Oops we broke it - Should have followed the process
After having great success last week, Real Progress - Engine works in two deployed environments, the organization really messed up this week. On Tues. one of the Core teams introduced a major change to our Data Protection subsystem, which may have passed their unit tests but they never tried to test it against the system. And that is ok because the "shred"/daily build process and smoke test did catch the problem. At this point, after it became apparent that a fix was not going to be easy, the organization should have imposed a code freeze and rolled back all the changes for Tues. back to Monday.
I suggested during our daily Scrum of Scrums on Tues afternoon that we needed to at least have a code freeze until the code was fixed. This was rejected because "we know what code is broken so it is ok if other code is checked in". So code, which could not be unit tested got checked in on top of a bad build.
Not for one day.
Not for two days.
But for three days.
We didn't impose a code freeze until Thurs. Of course by now there are hundreds of check ins by the undisciplined programmers who are not unit testing their code.
Fortunately my teams started complaining on the first day they couldn't continue developing because when they took latest code and reran their unit tests the unit tests failed. And they stopped developing code.
When I left on Friday afternoon, the system was still broken. Pretty sad really. We did end up taking a step back.