The assessors came for three days in October. The evidence pack ran to 412 documents, 61 of them written that fortnight. The verdict was level three of five, with a roadmap to level four. The certificate hangs by the lift.
In November a release went out without a rollback plan, the way releases had always gone out.
Policies, minutes, screenshots
Policies. Process descriptions. Minutes. Review schedules. Screenshots of the pipeline configuration. Nine interviews. None of it was untrue.
Nothing in the three days watched a decision being made.
The score for each process came from six questions on a five-point scale. Whether the thing exists. Whether it is written down. Whether it is reviewed. Whether the review is scheduled. Whether the schedule is met. Whether the whole is signed off.
The assessors asked nine people to describe the change process. Seven described the process as written. Two described what they do. The difference was recorded as inconsistent adoption, and the score for that process came down half a point.
The team that had never needed it
The highest-scoring evidence in the pack was the rollback procedure of a team that had never used it. Current, reviewed on schedule, signed off, one version, never handled.
Another team had used its rollback procedure three times and simplified it after each one. It could produce three versions, and approval records for one of them. It scored lower.
Evidence keeps better when nothing touches it. Use marks it.
Which document answered which question
The second assessment took three weeks instead of six. A coordinator had been appointed for three days a fortnight, and knew which document answered which question. Two of the templates were reused unchanged.
Before the second assessment the people due to be interviewed were sent a briefing note on how the change process is documented.
At the first assessment the question was how deployment approval works. At the third it was where the deployment approval document is.
The certificate by the lift was replaced with the new one. The old one is in the pack.
Finding 17
The evidence request had 140 items. The pack answered 130 of them without anyone opening a system. Two findings were raised, one closed by adding a paragraph to the change policy and one by scheduling a quarterly review of it.
The report recorded a mature and well-documented control environment.
The quarterly review produces minutes, and the minutes are evidence for the next assessment. The pack ran to 412 documents in October and stands at 470.
In the steering meeting nobody asked whether a release could be rolled back. They asked whether finding 17 was closed.
Everything the assessment had to work with was an interview or a document. Both return something simpler than the process, on Metier’s reckoning, and in most situations nobody has an overview of the real one. There are no verified independent maturity levels, which does not make the measuring pointless: without metrics any change is disputable.
The work of producing evidence now has a description, a schedule, an owner and a signature. It would score well.