Image: H. Zell · CC BY-SA 3.0 · Wikimedia Commons
Technology·🏆 Entry of the Week

The Software Was Right and the Line Was Down

QU
Nov 20, 2021 · 3 min read · Edited

The dashboard said the press was running at ninety-four percent utilisation. I was standing next to the press. It was not running.

This was 2016, in a stamping plant outside Toledo, and the discrepancy turned out to have an explanation so ordinary it is almost not worth writing down: the sensor counted ram strokes, the press was stroking, and the die had cracked, so every stroke produced a scrapped part. Ninety-four percent utilisation, one hundred percent scrap. The number was accurate and the plant was losing about eleven thousand dollars an hour.

I have spent nineteen years around machines and about eight around the software that claims to describe them, and the thing I keep running into is not that the software is wrong. It is usually right about the thing it measures. The problem is that the thing it measures gets promoted to the thing that matters, and then the plant starts optimising the number instead of the work.

You can watch this happen in real time. Put a utilisation metric on a screen where a supervisor can see it, and within a month the supervisor is making decisions that raise utilisation. Some of those decisions are good. One of them, reliably, is running a marginal die longer than you should, because stopping to change it puts a visible dent in a number somebody upstairs looks at, and the scrap shows up in a different report on a different day with a different owner.

Nobody in that chain is behaving badly. Every individual is responding sensibly to what is in front of them. That is what makes it a design problem rather than a people problem, and it is why I get irritated by the framing where the shop floor is the source of resistance and the software is the source of progress. The floor is not resisting measurement. The floor has watched measurement be wrong in a specific, expensive way, several times, and has updated accordingly.

The engineers I worked with wanted to solve this by adding a scrap sensor. That is a reasonable instinct and it does help. But it does not touch the underlying thing, which is that every measurement system produces a model of the work, the model is always simpler than the work, and the gap between them is exactly where the money leaks. Add a scrap sensor and you get a better model with a different gap. There is no version of this where the model catches up.

What actually worked, in that plant, was smaller than a software change. The maintenance lead started walking the line twice a shift, which he had stopped doing when the dashboard arrived because the dashboard was supposed to have made it unnecessary. He caught the next cracked die in about four minutes, by sound. He could not tell you what he was listening for in a way you could put in a specification, which is precisely why nobody had thought to preserve it when the sensors went in.

I am not making the argument that the old way was better. I have no interest in that argument; I have used enough badly-run machines to know that unmeasured work hides its own failures, often for years, and that the nostalgia for the pre-instrumented shop floor is mostly held by people who did not have to work on one. The dashboard is a real improvement.

The argument is narrower. A measurement is a claim about what matters, and installing one silently retires whatever it replaced. Nobody decided to stop the walk-throughs. The decision was never made; it was implied by a screen, and it took eleven thousand dollars an hour and a cracked die to make it visible again.

The die is on a shelf in that plant's break room now, with the crack facing out. Somebody labelled it with the date and the utilisation figure. It is the only piece of instrumentation in the building that has never given a wrong reading.

Corrected: the original said the sensor counted cycles. It counted ram strokes, which on that press is not the same thing, and a man who ran it for nine years took the trouble to tell me so.

◉ 105 views

Comments

3
RJ
rjh_writes· Nov 21

This is the clearest statement of Goodhart's law I have read that never mentions Goodhart. The die on the shelf is a better teaching object than the paper.

AR
arjunmehta· Nov 22

Same failure, different industry. We had a dashboard that was accurate about an event nobody had defined the purpose of, and it took an intern eleven months to say so.

QU
quietmachinistAUTHOR· Nov 22

The intern detail is the tell in both our cases. She had not learned yet that the number belonged to somebody.