Morale and Complex Malfunctions

Commenter Boonton has on a few occaisons mused about complex industrial accidents and the avoidence of the same.

Complex project development, in a book which came out in the 80s (Have Fun At Work, by Mr Livingstone) was an interesting read. The main thesis of the book was that complex projects (those are too large basically to fit in one smart persons brain … and he gave specific concrete ways to recognize those projects) fail. They all fail (or at the best have horrible delays and massive cost overruns). Much of the book devoted itself to orienting tech/engineer personel to recognize if your project was one of those which would fail and how to prevent that from career or psychic injury to self. As a sidelight he noted the only way that complex projects succeed. Complex projects succeed if heirarchical information pathways are removed and replaced with a model in which everyone can talk (and does talk) to everone. The cannonical such project is the Lockheed Skunkworks, which developed the SR-71, the U-2, and stealth combat aircraft. In their working environment, aerodynamicists and systems engineers sat next to draftsmen and machinists. “Can this …?” questions didn’t filter up and down the chain but you would ask the guy who might know the answer directly.

Big systems with complex working parts are put in place all over the world. Refineries, airplans, chemical plants, nuclear power plants and so on are all complex working systems. One way in which one might approach minimizing the occurance of complex accidents is to follow the Kelly Johnson/Skunkworks approach and shift it from project development to ongoing system operations. Why isn’t this done?

One reasons might be tied to morale. The Skunkworks team was a high morale operation. They had an impossible (basically) cutting edge project. They worked rediculous hours because of their excitement and the demands of the project and the basic urge human urge for success and to win, defined in this case as completion of the project, to scale that technical mountain. How can this translate to a multi-decade task of keeping equipment running safely, a far more mundane and routine task? If one identifies a clear difference in the two tasks as one of morale. High morale is essential for the operation of a non-heirarchical task/team project. High morale might also be an essential telling point in the operation of a long term operational facillity if one were to attempt to shift it to a more skunkworks-like approach to management. You can’t do that without high morale.

Ultimately government “regulation” of industrial workplace might be better served not trying to pretend it knows better how to drill offshore, run nuclear plants, and so on. It can on the other hand, have a better shot a spotting any number of ways in which workplaces are poisoned by poor morale and other working conditions conducive to failure (reckless risk taking has its own signature on morale). The point is, inspectors might be better served watching dynamics of workplace (social) chemistry and less on technical questions which they have, likely, less (or captive) expertise (not to speak of other agenda).  

Leave a Reply

Your email address will not be published. Required fields are marked *

8 comments

  1. Boonton says:

    Speaking of complex systems, I entirely missed this post. Why? Even though you mentioned my name and all! Well I’m so used to back and forth comments and seeing only you’re daily links of the day it turns out that every time I hit your blog I’ll miss any new post you make if it isn’t on top. I only expect to see ‘links of the day’ and nothing else. Then I look at your conversation section to see who posted new comments….if I’m on top then in my mind I’ve covered all that’s new on Pseudo-polymath and I’m off to other things.

    This ties in with Normal Accidents by Charles Perrow, an old book from the early 80’s but remarkably relevant. The difference between what he looks at and what you’re looking is the difference between complex systems and complex projects. Skunkworks undertakes complex projects, but a complex system is something that you run after a complex project has created it.

    He uses two variables to measure complext systems, Interactions and Coupling. Interactions range from linear to complex. Examples of linear systems include dams, rail transport, assembly lines, and single goal agencies (post office, DMV). Complex interactions include things like mutli-goal agencies (child welfare department, Homeland Security), universities, aircraft etc. Coupling refers to the slack or buffer between two items in the system. If one thing gives what happens to the other thing? Can it be controlled. A dam, for example, is highly linear (water on one side, dam, dry land on the other or stream for the water allowed thru the dam) but tightly coupled. If the gate fails the water immediately pours out. But you can have loose coupling in a complex system. For example a welfare agency is highly complex in its interactions, not linear, but loosely coupled. If the person who audits the food stamp applications falls sick for two weeks the system doesn’t collapse. A university is likewise complex but loosely coupled. If a class you need to complete your major is filled up this semester, you can usually take other classes and adjust your plans so you can still graduate on plan. A military operation is less coupled….if Pakistan refuses to let a supply convoy thru to Afghanistan, we can use more expensive air transport over Russia, Iraq, Turkey etc. (But not as decoupled as other things, if a massive storm hit on D-Day the entire invasion could have failed as a result).

    It so happens that nuclear plants give you the worst of both variables. They are tighly coupled and highly complex. While individual failures can be assigned ’causes’ (such as operator error, poor morale, not following manufacturer’s instructions….which I think you said was done at Japan’s reactor), the true cause is the nature of a large nuclear plant. Being both non-linear and tightly coupled they will fail and improvements in technology, training, management and regulatory systems are unlikely to change that. An interesting point while reading the book was how much the past sounds like the present. Reading about Three Mile Island and what almost happened as well as other less publicized incidents one is amazed at how much Russia and Japan’s accidents were essentially predicted by him. One idea might be to follow your suggestion for smaller plants, essentially reducing the complexity and coupling (i.e. if the water fails, the reactor won’t melt). I suspect, though, that they will be less economically viable.

  2. Boonton says:

    http://onjava.com/onjava/2002/03/27/graphics/couple.gif has a good chart the illustrates different systems and Perrow’s Coupling/Complexity matrix.

  3. Mark says:

    Boonton,
    Intrinsically safe reactors need not be small. It’s a different design criteria. Chernobyl vs other accidents and thinking they are similar ignore the enrichment difference and the use of graphite as moderator, both of which are horrible design decisions which are why Chernobyl was many orders of magnitude worse as an accident (and why Japan’s incident had almost zero environmental impact). Without those two matters the Chernobyl accident would probably have been unnoticed in the West (due to the Soviet control of internal media).

    I thought I remarked on the difference between complex projects and the long term running of complex systems. My suggestion is that you need some of the same features to run a complex system as those which you need to run a complex project (high levels of intercommunication and ease of talking across specialty).

    However, your complexity vs coupling means that part of system design is part of system design might do better to design a power plant that is less tightly coupled by designing around single points of failure and allowing for systems to have multiple dependency paths.

    You offer no remark on the detection of failing in the maintenance of complex systems as being located in morale.

  4. Boonton says:

    Question then, can you design a large scale reactor to run at full power with total loss of coolant for an indefinate period of time without melting?

    I’m progressing thru the book slowly but loosening the coupling seems to be a viable way to make nuclear power viable from a safety POV. But the issue isn’t just the profile but whose put at risk. There are other systems that exhibit tight coupling with non-linear interaction (maritime shipping, surprisingly, seems to be one) but their risks are mostly self contained to those directly engaged in them. (although some accidents do impact innocent bystanders, transporting large amounts of liquified natural gas seems to be a potential source of a lot of trouble).

    One coupling solution might be to establish centralized repositories for spent fuel rods. From what I’m reading, nuclear plants have to manage at least 3 ‘pools’ or pots of water:

    1. Water inside the reactor itself, this water doesn’t leave and is radioactive and very hot.

    2. Water outside the reactor, this gets turned to steam to produce electric.

    3. Water in the holding tanks for spent fuel. This water needs to be circulated otherwise it will boil off and the rods can potentially catch fire.

    Water seems to be the crux of the planets complexity problems. Water = plumbing and a huge amount of plumbing which makes for lots of complications. If nothing else #3 should be removed from the equation.

  5. Mark says:

    Boonton,
    Where was a time when I had a semi-longish essay posted nightly (or at least in advance of each weekday morning). I’m trying to get back to that. I’m leaning toward dropping the course I’m taking right now, which might give me the time to d that. We’ll see. Plans and manly mice and all that.

  6. Mark says:

    Boonton,
    The design criteria for a Gen IV reactor is that one be able to shut off the cooling entirely and remove the contro rods entirely (putting the reactor at full power) and it will stabilize in temperature with no fuel movement (that is the fuel and core will remain stable). Since this is the design criteria going in, and it there are a half-dozen designs on the table, I think the answer to your question is a strong affirmative.

    One thing to note, on water. Water is the main coolant in today’s reactors largely because of plant reactors in use are scaled up versions of the reactor design developed by the Navy for submarines and carriers, for which water cooling makes sense. Helium and sodium as the coolant are two the prominent reactor cooling media for Gen IV reactor. Those primary coolants (in the case of sodium) would then pass through heat exchangers to produce the super-heated steam to drive turbines. Sodium has an interesting advantage in that it remains liquid and as there is no need to handle high pressures.

  7. Boonton says:

    Hellium seems problematic to me since its hard to hold onto it if you get a leak in the system. I suppose, though, sodium suffers from the same issue.

  8. Mark says:

    Boonton,
    Helium’s advantage over water is that water becomes more and more reactive (chemically) as it heats. Helium does not, being a nobel gas. Sodium, if it leaks, becomes a solid metal. Not so big a deal.