Feedback as Regulation

Can we trust software? Bruce Schneier has recently been advocating software liability as a trust mechanism. In the latest issue of Crypto-Gram (December 2004), Geoff Kuenning contributes a contrary opinion.

If you've never read it, you should track down the CACM article on the history of steam boilers that appeared some time in the 80's. The brief summary is that after steam power was invented, there were lots of nasty boiler explosions. In the UK, the problem was dealt with by regulation. In the U.S., free-market advocates succeeded in arguing that liability law was sufficient: boiler makers would lose a few lawsuits and would then have an incentive to develop safer boilers.

The result was that boiler-related deaths dropped to near zero in the UK and continued at high rates for 20 more years in the U.S., until finally we broke down and regulated the industry.

The problem with liability as a feedback mechanism is that the negative feedback is strongly disconnected from the original action. Liability can help, but it's not at all clear to me that simply making MS liable for all the worms in the world would cause them to start making secure software.

From a systems engineering point of view, feedback may be regarded as simply a different mode of regulation. One of the first feedback control systems, invented by James Watt, has become known as the Governor.

Feedback corresponds to a kind of Network Regulation, producing Network Trust. As Kuenning points out, this doesn't work effectively when there are strong disconnects between cause and effect. Kuenning contrasts this with a kind of Authority Regulation, and argues that because the latter was more effective at dealing with boiler safety, it would therefore be more effective at dealing with software quality and trustworthiness.

However, Authority Regulation can also suffer strong disconnects between cause and effect. Is there really an Authority anywhere in the world that can keep Microsoft in check?

Drug Regulation

Dr David Graham, described by the FT as a safety official, told US senators last week that the FDA system was broken because of a conflict of interest.

According to the Financial Times, the FDA once enjoyed a high reputation. Few consumers challenged its judgement. The FT writes: "Such trust is one reason why the US public has been more willing than Europeans to accept foodstuffs containing genetically modified organisms."

So what has eroded this authority trust? The FT identifies two factors that may be relevant.

1. One can detect a regulatory cycle in drug approval. Testing standards became tougher after Thalidomide, were relaxed when AIDS stimulated demand for new and experimental drugs, and so on.
2. Division of responsibility and inconsistent handling of drugs at different stages of the innovation lifecycle. Approval follows one process; monitoring of approved drugs is done by a different department and follows a different process.
   

Oscillation and internal contradiction are both natural phenomena of complex systems, and tend to have a destabilizing effect on trust. One obvious response to this is to distrust complexity. But denial of complexity may merely create a false illusion of trust. Authentic trust may just have to accept oscillation and contradiction.

So whither the FDA - towards a restoration of authority, or towards an engagement with authenticity (if we can believe that)? Either way, a more intelligent and honest attitude to risk (as recommended by the FT) will be useful.

Source: FT Editorial No Pain-Free Option (November 20, 2004)