Matt Palmer discussed why water supplies are reliable and tries to generalise to IT.

http://www.hezmatt.org/~mpalmer/blog/general/water_tanks_reliability_and_redundancy.html

I think he misses some key points with water supply. Drinkable water falls from the sky (okay it picks up a few bacteria when it falls), so whilst gravity fed delivery is reliable, water quality needs a lot of active work to be maintained. Your supplier is constantly fiddling, yet still achieves these levels of reliability.

Societies organise themselves such that meddling with the water supply is a serious offence, and the provisioning is generally highly regulated.

But I think on provision of IT service he misses a point, many authoritative DNS providers have attained many years of reliable provision with minimal maintenance. In their case it is about protocol design, and simple precautions in deployment (multiple servers on multiple independent networks). But DNS is simple because it is effectively a broadcast technology, there is no meaningful interaction with the user.

I agree strongly with his point that layering in technology, redundant servers, and complex fail-over schemes, often isn't the right way to provision all services reliably. I've seen too many complex schemes fail in ways that simpler (nominally less redundant systems) simply could not fail.

In the case of web technologies, I've always been appalled at the "out of the box" performance of many web servers. But for real scalability the architecture is simply wrong, when more people access your content it should be cached wider, and thus accessed quicker. DNS does this, freenet does this, HTTP for static content would in the days when caching proxies were common, but we've now eaten away at the underlying technologies with cookies, encryption, and other variables, that a general purpose cache for HTTP usually only saves you ~50% of your bandwidth.

But again static content is the easy stuff to make redundant. As soon as you get to making update-able data redundant, there is a whole new ball game (think bank account balance). I don't think it permits of a magic bullet solution. There are inherent levels of complexity in the problem, which became very apparent when I first learnt about Oracle database replication. One can of course study any particular case to see how much doesn't have this inherent complexity to it. I note that withdrawing cash from cash machines doesn't always update the balance immediately. So presumably banks sometimes side step this issue, even in cases where you might initially think there is no sensible way around it.