In Praise of Silos

I’m not sure who first started to use the word “silo” as a term of abuse in the computing world, but it seems to be the go-to word for any systems that limits access to its data in any way that the author deems inappropriate. I’m not sure if whoever coined the term was thinking of grain silos or missile silos — perhaps the latter, which would fit the usage better.

Grain Elevator

Silos store and organize grain for distribution.

Anyway, silos are getting a bad rap. A grain silo is a container designed specifically to collect grain produced in a local market, sort it, classify it, and store it for shipment to global markets. Putting data in a silo is portrayed as a means of keeping other people from getting to the data. But putting grain in a silo is the very opposite of preventing other people from getting to the grain; the silo exists for the sole purpose of shipping grain efficiently to other people.

Imagine if we tore down all the silos and forced every farm to truck its grain directly to the shipping terminal, bypassing the silos and the railways. Costs would skyrocket and chaos and delays would be introduced all along the line. Is this really how we want to manage our content?

 

Rapunzel

Content is commonly stored in a guarded tower when it should be in an open silo.

A better analogy for the truly isolated pockets of content would perhaps be Rapunzel’s tower. The tower has neither door nor window and the only way to get to the data inside is to charm someone on the inside into letting down their hair. That is certainly the situation that you find with much corporate data — it is locked up in a private store, and only a special dispensation from within will gain you access.

The solution to this problem, though, is precisely to convert Rapunzel’s tower into a silo. Cut doors and windows and organize and classify what lies within so as to make it available for shipment to other people who need it. A silo is not the problem; it is the solution.

Okay, so the silo metaphor is flawed. Is that a big deal? Actually, yes. A flawed metaphor can shape the way we think about things, can change how we understand our problems, and what we look for in solutions.

The condemnation of silos is by no means disinterested. If someone is trying to sell you a large expensive enterprise system, you can be sure that the word “silos” will be often on their lips. “Your data is shut up in silos”, they will say, “and the only way to make it available is to tear down all the silos and put all the data into one big enterprise-wide management solution.”

In practice, of course, such systems are grossly expensive to purchase and install, hard to maintain, and often not well suited to the needs of individual groups. Cost overruns, schedule delays, and integration issues are rampant, and groups often resort to rebuilding their Rapunzel towers as a refuge in which they can actually get some work done and get their deliverables out the door. Such towers are often well disguised and/or heavily fortified specifically to keep out the soul-sucking forces of central control. Because there is a corporate mandate that all sharing must be done through the central system, they don’t share at all.

Is there another way? Yes, and the answer (unsurprisingly, by now, I hope) is silos. Rather than forcing everyone to use a single central system with a single central interface and data model, allow each group to set up their own system that suits their individual needs, but require that they provide an interface that allows others in the organization to query and retrieve their data as required.

Are there examples of a system of this type working well in a large organization? Actually there are many, but there is one that most people will be familiar with. It’s called the Web, and the organization is serves is planet Earth.

The Web is a network of independent nodes.

The Web has no hub. There is no central black hole around which all the stars revolve. It is a vast collection of individual nodes that communicate with each other using a set of agreed protocols and message formats.

Is the Web perfect? Certainly not. But no centralized system is perfect either, and the Web saw off its centralized competitors — AOL and Compuserve — in fairly short order. The Web continues to grow and to improve rapidly. But the great thing about how the Web grows and improves is that it never stops working. Nobody has ever taken the Web off line for a version upgrade. Individual nodes of the Web can upgrade themselves, introduce support for new protocols and message formats, and the Web itself just keeps rolling along.

The great thing about the web model is that it lets each site owner organize their work in whatever way makes the most sense for themselves. This lets each group optimize their own process, on their own schedule, while still making their data available for the rest of the world to use according to the agreed protocols.

Many corporate systems do, of course, organize themselves on the web model, which is, precisely, a network of independent well-managed silos. More and more, organizations are coming to appreciate the power and flexibility of an architecture of many small pieces loosely joined. If your content is currently living in a fortified tower, the optimal solution may not be to tear the tower down in favor of a single central fortress, be to renovate the local tower into a local well-managed silo.

 

 

 

 

Tags: , , , , , , ,

{ 2 comments to read ... please submit one more! }

  1. I thought you would like to read, if you haven’t already, the MoReq2010. It is a model of requirements for building record systems, describing modular end-to-end features. It goes along your claim for standard-ly interfaced silos as the means to address content transparency among organizations.

  2. Missed the link. Here you go: http://moreq2010.eu/

{ 0 Pingbacks/Trackbacks }