This story is the first part in a two-part series on record retention.

For boys in my day, it usually began with a cigar box—a place to keep the really important things: marbles and bottle caps, a particularly colorful leaf or the exoskeleton of an unusual insect. I think that most of us have, at some level, a compulsion to save things. Some of the things that we save are important—I have a collection of newspaper clippings about my father, without which I would know very little about him. However, many of the things we save are not important—I seem to collect owner's manuals and warranty information for products I no longer own. (I am going to sort through that stuff one of these days.)

This need to save things is generally manageable in our personal lives, but when we bring it to work, it can become a significant problem. The number of things that I can personally save is limited by the size of my two file cabinets and the hard drives on my personal computer. At work, there are lots more people and much more space, both physical and virtual. My first real job was as an electronics technician for Digital Equipment Corp., and I can still remember walls of file cabinets with product specifications. At Intel Corp., we had individual file cabinets, department file cabinets and archive file cabinets—records everywhere.

Perhaps counter-intuitively, the problem has grown worse as the ratio of information stored on computers to information stored on paper increases. Science magazine reported earlier this year that the percentage of information stored on computers rose from 0.8 percent in 1986 to 95 percent in 2007. I'd lay odds that the number is still growing. Computer data, however, has been managed much differently than paper records. In the paper world, companies faced a challenged measured in square feet. You can only keep so many file cabinets in a given space. Sooner or later, you've got to box up those records that are not routinely accessed and move them to lower-cost storage. The challenge of organizing those records evolved into the discipline of records management.

Information technology, on the other hand, was not about real estate. In fact, the physical space required to store electronic records grew exponentially smaller and less expensive as time went by. “Storage is cheap,” they said, so, true to our nature, we kept everything. The IT imperatives were access and uptime, and the users were left to decide how to organize the data. And organize we did. Unfortunately, we all did it a little differently because, unlike records management professionals, we were making it up as we went. Eventually we realized that cheap memory is a relative thing—one gigabyte is pretty cheap, but multiple petabytes translates into real money. When IT departments sought to rid the company of some of those terabytes of data, however, the lawyers went apoplectic. Justifiably concerned about inadvertently deleting information that may be subject to an existing preservation obligation, and without any effective way of determining whether such information even exists, lawyers generally prohibited its destruction.

I am not slighting the lawyers, mind you. I am one, after all, and we have good reason to be concerned. The external demands to preserve information are many, and come primarily from two sources: litigation and regulation. My very first recollection of a requirement that records of any type be retained for any period of time was hearing that I should keep my tax records for seven years. That was more than 30 years ago, and while it turns out to be a bit of a myth, it demonstrates how pervasively the government is interested in how long certain record types are kept. There are literally thousands of federal regulations requiring records to be maintained for fixed periods of time and, often, in prescribed fashions. State regulations simply compound the situation. Though the number of retention requirements for any particular company are finite, they are nevertheless complex, difficult to identify and even more difficult to apply. The end result is that many companies have a stated or de facto policy of keeping everything … forever.

Therefore, most companies find themselves spending enormous amounts of money (check with accounting – it's true) to store massive amounts of data (check with IT – true again) in a wide variety of repositories, from legacy systems and backup tapes to active file servers and e-mail archives. And most of that money is ill-spent.

The IDC Digital Universe study in 2010 indicates that only 25 percent of electronically stored information is unique – the rest are duplicates. Another IDC study estimates that, by 2012, about 35 percent of corporate data will be subject to some sort of compliance obligation. That means that about 65 percent will not. In other words, your company is spending a lot of money to store information that is duplicative and not required to be maintained, not to mention the substantial risk that unneeded information represents.

I am sorry to be the bearer of bad tidings, but, in my next article, I will talk about ways that companies can effectively address these challenges.