Sunday, October 26, 2008

3.5 Terabytes

Not that long ago, or so it seems, I built a computer with 8k of RAM memory.  Coupled with a little cassette player, I had all the data storage in the world.  I wrote my own little genealogy program in basic and my research life was great.

Skip forward 'a while'. 

After an all night session of scanning photos and documents, I named an image and hit save only to receive an out of space error.1tb_drive3   What?  Out of space?  How could a 750 gig drive that was less than a year old be out of space?  Looking at its properties, it really was full.  All that was on it was copies of my most used databases, photo and document images.  

Full.  In nine months it was full.  I'm sure you've noticed that you are filling up your hard drives quickly too.  It is so easy to accumulate large files with the scanning hardware and related software that we have today.

My data is backed up regularly and so of course additional hard drives are needed for that task.   Backing up to Mozy is also part of the mix, but I like to have my data on site for instant access too.

With the out of space error, I ordered another 1TB drive.  It arrived a few days later I and put it to use.  Here we are 1 month later and I just noticed that it is about 40% full.  Factor in backing up the new data and I'm almost out of space again. 

Where will it end? 

My off site plan of sending DVD's containing my genealogy data to our children in rotating order isn't working with that much data.  The DVD's just don't hold enough data to make that feasible any more.  Maybe I'll start mailing 1TB or larger hard drives to them that hold just the 'important' files.

I keep 16MB memory sticks full of just the direct line data and some photos on them in our emergency kits.  I keep them updated monthly so if all of our homes fall down in an earthquake or other disaster, hopefully that much data will survive.  It's not much, but it is a starter seed.

What are your data storage plans?   Do you keep copies of your data in both the format used by your genealogical and other software today as well as it being in gedcom format and lossless formats of your images?   Do you keep a copy of the genealogy and graphics software you use on your backups along with the program keys and related passwords?  

Additionally, formats will change quickly in the future.  We need to plan for ways to restore and read our old data formats if isn't part our regular backup routine.  I've helped a lot of folks retrieve data created with old software versions.  Fortunately, we've found old versions of software that would still install on today's operating systems and were able to update the data to a current standard.  Have you thought about this problem?

So, here I am, thinking about getting more storage for my data.  Its size has grown to an 'outrageous' number and no doubt will continue to grow.  It is time to do some serious thinking about my storage and backup plan for the next five years.  Is it time to go back to tape backup?  Do I need to build yet another storage server with raid 5 or better implementation or do I just load up my Mozy account or get one from a similar company and live with the time required to encrypt and incrementally upload my data on a daily basis?

I hope you are thinking about this issue in your own backup and storage planning.

3.5TB headed for 4.5TB.   How about you?


Sean Sexton said...

The sheer amount of data that we store (and should preserve) is staggering. When it was just raw family history data, we were talking in KB. But now with photos and--eventually--HD videos, we're looking at many, many TB of data that we should perserve for future generations.

My big fear is that family historians will collect and gather, but not preserve. We need to think more about how to ensure that a) data is never lost and b) data is passed on to our descendants.

Lots of food for thought.. Thanks for the article.

Here's a recent post I wrote, talking about a strategy for backing up your family history data: 10 Essential Steps to Protecting Your Family History Data