[Evildave]

I'd like to spend a few minutes on the most crucial piece of security for your PC: BACKUP YOUR HARD DRIVE!

Every time I hear about some worker or 'hacker' who managed to destroy 'millions of dollars' worth of work on a company network, it really bugs me because the ultimate blame lies with the management and system administrators who didn't back that valuable data up. Backups protect you from more than 'hackers' and 'vengeful employees'. it protects you from all manner of logical meltdowns, and even from your hard drive failing.

Having good backup practices will turn what would otherwise be a total calamity into a minor annoyance. If Windows melts down (or you screw up a script in Linux), you'll be 20 minutes away from having a stable and usable system again, rather than suffering through days of installation and configuration.

The first step is to buy a big USB2 drive just for this purpose. It's fast and super-convenient. Unlike tape or other 'cheap' backup media, it's HIGHLY reliable, has awesome capacity, and can be reused again and again more or less 'forever'. In general a huge USB hard disk will cost less than the equivalent pile of tapes or optical media to do the same thing, and doesn't need the media to be 'swapped'.

This drive should normally be put away safe in a cool, dry environment and not connected to the computer or power outlet at all. If something bad happens in the room with the computer, like a fire, the backup drive should be at least somewhat less likely to suffer the same fate as the computer. When you are ready to backup (or need to restore something) dig it out and plug it in. As a minimum, it should be powered off and completely unplugged, even if it's connected to a UPS. A UPS or surge suppressor can fail, too.

Split your computer's hard drive into two partitions, the 'Boot' and the 'Work' partition. The reason for this is that if you need to 'restore' your OS, you don't usually want to overwrite work. If you need to restore work, you don't usually want to overwrite the OS. Generally the 'boot' partition should be as small as you can get away with. In WIndows, this is pretty much moving most of the junk in your 'Documents and Settings' or 'Users' folder, such as 'My Documents' and 'My Pictures' to folders to the other partition. In Linux, it's more or less the same thing with symlinks in your home folder pointing at folders in a different logical or physical drive that's always mounted.

Windows Vista has a 'Resize Partition' built in, but don't be surprised if it's completely ineffective, or won't shrink the Windows partition by as much as it SHOULD. If you have Vista, this is about all you can use to 'safely' shrink the Vista partition (for now), short of deleting the Vista parition and starting over, and if you're like a lot of people, you'll start over with XP if you aren't a masochist. If you have XP or earlier, Partition Magic or a 'GParted Live' boot CD will shrink all normal Windows partitions to make room for a second partition. You could also add another physical drive, which will give you a 'poor man's RAID' boost in performance as a single drive will need to seek a lot between OS and program files and work data. I recommend against RAID 0 as many motherboards implement it. Most of the time, the OS won't boot off it without all kinds of magical configuration, and you more than double the likelihood of failure that takes the system down.

While you could download various 'tools' to manage file backups, I find just writing a script or batch to do it is easy, and that those tools are usually cumbersome and unreliable, and in the Windows world are often save data in proprietary formats that you need THAT particular tool in order to read, usually cripple-ware, and often come with a spyware installer and/or they want to run all the time. Norton Ghost is the worst.

A batch or script will generally be a list of copy commands to do, one after the other of the pertinent directory trees in your 'Work' partition to be copied your external USB drive. At the bottom of this post are a few examples.
* rsync -av --delete -l srcdir dstdir
* robocopy /MIR /xo /R:0 /A-:R srcdir dstdir
* 'xcopy /h /e /r /d /i /c /k /y srcdir dstdir

All the extra flags on those command line tools are to make them copy 'only newer or changed' files and/or to ignore some errors so that a large copy will complete even if one or two files couldn't be accessed. Normally XCOPY or ROBOCOPY will choke if the first file has an error and not back anything up. Since the copies only include 'changed' contents, you can quickly run it a second time if something had a 'lock' on a file. After the lengthy initial backup, subsequent routine work backups will take from seconds to minutes, according to what changed since the last time. Of the three, the xcopy invocation is the worst (or best) because it doesn't remove files that you deleted, so the backup will accumulate 'hair' in the form of previous versions of renamed or deleted files that will have to be picked over during a restore. I find a combination of mirror and 'hairy' backups is useful, in that sometimes you delete something you suddenly realize you wanted a few months later, and it'll still be hanging around... but when you need to restore a clean work copy of your work tree, you want all the 'old' stuff gone.

A restore is usually selective. You can just navigate there with the 'Explorer' or other GUI tools and drag/drop the backup files or folders back to the drive they're missing from, or you can use a version of the backup batch with dstdir/srcdir reversed, and parts you want to leave alone commented out.

Consider using version control, like SVN too. You can get 'TortoiseSVN' with a Windows SVN server in a single setup package and install it. Once you have the repository defined, you can review history or revert versions of your work back to older versions, and the database supports binary data well. The repository will need to be backed up, unless it's on a remote server that's backed up for you.
http://subversion.tigris.org/

VERY IMPORTANT: Once you manage your own backups, TURN OFF 'Windows Restore' on your work partition(s). It'll even be 'On' by default on your backup drive, eating lots of space. If you do 'Windows Restore' to fix a minor brain-fart in Windows, it will none-too-selectively crap all over your work drives as well with versions of files from whenever that restore point was made.

Use drive imaging software to copy the 'Boot' partition to a file on the big USB2 drive. Do this whenever you've made a 'big change' to the system, such as installing and configuring a major software package, and find it's still stable. Keep a couple of versions of this backup. If possible, include a 'clean' one of the system when it was new, before you ever did anything to it. This will come in handy if you ever need to sell or give the machine to somebody else.

You can use FREE software to resize and backup partitions. Try a CloneZilla / System Rescue CD ISO. Clonezilla even has a 'server' mode so you can clone one drive image to an unlimited number of computers on a network if you need to. Very handy if you have a lot of computers to administrate, and a VERY expensive capability to buy from Symantec. It'll have GParted on the 'Rescue CD' half as well. So you get a combination of 'Norton Ghost' and a version of something just like 'Partition Magic' that won't barf if there's a Vista partition present on the machine. Nothing needs to be installed on the machine at all. It boots off the CD. It handles NTFS just fine and reverts to a simple 'full partition' copy if the format is unsupported (i.e. it zips the whole partition contents and settings up and sticks it in a file without regard to used versus free data).
http://clonezilla.sourceforge.net/

If you have SENSITIVE files, like tax records, software regcodes, passwords, etc., move them onto a Flash drive of some sort. Back that up onto the other USB drive occasionally, too. Only plug the Flash drive in when you're going to use it. Normally leave it somewhere 'safe', and away from the computer. Malware can't upload what isn't there.

DO NOT store passwords in your web browser, nor allow most kinds of software to 'remember' passwords 'for you'. Type them in. It's slightly less convenient each time you use the system, but much more convenient than having all of your passwords automatically compromised. If you need to 'write them down', type them into a text file on that Flash drive. Also, most people who allow a browser to 'remember' passwords for them are at a complete loss if a crash of any sort takes that password cache with it.

Anyways, as long as you spend a couple of hours every few months backing everything up, then no matter why your Windows install turns to MUSH, you can use the backed up partition image to overwrite your OS completely, even restoring any 'activation' that has been broken. If your hard drive FAILS, you will still have your data, and upon replacing it, be able to boot your computer with all your pre-configured applications 20 minutes after you replace it.

Here's my old Windows Backup.bat. You can get 'robocopy' from the Microsoft web site. I put in an extra prompt for whether to do the VMWare folder, as just running a virtual machine will alter its virtual drive, and that makes for big, fat backups all the time.
{noformat}
set mflags=/MIR /xo /R:0 /A-:R
set aflags=/E /xo /R:0 /A-:R
set dst=K:
set mirror=%dst%BackupMirror
set archive=%dst%BackupArchive
set recover=%dst%BackupRecovery

@echo Recovery Backups
robocopy %aflags% "C:Config" %recover%Config
robocopy %mflags% "C:Documents and Settings" "%recover%Documents and Settings"

@echo Mirror Backups
robocopy %mflags% "D:My Documents" "%mirror%My Documents" /XD .svn
robocopy %mflags% "D:Work" %mirror%Work /XD .svn
robocopy %mflags% "D:svn" %mirror%svn

@echo Mirror Other Junk
robocopy %mflags% "D:Download" %mirror%Download
robocopy %mflags% "D:Movies" %mirror%Movies
robocopy %mflags% "D:Music" %mirror%Music

@echo Archive Backups
robocopy %aflags% "D:My Documents" "%archive%My Documents" /XD .svn
robocopy %aflags% "D:Work" %archive%Work /XD .svnge

@echo Mirror VMWare...
pause
robocopy %mflags% "D:VMWare" "%mirror%VMWare"
{noformat}

Here's my Linux Backup script. It uses 'zip' to keep the size of certain trees down.
{noformat}
#!/bin/bash
echo
echo Backup script!
echo
flags='-av --delete -l'
mirror='/media/Backup/Mirror'
read -s -n1 -p "Continue with backup?" keypress
if ||
then
echo
echo 'Backup work tree'
pushd /home/dave
pushd work
cleaner
popd
find -H work -type f | sed -n -e '//./!p' | sed s/(.*)/"1"/ | xargs zip -u $mirror/work.zip
echo
echo 'Backup Documents'
find -H Documents -type f | sed -n -e '//./!p' | sed s/(.*)/"1"/ | xargs zip -u $mirror/docs.zip
echo
echo 'Backup Home Tree'
zip -y -r -u $mirror/dave.zip *
echo
echo 'Backup Download tree'
rsync $flags /work/Download $mirror
echo
echo 'Backup Music tree'
rsync $flags /work/Music $mirror
echo
echo 'Backup ISO tree'
rsync $flags /work/huge/ISO $mirror
echo
echo "Backup Ancient (archive) tree"
rsync $flags /work/huge/Ancient $mirror
echo
read -s -n1 -p "Continue with VMWare backup?" keypress
if ||
then
echo
echo Backup VMWare tree
rsync $flags /work/vmware $mirror
fi
popd
fi
echo
{noformat}

Finally here's a subset of my Linux backup for a Flash drive. It zips up the data then moves it to the Flash drive, so it's faster and doesn't 'wear out' the bits.
{noformat}
#!/bin/bash
echo
echo SD Mirror script!
echo
mirror='/media/MIRROR/Mirror'
read -s -n1 -p "Continue with backup?" keypress
if ||
then
if [ ! -d $mirror ]
then
mkdir $mirror
fi
pushd /home/dave
echo
echo 'Backup home/work tree'
find -H work -type f | sed -n -e '//./!p' | sed s/(.*)/"1"/ | xargs zip work.zip
mv -v work.zip $mirror/work.zip
echo
echo 'Backup Documents'
find -H Documents -type f | sed -n -e '//./!p' | sed s/(.*)/"1"/ | xargs zip docs.zip
mv -v docs.zip $mirror/docs.zip
popd
fi
{noformat}

[smax013]

Very useful information.



I would add that while I agree that using a hard drive for routine backups, it is also a good idea to do a backup to an other medium such as tape or DVD. While DVDs can be a real pain in the rear, they are generally more robust than hard drives. The same problems that can happen to your primary drive can happen to a secondary/backup drive. Thus, while I tend to recommend typical and frequent backups to be done to a hard drive due to easy and speed, I also usually also recommend doing an occasional backup to DVD (or tape).



And to REALLY be on the safe side, it can be a good idea to use two hard drives for backups...and alternate between which drive you use. This way if somehow, someway a backup gets screwed up at the same time your computer does (yes, it is very improbable, but still possible), you will still have another back. Many IT groups in enterprise situations will actually run several backups that they alternate through...and typically keep some/many of them offsite when not actively being used to backup the system.

[mphenterprises]

Hi EvilDave. Thank you for your in-depth Discussion. If you could please, can you convert this Discussion into a Document? This way, your information will be maintained in our Document database. Click on the "Convert thread to document" link under the "Actions" section. Once it is approved by Kellie, the Community Manager, it will appear for the public to see.

[Adama]

That's great information, EDave. I think it's very timely and it would be good for people to read it.

[lilxkid24]

lol im one of those lazy person to backup and something bad happened 3 times causing me to reformat and ended up losing data which i was lazy to backup. Now i just keep stuff i need on a flashdrive

[Evildave]

Some time in 1993 or so I had moved all of my data to an obscenely expensive 1GB SCSI drive in my Gateway EISA machine. That drive failed a few weeks later and turned out to be the only copy left of several years of previous work. Since then, I took backups seriously.

While optical backups of data are good (assuming it FITS ON ONE DISK), I would recommend a flash drive instead, and I would ABSOLUTELY recommend against tape. Tape stretches. You can record and verify the tape (and it will take hours upon hours to do so), and when it comes time to restore it, it'll be 'blank', at least as far as the restore software can tell. Tape is very expensive, must be stored hyper-carefully, and still it's as stable as a drunk on a pogo stick. You can only use it a few times at best, and often as not, not even once. BITTER experience teaches me so. While I'm sure there are data centers that have better that 50/50 rate of recovery from tape, they also use tapes and tape drives that are hideously expensive, and store them in a climate-controlled vault.

The USB drive is to be connected ONLY when backing up and restoring. Powering it on and shutting it off every day with the computer (and in the case of Windows, mounting it every day) would be just dumb.

Your backup hard drive may have a MTBF of a hundreds of thousands, even millions of hours. Even if you religiously use it every single week, you're never even going to get past the leading edge of the 'bathtub curve' for hardware failure, so it's important to burn it in before relying on it, because most makers of USB drives just plug hard disk A into enclosure B, put in the screws and stuff it in a box. Do a full format and beat on it with repeated copies/compares or thorough invocations of fsck/scandisk to make sure it's a 'GO' before trusting it. Once it has a couple of days of continuous operation without failure, it's probably going to last for years and years. You shouldn't try to 'stretch it', though. It's perfectly OK to buy a new backup drive every couple of years and 'retire' (format and give away) an old backup drive. They all start dropping sectors eventually, but it's probably 'good enough' to give a friend for backups and file transfers, since chances are they'll never do either unless you show up and do it when you 'fix' some bonehead problem with their PC.

While IT staff in big companies routinely use multiple hard drives for backups, you at home probably won't. This is why the one backup (the Windows version) is a combination of mirrors and recursive copies. The mirrors make a quick 'restore', while the recursive copies that don't remove excess files allow recovery of 'accidents'. It's a decent middle ground. I think I only used the recursive-copy backup to restore something maybe twice in ten years, so it's absent from the more recent Linux backup script.

I personally use a hard disk, a 'private data' flash drive, a 'limited' backup flash drive that I keep on the desk, and a few different SVN repositories for my software development use. Sourceforge for the open source stuff, Assembla for the closed-source stuff, and whatever a contract employer has on their end for that sort of work. If you do 'real work' on your PC, then once you discover what version control is about, you will never go without it.

If you want to take it further, you can burn a CD or DVD of your most critical data and keep it at somebody else's house (someone you TRUST). You'll still have to destroy the media. After all, who wants to pay 10x as much for 'rewritable' media only sit for most of an hour 'formatting' it on top of swapping multiple disks into the drive as they fill up? Never mind the waste.

You can buy several USB drives to swap a hard drive on/off-site like that so you have the most recent close at hand, and the least recent at Grandma's... or you can just store the one drive over there. Now even if your house burns to the ground and then the embers get sucked up in a tornado before the remains are buried in a mudslide, you can get a semi-recent version of your data back. Of course there's still a risk you'll get into a wreck or the big bad wolf will startle you and make you drop the drive on the way to/from Grandma's house when swapping backup drives.

If you go to Pricewatch and shop the drives, chances are you can find a USB drive several times larger than the one that shipped in your PC for $100, mostly because PC manufacturers are stingy bastards. Especially if your PC is portable, and the USB drive you buy is not. Well, technically, they're all 'portable', since any 3.5" one will be in a box not much bigger than the drive and have a 'wall wart' and cable that's all pretty easy to wrap in a bundle and carry around. You can get a terabyte (trillion byte) drive for $200. You can also just buy 'naked' hard drives and use a USB->SATA or USB->IDE, or a universal adapter. The only problem with this approach is the 'naked' drives purchased through an online retail channel don't really cost any less than a similar USB drive, and they're easier to damage, since all the traces and ribbon cables and pins and such are right there on the outside to be bent, nicked and scratched.

Now go look at what tape drives and media costs, especially the ones that can fit your hard disk onto one tape, knowing the tapes have severely limited (or NO) reuse, are UNRELIABLE, even in the most anal-retentive clean room environments, and you will quickly change the page back to the hard drives, which are reasonably durable, hermetically sealed against dust and moisture and robust in a wide variety of filthy environments... like your home. Even if Martha Stewart couldn't find dust in your house with white gloves and nothing to do for a week, and you spend hundreds a month on heating and air conditioning to keep the temperature at the crisp temperature that tapes in a vault prefer, your backup tape media is likely to be 'bad' before you can ever use it to restore anything. Oh yes, the e-tailer will sell you one, no questions asked. You're the one who's supposed to know better.

Even CDR/DVDR media needs protection. It's more durable against a lot of things than magnetic media, but it can be marred and scratched, it will 'forget' if left in the sun for a few days, or even warp (especially if left in a car) and become unreadable. The dye on some even self-degrades in a few years no matter how it's stored. Optical media needs a cool, dry, dark place. In very warm, humid environments, biological stuff has been shown to actually eat the polycarbonate the disks are made of. Of course, the same kind of stuff will eat tape or floppy disks, too. I've seen it. Since your typical hard drive is sealed (except for a moisture-blocking membrane to keep pressure from building up inside), it will stand up to more general environmental conditions.

Another argument FOR multiple hard drives is getting two or more of the identical drive at the same time. If you have a logic board failure, you can usually swap the board from a working one (of exactly the same part number and stepping) to the failed one if you're in dire straits for that data. Yes, it's true, not all drive failures have something to do with the mechanical parts, and NO, you do not open the chamber that has the actual heads and platters. A year or so after you buy your hard drive(s), your chances of finding the exact same stepping of replacement drive to try this with diminishes to effectively-zero. If you're going to try it, make sure to back up anything from the 'donor' drive first, because you're about as likely to make two dead drives as resurrect the one with the 'lost' data unless you are VERY skilled and just a little bit lucky.