How To Perform a Bare-Metal Backup On Linux LVM

perlow writes "Using the free System Rescue CD you can perform bare-metal backups and restores of many types of computer systems. In this article, ZDNet columnist Jason Perlow explains the multi-step procedure in detail on an LVM-based system."

Another good disaster recovery solution

[lazy_nihilist]

http://www.mondorescue.org/ [mondorescue.org]

Re:

[perlow]

Mondo is cool, but it is not platform independent and usable on a live CD. It requires installation and compilation on each box to build its own rescue environent.

The objective of my procedure was to come up with a method that could back up ANY Linux distribution at a low level in a very standardized and platform independent way, without sacrificing storage or performance OR TOUCHING THE OS ON THE BOX IN ANY WAY. This works with any Linux system you boot the System Rescue CD on, it requires no package or so

Re:

[perlow]

I'll look into this and correct if necessary. It appears to be redundant because the config backup files contain the UUID.

Re:

[donaldm]

Mondo is excellent for disaster recovery of the Linux Operating System with the advantage of producing a recovery iso without shutting down the system. While you do need to compile the source it is fairly easy to do this and the compiled binaries can easily be installed on all other Linux OS's of the same type.

I have found that Clonezilla is an excellent tool for creating recovery or deployment images however it is not really a true disaster recovery tool since you have to create an image by shutting dow

Sure - Don't.

[pla]

Step 1 - If your LVM hasn't forgotten its own configuration yet, get that sucker off an LVM as fast as you possibly can.

When I first discovered LVM (and it had already reached a quite mature V2 at that point), I thought it the greatest thing in volume management since the invention of the RAID.

I didn't half-ass it, I took my time to learn how it works, set up and broke a few, added a new PV and grew it to make sure it worked, backed some up, recovered from deliberately broken volumes (which works great - as long as you never try to bring the broken volume online by, say, rebooting), etc. Then I turned it on and loved it.

Then I received a hard lesson in what it means to say that JBOD has a failure rate multiplicatively proportional to that of each individual drive. And, surprise surprise, LVM amounts to nothing more than fancy OS-level JBOD, without even the performance boost of a proper RAID controller.

Thanks but no thanks. If I want to span a volume across multiple disks, I'll use RAID 5 or 6 from now on, ThankYouVeryMuch. I can only thank Zeus that I hadn't yet gotten around to cannibalizing my old file server when the LVM one day decided to eat itself (even if I could have salvaged something from the individual drives, I couldn't once it "fixed" its configuration to reflect the new PV set).

Re:Sure - Don't.

[bernz]

You make sure that all your devices block devices (at least if you're in a "Real Enterprise") are at least mirrored. Just because LVM2 writes across all disks, doesn't mean you're in a single-point-of-failure mode (or RAID0 or JBOD). You just make sure your "disks" are not just a single Physical Disk, but rather a LUN composed of a RAID6 with multiple paths anf power-supplies.

And if you're not an Enterprise, this can be done on the "cheap" with RAID6 commodity boxes running as iSCSI targets.

If you're ever relying on a single device for ANYTHING you're doing it wrong. But LVM is perfectly safe if you do it in a responsible way. It just isn't necessarily super-cheap.

Re:

[pla]

If you're ever relying on a single device for ANYTHING you're doing it wrong. But LVM is perfectly safe if you do it in a responsible way.

I agree completely - And mostly meant that as my point (with an anecdote about myself as an otherwise-knowledgeable user making a really really stupid mistake).

LVM does work exactly as advertised - It just shouldn't have anywhere near the popularity it currently enjoys. For example, several Linux distros (Fedora comes to mind) set up an LVM by default. Ouch.

But ye

Re:

[pipatron]

I use LVM on all my single-drive setups. I use it on this 16GB compact flash-equipped laptop. It's very convenient to be able to shrink and grow partitions on the fly, when testing new distros or using virtualization. On the machines that I actually use it across more than one physical device, I make sure to have a good backup...

Re:

[petermgreen]

IMO the big thing that LVM offers is it makes reproportioning easier and safer.

With traditional partitioning you have to slide the data along the drive (which requires you to take the system offline, is time consuming and will probablly screw things up big time if power fails while doing it).

With LVM, I can add unallocated space to any volume I like quickly and easilly. If the filesystem supports online resizing I can even do it without unmounting it.

I agree though that while LVM does have JBOD and striping

Re:

[tzanger]

LVM does work exactly as advertised - It just shouldn't have anywhere near the popularity it currently enjoys. For example, several Linux distros (Fedora comes to mind) set up an LVM by default. Ouch.

What the heck's wrong with that? I've been doing that manually on my cheap slackware systems for ages. I don't want hard partitions, and those who do are generally too young to remember the hell of C: D: E: F: G: H: drives in the old DOS days. Take a 250G drive, put a 5G / on it and LVM the rest for /usr

Re:

[Spirilis]

Yeah, that's kinda how it works. LVM is a volume management system, not a data protection system. LVM is still decent, but use underlying RAID devices as your PVs. That way you still have the flexibility of LVM without being susceptible to single-disk failures.

Re:

[Henry V .009]

LVM shouldn't be used except on drives on a redundant RAID level. As long as you take that basic precaution, it's fine and can be fairly useful.

Re:

[robo_mojo]

Then I received a hard lesson in what it means to say that JBOD has a failure rate multiplicatively proportional to that of each individual drive. And, surprise surprise, LVM amounts to nothing more than fancy OS-level JBOD, without even the performance boost of a proper RAID controller.

Thanks but no thanks. If I want to span a volume across multiple disks, I'll use RAID 5 or 6 from now on, ThankYouVeryMuch.

Your problem wasn't that you used LVM, your problem was that you made it into a JBOD. Some of us

Re:

[gbjbaanb]

so lets get this straight, just for the record. You installed LVM and added additional drives to a single volume (in much the same way as you'd do with a RAID0 setup) and when 1 drive failed you were surprised that you lost all your data?

Its not magic, you need RAID1 or RAID5 to get protection from drive failure, or good backups. LVM has some features that make this a good choice (eg snapshots) but it is not a RAID solution. Stick it on a redundant array and you'll be fine.

Re:

[Chandon Seldon]

There are rarely free lunches when it comes to computer system configuration choices - and when there are free lunches, you can be sure that the hardware / software will be using that configuration by default.

The safest bet is this: If you aren't sure that you understand the trade-off you are making by using a tool like LVM, you should assume that it's going to come back and bite you in the ass. This isn't just true for software stuff either, hardware RAID 0 has basically the same reliability issues - but

Re:

[dnamaners]

Of course it would behave like JBOD if you applied LVM over many non-redundant volumes. To expect redundancy protection from a tool that does not provide it by nature is like expecting blood from stones. Use LVM to best effect on whopping great raid volumes. This way and you get the effect of partitions, the flexibility to change your mind about layout. Not to mention snapshot and other useful things. The redundancy is managed by the raid method you use. Choose your tools well, use more as needed.

If

Re:

[drsmithy]

I didn't half-ass it, I took my time to learn how it works, [...]
Then I received a hard lesson in what it means to say that JBOD has a failure rate multiplicatively proportional to that of each individual drive. And, surprise surprise, LVM amounts to nothing more than fancy OS-level JBOD, without even the performance boost of a proper RAID controller.

These two statements are in conflict.

No! You have to use BOTH!

[khasim]

LVM is NOT the same as RAID5.

LVM makes it easy to move space around on the disks, but it does NOTHING to prevent data loss from failed disks.

Put LVM on top of a RAID 1 or RAID 5 subsystem. Then you can add / replace disks and grow the volumes to use the new space.

Re:

[bigredradio]

Not RAID 5, but you can map two physical extents on different disks to a single logical extent. This would be similar to RAID 1.

Re:

[Saeger]

You can span LVM across many individual PV points of failure (i.e. single disks), but only morons do that. The right way to use LVM is to layer it above the redundant (RAID) storage, and/or make sure you have really good backups if you're only using LVM for its added features.

LVM itself can also do simple mirroring (and striping) internally across multiple PVs, but this is used far less commonly than layering it over s/w, h/w, or remote RAID.

Re:

[Malevolyn]

Now for the important question: what's the proper word for self-cannibalism? Autocannibalism?

Re:

[Eggplant62]

You mean you actually thought at first that the LVM was supposed to replace a RAID mirrored and striped array? Son, that's where you made your mistake. Create the raid, put the LVM on top of the raid.

That's just dumb.

Re:

[tzanger]

I don't know about the rest of y'all, but my filesystems are generally RAID arrays running LVM on top of them. i.e. my home fileserver has a pair of 300G drives, four 250G drives and a pair of 200G drives. I have it configured into two RAID1s and a RAID5, and the three /dev/mdx nodes are PVs which go into my VG. I can survive any single drive failure without losing the VG, and I can grow it as I want, without having to worry about my older "big" drives getting waylaid. It's perhaps not an ideal setup, b

Hmm.

[gbjbaanb]

other features for bare-metal backup/restore involves:

Systemimager (dead good, and also good to restoring an image to other machines too). Restores from a cd (or PXE) boot and pulls the necessary data from the server with minimal intervention.

Any virtualisation technology - depends on your definition of 'bare metal', but if you think of it as an easy-to-restore server, then I guess it counts.

r1soft's (paid-for) continuous backup solution. Not tried this, but many webhost type companies have said its the dogs danglies.

I prefer old school backup

[dnamaners]

Don't get me wrong, I love centralized backups and backup server/client software (bacula for example). However I use old school methods for a bare metal restore. For simple things blanking the free space and then make a "dd" image is all you need. However you do it, keep it simple, and there may be no need to do bare metal on the whole of a file servers data set. Once I get the core of it back up I can use my favorite software to get the system up-to-the-last-snapshot current.

LVM and Raid can be a little complex if you boot off it so ill give some tips. A nice small boot like SLAX (may favorite) or Knoppix is needed to start any bare metal restore. Pick something that supports the hardware and had LVM and or Raid. Either make a custom disk to add your backup data to the volume or have a 2nd DVD drive. Boot and restore.

I make my backups via simple methods like rsync and tar. I use a few simple scrips to do it all. So simple that it is easy to restore by hand if need be. This makes the restore safer as it can be done a host of ways. Not to mention these unix tools are present in all distributions. Probably the biggest reason for the old school way is that on a bare metal restore often hardware may change. New drives etc, basically failure can be a reason to upgrade etc. Many utilities can choke on big hardware changes, new drive sizes and such.

I take detailed system information and note it in the backup as human notes. Then grab your raid & LVM configuration and write a quick script to rebuild these features via the command line. Basically just a list of the actions you take to make a file system and format them the desired files system types manually. These same scripts will also re-apply boot loader and such as well. If the volumes change after a failure I can just issue updated commands (or edit the script) and make a similar layout of the filesystem. Then just open the tar balls via script or by hand and populate the new filesystem with the files you archived. With a bit of practice all of this can be automated by scripts. But if you need to vary the procedure due to hardware changes or as a way to clone a system you can.

Re:

[wertarbyte]

I'm using tar to store the stuff on-the-fly to an FTP site. I even made a flexible backup script [wertarbyte.de] to coordinate upload, snapshotting, encryption and everything else. It's completely based on standard unix tools, so I won't have to install any special backup/restore software when recoverying my server.

The RedHat shop way

[EvilAlphonso]

Kickstart from the satellite for the OS

CVS checkout in the %post section of the kickstart for /etc or use the configuration channels

amrecover for the data and applications not stored on the satellite

As long as you keep a sane filesystem plan, the entire thing can be kept in an easy to follow procedure and fully scripted (except for the tape change on the amanda server).