WEBlog -- Wouter's Eclectic Blog

Sun, 07 Aug 2011

Restoring from backups using NBD

During the past Debconf, Joerg borrowed my laptop. I'd told him that he could basically wipe all files if he wanted to, since I had a full backup at home. With that, I mostly meant to say that he could reinstall the box, or add or remove packages as much as he wanted to.

What I didn't mean for him to do was to give it back to me with a completely wiped hard disk that it wouldn't even boot off of anymore, but that's what he did. Oh well.

So I hadn't done the restore yet, last week, since I'd been busy doing other things. Now, my backup consisted of a file called 'mbr' of 2KB; a file called 'backup.tar.bz2' of 66G, and an LVM volume called 'windows' that was written with 'dd if=/dev/sda1 of=/dev/lvm/windows'[1]. Meaning, I could do a restore, but it was somewhat more involved than just running bacula and allowing it to do its stuff for a while.

I briefly considered mounting the external USB disk on which the backup was written to my other laptop, and streaming everything over the network, but quickly discarded that idea for speed issues. Instead, I created 4GB NBD export on that other laptop, ran d-i with my partman-nbd on the to-be-restored laptop, installed Debian to the NBD device, booted off of that, and could then do a restore while having a fully functional system. With the sole gotcha that I couldn't disconnect it from the network until the restore was complete, but that's not insurmountable.

The 'dd' restore has finished in the mean time, I'm still waiting for tar to complete its job. Once that's finished, I'll probably have to refresh grub to the internal hard disk, and then I'll be able to boot it.

I hope.

Having said that, doing a bare-metal restore while booting off of an NBD device is quite practical. I mean, sure, I could have downloaded a live system from the network somewhere, but that would've taken forever, would probably have had an impact on performance of the system while it was running, and overall isn't very ideal IMO. On the other hand, an NBD device is much faster than a local CD or DVD, can be written to without requiring a RAM disk, doesn't need squashfs to get you to reasonable sizes, and as far as the kernel is concerned it's just a local hard disk.

If the above paragraph sounds like I'm proud, then that's because I am. Two years ago, doing this would have involved much more manual work. Today, I had a system running root off NBD in minutes.

[1] Yeah, so my car has a builtin TomTom GPS that I can't do updates for under Linux, AFAIK. Plus some other stuff. Sue me.

Sat, 23 Jul 2011

IT BUILDS!

Finally.

Since the release of squeeze, I've been fighting to get nbd to actually build on "most" architectures. There had been several issues, not the least of which was the fact that it had categorically refused to build on the kFreeBSD ports. I'd tried to debug that several times, but could not understand the issues. So then I set up a VirtualBox image with kFreeBSD to test, and when I finally got it to build there, I did an upload with the changes that I'd applied. Only to find it didn't build on the autobuilders.

I briefly considered building it myself and uploading that, but then we have two kFreeBSD ports, so that wouldn't help. And, I guess, it would be severely frowned upon.

Eventually I found the final issue: the test suite has recently acquired a set of "integrity" tests, that replay a recorded session and verify that the responses from the server are correct. This implementation in nbd-tester-client, however, has non-blocking writes to the server, and the server also does non-blocking writes. The "integrityhuge" test tries to write rather large blocks in one go; so basically, what happened not only on kfreebsd-* but also on some Linux architectures, like s390 and ia64, was that the buffers of both the server and the tester client were full, and they would deadlock when both wait for the other to read before they could continue writing.

This is a bug, yes, but one that has existed for a very long time, and one which is unlikely to hit in actual usage—the integrityhuge test fires off a multitude of very large reads and writes, without waiting; a situation that is highly unlikely to occur in real usage. So for now, I will disable the integrityhuge test, until such a time when I can implement async writes—a feature that has been on my TODO list for a significant amount of time.

Tue, 27 Jul 2010

NBD: authentication and named exports

I just did some work on two NBD features: authentication and named exports.

The first was written by Brad Allen back in 2008. I'd been thinking about an authentication scheme back at DebConf 8, and discussed it on the nbd-general mailinglist. Brad picked it up and ran with it, but it never got finished. Basically, what I did today was forward-port the code so that it would compile (and work) with today's version of the NBD utilities.

I tested it and it works, but I'm not sure what to do with it. NBD is not encrypted; so there's no protection against TCP hijacking or some such. Oh well; I might still integrate it into nbd mainline.

The second feature is something way more interesting. It had been on my long-term TODO list for quite a while, but I never sat down to implement it. Until today, that is. This patch has been compile-tested, but I haven't functionally tested it yet; I'll do that tomorrow.

The main reason it took so long to implement, was that initial NBD handshake made that quite hard. Upon connecting, nbd-server as it currently exists will open the file or block device that it needs to export, and—before nbd-client has sent anything—sends information on that export (its size, whether or not it is read-only, etc) to the client. Unfortunately, this does mean that I cannot add new features to nbd-server while retaining full backwards compatibility if these features may involve changing what nbd-server sends during the initial handshake, as is the case here. This could be worked around by having the server first send information on a "default" export to the client, and then variate that if the client informs the server that it actually wants something else. But this would have been rather ugly, since it means that errors in open() or stat() calls for the initial "default" export would have to be non-fatal until the client actually tries to read or write to or from the device, at which point they would suddenly become fatal. This would open a can of worms near which I didn't want to tread.

The way I fixed it in the end, however, is fairly simple. If names are used to specify an export, nbd-server can live with just one port to export all block devices that it supports, rather than needing many. As such, I can "reserve" one port for the new-style named exports. On this port, nbd-server would expect that clients send a name, and would not function correctly with clients that don't understand names. At the same time, nbd-server could still export the same block devices on other ports as well, using the old-style protocol, for clients that do not support named devices.

This would of course require that both the client and the server know perfectly well what the port is, and that it does not change too easily. As such, I've just put in a request for an assigned port number over at IANA, so that I can be sure that the port will not change unexpectedly. This also means I won't make a public release with this change until the port number has been assigned, so that I don't need to support two 'default' nbd-server ports in the future (one "temporary" and one IANA-assigned port).

For now, however, I think I'll go grab some food.

Wed, 24 Dec 2008

Root on NBD, now with partitions

A while ago, I wrote an initrd script and hook that would allow one to run a system with root on a Network Block Device (NBD), that I maintain both upstream and in Debian.

Since then, Laurent Vivier submitted a kernel patch that got merged, and which will allow one to partition an NBD device. I had a patch for my initrd script that would support running root off a partitioned NBD device, but did not yet have the time to test and debug it.

Until today, that is. Well... not really. I made the time, but I didn't really have it. For reference, the time is 07:25 right now, and yes, I did stay up all night. But, well, who cares, right?

Right.

I'd really love to get this into lenny still, but since we're in deep freeze... not keeping my breath. We'll see, I guess. The upload is going to unstable, anyway—after this one final test.

So how does it work? Unfortunately, no d-i support as of yet. I'd been working on that, but then it turned out that in order to support root-on-NBD, I'd have to somehow, somewhere, teach partman that NBD is this strange thing which can be formatted, but not partitioned. I was way far in my research (but nowhere in code, yet) on that when Laurent sent in his partitioning patch, so it got postponed then, and I didn't get a chance to pick it up anymore in time.

But if you're willing to use debootstrap manually, it's really simple to set up:

# apt-get install nbd-client
# nbd-client server port /dev/nbd0
# fdisk /dev/nbd0
[... create partitions ...]
# mkfs -t ext3 /dev/nbd0p1
# mkdir /mnt/tmp
# mount /dev/nbd0p1 /mnt/tmp
# debootstrap --include=nbd-client sid /mnt/tmp
# echo KILLALL=false >> /mnt/tmp/etc/nbd-client

The "KILLALL" is necessary, since otherwise the nbd-client initscript will do nasty things with your root filesystem. Note that you still need nbd-client installed and run the initscript, even if it only uses nbd-client inside the initrd; because the initscript writes the nbd-client PID to /lib/init/rw/sendsigs.omit.d, which is also required if you don't want sendsigs to shoot the root filesystem from under your nose. This might see some changes in the future.

Now boot with your kernel and initramfs, and boot it with the following arguments:

nbdroot=server,port root=/dev/nbd0p1

Obviously, in the above two examples, you should replace 'server' by the IP address or hostname of your server, and 'port' by the port number on which your nbd-server is running. It is of course also possible to use nbd1p2 or nbd0p4, should you want that.

And that's all there is to it.

Well, almost; due to the way the network configuration is written in initramfs, I had to choose between reinventing the wheel, or hardcoding the NIC name to 'eth0'. I chose the latter. What this means is that if you want to boot off an NBD blockdevice that is only available on eth1 or wlan0 or some such, you lose.

Other than that, go ahead and try. Should be on a DMNY (Debian Mirror Near You) RSN.

Update: fill in the link to git.kernel.org, and mention the KILLALL thing. Oops.

View comments on this entry (2 comments)...

Wed, 02 Apr 2008

nbd 2.9.10

I Just released NBD 2.9.10. It had been upcoming for a while already, with a few interesting patches being applied. In addition, some bugfixes were applied to subversion, and some more interesting things had happened. Really, a release was getting overdue. Sometimes life just intervenes.

Anyway, apart from the upstream release, I of course also did a Debian update of these packages, which also contains some changes that had been pending for a while. Combined, the changelog mentions five different bugs of which none are l10n bugs, which is a record (there had been 5-bug uploads before, but those did include l10n bugs). On top of that, I closed some other outstanding bugs which really had been closed a while back, but either I forgot to close the bug, or I forgot that there even was a bug in the first place.

I should stop slacking so much.

While preparing this upload, it occurred to me that nbd has come a long way since I got involved with it back in 2001. The original nbd-server that Pavel Machek had written did work well, but it was a bit of a hack; it did not fork() upon startup or anything similar, so starting it was a pain. It could only serve one device at a time. It was not documented. It had no concept of a configuration file. The code, frankly, was also bit of a mess; it was written on a hack night, and not much design had gone into it.

I guess it was no surprise, then, that no single distribution had a package for nbd at the time. I only packaged it because I actually had some use for nbd; at the time, I'd just bought quickstep, my first functional m68k machine, and had installed Debian on its 250MB hard disk. However, running a buildd on that machine proved problematic, to say the least; with so little disk space, you could hardly boot Debian, and due to speed and clock skew issues, NFS was not an option. So I went with NBD, which worked great. After a while, quickstep got a bigger hard disk (it has an 18G disk today), so diskspace was less of an issue at that point; but, well, I'd been maintaining it already, and that was fun, so I didn't think of giving up.

When Pavel lost interest in nbd back in 2003, he gave maintenance of the kernel bits to Paul Clements, and maintenance of the userspace bits to me. I promptly went ahead with what I thought should have been done already, but felt reluctant to do at the time to suggest that: refactoring the code, adding features, etc. Today, nbd-server does support all of the above things of which I mentioned it was lacking; and, probably as a result of that, apart from Debian, there are now packages of nbd in, at least, SuSE, Fedora, Gentoo, uClibc, and FreeBSD (the latter obviously only containing the server). Well—the fact that LTSP uses it might have something to do with it, too ;-)

That's not to say there's nothing left to be done, but I'm quite happy about how far we've come. Really.

Sun, 30 Dec 2007

NBD 1:2.9.9-3

I just uploaded the Debian packages versioned 1:2.9.9-2 (almost immediately followed by 1:2.9.9-3, because I forgot to test part of it, and obviously that part turns out to be broken), containing some updates:

Added a switch in the configuration file which will cause the initscript to not kill all nbd-client instances, which it does currently.
Added initramfs scripts.

The two combined should allow for running a root filesystem on an NBD device. Note that the initramfs scripts haven't received any testing beyond my system yet; it's not impossible that they're broken in some regard. Treat them with some caution.

Having said that, testing would be welcome...

Next, I've also started work on partman-nbd. This is in an even more experimental and unfinished state, but the eventual goal is to support installing to an NBD device. Wouldn't that be nice.

Wed, 19 Sep 2007

NBD 2.9.7

Earlier today, I release NBD 2.9.7, containing a few fixes for segfaulting bugs, and some new features: one by Mike Snitzer to implement support for better handling of dead connections (which has been requested by a number of users for eons); one by myself to allow for running shell script snippets at connect/disconnect time, which should be helpful for the LTSP people (they now have an inetd-based wrapper written in shell, which seems like such a waste).

Source can be found on sourceforge.net, packages have been uploaded to Debian and are even installed by now; these also fix some outstanding packaging-related bugs. Enjoy!

Thu, 19 Jul 2007

I suck

It's good if you bring out a maintenance release of a certain specific software package that you maintain, if it segfaults on startup when a new feature is used.

It's pretty silly if that new version still segfaults if and only if that very same feature is used. What do you mean, testing?

Anyway, today I released NBD 2.9.6 that fixes those issues. And I extended the tests that are ran as part of 'make distcheck' so that it won't be segfaulting on startup anymore next time I do a release.

Silly me.

Wed, 27 Dec 2006

NBD 2.9.1

I released the userland NBD tools of version 2.9.1 today, incorporating a fix for a minor memory leak (it would allocate memory for a hash table for each and every port on which nbd-server was going to listen, rather than just once for the entire server), and a fix for some piece of code which followed a null pointer (I had actually found that bug earlier, but forgot to commit the fix up to now). These were minor, but important, changes.

Of course only now do I see the mails from Mike Frysing (the Gentoo maintainer for the userland NBD tools) containing a number of build fixes. Had I seen them earlier, I could obviously have included them in the 2.9.1 release. Ghaah!

Lesson learned: check your mailbox before doing an NBD release.

Oh well.

On a side note, there's a pretty annoying and disgusting heisenbug in nbd-server, involving reliability. If you know stuff about mutexes (I don't) and are willing to help, here's a chance to be my personal hero.