It is possible but inconvenient to manually clone a hard disk drive remotely, using dd and netcat. der Mouse, a Montreal-based NetBSD developer, has developed tools that allow for automated, remote partition-level cloning to occur automatically on an opportunistic basis. This facility can be used to maintain complete duplicates of remote client laptop drives to a server system. This network mirroring facility will be presented at BSDCAN 2005 in Ottawa, ON on May 13-15.
The facility has three main components: a client-side kernel component that monitors disk drive writes, a client-side userland program that initiates and handles client to server communication, and a server-side userland program that accepts incoming TCP connections and handles incoming encrypted data streams from multiple clients.
The system operates in a filesystem-agnostic basis, using 512 byte blocks as a lowest common denominator unit of exchange between client and server. At each client to server connection, the application identifies and maps changes to disk block states. Changed blocks are then encrypted and sent to the server. This indicates that a user could open his or her laptop in an airport, establish a WiFi link to an open access point, and remotely update their laptop backup without effort, knowledge or even good intentions.
The code is being released into the public domain free of license restrictions in any form. The initial proof of concept code has been written to NetBSD, but der Mouse expects the code to be easily portable to systems that allow hooks to be inserted into disk driver code. The code can be accessed via anonymous FTP at ftp.rodents.montreal.qc.ca:/mouse/livebackup/.
On May 13-14 at the University of Ottawa will be hosting BSDCAN 2005. BSDCAN 2005 is a technical conference for developers and people interested in the technology behind the BSD Operating Systems FreeBSD, NetBSD and OpenBSD.
For more information:
Live Network Backup
email: der Mouse <mouse @ Rodents.Montreal.QC.CA>
ftp: ftp.rodents.montreal.qc.ca:/mouse/livebackup
BSDCAN 2005:
email: Dan Langille <info @ bsdcan.org>
web: http://www.bsdcan.org/2005/
| Linux Bootup hangs after adding RealTime Premption and HR-Timer | 1 hour ago | Linux kernel |
| SATA 2 size problems | 2 hours ago | Windows |
| problem with 2.6 kernel driver for a USB MAG Stripe Reader as HID device. | 14 hours ago | Linux kernel |
| get_user_pages failure | 16 hours ago | Linux kernel |
| Reading linux kernel | 17 hours ago | Linux kernel |
| High level of Seagate 2.5" SATA drives failing | 23 hours ago | Hardware |
| Resetting the bios password for Toshiba Laptop | 1 day ago | Hardware |
| Linux 2.6.22 slowly RUNS OUT OF LOWMEM | 1 day ago | Linux kernel |
| Questions about modules | 1 day ago | Linux kernel |
| KDB | 2 days ago | Linux kernel |
From a user perspective
What are the advantages over a rsync/ssh solution? Performance (both in speed and disk accesses)?
If your 'master' disk fails, what would be the consequences?
lower level
This is backup on lower level. Rsync/ssh works on filesystem level, while proposed solution is block device based. They have some distinct differences, but I can't find good comparision on web right now.
Taking Linux-zealotry hat on: this could be also done on Linux, without additional kernel component. The key here is using LVM's Snapshot capability.
LVM should became ubiquitous. Fedora is doing great job making LVM default while install.
--
:wq
I think the main difference b
I think the main difference between this and Rsync is "real-time" backup. The duplication is done right away the modification occured in the main disk.
And I don't think the snapshot of LVM could be compared with this feature. One is for mirroring data, one is for preserving freezed data image.
There is already implemented
There is already implemented & well-tested software which does this very thing in Linux. A kernel module reports to a userland tool when a file has changed and rsync moves the changes over at the filesystem level. However I don't see why a custom implementation of rolling checksum in a librsync-fashion couldn't also send changes at the partition/whole-disk level, even from userland (disks are files too, you know..). The kernel module could just report a 'guess' of where in the partition the change was noted and binary deltas in a given range could be sent over to the server. It's difficult to get this type of program not to be complex whilst only sending specific filesystem changes...
Couldn't you accomplish simil
Couldn't you accomplish similar things by copying the contents of a snapshot volume (not what the filesystem usually sees) to a server on the other end? The server side would have to merge the changes in to the last snapshot, though.
Synchronizing user deletions
Synchronizing user deletions and application corruptions "without effort, knowledge or even good intentions." Please don't use this as your only backup.
Backup...
This system is akin to RAID. RAID is not backup. True backup is a semantic thing, you backup your important information.
Even in a perfectly clean filesystem, with no application corruptions, no disk crashes, your data can be lost. That is why there are things like CVS.
No, this does not appear to b
No, this does not appear to be a realtime thing (which RAID would be), rather the user initiates the process. Sending all drive writes over the network would be pathetically slow unless at gigabit speeds!
Combined with LVM on the server side would be even better
Then, you could use LVM on the server side to store snapshots, and still have the client keep a near real time update in sync on the server. That would allow multiple snapshot recovery of the clients getting past the "replicated corruption".
sounds like nbd to me
Is this a NetBSD version of Network Block Device (NBD)? NBD has been around in Linux for a few years. There is a variant of NBD called DRBD (Distributed Replicated Block Device) which has good integration with Heartbeat. (NBD is basically a network-raid-1.)
Hmmm... I'd like to see more layers on this.
If you increase the block size a bit (say 1K to 4K) and did checksums on the blocks, you could possibly optimize away many of the block transfers over the network and get a decent speedup (along with cheap compression) to boot.
Treat the checksum as a hashing function. (Pick something reasonably strong, like SHA-1.) The backup server can determine if that block's already backed up. This would be most useful if you had many people sharing one backup server, because something tells me the same files appear on many machines. Also, the same block will come and go. And, you still get a bit-image backup of everyone's HDs.
Another good feature to have would be "snapshot points." Since you're only sending a delta over the network anyway, nothing stops you from storing things such that you can move between different FS revisions over time.
Way to not read the article.
Way to not read the article.
"Changed blocks are then encrypted and sent to the server."
So unchanged blocks aren't. Thanks for playing.
Somewhat like Norton Ghost
This is similar to another netBSD based network disk imaging system (forget the name). I did not get the impression from the writeup that this is a 'live' backup, but rather that you can save partitions to an image file and load them over the network. Say a 'doze PC gets a bad virus.. assuming you have important data backup facilities, you could reimage the entire machine remotely and start over with a good HD image for that machine.
If I am correct, netBSD only has to be involved in this for the server and the restoring client. You could conceivably have a netbsd boot cd that reloads the hard disk remotely. After loading, boot whatever OS it is that is on the image.
You'll also notice that the b
You'll also notice that the block-level changes, as they are tracked, are updated by a client-side program. The client does, on some level, have to be running netBSD. However, if you ran a VM on top of netBSD that was running 'doze, then you could work within this setup.
RE: "... another netBSD based network disk imaging system"
g4u - http://www.feyrer.de/g4u/
It's cool!
:)
Very nice chance backup serve
Very nice chance backup servers www servers for example , good. thnx