Transfer disks, partitions, folders to a new machine (Linux)

Thomas FischerBlog

About

You may notice that I really like Open Source stuff and for sure GNU Linux.

One of the uncountable reasons is: the straight-forward handling of the filesystem.

Everything is a file – even a partition. This comes very handy in many situations and one of the coolest things is when it comes to backup. You can also grep on a whole partition and so with a bit of regex it is easily possible to restore text files (and more ofc).

But lets say you want to migrate from one machine to another. Different location, different networks and the Internet between. There are several ways to solve this and there is even Clonezilla which has a “remote” option allowing to do that. I haven’t checked if that data gets encrypted but I guess not. Anyways I tried it 2 or 3 times to use Clonezilla but it either was not able to detect all my disks (ofc a special setup here) or I didn’t got it to run at all.

One step back: When everything is a file in Linux doesn’t that mean we could just tar the whole filesystem and copy it to the new disk on the new machine and it should just work? YES. While that remains true tar will put some (unnecessary) overhead and by itself it is “just” an archive tool not meant for syncing data.

With that in mind here comes a super cool approach to achieve exactly that:

Copy the full disk from a source to a new target, encrypted, over the Internet, in the fastest way possible!

The following explains the process by a (real world) example which I have used very very often in the past.

The setup

TARGET (the fresh blank system)

  1. the partition you want to get the data on is /dev/sda1 (must exists or use the full disk /dev/sda when you want to write/get all partitions). Obviously you have to adjust this to your system and needs.
  2. the receiving port is 8080 (ensure your firewall rules are properly set!)
  3. the encryption password is abcdefgHi (use at least 30 characters here and avoid special chars as these would require specific escapes on the cli later)
  4. the host IP is 1.1.1.1
  5. the small utility pv is installed (either on the SOURCE or the TARGET or even both. Its useful for showing the progress but optional at all)

SOURCE (the system which you want to migrate)

  1. the partition you want to send the data from is /dev/mapper/vg_root-os (this can be everything: a disk, a partition or any other single file)

Partition (or disk) transfer

While this example uses a partition as target and source you can also use a complete disk or mix it between or write to a file.img instead.

  1. Boot the target system into a Linux live system. Ensure internet is properly set and the receiving port is allowed by your firewall rules.
  2. On the source system you have the choice and while it is possible to do everthing within the current running OS it is recommended to umount the partition you want to sync. In case of the root filesystem it is recommended to boot a live system as well. .. but again it would work even with the root fs the described way but might need a filesystem check on the target later.
  3. Start the receiving command on the TARGET system (yes that has to be started at first).

    TARGET     (with progress):

    nc -l -p 8080 | pv | openssl enc -d -aes128 -pass pass:abcdefgHi -in - -out /dev/sda1

    TARGET (without progress):

    nc -l -p 8080 | openssl enc -d -aes128 -pass pass:abcdefgHi -in - -out /dev/sda1

    you start a netcat listener which forwards (whatever comes in) over to openssl for decrypt which then gets written directly to the target partition. As said the output can be everything so a file or like in this case a partition.

  4. Start the sending command on the SOURCE system. You can enable pv here as well but usually it is enough to see that on 1 site ;)SOURCE (without progress):

    cat /dev/mapper/vg_root-os | openssl enc -e -aes128 -pass pass:abcdefgHi -in - | nc -c 1.1.1.1 8080

    SOURCE (with progress):

    cat /dev/mapper/vg_root-os | openssl enc -e -aes128 -pass pass:abcdefgHi -in - | pv | nc -c 1.1.1.1 8080

    you cat the whole partition and pass its content to openssl which encrypts it with the given password before sending it to netcat which by itselfs just send the encrypted data to the target.

  5. When the above has finished (e.g. progress shows no incoming data or check network traffic or top etc) you can try to mount the partition, set boot flags, chroot to install grub etc.
    Note: the above even works when the target disk is bigger then the source! After the command finished just use resize2fs to have the new disk size (requires a filesystem which supports that like ext4, xfs , ..)

 

The overhead of the above is very small unlike in SCP which was (even with tuning) noticeable slower (avg. 40MB/s with SCP vs. avg. 90 MB/s with the above method).

Of course there is much more you can do with netcat, you can even use it as a chat or doing port scans , or …  ;)

Folder transfer

Here a quick other example if you want to transfer not a whole disk/partition/single file but one or more folders only.

  1. Start the listener to get folders or multiple files on the target system:TARGET (with encryption): 
    nc -c -l -p 8080 | openssl enc -d -aes128 -pass pass:abcdefgHi -in - | tar xvp

    TARGET (without encryption):

    nc -c -l -p 8080 | tar xvp

    all incoming data will be decrypted and then the resulting tar extracted (to the directory you are currently in). Note: there is no pv here as we use "tar v" but ofc you can leave that out and instead use | pv | before the tar to get the generic progress view. the “p” option of tar preserves file/folder permissions which usually is useful as otherwise all files are owned by the owner who started the command.

  2. Send folders to the target system:SOURCE (with encryption): 
    tar cvp - /var /opt /usr /home | openssl enc -e -aes128 -pass pass:abcdefgHi -out - | nc -c 1.1.1.1 8080

    SOURCE (without encryption):

    tar cvp - /var /opt /usr /home | nc -c 1.1.1.1 8080

    the given folders gets encrypted and then the resulting tar will be send by netcat. Note: there is no pv here as we use "tar v" but of course you can leave that out and instead use "| pv |" before the nc command to get the generic progress view. the “p” option of tar preserves file/folder permissions which usually is useful as otherwise all files are owned by the owner who started the command (on the target).