Backup

Documenting my backup setup as it changes over time.

2017-09-04 Backup

I bought three new 4T disk drives for my backup needs! Two of these will be used in rotation, one of them always at my wife’s office. I’d like to encrypt them. Do I use the Apple tools to do it? Maybe I should.

I think this is what I want to do:

  1. I want to backup my laptop’s internal drive, of course.
  2. Use one of the 4T disks as the new external disk, replacing the 1T disk I currently use (called “Extern”).
  3. I also want to replace the other external disk we use for media (called “Movies”)
  4. Backup my websites using rsync.
  5. Use Time Machine for the two other 4T disks. This means that eventually, as the first disk starts to fill up, a complete backup will no longer be possible. But since all my backups are currently on 1T disks, this should be possible for quite a while.

And these are the steps I need to do:

  1. Pick a nice long password.
  2. Use Disk Utility to erase the first 4T disk and create a Mac OS Extended (Case-sensitive, Journaled, Encrypted) partition. Let’s call it “Data”.
  3. Copy all the data from “Extern” to “Data” in archive mode. Can I use cp -a for this? I think I’m better off using what I know: sudo rsync --archive --itemize-changes /Volumes/Extern/ /Volumes/Data
  4. Copy all the data from “Movies” to “Data” in archive mode. sudo rsync --archive --itemize-changes /Volumes/Movies/ /Volumes/Data should merge these without problems, as far as I can tell from the top level directories.
  5. Fix the existing backupscript such that it downloads the sites and /etc to the new “Data” drive; remove the rsync invocations for the local drives.
  6. Use Disk Utility to erase the second 4T disk and create a Mac OS Extended (Journaled) partition. Let’s call it “Time Machine 1”. Tell Time Machine to use it, and make sure the backup is encrypted. Send it off site.
  7. Use Disk Utility to erase the third 4T disk and create a Mac OS Extended (Journaled) partition. Let’s call it “Time Machine 2”. Tell Time Machine to use it, and make sure the backup is encrypted, too.

Done?

“About This Mac” reports:

  1. macOS Sierra, Version 10.12.6
  2. MacBooc Pro (13-inch, Mid 2010)

Disk Utility reports:

  1. Hitachi HTS545025B9SA02 Media (the internal disk, 250GB)
  2. TOSHIBA External USB 3.0 Media (”CANVIO for Desktop”, 4TB), twice

Tags:

Comments on 2017-09-04 Backup

Many hours later, I copied the contents of my old “Extern” disk and my old “Movies” disk (a hold over from the old sneaker net days when people would visit one another with hard disks in order to share) to the new “Data” disk.

So on to the next step: I plugged in the second disk, used Disk Utility to rename it to “Backup” and used Time Machine to set it up as an encrypted backup. I made sure to look at Options and removed the exclusion of the external “Data” disk. I want it included, after all.

I do wonder how good this Apple disk and backup encryption is.

– Alex 2017-09-05 13:28 UTC


Wow. Many hours later and we have 440GB of an estimated 1.8TB written. Time Machine is slooow.

– Alex 2017-09-05 19:38 UTC


OK, today I learned: one full backup takes more than 24h. The next laptop definitely needs USB 3.

– Alex 2017-09-06 13:29 UTC


Ok, disk “Backup” is done. Time Machine said “Encrypting Backup: 7%.” What is this? I thought it was all encrypted?

Oh well, since I was able to unmount it, I just went ahead and plugged in the third disk, erased it, called it “Backup 2”, and told Time Machine to use it without discarding the first Backup disk. So now it will backup to both. This is good.

And now that I have a new set of disks, I should definitely check the disks. But before doing all that, I will have to prepare:

  1. install the SMART driver, check all three disks
  2. much later, uninstall the SMART driver, mount all the old disks and wipe them
  3. install the SMART driver again

Gah!

– Alex 2017-09-06 14:44 UTC


OK, second backup done. When I ejected the disk it said “Encrypting Backup 6%”. I still wonder what that means.

Just to get a feeling for how things work, I decided to put the first backup disk back in and clicked “Backup Now” in the menu. To be honest, I thought Time Machine should detect the old backup disk immediately, notice that the last backup was older than one hour and immediately do another backup. Not so, unfortunately.

“Preparing Backup...”

– Alex 2017-09-07 12:36 UTC


I am happy to learn that this new backup is “490MB of 28.38GB” done.

– Alex 2017-09-07 12:39 UTC


Sadly, this is where it remains. Currently: 492MB. The estimate is: 2h remaining.

– Alex 2017-09-07 12:54 UTC


Ugh, status unchanged. This is not cool.

– Alex 2017-09-07 14:06 UTC


I was unsure of what to do and so I turned to the age old trick: I rebooted the system.

– Alex 2017-09-07 14:32 UTC


Rebooting with the drives connected left me with the grey apple screen and ventilators at 100%. I disconnected the new USB drives and held down the power button until it powered down. I am not liking this!

– Alex 2017-09-07 14:53 UTC


After rebooting and reconnecting the drives I was asked for the two passwords and the icon for the backup drive turned into the petrol colored backup icon. Good!

Picked “Backup Now” from the menu. Current status: “Preparing Backup...”

– Alex 2017-09-07 14:58 UTC


Status:”Encrypting Backup Disk: 9%”

– Alex 2017-09-07 19:04 UTC

Add Comment

2014-02-14 Backup

This got started on 2006-08-03 Backup using rsync. We’re using several external USB harddisks to keep backups. The most important part is that we have two external disks and we keep at least one of them at Claudia’s office. Just making sure that a simple fire in our flat cannot destroy all our data. I do backups using rsync. Note my Restore page.

Look at how the external drives are setup. The Extern disk is always connected and it’s where both I and Claudia keep files such as the iTunes library. That’s why it has the “noowners” flag set. The Backup disk, on the other hand, needs to preserve ownership.

alex@Megabombus:~$ mount
/dev/disk0s2 on / (hfs, local, journaled)
devfs on /dev (devfs, local, nobrowse)
map -hosts on /net (autofs, nosuid, automounted, nobrowse)
map auto_home on /home (autofs, automounted, nobrowse)
/dev/disk2s3 on /Volumes/Backup (hfs, local, nodev, nosuid, journaled)
/dev/disk1s2 on /Volumes/Extern (hfs, local, nodev, nosuid, journaled, noowners)

I run this from my main machine, Megabombus.

#!/bin/bash

if [ ! -d /Volumes/Backup ]; then
    echo you need to mount the backup drive, first
    exit
fi

if [ -z "$1" -o "$1" == "Megabombus" ]; then
    echo Megabombus
    sudo rsync --archive --delete --delete-excluded \
	--itemize-changes \
	--exclude=.Spotlight-V100 --exclude=.DS_Store \
	--exclude="/Users/*/Library/Caches" \
	--exclude="/Users/*/.Trash" --exclude="/.Trashes" \
	--exclude="/Volumes" --exclude=".fseventsd" \
	--exclude="/Users/*/Library/Application Support/Wuala/Data/Temp" \
	/ /Volumes/Backup/Machines/Megabombus
else
    echo skipping Megabombus
fi

if [ -d /Volumes/Extern -a \( -z "$1" -o "$1" == "Extern" \) ]; then
    echo Extern
    sudo rsync \
	--itemize-changes \
	--partial --archive --delete --delete-excluded --verbose \
	--exclude=.Spotlight-V100 --exclude=.DS_Store \
	--exclude=/Extern/.Trashes --exclude=/Extern/.fseventsd \
	/Volumes/Extern /Volumes/Backup
else
    echo skipping Extern
fi

if [ -z "$1" -o "$1" == "Psithyrus" -o "$1" == "net" ]; then
    echo Psithyrus
    rsync --archive --verbose --compress --delete --delete-excluded \
	--itemize-changes \
	--exclude '/logs' \
	--exclude '/planet/rpg' \
	--exclude 'temp/' \
	--exclude 'pids/' \
	--exclude 'visitors.log' \
	--exclude 'referer/' \
	--exclude '.git/' \
	--iconv=UTF8-MAC,UTF-8 \
	alex@psithyrus.epfarms.org:/home/alex/ \
	/Volumes/Backup/Machines/psithyrus/
else
    echo skipping Psithyrus
fi

if [ -z "$1" -o "$1" == "Emacs Wiki" -o "$1" == "net" ]; then
    echo Emacs Wiki
    rsync --archive --verbose --compress --delete --delete-excluded \
	--itemize-changes \
	--exclude '/org.emacswiki/logs' \
	--exclude '/org.emacswiki/htdocs/*/visitors.log' \
	--exclude '/org.emacswiki/htdocs/*/pids/' \
	--exclude '/org.emacswiki/htdocs/*/temp/' \
	--exclude '/org.emacswiki/htdocs/*/referer' \
	--exclude '/org.emacswiki/htdocs/emacs/git' \
	--iconv=UTF8-MAC,UTF-8 \
	aschroeder@web.moinmo.in:/org/org.emacswiki \
	/Volumes/Backup/Machines/moinmo.in
else
    echo skipping Emacs Wiki
fi

if [ -z "$1" -o "$1" == "Raspberry Pi" -o "$1" == "net" ]; then
    echo Raspberry Pi
    if ping -q -c1 raspberrypi.local > /dev/null; then
      for ME in pi alex; do
        echo ... $ME
	mkdir -p /Volumes/Backup/Machines/raspberrypi.local/home/$ME
	rsync --archive --verbose --compress --delete --delete-excluded \
	    --iconv=UTF8-MAC,UTF-8 \
	    $ME@raspberrypi.local: \
	    /Volumes/Backup/Machines/raspberrypi.local/home/$ME
      done
    else
	notify Raspberry Pi cannot be reached
    fi
else
    echo skipping Raspberry Pi
fi

if [ -z "$1" -o "$1" == "Subterraneobombus" -o "$1" == "net" ]; then
    echo Subterraneobombus
    if ping -q -c1 subterraneobombus.local > /dev/null; then
	rsync --archive --verbose --compress --delete --delete-excluded \
	    --exclude '/tmp' \
	    --exclude '/dev' --exclude '/proc' --exclude '/sys' \
	    --exclude '/home/alex/.local/share/Trash' \
	    --exclude '/home/alex/.mozilla/firefox/*/Cache' \
	    --exclude '/home/alex/Videos' \
	    --iconv=UTF8-MAC,UTF-8 \
	    alex@subterraneobombus.local:/ \
	    /Volumes/Backup/Machines/subterraneobombus.local/
    else
	notify Subterraneobombus cannot be reached
    fi
else
    echo skipping Subterraneobombus
fi

notify Backup finished.

Tags:

Add Comment

2012-07-20 rsync backups

I have a job that creates backups of my sites using rsync. My sites are in Germany and the USA, the backups are in Canada and Chile. The point was to protect myself against hosting services disappearing and my sites getting lost. Recently I was thinking about data corruption, however. As soon as the cronjob writes the corrupted data to the backups, there is no way to retrieve my data. (There is in fact a third backup: every few weeks I use rsync to copy the remote sites to one of a rotating set of mobile disks, one of which is always outside our apartment.)

There is in fact an option for rsync which will allow you to create copies of your file tree at certain intervals using hard links for the files that haven’t changed. I found a tutorial on how to do it: Time Machine for every Unix out there subtitled “Using rsync to mimic the behavior of Apple’s Time Machine feature.”

And that’s exactly what I did.

Update: I soon disabled it again because I was running out of disk space. :)

Tags:

Add Comment

2008-06-17 Advanced Rsync Issues

So I’m trying to use rsync to backup stuff on my various servers out there to my newly acquired external disks. Just in case the hosting providers are having backup issues. ;)

The source system is a GNU/Linux system using ext3, the target is a Mac using HFS+. That is, the target uses a slightly modified normal form D instead of normal form C UTF-8 , and the target is not case sensitive – it is case-preserving only. The mind recoils in fear!

The first part of the problem is addressed in the rsync FAQ rsync recopies the same files:

rsync --archive --verbose --compress  --delete --delete-excluded \
--exclude '/org.emacswiki/logs' \
--iconv=UTF-8,UTF8-MAC \
aschroeder@thinkmo.de:/org/org.emacswiki thinkmo.de

Except:

rsync: --iconv=UTF-8,UTF8-MAC: unknown option

Oh no, here we go again... :(

Update: Building rsync from source was painless. Yay!!

But now:

rsync: on remote machine: --iconv=UTF8-MAC: unknown option

Noooo!

Tags:

Add Comment

2006-10-14 iPhoto Corruption

Claudia got back from her holidays in Crete. We connected the camera to the Mac Mini, switched it on, iPhoto started, we imported the pictures. Then we noticed: How weird, the new pictures came right in the middle of the archive. Somehere in 2005. When I searched for “Kreta”, we got all the pictures from Crete 2006 and all the pictures from Beirut 2005. W00t!?

Some rearranging, investigating, restarting... And suddenly the complete archive only lists all the pictures up to Beirut and all later pictures were lost, unless you clicked down to the archive for 2006. Another restart, and iPhoto offered to reimport some new pictures. And it turned out that it reimported two more sets of the Crete pictures. When I rebuilt the library using the special key-combo while starting iPhoto, everything looked ok (and the Crete images were gone), but once I reimported them from a folder, things got mixed up again. Three pictures from older days suddenly got reassigned to the Crete folder.

I think I’m going to restore the iPhoto library from backup and try again.

This sucks!

Time passes. Restore from backup, everything looks ok. Import pictures from Crete folder, and there we go again!! AAAARARRRRGHH!! >{

So what we had before was a folder the Pictures/iPhoto Library/Data/2005/2005-02-22 Beirut. After importing the pictures from a folder called “Kreta 2006”, the old Beirut folder disappeared and all the pictures got moved into a new folder called Pictures/iPhoto Library/Data/2005/Kreta 2006. All the Crete pictures, got moved into Pictures/iPhoto Library/Data/2005/Kreta 2006_2.

Well, at least they didn’t all end up in the same folder like last time. But it sure sucks like hell!

What are we supposed to do now?

I think what I will do is restore a library from the backup, but keep it under a different name. And we’ll delete the current library, so that iPhoto will start a new one. At least we can navigate both libraries. Both libraries will not be corrupted. And we could even merge them later, should that be necessary.

Tags:

Add Comment

2006-10-12 VFAT and rsync

Gah. I decided to copy all my audio files to a small extra portable disk I have so that I could easily transfer stuff between home and office. It turns out that VFAT has some curious properties, which make sure that some files get retransfered again and again. Grrrr!

What should I do? In theory I could install ext2 drivers on both systems and use that, hehe. But honestly: There must be a better way!

I still remember my ext2 woes where OSX would kernel panic if a command-line tool read Latin-1 encoded filename (OSX finder would just truncate right there).

I guess I could format it as UFS and use ufs2tools to access the files from the command-line. Extra hassle... : And nobody knows whether they’ll work for external USB drives, either! I think we’re stuck with a crappy VFAT filesystem for this kind of task.

man rsync suggest using --modify-window: In particular, when transferring to Windows FAT filesystems which cannot represent times with a 1 second resolution --modify-window=1 is useful.

And it works!!

sent 267517 bytes  received 20 bytes  59452.67 bytes/sec
total size is 38204530806  speedup is 142800.92

Tags:

Comments on 2006-10-12 VFAT and rsync

I’ve seen this fortune on irc and thought about you :)

Only wimps use tape backup: real men just upload their important stuff on ftp, and let the rest of the world mirror it ;) – Linus Torvalds, about his failing hard drive on linux.cs.helsinki.fi

– PierreGaston 2006-10-13 14:33 UTC

Haha! :)

– Alex

Add Comment

2006-08-03 Backup using rsync

Here’s how I backup my current home directory on the Mac Mini onto an external USB harddisk using rsync.

rsync --archive --verbose --delete --delete-excluded --exclude=/Library/Caches --exclude=/.Trash "/Users/alex/" "/Volumes/Media Backup/Alex Pyrobombus"

For my laptop:

rsync --archive --copy-unsafe-links --verbose --delete --delete-excluded --exclude=/Library/Caches --exclude=/apache2 --exclude=/.Trash "/Users/alex/" "/Volumes/Media Backup/Alex Alpinobombus"

I don’t have any music on my laptop... ;) I’m excluding apache2 because that is a symlink to %%usrlocal/apache2%% and this causes a chgrp error when running rsync. I use copy-unsafe-links because some of the links in %%Library/NeoOfficeJ-1.1%% point outside of my home directory.

That reminds me, by the way, that I need to buy a second enclosure for the second internal 120G IDE drive I have left-over from my old Confusibombus machine. The old dead and empty hull is all that remains of my self-assembled Pentium 4 SlackWare. BumbleBees don’t last forever...

Update: I bought an “M9-DX Mini Pod” with USB 2.0 1 port upstream (B Type) and 3 ports downstream (A Type), 3 ports Firewire 400, a passive heat sink, and a thermal probe to regulate fan speed.

While backing up, I noticed that my use of the CPAN shell was wrong, and I reran o conf init and used “sudo make“ and “sudo .Build” instead of the defaults. Now I can run the CPAN shell directly without sudo, and it will sudo when installing. That (hopefully) means that my .cpanbuild and .cpan/sources will have the correct ownerships: alex instead of root. ;)

For Claudia’s account, we also need to backup the DVD stuff. The DVD stuff has no space on the system disk. So we have a disk called “Extern” for DVD stuff.

  1. UsersClaudia → VolumesClaudias Backup/Claudia Pyrobombus
  2. VolumesExtern/DVD → VolumesClaudias Backup/DVD

Here we go:

rsync --archive --copy-unsafe-links --verbose --delete --delete-excluded --exclude=/Library/Caches --exclude=/.Trash "/Users/claudia/" "/Volumes/Claudias Backup/Claudia Pyrobombus"

rsync --archive --copy-unsafe-links --verbose --delete --delete-excluded --exclude=/.Trash "/Volumes/Extern/DVD/" "/Volumes/Claudias Backup/DVD"

Tags:

Add Comment

2006-07-11 Restoring an iPhoto Library from DVD

I burnt a copy of Claudia’s iPhoto library on a DVD, for backup purposes. I also looked at the files in the iPhoto Library folder and found that the movies (little 30s AVI files from my digital camera) where not to be found in the iPhoto Library. So I decided to try the following experiment: Renaming the existing library and restoring the library from DVD. After all, that’s my worst case scenario anyway.

I renamed the existing library, and when iPhoto started, I had to create a new one. That seems reasonable. When I inserted the DVD, I was able to browse it. Cool. But how to merge them, including originals?

After some googling “merging iphoto library” I decided I just wanted to drag the folder from the Finder into iPhoto. This worked, in a way: All the pictures got imported, but since I had about twice as many images after the import, and I spotted several duplicates, I’m assuming that it imported all the files from the Data, Originals, and Modified folders. No good! I deleted the new library folder again.

Then I tried the thing that seems like the most natural thing to do: I just copied the iPhoto Library folder from the DVD into my Pictures directory. Why merge a new library with an empty one, if I can just copy it? After restarting iPhoto it regenerates the thumbnail cache, which seems reasonable enough. The number of images is correct. And the AVI files are part of the library folder, too. Great!

Simpler than I thought. It’s not always the system’s fault. I wonder how I turned out to be as paranoid and irrational about software... 8-)

I also used rsync -av Pictures/iPhoto\ Library/ /Volumes/Extern/iPhoto\ Library/ to copy all the data to an external harddisk. Just to be safe. ;) I heard rumours about the resource fork being used, and wonder I could use this directory to restore the data... Even though I am lazy by nature, I’ve become paranoid, so I’ll try it.

I also wonder how useful it is to run rsync when copying data between local discs. It seems to me that this would be only beneficial if writing data takes significantly longer than reading data. That might be true. But I don’t feel like measuring it. Well, actually I do, because now I decided to rsync the entire Pictures folder. It contains the iPhoto Library and iChat Icons folders. So I created a new Pictures folder on the extern disk, moved the 1.9G iPhoto Library into it, and ran rsync -av Pictures/ /Volumes/Extern/Pictures/:

Pyrobombus:~ claudia$ rsync -av Pictures/ /Volumes/Extern/Pictures/
building file list ... done
./
.DS_Store
iChat Icons -> /Library/Application Support/Apple/iChat Icons/
iPhoto Library/
iPhoto Library/.ipspot_update

sent 73505 bytes  received 60 bytes  147130.00 bytes/sec
total size is 2019142949  speedup is 27447.06

Nice! It was blazing fast. Thanks rsync!

Anyway, back to my test: I now renamed the iPhoto Library I had copied from DVD and copied the rsync copy from the external harddisk back into my Pictures folder. (Well, Claudia’s folder...) Restarted iPhoto. The number of images is correct. The thumbnail cache does not need to be rebuilt. It seems to work. Awesome! I picked an image I had rotated, and reverted it back to the original. It worked, too! Awesome!! :-D

You might be wondering why I’m doing all this if her pictures fit on a single DVD. Well, here MP3 collection does not... I have more backing up to do today!

Here’s what I’m using right now:

#!/bin/sh
if [ -z "$1" ]; then
    echo Missing volume name
    echo Currently available:
    ls /Volumes/
    exit 1
fi

if [ ! -d "/Volumes/$1" ]; then
    echo Volume $1 does not exist
    echo Currently available:
    ls /Volumes/
    exit 1
fi

for d in Desktop Documents Movies Music Pictures; do
        rsync -av ~/$d/ "/Volumes/$1/$d/"
done

Tags:

Add Comment

Comments

Gah, with three external disks for me and four external disks for Claudia (one of them dedicated to movie material from her camera), bringing the appropriate two from the office, running the backups (my laptop to two disks, my mini to two disks, Claudia’s mini and her movies to one disk) still takes about an hour. :(

AlexSchroeder 2006-09-29 07:54 UTC


Please make sure you contribute only your own work, or work licensed under the GNU Free Documentation License. See Info for text formatting rules. You can edit the comment page if you need to fix typos. You can subscribe to new comments by email without leaving a comment.

To save this page you must answer this question:

Please say HELLO.