I started this particular series of posts back in 2015 by detailing my photography backup routine and outlining some of the basic strategies for a safe and redundant backup system. If you’re uncertain about the basics, you should start with that post first.
Time and time again I hear stories of people losing their precious images, and it’s not just amateur photographers either, I’m constantly reading about professionals running into trouble too and frankly they should know better.
The biggest problem is that too many people still think of digital photography as being “free” because there are no film costs or costs to develop the photos. This is absolutely the wrong attitude to take though, because that puts your back up against the wall financially, when it comes to purchasing hard drives and storage arrays to keep your images safe. All of a sudden people shrink away from correct proceures when I tell them they need to spend a few hundred dollars on some hard drives, even when they have spent thousands of dollars or even tens of thousands creating the images. Not to mention spending life’s most precious commodity: Time.
Repeat after me: Digital photography is NOT free! Your images will be a lot safer when you factor in storage and backup costs while you plan your photographic spending.
Ok, enough of the ranting, let’s take a look at the changes I’ve made this year so far, and then I’ll lay out an overview of my current backup routine and setup.
2017 Change #1 – Cloud Backup!
This isn’t so much a change, as an addition to my routine, albeit a totally automated one. I’ve actually written a dedicated post about this already called Cloud Backup for Photographers, but the gist of it is that I’m now using CrashPlan to keep a cloud backup of the contents of my computer’s hard drive. My upload speeds in semi-rural Canada still prevent me from being able to keep a cloud backup of my photo archive, but after 6 months of testing CrashPlan on my main computer, it’s definitely here to stay. I have it set up so that it only backs up to the cloud at night while I’m sleeping, and whilst it took about 4 months (!!) to do the initial upload, I’m happy to pay the $5/month fee. I must admit that before researching this at the tail end of 2016, I had no idea it was so cheap! I felt kind of stupid when I found out. Why would you not be using this service when it costs less than the price of a Starbucks coffee for unlimited backup?
If you aren’t familiar with the CrashPlan concept or service, and want to know why I chose CrashPlan over Backblaze (and which you should choose), then I’d recommend you take a look at the aforementioned cloud backup post. This doesn’t replace any of the other layers of my backup routine, it sits alone as a totally new layer of protection. In the past, my computer’s internal drive was backed up using Time Machine, and I also created a bootable clone using Carbon Copy Cloner. Those are both still in place, but CrashPlan now serves as an offsite copy of that internal drive data, something that was previously missing as my offsite backup only included my photo archive.
2017 Change #2 – Drobo 5D3 with Thunderbolt 3
When Drobos first launched there was a 16TB limit on the size of a single volume. If you added drives that gave it more drive space than 16TB, it created a second volume and therefore showed up as two hard drives when plugged into your computer – not really what you want! At some point in the last couple of years this limit was upped to 64TB with a firmware update, but the kicker was that you had to reformat your Drobo to set it up. In order to do this, I’d have to move about 14TB of data into a temporary location while I performed the formatting, and I simply didn’t have a good way to do that. My Primary Drobo was about 90% full coming into mid 2017 and I knew this this was probably a good opportunity to update my primary backup solution to something larger, and faster, and get around that 16TB limit I was facing.
I wanted a future proof solution so Thunderbolt 3 looked like the way to go. There’s no speed benefit of the TB3 interface because mechanical hard drives are still saturated well before theoretical TB3 speeds, but as TB3 units are also the newest units, they often have other new tech or processors in them that do offer some benefits.
At this point I still don’t have a computer that has Thunderbolt 3, but I’m sure I’ll have one within then next 12 months and since they are backwards compatible it just made sense for me to look at the newest tech, with an eye to using this primary storage array for the next 4-5 years.
I was just about set to buy a G-Technology G-Speed Shuttle XL with Thunderbolt 3, which would have given me 20TB of storage in a RAID5 configuration, when all of a sudden Drobo launched the 5D3. The 5D3 is a Thunderbolt 3 enabled version of the 5D and it’s priced much more competitively than the G-Speed Shuttle. In fact, it would save me well over $1000 if I went with the Drobo 5D3 instead. I’ve had mixed feelings about Drobos in past years, and previously mentioned my intentions to phase them out of my setup, so it came as a bit of a surprise to me when I so readily purchased a 5D3 the very same day I learned they had been released!
I still have a desire to set up a NAS device in the office for remote file access, but I shelved that plan temporarily when I started to realize how complex Synology and QNAP NAS setups can be, and I wasn’t willing to deal with that, for what amounts to only a very occasional need to access files remotely. I may revisit the idea of a NAS at some point, but at this point it became more important to replace my primary DAS (Direct Attached Storage).
The more research into storage arrays I did, the more complicated things started to get. I began to realize just how good the Drobo 5D units are for people like me, who simply don’t have the time for complexity in yet another part of their work. Going down the route of getting another Drobo meant I could gradually fill the unit with whatever brand and size of drives I wanted, rather than having to shell out for a whole set of identical drives right from the get-go, as you have to do with a RAID array such as the G-Speeds. This is where much of the cost saving comes from, but it also allows me to expand to much larger sizes for a much lower cost over time. Couple that with some early, favourable speed tests on the 5D3 from other reviewers, and there was just too many things in the “pros” list for the 5D3 to ignore. Heck, they even lowered the price on them compared to previous models!
When B&H Photo had HGST 8TB drives on sale the same day I ordered the new Drobo, I knew it was meant to be, so I grabbed 4 of those 7200rpm drives and set up the new Drobo 5D3. It has 5 drive slots, but for now, the 4 x 8TB drives give me 21.7TB of useable space which is plenty. I’ve no doubt that the 5D3 is going to be a little slower than the G-Tech RAID array I was also considering, but I’m not editing a lot of 4K video, and this new Drobo is still over 5x faster than the ones I was previously using! In other words, it’s still lightning fast – more than fast enough for photographers working with large files in my opinion.
Time Machine Location Change -> Primary Drobo
I used to use a separate drive as the Time Machine backup for my Mac, but I’ve now switched to using the primary Drobo 5D3. When my old Drobo 5D was maxed out as a 16TB volume (see note in above section), I didn’t want to take up any of that space for Time Machine. Now that my new 5D3 can handle up to 64TB I’m using the built-in Drobo functionality to partition off a section of the internal volume for Time Machine usage. This is awesome! One less drive on my desk and one less drive to worry about failing. Since Time Machine now lives in my primary Drobo, it also has redundancy against drive failure, which it didn’t before when I was using a single external drive. A simpler setup now, but also one that offers more protection. Win, win!
Offsite Backup Change -> Old Drobo
My offsite backup was previously a 12TB WD MyBook Pro (still GREAT value). This was dangerously close to being full, so I’ve taken my old Drobo 5D that used to be my primary storage unit, and moved that into the role of offsite backup device. I took the opportunity to reformat it and update the firmware to allow up to 64TB of volume size, and now I have an offsite array that I can expand with larger drives as needed. The Drobo 5D is a little long in the tooth in terms of speed when you compare it to my shiny new 5D3, but it’s only used once every few weeks when I bring it to the office to perform a sync with my primary storage array. It can sit and do that overnight, so frankly I don’t care how fast it is as long as it works.
For the Love of Simplicity
I tend to be the kind of person that does a lot of research into products before I shell out my heard-earned cash for them. I easily spent 30 hours researching QNAP and Synology NAS units, as well as the ins and outs of RAID5 vs. RAID10 for DAS and NAS arrays. What I really began to realize is that I have far more interesting things going on in my life than learning how to set up all of these kinds of devices. I know I could do it, but I just don’t have much inclination to learn about it any more. I’d rather be outside with my camera! This is the point at which it really began to dawn on me how great my Drobos have been over the last few years. No, I still don’t agree with all their repair policies but damn if they haven’t saved me a hell of a lot of time since I’ve been using them. They are just so much simpler than anything else on the market, and the products seem to improve in both performance and stability with every new release.
I’ll be the first to admit that this was not the conclusion I expected to come to when I first started to research possible new primary storage arrays, but when I did the math on the pricing, and kicked myself for wasting so much time delving into the intricacies of other products that would give only a minor performance boost, the Drobo 5D3 just made the most sense.
My 2017 Backup Routine
With these new changes in place, here’s a rundown on my current backup routine.
Primary Photo Storage – Drobo 5D3
Photos are downloaded from memory cards using Photo Mechanic and placed into my primary photo archive on the Drobo 5D3. The 5D3 is a redundant array using Drobo’s BeyondRaid technology, so if a drive fails in the 5D3, I can simply slot in a replacement and no data is lost. This is my first layer of redundancy.
Secondary Photo Storage – Drobo 5D
This remains the same as previous years. Every night, Carbon Copy Cloner mirrors the primary storage onto a second Drobo which happens to still be an old 5D. Since this unit only gets used as a backup, and never worked from, I’m not worried that it is the slower, older 5D model. Again, since it has redundancy built into it, a drive can fail and no data is lost. Even if a drive fails, it can still perform its backup duties that night whilst simultaneously rebuilding the replacement drive that you put into it. Even when a drive fails in a Drobo, you still have continued access to all your files during the rebuild process.
The primary reason for having two Drobos containing the same data is to protect against hardware failure of the Dobo itself. If the Drobo 5D3 unit fails, even if all the drives are fine inside it, this would knock me out of action for several days while warranties or replacements are organized. I can’t afford for that to happen, and I learnt this lesson the hard way many years ago. It also protects against file corruption because a RAID (or in this case a BeyondRaid system) is not a backup. Period. Read this is you don’t know what I’m talking about.
Offsite Photo Storage – Drobo 5D
This additional Drobo 5D lives at a friend’s house. After a big job, or several weeks of shooting, I bring it home and plug it in to the computer. Carbon Copy Cloner immediately jumps into action and mirrors the Primary Photo Storage 5D3 onto this 5D, then it goes back to my friend’s house.
This protects agains fire, flood or theft of the Primary Storage 5D3 from my office location.
Internal Drive backup – Time Machine
Time Machine backs up the internal drive on my Macbook Pro 15″
Internal Drive backup – Offsite with Crashplan
CrashPlan backs up my internal drive to the cloud. This protects agains fire, flood or theft of my Time Machine drive.
Internal Drive backup – Bootable Copy
Time Machine is great for getting back a single file that you may have accidentally deleted, but if your entire internal drive dies, it takes time to replace the drive and get back up and running. I use Carbon Copy Cloner to occasionally create a bootable clone of my internal drive onto a G-Technology ev Raw drive. If my internal Macbook Pro drive fails, which has happened before, I can instantly boot a mirror of my computer from the clone and be back up and working in a couple of minutes. Of course the internal drive would still need to be replaced, but this means I can take the time to do it at a convenient moment.
This setup has me covered for some pretty serious hardware and hard drive failures. Even if a drive fails in my primary Drobo, and a second drive fails in the same Drobo before it is rebuilt with a new disk which would cause a total loss of data, I still have a redundant copy on site in the backup Drobo 5D AND another redundant copy offsite in the second 5D. In other words, I could lose two drives from one unit simultaneously, and my office could go up in smoke ten minutes later and I’d still be fine, even if a third drive then failed in the offsite copy. It’s a pretty solid setup!
On top of that, all my day-to-day personal and business files are backed up onsite and offsite with Time Machine and CrashPlan.
I haven’t changed this setup in a few years. When I travel I have a pair of G-Tech ev Raw drives labelled Photo 1 and Photo 2. Images get copied onto both drives during memory card ingestion with Photo Mechanic. The drives are stored in different locations and whenever possible I try not to format memory cards until I’m back home, leaving me with three copies of my images while I’m away. If I get that once-in-a-lifetime shot, I stick it on Dropbox too.
Changes to Come…
I’d still LOVE to automate my offsite photo archive backup somehow. Any suggestions? I can’t use typical cloud storage services because the initial upload of 12TB of images would take about two years with my upload speeds.
15 thoughts on “My Photo Backup Routine – Mid 2017”
Have you kept your old offsite back up drive? I’m always most nervous when my office back up is onsite being updated. I’m sure if there was to be a fire that would be when it happens.
I am considering Amazon AWS Glacier storage for an extra layer of offsite backup. Not sure how I will use it yet though.
I’ll probably get rid of it because it was about full anyway. Having your offsite backup onsite for a few hours is a concern for sure. In an ideal world I would rotate a matching pair of offsite drives I guess, but I minimize the risk by performing the offsite sync as soon as I bring the drive home, and then I literally don’t leave it alone until it’s done and removed from my place. The sync is incremental, so it usually only takes an hour or two.
Glacier would be cool if I had more upload bandwidth but as it is, I’m maxed out with CrashPlan at night anyway. Another thing that interests me for the future is DroboDR. This is a features that mirrors two Drobo 5N2s, one on and one offsite. It might be the first version of such a system that I think would be simple enough for the layman (me) to use, and it would also give me remote access to my files too. It’s likely I’ll look at swapping out these two old 5Ds for 5N2s at some point, and putting that offsite automation in place.
I’m really so impressed lately by the simplicity of the systems that Drobo have created. They have added some seriously powerful features in such a user-friendly way.
For the offsite backup that where synology can help..
They can create a backup in a similar network drive somwhere else, from a manual command or automatically..
Hey Sylvain! Yeah I’m definitely looking at this. One concern for me is that we have limited bandwidth for most services here in Canada. If i put the offsite unit at someone else’s house, I don’t want them to incur bandwidth overage fees because I’m dumping gigabytes of data on them every night :p
Be careful if you bought four identical drives in that B&H sale. It’s well established practice to mix drives from at least different batches when building arrays, and even different manufs where you can identically match the platter and track count. If the batch turns out to be dud such as if a drive dies vastly under it’s MTTF, there’s a good chance the identical drives next to it will be only days if not only hours behind. I’ve experienced this myself and suffered huge data loss despite all the “proper” redundancy in place. With your multiple backups and offsite you’d never lose everything but you could lose every disk in your primary array at once, effectively rendering the disk redundancy useless.
As a note, I am using multiple different size disks from multiple manufs exposed as a singular network volume using mhddfs on an OMV (OpenMediaVault) NAS homebrew box, with nightly delta (rsync) to backup disks, and plug-in-detect sync for monthly offsite backup device. Also been pondering S3 or Glacier for archiving. Am also a massive IT nerd and Linux fan, which helps.
First comment: I didn’t think about the drive thing you mentioned. I wonder what their serial numbers are… this is definitely a good thing to think about.
Your second comment: Haha! I have NO idea what you just said, which is exactly why Drobo is great for me.
Thanks for chiming in, Dom!
Hi, nice write up! I have a somewhat similar setup with a homebrewed synology and Crashplan (and Crashplan is also running headlessly on the synology, too). One thing to consider with the offsite backup, is that Crashplan charges for uploading to their servers, but you can also sync data to other machines that are also running Crashplan (for free), such as a home/office or friend to friend setup. You could seed the destination drive(s) locally to get started. If you have somewhere else with a computer/server running all the time, it could be just what you need. Here’s more info: https://support.crashplan.com/Backup/Backing_Up_To_A_Friends_Computer
Also, make sure you have Crashplan set to encrypt the data locally before it transfers it!
Excellent tips, Brian! Thanks for chiming in. I’m going to take a look at that Crashplan friend sharing thing in closer detail.
Dan, great article. I just did tons of hours of research debating as to NAS vs DAS to manage primarily my photos and videos. I realize I need a system like years, after a recent internal HD crash, and recognizing my back up system was incredibly disorganized. I debated between the Pegasus 3 and the Drobo 5D3. I noticed that the sales people at Promise seemed to know a lot more about the details, and Drobo folks were inferior. This concerned me, however their website is informative. I bit the bullet, and ordered the 5D3 for its flexibility with drive sizes, and opportunity to growth without having to transfer my files off it and then rebuild, if I expand and add a drive. So to start have 2 10 TB 7200 drives. A new iMac, and old iMac. 2 questions: You mentioned that you don’t agree with Drobo’s repair policies, what exactly are your concerns, and any other cons. I also read somewhere that the 5D3 will be faster with a T3 active cable versus the included cable (they state is passive…20gb per sec.). I could not find this information on there website, nor did anyone respond from my email query to the company as of yet. The Promise website stated their P3 does come with a T3 cable, at 40gb per sec (again seeing how detail oriented this company is compared to Drobo – better for the OCD Tech Nerds! So happy I found your post!
I must say that I don’t even know the difference between a passive and active cable. Can’t help you there! Seems odd… I don’t really believe it. Th 5D3 doesn’t even come close to saturating the Thunderbolt throughput anyway. You’ll never get 20gb sec.
As for repair policies, i mentioned that because once I had a fan fail in a Drobo 5D. It’s a $5 part, but they didn’t offer any way to repair it. Their suggestion was that I buy an entire new unit. They said they don’t offer repair services. Take that for what you will, I think they can do better than that.
HI Mary, I was also considering the Pegasus3 R4 for awhile. I actually found the customer service at Promise Technology to be inferior to Drobo which is just the opposite of what you experienced. I have finally decided to go with Drobo though, because of the ease of upgrading to larger storage when the need arrises. Just as Dan has said, in a traditional RAID 5 config, you would have to migrate ALL the data off the drives, put new larger drives in the array, format and build the RAID 5, then migrate all the files back onto it. I simply don’t have a place to even offload all that data, let alone the time to do it. My initial concern was the speed differences of the P3 versus the Drobo 5D3… there is NO comparison. The P3 blows most systems away. Then I have read some reviews that seemed to indicate the 5D3 will be plenty for my needs. I will be using it as a primary working drive in which I will be reading and writing from rather than my internal SSD. WHY would anyone do this, you might ask. Well, the simple truth is that Lightroom is fickle and I don’t want to be moving files back and forth from one drive to the next and possibly lose the catalog connection to Lightroom. So yes, I’ll be operating everything from the Drobo and speed is a concern. Anyway, I have yet to get my unit and am cautiously hoping for it to be what I need. I’m nervous, but also excited that I chose what I did. I was put off by my experience with Promise Tech and wish perhaps that I had your experience with them. On the other hand, I was pleasantly surprised at my workings on the phone with Drobo, as well as talking fact-to-face with the lead product guy at IMAGING USA last week in Nashville. I do know that Drobo has a new CEO (for a couple of years now) and changes are being implemented for the better. He has come into the company and cracked some heads… lets hope that the whole repair policy thing gets resolved, too. Cheers!
Thanks for adding your comments to the conversation! Really great to get such detailed info from other readers!!
Have you looked at AWS Snowball or storage gateway. Given your data size and rural location snowball seems like the obvious solution?
First I’ve heard of it. It looks to be a pretty serious service for bigger businesses than me, with some big costs involved if you want to do regular backups. Do you use it? I can’t see myself choosing to pay hundreds of dollars and ship a device from rural Yukon to Amazon, instead of driving 30 minutes to my off-site storage location to make the swap. I’d have to drive the snowball the same distance just to ship it (no collections here). it’s an interesting service I didn’t know about, but I think it’s beyond the scope of most photographers.