Data Security

[David Paterson]

In my "200-500mm first results" thread, Frank wrote -

------------------------------------------------------------------
"That is so not true. I 1996 I reported for c't magazine about data storage in Banks and Broadcasting. It works reliably.

Think of having your files stored unchanged. Bit for bit.
Do not think about the media, media fail.
Do not organize it yourself.
Store the data in a reliable professional data center.
pay a monthly fee.
Keep your access and encryption keys on different physical media in different places.

That is data storage."
--------------------------------------------------------------------

I am sure such systems exist everywhere - for the military, for the government, for banks and financial institutions, etc etc and they may well be available to the public for a fee, but that is not what I was talking about. And not what I want. Surely what we need is a desktop solution to the problem of secure and long-lasting data storage, and if not desktop, then at least within one's home or office. Unfortunately this would involve "media" of some kind - my point was that such media need to be built to last. It seems probable that "hard" systems already exist - you do see mention of "military-grade" drives occasionally, and there will be other devices. Price may be a factor, but after more than 2 weeks of constant work restoring my systems, software and (non-photographic) data, I am still a long way from where I was on 12th December. If I costed my time at a modest rate of say €30.00 per hour, this breakdown has already cost me about €3000 plus all the hardware and software I have had to buy, to ensure some sort of restoration and continuity. If a really permanent solution to  secure personal data storage was available, I'm sure many professionals (not just photographers) and others would consider it very carefully.

My disk failure became an almost-catastrophe because several factors came together to create a perfect storm (the numbered items are the crucial ones) :

1. The system disk was SMART-enabled but made no reports of problems.
2. Regular inspections using Disk Utility and the fsck routine found no problems. 
    Nevertheless, I knew there was a problem - strange behaviour, hangs, freezes, very slow, etc.   
    I bought and installed a new 1TB drive, intending to install a complete Time Machine backup on the new drive.
3. Partitioning the new drive went disastrously wrong, leaving the drive completely inaccessible to both Disk Utility and Drive Genius.
    I contacted D. Genius who said their latest version could definitely fix this problem. I paid for a download of the newest version.
4. Drive Genius's download system would NOT download the software. Multiple attempts and email exchanges with those Geniuses all failed to bring me the software.
5. The system-drive crashed, terminally.
    I removed the broken drive, dumped all the data on to an external drive from another internal drive, and tried to install the Time Machine backup on this drive.
6. Time Machine failed. There are various ways to approach a TM restore;I tried them all; I ran Disk Utility over the TM drive, and tried the restore again, several times.
    Time Machine failed completely.

I gave up all attempts at machine-led restoration, installed OS 10.6.8 on the now vacant internal drive and started piecing together as much as possible of my missing software, documents,  emails, e-addresses, contacts, etc etc, as I could, from whatever sources were available.

Yesterday, Drive Genius set up a direct link to allow me to download their software, two weeks too late.
I fixed the unusable 1TB drive with its faulty partitioning, and re-partioned it into system and data partitions.
I again ran Disk Utility over the Time Machine drive several times, and tried to restore on to the 1TB drive.
TM failed again.

This is where I am today.

[ColinM]

I feel your pain Dave. I've previously had standalone RAID disks which have failed and required me to become familiar with Linux to recover the data (probably poor choice of gear by me).

Any reason you can't use a reliable cloud storage option?

This will give you an offsite backup, removing risks of things like flooding or fire that could render a home backup unusable.

[pluton]

In the 200-500 thread, MFloyd stated
 "(1) use TM the way it is build i.e leave it on, and don't use it from time to time (2) always put (at least) a second backup strategy in place e.g. every month (or less) I take a CarbonCopy clone on HDDs (copies which are then safely stored away: fire and theft proof). This way, I was kept whole from major disasters, all these years."
I specifically do not use Time Machine continuously, due to early concerns(well-founded or not) that a constantly updating backup might not work well with Lightroom's constantly updating database.
Recently, I abandoned the additional "full clone" copy of the computer that I had been maintaining, and instead created a second Time Machine drive, since on those few occasions when I have had to restore the complete computer from backup,  Time Machine has been the more complete and trouble-free "one-stop" way to go.
Of course, I am aware that Apple could make a few software mistakes and render Time Machine temporarily crippled or useless.  I seek to avoid that scenario by not updating software without waiting for user reports. 

[Frank Fremerey]

I use  the Microsoft cloud in addition to my private off site storage and Google Cloud.

The principle is "all data on two separate physical devices any time."

Continued:

1) Come back with full cards from shooting

All my cards together are now big enough for two weeks shooting. On Cameras with two drives, use two identical cards creating two copies.

2) If possible copy one copy to HD of Laptop on the way, so now it is three copies I can store seperately when not shooting on the road.

3) At home create one consistent body of work in one place with no doubles

4) Backup the drive


• I never store data on the same drive I store programs or OS.
• I have two SSDs for the system, so I can easily take a fresh current system from the cupboard whenever I need it and I am back in business after 2 Minutes.
• Additionally I have current drive images that live on the data drive. I use "Acronis True Image".
• All backups are done using Beyond Compare "mirror left to right".
• Data live on identical HDs, the last few months here, the time before one copy here, one copy at a friend 10 Kilometers away.
• As reliable drives I recommend "WD green", which are now called "WD blue". I never experienced any problem with these drives in many years. running slow & cool.
• I strictly disregard RAID systems for home use.
• Ask Bjørn Rørslett how many drives he lost, I think it is more than 200 drives. IMO that has a lot to do with his using RAID systems.
• I lost one drive out of hundreds since I started using computers in 1992. It was a rare drive failure due to a firmware issue.

Why?


• A RAID System is a good solution if you want to combine cheap slow disks to for a virtual big fast disk.
• So in a video production unit RAID is a good thing, in a broadcasting unit RAID is a good thing.
• In these systems RAID is used to edit or transmit content that is also safely stored somewhere else.
• RAID is a good thing for any fast production unit, but RAID actually adds more points of failiure.
• My backup drives are nearly never online (every two or three days for a few minutes)
• RAID drives used for backups are often online 24/7, so they are subject to any power issues, software issues and internet issues
• You have to backup the RAID, it is only a drive with different properties, but still all together one drive that needs another sinmilar system for backup. As a whole a RAID system is not more secure than a single drive, it also has a controller, firmware, has its own power supply and many more points of failiure


I never use Apple because I feel the system is intransparent
I would use LINUX, but I like a lot of software that is not available for LINUX

[simato73]

This year I have stepped up my data security strategy. I just stopped short of off site backup.
As a first safety net I have Time Machine running.
In addition I have a clone of the boot drive being updated daily using Carbon Copy Cloner.
Other drives containing photos and other media are also backed up regularly with CCC (in addition to having a TM copy).
My photos are spread in a 1TB SSD (recent pictures) and a 4TB partition of a RAID 5 system (photo archive).
The RAID 5 is built using SOFTRaid, which provides constant SMART monitoring and hardware diagnostics that goes well beyond what Disk Utility can do.
It has correctly predicted a disk failure recently and rebuilding the array has been a smooth operation.

I am feeling very comfortable with the multiple safety net I have in place and would recommend it - it is actually not very expensive to run.
The only thing missing is offsite backup, but in case of a major accident to my home the lack of photos would be the last of my problems. Besides I am wary of "cloud" storage.

[Bjørn Rørslett]

Frank overestimates my disk device loss rate by nearly an order of magnitude. Also, his focus is on the wrong end of the stick: actually, thanks to pre-emptive storage schemes and RAID arrays, my actual data loss over the last 20 years is definitively negligible, perhaps a handful files at most out of millions of file instances. Most of these also occurred in the earlier stage before I had fully automated the backup strategies.

[ColSebastianMoran]

Losing data sucks.

Here's what I do:

I assess my risk profile this way (decreasing order of possible loss):
 - House broken into, everything stolen. Or house burns down.
 - Hardware failure of a hard disk or system failure that kills it
 - Industry changes: key software I use disappears, older versions of RAW no longer supported, etc.
 - Operator error (overwrite or erase a file)

Strategy:
 - Apple Time Capsule backup running all the time over WiFi. Fabulous for retrieving earlier versions of a file.
 - Carbon Copy Cloner hard disk images created every couple weeks. I have two backup disks: One locked in a file cabinet in my home office, the other at the bank in a safe deposit vault. At least once a month, I go to the bank and switch the backups.
 - Photos start on a memory card in a camera. I download and process the images. But, I leave the originals on the camera cards, switching cards when full. Cycle a half dozen cards so I am sure to have images on cards long after they have been backed up.
 - After post processing, save a full resolution un-sharpened JPG of the final rendering on all selected images. Save a 16bit tiff for very special images. My kids can go back to these jpg's and tiff's for 50 years no matter what happens to Lightroom or Nikon.
 - New hardware every three years. Especially the laptop, also the backup hard drives.

[David H. Hartman]

Correction: DISKPART.exe is a utility found in the C:\Windows\System32\ folder of my Windows 7 computer. It's been a long time since I've used DISKPART. It's not a command built into CMD.exe as implied below. CMD.exe is also fond in the C:\Windows\System32\ folder. I strongly recommend disconnecting all data drives before using DISKPART.exe

A note concerning a HD petitioned and formatted for a MAC may be unusable for Windows. The problem is Windows recognizes a partition but can't delete it. I had this problem with two 1TB G-Drives. After some research I used DISKPART from the command line processor running in Windows, CMD. CMD is a command line interface like MAC's Terminal. With DISKPART you type the command and add arguments. Even DISKPART would not delete the MAC partition until an additional argument was added that forced it to proceed.

CMD and DISKPART would scare the hell out of most computer users who had not used a command line processor and command line partitioning software. If I had, had a MAC at the time delete the offending partition would have been easy. The drive came with no utility for Windows to remove the MAC partition. Windows Disk Management rightly would not delete the partition as it might contain valuable data for another operating system. As it was the partition was empty and ready for MAC's Time Machine or other data storage.

Just to toss it out Windows DISKPART can delete an unknown partition so Windows Disk Manager can partition a HD for Windows. Just be sure there is no data that is important on the partition that's being deleted (which may not be possible using Windows).

Dave Hartman

[Frank Fremerey]

Frank overestimates my disk device loss rate by nearly an order of magnitude. Also, his focus is on the wrong end of the stick: actually, thanks to pre-emptive storage schemes and RAID arrays, my actual data loss over the last 20 years is definitively negligible, perhaps a handful files at most out of millions of file instances. Most of these also occurred in the earlier stage before I had fully automated the backup strategies.

Happy to hear it is not as bad as I remember from our last discussion on that topic here last year.

Wish you all a happy transition to the next year. 02017 full of plans and hope and hopefully success!!!

[John G]

Amazon offer a 3 Tier Storage Service,
5gb free,
S3 offers $0.03 per month/ per Gb, with a uncomplicated data retrieval set up.
Glacier offers $0.01 per month/per Gb, Data Retrieval is charged for and Glacier has a more complex data retrieval set up, this is for long term storage, not intended to have frequent retrievals, there are reports of complications with large retrievals, but the knowledge of how to work with Glacier retrievals is getting more common on the net.
Amazon is a Service that seems here to stay, so the idea of finding your cloud service disappear is not likely.
Please feel free to post your thoughts on this as a solution, as we all want valuable data storage support.     

[John G]

There seems to be a mixed bag of calculations, when it comes to Glaciers Data Retrieval costs.
I have shown this pricing scheme, as it seems more realistic in terms of how much data a individual might require to extract with a heavy workflow. 
Below is a entry from a forum, attempting to help explain Amazon Glacier Data Retrieval pricing.


Re: Price to restore data...
Posted by:   Colin@AWS
Posted on: Aug 21, 2012 7:53 AM
in response to: S. Bouvard in response to: S. Bouvard
    Click to reply to this thread   Reply
Hi,

A few clarifications in your example. The peak retrieval is calculated based on the retrieval time, not the download time. Your data would be retrieved in approximately 4 hours and so therefore your billable peak hourly retrieval would be 30GB/4Hr = 7.5 GB.

Per the example in your link with 12TB stored you could retrieve 20.5GB for free every day. Assuming that you did no other retrieval operations that day the charge would look like this:

Billable peak hourly retrieval = 30GB/4HR = 7.5GB
Free daily retrieval amount = 12TB * 0.05 / 30 = 20.5GB
Free retrieval applied to peak hour = 20.5/4 = 5.125GB

Billable peak hourly retrieval = 7.5GB - 5.125GB = 2.375GB

Retrieval fee = 2.375GB * 720 * $0.01 = $17.10

As this is based on a peak monthly retrieval rate, there are further opportunities to reduce this cost. If you spread the retrievals across 2 days it would cost nothing as 30GB/2=15GB is less than the 20.5GB daily free retrieval amount in this case. Similarly, if you were to spread the retrievals more smoothly across the day, lets say over the course of 24 hours the bill would look like this:
Billable peak hourly retrieval = 30GB/24HR = 1.25GB
Free daily retrieval amount = 12TB * 0.05 / 30 = 20.5GB
Free retrieval applied to peak hour = 20.5/24 = 0.85GB

Billable peak hourly retrieval = 1.25GB - 0.85GB = 0.4GB

Retrieval fee = 0.4GB * 720 * $0.01 = $2.88

[Hugh_3170]

Don't forget that many HDD failures are due to the controller boards failing, whilst the disk and spindle are all OK  -  about 2 out of 3 failures according to some sources are due to the controller boards themselves.

Here is a Canadian outfit that I have had good results from that can rebuild most controller boards:

Link: http://site.hdd-parts.com/home

[Bjørn Rørslett]

That might be a life(data)-saving link, Hugh. Thanks 

Most issues will be alleviated if you keep in mind all disk drives *will* die, and take the necessary mitigating steps from this sad fact.

Don't for a second believe transitioning to SSD will mean your data is safe for eternity. On the contrary, SSD longevity at present is a black hole of uncertainty and might be far shorter than that expected of spinning disk.

Also, the practice of only powering disks on for a short time to do data transfers might shorten their life instead of prolonging it.

In short, disk storage is volatile. Thus redundant storage is the key to any long-term storage scheme.

[MFloyd]

Bjørn's summary is what's all about. Network backup for image storage is not workable from a cost viewpoint; I use Apple Cloud storage within its small scope i.e. its saves the Bureau and the Documents folder where all non-image related data is stored; together with ICloud recovery data for my other iDevices: iPad, iPhones. All this goes into less than 50GB and costs me 1€ / month. The physical disks go into the house's nuclear shelter which is mandatory in Switzerland and which is fire proof, and, off course, NRBC proof, the advantage being that it is only a few steps away.

[Bjørn Rørslett]

Correction: DISKPART.exe is a utility found in the C:\Windows\System32\ folder of my Windows 7 computer. It's been a long time since I've used DISKPART. It's not a command built into CMD.exe as implied below. CMD.exe is also fond in the C:\Windows\System32\ folder. I strongly recommend disconnecting all data drives before using DISKPART.exe

A note concerning a HD petitioned and formatted for a MAC may be unusable for Windows. The problem is Windows recognizes a partition but can't delete it. I had this problem with two 1TB G-Drives. After some research I used DISKPART from the command line processor running in Windows, CMD. CMD is a command line interface like MAC's Terminal. With DISKPART you type the command and add arguments. Even DISKPART would not delete the MAC partition until an additional argument was added that forced it to proceed.

CMD and DISKPART would scare the hell out of most computer users who had not used a command line processor and command line partitioning software. If I had, had a MAC at the time delete the offending partition would have been easy. The drive came with no utility for Windows to remove the MAC partition. Windows Disk Management rightly would not delete the partition as it might contain valuable data for another operating system. As it was the partition was empty and ready for MAC's Time Machine or other data storage.

Just to toss it out Windows DISKPART can delete an unknown partition so Windows Disk Manager can partition a HD for Windows. Just be sure there is no data that is important on the partition that's being deleted (which may not be possible using Windows).

Dave Hartman

Strange. The Disk Management module of computer Management will happily remove partitions of any disk, be it created by Windows, Mac, or Linux. I have done that numerous times so know it'll work. Besides, it has a graphical UI too if the command line is an obstacle.