some tools I found in my room

[Björn Carlén]

Some day I'll shoot my collection of crashed hard disks - been stacking them for a year now and there must be at least twenty in the stack. ...

Twenty crashed hard disks – that's quite impressive. I myself recently had an SSD that failed totally and permanently, for no apparent reason. Does that sound familiar to anyone? It was a Samsung 120 G from the 840 series.

[David Paterson]

It is a predictable fact of life that hard disks fail unpredictably, but always at the worst possible time. Thus, build your data storage routines to be able to handle a sudden loss of data in a manner such that you avoid a permanent loss of data.

I am actually at the moment trying put my archives and back-up archives into some sort of order, and to establish a sensible routine for backing up; until very recently this has all been rather haphazard and "when I remembered" - not a good idea. Last week I suddenly discovered I had no back-up at all of any of my 2015 image-files ; even the raw files were on the same drive.    

Frank - my very first CF card was 340mb Kingston, for one of the Nikon Coolpix 5000 series - I forget which one (maybe the 5200) - which I absolutely hated and which did NOT start me on the way to being a digital shooter (a year or two later the D70 did that).

[Bjørn Rørslett]

Björn: that is twenty disks within the last 24 months ... About 1 disk crash per month on average. Never assume anything about the longevity of any hard drive, be it an old-fashioned "spinner" or the modern SSD variant. I have had disks fail within 1 hour after installation.

[Bjørn Rørslett]

And lest I forget: do purchase *new* spare disk(s) of the same type or storage capacity IMMEDIATELY whenever a disk fails. I now get two new ones each time I encounter a crash. Thus, I try to keep a shelf with enough spare disks to keep all my storage units running for years to come. Perhaps having a store of 20-30 replacement disks seems superfluous at present, but just wait 1 year or two when replacements no longer might be available and your RAID array has crashed again ...

It goes without saying that in addition to RAID arrays, you also need independent backups.

[Gary]

Björn: that is twenty disks within the last 24 months ... About 1 disk crash per month on average. Never assume anything about the longevity of any hard drive, be it an old-fashioned "spinner" or the modern SSD variant. I have had disks fail within 1 hour after installation.

I cannot remember when I've had a hard drive fail. Maybe ten years ago.

[Frank Fremerey]

I myself recently had an SSD that failed totally and permanently, for no apparent reason. Does that sound familiar to anyone? It was a Samsung 120 G from the 840 series.

My routine says: never save data on SSD.

current systems live on mirrored SSDs like Windows on a 250GB 850Pro mirrored to a 840Evo
on the shelve. Ditto for Notebook. Current image of SSD automatically logged by Acronis.

Only a few weeks ago an Intel120 from the 330Series died without prior warning. Intel collected
the Item and replaced it with a 180GB from the 520Series free of charge.

All data are on mirrored HDDs with older Backups 20 km away to have my archive save too.

Any significant job is backupped to at least two physical drives before I start editing.

[Frank Fremerey]

I managed my way trough lots of HDDs.

Since I switched brand to Caviar green by WD I have had Zero disk fails. 50 disks. About 10 years.

As IT journalist and ex member of a renowned test lab I can only
inform any interested party to view a RAID system as one. One
single big HDD with multiple points of failiure.

The combination of system SSD and Data Grave huge and fast
HDD has made RAID systems obsolete except for some very special
production environments in Broadcast where they are not used to
store data but as availability device.

so unless you broadcast 4k or 8k signals and need the matetial
for the next programmes online fast you are imo not in a situation
that justifies RAID

[Bjørn Rørslett]

I strongly beg to disagree to such a blanket statement.

RAID 0 has no redundancy and can only provide a speed advantage. All other RAID arrangements offer data redundancy (unlike a single drive) and thereby added real-time data protection. That is a fact.

Except for a simple  RAID 1, all other RAID arrangements provide a larger storage capacity, many times more than any single disk can give. That is also a fact.

Not adding any redundancy to one's storage scheme is very unwise and over time you are certain to lose data. Another fact.

As to the failure rate, no brands or product models are exempt to the rule that all disks fail at some point in time. Yet another fact. Use hard disks by the hundreds or more and this is uncomfortably easy to observe.

While the failure rate of SSDs might be lower than that of the traditional hard disks, they are subject to failure by wearing out. You actually ought to set aside a sizeable part of the disk capacity to help the disk controller to distribute write wear more optimsal and thus extend SSD longevity, but in my experience few bother to perform such basic measure of prevention. Do note that very few SSDs are warranted for more than 3 years. That is a bit of information worth some thoughts.

[Frank Fremerey]

Bjørn. Your statement contains zero contradiction to mine except for the fact that you seem to trust RAID and SSD
more than I do.

The data structure of a RAID like say 5 or 6 type array is made to compensate for single or even dual disk failiures
which is kinda nice.

All the disks are managed by one controller wich might fail by many reasons and render all disk contents useless.

Even two Disks run in parallel offer more safety because they are still controlled independently.

If you choose to drive the two disks behind independent uninterruptable power systems you are in safety heaven.

To get anywhere near this level with RAID you have to have a backup RAID for every RAID.

That put we look back into history and ask: what do we need a RAID for?

1) make big disk from smaller disks and compensate for extra failiure points by redundancy

2) make faster disk from slower disks and compens....

When I wrote about this topic in c't we had data rates of 2MB/s at home and a few Terabytes were a few Millions of DM

Today SSD offer system speed of 500MB/2 rw and a Terabyte for Notebooks is 50€.

Conclusio. Just like above.

[Bjørn Rørslett]

On the contrary, our conclusions are fundamentally different. I say "NEVER trust a SINGLE disk, ALWAYS use a system with REDUNDANCY".

Spare disks are a necessity of life and so is a spare RAID controller. Of course, redundancy at play again.

I'm not talking about 2 TB systems, but arrays of size say 40 to 80 TB.

Also not to be forgotten, RAID is not equivalent to backup. Thus you also absolutely require sensible backup plans.

By the way, I can count on one hand the number of files lost during the last 20 years of using computerised image storage.

[PeterN]

That's quite a collection. I will tell my wife that others have more stuff stored than I have. Not sure if it will convince her. ;-)

[Frank Fremerey]

Bjørn. I feel your current understanding is a subset of what I say. That is how your statements
are true  but you do not see the big picture. No harm done.

If you say that you lost 20 HDD in 24 month it is clear to me from a pure statistical vantage point
that HDD stamnia is not the limiting factor here. An average HDD lives minimum 5 years without
faliure. Do the math.

[afx]

Björn: that is twenty disks within the last 24 months ... About 1 disk crash per month on average. Never assume anything about the longevity of any hard drive, be it an old-fashioned "spinner" or the modern SSD variant. I have had disks fail within 1 hour after installation.

So what are you doing Bjørn?
Even our constantly mistreated laptop disks at work have a significant lower failure rate.
Your failure rate is way out of the statistic average.

cheers
afx

[Bjørn Rørslett]

The statistical distribution of overall failures is an extreme-value type? Sorry, I cannot see anything unusual here. I know many people having similar experiences.

[ColinM]

After reading this thread, it was interesting to come across this news story

http://www.bbc.co.uk/news/technology-33989384

In case the link isn't available in some countries, the gist of this was:

"Google says data has been wiped from discs at one of its data centres in Belgium - after the local power grid was struck by lightning four times.
Some people have permanently lost access to the files on the affected disks as a result.
A number of disks damaged following the lightning strikes did, however, later became accessible.

In an online statement, Google said that data on just 0.000001% of disk space was permanently affected.

"Although automatic auxiliary systems restored power quickly, and the storage systems are designed with battery backup, some recently written data was located on storage systems which were more susceptible to power failure from extended or repeated battery drain," it said."

The main Google statement is here
https://status.cloud.google.com/incident/compute/15056#5719570367119360

One fascinating section reads:
"This outage is wholly Google's responsibility. However, we would like to take this opportunity to highlight an important reminder for our customers: GCE instances and Persistent Disks within a zone exist in a single Google datacenter and are therefore unavoidably vulnerable to datacenter-scale disasters. Customers who need maximum availability should be prepared to switch their operations to another GCE zone. For maximum durability we recommend GCE snapshots and Google Cloud Storage as resilient, geographically replicated repositories for your data."