NikonGear'23
Images => Life, the Universe & Everything Else => Topic started by: Frank Fremerey on August 22, 2015, 02:05:32
-
Tonight we were talking about resources we use or waste and my eye fell on some of the things, a small fraction of things, I use or not use for taking photos.
Lets start with a big and a small camera:
-
Something ABSOLUTE and something obsolete:
-
Some supporting acts and some collateral damage (of 40 or 50 HDD/SDD):
-
Enlightenment and Drawing:
-
What once was film and pars pro toto for everything Acra compatible:
-
That's actually a very nice series Frank!
Nice to see there are more people who have amassed stuff and not use it all.
-
This is sctually a small fraction of the things I use regularly.
To begin with neither the cam I used to take this set nor the
lens or the lamps are in the frame
Tripods Background system all the batteries
computers and assessoires cables and transport boxes the whole
Sinar sytem diverse adapters decoration small set of portrait
make up a shovel I use in the field as s make shift toilet.
Not even my most important tool: pen and paperclipboards my
compass snd the flashlights mobile presrntation equipment
reflectors diffusors 23 kinds of glue tape plus white plastoform
for posutioning.
New equipment amasses with every project.
-
This is sctually a small fraction of the things I use regularly.
To begin with neither the cam I used to take this set nor the
lens or the lamps are in the frame
Tripods Background system all the batteries
computers and assessoires cables and transport boxes the whole
Sinar sytem diverse adapters decoration small set of portrait
make up a shovel I use in the field as s make shift toilet.
Not even my most important tool: pen and paperclipboards my
compass snd the flashlights mobile presrntation equipment
reflectors diffusors 23 kinds of glue tape plus white plastoform
for posutioning.
New equipment amasses with every project.
Ehhh ... wow!
I don't have so much stuff, i'm just a hobbyist. Still have quite some stuff not used anymore. And frankly, also some stuff never even used.
-
Frankly I think I am, one of the guys with small amounts of equipmenmt here...
-
Interesting to see what someone else has in their camera cupboard. After I bought my first "proper" digital camera (D2x) I began a ruthless clear-out of everything that remained of my once comprehensive kit of film-based cameras, plus lighting, tripods, darkroom equipment and so on. Within less than a year it was all completely gone. Soon I plan to do the same with my much smaller digital kit, reducing cameras from 3 to 2, and lenses from 10 to perhaps only 3. Simplify, simplify.
-
Some day I'll shoot my collection of crashed hard disks - been stacking them for a year now and there must be at least twenty in the stack. Earlier I just threw them away, or opened up to make photos of the disk platter(s) and read-out head.
-
Frankly I think I am, one of the guys with small amounts of equipmenmt here...
More than me, I think. I don't have any studio equipment. But I do have 'complete' systems in three different formats, FF, APS-C and MFT.
-
Some day I'll shoot my collection of crashed hard disks - been stacking them for a year now and there must be at least twenty in the stack. Earlier I just threw them away, or opened up to make photos of the disk platter(s) and read-out head.
Since 1992, when I bought my first Mac, I've only had one hard disk fail, and one CF card (Kingston 1gb - remember 1gb cards?). I'm touching wood as I write this and will continue to do so for the foreseeable future. ::)
-
It is a predictable fact of life that hard disks fail unpredictably, but always at the worst possible time. Thus, build your data storage routines to be able to handle a sudden loss of data in a manner such that you avoid a permanent loss of data.
-
Dave. I remember my first CF was a Sandisk Ultra 512MB for 115 Euro. Now the smallest card is 2GB
and a 90MB/s Samsung SD is 55 Euro at Amazon.
Bjørn. These guys look great and shiny inside. My backup routine is such that data loss is rather improbable.
Worst case is loss of a day's work but only if I am multifailing in my routine.
correct. I once had a rented Casion Q10 at IFA in Berlin. The card was much Smaller. 20 MB or something
-
Some day I'll shoot my collection of crashed hard disks - been stacking them for a year now and there must be at least twenty in the stack. ...
Twenty crashed hard disks – that's quite impressive. I myself recently had an SSD that failed totally and permanently, for no apparent reason. Does that sound familiar to anyone? It was a Samsung 120 G from the 840 series.
-
It is a predictable fact of life that hard disks fail unpredictably, but always at the worst possible time. Thus, build your data storage routines to be able to handle a sudden loss of data in a manner such that you avoid a permanent loss of data.
I am actually at the moment trying put my archives and back-up archives into some sort of order, and to establish a sensible routine for backing up; until very recently this has all been rather haphazard and "when I remembered" - not a good idea. Last week I suddenly discovered I had no back-up at all of any of my 2015 image-files ; even the raw files were on the same drive. :-\ ::)
Frank - my very first CF card was 340mb Kingston, for one of the Nikon Coolpix 5000 series - I forget which one (maybe the 5200) - which I absolutely hated and which did NOT start me on the way to being a digital shooter (a year or two later the D70 did that).
-
Björn: that is twenty disks within the last 24 months ... About 1 disk crash per month on average. Never assume anything about the longevity of any hard drive, be it an old-fashioned "spinner" or the modern SSD variant. I have had disks fail within 1 hour after installation.
-
And lest I forget: do purchase *new* spare disk(s) of the same type or storage capacity IMMEDIATELY whenever a disk fails. I now get two new ones each time I encounter a crash. Thus, I try to keep a shelf with enough spare disks to keep all my storage units running for years to come. Perhaps having a store of 20-30 replacement disks seems superfluous at present, but just wait 1 year or two when replacements no longer might be available and your RAID array has crashed again ...
It goes without saying that in addition to RAID arrays, you also need independent backups.
-
Björn: that is twenty disks within the last 24 months ... About 1 disk crash per month on average. Never assume anything about the longevity of any hard drive, be it an old-fashioned "spinner" or the modern SSD variant. I have had disks fail within 1 hour after installation.
I cannot remember when I've had a hard drive fail. Maybe ten years ago.
-
I myself recently had an SSD that failed totally and permanently, for no apparent reason. Does that sound familiar to anyone? It was a Samsung 120 G from the 840 series.
My routine says: never save data on SSD.
current systems live on mirrored SSDs like Windows on a 250GB 850Pro mirrored to a 840Evo
on the shelve. Ditto for Notebook. Current image of SSD automatically logged by Acronis.
Only a few weeks ago an Intel120 from the 330Series died without prior warning. Intel collected
the Item and replaced it with a 180GB from the 520Series free of charge.
All data are on mirrored HDDs with older Backups 20 km away to have my archive save too.
Any significant job is backupped to at least two physical drives before I start editing.
-
I managed my way trough lots of HDDs.
Since I switched brand to Caviar green by WD I have had Zero disk fails. 50 disks. About 10 years.
As IT journalist and ex member of a renowned test lab I can only
inform any interested party to view a RAID system as one. One
single big HDD with multiple points of failiure.
The combination of system SSD and Data Grave huge and fast
HDD has made RAID systems obsolete except for some very special
production environments in Broadcast where they are not used to
store data but as availability device.
so unless you broadcast 4k or 8k signals and need the matetial
for the next programmes online fast you are imo not in a situation
that justifies RAID
-
I strongly beg to disagree to such a blanket statement.
RAID 0 has no redundancy and can only provide a speed advantage. All other RAID arrangements offer data redundancy (unlike a single drive) and thereby added real-time data protection. That is a fact.
Except for a simple RAID 1, all other RAID arrangements provide a larger storage capacity, many times more than any single disk can give. That is also a fact.
Not adding any redundancy to one's storage scheme is very unwise and over time you are certain to lose data. Another fact.
As to the failure rate, no brands or product models are exempt to the rule that all disks fail at some point in time. Yet another fact. Use hard disks by the hundreds or more and this is uncomfortably easy to observe.
While the failure rate of SSDs might be lower than that of the traditional hard disks, they are subject to failure by wearing out. You actually ought to set aside a sizeable part of the disk capacity to help the disk controller to distribute write wear more optimsal and thus extend SSD longevity, but in my experience few bother to perform such basic measure of prevention. Do note that very few SSDs are warranted for more than 3 years. That is a bit of information worth some thoughts.
-
Bjørn. Your statement contains zero contradiction to mine except for the fact that you seem to trust RAID and SSD
more than I do.
The data structure of a RAID like say 5 or 6 type array is made to compensate for single or even dual disk failiures
which is kinda nice.
All the disks are managed by one controller wich might fail by many reasons and render all disk contents useless.
Even two Disks run in parallel offer more safety because they are still controlled independently.
If you choose to drive the two disks behind independent uninterruptable power systems you are in safety heaven.
To get anywhere near this level with RAID you have to have a backup RAID for every RAID.
That put we look back into history and ask: what do we need a RAID for?
1) make big disk from smaller disks and compensate for extra failiure points by redundancy
2) make faster disk from slower disks and compens....
When I wrote about this topic in c't we had data rates of 2MB/s at home and a few Terabytes were a few Millions of DM
Today SSD offer system speed of 500MB/2 rw and a Terabyte for Notebooks is 50€.
Conclusio. Just like above.
-
On the contrary, our conclusions are fundamentally different. I say "NEVER trust a SINGLE disk, ALWAYS use a system with REDUNDANCY".
Spare disks are a necessity of life and so is a spare RAID controller. Of course, redundancy at play again.
I'm not talking about 2 TB systems, but arrays of size say 40 to 80 TB.
Also not to be forgotten, RAID is not equivalent to backup. Thus you also absolutely require sensible backup plans.
By the way, I can count on one hand the number of files lost during the last 20 years of using computerised image storage.
-
That's quite a collection. I will tell my wife that others have more stuff stored than I have. Not sure if it will convince her. ;-)
-
Bjørn. I feel your current understanding is a subset of what I say. That is how your statements
are true but you do not see the big picture. No harm done.
If you say that you lost 20 HDD in 24 month it is clear to me from a pure statistical vantage point
that HDD stamnia is not the limiting factor here. An average HDD lives minimum 5 years without
faliure. Do the math.
-
Björn: that is twenty disks within the last 24 months ... About 1 disk crash per month on average. Never assume anything about the longevity of any hard drive, be it an old-fashioned "spinner" or the modern SSD variant. I have had disks fail within 1 hour after installation.
So what are you doing Bjørn?
Even our constantly mistreated laptop disks at work have a significant lower failure rate.
Your failure rate is way out of the statistic average.
cheers
afx
-
The statistical distribution of overall failures is an extreme-value type? Sorry, I cannot see anything unusual here. I know many people having similar experiences.
-
After reading this thread, it was interesting to come across this news story
http://www.bbc.co.uk/news/technology-33989384
In case the link isn't available in some countries, the gist of this was:
"Google says data has been wiped from discs at one of its data centres in Belgium - after the local power grid was struck by lightning four times.
Some people have permanently lost access to the files on the affected disks as a result.
A number of disks damaged following the lightning strikes did, however, later became accessible.
In an online statement, Google said that data on just 0.000001% of disk space was permanently affected.
"Although automatic auxiliary systems restored power quickly, and the storage systems are designed with battery backup, some recently written data was located on storage systems which were more susceptible to power failure from extended or repeated battery drain," it said."
The main Google statement is here
https://status.cloud.google.com/incident/compute/15056#5719570367119360
One fascinating section reads:
"This outage is wholly Google's responsibility. However, we would like to take this opportunity to highlight an important reminder for our customers: GCE instances and Persistent Disks within a zone exist in a single Google datacenter and are therefore unavoidably vulnerable to datacenter-scale disasters. Customers who need maximum availability should be prepared to switch their operations to another GCE zone. For maximum durability we recommend GCE snapshots and Google Cloud Storage as resilient, geographically replicated repositories for your data."
-
The broadcasting companies I reported about earlier set up their Data Centres made of tape libraries of the terabyte scale interconnected by fibre optical cables each roughly 150km away. A three way mirror with uninterruptable Power supply and mains from different parts of the electricity grid.
-
I fail to comprehend the relevance of comparing tape libraries with hard drives ...
-
backup storage on tape is more reliable than on disk
-
Perhaps. Perhaps not. Data errors are more common. But error corrections might function better. Access time shouldn't be mentioned at all of course.
I have a backup server cluster running multiple LTO tape drives myself.
-
(https://farm1.staticflickr.com/653/20830004912_410afd67fc_b.jpg)
Anyone familiar with this contraption and what you can do with it?
-
Anyone as in noone? ;)
-
Projection of some kind or a beam splitter? Shine a light into the lens and see if the opaque glass lights up.
-
Camera obscura?
-
Frank:
1) I don't understand why you say never store anything on SSD's? Yes, I know there were issues early on with SSD's, but today? I haven't seen any statistics but my hunch is that SSD's are more reliable than HDD's.
2) If I understand you correctly, you say that RAID is something that few people should ever need or use. You are completely correct that RAID can fail in many ways, among them the controller can fail and corrupt the data on your RAID system, in which case you'd simply have redundant, corrupt data. That said, RAID controller failures are much more rare than HDD's. This means you'll have far fewer systems crashing due to disk issues. At least, that is my experience from working in a data center. One issue with RAID systems today is that they take a long time to re-sync because the disks have become so big. Unless you go for enterprise storage systems, the primary way to create bigger storage systems, is RAID. I'm not sure what your suggestion is if you need +10TB storage?
There is of course Samsungs new +15TB SSD: http://arstechnica.com/gadgets/2015/08/samsung-unveils-2-5-inch-16tb-ssd-the-worlds-largest-hard-drive/
As for the endurance of SSD's, they are generally performing quite well: https://techreport.com/review/27909/the-ssd-endurance-experiment-theyre-all-dead
-
Jørgen:
1) I store operating systems and programmes there, sometimes the one big file I am currently editing and its children.
2) My main safety principle is to only have the data "online" I really need currently. Archives and everything else are external drives mirrored and outsourced to a building on the other end of the city.
One copy I have here sitting on the shelve and 5 free hot plug bays in my workstation. If I need archive files, I sATA hot plug what I need, get it and eject it.
Soooo:
RAID is IMO for people who need large amounts of data ONLINE which is not the case for most photographers
In rare cases, like checking through ALL DRIVES to eliminate obsolete redundancy or integrate files over a huge amount of disks into a new file structure, I can fill all 7 Bays of my Lian Li with drives and have a lot of data online.
Most of my drives are 2TB still, but there are some 3TB and 4TB and still some earlier generation 1TB drives.
Which user scenario do you think of?
Frank
-
"RAID is IMO for people who need large amounts of data ONLINE which is not the case for most photographers"
A curious blanket statement. Surely this should be left to each photographer to answer? The average photographer is no photographer.
-
1) That doesn't explain why you mistrust SSD's?
2) Are you keeping your storage offline to hopefully keep the drives in good shape? You do know that, a lot can also go wrong when you hot plug HDD's?
For me, convenience is king. I'm lazy and I know I will fail at some point and Murphy would make sure I would forget to archive some data right when my laptop crashes. Why should I manually keep a schedule when I know the machine is much better at doing that?
I have my LR catalog on my laptop and it's backed up hourly (or something like that) through Time machine to my NAS. When I import new images, they are stored on my laptop until I have edited them, at which point I move them to the NAS over the network. I have automatic daily local and remote backups of my NAS. I'm of course notified if something fails. Where I honestly do fail, is checking my backup's frequently enough. I do test them once in a while as you should, but probably not often enough.
-
Early SSDs had issues with "stuttering" or untimely demises. Sequential reading might go pretty fast, but randomly writing lots of small files, like an OS does, could slow the drive dramatically.
These days SSD controllers apparently are much improved and much of the early problems sorted or at least brought under better control. Still one should leave a significant part of the drive allocated for over-provisioning to extend the longevity and durability of the SSD.
Although modern SSDs should be amazingly efficient in withstanding massive write operations over time, it is a worrying fact that the drives might suddenly die when the wear levelling no longer can provide healthy data blocks for writing. Even more worrisome, the read-out capability tends to be intact at the point of drive death, but of no use since the drive throws in its towel completely to become totally bricked.
Perhaps as a safety measure one should replace SSDs with fresh drives on a 3-5 year cycle. One tends to get equal or bigger capacity for less money as well in this scheme.
-
Anyone familiar with this contraption and what you can do with it?
Kind of measuring device ?
Ancestor of laser spirit level or length ?
-
I keep drives offline to not waste energy.
Everbody might keep their dogs in any box.
I do not use more than 10.000 files at a time
and my biggest single drive contains more than
200.000 RAWs
If I summerise my impression of this RAID
discussion, Bjørn seems to enjoy keeping
his files on RAID systems and killing HDDs.
People are different.
-
Early SSDs had issues with "stuttering" or untimely demises. Sequential reading might go pretty fast, but randomly writing lots of small files, like an OS does, could slow the drive dramatically.
Indeed, there was numerous issues with early SSD's. One of the fixes was TRIM. In the early SSD days, which is only a few years back, sequential read was actually slower compared to HDD's, it was random read/write performance where SSD's really made a difference. These days, SSD's have improved massively and you need to use a PCIe interface to take full advantage of the SSD.
These days SSD controllers apparently are much improved and much of the early problems sorted or at least brought under better control. Still one should leave a significant part of the drive allocated for over-provisioning to extend the longevity and durability of the SSD.
That is certainly true and picking the right drive for your use case certainly makes a difference, just take a look at the endurance test I linked earlier.
Although modern SSDs should be amazingly efficient in withstanding massive write operations over time, it is a worrying fact that the drives might suddenly die when the wear levelling no longer can provide healthy data blocks for writing. Even more worrisome, the read-out capability tends to be intact at the point of drive death, but of no use since the drive throws in its towel completely to become totally bricked.
Again, based on the endurance test linked earlier, it does indeed seem to be the case that the controllers aren't entirely bug free (nothing really is), which means drives might become brick'ed before expected. That happened to the Samsung 840 Pro drive, but it didn't happen until after they had written 2.4PB of data to the drive. The other drives didn't last as long, but gave warnings well before they failed.
Perhaps as a safety measure one should replace SSDs with fresh drives on a 3-5 year cycle. One tends to get equal or bigger capacity for less money as well in this scheme.
I personally have bigger faith in SSD's, but with your history of failures, I understand the concern. However, in laptops like my rMBP, it's not really possible.