Author Topic: A friendly warning ...  (Read 15475 times)

Michael Erlewine

  • Close-Up Photographer
  • NG Supporter
  • **
  • Posts: 1422
  • Close-Up with APO
    • Spirit Grooves
Re: A friendly warning ...
« Reply #30 on: December 07, 2015, 22:46:52 »
Unfortunately, although the media itself just might survive a millenium, the chance that the reader will do the same is virtually nil.

Thus we push the problems ahead of us.

I understand, but the standard response to the view: if there is a reason (or money) to keep it current, someone, somewhere, will offer it as a service. And, it is not that we have much choice, just now. If the M-Disks were 100 GB, we would all be using them. At least they give us a little time.
MichaelErlewine.smugmug.com. Founder MacroStop.com, MichaelErlewine.com (articles), https://www.youtube.com/user/merlewine (video tutorials), All-Music Guide, All-Movie Guide, Classic Posters.com, Matrix Software, SpiritGrooves.net, DharmaGrooves.com

Thomas G

  • NG Member
  • *
  • Posts: 1929
  • lumofisk
    • Iceland round trip 2016
Re: A friendly warning ...
« Reply #31 on: December 07, 2015, 22:52:12 »
RAID6 would be the obvious next step. It's depends on now critical the system is and the discussion isn't relevant for many of the most critical systems - they are connected to a SAN instead.

We live with the possibility of hardware failure in our cameras too - flash cards fails regularly too, which is probably why it's possible to create a mirror with two cards in many cameras.

Well, SANs consist of (raid) disk sets in the background, and the disks share a quite consistent dying rate with what we discuss here.
SANS are abstractions, they just help to operate disk space more efficiently and independently from servers.
And they often have supportive measures build in to deal with the disk imperfections.
For our purpose:
There are quality differences between drives: desktop SATA worst, server SAS/SCSI best. Server SATA better than average. You get what you pay for. Think in price factors.
Always add hot spares. Always.
Cold spares (sometimes called stand-by disks) only make sense when they are not sharing the same mechanical vibration domain of the main cage or case.

Read performance: Raid 5,6, and 10 deliver about the same read performance.
Write performance: Raid 10: 50% write performance, Raid 5: 25%, Raid 6: 17%.
Space effficiency (minimum 4 disks): Raid 10 has only 50% constantly, Raid 6 at 50%, Raid 5 at 75%.
Space effiiciency (                8 disks): Raid 10 has only 50% constandly, Raid 6 at 70%, Raid 5 at 85%.
Reliability (4 disks): Raid 6 is 450 times raid 10 and raid 10 is 10 times Raid 5

Due to the propability of a disk having bit errors it is good advice to keep arrays of very cheap disks, say SATA/P-ATA desktop disks, small. 10 Gbytes (4 disks) or so.
Size and time make up for failure.

Keep copies. Use different media. Encrypt and load to the cloud.

Did I talk about power supplies?







Jørgen Ramskov

  • NG Supporter
  • **
  • Posts: 1103
  • Aarhus, Denmark
Re: A friendly warning ...
« Reply #32 on: December 07, 2015, 22:52:44 »
Data Storage on DNA Can Keep It Safe for Centuries:http://www.nytimes.com/2015/12/04/science/data-storage-on-dna-can-keep-it-safe-for-centuries.html?_r=0

Not yet available unfortunately :)
Quite fascinating, haven't heard about that before. Sounds like part of the technology needed to create a "babel fish" :D
Jørgen Ramskov

Jørgen Ramskov

  • NG Supporter
  • **
  • Posts: 1103
  • Aarhus, Denmark
Re: A friendly warning ...
« Reply #33 on: December 07, 2015, 22:55:54 »
Thomas G: You are of course correct. We are moving away from HDD's to SSD's and such. Samsung recently displayed a 16TB SSD.
Jørgen Ramskov

Bjørn Rørslett

  • Fierce Bear of the North
  • Administrator
  • ***
  • Posts: 8248
  • Oslo, Norway
Re: A friendly warning ...
« Reply #34 on: December 07, 2015, 23:24:25 »
Longevity of SSDs needs to increase really fast, and prices come down, are they to become a viable alternative for large-scale data storage. Seen in an overall perspective they are early days and their technology is immature. Not unlike our digital cameras in that respect :D

Bruno Schroder

  • NG Supporter
  • **
  • Posts: 1056
  • Near Waterloo, Belgium
Re: A friendly warning ...
« Reply #35 on: December 07, 2015, 23:38:50 »
Quite fascinating, haven't heard about that before. Sounds like part of the technology needed to create a "babel fish" :D

It is brand new stuff indeed. Read only was demonstrated in 2012. Read/Random Edit was published in Nature in September this year and the Microsoft paper will be published next year only.

Babel fish for instant translation?
Bruno Schröder

Øivind Tøien

  • NG Supporter
  • **
  • Posts: 1063
  • Fairbanks, Alaska
Re: A friendly warning ...
« Reply #36 on: December 08, 2015, 02:18:48 »
Data Storage on DNA Can Keep It Safe for Centuries:http://www.nytimes.com/2015/12/04/science/data-storage-on-dna-can-keep-it-safe-for-centuries.html?_r=0

Not yet available unfortunately :)

So now we just need to worry about real viruses attacking our storage...
Øivind Tøien

Bjørn Rørslett

  • Fierce Bear of the North
  • Administrator
  • ***
  • Posts: 8248
  • Oslo, Norway
Re: A friendly warning ...
« Reply #37 on: December 08, 2015, 02:36:28 »
Or evolutionary mutations :D

Øivind Tøien

  • NG Supporter
  • **
  • Posts: 1063
  • Fairbanks, Alaska
Re: A friendly warning ...
« Reply #38 on: December 08, 2015, 03:04:26 »

That could lead to new types of creativity...
Øivind Tøien

Bruno Schroder

  • NG Supporter
  • **
  • Posts: 1056
  • Near Waterloo, Belgium
Re: A friendly warning ...
« Reply #39 on: December 08, 2015, 08:38:44 »
and organic editing ...
Bruno Schröder

Jørgen Ramskov

  • NG Supporter
  • **
  • Posts: 1103
  • Aarhus, Denmark
Re: A friendly warning ...
« Reply #40 on: December 08, 2015, 08:51:33 »
Babel fish for instant translation?
Yes, the babel fish from the Hitchhikers guide to the galaxy :)
Jørgen Ramskov

PeterN

  • NG Supporter
  • **
  • Posts: 655
Re: A friendly warning ...
« Reply #41 on: December 08, 2015, 13:40:51 »
Interesting, could you share some more details about your setup?

 I got tired of keeping everything in sync across devices and computers.  Although I do have a 12TB Mirrored Storage Unit (Promise) and several other units, I use Dropbox as my cloud storage. They take care of backups and when all of my computers crash and need new computers, I just download everything.

Files (photo libraries and raw files) are synced across laptop and mac pro automatically. I decided to keep everything in NEF and RAF because software evolves and what is impossible to do with today's software might be possible tomorrow. I know I should only do it for precious images but I don't spend time on selecting the precious ones so I keep everything in raw. I made once the mistake to convert to DNG and/or JPG and won't do that again.

I can even access from my iPad if I want/need to.

So life has become substantially easier, also in migrating to another computer. The only tricky thing is to make sure plugins etc are in sync across computers.

Having said so, the old dog in me still tells me to backup occassionally on my (mirrored) network drives. ;-)

PS: If you worry about security, I think data in their centers is more secure than data in our own personal networks. They have dedicated firewalls and security and network exports. I don't although I customized my firewall.

Bjørn Rørslett

  • Fierce Bear of the North
  • Administrator
  • ***
  • Posts: 8248
  • Oslo, Norway
Re: A friendly warning ...
« Reply #42 on: December 08, 2015, 13:54:08 »
Appears you need a substantial bandwidth to make remote storage efficient?

Will your solution scale to say storage requirements of 1-200 TB within acceptable economic overhead?

Jørgen Ramskov

  • NG Supporter
  • **
  • Posts: 1103
  • Aarhus, Denmark
Re: A friendly warning ...
« Reply #43 on: December 08, 2015, 14:35:27 »
Dropbox have a €12/month plan with unlimited storage. I use their basic, free offering and it seems decently fast, but you're limited to one central folder under which everything needs to be stored. I don't know if that's different if you switch to one of the paid versions.
Jørgen Ramskov

PeterN

  • NG Supporter
  • **
  • Posts: 655
Re: A friendly warning ...
« Reply #44 on: December 08, 2015, 14:54:47 »
Appears you need a substantial bandwidth to make remote storage efficient?

Will your solution scale to say storage requirements of 1-200 TB within acceptable economic overhead?

Mine is 1TB so I can work with one local folder/disk. Your situation is more comparable to that of businesses who use cloud services.  I haven't checked the availability of 200TB solutions but there is a Dropbox solution for businesses that has unlimited storage. I bet that file management differs too so perhaps you could contact them. Or another provider. There are multiple vendors for cloud storage for large data amounts.

The sync solution of dropbox is fast and only syncs changed files. A 200TB databse might be a problem but I assume you have multiple libraries and files.