Author Topic: A friendly warning ...  (Read 23737 times)

Jørgen Ramskov

  • NG Supporter
  • **
  • Posts: 1103
  • Aarhus, Denmark
Re: A friendly warning ...
« Reply #45 on: December 08, 2015, 15:05:06 »
That wouldn't quite work for me as I use a laptop with a 512GB SSD and move processed images off it to my NAS as I don't have enough room on my laptop to store all my images. Other than that, I can certainly see the advantage of using Dropbox, they do offer a nice service.
Jørgen Ramskov

Sash

  • NG Member
  • *
  • Posts: 158
  • From behind the Irony Curtain
Re: A friendly warning ...
« Reply #46 on: December 10, 2015, 21:09:04 »
That is all really weird to me. In 20 years I have not had a single crashed hard drive.
Alexander

Bjørn Rørslett

  • Fierce Bear of the North
  • Administrator
  • ***
  • Posts: 8252
  • Oslo, Norway
Re: A friendly warning ...
« Reply #47 on: December 10, 2015, 22:21:34 »
I don't believe my experiences with hard disk failures are exceptional. Lots of friends have had similar encounters with reality.

If failure rate is approx. constant, the expected number of actual failures becomes a function of the total number of disks only. A small rate multiplied by a large number of units rapidly becomes a quantity > 1.

bobfriedman

  • NG Supporter
  • **
  • Posts: 1243
  • Massachusetts, USA
Re: A friendly warning ...
« Reply #48 on: December 10, 2015, 23:43:37 »
I use level 5 RAID with a redundant HDD backup archive of the active RAID arrays
Robert L Friedman, Massachusetts, USA
www.pbase.com/bobfriedman

stenrasmussen

  • Guest
Re: A friendly warning ...
« Reply #49 on: December 10, 2015, 23:44:35 »
I have experienced CF card, computer HD, mainboard and memory failure. Not too often but it happens.

arthurking83

  • NG Member
  • *
  • Posts: 240
  • Good to be back on NikonGear
Re: A friendly warning ...
« Reply #50 on: December 11, 2015, 12:28:43 »
... to heed. Hard disks do not last forever. ....


timely reminder .. and what do you think happens in the days since this post and yesterday! .. crashed primary (image) storage drive.
I have no idea how long it's been dead for, but noted it only yesterday.
Work has kept me too busy to do anything photography related over the past few months, and haven't had much time to log in since about last weekend.

Of course I have a secondary backup of the image archive .. but work is so busy ATM, I don't even have the time to get to the local store to acquire the necessary replacement drive too.

Secondary drive has now been taken off line(to maximise preservation) until a new drive(or two) is/are eventually purchased.

This is my first drive failure in about 15 or so years when the last one dies(an old 80G drive, then currently a large capacity drive) .. so drives have been good to me over the years considering my usage pattern.

I've had many friends come to me to sort out dead drives for them, which seemed to be all too often up to about a year or so ago(maybe two) .. when it got to the point where I had to acquire a docking station just to make it easier to sort them out, rather than forever opening my PC case to connect it to a cable .. etc.
So of course, not long after I acquired the docking station my friends/family's HDD's all behave properly now, so the docking station has become almost redundant(and there was much rejoicing).

moral of the story:

 .. if you want security in failure resistant hardware, acquire the necessary tools to fix the problem(s) before the failure of the hardware .. this ensures that you'll never get to use those tools  ;D
Arthur

Bernard Delley

  • NG Member
  • *
  • Posts: 127
  • You ARE NikonGear
What are you doing to prevent accumulating corrupt data ?
« Reply #51 on: December 11, 2015, 14:21:43 »
This could easily a problem if you deal with a few disks containing mirror copies of the archive. Before a disk obviously fails, it already may contain corrupt data. These corrupt data/images may then get copied along onto the upcoming mirror disks.

Erik Lund

  • Global Moderator
  • **
  • Posts: 6480
  • Copenhagen
    • ErikLund.com
Re: A friendly warning ...
« Reply #52 on: December 11, 2015, 14:44:43 »
I have had my fair share of dying HDD over the years and hear about them from our IT dept at work all the time, so many possible things can go wrong trying to install the backup that it's puzzling that all is not lost more often...

Currently I use multiple HDD and Drop box but looking in StremNation.com since its cheaper and has full support for NEF and DNG raw files on top of the JPG files.

I'm testing a 4G on top of my normal connection to keep upload speed as high as possible.
Erik Lund

Jan Anne

  • Noob
  • Global Moderator
  • **
  • Posts: 2042
  • Holland
    • Me on Flickr
Re: A friendly warning ...
« Reply #53 on: December 11, 2015, 17:10:57 »
I'm testing a 4G on top of my normal connection to keep upload speed as high as possible.
I am planning to do the same, in my new place the closest Internet switch box is 6km away so the fixed line speed will be something like 1-3Mbit/s :-\

So I've chosen a provider that offers an additional subscription with 100GB of 4G data for €45 a month available for people living in rural areas.

This way I can still do NG admin work while watching Netlfix or HBO ;D
Cheers,
Jan Anne

Erik Lund

  • Global Moderator
  • **
  • Posts: 6480
  • Copenhagen
    • ErikLund.com
Re: A friendly warning ...
« Reply #54 on: December 11, 2015, 21:07:23 »
I chose 45€; 300GB free data at between 20 and 30 Gbit/s down and up
Erik Lund

Jan Anne

  • Noob
  • Global Moderator
  • **
  • Posts: 2042
  • Holland
    • Me on Flickr
Re: A friendly warning ...
« Reply #55 on: December 11, 2015, 21:22:39 »
I chose 45€; 300GB free data at between 20 and 30 Gbit/s down and up
Niiiiice, no such subscriptions in Holland I'm afraid :'(
Cheers,
Jan Anne

arthurking83

  • NG Member
  • *
  • Posts: 240
  • Good to be back on NikonGear
Re: A friendly warning ...
« Reply #56 on: December 14, 2015, 18:27:47 »
This could easily a problem if you deal with a few disks containing mirror copies of the archive. Before a disk obviously fails, it already may contain corrupt data. These corrupt data/images may then get copied along onto the upcoming mirror disks.

On the topic of corrupt data:
Is there some software that can quickly confirm the integrity of a collection of NEF files without the need to try to view them all.

I was going through my collection a while back and got back into tagging/keywording what I could reasonably do in a new session.
I noted that some images(only in one directory) had been corrupted. No program would display them as an image nor the preview file(embedded jpg file).

I tracked the corruption of those images down to an issue I had with the docking station when I initially got it and started using it.
On USB3 the docking station would suddenly quit transferring data and hang. (turned out to be a badly made driver for the device).
I initially started to transfer my archive to a new HDD connected via USB3, and the transfer would hang after a short time.
Gave up on that, connected the docking station to USB2 and it would transfer the archive without issue.
Finally go the driver sorted and USB3 would work fine .. but unknown to me in the few attempts I made to transfer the data via USB3 where the transfer quit, the hanging process must have corrupted some of the images.
From memory I think 8 images were lost and I can't figure out a way to recover them(but that's not the issue).
The sudden loss of connection must have caused the loss, as the last written file date of those images corresponds with the time when I got the docking station(and new HDD and started the secondary backup process).

I know you can do a md5 hash checking thingy to compare one archive set compared to another, but this is no good if both archive sets are corrupted in the same way.
Is there a program that can simply scan or check to see if the image file(tiff or NEF) is corrupted in any way.
The only reason I noted those files was that the preview files in that folder weren't displaying at all, so went in to view what those files were.
Arthur

Asle F

  • NG Member
  • *
  • Posts: 320
  • Hovet, Norway
    • Fjell og foto, my mountain and photo blog in Norwegian
Re: A friendly warning ...
« Reply #57 on: December 14, 2015, 21:46:10 »

I know you can do a md5 hash checking thingy to compare one archive set compared to another, but this is no good if both archive sets are corrupted in the same way.
Is there a program that can simply scan or check to see if the image file(tiff or NEF) is corrupted in any way.
The only reason I noted those files was that the preview files in that folder weren't displaying at all, so went in to view what those files were.

I use dcraw for this. dcraw -c filename.nef >/dev/null will print an error if the nef-file is corrupted. I use this as part of my copy-from-memorycard-script. I also use md5sum and am saving the result for comparing with this in the future. In this way I can easy check if the file is the same as it was from the camera or if it has changed.
There is no illusion, it just looks that way.

David H. Hartman

  • NG Member
  • *
  • Posts: 2778
  • I Doctor Photographs... :)
Re: A friendly warning ...
« Reply #58 on: December 23, 2015, 21:29:14 »
Some years back I had the hard drive that contained my operating systems and programs was over heating. I was in the process of setting up and new HD to replace the one that would surely fail. I forgot a major rule of mine and did not disconnect my data drives. The OS was WinNT which was quite stable but may have hiccuped due to bit rot. The system crash took out the master boot records on all four HD that were spinning.

I had one 640MB portable USB HD with most of my NEF files. It was about 30-60 days out of date. I bought a 1T portable HD and Phoenix data recovery software. After many hours of work I had lost only 30 Photoshop PSD files and one TIF. The PSD files were very small and contained only a thumbnail JPG not the full image. In the end nothing important was lost as PSD could be remade and would usually be better that the one's lost. This was a very close call for me.

Now I have all my photos, original NEF(s) and JPG(s) from P&S cameras on four HD(s). I backup to three HD every time I upload photos. My system is well short of perfect but I'm better covered than I was some years back.

For those with no plan please start now. When I had the system crash I wasn't cloning the drive. I was only partitioning the new one. I restored my operating system from a ghost image. Starting a consistent back system would be a good new years resolution, one to keep. Maybe buy some new HD(s) on December 26 and start before the New Year.

Best,

Dave Hartman
Beatniks are out to make it rich
Oh no, must be the season of the witch!

Bjørn Rørslett

  • Fierce Bear of the North
  • Administrator
  • ***
  • Posts: 8252
  • Oslo, Norway
Re: A friendly warning ...
« Reply #59 on: December 23, 2015, 21:49:52 »
Logical disk errors (corruption of file system or partitions) can be handled pretty well by many easily available software packages - provided you pull the brakes immediately and don't try to do anything further with the affected drive(s).

I have enclosures that enable me to insert troublesome disks and read them independently from their hosting environment.