Handling Large Image Files

[Michael Erlewine]

I have many large image files, most of them TIF files, the resulting images of years of focus stacking. As for the original layers, there are probably half a million or more of those. Let’s just look at these large TIF files.

Right, now I am trying to work (Windows 10) with the very large TIF files. I have tried Infraview, Bridge, PhotoMechanic, Windows Explorer, and so on. All of them seem to kind of choke up.

I have a pretty fast computer, yet nothing seems be very fluid. I am wondering what other photographers do with this situation. Here are some of my questions:

(1) What kind of external hard drives are best?

[I have tried SATA, SSDs, and all kinds of external drives, but none seem to be a charm, so to speak.]

(2) What size external hard drives are best?

(3) What viewing software do you use?

[As mentioned above, Adobe Bridge, Infraview, and so on see to bog down, take forever to load so I can see thumbnails, and generally prove almost unusable except at a snail’s pace.]

All suggestions are welcome as to how to deal with this ever-growing problem. Thanks.

[Jack Dahlgren]

Michael,

1) SATA is a type of interface, not a type of drive. You will want a fast Nvme (solid state) drive connected through a thunderbolt interface to reach top speeds currently - assuming that data transfer is the bottleneck.

2) Depends on how much data you have. Price/capacity is linear in the middle of range so probably good to choose from that range.

3) I use a bunch of things, but if you are concerned about speed, pictureflect https://pictureflect.com/ is pretty snappy. But have only used on .jpg  Free also.

4) I'd probably put on a fast internal m.2 or NvME drive instead of external.

[ColinM]

Thank you Jack - I learned a lot from your post.
I'm sure Michael and others will also be googling, but this article (over a year old now) gave me some brief background, plus also indicated adding these drives to an existing machine might be harder than buying a new one that was built ready to use them

https://www.pcworld.com/article/2899351/everything-you-need-to-know-about-nvme.html

Michael, I can't vouch for any speed improvement on really big TIFF files as I don't use them, but I've found FastStone to be faster & more convenient than some other image browsers

https://www.faststone.org/FSViewerDetail.htm

[Michael Erlewine]

Michael,

3) I use a bunch of things, but if you are concerned about speed, pictureflect https://pictureflect.com/ is pretty snappy. But have only used on .jpg  Free also.

Yes this program, "Pictureflect" is indeed powerful, free to try, and only around $2.00 to purchase, and it works on large TIF files, better than most. The program "FastStone Image Viewer" was the same old too-slow to work on large file.

Tanks Jack!

[Michael Erlewine]

Since I usually stack images from several to hundreds of layers, the resulting stacked image is a fabrication and does not carry all the info one finds in raw files, which is what I use for layering. I keep the raw files, so EXIF data is available if I am interested.

Actually, for my interest, I am only interested in a few pieces of data.

Filename
Lens Name
Keyword

I don’t really care about most of the other data because, in general I use APO lenses wide open (or nearly so), so since I know the F-Stop of the lens I am using. And most of the lenses I use are “exotics” meaning they do not usually interface to the camera to provide the standard EXIF data.

And I almost always use the lowest ISO, which for the Nikon cameras I use is ISO 64. So, the variable is the shutter speed and I don’t care about that.

An image filename might look like this:

8AE-0142-EL105-Morning-Glory

Where:
8AE-0142 =  the camera-generated filename
EL105 = APO El Nikkor 105 f/5.6
Morning Glory = the kind of subject

That all I need. From the filename (and the original NEFs (raw files) I can look up EXIF data if I need it.

However, what I am interested in is the particular lens used and not much else. I have a lot of lenses, mostly APO lenses, and their differing IQs vary, often subtly, so I want to know what lens was used.

[Bob Foster]

As I’ve been able to make time over the past 10 days I’ve done some experimenting with rendering 250-265 GB .tif files on a basic commodity computer (i5 4 core @2.7GHz,  8 GB RAM, video card- NVIDIA GT 730, 2 TB HDD, and USB 3.0 ports). The results on Windows 10 were a surprise. I have two suggestions.

1. In general, with all file systems, the smaller the number of files (in this instance .tif image files) within any particular directory (folder) the better that any program that "prefetches” files will perform. If it will not affect whatever means you presently have of grouping and/or indexing your files I'd strongly suggest the use of smaller directories/folders. If you wish to test this try limiting the contents of any particular directory to those images, by filetype. E.g.:

(Directory) Subject-Lens Identifier- (in any order)

(Sub-Directory) .NEF files. I keep the originals because software has arguably become better over the years at rendering nuances from the extreme ends of the dynamic range.

(Sub-Directory) .TIF files. I store these in compressed form to save disc/ssd space but decompress them prior to any use. To these filenames I add by batch lens and subject info leaving the camera filename.

(Sub-Directory) The directory generated by Zerene Stacker.

2. Contrary to what might seem like common sense, the use of prefetch routines in computer code can have a deleterious effect on computer performance when data sets are very large. RAM usage can sometimes have a very negative impact on program execution, it’s also possible to choke a drive with excessive demands on its read speed and buffering capabilities. Also, not all USB host controllers are equally capable; particularly those rated USB 3.X.  In some cases going to the USB controller manufacturers website and downloading the current driver can make a very significant difference in throughput. For many devices Microsoft’s update service has never kept drivers current.
 
I created a directory containing over 2000 .tif 250+ GB files on a QNAP TR-4 USB 3.1 External RAID Unit. The directory is over 500GB in size. The directory was also copied onto each of a pair of 5TB Seagate Backup Plus drives for testing purposes. I’ve avoided the use of all USB 3.X TB thumb drives because all apparently responsible reviews that I’ve seen indicate that to date performance of these drives is impaired by heat or purposely limited by the manufacturer.

I’ve stuck with Fast Raw Viewer because it is highly configurable and as a long time user I’m fairly familiar with how to tweak it to attain better performance.

The biggest gain I've been able to realize on the test computer was accomplished by turning OFF the prefetch function in Fast Raw viewer. I'm easily able to view each image at 1 second intervals using the QNAP RAID Unit either plugged directly into the test computer or plugged into another computer on my network with the appropriate directory mounted (Map Network Drive) on the test computer. I used a hardwire (not wireless) connection. See pages 122-125 of the current version of the Fast Raw Viewer manual. The Seagate Backup Plus drives do not perform as well: one is “comfortable” rendering at 1.5 second intervals, the other (older) unit takes about 2.1 seconds to render an ~260MB image in series.

Best wishers, Bob

[Seapy]

Michael, I believe the fastest generally available processing storage (not long term storage) is an M2 blade drive on a PCIe card.  That is what my latest MacPro 5.1 has for the boot drive.  I also have a second small(ish) 250Gb version for my working image storage where I place my latest import from camera while I work with and cull files before producing the final TIFF images. This is also my Photoshop scratch disk.

You can also place SSD's on PCIe cards which give them direct connection to the processor(s) which speeds things up considerably, allowing the full potential speed of the SSD to be enjoyed.

The M2 drives are extremely fast, instant almost, on my ancient MacPro but not particularly capacious.  So once processed the finished image files are moved over to longer term storage and archive to free up space for the next batch.

The PCIe cards for the M2 drives are cheap enough, the M2 blades are less cheap, but in my view worth every penny (cent?).

https://www.megamac.com/collections/storage-solid-state-drives/products/480gb-owc-aura-ssd-for-mid-2013-and-later-macbook-air-and-macbook-pro-with-retina-display-owcssdab2mb05

[arthurking83]

Hi Michael.
What Bob Foster mentioned can be a major source of frustration. The USB controller you use can be a major contributor to file access speed .. no matter how 'fast' your computer may be.
It'll only be as fast as the slowest link will allow it to be.
Bob mentioned that USB host controllers are not all created equal. If you are not computer savvy, it's not the easiest device to find info on, as more than likely you will have more than one type/model on your motherboard.

I've found that over the years, if the USB3.x controller is not Asmedia branded(whatever model) .. it'll most likely be slower .. a lot slower than the marketing hype will state it to be capable of.
If your motherobard has either VIA USB3.x controller or JMicron USB controller .. avoid them for external drive devices. OK for peripherals(mouse/keyboard/card reader/USB thumbrives/etc) .. but not for any type of hard drives .. SATA or otherwise.
JMicron USB controllers in particular can be literally diabolical in their (in)ability to transfer data .. so it's become an important hardware aspect of any device I need to acquire. Had more than one JMicron built in controller(both USB and SATA) that have caused me much grief.

You have a twofold issue:
1/. 500K tif files are going to create a huge storage reservoir .. how many Tb of data have you collected?
2/. At the Tb/$ rate .. SSDs are simply not economical compared to magnetic storage. Also be weary of some magnetic storage drives(recent WD issues have been made public re some of their recent SATA products .. WD Red HDDs).

Another thing I've found over the years is that ready made(plug'n'play) external drives don't use the highest quality components(neither the physical hard drive, nor their controllers) .. so building your own external drives from better components works better .. 25-50% better speeds.

Robert(Seapy) mentions a reasonable possible solution. M.2 drives are insanely fast, and not always require a second mortgage to build a decently fast and large storage solution.
I recently (18mths ago) built a PC for my son and took 'a punt' on a no-name M.2 SSD manufacturer  ... and it paid off nicely (for him). Much faster and higher quality SSD than I expected.

ADATA XPG SX8200 series M.2 SSDs .. can come in up to 4Tb drives if required. They were about 50% cheaper than competing spec M.2 drives .. recommended if wanting 'fastest' transfer rates.

There are devices you can install into your PC(onto PCI slots) that allow you to add multiple of these M.2 SSD drives.

eg. ASUS Hyper M.2 x16 PCIe 3.0 x4 Expansion .. allows 4 M.2 drives to be installed onto the motherboard.

So if you take the ASUS expansion card, add 4x 4Tb ADATA M.2 drives, you get 16Tb of some of the fastest storage you can get for your PC.
There are some caveats tho .. it may not 'just work' .. sometimes specific hardware maybe needed, and/or some hardware configurations can clash(eg. at boot up) so some things may require some tweaking here and there(eg. BIOS). 

If you can elaborate some the PC specs you have, I(or anyone else) can advise if this kind of setup will work on your current PC.

Magnetic drives .. bigger is usually better(faster) for a given brand/model. I've found that currently WD Black drives are faster on my PC than other brands(Seagate) .. with the side benefit that they don't fail .. at all. My 3 Seagates have all failed!
Anyhow, traps not to fall into with external drives: Don't waste money on fast SSDs for external drives, when there is little hope to ever use that SSD speed, where the interface just can't reach those speeds itself.
You may have 500Mb/s SSD, but most USB3.x interfaces can't reach 150Mb/s in the real world .. that SSD is wasted money for that connection type. .. better to add it into the PC, directly to the M/board.

Also think about your filing system too. Do you keep all those 500K+ tif files in the one directory, or are they in separate directories? Separating lots of files across many folders usually makes caching faster.
I found the fastest image reviewing program I use is Fastone Image Viewer. Strangely!!! .. for tif files, Nikon's (usually reviled) ViewNX2 is next best. I don't use many tif files .. any NEF file that has been converted to tif file(only for stitching here) is deleted pretty much straight after the image has been used. Once you have the NEF, a tif file is trivial to create.
Only tif files I have kept are old scans of films I have done over the years. I stopped scanning and started to use the cameras to digitize films ... so NEFs now.
While I don't have 500K tif files .. maybe a few hundred at most, they aren't very large either .. so FSViewer works quite well for me.

The other software I will give a shout out for:
XnViewMP. (free)
It works really well if you follow one small tip(if you try it) ..
Open the program -> look in the toolbar for a folder icon with two squares attached(it's called Show files in subfolder). click this, and then you really need to wait a bit. it builds up a database/thumbnail system .. which will become quite large(12Gb on my 1.8Tb of Photos).
It's not a necessary step, all it does it build the database/thumbnails all at once, and from then on, it will run very fast. If you have files in subfolders, it may take some time to build thumbnails. Once built up, it's quick.

Reason I mention this is that the only use I have for this program is to manually verify that my 'files'(images) are not corrupted. I once had a problem with one of those diabolical USB controllers(the JMicron type) that created a host of corrupted NEF files I didn't know about. Files looked seemingly fine .. same size as they were supposed to be, file name OK, nothing looked wrong. But some anomaly corruption during a file transfer(via the JMicron USB device) and files could never actually be opened again. I confirmed that some of the corrupted files were the same size as some archived files I had, but no good.
One issue with having files across many subfolders is the ability to see each image .. I mean with my eyes. Confirmation that the file existed and looked like it was OK wasn't enough for me .. I had to 'eyeball' it to make sure it's working.
This is where XnViewMP comes in .. only program I can find that will display all files at the same time in a single window when they are all in subfolders .. tifs/NEFs/jpgs/whatever. if I don't see the thumbnail image as I hold the page down button as they flick across the screen, there may be something wrong. I do this(manually) every time before I commit to updating my backup locations with files.
I can scroll 1.5Tb/200K files(nefs/tif/jpgs/etc) continuously with XnViewMP without any pause. As they flick by and if I see any white square(ie. no thumbnail image) I have to investigate.
The image files(to be backed) when I do this process are all stored on a fairly lowly specced WD Black drive(HDD) in my PC. It's max throughput has been measured at 140MB/s on it's SATA controller .. ie. It's not slow, but it's not the fastest storage connection I have on this PC.
 
Have found one other corrupted file from my old D70s over the years since the JMicron debacle some years back.
Also .. as side note .. I have two or three corrupted images from my D70s(my son shot them), where the files got corrupted when written too on the card at the time.
Only programs that display the corrupted image as the thumbnail are Nikon's programs(ViewNX2 and CaptureNX2 AND XNViewMP) .. all other raw file capable software don't show an image of those corrupted files.

Hope this helps in some way.

[Michael Erlewine]

Thanks Arthur


I have hundreds of thousands of photos

System is:

Windows 10 Pro
Intel® Core™ i9-9900K CPU @3.60GHz 3.60 GHZ
64 GB
Motherboard Z390 Designare
Video Card PNY Quadro P5000 PCI-E 16GB

[arthurking83]

Thanks for that info Michael, helps a bit.
From my understanding that ASUS 4x M.2 add on card will work on your motherboard if you want more internal storage space within the PC itself.

What other specs for your PC?
What storage config?
ie. do you have M.2 storage? if so, one or two? What model SSDs?
 Do you use plain old SATA drives? if so, SSD or HDD?

Can we assume that you have backup storage available for your images?

eg. I have a NAS(QNAP) where I archive my images, but still have all those backed up images available on a HDD in the PC too.
Plain old simple redundancy.
Because the NAS is over a Gb Ethernet connection, it's not particularly fast(access) and all programs slow down when trying to access the files on the NAS.
So a relatively fast (like a cache) storage space makes file access quicker for using those files on the PC. Once edited, back them up to the NAS for data integrity.


if you have this setup, then it's not a totally mad idea that if you create a RAID0 internal storage space in the PC for the everyday use of your files .. will speed up file access to them.
RAID0 is not a totally secure way to 'archive' files .. if it fails in any way you lose all data .. no redundancy.
Of course there are other RAID methods that can also help access speed, but you will be looking to balance space vs speed .. etc.

I used to have a RAID0 config on a very old PC, but back in the day before SSDs were a real thing for us mortals. Old slow drives used to benefit from RAID0. But it was only a setup for speed, not backup.
Just my experience but I just don't trust RAID at all any more, so just keep multiple backups and do it semi manually.
I'm not entirely sure that RAID0 on a set of very fast M.2 drives may produce any real benefit either tho(never done it). But on plain jane(ie. slower) SATA SSDs it can help a bit. Benefit wasn't worth it tho.

Those ADATA M.2 XPG SSDs do have amazingly fast read/write times(BTW, your primary concern here is read times going by your OP). Write speeds are usually also tied to read speeds tho, so one will go hand in hand with the other.

I have two 400-500MB/s capable SSDs in my PC, connected to SATA. configured in RAID0 they give 600-700MB/s, faster for sure, but the benefit isn't worth the effort.
My M.2(Samsung) isn't as fast as the ADATA M.2 I got for my sons PC. The motherboard limits it's speed to a degree. My PC is now about 5+ years old tho. My sons was built last year, so it's newer tech.

Where the ADATA XPG SSD tops out at high 1800MB/s on my motherboard, on his newer motherboard it reaches 3000-3200MB/s easily(all read speeds). Write speeds are a lot slower at about 1500MB/s and 2500MB/s respectively! 
What this shows is that just spending big $s on the fastest hardware doesn't necessarily equate to faster hardware .. remember the comment that the weakest(or slowest) link in the chain will be the limiting factor.

eg. lets say you decide to get yourself an external M.2 enclosure and connect it via Thunderbolt/USB-C. You can easily get 3000+MB/s transfer rates from a M.2 drive. Thunderbolt(on the PC) may give you about 2000MB/s transfer rates.
Thunderbolt(on the external enclosure) may only give you 1000MB/s tho. Remember the point I highlight that Bob Foster first noted ... not all chips are created equal. Each device has it's own chipset(interface) device that allows communication(data transfer) between the various devices. This is just a hypothetical eg. and to be sure .. 1000MB/s storage transfer rates is not slow, and would(or should) never slow down your preview experience of your images the way you described in your OP.
The process I described that I do in XnViewMP never causes me concern.

Have you done any storage speed testing?
There are many free programs out there, one I use for all storage testing(easy to read/use/understand) is Crystal Diskmark. You don't need to install it(which I hate to do uncessesarily) .. just run it once you unzip the download.
It then outputs a text that easy to understand.

eg.

<sub>CrystalDiskMark 5.1.2 x64 (C) 2007-2016 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

   Sequential Read (Q= 32,T= 1) :  1667.959 MB/s
  Sequential Write (Q= 32,T= 1) :   969.903 MB/s
  Random Read 4KiB (Q= 32,T= 1) :   265.704 MB/s [ 64869.1 IOPS]
 Random Write 4KiB (Q= 32,T= 1) :   200.334 MB/s [ 48909.7 IOPS]
         Sequential Read (T= 1) :  1400.098 MB/s
        Sequential Write (T= 1) :   963.801 MB/s
   Random Read 4KiB (Q= 1,T= 1) :    48.157 MB/s [ 11757.1 IOPS]
  Random Write 4KiB (Q= 1,T= 1) :   122.435 MB/s [ 29891.4 IOPS]

  Test : 1024 MiB [C: 14.3% (33.9/237.9 GiB)] (x5)  [Interval=5 sec]
  Date : 2016/10/02 0:30:48
    OS : Windows 10 Professional [10.0 Build 10586] (x64)</sub>

it just gives an idea of how fast your storage devices are operating in your system.

[David H. Hartman]

3) I use a bunch of things, but if you are concerned about speed, pictureflect https://pictureflect.com/ is pretty snappy. But have only used on .jpg  Free also.

I'd like to try this program but I'm using a school computer where the Microsoft Store is blocked. A tech guy setup the account I'm using as an admin so I can install programs but the Microsoft Store is blocked. I've tried to edit the group policy in the windows policy editor gpedit.msc and the registry editor regedit.exe but the the Microsoft Store is still blocked.

Does anyone know where I can download a non-Microsoft Store, Offline Installer? Any help will be appreciated.

Dave