Author Topic: Handling Large Image Files  (Read 412 times)

Michael Erlewine

  • Close-Up Photographer
  • NG Supporter
  • **
  • Posts: 1632
  • Close-Up with APO
    • Spirit Grooves
Handling Large Image Files
« on: August 21, 2020, 11:19:55 »
I have many large image files, most of them TIF files, the resulting images of years of focus stacking. As for the original layers, there are probably half a million or more of those. Let’s just look at these large TIF files.

Right, now I am trying to work (Windows 10) with the very large TIF files. I have tried Infraview, Bridge, PhotoMechanic, Windows Explorer, and so on. All of them seem to kind of choke up.

I have a pretty fast computer, yet nothing seems be very fluid. I am wondering what other photographers do with this situation. Here are some of my questions:

(1) What kind of external hard drives are best?

[I have tried SATA, SSDs, and all kinds of external drives, but none seem to be a charm, so to speak.]

(2) What size external hard drives are best?

(3) What viewing software do you use?

[As mentioned above, Adobe Bridge, Infraview, and so on see to bog down, take forever to load so I can see thumbnails, and generally prove almost unusable except at a snail’s pace.]

All suggestions are welcome as to how to deal with this ever-growing problem. Thanks.
MichaelErlewine.smugmug.com. Founder MacroStop.com, MichaelErlewine.com (articles), https://www.youtube.com/user/merlewine (video tutorials), All-Music Guide, All-Movie Guide, Classic Posters.com, Matrix Software, SpiritGrooves.net, DharmaGrooves.com

Jack Dahlgren

  • NG Member
  • *
  • Posts: 1180
  • You ARE NikonGear
Re: Handling Large Image Files
« Reply #1 on: August 21, 2020, 21:05:29 »
Michael,

1) SATA is a type of interface, not a type of drive. You will want a fast Nvme (solid state) drive connected through a thunderbolt interface to reach top speeds currently - assuming that data transfer is the bottleneck.

2) Depends on how much data you have. Price/capacity is linear in the middle of range so probably good to choose from that range.

3) I use a bunch of things, but if you are concerned about speed, pictureflect https://pictureflect.com/ is pretty snappy. But have only used on .jpg  Free also.

4) I'd probably put on a fast internal m.2 or NvME drive instead of external.


ColinM

  • NG Supporter
  • **
  • Posts: 1011
  • Bristol, UK
    • My Pictures
Re: Handling Large Image Files
« Reply #2 on: August 21, 2020, 21:49:21 »
Thank you Jack - I learned a lot from your post.
I'm sure Michael and others will also be googling, but this article (over a year old now) gave me some brief background, plus also indicated adding these drives to an existing machine might be harder than buying a new one that was built ready to use them

https://www.pcworld.com/article/2899351/everything-you-need-to-know-about-nvme.html

Michael, I can't vouch for any speed improvement on really big TIFF files as I don't use them, but I've found FastStone to be faster & more convenient than some other image browsers

https://www.faststone.org/FSViewerDetail.htm

Michael Erlewine

  • Close-Up Photographer
  • NG Supporter
  • **
  • Posts: 1632
  • Close-Up with APO
    • Spirit Grooves
Re: Handling Large Image Files
« Reply #3 on: August 21, 2020, 23:39:08 »
Michael,

3) I use a bunch of things, but if you are concerned about speed, pictureflect https://pictureflect.com/ is pretty snappy. But have only used on .jpg  Free also.

Yes this program, "Pictureflect" is indeed powerful, free to try, and only around $2.00 to purchase, and it works on large TIF files, better than most. The program "FastStone Image Viewer" was the same old too-slow to work on large file.

Tanks Jack!
MichaelErlewine.smugmug.com. Founder MacroStop.com, MichaelErlewine.com (articles), https://www.youtube.com/user/merlewine (video tutorials), All-Music Guide, All-Movie Guide, Classic Posters.com, Matrix Software, SpiritGrooves.net, DharmaGrooves.com

Michael Erlewine

  • Close-Up Photographer
  • NG Supporter
  • **
  • Posts: 1632
  • Close-Up with APO
    • Spirit Grooves
Re: Handling Large Image Files
« Reply #4 on: August 22, 2020, 10:00:20 »
Since I usually stack images from several to hundreds of layers, the resulting stacked image is a fabrication and does not carry all the info one finds in raw files, which is what I use for layering. I keep the raw files, so EXIF data is available if I am interested.

Actually, for my interest, I am only interested in a few pieces of data.

Filename
Lens Name
Keyword

I don’t really care about most of the other data because, in general I use APO lenses wide open (or nearly so), so since I know the F-Stop of the lens I am using. And most of the lenses I use are “exotics” meaning they do not usually interface to the camera to provide the standard EXIF data.

And I almost always use the lowest ISO, which for the Nikon cameras I use is ISO 64. So, the variable is the shutter speed and I don’t care about that.

An image filename might look like this:

8AE-0142-EL105-Morning-Glory

Where:
8AE-0142 =  the camera-generated filename
EL105 = APO El Nikkor 105 f/5.6
Morning Glory = the kind of subject

That all I need. From the filename (and the original NEFs (raw files) I can look up EXIF data if I need it.

However, what I am interested in is the particular lens used and not much else. I have a lot of lenses, mostly APO lenses, and their differing IQs vary, often subtly, so I want to know what lens was used.
MichaelErlewine.smugmug.com. Founder MacroStop.com, MichaelErlewine.com (articles), https://www.youtube.com/user/merlewine (video tutorials), All-Music Guide, All-Movie Guide, Classic Posters.com, Matrix Software, SpiritGrooves.net, DharmaGrooves.com

Bob Foster

  • NG Supporter
  • **
  • Posts: 2
  • You ARE NikonGear
Re: Handling Large Image Files
« Reply #5 on: September 03, 2020, 19:47:40 »
As I’ve been able to make time over the past 10 days I’ve done some experimenting with rendering 250-265 GB .tif files on a basic commodity computer (i5 4 core @2.7GHz,  8 GB RAM, video card- NVIDIA GT 730, 2 TB HDD, and USB 3.0 ports). The results on Windows 10 were a surprise. I have two suggestions.

1. In general, with all file systems, the smaller the number of files (in this instance .tif image files) within any particular directory (folder) the better that any program that "prefetches” files will perform. If it will not affect whatever means you presently have of grouping and/or indexing your files I'd strongly suggest the use of smaller directories/folders. If you wish to test this try limiting the contents of any particular directory to those images, by filetype. E.g.:

(Directory) Subject-Lens Identifier- (in any order)

(Sub-Directory) .NEF files. I keep the originals because software has arguably become better over the years at rendering nuances from the extreme ends of the dynamic range.

(Sub-Directory) .TIF files. I store these in compressed form to save disc/ssd space but decompress them prior to any use. To these filenames I add by batch lens and subject info leaving the camera filename.

(Sub-Directory) The directory generated by Zerene Stacker.

2. Contrary to what might seem like common sense, the use of prefetch routines in computer code can have a deleterious effect on computer performance when data sets are very large. RAM usage can sometimes have a very negative impact on program execution, it’s also possible to choke a drive with excessive demands on its read speed and buffering capabilities. Also, not all USB host controllers are equally capable; particularly those rated USB 3.X.  In some cases going to the USB controller manufacturers website and downloading the current driver can make a very significant difference in throughput. For many devices Microsoft’s update service has never kept drivers current.
 
I created a directory containing over 2000 .tif 250+ GB files on a QNAP TR-4 USB 3.1 External RAID Unit. The directory is over 500GB in size. The directory was also copied onto each of a pair of 5TB Seagate Backup Plus drives for testing purposes. I’ve avoided the use of all USB 3.X TB thumb drives because all apparently responsible reviews that I’ve seen indicate that to date performance of these drives is impaired by heat or purposely limited by the manufacturer.

I’ve stuck with Fast Raw Viewer because it is highly configurable and as a long time user I’m fairly familiar with how to tweak it to attain better performance.

The biggest gain I've been able to realize on the test computer was accomplished by turning OFF the prefetch function in Fast Raw viewer. I'm easily able to view each image at 1 second intervals using the QNAP RAID Unit either plugged directly into the test computer or plugged into another computer on my network with the appropriate directory mounted (Map Network Drive) on the test computer. I used a hardwire (not wireless) connection. See pages 122-125 of the current version of the Fast Raw Viewer manual. The Seagate Backup Plus drives do not perform as well: one is “comfortable” rendering at 1.5 second intervals, the other (older) unit takes about 2.1 seconds to render an ~260MB image in series.

Best wishers, Bob

Seapy

  • NG Member
  • *
  • Posts: 813
Re: Handling Large Image Files
« Reply #6 on: September 04, 2020, 10:51:26 »
Michael, I believe the fastest generally available processing storage (not long term storage) is an M2 blade drive on a PCIe card.  That is what my latest MacPro 5.1 has for the boot drive.  I also have a second small(ish) 250Gb version for my working image storage where I place my latest import from camera while I work with and cull files before producing the final TIFF images. This is also my Photoshop scratch disk.

You can also place SSD's on PCIe cards which give them direct connection to the processor(s) which speeds things up considerably, allowing the full potential speed of the SSD to be enjoyed.

The M2 drives are extremely fast, instant almost, on my ancient MacPro but not particularly capacious.  So once processed the finished image files are moved over to longer term storage and archive to free up space for the next batch.

The PCIe cards for the M2 drives are cheap enough, the M2 blades are less cheap, but in my view worth every penny (cent?).

https://www.megamac.com/collections/storage-solid-state-drives/products/480gb-owc-aura-ssd-for-mid-2013-and-later-macbook-air-and-macbook-pro-with-retina-display-owcssdab2mb05
Robert C. P.
South Cumbria, UK