Advice needed on extracting date and time from 15000 JPEGs

[Bruno Schroder]

I'm looking for advice on the easiest way to extract date and time of shot of about 15.000 jpegs. The images are stored in multiple directories of roughly a thousand files each.
There are two ways of accessing the info: at file level and in the exif. Both could work but the images are from an IR camera trap and also store temperature and moon phase. It is possible (likely) the data scientists might need these additional exif parameters in a second stage, so an exif based method is preferable.

Project background:
The participants at the Scottish meeting could note my interest in bats, they had to suffer a stroboscopic session when I flash photographed the emergence of about 100 bats from under the roof of the lodge. You might have noticed the composite image Birna made of a few of the shots.
I'm involved in a multiyear bat conservation project trying to measure the influence of microclimate conditions in occupation of bats shelters. I'm in charge of the data acquisition, meaning temperature and humidity sensors and IR recording of the presence of bats.
Roughly speaking, I'm getting 600K datapoints and 50K images a year. Images are reviewed by volunteers and when bats are present, images are tagged and a pointer is added to the database of the sensor data at the right time and date. In a second stage, data scientists will look for correlations between whatever interior or exterior climate data and the presence of the bats.

All in all, it means 15000 images to handle. Automatisation is very welcome 

How would you do it?

[Stephan]

On Mac I use "exiftool", don't know if it's available on Windows though.

Renamed and organized fully automatically >50.000 pics according to date/time shot, parsed through subdirectories and created new ones (Y/YMD/YMD.nef in my case)

unbelievable powerful which also means complex, needed a lot of advice from the net, but there is a lot available -> so highly recommended!

[Bruno Schroder]

Thanks, Stephan. It is also available on Windows. I'm occasionally using it in drag and drop mode for 1 image. I'll dig deeper in the other functionalities.

[Matthew Currie]

There is a program called "Exiftool GUI" which piggybacks on Exiftool, and runs in Windows.  It allows you to export EXIF data to separate files, HTML, Text, etc., one data file for each image, and can do a whole folder in bulk. The new files will be added to the folder, so accompanying, say, DSC5000.JPG would be DSC5000.TXT.  So far I see no way to have it export only selected data, so every accompanying file has the whole lot.  But you might be able to do some kind of filtering or selecting from those resulting files afterwards.

You can probably do just about anything in the core Exiftool program if you can figure out how. But it's mighty complicated, and being DOS, it is pretty finicky about formatting.

I miss our bats (Vermont).  The white nose die-off has made the little brown bats nearly extinct here. 

e.t.a.  there is also a program called "exif pilot," which allows the display of selected exif values, but it does not seem to support copying or transferring the view in bulk.  The program is free but the bulk extension is sold, and I'm not sure how the export function works, as I so far haven't been able to make it actually export any information anywhere even on a single file, though if I read the instructions right it might 9with bulk extension added) be able to export to an excel file.  Unfortunately, you cannot select the date displayed, so the only way I can see right away to get it from the screen would be to screen print it piece by piece.  But on the computer screen it's pretty easy to read the selected data, with a little thumbnail optional for each picture.

[mxbianco]

You can look into the XDIR program: works in Windows, in DOS mode, but then you'd have to work with BAT scripts (pun intended).

In order to use XDIR you have to be sure that the creation date/time, or modified date/time are equal to the date/time shot, Exiftool GUI has a batch function for equalizing (copying the date/time shot to Create date/time *and* Modified Date/time). For example, if you copy iPhone pics from the cloud to your local drive, they get a create date/time that's different from the date/time shot. You solve it by using the equalize function.
An example of an XDIR command :

XDIR *.JPG *.NEF *.TIF / "FORM=*d[7890]*d[345]*d[312] *t,`*F*E`,`*D*P`" >> myname.txt

Pretty cryptic, huh? You have to read the documentation! the output is like this (CSV format):
...
2022/10/06 11:17:48,"DSC_7982.JPG","J:\DCIM\DA_DF\"
2022/10/06 11:17:50,"DSC_7983.JPG","J:\DCIM\DA_DF\"
2022/10/06 11:17:50,"DSC_7984.JPG","J:\DCIM\DA_DF\"
...
that is, date/time shot, filename, folder

Exiftool has the ability to extract selected EXIF fields to a TXT file, there's a limit of 1024 characters to the command line you can create

As an example, from pic DSC_7983.JPG you can extract the make, model, Lens model, exposure date/time, focal length with this command:

exiftool -S -G -tag  -exif:Make -exif:Model -exif:LensModel -exif:ExposureTime -exif:FocalLength  -c "%.6fº" DSC_7983.JPG

output goes to standard console ( = SCREEN), but you can pipe it to a text file by adding at the end    >myname.txt
or append it to a file by adding at the end   >>myname.txt

This is the command that creates a new file "myname.txt"
exiftool -S -G -tag  -exif:Make -exif:Model -exif:LensModel -exif:ExposureTime -exif:FocalLength  -c "%.6fº" DSC_7983.JPG >myname.txt

And this is the command that appends information to existing file "myname.txt"
exiftool -S -G -tag  -exif:Make -exif:Model -exif:LensModel -exif:ExposureTime -exif:FocalLength  -c "%.6fº" DSC_7983.JPG >>myname.txt


This is an incomplete list of the exif fields you would use in your script:

-IFD0:ImageDescription
-exif:Make
-exif:Model
-exif:LensModel
-exif:ExposureTime
-exif:FNumber
-exif:ISO
-Nikon:FocusDistance
-exif:FocalLength
-ExifIFD:FocalLengthIn35mmFormat
-ExifIFD:ExposureMode
-Nikon:WhiteBalance
-Nikon:FocusMode
-Nikon:NikonMeteringMode
-Nikon:AfAreaMode
-exif:Flash
-Nikon:VibrationReduction
-Nikon:ActiveD-Lighting
-exif:Orientation
-Nikon:HDR
-exif:DateTimeOriginal
-exif:CreateDate
-Nikon:TimeZone
-Nikon:DaylightSavings
-exif:Artist
-exif:Copyright
-exif:UserComment
-exif:Software
-Gps:GPSLatitude
-Gps:GPSLongitude
-Nikon:Altitude
-nikon:ShutterCount
-Nikon:MechanicalShutterCount
-xmp-dc:Title
-xmp-dc:Subject
-nikon:SerialNumber
-ExifIFD:LensMake
-nikon:Lens
-ExifIFD:LensSerialNumber
-nikon:DirectoryNumber
-nikon:FileNumber
-exif:exifImageWidth
-exif:exifImageHeight

commands can be quite long, for example I use this one to enumerate most EXIF fields:

exiftool -S -G -tag -IFD0:ImageDescription -exif:Make -exif:Model -exif:LensModel -exif:ExposureTime -exif:FNumber -ExifIFD:ExposureCompensation -exif:ISO -Nikon:FocusDistance -exif:FocalLength -ExifIFD:FocalLengthIn35mmFormat -ExifIFD:ExposureMode -Nikon:WhiteBalance -Nikon:Quality -Nikon:FocusMode -Nikon:NikonMeteringMode -Nikon:AfAreaMode -exif:Flash -Nikon:VibrationReduction -Nikon:ActiveD-Lighting -exif:Orientation -Nikon:HDR -exif:DateTimeOriginal -exif:CreateDate -Nikon:TimeZone -Nikon:DaylightSavings -exif:Artist -exif:Copyright -exif:UserComment -exif:Software -Gps:GPSLatitude -Gps:GPSLongitude -Nikon:Altitude -nikon:ShutterCount -Nikon:MechanicalShutterCount -xmp-dc:Title -xmp-dc:Subject -nikon:SerialNumber -ExifIFD:LensMake -nikon:Lens -ExifIFD:LensSerialNumber -nikon:DirectoryNumber -nikon:FileNumber -exif:exifImageWidth -exif:exifImageHeight -c -n %1 >>  C:\Programmi\EXIFTOOL5\ListaEXIF.TXT

and the output is something like this:

[EXIF] ImageDescription: Passeggiata nel vallone delle 7 Fontane, Altopiano des Baïsses 02-08-2021
[EXIF] Make: NIKON CORPORATION
[EXIF] Model: NIKON Df
[EXIF] LensModel: 70-180mm f/4.5-5.6 AF-D micro-Nikkor
[EXIF] ExposureTime: 1/1250
[EXIF] FNumber: 6.3
[EXIF] ExposureCompensation: -2/3
[EXIF] ISO: 100
[MakerNotes] FocusDistance: 1.50 m
[EXIF] FocalLength: 180.0 mm
[EXIF] FocalLengthIn35mmFormat: 180 mm
[EXIF] ExposureMode: Auto
[MakerNotes] WhiteBalance: Auto1
[MakerNotes] Quality: Normal
[MakerNotes] FocusMode: AF-C
[MakerNotes] AFAreaMode: Single Area
[EXIF] Flash: No Flash
[MakerNotes] VibrationReduction: Off
[MakerNotes] ActiveD-Lighting: Auto
[EXIF] Orientation: Horizontal (normal)
[MakerNotes] HDR: Off
[EXIF] DateTimeOriginal: 2021:08:02 09:37:06
[EXIF] CreateDate: 2021:08:02 09:37:06
[MakerNotes] TimeZone: +01:00
[MakerNotes] DaylightSavings: Yes
[EXIF] Artist: Massimo Bianco
[EXIF] Copyright: ©2021 Massimo Bianco
[EXIF] UserComment: ©2021 Massimo Bianco
[EXIF] Software: Ver.1.03
[MakerNotes] ShutterCount: 175821
[MakerNotes] SerialNumber: 8405658
[EXIF] LensMake: NIKON
[MakerNotes] Lens: 70-180mm f/4.5-5.6
[EXIF] LensSerialNumber: 200883
[MakerNotes] DirectoryNumber: 131
[MakerNotes] FileNumber: 1761
[EXIF] ExifImageWidth: 4928
[EXIF] ExifImageHeight: 3280


Ciao from Massimo

[Bruno Schroder]

Thank you, Matthew and Massimo, this is very useful, particularly your XDIR command example, which seems to be exactly what I need. I'll test later today and report here.
Thanks a lot.

[mxbianco]

Thank you, Matthew and Massimo, this is very useful, particularly your XDIR command example, which seems to be exactly what I need. I'll test later today and report here.
Thanks a lot.

I use the XDIR command to aggregate filenames from 54 folders, each one hosting the pics from the camera(s) I use on a particular day or period. Empty folders do not produce entries in the CSV. Then I SORT the CSV file on the first field (date/time shot) and I can reread the filenames and generate a GLOBAL serial number that's exactly sorted on the date+time of picture taken, almost exactly what you are planning to do. Naturally the cameras' dates/times have to be synchronized  for the process to be of any accuracy...

An added challenge is that I wanted the global serial # to be the same if I have both the .NEF and the .JPG versions of a picture, and eventually the associated vocal comment (a .WAV file). Solved this one too.

The global serial number can be used to rename the file(s), or it can be injected to an appropriate EXIF Field by using Exiftool (canNOT inject on the .WAV comment). It can all be automated with very little coding and a very large .BAT script.

Ciao from Massimo

[ColinM]

I can't help you at all with the question, but just wanted to say

- your BAT puns were as excellent as your solution was comprehensive Massimo

- I'm a newbie into the world of bats Bruno
I've watched them fly around the garden of the house we recently moved into and have just put up a bat nesting box.

There's a local group who are probably more at your level of expertise. But if you have any good links or suggestions for a newbie Bruno. I'd love to see them

[Bruno Schroder]

Colin,

in English, I'm using these web sites: www.bats.org.uk, www.batconservationresearchlab.co.uk and www.batconservationireland.org. These 3 combined do provide a good range of info, from very introductory to scientific publications. Besides that, forums are also a good source of info but I only know French speaking ones.
There is a also good number of bat conservation Twitter accounts regularly posting. The 3 organisations above are a good start.

Engage your local group if you can. The bat conservation groups are sometimes difficult to join because bats are easily disturbed and hibernation disturbances cause early exhaustion of fat reserves, which is often lethal in spring. Cautiousness and adherence to rules is critical and conservationists try to filter 'tourists' out.

That said, bats conservation is full of intriguing questions and the bonus is that once you're in, you get access to some unbelievable places.

Cheers