NikonGear'23
Gear Talk => Processing & Publication => Topic started by: charlie on March 06, 2017, 23:45:22
-
I have a PC desktop and macbook laptop. The PC is used for post processing & cataloging. The Macbook is used for shooting tethered on location. Sometimes sharing larger amounts of files between the two can be a bit of an inconvenience being that my external hard drives are formatted for either one system or the other so my data transfer currently takes place using a 32GB memory card.
I was hoping someone might be able to share with me the potential risks, problems, or draw backs I might encounter if I were to format a larger external drive (2TB) to exFAT and use between the two systems. Mostly transferring files from the Macbook to the PC.
Perhaps there is an alternative solution I am overlooking?
-
Is using a NAS an option? They typically use an embedded Linux distribution and run Ext4 or other journaled file system(s). Such systems can handle huge disk volumes (I have them up to 40 TB). Access from Windows, Mac, or Linux-based installations is a breeze and completely transparent. The NAS is self-serviced and will take care of itself 99.9% of its up time. You can plug in and remove your machines to the network at any time without paying the NAS station(s) any attention. They just are there as a common resource.
Failing that solution, creating a common drive with exFAT ought to work. However, you need to pay special attention to details such as properly ejecting the drive once you are done with it, this is to avoid file corruption and other file system issues. Running CHKDSK frequently should be a mandatory maintenance. Beware that moving a disk cabinet around between computers will add to the risk of something going wrong. I have mixed experiences with this approach and prefer not to recommend it unless it's the only viable solution.
-
I actually had the thought of NAS in the back of my mind when writing this post. I've always shied away from NAS because of the initial cost of investment though it would be the better long term solution. I will look into it, thanks for the suggestion.
Are there any particular NAS enclosures that are more highly regarded than others?
-
I have settled on the Synology range, but there are certainly alternatives aplenty. Just get whatever is reviewed favourably and has an acceptable price.
When you decide the storage volume size, don't forget the overhead created by the particular RAID type the NAS is set up to use. RAID 5 is most economical, however, its total dependency on error-free disks for later replacement indicates one shouldn't use building blocks larger than 2 TB. Do get the dedicated 'NAS' version as well. Standard desktop disks will fail easier in an NAS ecosystem.
-
Qnap should be as good. Your choice may be a bit depending on local/regional offerings and pricing. After some varying experiences I now run WD Red disks of the 7/24 type with
greater better success.
-
Using a NAS also give the extra security if used in a RAID configuration that enables change of a failed hard disk. I have used this for some years now and have changed a couple of hard disk in that time. Now buying the WD red HD for exchange. I have a four bay QNAP, but I gues that the brand name is not that important, just buy a well regarded name.
-
Glad to hear about other well performing brands.
As to the NAS and its RAID setup, a few points to consider;
- Maintain the fan(s) of the NAS so they run silently. A NAS should merge into the environment and not be obviously present
- Contrary to commonly held views, RAID is *not* a backup. It provides an extra copy of your data, with possibility of real-time protection against a disk failure
- If you select RAID 0, you gain possibly access speed, and concurrently run a much higher risk of total data loss if one of the disks fail. The single-disk NAS station belongs to the same risk group
- Always ensure there is a backup to the data of the RAID station. Eventually you'll need it
- RAID 1 or 10 (mirroring) is least and RAID 5 most economic in terms of storage space available
- The underlying file system for the RAID station needs to be journaled and capable of self-repair. I believe most if not all products today delivers that functionality
- The RAID station should be tested for its handling of a failed disk. Do this with unimportant data and just pull one of the drives to simulate a disk crash. Can the data volume be restored in its entirety? Better to ensure this works before a disaster strikes
- Buy ample spare disks at the same time as the NAS is installed, and keep these replacements in a safe place
- Getting a spare NAS of the same type as well makes one sleep better :D
The overview might seem a handful but really is just common sense put to practice.
-
I'm using a Synology NAS, but have heard good things about Qnap as well. I guess it's a matter of taste and what user interface you prefer.
It's important to stress the fact Bjørn mentions: RAID is not backup. Another thing to remember when doing backup is that sync is not backup either.
Some of the latest Synology NAS and I believe Qnap too, support BTRFS, a next gen filesystem for Linux (these NAS boxes generally run Linux). BTRFS got some nice advantages compared to previous filesystems: https://www.synology.com/en-global/dsm/Btrfs
-
For backup I have an external disk connected to the NAS, doing backup every 12 hours. I hope this will do.
The backup drive is unfortunate not in my native format (Apple) but a Linus variant, but I can connect it to my mac if needed, tried it ;) just to be sure.
I actually bought the QNAP because they claimed HFS compatibility, buy that was not completely true ??? Otherwise it has worked fine.
-
I actually had the thought of NAS in the back of my mind when writing this post. I've always shied away from NAS because of the initial cost of investment though it would be the better long term solution. I will look into it, thanks for the suggestion.
....
Same here too.
I looked into building my own NAS box, as I have several PC boxes sitting gathering dust .. but in the end I opted for a QNAP myself.
4 bay multimedia device, rather than a pure NAS for data storage .. TS459A model.
I had a couple of drives already, and purchased a couple more when I bought the QNAP.
By the time you've added up the cost of getting larger and larger USB drives .. the NAS box(once bought and populated with excess capacity) isn't really that much more in the long term.
I'd just advise to have a plan for the future too.
Main reason I went with the QNAP I got, was that it had to justify itself with the multiple use of 1/ data storage, which is a backup to the single USB drive I still have connected to the PC of just my photos .. and 2/ the multimedia usage.
NAS is connected directly to the TV so that kids can play their movies too .. they just haven't learned how to use it for their music streaming yet(although it's easy).
So there are ways to justify it's cost, if you have multiple requirements .. and as you have multiple operating systems to work with, that in itself helps justify it's cost.
You can find cheaper NAS type devices, but I think in the long run a good quality, well featured NAS box will serve you well, and you'll never have to worry about it's initial cost outlay for many years to come.
Same can't be said about external USB storage.
I initially purchased a smaller (very) consumer model NAS box, 2 bay not very well featured device from DLINK. OK device, but very limited in what it could do, and a type of device predominantly sold on the basis that it can connect to a DLINK provided cloud based service!
Not some thing I cared for.
It worked, both for me(my photo backup storage) and kids(for their multimedia uses) .. but it was a very slow device, and limited in features.
Then I upgraded to the QNAP. I doubt I'll need to spend any more money on storage for a very long time to come now.
A quick word on backup too: there are two types of backups when it comes to NAS(boxes). Data backup and device backups. Found this out the hard way.
A few days after I got my QNAP, it played up.
Initially I thought it was a dead disk, as one (of the 4) went missing from the NAS system. Got a new disk to get it all back online, and nothing!
Turned out to be a dead HDD bay/connection(internally).
Not a happy camper here!
Anyhow, contacted QNAP, they got back to me immediately and advised to get it back to the retailer .. thank god for local suppliers! .. and within two my replacement was arranged.
So while I had a bad experience re the hardware, QNAPs customer service was top notch .. no attempt from them to wiggle out of warranty issues(as seems to be the norm nowadays!) and no questions asked re my ability to look after my gear.
With that in mind, I'd still recommend the QNAP, simply based on their customer support services(pity same can't be said of Nikon :p)
Back to the backup anomaly: So I had my QNAP NAS running as I liked, and I assumed that if I replace all the hardware(ie. 4 HDDs) in the exact configuration from the old box into the replacement box, that it should just start off from where I left it!
Nup! nothing .. had to start all over again, even formatting the drives all over again. It wouldn't recognise any of the data on the drives, even tho it was the exact same device!
So backup your data as already suggested, but also backup the actual NAS device too.
Or as Bjørn commented ..
- "Getting a spare NAS of the same type as well makes one sleep better :D"
A feature I never even considered when looking at the options available was 'port trunking'. It's a system where you aggregate multiple connections into a single faster connection.
I didn't go for the QNAP I got for this feature, mainly for the storage and multimedia ability .. but the best feature is by far the port trunking.
The QNAP was already much faster in data transfers(over gigabit cabling) than the cheap little DLINK box, but I then enabled port trunking(over double gigabit cables) and data transfer upped from mid 40's to 50's Mbyte/s to over 60Mbyte/s easily(15-25% just with a simple extra wire).
Doesn't sound like much, but when you transfer 1.6Tb of data, it counts!
My recommendation for a NAS .. look for a proper(slightly more expensive) NAS solution, rather than a consumer(cheaper) solution.
Just because you get a 4 or 6 bay box, doesn't mean you HAVE to fill it with hard drives, those slots can be empty, and used for future expansion.
It's tempting to go for a cheaper option, but in the long run, I dare say .. you may want options or features that are only possible in the more robust devices.
-
Thanks everyone for your input.
The more I look into NAS the more I like it for its versatility of connecting to multiple devices and streaming capabilities though its primary function would be a working catalog for my photographs and transferring files between machines. So far I've looked into WD and Synology though it seems Synology and QNAP are the more widely used enclosures amongst the data storage professionals of the world. I like that Synology has an esata option for a 5 bay expansion enclosure should the need arise. I will look into QNAP next.
My thoughts are starting with a 4 bay enclosure running 4TB WD RED drives. My desktop (editing machine) has faster 7200rpm HDDs where I would import currently 'active' work to which would be edited, post processed, and backups sent to external HD's. Once the files are finished being edited & selects sent to clients they would be moved to the NAS for longer term storage/access freeing up the desktop drives. Not certain which RAID configuration I'd use as of yet.
If anyone see's something flawed with that plan or has a better alternative I'm all ears, though I am on a budget.
-
For backup I have an external disk connected to the NAS, doing backup every 12 hours. I hope this will do.
The backup drive is unfortunate not in my native format (Apple) but a Linus variant, but I can connect it to my mac if needed, tried it ;) just to be sure.
I actually bought the QNAP because they claimed HFS compatibility, buy that was not completely true ??? Otherwise it has worked fine.
Perhaps by HFS compatibility, they mean you can connect to it using AFP and use it as a Time Machine backup solution. I expect Time Machine to change in the not too distant future though since Apple has created a replacement for HFS called APFS (https://en.wikipedia.org/wiki/Apple_File_System), it looks like they will actually roll it out on IOS devices with the next IOS release 10.3 which is in beta now. They might very well make it the default filesystem for the next MacOS release as well. APFS includes some features (just like BTRFS) that makes backup safer, better and more efficient, which means Time Machine will have to be updated as well to support it.
-
You may be right regarding my misinterpretation of the HFS compatibility.
It for sure will be interesting to see what happens.
I hope that QNAP will update if needed.
Is using 10.3 beta on my iPhone allready 8)
-
I know my Synology NAS will not, but it's so old it's only getting security updates from Synology.
-
.....
My thoughts are starting with a 4 bay enclosure running 4TB WD RED drives. My desktop (editing machine) has faster 7200rpm HDDs where I would import currently 'active' work to which would be edited, post processed, and backups sent to external HD's. Once the files are finished being edited & selects sent to clients they would be moved to the NAS for longer term storage/access freeing up the desktop drives. Not certain which RAID configuration I'd use as of yet.
If anyone see's something flawed with that plan or has a better alternative I'm all ears, though I am on a budget.
I don't see a flaw(as such) but consider the implications of RAID!
RAID 0 you don't want .. simple as that.
RAID 1 and 5 are similar, but different enough to warrant their existence.
Problem is the wasted disk space if the need ever arises.
I have 4x 4T WD Reds myself in the QNAP, and each individual Disk allows me a few years of storage headroom.
Currently have 1.6Tb of image storage needs, and I estimate that 4Tb will last me about another 5-10 at least.
So in 5-10 Years time, I'm allowing the need for larger (say 8 or maybe 16Tb) drives by then. in the years I have what I have, they'd have 'paid' for themselves in the work they'd done for me.
Problem with raid:(eg. RAID5) you need 3 disks to cater for the space allowance of 1 disk. Great that they're safe with parity and pretty much disasterproof relaiability .. but the wastage of requiring 3 disk for the privilege having one disk at your disposal isn't what I'd called efficient.
In your situation above 3 of your 4 4TB drives that total 12Tb will only give you 4Tb of usable space! The last 4Tb drive will hold the NAS box's OS and you can store other data there too. so you have just under 8Tb of storage space available.
So what I've done, is a psuedo RAID 5 but manually done.
2x 4Tb drives on the QNAP(ie NAS box) and the remaining external USB drive still hooked up the PC.
(Originally I had 2x USB drives on the PC as my 'backups')
The USB disk at the PC is USB3 and so data transfers are fast(100+Mb/s) so accessing images is painless .. no waiting for loading and so on.
The 2x 4T drives on the NAS is simply for backup purposes. I rarely access them.
In fact the way I have it setup, is that one of those drives on the NAS is mapped in Windows, so that I can access it easily, but the other is not.
Reason is for security, if the need arises. eg. if I accidentally attract some virus/trojan that scans all drives and deletes/destroys data(my sister had that happen) at least the one drive is unaffected .. it's on an alien OS relative to the Windows malware.
*Note that this above scenario is unlikely, in that I've never had a virus of any kind(I'm one of those cautious types that avoid rather than curious) .. but it's a 'plan I have'
Anyhow.
So my backup routine consists of:
Take images, download to PC.
On the PC I use a 1Tb SSD for the days images. This drive only stores images for the year. Every year they get deleted and I start again.
The 2TB USB drive on the PC is my archive. That drive is the one that is backed up on the 2 4Tb drives on the NAS. Both manually done. the networked drive is backed up regularly(as it's easy to do because that drive is mapped)
The non mapped 4Tb drive on the is the failsafe(as long as it doesn't physically die!!)
I only back this one up at half yearly or yearly intervals.
Main reason for that failsafe backup drive is that a while back I was in auto dumb mode(as we sometimes tend to get into) and was regularly backing up HDD to HDD of my images.
Little did I know, that at some point there was some data corruption of some of those NEF files. So I had no idea that I was backing up corrupted data. I lost about 20-30 NEF files, but I don't think they were a major loss.
So this unmapped drive is my 'snapshot' drive. before I backup to it from the mapped NAS drive, I thoroughly check close to every NEF file via whatever software allows me to do so quickly.
I used to use ViewNX in raw mode, but it's slow over the network connection, so I tried Microsoft's Photo Gallery(works well enough) and then I tried XNViewMP fast over the network connection.
This involves a very brief manual inspection of each thumbnail of the NEF images.
I don't know of any program that can verify the integrity of each NEF file automatically.
When I do a backup/sync of the drives, I use FreeFileSync to inspect the differences between each drive. It just displays if there are, and what differences there are. But I use FastCopy to do the actual backup/syncing duties, using the 'verify' option.
Reason I use FreeFileSync is that I'd like to see visually what, if any, differences there are ... rather than just do a backup for no reasoning.
RAIDs take a while to build up. and if you have to replace a drive, you have to rebuild the raid again. If you need more storage space, you need to update all drives, as the storage space is limited to the size of the smallest HDD.
I have close to 16Tb of flexible storage on my NAS box, if I needed backups of more than 4Tb of data, I can easily acquire just a single >4Tb drive and use that .. singularly .. and update the other drives singularly as needed.
(you did make the point that you are on a budget!).
While three individual HDDs acting as a pseudo RAID 5 is more work, and not quite the same in terms of being failsafe, I think the prospect of those 3 drives all physically failing at the same time is too remote to concern myself with.
I used to have 2 x external USB drives connected to the PC, and when the one failed about a year ago, I looked for the NAS solution rather than to replace it .. so I've had HDD failures(usually Seagates too!).
Not a good feeling only having to rely on the one USB drive, that Murphy's Law dictates is likely to go wrong just when you least need it too :p
-
This must be a misunderstanding as to how RAID 5 or other schemes work.
With N disks each of size M or larger, the storage capacity is (N-1)*M for a RAID 5 layout. The parity information is distributed all over the members of the disk set. For example, with 4 (N) disks each of size 4 TiB (M), you get an RAID 5 volume of 12 TiB.
The least "efficient" RAID scheme is RAID 1/10, which in the example above would provide N*M/2 or an 8 TiB volume. RAID 6 results in (N-2)*M thus in this example also 8 TiB. There are numerous other RAID schemes but these are hardly practical for the comparatively small disk arrays of the home user.
The storage used for the operating system of the NAS is very small so can be safely ignored in these calculations.
-
I read somewhere exFAT is not as robust as other file systems because there is only one copy of essential system areas.
-
Journaled file systems are more robust, true enough. exFAT in the normal implementation lacks this feature. It does have checksum as integral parts of the file system, and there is a transaction-adopted version (TexFAT) as well.
An NAS uses journaled file systems thus are not vulnerable in the same sense as a USB stick or SD card. Communication over the network to the computers accessing the NAS is through SMB/CIFS or other universal protocols.
-
Hi Charlie
I also use PC desktop and Mac laptop. My external image drive is formatted NTFS and I use Paragon NTFS driver for W7/Mac read/write compatibility.
I also have a 240GB Sandisk 500 USB with two partitions (partitioned with iPartion which supports HFS and NTFS), one NTFS for image files and LR library, and the other smaller partition for HFS as a swap drive for CS when I hook it up to the Mac. So far works quite nicely for this application and swapping LR library back and forth between PC and Mac.