I recently got the 400/4.5 and it was indeed a surprise. The combination of high optical quality, (apparently - so far no surprises) high build quality, and very low weight make for an unbeatable combination. I can easily operate the lens hand held at slow shutter speeds unthinkable for such long focal lengths in the past.
On the other hand, the standard f/1.8 lenses such as 24, 35, 50, and 85 are rather beefy and handling them is not as easy as using their F mount equivalents, unless one aims to rely solely on AF.
For the Z/DX, third-party offerings from makers such as Viltrox, Laowa, and Voigtländer are pretty small and most handle great due to better user interface than the sleek Nikkors.