Special lenses such as the fisheye should be more tricky to assemble because of the tolerances tighter than, say, not-so-fast standard primes.
When I tried non-Ai Nikkor 20/3.5 (72mm filter thread) for the first time on F3, I was surprised by its pronounced barrel distortion. On the contrary, the one I finally purchased later didn't suffer from any perceivable distortion. The optics of the first sample was perfectly clean. I think that the first sample was badly assembled after it was cleaned and lubricated.
Unlike more modern lenses, the older MF Nikkors needed to be assembled by well-trained technicians using a handful of specialized tools.