At what distance? The 105/4 is a unit focusing lens - no floating elements - so it can be optimised for only one distance. For example the AI 55/3.5 micro is optimised at 1:10, I'm not sure if the 105/4 is the same. At distances outside the optimum range I expect field curvature and spherical aberrations to become progressively worse. With a well designed lens, the drop in performance can be kept minimal so it can be used successfully at all distances.
The AIS 105/2.8 micro does have floating elements, the basic structure is a double-gauss 75mm f/2 lens with close range correction applied between the front and rear groups, and a 1.4x teleconverter fixed at the rear, so there are three groups of lenses which should prove a higher level of correction at all distances. Surely the Nikon designers would have wanted their new f/2.8 macro to be superior to the older model over the entire focus range.
The increase of one stop and CRC means the new lens has twice as many elements, the extra glass/air surfaces means is has lower contrast when shooting into strong light (the 105/4 isn't perfect either in spite of only 5 elements in 3 groups). Also, the AIS 105/2.8 micro has a shorter focus distance at close range and doesn't work so well on extension, so you could question whether the newer lens is an all-round improvement, but I would be surprised if it isn't sharper between infinity and 1:2.
I have heard the CRC mechanism of the AIS 105/2.8 is delicate, a knock to the front could put it out of alignment. Maybe that, or the lower contract could account for the lower perceived sharpness?