That is not correct. As the filter is part of the optical system (by virtue of being in the light pathway) it most definitively will influence focus. Expect a focus shift approx. 1/3 of the filter thickness. Perhaps not much in absolute figure, but for a short focal lens such as the 15 mm, that will throw infinity focus well and truly off, as you already discovered.