For me, between the three format it comes down to lens choice. There is very little choice for CX and relatively little choice for DX, although if you rely mostly on zooms you can get covered with wide-angle, normal, and tele-zooms on CX and DX as well. For FX, there is a vast choice of lenses from old to new.
Including other formats than the three mentioned, there is also a lot of lenses to choose from in micro 4/3, and Fuji has a small, but very reasonable line-up. If size and weight is high on the list of priorities, those two systems should be considered as well.