I don't think switch drivers would affect the linearity of the DAC arrays.
However, sampler switches would do as well as the comparator.
So, there are trade-offs among those things wrt. your design targets.
Or, you need to alter your architecture for the LSBs.