Latency is about speed - so changing buffer sizes is a tweak, however, usually the best improvements come from the drivers. What interface and what DAW. Some just work poorly. What is the latency in mS? Most folk can work up to 10mS or so, but more than that and you start to hear the delays.
Latency in what, the input monitoring or actual offset between tracks that have been recorded? Are you using the interface's provided input monitoring feature?