Latency is about speed - so changing buffer sizes is a tweak, however, usually the best improvements come from the drivers. What interface and what DAW. Some just work poorly. What is the latency in mS? Most folk can work up to 10mS or so, but more than that and you start to hear the delays.