If you're hitting a wall at four or five tracks, something is very, very wrong. I used to do eight tracks at once on a dual 2 GHz G5, stacking up twenty or thirty tracks at a time, without any problems. It takes a tiny fraction of those CPUs' horsepower to handle audio recording (unless you're also stacking up lots of CPU-intensive effects like convolution reverb).
If you're using an external hard drive for recording, stop right now and add another internal SATA drive for recording. I've seen some badly written audio apps that misbehave horribly when used with external hard drives (even FireWire) because they don't ask for data to be read from disk early enough to have the data ready when it is needed. If Cubase has an option to adjust its disk cache size, try cranking it up.
Second, that particular audio interface is DICE-based, which IIRC were notorious for these sorts of problems at first. Make sure you're using the latest version of the manufacturer's drivers.
If none of those things help, part of the problem may be Cubase. Try switching to Garage Band and see if the problem goes away. If so, buy yourself a copy of Digital Performer (current version) or an older, G5-compatible version of Logic.