You can do exactly what you want using Sonar and a MOTU 2408. I have the exact same setup. In Sonar, you can send each track to any pair of analog outs on the MOTU 2408. For example, you can send tracks 1-8 to analog outputs 1+2 (stereo) and tracks 9-16 to analog outputs 5+6 if you want.
By doing this, you can send any track or group of tracks out to an outboard processor. From there you can do two things. The first doesn't require a mixer, but it does require a fast computer and may not even work doing that.
1) You can send the output of the processor back to a pair of analog inputs, then setup Sonar to monitor that pair of analog inputs. There may be a slight delay, though, between the rest of the tracks and the processed tracks.
2) Put the output of the processor into an outboard mixer. Also, route the output of the other tracks to the mixer. Combine them through the mixer and send them to your speakers.
Also, what you can do is after you have monitored everything and found the right processor settings, record the processed signal on to Sonar so that you can free up that output if necessary.