First there are distros optimised for audio like Planet CCRMA and Agnula, find them here:
http://ccrma-www.stanford.edu/planetccrma/software/
http://www.agnula.org/
Those are the best known, but there are others (see last link). The very small distros are usable as audio platform, but you will have to add soo much libraries and applications that it is easier to use a "normal" distro.
As for drivers, all distros can use OSS and have it included. The newer drivers are alsa
www.alsa-project.org but if you use a 2.6.x series kernel, the alsa drivers are already included into the kernel. Check the site for supported soundcards (but the page is in rework).
For some apps you will need Jack, certainly for the newer and more advanced.
http://jackit.sourceforge.net/ Once you find your way in jack, you will never understand why there is nothing comparable for windows.
Efect plug-ins are provided by LADSPA:
www.ladspa.org
And a very extensive overview of possible applications:
http://sound.condorow.net/
If you want a reasonably lean distro to run audio, you can try
gentoo if you are desparate. That's what I run now with an RME Hammerfall under Alsa/Jack and a cheapo no brand running OSS emulation under Alsa. Other apps I use regulary:
Ardour, Rezound, Audacity, Sweep and Mhwaveedit as sound recorders editors depending on needs, loads of ladspa modules. Jackrack, Qjackctl and meterbridge as utilities with Jack.
Have fun and good luck with your search.