You are definitely on the right track with a pc - again, my opinion.
Pcs are great as long as you don't try to use it for more than one purpose. So if you use a pc for music, use it for music ONLY. No games, no internet, no graphics (like Photoshop) and for heaven's sake no microsoft office.
The pc should have nothing else on it than your OS (I prefer Win 98se over ME yet, but win2000 is very reliable, too. With Win 2000 you will [at this time] run into problems getting drivers for certain soundcards and some plugins don't run under 2000)
So, again:
OS:
Win 98se
Software:
Sequencer Program (Cubase or Logic Audio or Cakewalk or...)
If you need, a Wave Editor (WaveLab, CoolEdit...)
PlugIns (VST and/or DirectX)
Keep in mind that if you decide to go with Cakewalk, you will not be able to use VST-plugins, only DirectX - and there are tons of free VST-plugs out there!)
That is it.
Oh, Microsoft DirectX7a or 7.1 - not DX8!
Hardware:
Athlon or PIII (PIV still a bit buggy)
Graphics: Matrox
G400 / 450 (dual head) or, for low budget, ATI Xpert 2000
Audio (NOW here we are!!!)
Look at these:
http://www.rme-audio.com
my favorites!!! All time. Reliable, professional, expandable.
Or
http://www.midiman.com
has nice solutions as well. Cheaper.
There are tons of other cards/systems of course, but i like these and they have proven to run smoothly with a lot of systems.
I'm running my pc for about one year (every day use, music only) and it never crashed on me! (Well, except you put in too many FX and thus overrun the CPU, but that is human failure.
btw, RME have good tips on their site how to optimise your machine for audio-use!