Jumping in a little late here, but I also recommend using a soft synth to start out with. There are a number of soft synths which are available as free VST plug-ins, such as the MiniMogue and the Arppe2600, making it very easy to start learning how to program and play a synth without having to actually buy one. Synths can be extremely expensive, especially a good one. I wouldn't recommend purchasing an actual physical synth until after you've thoroughly explored what can be done with a soft synth.
One thing about soft synths-- and virtual instruments in general-- is that you generally need to watch out for latency issues. If you use an ASIO driver for the audio output, it will help to greatly reduce or eliminate any latency. And having a fast PC doesn't hurt, either.
As far as a MIDI keyboard controller, I think you should carefully consider its features before you buy one. You can buy one for pretty cheap, but the cheaper ones tend to have very few controls, and/or they have keyboards with only 2, 3, or 4 octaves, sometimes with mini-keys or button-keys. If you can afford it, consider spending a bit more to get a controller that has more buttons, dials, and sliders, as they'll let you modify the soft synth's patch settings directly without having to do it on the computer screen. I was looking at buying a MIDI keyboard controller a couple of years ago, and at first I was tempted to buy a cheap one (under $100), but I eventually decided to get a more expensive one instead-- an M-Audio Axiom 61-II for $330 US-- because I wanted the extra controls and a 5-octave keyboard with full-size keys. I've seen some negative feedback from users of M-Audio's cheaper Keystation controllers, but I haven't had any problems with my Axiom.