In a nutshell, the minimum you need is
a MIDI keyboard or some other means of playing notes so that they can be captured as MIDI data;
at least one MIDI sound module that can make sounds in response to MIDI messages;
a computer with a MIDI interface and sequencing software, or a stand-alone sequencer of some kind, that you can use to capture, save, and manipulate MIDI data;
a sound system -- powered speakers, an amp, a mixer if you have more than one sound source, maybe an effects box if you want to add some reverb and delay to the sounds, etc. -- so you and others can hear the end results of your captured and manipulated MIDI data. You also probably want to have a way to make a final end product that you can hand someone -- a cassette tape or a CD they can listen to in the car or at home, away from your setup.
It's conceivable to do everything in one device -- a nice workstation keyboard with a built in sequencer and on-board effects can be found in the $1000 area. (well, you still need an amp and speakers and a way to tape it or burn a CD.) Or it can be completely in the computer -- a powerful PC or Mac can run softsynths as well as sequencing software, and if you don't have a keyboard you can draw notes on a staff or step-enter them using the computer keyboard, depending on the software.
Most people find that a computer with some software and a MIDI interface and audio interface, at least one external synth or sampler or other sound source, a controller keyboard, and some way to hear it all, is the most flexible and comfortable way to work. A typical modern PC is powerful enough to record and play back upwards of a couple of dozen of independent audio tracks, and software plug-ins give you amazing possibilities with effetcs processing. Most computers these days include CD-RW drives so it's easy to make mixes entirely within the computer.
One more note, vicevursa recommends a whole bunch of software, but most of the ones he named are predominantly audio recording packages. If it's predominantly MIDI you want to work with, you want to have software where the MIDI capabilities are not just an afterthought, or not just for synching audio to external devices and applications. Your prime choices are probably Cakewalk/SONAR, Cubase, and Logic. Each of these tyoically have a range of versions at various price points, ranging from under $100 for the Lite versions to over $500 for the full-bore professional versions.
Good luck!
-AlChuck
