Depending on budget and need, you'll start at about $500 without monitors, to tens of thousands if you so choose. For a hobbyist, it's best to start at the lower end, but if you are serious and want to get going, set a decent budget and ask questions about how to apply it.
Okay, first a quiet, controlled room. If you clap and hear high frequencies after, that's called flutter echo. If you clap and hear a clap after, that's regular echo. If you clap and you get a pleasant tonal ambiance back, that's called reverb. You definitely want to control the first two (room treatment is in order). The last is passable if you don't mind that being the only reverb you'll get. Corner trapping is fairly imperative. Bass collects in corners and needs to be tamed. Room treatment is the most overlooked aspect of recording/mixing. Most bad mixes are due to poorly recorded tracks during the tracking process and your room lying to you when you're mixing.
Second. Mike, interface, headphones and DAW. LOTS to choose from, but start somewhere basic. Grab a package deal like the PreSonus or Focusrite studio bundle. Make sure you get a pop filter with the mike if it's condenser.
If you can afford, take a step up to the NI KA6. Excellent starting unit.
Get a decent mike (check the microphones equipment forum for lots of conversations).
Sennheiser 280s get a lot of nods in the $100 bracket as do Sony's 7506. Lots of others to consider as well.
Many people learn on Reaper as it's cheap ($60 for the full version) and VERY powerful/flexible. Many others use the bundled software that comes with their package and eventually spend $100s to get the full version. Then more quite often for upgrades... Others go against the grain
and get DAWs that aren't included in bundles and aren't inexpensive (Acid/Samplitube/Reason/etc.)
Third. Ear training. Find a system out there (they're all over youtube) that gets you frequency training. Get your ears in tune with sound. A good set of ears is a tech's best tool! If you're the kind of person that can sit in a crowded room and make out 4 or 5 conversations at the same time, that's a great start. It's kind of what you have to do.
Lastly, studio monitors. You really don't need these to record. But when it comes to mixing, they are inevitable. Again lots to choose from and your need will be based on the size of room you mix in and other considerations.
Hope this helps.