What I did was I started out with a 2-track machine, and used it to record mixes. That gave me a basic familiarity with the machines. After that I acquired a broken Fostex A8 and shortly afterwards a TASCAM TSR-8 which I got for a song on ebay.
I think the first question is, do you already have a mixing desk you can use? If not, you might want to look at the TASCAM 388 which has a mixing desk built into it like a giant Portastudio. It also uses 7" 1/4" tapes, which are pretty cheap. The half-inch ones used by the TSR-8 and its siblings are rather more expensive.
This is assuming that you want to do the entire tracking process on tape, of course. What some people prefer to do is integrate it with a DAW or something, and that gets a bit more complicated. I'm not sure the 388 would be good for that approach. What some people do is record on the tape deck, dump the tracks into the DAW and proceed as before. With a 3-head machine you can actually record through the tape deck into the DAW, though you'll have a delay of about 1/15 second - this is not strictly what the machine was designed to do anyway. These approaches do mean that you get to recycle the tape since it's just being used as in intermediate step.
Anyway, at the risk of being over-simplistic, the traditional approach which I use is this:
You have an 8-track machine or similar. I should perhaps point out that this is 8 MONO channels - some people coming from a DAW expect it to be 8 stereo channels and unfortunately it's not. A stereo instrument will require two tracks, for left and right.
Anyway, you connect the outputs of the tape deck to the inputs of your mixing desk. Once the tracks are recorded, you play it back, and use the mixing desk to tweak the EQ and position each track in the stereo image (left, right, centre etc).
The mixing desk will have a stereo out and you can record this into a 2-track mastering deck, cassette, DAT, computer soundcard, whatever takes your fancy.
To record the tracks, you connect your instrument(s) up to the tape deck's inputs. There is one input for each channel. If it's a band or a collection of synthesizers or something, you can set it up to record all the tracks simultaneously, or if you're a solo artist you can record them one at a time. It's possible to connect most instruments to it directly (a mic or guitar will need a preamp, though), but to save wear on the sockets it's probably best to go through a patchbay.
The best solution, if you can manage it, is to have two mixers - one to mix down the song (as described above), and one connecting to the inputs. If the mixer supports four channel groups, you can use the mixing desk itself to direct each instrument to a track on the machine without plugging in or unplugging things. Having a mixer as the front end also means you can easily adjust the EQ and add effects to the instruments as they are recorded, which can be handy.
The setup I had in 2004 was this:
http://dougtheeagle.com/lab/lab2004.jpg
...the left mixer has all the synthesizers connected to it. I used that one for routing the instruments to the tape deck, adding effects and so forth.
The mixer on the right was used to mix the song down. It was connected to the deck in the middle, the 8-track TSR-8 machine. On the right was the 2-track mastering deck.
That was the first fully working analogue setup I had, and therefore the simplest. After that they got more complicated :3
Hope that helps...