It works the same as stereo really, except you have 5 channels instead of the two. All you really need to actually do the work is a sequencer that supports 5.1. As you can generally set up your own outputs, you won't have a problem there. You just need to create those outputs (ie front-centre, front-left, front-right, rear-left, rear-right), and make sure you assign the relevant outputs to the relevant track.
Of course, if you actually want to listen to what you've done in 5.1, you'll need a soundcard with 5 mono outputs to accomodate the 5 speakers. You'll not necessarily need a sub (ie the .1) for mixing. You could probably manage just as well without a sub and 5 decent speakers, especially for music as opposed to movies. But then there is a school of thought that may disagree I'm sure.
As far as filetype to mixdown to goes, I'm not sure. DVD-audio handles 5.1, but I'm not entirely sure what sequencers can mixdown to that. I never got as far as mixdown when I've experimented with surround mixing in the past.