(This reply cut + pasted from the 'Roland and Boss...' forum)
First of all, your custom rhythm track in the BR-864 can only be transmitted to a computer in the form of midi notes, and then only over the midi out. The USB interface is reserved for audio transfer only.
You need to have a drum sampler --- meaning one note for the kick, one note for the snare and so on for each drum you want to hear --- to interpret the midi data in your software of choice. The midi notes will be transmitted in real time as you play the song on the BR-864. Plug the midi out of the BR-864 to the midi in on your computer interface, set up a midi track, set the software to sync to midi clock or midi time code, push record on the software and start playback on the BR-864.
.