It depends on what your purpose is for making MP3s. The difference in quality between the original source WAV file and a 320 kbps file will be pretty negligible, but the quality comes at the cost of file size. If you're putting them on a website, you really want to put them in as compact a size as possible without impacting the audio quality too much. Most people seem to find 96-128 kbps to be pretty reasonable quality and yet small enough for someone with a 28 kBaud dial-up connection to be able to download in a reasonable amoung of time.
Yes, there is a pretty dramatic difference between 64 and 320 kbps. I find 64 to be too noticeably lo-fi. But I find 96 or 128 to be nearly indistinguishable from the original, at least with all my noisy lo-fi crap...