You are correct. We started out with the mic (sm57)but couldn't make the sound very good so we plugged straight in to the Tascam us1800 with minimal effects.
Well it's not the mic, unless it's broken. The SM57 is , for better or worse, an "industry standard" for recording electric guitar, right up to the top flight studios. There are some basics to address.
1. Tune and intonate your guitars. This is easier if you have new, properly fitted and stretched strings.
2. Plug the guitar into the amp and get a good sound from it, one that you want to capture on your recording.
3. Put your mic in front of the amp, up close. You should experiment with mic placement, but as a general guide, your mic should point at a spot halfway between the dust cap and the edge of the cone.
4. Play the guitar and set up the record level so that it's somewhere around -12 dB.
If you follow these steps and it still sounds shite, then you have one of the steps wrong. It will probably be step #2.
Then, when you have a decent guitar sound, use your ears to balance it with the other instruments. Listen carefully and decide whether you can hear all the parts in the balance that you want.
This is just the basics.
One point I ought to make is that your ears really don't like to hear a guitar without some kind of room sound blended in with it. This can be a digital reverb or a real room. I would experiment with your digital reverb, it's easier. Try a small-to medium sized Hall reverb blended in with the sound, not too loud - your ears will thank you and the guitar will sound 100% more "real".