Us home-recording types that do what I call "pseudo-mastering" probably use software like Izotope's Ozone or IK Multimedia's T-RackS, to name a couple. I use Ozone (old version 5) and have found it's useful for what I do, and I've learned a lot by spending time understanding how the different pieces (multi-pressor, limiter, dither, exciter, et al) work, and creating my own presets for the stuff I do most often.
But, as others have said, mastering is something else, and for "polishing" up your home stuff, I honestly think you can probably get by with a few stock plugins on the master bus these days, to make things sound a bit more finished. (Success or failure depends on how well your room works, TBH.)
Doing a professional job on a bunch of disparate tracks, perhaps recorded with different equipment at different times and places, like what I've heard a friend do, can be a pretty massive (
) undertaking, and his "room" probably cost half what my house did, so I don't pretend what I do is anything like that.
I'll suggest spending time watching videos, maybe anything specific to "mastering in [insert your DAW]" and see if you see some common thread there before buying software. Good luck.