Is mastering truly necessary these days?

I think if you are going to do vinyl, one has to master.

I am going to try a concept of releasing an "album" as one file, rather than cut it up as individual. In this case, I will "master", but not like old school real mastering, where they have to rework the lower end (I understand some bottom is removed and the phone cartridge make it up), make sure the songs will fit on one side, true mastering for a vinyl release. From what little I know was very much a technical expertise.

In other words, except for loudness and sound consistency, not sure it really exist in the digital world, or it is redefined.
"Mastering" just means preparing the mixed recording for final release in whatever context.

Sure, if you aren't going to release the album as vinyl, mastering for vinyl is a pointless waste of time and money.

But if you're going to release a CD, matching the volumes of the various tracks and configuring the timing between them is still mastering.
If you're only releasing singles directly to the streaming platforms, EQing and compressing the track for best sound in that context is still mastering.
Is mastering an arcane/obsolete process for vinyl now used as a grift?
As compared to when? I ask because there are many more tools available to home recordists now compared to the time prior to the arrival of the DAW. When I was recording with tape I couldn't afford all that I would require to master my songs.
I'm half way through a course on mastering and have realized a few things:

1. If the mix is balanced amazingly, the stereo image is good, no phase issues, and everything is where it should be, then mastering is the stamp of approval and loudness management based on Ulta linear monitoring and the ears of the ME who has developed the skill of being able to perceive small changes in pitch, harmonic distortion, and tonal balance.

2. For every other scenario, mastering is absolutely essential as a process of adjusting tonal balance, dynamics and loudness.

3. Ai mastering isn't mastering unless it's done by a mastering engineer who can hear exactly what the song needs so that it can translate as exceptionally well as the original mix will allow to all playback systems. 9/10 times a decent M.E will be able to blow even current AI out of the water.

4. Mastering on the same playback system you mix on will keep you blind to problems in the mix. A second set of skilled ears whose playback system is tuned for mastering is essential for competitive sounding releases.

5. Get things right at the source and everything will sound better throughout the entire creation process, including in mastering. A good me can bring out the qualities needed.
And more, if mastering for vinyl, there are additional considerations. Too much bass or high frequency content will mess up the lathe and you won't get a clean pressing. So proper mastering is essential for pressing to vinyl.
The one problem with "mastering" is that it's not a cut and dried affair.

A fellow recently posted a track that he sent to 5 or 6 different mastering places. They ranged from some fellow on Fiver to Abbey Road Studios. Each was given the same track and sent back a mastered track. The various masters were grossly different. I thought that two of them actually sounded worse than the unmastered original track. So now, you're put in a position of finding an ME that will improve your track, not degrade it. Unless you have someone that you have worked with previously, or have heard and like their work, it can be a bit of a crap shoot.

Make sure the ME you choose will be willing to go back and do things until you get something that satisfies you.
For most of us, adding an extra person is something we don't want to do, or cannot afford to do - so we don't. The more mixes you do and the better your ears get means that if you did want to use one, you have to find one better than you, who you trust. I figure that is why most of us just don't bother. I've found a few on youtube who seem to think that because they have X or Y plug ins, they are a mastering engineer. They're not. It's a bit like the folk who now re-write the code for the engine management systems in cars. Some don't appear to have ever owned a car?
Isn't "overall sound" a very subjective thing?
I AM one of those who listens to EVERYTHING on the same system adjusted the way I LIKE to hear it. Everything from Tubi and broadcast news, to talk radio and my favorite old albums. It's like a "built in" reference that my ears are trained to, but it's still MY preference.
My point is, my amp/receiver and speakers are set where I like and I never touch them. However, all I have to do is change channels or switch music CDs to notice subtle differences in tone/eq. There's obviously no "industry standard" with regard to mastering...
I went to a music biz meeting last night aimed at up and coming artists and was very interested to hear Karen Kemp from BBC Introducing say that, no matter how much else you DIY, you should get your music mastered properly. I guess she gets to hear all kinds of material from up and coming artists who are trying to get their first radio play so it seems that properly mastered material has an advantage.
Of course, it could simply be that they get an awful lot of poorly mixed and non-standard media sent in - when I was examining, the huge quantity of material that arrived with levels all over the place was extreme! Some had pushed the edge so far that it was micro dBs from digital zero, while other had you scrabbling for gain, and you had a noisy mess. Beatles style 60's -panning and all kinds of crazy 'mistakes' often detailed in the notes as deliberate stylistic choices when they were relaly bad mistakes.
It's nice to get a final set of ears on a project. A good mastering engineer may do nothing or very little if you do a great job on your mix. I took a mastering class in college (almost thirty years ago!), and Bob Ludwig came in a couple of times and said as much.

The big problem today is that many people need to understand how to record music or care enough to do things correctly, so the mixing engineer or mastering engineer has a lot of problems to solve.

Maybe it's because I came from tape, and when I was learning, having moved on from a Portastudio, it was mainly tape. SDII was getting popular, which was amazing. I still record the same way - commit and build in 85-90% of what I want. My goal is always to build things in and have sessions where if everything is at 0, it sounds good.

I understand people have different ways of working, but I get sessions that take two or three hours to get them ready to be worked on more than I would like.

A lot of the voodoo around working with vinyl is overstated, and if you take care in recording, you won't have any issues. The first music I ever released commercially was a vinyl single that I recorded on 8-track and there was no ME involved. I probably did about eight records on vinyl before I did anything on CD.
Mastering. If you know, you know.
If you don't, you'll hopefully learn through trial, error, and experience.

The right ME can make a good mix sound great, and your collection of material sound cohesive. The wrong ME can make a good mix sound like hot trash. Finding one you can work with is worth the cost, in my opinion. Vinyl, download, streaming, or CD. It's all relevant. And a greater understanding/standard on how to approach the different platforms now.

20 years ago, I sent an album to an ME. I had digital overs on a few tracks and had not caught them before sending my files. It was a 3-week write, track, and mix marathon. My ears were toast. When I got the master back, the digital overs were very noticeable on those tracks. As expected, they were nasty and harsh, and we could not listen to those mastered tracks. I asked the ME about it; he said, "those digital artifacts are in your mix," - But he chose to master the tracks anyway. We had to correct the tracks and pseudo-master them ourselves. It still shows.

Doesn't matter that the album wasn't great or that I had too hot of tracks to start with; what mattered to us was the attitude and lack of willingness of the ME. He wasn't willing to be our safeguard and last set of good ears. I never used him again, but I have also never sent files with overs again. 20 years ago, digital wasn't as well defined, loudness wars were in full swing, and we were all still working to balance the sterility.
I'm a chump. A wannabe. A hobbyist. A virtual know-nothing. Who likes to fiddle with mixing and then using the "mastering" software and hardware that I own to enhance my home recordings. I'm quite content to record a track, mix it roughly to my taste, then running it through the "mastering" equipment that I own to enhance it for cohesion... a final polish. It's almost always is an improvement.

If I were to miraculously record something that I deemed really worthy - I would probably consider paying a pro to help polish the final product. But I don't see that happening.
Yes, but only by an experienced and qualified mastering engineer who fully understands the process and has the professional gear to get the job done properly. There is no substitute as I look at it.
Now we don’t have absolute requirements for the medium, for example, vinyl pressings, what exactly would a proper, professional mastering engineer do? Change what you have done? They alter your dynamics, they alter the spectral content and they change the level. If your recording is supposed to be say EDM, or metal, or some other very defined ‘club’ your music fits, membership wise, then they can fix faults. If there are no faults, they’re changing your mix, to their preference. I can understand that some music is poorly put together and needs an expert to repair. Some people produce music and put it on here for us to listen to, and their sound is consistant, and nice. They’re happy with it. They made good choices. Sometimes, not choices I would have made, but it is their choices that make it ‘them’. What benefit would paying a 3rd party bring? Something different? They might hate it.

For most people who now self release, would a mastered product result in more streams or downloads? If yours was sh*t originally, yes, for certain. For good stuff, I suspect not at all.