To further expand on the arrangement, but this comment is production oriented, a low-fi start, like 78 record low-fi on the vocals. That would compliment that distorted organ. IMO, the song never seems to find a groove. I liked the first part, but in my head, I was waiting for it to go to "its" place. I think starting the way you have it, then move into a more "fuller" rhythmic song. It just seemed to meander without form or purpose.
Just an opinion, even if you have another objective for the song, the purpose never becomes clear. Hope you find this constructive.