George Martin had a series on BBC TV a few years ago concerning this concept. He divided (in general order of importance) the foundation of music into 3 parts: Rhythm, Melody & Harmony. From what I remember, he said that the Rhythm is most important to start with, as it is something we have all been attracted to since birth (i.e. our mother’s heartbeat). Rhythm is also around us in nature at all times (i.e. change of seasons etc.) & everywhere we walk, we are walking in some sort of rhythm.
A lot of tribal music is exclusively based on rhythm, as it’s a celebration of nature, human nature, birth, death etc. I initially agreed with the Old McDonald theory, but now I think about it, the most attractive thing children find in this song must be the strong, forceful rhythm (esp. the “With a …” section).
Even the most distinctive classical pieces have a strong foundation of rhythm. Imagine Beethoven’s 5th with the same melody but different rhythm. It wouldn’t work nearly as well as a different melody with the same rhythm. I find the same thing with many of the great guitar riffs – the melody can be changed around a bit, but the rhythm is essential to keep the original feel (or intention) intact.
It’s the same with modal/scalar writing or improv – the notes alone (although a great tool) cannot duplicate the mood you’re intending to put across to the listener. A Major mode can be phrased in such a way as to sound very sad, as a harmonic minor scale can sound very happy if phrased in a certain manner. It’s all down to a strong foundation of rhythm.