It's the one thing about Reaper that I don't like. It's getting better at Midi, but program changes are poorly handled (in my view).
Here's one way:
Open up a midi item (e.g. double click on it). You will see the piano-scroll editor. At the bottom left (above the transport controls) is a drop down menu, which, by default, should show 'program'.
If not, click on the down arrow and select 'program'
If you now click somewhere in the area beneath the paino scroll and drag upwards, a bar appears, and its value appears in the bottom right corner of the screen, changing as you drag up or down. This is the program value, representing a midi instrument.
(This form of control is suitable for notionally continuous variable changes, such as volume, reverb or pan, but is not the best way of deleting with discrete, usually global changes, such as selecting an isntrument.)