Absolutely safe?
Well, if you have removed your project files from the system and they are not accessible, it's possible you have WAV files lurking around that belong to something that you really might want to have. But of course the Audio Finder Tool cannot know this, so these will turn up in the list of orphaned files.
I suppose there are other scenarios under which you might defeat the assumptions built in to the tool.
So it's not absolutely safe, but it's safe as it can be within reason.