Everything you've said basically matters. I think 320mp3s are fine personally for home listening and off an iPod etc. You won't be able to tell the difference generally between wav and a 320, the first thing you should notice drop off compressed music is the subbass.
Anything lower than 320 and I can tell the difference with my earbuds immediately.
If you compare a 15 quid set of earbuds with an 80 quid pair you will to start to notice sounds and differences that you didn't know were there before its almost like listening to a track for the first time again.
The first thing you'll notice when you listen to a compressed music file affects the opposite end of the frequency spectrum. It's the high end of the frequency spectrum not the low end. A 320kbps MP3 will have the frequencies above 18Khz rolled off. The high frequencies are compressed to make the 320 mp3 file a quarter of the size of an uncompressed WAV.
Personally I can't tell the difference between a 320 mp3 and WAV. I can't tell the difference between a 256 mp3 and a WAV. I can only start telling the difference when the mp3 is at 192kbps. The brightness of the hi hats starts to deteriorate.