Sure, you can notice changes in the noise floor sometimes...I guess it depends on how the record was mixed...I tend to work on projects that are mostly live recordings, and have pretty much the same set up for each song, so in cases like that the noise floor is pretty consistent. If you're treating each track as a very distinct individual entity, then there's probably more of a chance that things will be different from track to track.
Personally, I like noise. I think it adds character, and can contribute to the overall aesthetic of a record.
I also tend to sequence albums so that everything flows real nicely - lots of cross fades and whatnot to make the whole record a cohesive unit. In these instances, I find myself leaving room tone or noise floor or whatever running right through the whole thing instead of doing fades at the end of songs.
But, I guess it really depends on the project you're working on.
I vote for noise!