I should consider myself lucky.
I have no downstairs or next door neighbors right now. And the guy who lives above me...his only annoyance is that he's active in the middle of the night. So if I happen to wake up at 3AM, I'm going to hear his footsteps. But even that's not a big deal. I'll just watch some tv then go back to bed.