I had a good 5 months off chess to think about this.
In my case if someone on a neighbouring board is playing an interesting game I'm likely losing concentration on my comparatively boring game.
That's one of a number of failings. But I'll let others list their reasons before listing more...