Fears that new Brighton Royal Mail site ‘will deteriorate quiet village life’