Workmen ‘spotted at factory site’ as council ‘unaware of development’