Brexit contributed to Port of Dover delays, Downing Street says