Conservative MP asks Brexit bill critics: 'Where were you?'