The Aged Care Services Minister has dismissed suggestions he should resign, despite criticism of the government's handling of COVID outbreaks and deaths in aged care homes.