“You’re talking about probably a million and a half people, and we just clean out that whole thing and say, ‘You know, it’s over,’” he says