Colbert refused to stay quiet