Speaker refuses to step down