He has the right to say that. You are not racist for agreeing with his right to say it.
He DOES NOT have the right to actually deny someone based on race. Nor do I agree with that.
That
Is
Racist
One would be racist if they agreed with the statement and/or the act of denial based on race.