Is there a way to make environment light only visible for reflections but not refractions?

I am a bit confuse as the refraction mask in objects only uses geo as inputs not lights.