Hi Martijn,
I think I misunderstood your question. You are trying to get rasterio and geopandas working, is that correct?
If so, then the solution would be quite easy. Can you update the version of rasterio to the latest version 1.3.6?
I believe you can now remove the wheels completely and everything should work out of the box.
viktor==13.8.0
rasterio==1.3.6
geopandas==0.12.2