GDAL error when installing Geopandas/Fiona in venv isolation mode

martijnkriebel · 3 August 2022 11:49

I want to switch from using the isolation mode from Docker to venv, so I used the viktor-cli configure command and provided the path to my Python installation. This env contains several packages including Geopandas, which has Fiona as a dependency.

While I encountered no problems using Docker, I get the error below regarding Fiona and GDAL when using venv. The same error pops up when simply installing Geopandas in a regular venv, so it is not necessarily a problem related to VIKTOR, but it does prevent me from the venv isolation mode. Does anyone know a solution for this?

WARNING: you are developing with Python 3.8, while the published app runs on Python 3.7. The Python version of the published app can be specified by setting 'python_version' in the viktor.config.toml file

SDK version: 13.2.1 Connector version: 5.16.0
Creating app specific install dir signals
Creating virtual environment
Looking in indexes: https://martijn.kriebel%40wsp.com:****@sys.viktor.ai/api/v1/packages/
Collecting viktor==13.2.1
  Downloading https://viktor-system.s3.amazonaws.com/packages/viktor/viktor-13.2.1-cp38-cp38-win_amd64.whl?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA53GKZRCUAHO7GC7K%2F20220803%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Date=20220803T095518Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&X-Amz-Signature=21b0a5cabe569f9d52f682dd8b73fca18d70173460441e99a0fda15410f48325 (7.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.7/7.7 MB 9.9 MB/s eta 0:00:00
Collecting viktor-connector==5.16.0
  Downloading https://viktor-system.s3.amazonaws.com/packages/viktor-connector/viktor_connector-5.16.0-cp38-cp38-win_amd64.whl?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA53GKZRCUAHO7GC7K%2F20220803%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Date=20220803T095520Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&X-Amz-Signature=f711a27a95c3853e0fc65de315d27c8d9a6348b642ba2ebc77e9700e5a5e5125 (674 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 674.4/674.4 KB 10.7 MB/s eta 0:00:00

Collecting geopandas==0.9.0
  Using cached geopandas-0.9.0-py2.py3-none-any.whl (994 kB)
Collecting pandas<2.0.0,>=0.23.4
  Using cached pandas-1.4.3-cp38-cp38-win_amd64.whl (10.6 MB)
Collecting fiona>=1.8
  Using cached Fiona-1.8.21.tar.gz (1.0 MB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [1 lines of output]
      A GDAL API version must be specified. Provide a path to gdal-config using a GDAL_CONFIG environment variable or use a GDAL_VERSION environment variable.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Exiting because of an error: installation of the app failed

rdejonge · 3 August 2022 11:56

Hi Martijn,

On Windows GDAL needs to be built from source. The error is caused by the fact that not all build dependencies required for this process are correctly set up. You can either try to get everything sorted in order to built the Python package from source or you can use an unofficial distribution as explained in the second bullet of this link: Getting Started - Troubleshooting | VIKTOR Documentation. Please not that you will need to account for the fact that published apps will run on Linux.

Regards,

Raoul

martijnkriebel · 4 October 2022 12:47

Hi Raoul,
I finally found some time to try this out. Unfortunately, I’m getting this error upon installation using the venv isolation mode:

SDK version: 13.2.1 Connector version: 5.17.1
Creating app specific install dir signals
Creating virtual environment
Looking in indexes: https://sys.viktor.ai/api/v1/packages/
ERROR: GDAL-3.4.3-pp38-pypy38_pp73-win_amd64.whl is not a supported wheel on this platform.
Exiting because of an error: installation of the app failed

My requirements.txt looks like this:

# viktor==13.2.1
d-geolib-plus==0.1.2
openpyxl==3.0.10
https://download.lfd.uci.edu/pythonlibs/archived/GDAL-3.4.3-pp38-pypy38_pp73-win_amd64.whl; platform_system == "Windows"
https://download.lfd.uci.edu/pythonlibs/archived/Fiona-1.8.21-pp38-pypy38_pp73-win_amd64.whl; platform_system == "Windows"
geopandas==0.9.0

I’ve tried to change platform_system to "Linux" hoping this might solve the problem (as the docs say something about accounting for the fact that the apps run on Linux) but this simply makes the installer ignore the wheels.

Do you have any idea what’s wrong?

sonomirco · 5 October 2022 04:11

Have a look to this thread.

martijnkriebel · 5 October 2022 10:24

Thanks Mirco, the thread clarified a lot!

Three things were going wrong:

I only defined the installations for Windows, not for Linux.
I tried to directly download the wheels from Gohlke’s website in the requirements.txt, which didn’t work. But downloading them manually to a local subfolder fixed this.
I mistakenly used a PyPy-wheel instead of CPython.

For future reference: my requirements.txt now looks like this. Haven’t tried to publish it yet, but local installation works like a charm.

# viktor==13.2.1
d-geolib-plus==0.1.2
openpyxl==3.0.10
./local-whls/GDAL-3.4.3-cp38-cp38-win_amd64.whl; platform_system == "Windows"
./local-whls/Fiona-1.8.21-cp38-cp38-win_amd64.whl; platform_system == "Windows"
fiona==1.8.21; platform_system != "Windows"
geopandas==0.9.0

sonomirco · 5 October 2022 10:37

Nice!!! I am glad to hear was useful to solve your issues.
For sure the documentation needs to be updated and improved on this topic.

martijnkriebel · 28 March 2023 09:13

Hi all,

I’m currently experiencing a similar GDAL-error for the Rasterio package, which apparently cannot be solved in the same way as for Geopandas/Fiona described above / in the docs.

Publishing my app using the viktor-cli results in an error. I also get an error in my CI-workflow on Github Actions. When using the app locally, everything works fine with the local wheels, so the error only occurs on Linux systems.

My requirements.txt looks like this (only relevant packages):

viktor==13.8.0
./local-whls/GDAL-3.4.3-cp310-cp310-win_amd64.whl; platform_system == "Windows"
./local-whls/rasterio-1.2.10-cp310-cp310-win_amd64.whl; platform_system == "Windows"
rasterio==1.2.10; platform_system != "Windows"
geopandas==0.12.2

This results in the following error when installing Rasterio:

Collecting rasterio==1.2.10
  Downloading rasterio-1.2.10.tar.gz (2.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 251.5 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error'
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [2 lines of output]
      WARNING:root:Failed to get options via gdal-config: [Errno 2] No such file or directory: 'gdal-config'
      ERROR: A GDAL API version must be specified. Provide a path to gdal-config using a GDAL_CONFIG environment variable or use a GDAL_VERSION environment variable.
      [end of output]

I noticed I didn’t include GDAL in the installation for non-Windows platforms, so I changed the requirements to:

viktor==13.8.0
./local-whls/GDAL-3.4.3-cp310-cp310-win_amd64.whl; platform_system == "Windows"
./local-whls/rasterio-1.2.10-cp310-cp310-win_amd64.whl; platform_system == "Windows"
GDAL==3.4.3; platform_system != "Windows"
rasterio==1.2.10; platform_system != "Windows"
geopandas==0.12.2

This causes a similar error, but is raised when installing GDAL (instead of Rasterio like in the previous error):

gdal_config_error: [Errno 2] No such file or directory: 'gdal-config'
      
Could not find gdal-config. Make sure you have installed the GDAL native library and development headers.

Jelle · 28 March 2023 14:16

Hi Martijn,

Looking on the PyPi page of Rasterio the following is stated:
“The rasterio wheels on PyPI include GDAL and its own dependencies.”

This means that if you include rasterio like this in the requirements, it will try to install another version of gdal. Can you try to remove “rasterio==1.2.10” from the requirements?

martijnkriebel · 29 March 2023 09:07

Hi Jelle,

Removing Rasterio indeed solves the problem. However, I still need to use it in my app. Is there any way to solve this?

Jelle · 29 March 2023 11:27

Did you still include the rasterio-wheel in the requirements file?
Essentially, the wheel replaces the statement “rasterio==1.2.10”.
If this doesn’t work, can you try to add “rasterio==1.2.10 --no-binary” ?

Please note that this is different than the installation of GDAL, for GDAL both the wheel and “GDAL==3.4.3” are required.

martijnkriebel · 30 March 2023 10:35

My requirements look like this, so the wheel is used when on Windows (local machine) but ignored when on Linux (Github/Viktor):

viktor==13.8.0
./local-whls/GDAL-3.4.3-cp310-cp310-win_amd64.whl; platform_system == "Windows"
./local-whls/rasterio-1.2.10-cp310-cp310-win_amd64.whl; platform_system == "Windows"
gdal==3.4.3; platform_system != "Windows"
rasterio==1.2.10; platform_system != "Windows"
geopandas==0.12.2

This results in this GDAL error:

Collecting viktor==13.8.0
  Downloading viktor-13.8.0-cp310-cp310-manylinux2014_x86_64.whl (30.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 30.8/30.8 MB 234.5 MB/s eta 0:00:00
Collecting gdal==3.4.3
  Downloading GDAL-3.4.3.tar.gz (757 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 757.6/757.6 kB 355.1 MB/s eta 0:00:00
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'error'
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [126 lines of output]
      WARNING: numpy not available!  Array support will not be enabled
      running egg_info
      creating /tmp/pip-pip-egg-info-xizxg1wx/GDAL.egg-info
      writing /tmp/pip-pip-egg-info-xizxg1wx/GDAL.egg-info/PKG-INFO
      writing dependency_links to /tmp/pip-pip-egg-info-xizxg1wx/GDAL.egg-info/dependency_links.txt
      writing requirements to /tmp/pip-pip-egg-info-xizxg1wx/GDAL.egg-info/requires.txt
      writing top-level names to /tmp/pip-pip-egg-info-xizxg1wx/GDAL.egg-info/top_level.txt
      writing manifest file '/tmp/pip-pip-egg-info-xizxg1wx/GDAL.egg-info/SOURCES.txt'
      Traceback (most recent call last):
        File "/tmp/pip-install-_6p2hmob/gdal_b087abaacd4f42e19bea18d75244a645/setup.py", line 105, in fetch_config
          p = subprocess.Popen([command, args], stdout=subprocess.PIPE)
        File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/subprocess.py", line 971, in __init__
          self._execute_child(args, executable, preexec_fn, close_fds,
        File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/subprocess.py", line 1847, in _execute_child
          raise child_exception_type(errno_num, err_msg, err_filename)
      FileNotFoundError: [Errno 2] No such file or directory: '../../apps/gdal-config'
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "/tmp/pip-install-_6p2hmob/gdal_b087abaacd4f42e19bea18d75244a645/setup.py", line 188, in get_gdal_config
          return fetch_config(option, gdal_config=self.gdal_config)
        File "/tmp/pip-install-_6p2hmob/gdal_b087abaacd4f42e19bea18d75244a645/setup.py", line 108, in fetch_config
          raise gdal_config_error(e)
      __main__.gdal_config_error: [Errno 2] No such file or directory: '../../apps/gdal-config'
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "/tmp/pip-install-_6p2hmob/gdal_b087abaacd4f42e19bea18d75244a645/setup.py", line 105, in fetch_config
          p = subprocess.Popen([command, args], stdout=subprocess.PIPE)
        File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/subprocess.py", line 971, in __init__
          self._execute_child(args, executable, preexec_fn, close_fds,
        File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/subprocess.py", line 1847, in _execute_child
          raise child_exception_type(errno_num, err_msg, err_filename)
      FileNotFoundError: [Errno 2] No such file or directory: 'gdal-config'
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "/tmp/pip-install-_6p2hmob/gdal_b087abaacd4f42e19bea18d75244a645/setup.py", line 195, in get_gdal_config
          return fetch_config(option)
        File "/tmp/pip-install-_6p2hmob/gdal_b087abaacd4f42e19bea18d75244a645/setup.py", line 108, in fetch_config
          raise gdal_config_error(e)
      __main__.gdal_config_error: [Errno 2] No such file or directory: 'gdal-config'
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-_6p2hmob/gdal_b087abaacd4f42e19bea18d75244a645/setup.py", line 389, in <module>
          setup(**setup_kwargs)
        File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
          return distutils.core.setup(**attrs)
        File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 968, in run_commands
          self.run_command(cmd)
        File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/setuptools/dist.py", line 1217, in run_command
          super().run_command(command)
        File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
          cmd_obj.run()
        File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 308, in run
          self.find_sources()
        File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 316, in find_sources
          mm.run()
        File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 560, in run
          self.add_defaults()
        File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 597, in add_defaults
          sdist.add_defaults(self)
        File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/setuptools/command/sdist.py", line 106, in add_defaults
          super().add_defaults()
        File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 252, in add_defaults
          self._add_defaults_ext()
        File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
          build_ext = self.get_finalized_command('build_ext')
        File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 306, in get_finalized_command
          cmd_obj.ensure_finalized()
        File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 109, in ensure_finalized
          self.finalize_options()
        File "/tmp/pip-install-_6p2hmob/gdal_b087abaacd4f42e19bea18d75244a645/setup.py", line 258, in finalize_options
          self.gdaldir = self.get_gdal_config('prefix')
        File "/tmp/pip-install-_6p2hmob/gdal_b087abaacd4f42e19bea18d75244a645/setup.py", line 201, in get_gdal_config
          raise gdal_config_error(traceback_string + '\n' + msg)
      __main__.gdal_config_error: Traceback (most recent call last):
        File "/tmp/pip-install-_6p2hmob/gdal_b087abaacd4f42e19bea18d75244a645/setup.py", line 105, in fetch_config
          p = subprocess.Popen([command, args], stdout=subprocess.PIPE)
        File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/subprocess.py", line 971, in __init__
          self._execute_child(args, executable, preexec_fn, close_fds,
        File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/subprocess.py", line 1847, in _execute_child
          raise child_exception_type(errno_num, err_msg, err_filename)
      FileNotFoundError: [Errno 2] No such file or directory: '../../apps/gdal-config'
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "/tmp/pip-install-_6p2hmob/gdal_b087abaacd4f42e19bea18d75244a645/setup.py", line 188, in get_gdal_config
          return fetch_config(option, gdal_config=self.gdal_config)
        File "/tmp/pip-install-_6p2hmob/gdal_b087abaacd4f42e19bea18d75244a645/setup.py", line 108, in fetch_config
          raise gdal_config_error(e)
      gdal_config_error: [Errno 2] No such file or directory: '../../apps/gdal-config'
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "/tmp/pip-install-_6p2hmob/gdal_b087abaacd4f42e19bea18d75244a645/setup.py", line 105, in fetch_config
          p = subprocess.Popen([command, args], stdout=subprocess.PIPE)
        File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/subprocess.py", line 971, in __init__
          self._execute_child(args, executable, preexec_fn, close_fds,
        File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/subprocess.py", line 1847, in _execute_child
          raise child_exception_type(errno_num, err_msg, err_filename)
      FileNotFoundError: [Errno 2] No such file or directory: 'gdal-config'
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "/tmp/pip-install-_6p2hmob/gdal_b087abaacd4f42e19bea18d75244a645/setup.py", line 195, in get_gdal_config
          return fetch_config(option)
        File "/tmp/pip-install-_6p2hmob/gdal_b087abaacd4f42e19bea18d75244a645/setup.py", line 108, in fetch_config
          raise gdal_config_error(e)
      gdal_config_error: [Errno 2] No such file or directory: 'gdal-config'
      
      Could not find gdal-config. Make sure you have installed the GDAL native library and development headers.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Error: Process completed with exit code 1.

For testing the --no-binary with Rasterio flag I removed the GDAL-requirement (since otherwise an error is already raised at GDAL installation, before Rasterio installation is executed), results in this error:

Collecting rasterio==1.2.10
  Downloading rasterio-1.2.10.tar.gz (2.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 204.6 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error'
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [2 lines of output]
      WARNING:root:Failed to get options via gdal-config: [Errno 2] No such file or directory: 'gdal-config'
      ERROR: A GDAL API version must be specified. Provide a path to gdal-config using a GDAL_CONFIG environment variable or use a GDAL_VERSION environment variable.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
Error: Process completed with exit code 1.

Jelle · 30 March 2023 13:59

Hi Martijn,
I think I misunderstood your question. You are trying to get rasterio and geopandas working, is that correct?
If so, then the solution would be quite easy. Can you update the version of rasterio to the latest version 1.3.6?
I believe you can now remove the wheels completely and everything should work out of the box.

viktor==13.8.0
rasterio==1.3.6
geopandas==0.12.2

martijnkriebel · 30 March 2023 16:05

Thanks Jelle, that worked! I tried something similar a couple hours ago following instructions I found online, but apparently I installed the packages in the wrong order.

Out of curiosity, any idea why the new version works like a charm compared to the version I tried?

rdejonge · 30 March 2023 17:01

The reason for this is that the latest versions of rasterio and fiona (a dependency of geopandas) have precompiled wheels available for Windows, Linux and Mac on PyPI. The older versions only shipped the source for Windows. This meant that you also needed the build dependencies to properly install these packages.