Hi,
We want to push our Viktor app from a GitHub repository via GitHub action but are having a problem with how we should handle larger files. The Viktor app developed requires a large .shp file (<500MB) to function properly, and GitHub doesn’t allow the upload of files greater than 100MB.
We could use https://git-lfs.github.com/ to store the file, but we are unsure will be collected during the process of packaging the app, because git-lfs provide a pointer.
We could make the user upload the required files, but they will be the same files, this is why we want to store them. Are the files kept persistently once are uploaded?
Pushing the app via cmd command and for this solution forget the use of github action.
We aren’t sure the best practise for this situation and would like to hear your recommendation or show us an example of a similar problem.
Thanks,
Charlie
Hi Charlie,
Interesting question! I can’t really give you any info about Github, because I have no experience with it myself. But I will ask some colleagues, perhaps they can chip in. As to your other questions:
- You don’t necessarily have to get the user to upload the file. Another option could be to have a root-entity called something like “shape file library” that the other entities refer to through the API. Then you can upload the required shape file there after publishing, or after the file is updated. In between publishes and use the file would indeed be stored persistently.
- Examples of this that I have seen are also about shape files. One example that comes to mind is: calculations on dykes. On the top level entity (the Dyke) the user could, through a filefield or file-entity upload the file, and then sub-entities (dyke cross sections) would make use of that uploaded file through the api
Hope that helps!
I also have no experience with git-lfs, but this thread on Stackoverflow shows how you could fetch the file from git-lfs within Github actions. I assume that after fetching the file can be uploaded to the packer just like any other (although it may take long because of the size).
I agree with Roeland that making a top-level file entity (managed by some owner of the large file) might be the best alternative, if you cannot get git-lfs working.
Thanks both for the answer.
The point is that the shp file and the correlated files it uses they shouldn’t change so often to require be updated for every use of the app, but when we will publish a new version these files need to be packaged every time, and if they are big files seams a problem for a CI/CD integrated workflow.
We will try the solution on linked, but as you mention for a big files will take time… but is ok if it works 
Ok, finally we have a solution.
The solution was to add and commit the change without pushing them.
Thanks, @rdejonge for the help.
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v3
with:
python-version: '3.9'
- run: curl -Lo viktor-cli 'https://developers.viktor.ai/api/v1/get-cli/?platform=linux&format=binary' && chmod +x viktor-cli
- run: ./viktor-cli ci-install
- run: ./viktor-cli ci-test
publish:
runs-on: ubuntu-latest
needs: test # start when job 'test' is finished
if: ${{ success() && startsWith(github.ref, 'refs/tags/') || github.event.inputs.tags}} # only run for a tag and when previous job completed successfully
steps:
- uses: actions/checkout@v3
- name: 7Zip server
run: |
7z x ./borehole.7z -o${{ env.BH_PATH }}
for i in $(ls -d ${{ env.BH_PATH }}/*); do echo ${i%%/}; done
echo ${{ github.ref }}
git add .
git config --global user.email ${{ secrets.GIT_EMAIL }}
git config --global user.name 'Mirco Bianchini'
git commit -a -m 'commit files'
git status
- uses: actions/setup-python@v3
with:
python-version: '3.9'
- run: curl -Lo viktor-cli 'https://developers.viktor.ai/api/v1/get-cli/?platform=linux&format=binary' && chmod +x viktor-cli
- run: ./viktor-cli ci-publish ${{ env.APP_NAME }}