Auto-publish to PyPi using Gitlab CI/CD
Intro
Have you ever found it tedious to manually build the distribution packages and using twine to publish them to PyPi every time you want to make a new release? We could make use of GitLab's CI/CD to do that for us. Let's see how.
Gitlab CI/CD
If you already have set up a .gitlab-ci.yml for your project, you may skip this part.
Gitlab CI/CD is a tool from Gitlab for what is known as Continuous Integration/Delivery/Deployment. Practicing CI/CD can help you find bugs quickly before they get into production. You can set up scripts to build, test or deploy your software and integrate it to Gitlab's CI/CD tool to trigger those scripts whenever changes are made to your codebase.
You can get started by simply including a .gitlab-ci.yml file in your project. (assuming you have a personal or shared runner installed for your project, go to Settings > CI/CD and expand Runners)
Every .gitlab-ci.yml file must have jobs and stages. Stages and jobs, by default, happen one after the other. Let us look at an example .gitlab-ci.yml file.
stages:
- build
- test
build-job:
stage: build
script:
- echo "Build script running.."
- echo "More build scripts..."
unit-tests:
stage: test
script:
- echo "Run test1"
- echo "Run test2"
- echo "Run test3"
integration-tests:
stage: test
script:
- echo "Run test1"
- echo "Run test2"
- echo "Run test3"
First, we define the stages. Here we have two stages - build and test, which we want to run in that order. Then we define the jobs. First, we have build-job which runs under the build stage.
After the build job finishes successfully (note that if any job fails at one point, the rest of the jobs won't be executed), we move on to the test stage where we have two jobs unit-tests and integration-tests.
Note that you have a variety of options to customize when and how each job should run. You could set constraints for jobs such that it runs on certain branches, runs if there is a specific keyword in the commit message, runs if commits are from specific people etc. You can also specify docker images for jobs to run. For a complete list of stuff, you can refer to gitlab's docs.
Auto publish to PyPi
First, you need to add your PyPi username and password as variables under your project's CI/CD settings. You need to set the key name for the username as TWINE_USERNAME and the password as TWINE_PASSWORD.
Then you need to add a job in the .gitlab-ci.yml file to do the publishing for you.
Here is an example.
stages:
- test
- publish
test:
stage: test
image: python:3
script:
- pip install . && cd tests/ && python tests.py
publish:
stage: publish
image: python:3
script:
- pip install twine
- python setup.py sdist bdist_wheel
- twine check dist/*
- twine upload dist/*
only:
- main
Note the only keyword set in the publish job. This job only starts when something is committed to the main branch. (If you only need releases from the main branch)
That's it, now you can focus on developing in a separate branch, and whenever your software is ready for a release, just merge them to the main branch to trigger the publish job. (Make sure you bump up those version numbers :P).