So a simple CI pipeline that uses Anaconda Action for Python package management might look like this: I’ve set up a very simple and straightforward Github repository that follows this approach so that we can easily visualize the different pipelines that we’re going to use. I use Conda environments for Python package management as it is a very powerful tool.
So these dependencies and others like Pytorch need to be installed in our environment before we can use them properly. Also, I am used to working with pytest because I consider it a very flexible and intuitive framework to run all my tests.
Github actions cache code#
The classic approachĪ common way to automate tasks such as Python package code formatting is to use pre-commit hooks. In this post I pretend to provide some intuition of how have I optimized some of these pipelines for building Pytorch and Cuda using a Conda environment in a pretty efficient way. Problems come when you deal with larger packages which is usually the case when working with Deep Learning. When these dependencies are small and they can be installed in a few seconds, you do not need to worry about them. The first one installs all dependencies that are going to be needed for running all jobs that compone the second block (test, lint, etc.). This is very annoying since you need to run all setup steps on every build even though your environment does not change.Ī classic CI workflow can be split into 2 two different blocks. In most of our projects we have to deal with large dependencies like Pytorch or CUDA, which are needed to run our test suite and some others like pre-commit that help us to lint our code.
I’ve been bumping my head around Github Actions recently, as most of our Continuos Integration (CI) builds time was spent installing third party libraries.
0 kommentar(er)
