Turbocharge your monorepo with Lerna and Github Actions
Problem
At Scalable Capital we use a Whitelabel platform which is a common place for reusable react components, configs, helpers, etc...
The platform is organised in a monorepo
to manage it's packages. We're using Lerna to handle the packages and their dependencies.
Now Lerna does not uses any cache system. Every time you build or run tests on root level, it will run everything for all packages even if you changed a file in one of the packages.
This is bad because building and testing takes time and can take up lot of development time.
What if somehow we build and run tests only for the package that is changed and not all packages?
What if we can cache the Github action workflows for the Pull requests subsequent commits for PR checks?
Enter Turborepo. ⚡️
Changes
Let's get right into changes.
Part 1 - Add Turborepo (assuming Lerna
is already setup)
To add turbo, run the following command in your terminal
yarn add turbo --dev -W
Create a turbo.json file in the root of your project.
{
"$schema": "https://turborepo.org/schema.json",
"pipeline": {
"build": {
"dependsOn": ["^build"],
"outputs": [".next/**"]
},
"test": {
"outputs": []
},
"test-coverage": {
"outputs": ["coverage/**"]
},
"start": {
"cache": false
},
"typecheck": {
"outputs": []
},
"prepublishOnly": {
"outputs": []
},
"@scacap/whitelabel-app-cockpit#prepublishOnly": {
"dependsOn": ["@scacap/whitelabel-components#prepublishOnly"]
},
"@scacap/whitelabel-app-onboarding#prepublishOnly": {
"dependsOn": ["@scacap/whitelabel-components#prepublishOnly"]
}
}
}
In the above turbo.json
file you need to define pipeline
.
So everything which you want to run in NPM scripts of root package.json
using turbo run <tests | build>
, same field should be present in pipeline
.
turbo
applies the pipeline task to the NPM scripts during execution.
Most important thing to note here is dependsOn
. In above config @scacap/whitelabel-app-cockpit
package prepublishOnly
NPM script will run only when @scacap/whitelabel-components
's prepublishOnly
script has been run.
More info on setting up pipeline is here
Finally changes to root package.json
Add packageManager
to package.json
if you are using Yarn Workspaces
with Lerna
"packageManager": "yarn@1.22.17",
Part 2 - Setting up Github actions with turbo cache
We use Github actions PR checks to maintain the quality of code.
The problem is every commit to a PR will trigger these checks (running tests, type checks etc..) and without some caching mechanism these tasks will run for all packages.
Let's add Turbo magic to Github actions to speed up the the action workflow.
Let's take example of running tests.
If we change some tests for a package, without Turbo Cache Lerna
will run tests for all packages. And this happens for every commit. 😱
Let's change that
tests:
name: Unit tests
runs-on: ubuntu-18.04
steps:
- name: Checkout repo
uses: actions/checkout@v2
with:
fetch-depth: 0
- name: Setup node
uses: actions/setup-node@v2.1.5
with:
node-version: 14
- name: Restore node_modules cache
uses: actions/cache@v2.1.5
with:
path: '**/node_modules'
key: ${{ runner.os }}-modules-${{ hashFiles('**/yarn.lock') }}
- name: Turbo Cache
id: turbo-cache
uses: actions/cache@v2
with:
path: .turbo
key: turbo-${{ github.job }}-${{ github.ref_name }}-${{ github.sha }}
restore-keys: |
turbo-${{ github.job }}-${{ github.ref_name }}-
- name: Install packages
run: yarn --frozen-lockfile
- name: All tests
run: yarn test-coverage --cache-dir=".turbo"
- name: Add coverage comment
continue-on-error: true
uses: eeshdarthvader/code-coverage-assistant@master
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
monorepo-base-path: './packages'
Here we are adding mem cache in Github action workflow and store our cache data in .turbo
which will be reused between commits in the same PR.
Results
Normal CI PR checks on each commit looks like this.
ℹ️ Note the time. This much time is taken for every tiny commit you push to PR.
CI runs with TurboRepo
ℹ️ The test run has decreased from
4 min
to1 min
and Typecheck reduced from3 min
to52 sec 😱
Let's see how it helps when you are developing locally.
After first run of tests using yarn test
, try running it again.
✅ ==> FULL TURBO ✅
The tests which usually takes
~20 sec
to run everytime, with Turborepo it ran for freaking0.7 sec
with its caching system. 🎉
That's all folks! Hope you all Turbocharge your monorepo's with Turborepo
More resources- Official guide Migrate from Lerna
- Pull request Github Action workflow mentioned in the Post
- Code coverage Github action built for Monorepo's