How to deploy NPM modules in an NX monorepo and github actions?

How do you maintain and deploy multiple NPM modules? How do you make sure versions do not mismatch or that nothing breaks while upgrading dependencies? And how do you deploy multiple packages at the same time in one CLI command?

Table of Contents

The problem of maintaining multiple modules and services

There are two main ways to manage multiple NPM modules.

The first and the “classic” is to maintain multiple repositories. Some say that developing this way, every module is independent from the others. You just develop the code in the module’s repository and publish a new version whenever you want. Is it true?

What if module A has a dependency (in its package.json) with module B? Now, we develop module B, which has a critical bug, and publish a new version. At first, it seems like they are independent. The problem is, module B changed, and we wouldn’t know if the changes had any implications on module A until we upgrade the dependency in A.

So the first thing we see here is that they are dependent on each other, even if they a 1000 repositories away. The second thing is that it is hard to catch integration regressions even though we are the maintainers of both A and B.

Another thing is – what if we changed A, B and C at the same time to solve some mutual bug? We need to update all of them and publish a new version of each. That’s touching 3 repositories. What if we have 10 modules? Or 50 microservices that require these modules?

These problems (and some more) are being solved by the second options – a monorepo. In this article, we will see how to easily publish and maintain NPM modules in an NX monorepo.

How to maintain NPM modules in an NX monorepo?

There are several flavours to an NX monorepo. For this tutorial, let’s assume we are building an angular modules library.

In NX, we will build the repository this way:

Initialization: npx create-nx-workspace --preset=angular
This will create a project for us with an angular application (which can be our library documentation or playground)
Now that we have a repo, we can start building our independent modules: nx g @nrwl/angular:lib form-association --publishable
This command creates a new publishable (e.g. npm publish) angular module called form-association.
This is a true use case, since in our team we had to integrate our native web components inside angular applications. The form elements had to have some custom angular directives to be able to interact with angular reactive forms or ngModel.
Other modules we created exposed some custom functionality for angular users that are using our web components. For instance, our dialog component. All in all, it was angular wrappers, directives and services that enabled angular users to more easily use complex components.

That’s about it. For every module we want to create, we run the lib generation command. We develop it and test it. We also use it in the demo app that was conveniently created for us when we created the NX workspace in step 1.

Now that we have tested libraries and an app that uses the libraries and tests them as well (integration tests), we can move on to publish our multiple libraries.

How to publish multiple libraries in an NX monorepo with github actions?

In this part, we will delve a bit into our CI/CD infrastructure. The example here is github actions syntax, but it can be easily “translated” into any CI/CD flow.

The CI Process

The first step would be to test and build our code. That’s pretty self explained in this code:

	name: Test and Build

	on:
	pull_request:
	branches:
	- main

	jobs:
	build-test:
	runs-on: ubuntu-latest

	steps:
	- name: Checkout
	uses: actions/checkout@v2
	with:
	fetch-depth: 0

	- name: Setup NodeJS 14
	uses: actions/setup-node@v1
	with:
	node-version: 14

	- name: Install yarn
	run: npm install -g yarn

	- name: Get yarn cache directory path
	id: yarn-cache-dir-path
	run: echo "::set-output name=dir::$(yarn config get cacheFolder)"

	- name: Cache yarn dependencies
	uses: actions/cache@v2
	id: yarn-cache
	with:
	path: \|
	${{ steps.yarn-cache-dir-path.outputs.dir }}
	**\node_modules
	key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
	restore-keys: \|
	${{ runner.os }}-yarn-

	- name: Install dependencies
	if: steps.yarn-cache.outputs.cache-hit != 'true'
	run: yarn install

	- name: Test
	run: \|
	RUN=CI yarn nx run-many --target=test --all --parallel 5
	yarn nx run-many --target=lint --all
	yarn nx run-many --target=build --all --prod

view raw ci.yml hosted with ❤ by GitHub

On lines 3-6 we state the trigger. In our case, a pull request (or push to pull request) to the main branch.

From here on we create one job that runs on ubuntu. It installs node, checks out the branch, verifies for cache or installs the dependencies and finally runs the tests, linting and build. If all passes, we can merge.

Once the code is merged, the Pull Request is closed and we can start the process of finalizing our integration and start the deployment.

	name: Build and release

	on:
	pull_request:
	types: [closed]

	jobs:
	build-test-release:
	if: github.event.action == 'closed' && github.event.pull_request.merged == true
	runs-on: ubuntu-latest

	steps:
	- name: Checkout
	uses: actions/checkout@v2
	with:
	fetch-depth: 0
	token: ${{ secrets.CI_REPOSITORY_ACCESS_TOKEN }}

	- name: Setup NodeJS 14
	uses: actions/setup-node@v1
	with:
	node-version: 14

	- name: Install yarn
	run: npm install -g yarn

	- name: Get yarn cache directory path
	id: yarn-cache-dir-path
	run: echo "::set-output name=dir::$(yarn config get cacheFolder)"

	- name: Cache yarn dependencies
	uses: actions/cache@v2
	id: yarn-cache
	with:
	path: \|
	${{ steps.yarn-cache-dir-path.outputs.dir }}
	**\node_modules
	key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
	restore-keys: \|
	${{ runner.os }}-yarn-

	- name: Install dependencies
	if: steps.yarn-cache.outputs.cache-hit != 'true'
	run: yarn install

	- name: Test
	run: \|
	RUN=CI yarn nx run-many --target=test --all
	yarn nx run-many --target=lint --all

	- name: Build components
	run: \|
	yarn nx affected:build --prod --with-deps --base=main

	- name: Raise version of affected libraries
	run: \|
	LATEST_TAG=$(git tag -l "v*" --sort=-version:refname \| head -n 1)
	LIBS=$(yarn nx affected:libs --base=$LATEST_TAG --head=HEAD --plain \| awk 'NR > 2 && $1 != "Done" { print $1 }')
	for LIBRARY in $LIBS
	do
	cd ./libs/$LIBRARY
	npm version minor --no-git-tag-version --no-push
	echo "Bumping $LIBRARY"
	cd ..
	cd ..
	done
	npm version minor --no-git-tag-version --no-push

	- name: get-npm-version
	id: package-version
	uses: martinbeentjes/npm-get-version-action@master

	- name: Push changes
	run: \|
	git fetch
	git config user.email "[email protected]"
	git config user.name "Vivid CI"
	git add --all
	git commit -m "update versions to ${{ steps.package-version.outputs.current-version }}"
	git push

	- name: Tag release
	run: \|
	git tag -a v${{ steps.package-version.outputs.current-version }} -m "tag release v${{ steps.package-version.outputs.current-version }}"
	git push --follow-tags

view raw prepare-for-release.yml hosted with ❤ by GitHub

Here our trigger is a closing Pull Request and we also make sure this is a merged pull request (line 9). From then on, we have a single job again. It’s a bit more complex, but most of it is familiar – checkout, yarn, test, lint.

Now the build is a bit different (lines 51 to 53). Here we don’t want to build all the libraries – just those that we need to update. In other words, we would like to update only the libraries that were affected by the changes in this Pull Request.

The command: yarn nx affected:build --prod --with-deps --base=main does exactly that for us. Nx has a dependency graph of all our libraries and applications in the repository. It then gets the changed libraries between main and the PR’s commit and the libraries that are dependent on them. It then builds them because that is what we asked Nx to do: nx affected:build.

The rest of the code raises a version and pushes the version change to main as well as tag the new head of the main branch.

The CD Process

The code is merged and a version was raised. Now it is time to publish and deploy. For this, we have two files – one to publish and another to deploy our demo app.

Deploy the Demo

	name: Deploy Demo Site

	on:
	push:
	tags:
	- v*

	jobs:
	deploy:
	runs-on: ubuntu-latest

	steps:
	- name: Checkout
	uses: actions/checkout@v2
	with:
	fetch-depth: 0

	- name: Setup NodeJS 14
	uses: actions/setup-node@v1
	with:
	node-version: 14

	- name: Get yarn cache directory path
	id: yarn-cache-dir-path
	run: echo "::set-output name=dir::$(yarn config get cacheFolder)"

	- name: Cache yarn dependencies
	uses: actions/cache@v2
	id: yarn-cache
	with:
	path: \|
	${{ steps.yarn-cache-dir-path.outputs.dir }}
	**\node_modules
	key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
	restore-keys: \|
	${{ runner.os }}-yarn-

	- name: Install dependencies
	if: steps.yarn-cache.outputs.cache-hit != 'true'
	run: yarn install

	- name: Build Demo
	run: yarn build:deploy

	- name: Deploy 🚀
	uses: JamesIves/[email protected]
	with:
	GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
	BRANCH: gh-pages # The branch the action should deploy to.
	FOLDER: dist/apps/angular-vivid # The folder the action should deploy.
	CLEAN: true # Automatically remove deleted files from the deploy branch

view raw deploy-demo.yml hosted with ❤ by GitHub

This code is quite similar to what we saw so far. We have a trigger at the top – this time, a tag that starts with v (meaning, we have a new version tag). We then checkout, install, build and deploy. In our case, we deploy the demo to github pages. You could also deploy it anywhere else (AWS, heroku, GCP etc.).

Publish to NPM

	name: Publish

	on:
	push:
	tags:
	- v*

	jobs:
	publish:
	runs-on: ubuntu-latest

	steps:
	- name: Checkout
	uses: actions/checkout@v2
	with:
	fetch-depth: 0

	- name: Get the two latest versions
	run: \|
	CURRENT_VERSION=$(git tag -l "v*" --sort=-version:refname \| head -n 1)
	LAST_VERSION=$(git tag -l "v*" --sort=-version:refname \| head -n 2 \| awk 'NR == 2 { print $1 }')
	echo "current_version=$(echo $CURRENT_VERSION)" >> $GITHUB_ENV
	echo "last_version=$(echo $LAST_VERSION)" >> $GITHUB_ENV

	- name: Setup NodeJS 14
	uses: actions/setup-node@v1
	with:
	node-version: 14

	- name: Install yarn
	run: npm install -g yarn

	- name: Get yarn cache directory path
	id: yarn-cache-dir-path
	run: echo "::set-output name=dir::$(yarn config get cacheFolder)"

	- name: Cache yarn dependencies
	uses: actions/cache@v2
	id: yarn-cache
	with:
	path: \|
	${{ steps.yarn-cache-dir-path.outputs.dir }}
	**\node_modules
	key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
	restore-keys: \|
	${{ runner.os }}-yarn-

	- name: Install dependencies
	if: steps.yarn-cache.outputs.cache-hit != 'true'
	run: yarn install

	- name: Build libraries
	run: \|
	yarn nx affected:build --prod --base=$last_version --head=$current_version

	- name: Publish components (Github packages)
	run: \|
	for LIBRARY in $(yarn nx affected:libs --base=$last_version --head=$current_version --plain \| awk 'NR > 2 && $1 != "Done" { print $1 }')
	do
	cd ./dist/libs/$LIBRARY
	npm publish --registry https://npm.pkg.github.com --no-git-tag-version --no-push --yes
	cd ..
	cd ..
	done
	env:
	GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

view raw publish.yml hosted with ❤ by GitHub

This step starts much the same as the other deployment. Triggered by a version tag, it starts the job but also runs Get the two latest versions which does exactly what it says it does.

In lines 54-52 it builds the affected libraries between the two version tags. That means, we build only what we need to, just like we did in the prepare-for-release workflow.

Now comes the publish part (lines 56-66). We run the affected command, only this time we get its output as text. Line 58 starts a loop that goes over every affected library. Inside the loop we cd into each library (line 59) and then publish it (line 60).

That finalizes the whole CI/CD flow of our NPM modules and our demo app.

Summary

Nx is a very powerful tool. The CI/CD example above is just a fraction of what Nx gives you.

In the article we saw how easy it is to create ready-to-publish libraries (with the --publishable flag) and how easy it is to find libraries (and applications) that were affected by certain commits using the affected command.

The techniques in this article can be used not only for NPM modules. You can use them for microservices, so the flow is kind of the same, only the end is not npm publish but an AWS or some other cloud provider’s CLI command to deploy your service.

Nx also comes with other very useful tools for CI/CD and standardisation. One of them is the Nx cloud service which helps you with caching builds and tests. This speeds up CI/CD as well as development processes.

I urge you to check out Nx if you haven’t so far.

Nx had a great conference with all the content free to watch on YouTube:

Day 1 link: https://youtu.be/oG2QbFquraA
Day 2 link: https://youtu.be/hlGOaGDsWKg

Would love to hear/read your feedback about your usage of Nx.

Facebook Tweet Pin LinkedIn

github actions

How to deploy NPM modules in an NX monorepo and github actions?

The problem of maintaining multiple modules and services

How to maintain NPM modules in an NX monorepo?