How to Use Lerna in a Monorepo to Deploy Terraform Modules

Written by: Igor, Clevertech DevOps Engineer

If you work with Terraform on a daily basis, you’ve already struggled with using and managing Terraform Modules. Some say that it's better to have each module in a single repository because this makes it easier to version and deploy them to the Terraform Cloud registry. But when you have a ton of modules, things usually start to get out of hand. Others say that monorepos are better because all your modules are in one place which allows you to follow patterns and validations. Versioning is harder however because tags and releases wrap the entire repository which means you can't have single versions for each module.

Here at Clevertech, we started out using multiple repositories but ended up facing two major issues. The first was that we eventually had too many repositories; and second, each repository had its own patterns (e.g. variable naming conventions, documentation, applicable use cases). So, we decided to switch to a monorepo to store all of them. However this required us to use those ugly as hell Git sources in the modules, and (obviously) versioning was off of the table.

But, what if we could bring together the advantages of both worlds and leave behind the mess? Can we have a monorepo that still allows each module version to be unique? If you’ve made it here, you’re probably guessing that the answer is yes. (Imagine coming all this way just to confirm that something doesn't work, right?)

Now enters Lerna, a tool to manage JavaScript projects with multiple packages. Many JavaScript projects like React, Angular, and Ember, use monorepos to hold their packages, so it would make sense that a JavaScript tool has been written to solve this kind of problem.

Lerna uses its lifecycle management to handle package (or module) versioning, and the only two things it requires are:

• Each module must have a package.json file with its package version;
• The repository root should have a lerna.json configuration file;

So, when you stop to think about it, we can use Lerna to version anything in a monorepo. We just need to put a package.json into it and it's done! But then the question becomes how do we use this with Terraform Modules?

First, we put the package.json inside each module so we can see how Lerna sees them. And as a bonus, Lerna can both list all the modules in the repository or list only the modified modules. This is super handy when you need to change only a part of a repository and don't want to build/lint/validate the entire codebase. With this list, we first create the docs for our modules using terraform-docs, and then we validate them with terraform fmt -check -diff (with the -diff option, terraform can warn us if anything is wrong and tell us how to fix it) and terraform validate. We also use super-linter from GitHub to lint all the files in the repository, not only the terraform ones.

With everything done and validated, we’re ready for deployment (this is what we were aiming for!). The process is triggered by running the lerna version command. ne of the main advantages to using Lerna is its lifecycle. The version command has three lifecycle scripts: preversion, version, and postversion. First Lerna will get all modified modules, track the commits of each one and determine what will be the new version, and then it will run the preversion of the root folder. After that, Lerna will enter each module and run its lifecycle scripts. Here is where we send our modules to the Terraform Registry. To achieve this, we use the Terraform Module's API. We gzip all files of the module together, create the version on TF Registry and upload the gzip. With everything uploaded to the TF Registry, we can now commit the new versions and generate GitHub releases for each module.

Looks simple enough? That’s because it is! We can do our work and let Lerna take care of the rest. This pattern can be applied to any monorepo you are working on, not only a JavaScript or Terraform one. This improves organization without the cost of lose versioning. If you want to see it all in practice, you can see working examples using Jenkins and GitHub Actions here . And if you still have any questions, feel free to open an issue on the example repository.

Happy terraforming!