Shared Vagrant setup across multiple projects
This post is intended for automation lovers, who want to automate the setup of the development environment in microservice-based projects. One option is to use Vagrant.
If you are setting up a project based on microservices (or any project organized into a few distributed, self-contained components) you will probably end up hosting them in separate Git repositories. This is to allow for division of work, keep an organized workflow, prevent bottlenecks, etc.
Some of these components may share the same development environment configuration and may even be designed to run on the same machine.
A quick digression
We have different approaches to automating the development environment configuration for our dev team.
One option is to use a custom bootstrap script to setup our machine. This option has some drawbacks, which include constraining the variety and types of workstations our developers can use and making it difficult to select and install specific versions of our project’s dependencies, due to project restrictions — e.g. when the project requires PostgreSQL 9.2, but we have 9.4 installed on our machine.
A second option is to use Vagrant. Vagrant allows us to create and configure guest virtual machines (VMs) easily on our developer workstations, using a single description file. This file is called
and is normally located in the project’s base directory. Using this file, we can tell Vagrant to choose a variety of VM providers (e.g. VirtualBox or AWS), to define a list of tasks to be run once the VM has been provisioned, to configure the network, and so on. Then, Vagrant will quickly spin up a VM containing our development environment and keep it in sync with our source code in the host OS. Changing the source code in the host OS will automatically reflect in the VM, and vice versa. One of the benefits of this mechanism is that we can edit the source code with our preferred IDE installed in the host OS (e.g. OS X) and run/debug/test the application in the VM (e.g. an Ubuntu Server 12.04, including a particular set of dependencies). Team members using Vagrant can reproduce exactly the same configuration on their machines.
A third option is Docker. Docker is a more efficient alternative to using VMs. In a nutshell, Docker’s service layer runs over the host OS and creates isolated containers that include the app under development and all of its dependencies. VMs, on the other hand, don’t share the same OS instance. For example, two apps placed in two separate Docker containers will run on the same host OS. Conversely, two apps placed in two separate VMs will run on two different OS instances — and will therefore consume more resources.
To define the scope of the problem without losing generality, our goal is to automate the development environment’s installation and configuration for a project that has the following constraints:
- It is organized in many Git repositories.
- Each repository contains the source code of an individual component (e.g. a service, an app, the project’s core).
- Components run independently from each other.
- Components can communicate between each other (e.g. through RESTful endpoints).
- Components (may) share the same development environment.
- Components (may) run on the same (virtual) machine.
- The environment setup should be easily portable.
- The solution should encompass at least Linux or Mac workstations.
- The solution shouldn’t constrain any development, communication or management tool that developers are used to.
Our solution involves using Vagrant and Git subtrees to reference and checkout the external components from a single repo.
We are not using Docker-native on the host machine, this time, because we want our solution to be as generic as possible: Docker would narrow our solution down to only using Linux machines and we want to be nice to OS X devs. If we still want to use Docker, Vagrant comes with a Docker provisioner out of the box that fits in with our solution (see Vagrant’s Docker provisioner).
Let’s call this single repo
will contain the
with all the development environment configuration, a Readme file, including any manual setup required after provisioning the VM, and the subtrees.
Git subtrees are great because they allow us to work on our external repositories, right from our super project —
, in our example — with relative ease. You only have to learn a new merge strategy — i.e.
— and be a little disciplined when pulling and pushing.
To simplify the explanation, let’s suppose we have three different repos,
, each one containing the components we want to configure.
As a first step, we will create the repo and initialize it, using Git:
$ mkdir dev-env $ git init
Then, we are going to add a
‘s base directory. (The
is not strictly required, but it is recommended that we devote some time to writing one.)
Assuming that our components will listen to different TCP ports, our
would end up looking like this:
VAGRANTFILE_API_VERSION = "2" Vagrant.configure(2) do |config| config.vm.box = "ubuntu/trusty64" config.vm.network :forwarded_port, host: 8080, guest: 8080 # core config.vm.network :forwarded_port, host: 8081, guest: 8081 # service-auth config.vm.network :forwarded_port, host: 8181, guest: 8181 # service-notifier end
At this point, we are ready to set up the subtrees.
The first step is to add one Git remote per external component. This is a trick that will allow us to refer to the subtrees in a shorter form:
$ git remote add core firstname.lastname@example.org:moove-it/core.git $ git remote add service-auth email@example.com:moove-it/service-auth.git $ git remote add service-notifier firstname.lastname@example.org:moove-it/service-notifier.git
base directory, we will add the subtrees in their respective
folders, referring to the remotes we have just created:
$ git subtree add --prefix core core master --squash $ git subtree add --prefix service-auth service-auth master --squash $ git subtree add --prefix service-notifier service-notifier --squash
(We should only have to do this once, or just each time we want to include a new external component in our setup, which won’t happen very often. The general command to add a new subtree is
git subtree add --prefix [prefix_folder] [remote_name] [branch] --squash
Note that we are using the
modifier. This is to avoid storing the entire history of the components’ repos in
After adding the subtrees, our commit log will look something like this:
9298f0a Merge commit 'b376ce343d7ebfa47811e1fb98ced870ce346a4c' as 'service-notifier' b376ce3 Squashed 'service-notifier/' content from commit 3d51460 a5a0407 Merge commit '56c06a59c1fc79d1a60a6712f467517c22fbdc05' as 'service-auth' 56c06a5 Squashed 'service-auth/' content from commit 1a7d7a3 1ce0b38 Merge commit '049d2492ae3fe15ec65d3663745e758638746414' as 'core' 049d249 Squashed 'core/' content from commit f51b5ba
command, we can keep our individual components up to date and push changes for review.
We will then use
git subtree pull
to update the components. For example, to update
‘s master branch:
$ git fetch core master $ git subtree pull --prefix core core master --squash
If we are working on a
branch, we will then use
git subtree push
to push our
$ git subtree push --prefix=core core feature
Finally, we can push
to our remote repo.
One of the benefits of this solution is that the team maintaining the original component’s repo doesn’t necessarily have to be aware of the existence of
and all the infrastructure we have created to automate the development environment setup.
This solution, to set up a common development environment for autonomous components living in separate repositories, may or may not make sense, depending on your organization’s policies and/or the project’s complexity.
Scenarios that may benefit from this approach include: projects that are difficult to set up, requiring specific dependency versions, overwriting certain libraries or copying certain files to specific folders; or, projects that require distributing the development environment on separate machines. Some developers may benefit from using the same Vagrant setup across a lot of projects (for example, WordPress sites), having a single custom base box, so they can update them from a single place. You will be able to tell if this solution suits your needs.
This blog post follows a discussion I started on Stack Overflow. There, you will find alternative solutions that may be a better fit for your project.
Vagrant is an excellent solution for automating your project’s development environment setup. When the project’s architecture is organized in several components, living in separate repos, and you need to run them in the same guest machine, while still keeping the ability to edit/debug/test/interact with them from your host machine, you will want to avoid adding your Vagrant setup to each project repo.
Instead, you need to find a way to end up with a single Vagrant folder pointing to the sub-projects’ folders. In addition, you will want a solution that keeps the folders in the guest machine synchronized with the ones in the host machine.
The approach outlined in this article describes creating a Git repo, containing a single
and Git subtrees, each one linked to an external project. Once the VM has been provisioned and loaded, the external projects will be placed in Vagrant’s default directory in the VM, where they can run in the proper environment.