Kloudless Blog

Kloudless Unified APIs enable you to code once and integrate many

Vagrant and Puppet!

Your Situation

This could be quite varied, you could be:

  1. A solo developer looking for a fast/easy way to have a local dev environment that resembles your production environment (say you develop on OS X, but are deploying to an infrastructure running some distribution of Linux. As a bonus, there is also an easy way to deploy to Amazon’s EC2 if you have a solid setup locally.
  2. A member of a team, where everyone has their own development style and want to avoid the headaches of cross-platform support.
  3. Someone who normally sets up servers in a third-party hosting environment, but you want to test your deployment without paying a bunch of money in wasted servers (this is where I am!)

What is Vagrant?

Vagrant is essentially a wrapper around a variety of virtual machine providers. If you have ever used Make to build a piece of software, it is kind of like that except with virtual machines. It provides a single command that uniformly creates, provisions, destroys, and connects to machines. You can use many different VM providers, but I will be using VirtualBox because it is free and easy to use. Usually it is a pain in the butt to create a virtual machine, install the operating system, etc. Vagrant makes it super easy, and there are lots of premade “boxes” for you to use (more on this later).

What is Puppet?

Puppet is an infrastructure automation tool and we are going to use it to take the hard work out of setting up our systems. We can do this because lots of people have put a ton of effort into writing modules that we can use. This won’t be a tutorial on Puppet, but it will go over the basics so that you can use modules that other people have written.

But I don’t like/use Puppet!

That is fine. Thankfully Vagrant is flexible in its provisioners and you can read more about the alternatives. For simplicity I am just going to cover Bash and Puppet since that is what I am familiar with. The overall process should be the same if you decide to use Chef or Ansible, but because I don’t know much about them, I won’t discuss them further.

Getting Started

Installation

Vagrant is a breeze to install: you can read over their installation instructions. Essentially, download the package that is relevant for your platform of choice and install it in the way you would normally install a package.

You will also need to install your virtual machine provider, in my case VirtualBox.

First VM

Once it is installed, you will want to create a new project directory and initialize it.

This will create a new directory and get it ready with a file called Vagrantfile which will contain all the information that Vagrant needs to manage your dev environments.

In that file there will be a bunch of comments about the different things you can put in that file. For now we just care about configuring a base “box” (which is just a virtual machine image) and some other machine properties. I ended up with a file that looked something like this:

Here are the things that we configured so far:

  1. config.vm.box: The name of the box that you are bringing up.
  2. config.vm.box_url: The location where Vagrant can look to download the box if you don’t already have a copy on your machine.The box that I chose is provided by Puppet Labs and doesn’t have any pre-installed provisioning software. This will be installed later as part of the bootstrapping process so that it can always be up to date.
  3. config.vm.hostname: The hostname of the VM.
  4. config.vm.network :private_network: The private IP address that the VM will have on the private VM network.
  5. config.vm.network :forwarded_port: The port labelled “guest” on the guest VM will be accessible on the port labelled “host” on your machine.
  6. VM memory: I gave my VM 1GB because it seemed like it would be enough (Note: this is provider dependent).

Once you have the file in place you can create and provision your VM from the same directory:

After that command finishes running, you will have a VM ready for you to connect and start messing around with. You can access the machine via SSH using:

You will now have a shell on your virtual machine as the vagrant user. The user has passwordless sudo access on the machine, it full fledged Ubuntu 12.04 LTS VM and you can do whatever you want! This is great and all, but we want to make things more automated, so you will want to exit your SSH session and get rid of the VM with:

Automating All the Things

As was talked about earlier, the nice thing about Vagrant is that it is really easy to offload the configuration to an automated tool, in this case Puppet. Since our VM is pretty bare bones, there is some extra work that we want to do to prepare it.

Bootstrapping

The box that I chose doesn’t by default come with puppet installed on it, this was a deliberate choice to make sure that I could use the same version I am using in production without having to change the box all the time. As such we need to do a little extra work. Preparing the machine to be puppeted is relatively straightforward and we are going to take advantage of the fact that you can use multiple provisioners on a single machine. In order to use the shell provisioner we add the following lines to our Vagrantfile before the final end:

Then we create a file called bootstrap.sh in the same folder as our Vagrantfile that contains the following:

One important thing to notice is that the script is idempotent (meaning that it can be run multiple times without having any bad effects), this is important because we can run the provisioners without creating a new machine. Your virtual machine will now be ready to be controlled via Puppet!

Adding Puppet

Since we don’t want to install Puppet on our host machine, bring up the VM and connect to it. Puppet is already installed. We can do all of our initial Puppet configuration directly within the vm. By default, the folder containing the Vagrantfile is shared on the virtual machine in the path /vagrant and this will make a good location to store our puppet configurations. Once in that directory you will want to create a skeleton of your puppet dir:

From here we will want to install some puppet modules that will make setting up a basic LAMP server easy (for now we will just keep everything on the same box). There are lots of different ways to install puppet modules, but the most straightforward is using the way that is built into puppet:

Now we will want to actually write a puppet manifest , so we will want to create /vagrant/puppet/manifests/site.pp with the following:

This describes how the server gets configured, the basic function is setting your vm with apache, mysql, and php along with a test page. Here are some more details about the different parts:

  1. The node definition is how we collect configuration for the machine, the label ‘development.kloudless.vm’ matches the hostname that we configured the vagrant box with.
  2. This statement uses the mysql class to install the mysql server and sets up the admin password for the db as ‘herpderpderp’.
  3. That is how the php bindings for mysql get installed
  4. Those two lines install the apache server package and mod_php respectively
  5. We want an apache vhost where we can access our basic test application.
  6. This actually creates the directories where the vhost content will live
  7. That is our application! Right now it doesn’t actually use the database, but it is a good example, we can define the contents of the file inline relatively easily this way. We will replace this later.
  8. This is a way of realizing virtual resources which in this case configures your machine’s firewall rules.

Now that we have the puppet configuration ready, we need to have Vagrant use it. This can be done easily by adding the following lines to your Vagrantfile after the shell provisioner lines:

This just tells vagrant where to find the modules we installed and the manifest we wrote. So now you are ready to vagrant up. Once your box is built, you should be able to visit http://localhost:8080 and see the output of your test page.

With a few small adjustments, you could use this to develop a full blown php app. The adjustments you would probably want to make are as follows:

  1. Remove the index.php file block from the sites.pp file
  2. Change the webroot value in the vhost code block to be “/var/www/app”
  3. Configure a shared folder that apache can point to as the webroot. For example, if you have a folder called app that contained your application and it is in the same directory as your Vagrantfile, you would add the following line to the Vagrantfile:
    config.vm.synced_folder "app", "/var/www/app"

Once those changes are made, you should be able to just run vagrant provision to update the settings and you can start dumping your project files into the app directory and you should see those changes on the vm.

Learning More

If you wanted a LAMP server to develop a php application, you don’t really need to go any further. Odds are, however, that you want to do something more than just this. If you want to run a more complicated application you can do this pretty easily if you can find a module to do it. The specific details of how you use a module depends on what it is, but the documentation is usually ok. If you want to more seriously manage your dev box with puppet, you should do some more reading so you can use the different modules people have written more easily. Here is some recommended reading :

  1. The Learning Puppet Series
  2. Example42 Tutorials. They also have a bunch of good modules you can use
  3. Puppet 3 reference manual

Hiera

If you are a more experienced puppet user, you might be familiar with Hiera and want to use it with the modules/classes that you have written. It is pretty easy to do. First, change the value of puppet.options in your Vagrantfile to “–verbose –debug –hiera_config /vagrant/puppet/hiera.yaml”. Now you need to populate that file with your hiera configuration, depending on how you are using it you might end up with something like this:

Now you will want to make a directory called hieradata in the puppet directory. From there you can put all of your hiera variables in a file called common.yaml.

Multiple Servers

One server is great, but if you have a real production infrastructure, it most likely doesn’t consist of a single machine. Vagrant is pretty nice in that it inherently supports it, you just need to do some modifications to your Vagrantfile. If we wanted to have our database server be separate from our web server. Here is our new Vagrantfile:

The primary change are the blocks labelled [1] and [2], those are just adding the host specific configurations for the network settings, so that they can be dealt with separately and talk to each other. Now that there are two boxes vagrant can refer to, for example if you wanted to just bring up the webserver you would do vagrant up web. The name you refer it to is the key that is the argument to config.vm.define.

In order to have puppet provision both of the servers, we also need to modify the site.pp file so we take into account the fact that we have two nodes. This is pretty straightforward and ends up essentially splitting the single node declaration into two, resulting in:

Essentially all that happened, is that we split the original node definitions into two separate ones. Once you have these manifests in place, running vagrant up brings up both virtual machines in sequence. Once they are up, they can communicate over the private network via the configured ip addresses. The hostnames you configure can’t get resolved (it wouldn’t be too hard to put the ip’s and hostnames in each server’s /etc/hosts file through puppet, but that isn’t too relevant here). Now this is more like something you would see in your actual infrastructure.

Moving to the Cloud

Once you have a real application developed and configured you probably want it to be accessible to everyone, so why not push it out to Amazon’s EC2! This can be done easily through the AWS provider add on to Vagrant. This will basically be a different Vagrantfile that you will use specifically to push to EC2. In order to replicate the configuration we had locally, you first need to install the plugin:

Once you have the plugin installed, we are going to take advice from the plugin’s docs to get started quickly using a dummy box. All this means is that our configuration will be explicit within the declaration of the box. To register the dummy box, we do the following:

Then we need to modify our existing Vagrantfile to create and provision the box, since it is going to be pretty different, we can create the different configuration blocks:

These blocks can just be added into your Vagrantfile before the final end. This assumes that you already have an account and a key-pair set up, so you will need to substitute your credentials into the proper place. I have chosen the Ubuntu 12.04LTS AMI because it is easy to use and us-west-2 because it is pretty close to where I am located (it is in Oregon). Now here is the somewhat tricky bit, because of the way that EC2 works, you won’t really know the hostname of the machine before you bring it up and the network configuration options of Vagrant don’t support setting the hostname. There are a couple ways around this:

  1. Nodeless Puppet: A pretty novel approach that is fact driven. It is interesting and probably what I would recommend if you are going to use this in a real production environment.
  2. Just bring it up and do provisioning afterwards: This is clunky, but easy and what I will do for this blog post.

So you will bring up your vm with vagrant up web_aws. The shell provisioning will go ahead just fine, but the puppet provisioning will fail. Ths is ok, we just need another puppet node definition. Basically I am going to just copy the node definition on dev-www (if you wanted to bring up the database server in EC2 it would be the same kind of process):

You can replace THE NEW HOSTNAME with the short domain name that the node thinks it has, in my case it was ip-10-251-32-195. Now you can actually provision your vm with vagrant provision web_aws. In order to actually view the test page, you will need to have your security groups set up properly, but if you just want to check you can vagrant ssh web_aws and then curl http://localhost and see that it works.

Extra Notes

Some things work differently when you are using the AWS provider and the main one you will notice is the shared directories. These get sync’d with rsync every time you run vagrant up, vagrant reload, or vagrant provision. You can also build a bunch more aws configuration into your box definition, so you don’t have to specify it by hand.

Since it is in EC2 you can take advantage of tags and user data. In our production puppet environment, we have an enc that decides what classes to give an instance based on its tags, but that requires a puppet master, which I didn’t really talk about (right now, vagrant just uses puppet apply).

Conclusion

Hopefully I have given you a good taste of how easy it is to put these two tools together to make your development and deployment a lot easier! Puppet is a really great tool and I highly recommend learning more so you can take full advantage of this work flow. I am still working to fully utilize all these great tools and it would be great to hear about other peoples’ experiences.

Categories: