Using vagrant and puppet to setup your symfony2 environment

As you may now by now, I’m a big fan of using Puppet for configuration management. Since the rise of virtualization, these applications are becoming one of the more dominant tools in a developers tool chain. Together with other tools, setting up a complete development environment with just a single command is not only reality, but it’s becoming for a lot of developers a daily practice. But even for open source projects like and are seeing the benefits of  having "development environment on the fly". New contributors don't have to spend a lot of time setting up their environment, but it's automatically generated: the code setup, the database server together with a filled set of data, any additional components like varnish, memcache, reddis etc. This blog post gives an overview on how to setup a symfony2 project with the help of vagrant and puppet.

Puppeteering your symfony2 project by itself isn’t a very complex task, but there are some catches you have to think about. First of all, make sure you have all the basic necessities to run Vagrant:

• VirtualBox (https://www.virtualbox.org/)
• Ruby (http://www.ruby-lang.org/)
• Vagrant (http://vagrantup.com/)

These systems are pretty easy to install and work on Linux, OS/X and Windows machines without problems.

Let’s start with our basic Vagrantfile:

Vagrant::Config.run do |config|
    # This vagrant will be running on centos 6.2, 64bit with puppet provisioning

    config.vm.box = 'centos-62-64-puppet'
    config.vm.box_url = 'http://packages.vstone.eu/vagrant-boxes/centos/6.2/centos-6.2-64bit-puppet-vbox.4.1.12.box'

    # Use :gui for showing a display for easy debugging of vagrant

    config.vm.boot_mode = :gui

    config.vm.define :project do |project_config|
        project_config.vm.host_name = "www.project.dev"

        project_config.vm.network :hostonly, "33.33.33.10"

        # Pass custom arguments to VBoxManage before booting VM

        project_config.vm.customize [
            'modifyvm', :id, '--chipset', 'ich9', # solves kernel panic issue on some host machines

            '--uartmode1', 'file', 'C:\\base6-console.log' # uncomment to change log location on Windows

        ]

        # Pass installation procedure over to Puppet (see `support/puppet/manifests/project.pp`)

        project_config.vm.provision :puppet do |puppet|
            puppet.manifests_path = "support/puppet/manifests"
            puppet.module_path = "support/puppet/modules"
            puppet.manifest_file = "project.pp"
            puppet.options = [
                '--verbose',
#                '--debug',

            ]
        end
    end
end

This is a simple configuration that will setup one virtual machine. The base system that will be used will automatically downloaded from the config.vm.box_url. As you can see, this is a public available box, which is no problem since we’re doing development on it. These boxes are minimal installs of the OS but has some extra stuff like a predefined root password, a vagrant user and some small things. It’s quite easy to create your own baseboxes, but 9 out of 10 times, you can get away by using a publicly available box.

Note that you have to decide on using a 64 or a 32 bit OS here. A 64bit system is better and most host machines can run it with no problem, but you can get into trouble when you want to run a 64bit virtual box on a 32bit host machine (or at least, a 32bit OS). Picking a 32bit virtual machine means it will be able to run on any machine without trouble.

Next up we can define if we want to have a virtual box console or not (the config.vm.boot_mode = :gui). This makes it easier to debug if something goes wrong, but as soon as your box is running perfectly, you can remove this setting, which means it will run headless.

The next lines are the configuration per box, since we have only one virtual machine, we only have one configuration, but such a setup makes it easier to setup multiple machines (for instance, a web machine and a database machine). It’s even possible to mimic your whole production infrastructure through vagrant!

In the config we define the hostname of the machine, the network (we use a host-only with a specific IP address, you can also use NAT, but i wouldn’t recommend it, since it means you have to deal with port forwarding as well).

You can configure the whole machine if you want, for instance, we make sure we use the ich9 chipset, as the default chipset might cause some issues on some host machines.

At the end of the configuration we are looking at the provision part, where we will be using puppet, where we can define some settings and parameters on it.

To actually run this system, all you have to do is issue:

vagrant up

That’s it. From that point, Vagrant will look for a Vagrantfile in the current directory and start working. First off all, it will try to find the basebox. If the basebox isn’t downloaded before, it will download it and place it inside its own  box-directory. This means you only have to download a box once. The next time you start Vagrant, it just uses the same box again.

Once the box is downloaded, it will create a virtual machine from that box. As you can see, each virtual machine that is created from vagrant has a additional timestamp added to it (ie: project_1340793356). Once the machine has been automatically configured, Vagrant will start up the machine. Once the machine has been completely started, the provisioning will kick in.

Since we are using puppet, it will actually do a puppet run based on the configuration we provided. The most important configuration is the puppet.manifest_file, which tells puppet which file to run. This will be our “bootstrap”.

Puppet

There are many ways to puppetize your system and even though there still can be a lots of things improved in the current setup, it pretty much does what it needs to do and still is flexible enough that you can change to multiple systems if needed on a later date. The puppet scripts are located in the /scripts/puppet directory. The manifests directory is nothing more than a placeholder where we can define our nodes and parameters. If needed we can do our adjustments here. The projects.pp file is our main entry point (as specified by our Vagrantfile). The default node (you can specify per hostname), includes our params and project. The params is a param.pp file located in the same directory, while our project is a module, which resides in our /scripts/puppet/modules/project directory. As you can see, the project.pp file doesn’t really hold a lot to it. The only thing we really set is the “Exec” thing, which tells that whenever we call the “exec” resource (lowercase), the paths will be set correctly.

# /support/puppet/manifests/project.pp

# Set default path for Exec calls

Exec {
    path => [ '/bin/', '/sbin/' , '/usr/bin/', '/usr/sbin/' ]
}

node default {
    include params
    include project
}

The params.pp file holds parameters for our project. For instance, if we have the possibility to tweak our environments, it should be done here. For instance, in this params.pp file, we have the option on enabling or disabling phpmyadmin. Furthermore, we have some global settings like database credentials, some paths and some hosts.

# /support/puppet/manifests/params.pp

class params {
    # Hostname of the virtualbox (make sure this URL points to 127.0.0.1 on your local dev system!)

    $host = 'www.project.dev'

    # Original port (don't change)

    $port = '80'

    # Database names (must match your app/config/parameters.ini file)

    $dbname = 'project'
    $dbuser = 'project'
    $dbpass = 'secret'

    $filepath = '/vagrant/support/puppet/modules'

    $phpmyadmin = true
}

The most important part of all are the modules located in the /support/puppet/modules directory. As you can see, we have 5 modules, but 4 of them are actually third party ones coming from example42, which is a great site with lots of puppet modules. We can use them as git submodules, but in our case we just copied them over. For now, they aren’t really that important but we will focus on the “project” module (which should be the name of your project).

A puppet module consists of a standard hierarchy with files, manifests, templates and sometimes other stuff. The manifests directory has an init.pp file which will be automatically called. So when I include the “project” module,  the file /support/puppet/modules/project/manifests/init.pp will be automatically called. As you can see inside this file, it’s all neatly namespaced and inside a “project” class. (again, change this to your project name). This project consists of 4 stages, a setup, a sql, a web and a symfony2 stage. Note however, that puppet is declarative and does not rely on any order. If puppet decides on doing the web stuff before doing the setup, it’s his choice unless we explicitly add some ordering.

# /support/puppet/modules/project/manifests/init.pp

class project {
    include project::setup
    include project::sql
    include project::web
    include project::symfony2
}

Let’s start with the setup.pp file, in which we will setup the environment so it can be provisioned and installed. First of all, we need some base packages, since i’m a midnight commander nut, the first package that needs to be installed is “mc”, but things like “git” are also important.

# /support/puppet/modules/project/manifests/setup.pp

class project::setup {

    # Install some default packages

    $default_packages = [ "mc", "strace", "sysstat", "git" ]
    package { $default_packages :
        ensure => present,
    }

    # Setup a EPEL repo, the default one is disabled.

    file { "EpelRepo" :
        path   => "/etc/yum.repos.d/epel.repo",
        source => "${params::filepath}/project/files/epel.repo",
        owner  => "root",
        group  => "root",
        mode  => 0644,
    }
}

The next part will setup an Epel repository. This is needed since our standard basebox (running CentOS) has got the Epel Repository installed, but has it disabled by default. We cannot tell puppet it needs to enable the repository, so what we do is tell puppet to overwrite the epel repository file with our own version (notice the source and path values). The {$params::filepath} actually points to the filepath as given in the params.pp file (yes, this means a tight coupling, they are better ways to do this). Since our template has got the epel repository enabled by default, we can easily let puppet install packages that are part of the Epel Repository.

Next up it the sql.pp file. This is again nothing more than a namespaced class where we will include “mysql”. This means the mysql module will be loaded and executed, which in turn will install the mysql server (this is the module we downloaded from example42).

# /support/puppet/modules/project/manifests/sql.pp

class project::sql {
    include mysql

    exec { 'create-db':
        unless => "/usr/bin/mysql -u${params::dbuser} -p${params::dbpass} ${params::dbname}",
        command => "/usr/bin/mysql -e \"create database ${params::dbname}; grant all on ${params::dbname}.* to ${params::dbuser}@localhost identified by '${params::dbpass}';\"",
        require => Service["mysql"],
    }
}

The next definition will execute some create-db statements and creates a new user, again with the parameters from our param.pp file. Notice that we have added a “required => Service[‘mysql’] line? This means this command should not be run BEFORE the service mysql will be started. And off course, the mysql service can’t be started before actually installing the mysql packages. Even though ordering does not matter, it might be possible that puppet will install the mysql server BEFORE installing the git package, or the Epel Repo.

The web.pp file. First thing we do (or at least, we configure) is the installation and starting of the Apache webserver (include apache). Again, this is a example42 module located in /support/puppet/modules/apache. Now we see that we have a conditional statement: if the $params::phpmyadmin is true, it will include the project::web::phpmyadmin class. This class is responsible for installing phpmyadmin, so it makes it really easy to tell the system we need phpmyadmin or not without messing around in our manifests.

# /support/puppet/modules/project/manifests/web.pp

class project::web {

    include apache

    if $params::phpmyadmin == true {
        include project::web::phpmyadmin
    }

Now, we are are some of the trickery stuff of having a symfony2 project:

With symfony2, you normally run the application as the same user as the webserver, which is either “www-data” or “apache” based on your Debian or Cent-OS system. However, sometimes we need to do some commandline stuff with symfony. For instance: generate entities, create database or update schema’s, install new users etc. This is done through the standard users with which we login, which happens to be the “vagrant” user. The thing is that you get a lot of conflicts if you don’t setup your rights properly, meaning you should have the vagrant user inside the webserver-group, and make sure everything is group-writeable, or maybe even make it other-writeable by issuing a “chmod 777 -R” on your /app/cache/dev and /app/cache/prod directories (don’t lie: you all did it at least once!).

Another way to dealing with this. Off course, this is a development system so it’s allright to it here, so please don’t do this on a production system, is to change the webserver user from the default user to the Vagrant user. This is really easy since you can set the user in the apache configuration. However, you must also make sure that the php session directory is writable for the vagrant user as well. From that point, we can run commandline tools, and browse both from production and development environments, which getting the nasty “can’t write to app/cache directory” exceptions. There is also another method by using ACL’s (chmod +a or setfacl commands). However, this doesn’t always work on all systems, but this is probably the way you want to setup your system on production.

This is what the following two definitions. The first one will change the user and the group in the apache configuration file. It can only be done if apache has been installed so that’s why we add the “require => Package[‘apache’]” line. Once we have done this, we must restart apache, which is done through the “notify” line.

The /var/lib/php/session directory must be group writeable by vagrant (instead of apache), so we change that too. However, this can only be done after install php.

# Change user / group

    exec { "UsergroupChange" :
        command => "sed -i 's/User apache/User vagrant/ ; s/Group apache/Group vagrant/' /etc/httpd/conf/httpd.conf",
        onlyif  => "grep -c 'User apache' /etc/httpd/conf/httpd.conf",
        require => Package["apache"],
        notify  => Service['apache'],
    }

    file { "/var/lib/php/session" :
        owner  => "root",
        group  => "vagrant",
        mode   => 0770,
        require => Package["php"],
    }

Next definition will setup our vhost, for which we use a template. The docroot will be /vagrant/web, since inside our virtualbox, our root directory of the project is automatically mounted to /vagrant. The template is /project/vhost.conf.erb, which maps to /support/puppet/modules/project/templates/vhost.conf.erb. This template will be automatically filled with the correct values like port number and hostname and such.

Again, the apache::vhost is an external definition found in the example42 modules so we don’t need to worry about it too much.

# Configure apache virtual host

    apache::vhost { $params::host :
        docroot   => "/vagrant/web",
        template  => "project/vhost.conf.erb",
        port      => $port,
    }

Next up, installation of some phpmodules, and thus php itself (we don’t specify php directly, it will be automatically added as soon as we add a module). In our case, we want the mysql, xml and pecl-apc modules.

# Install PHP modules

    php::module { "mysql" : }
    php::module { "xml" : }
    php::module { "pecl-apc" : }

We also want the pecl-xdebug module, but this module is found in the EpelRepo. We must add this dependency, so we will know for sure the xdebug will be installed AFTER it installed the Epel repo.

php::module { "pecl-xdebug" :
        # xdebug is in the epel repo

        require => File["EpelRepo"],
    }

Next augeas. This is a really cool tool about which i blogged before. It basically can load all kind of configuration files and formats and displays them as a hierarchical system. It’s easy to add, update or remove configuration settings this way, without using sed. (notice that I DID use set for changing the apache configuration, because augeas isn’t capable of handling apache httpd configuration decently (yet). We change things like the error reporting, error settings and the datetime zone. Again, this can only be done after installing php, and afterwards we restart the apache webserver (meaning it relies on having the webserver installed as well!)

# Set development values to our php.ini and xdebug.ini

    augeas { 'set-php-ini-values':
        context => '/files/etc/php.ini',
        changes => [
            'set PHP/error_reporting "E_ALL | E_STRICT"',
            'set PHP/display_errors On',
            'set PHP/display_startup_errors On',
            'set PHP/html_errors On',
            'set Date/date.timezone Europe/Amsterdam',
        ],
        require => Package['php'],
        notify  => Service['apache'],
    }

An additional augeas block allows us to set the correct configuration for xdebug. Since we are doing a development environment, this is a simple way to setup xdebug (it might be possible to add a parameter to say if we want xdebug or not). Again, requires php, and restarts apache.

augeas { 'set-xdebug-ini-values':
        context => '/files/etc/php.d/xdebug.ini',
        changes => [
            'set Xdebug/xdebug.remote_enable On',
            'set Xdebug/xdebug.remote_connect_back On',
            'set Xdebug/xdebug.remote_port 9000',
            'set Xdebug/xdebug.remote_handler dbgp',
            'set Xdebug/xdebug.remote_autostart On',
            'set Xdebug/xdebug.remote_log /vagrant/xdebug.log',
        ],
        require => Package['php'],
        notify  => Service['apache'],
    }

Last up, we install the php-pear package, which we probably need in some cases.

# Install PEAR

    package { "php-pear" :
        ensure => present,
        require => Package['php'],
    }

}

The last file is the symfony2.pp, where specific symfony2 things will be done.

First we need a vendor update. Normally you can do this always, which means provision takes a bit longer (since it needs to check all the bundles), but on some environments I’ve worked at, had such a poor internet line quality, that we got a lot of failed connections when the vendors tried to install. This is why this definition has a timeout and tries parameter, and will only be run when the /vagrant/vendor/twig directory does not exist (if it does, we assume it’s already installed properly).

# /support/puppet/modules/project/manifests/symfony2.pp

class project::symfony2 {

    # Install / Update the vendors

    exec { "vendorupdate" :
        command => "/usr/bin/php /vagrant/bin/vendors install && /usr/bin/php /vagrant/bin/vendors update",
        creates => "/vagrant/vendor/twig",
        require => [ Package["php"], Package["git"] ],
        timeout => 0,
        tries   => 10,
    }

The parameters.ini should hold all your personal configuration settings. Best practice is not to add this to your repository but create a parameters.ini-dist file which you can setup. However, if you didn’t do this, we just create a ini file from the dist file. If it’s already present, we don’t touch it. This must be done BEFORE the vendorupdate, since symfony2 relies on the parameters.ini to be set correctly.

# Setup parameters.ini if it does not exist yet

    file { "parameters.ini" :
        path => "/vagrant/app/config/parameters.ini",
        source => '/vagrant/app/config/parameters.ini-dist',
        replace => "no",                    # Don't update when file is present

        before  => Exec["vendorupdate"],
    }

The last two things we do are initializing the database and populating is. We initialize it by issuing a doctrine:schema:create call. We have added an “|| true” in case something goes wrong. In that case, we just want to continue as nothing strange has happens and this will always make sure it will return a correct return code. Notice that we have three dependencies here (which is a lot, but still, we need them all). This will only be run when the /tmp/.sf2seeded file is not present.

the last block will be seeding the database. In our case, we seed directly into MySQL, but you could also use symfony2 fixtures for this. Make sure that at the end you create a /tmp/.sf2seeded file, so it will not run again when you want to provision again. off course, this can only be run AFTER the initialization of the database.

# Create our initial db

    exec { "init_db" :
        command => "/usr/bin/php /vagrant/app/console doctrine:schema:create || true",
        creates => "/tmp/.sf2seeded",
        require => [ Exec["vendorupdate"], Service["mysql"], Package["php-xml"] ],
    }

    exec { "seed_db" :
        command => "cat /vagrant/doc/db/seed_data.sql | mysql -u${params::dbuser} -p${params::dbpass} ${params::dbname} && touch /tmp/.sf2seeded",
        creates => "/tmp/.sf2seeded",
        require => Exec["init_db"],

    }
}

The actual code can be found on github.

Tips:

• Don’t use NFS. It won’t work with windows hosts.
• Try to use a 32bit basebox to make sure everybody can run vagrant.
• VboxFS cannot do permissions. You cannot set permissions or file ownership on the /vagrant directory.