Creating MCollective clients in PHP - The hard way

If you haven’t heard of MCollective, think of.. The Borg.. Except without the laser-eye, or the spaceship-cube, or the scary voices. Come to think of it,.. it doesn’t really have anything to do with the Borg, except they are both a collective, and you are in charge.. just like the Borg-queen. And everything else is futile..

ActiveMQ

MCollective is a ruby framework written by R.I.Pienaar and is based on top of ActiveMQ (active message queue). This message queue system takes care of sending and receiving messages by subscribing to queues. If you send a message to a specific queue, every node connected to that queue gets that message and can respond to it. It’s also possible to send a message directly to a specific node as well. Think of a message queue system like a big IRC server with the nodes being the users, and the queues as the channels In fact, it’s probably easy enough to connect all your MCollective nodes to an IRC server, from which you can control all nodes (and you’ve got yourself a pretty little bot-net :)).

MCollective is a ruby framework to send out messages and receive replies. It controls what kind of messages are being send (and where to), and will parse the replies. MCollective itself can send and receive messages through the help of clients. For instance, there is a “ping” client, that can send out a ping-message to other nodes, and if they have a ping-client as well, they will return a ping reply message (a pong). So with that client, you’ve got a system to see the nodes that are alive in your collective.

There are many different clients available: there are clients that can run software on a node, that returns the process list of a node (so you get a “remote ps”  more or less), and it would be very easy to create a ftp client that will transfer files between nodes (although not very efficient and better ways, it would work).

So what you want to do with MCollective is all based on which clients you install. One of the commonly used purposes would be to start a puppet client remotely in real time (if you don’t know puppet, you should take a look at it!).

Filtering

MCollective itself can filter out to which nodes you want to send your messages to. For instance, you can let MCollective filter out messages so only nodes for a specific operating system will receive them, or systems that have at least 16GB of memory and 50% disk space in use, or any other fact or set of facts. Actually, the messages are send to all systems, but the receiving side decides if it will accept the message or not. This means the sending server doesn’t have to know anything about any other node connected to the system.

For this type of filtering MCollective makes use of an application called Facter. This is nothing more than a simple data-aggregator that collects system information. With Facter, it’s easy enough to add your own facts, for instance, a fact that returns “true” or “false” based on the fact if ZendServer is running on the server. Through this system, it becomes very (VERY) easy to apply actions to a specific set of nodes without knowing any hostnames. There are other ways of filtering out (through subcollectives, classes etc), but through facts is the most basic and most used way.

Ping all CentOS systems in your collective:

$mco ping -F operatingsystem=CentOS
puppetmaster.enrise.local                time=3709.46 ms

---- ping statistics ----
1 replies max: 3709.46 min: 3709.46 avg: 3709.46</pre>

Get the number of CentOS systems that have 2 or higher (v)CPU’s:

$mco facts -F operatingsystem=CentOS -F processorcount>1
puppetmaster.enrise.local

As you might have figured out by now, MCollective is served best with hundreds or thousands of nodes, but even implementing MCollective on 10 systems can make your administration life much easier. In the end, you don’t need to keep a list with hostnames / ip-addresses which simplifies your work.

With the help of the filtering and clients, the amount of power you can wield over a collective is massive. How cool is this scenario: you are able to restart all apache processes on CentOS 6.0 machines in specified timezone, with just a single command. Or maybe just you want to retrieve a list of all zombie-processes on ALL your systems (if any). Or maybe just get a list of the uptimes from those systems? Or which servers are running a mysql-server and have less then 8GB of memory? It’s all single commands to send out to the MCollective. Resistance truly is futile..

Writing your own client

If you want to write your own client, the best solution would be to write them in ruby, the language MCollective (and other cool stuff, except for rails) is written in. But because my main language nowadays is PHP, I’ll wanted to try it out in PHP. So we needed to figure out how MCollective (and ActiveMQ worked internally to get that up and running.

Stomp

MCollective uses ActiveMQ with the Stomp protocol. Stomp itself is supported in PHP through the help of a Pecl extension. There are a few bugs inside the extension that makes it unusable but this will hopefully be fixed in the near future. For now you can manually compile the extension with the help of this patchfile.

With this (patched) extension we can connect directly to the ActiveMQ and receive MCollective messages and send replies. (there is a problem still though, the proof-of-concept I’ve posted on github will only receive one message. Somehow a second message is not received. I still need to figure out of I’m doing something wrong (acking the message maybe?), or the stomp extension fails, or something else. It’s still work in progress :).

Marshal

So sending and receiving messages is not a problem through PHP. But consuming those messages actually is. MCollective can send it messages in two different formats: Marshal or YAML. YAML is a pretty well known markup-language, and Marshal is ruby’s internal serialize()/unserialize() system that stores data in a binary format. PHP has got a YAML parser (but haven’t tested it yet) but when you use YAML as a format for MCollective you must use SSL to encrypt your messages, which is a bit more of work to setup. The standard way (PSK, or Pre-shared key) is much easier to setup (out of the box), but cannot use YAML.

So our options are:

• Setup SSL and use YAML, which means you need to reconfigure the whole MCollective.
• Patch MCollective so it uses YAML with PSK.
• Leave MCollective as-is, and handle marshal messages ourself.

Of course, the last option is the hardest, so that’s the option I’ve chosen :)

So Marshal is ruby’s equivalent of serialize/unserialize is a binary format so we need to parse it manually. There is no complete format description on how the binary format is made up from, but the source code for the ruby marshal functionality is available. If you can read a bit of C, you’ll see it’s not really that complex. Half the file is about converting ruby data into marshal, and the other half is about converting it back.

Now, for the purpose of our proof of concept, we don’t really need to implement the full marshal specifications. I don’t even think that would work because of all the different types ruby has (and PHP hasn’t). So converting back and forth will lead to incorrect data.

MCollective likes to put most of it’s stuff in hashes and integers (bignums) which makes our live much easier. I’ve decided to just start parsing a marshal message and see which types I needed to support. Turns out that for my purpose, strings, hashes and bignums where the only ones used.

The biggest problem on the marshall conversion is the integer conversions. The mechanism to convert a long to binary format eludes me, and because it makes heavy use of signed numbers, I’m actually surprised if the actual code works properly for all numbers. I’ve made a little class around it so feel free to update and fix bugs :)

Now that we can “unmarshal” messages send through the ActiveMQ, it’s time to actually parse those messages. As it turns out, those messages are actually quite simple: a body containg a command (plus arguments), some meta-information like timestamps, destinations, filters (if needed) and a hash when dealing with PSK (which is nothing more than a MD5(body + psk).

So with that, it’s easy to create a simple proof-of-concept that connects to the ActiveMQ, subscribes to the correct queues, receives messages, parse them and return a message back if needed.

Our PHP client

So what we’ve done is creating a simple client that responds to “ping”. The only thing it needs to do is send out a response with the body “pong”. It’s actually that simple. So our client will connect to the ActiveMQ server and subscribes to the correct queues. After that it will loop while it waits for messages, checks if those messages have a valid hash signature, and if so, it will check the command. If it’s a “ping” command, it will reply with a pong, and wait for the next message. It’s that easy..

Ways of improving

If you want to create your own clients with PHP, you shouldn’t use this method. Either patch up MCollective to use YAML over PSK, or setup SSL. But even when using such a method requires to have a separate PHP daemon running besides MCollective, which is not really what you want to begin with. It would be nice to see a MCollective “blank” plugin that can execute a command (and thus, a PHP file) and send/receive the data through YAML or another format.

Such an client would not only be able to cope with PHP, but everything that can parse stdio (from bash, to C, to Erlang, to Haskell, and everything else in between). It would make MCollective truly language-agnostic and still keep all the power of mcollective like the filtering, message-sending etc..

I’m not sure if such plugin is available (otherwise, it will be soon), but in the meantime, just play with the PHP one:

https://github.com/jaytaph/MCollectivePHP

Conclusion

I hope I don’t have to explain to you that using this in any production environment is the equivalent of lighting a handful of firecrackers and sticking them in your mouth. You just don’t do it :). But isn’t it fun to find the hardest way to get things done, and actually achieve your goal? :) There are other, better and easier ways to connect to MCollective but of course, they are less fun to implement. And isn’t having fun in what you are doing the most important factor anyway?