Three people in particular have contributed awesomely to making Merlin better for distributed environments. The first to pick it up for this purpose is a guy named Russel Jennings. He's written several concise bug-reports and done tireless testing with various versions to find where some bug was introduced and which versions of the Merlin daemon work well together. A great big thanks for that, Russel!

The other person is a guy named Sean Millichamp. He's gone the extra mile and has started sending in patches for bugs he's found. So far, he's contributed with 10 patches, making him the second most prominent developer for the merlin module-daemon pair. So far, his patches hold very high standard indeed and I have great hopes that we'll see more of his excellent contributions making it into future releases.

Jean-Marc Le Fevre has also contributed some minor patches that he deserves recognition for.

Numerous other people have reported issues with Merlin and contributed to the Wiki and HOWTO's. Thank you all

So I've implemented the state retention stuff in Merlin. Turns out that all that was really required was to make sure the status and object import works ok and is up-to-date (so I implemented an automagic way of making that happen). Then I can just read the current status from the database. I use an array sorted by object name ("host_name" for hosts and "host_name;service_description" for services) so I can use a binary search. 3000000 lookups of some randomly chosen nodes in a config with 15k hosts complete in just under 2.2 seconds. Quite impressive. Especially so when about 0.8 seconds is spent loading all the states and sorting the array in the first place.

I've also had a chance to look over the cross-host event transport stuff, which was subtly broken due to a brainfart of mine in a tertiary operation. I've tested it and it works just fine now agan.

With the changes mentioned above, I Merlin is rapidly approaching production quality in terms of its planned feature-set, so I've just released v0.5 with the hopes of attracting some more testers.

Let it rip, people, and make sure to let me know how it's going. Merlin can be downloaded from our git repositories using the following command:

  git clone git://git.op5.org/nagios/merlin.git

Cheerios for now.

However, this post is mostly about one particular patch from a guy named Jean Gabès. The patch speeds up Nagios' circular-parent-child dependency checks a *lot*. In a network 300 levels deep (root-host -> lvl1-child -> lvl2-child -> ... -> lvl300-child) where each level in itself has 500 hosts, vanilla Nagios had to be Ctrl-C'd out of after 53 minutes, while Nagios with Jean's patch completed in less than seven seconds (a speedup of more than 51000%!).

For a more modest network of 15000 nodes, (30 levels deep, 500 hosts in each level), vanilla Nagios completed a configuration check in 3 minutes and 33 seconds, while patched Nagios did the exact same job in less than one second.

Awesome, Jean. Thanks a lot indeed :-)

Shawn has been busy quite a long time (understandable, as he still makes significant contributions both to git.git and jgit.git), and only me and Ramsay Jones have made any significant contributions to libgit2 since it was announced back in November 2008.

Hopefully, this will mean whatever patches there are will get applied faster and that development will move forward at a quicker pace.

Personally, I've got a lot of un-committed work lying around that pertains to index reading and writing. I'll be working on finishing that up and sending it out to git@vger for review. The sooner we can get *some* part of the library working properly the better, imo, as git.git can't start using it until it does.

Oh, and libgit2 is currently looking for additional contributors. It doesn't really matter what you want to work on. Just clone the repo from git://repo.or.cz/libgit2.git and start hacking. Patches could go either to me (ae@op5.com) or to the git mailing list (git@vger.kernel.org).

• The fork was instigated largely by german members of the community. It appears to have been spearheaded by a german company (though I don't know this for sure) that makes its living selling customized Nagios solutions and/or support. I don't know this for sure, but it sure looks as if that's what's happened.
• The german company have unlawfully used the Nagios trademark after being asked not to do so. It has also registered Nagios as a trademark in Germany, to which is a huge slap in the face of an opensource project. They are naturally not on the best of terms with Nagios' founding father, Ethan, at the moment.
• Ethan has been absent working with the aforementioned lawsuit (or whatever it is a trademark violation results in when friendly talk is no longer enough), and also trying to put together a new webbased user interface for Nagios.
• Patches from all levels of the community have been erratically ignored during Ethan's absence. Some were picked up, but as many or more slipped between the cracks.
• Ethan has always been the single person with commit access to the Nagios CVS (yuk) repository.
• The fork uses git to track their patches.

The community developers have voiced a complaint that they cite as the primary reason for the fork:

Nagios is not being developed fast and openly enough.

I agree with this, and I'm currently discussing with Ethan about expanding the developer-base. Unfortunately, the scarce resource "trust" is even scarcer for those developers who joined the fork, which leaves the available candidates rather few. Happily, I count myself among them, and apparently so does Ethan. He emailed me away from public channels asking if I'd be willing to become a core developer, and op5 has graciously given a tentative promise to devote one to two days per week to Nagios development / patch management. Nothing's settled yet, but development has to continue even if the core maintainer takes a leave of absence, so one way or another, we'll make sure this happens.


In a perfect world (ie, one where I get to decide everything ), here's what will happen:

• Nagios incorporates the good changes that the fork produces.
• The benevolent but previously frustrated developers from the fork hop back to working on Nagios when they see it's once again moving forward. They could actually do that by keeping on working on their fork, although that would set them apart from the Nagios community a bit rather than make them members of it.
• Nagios development picks up its pace and a new GUI is added to it which fulfills everyones wildest dreams.
• Nagios development moves to using git instead of CVS. Since git actually invites people to fork the code but makes it incredibly simple to merge those changes back to the pre-fork project again, there could be any number of forks and Nagios would be the grand total of the best of all of them. Who would win on that? Well, the Nagios users for a start, and Nagios itself, and Ethan, and every company making a living off of Nagios one way or another. So that'd be a win-win-win-win-win situation? I like it.

For those who wonder where I'm standing in all this, I'll be working with Ethan to make the community developers happy while at the same time trying to prevent the community users from living through the confusion that a long-lived fork means. In the end, I hope Nagios becomes a better product with a stronger and better community backing it, which seems rather inevitable now that more people than ever are working frantically at making it so. Hopefully it results in a happy community where The Right People(tm) are part of a Nagios steering committee or some such.


Time will tell. It always does ;-)

What with me being the company's die-hard C programmer, I'm naturally taking care of finishing off the Merlin module.

As some of you know, the merlin module was originally designed to be an event transport for effortless redundant and loadbalanced network monitoring. Since modules running inside Nagios have certain restrictions put upon them, we decided to empower the Merlin module with the capabilities to insert events into a database (a rather straightforward patch). The really cool part about it is that Merlin still retains its multiplexing networking capabilities, which means that you can now use Merlin as a (very, very fast) way of communicating Nagios events to other servers.

Since merlin is designed to work with a plethora of different topologies, this means that Nagios will be the easily most scalable network monitoring system of them all. If you want to monitor Google's server-park from a single tool, you'll have to use Nagios. If you want to monitor Second Life's vast and widespread server network, Nagios is the only choice. If you want to monitor the entire internet, Nagios can do that (provided you spend "some" money on hardware ;-))

If you're a handy guy when it comes to doing certificate authentication in C, I might have a job for you though. Currently all nodes have to be configured upstream in its chain of responsibility. The capability to add random servers without modifying the configuration of running servers would be even more awesome

A cross-site request forgery means that one site includes a <form> tag with an "action" value pointing to a different site. The idea is to utilize a user's already valid session with a site requiring authentication to submit forms to that site that the user didn't intend to submit.

For Nagios, the scenario looks like this:
1. Random Nagios Admin (RNA) logs in to nagios, supplying valid credentials.
2. RNA goes to evilsite.com, where some lurid java-script checks his browsers history and notices that RNA has a Nagios installation by looking at the previously browsed pages.
3. evilsite.com creates a form which, using hidden variables, submits a command to the Nagios site where RNA is an admin.
4. Since RNA is authenticated with valid credentials, the command is accepted and Nagios loads it as if RNA had submitted it himself (which, for all that cmd.cgi on the nagios server knows, he/she has).

With Nagios 2, the worst that could happen is that evilsite.com disables monitoring of the network, or submits any of the other commands that Nagios accepts (invalid commands are simply discarded by the Nagios core).

The remedy to this is a patch that I wrote, which I hope will go into Nagios 3.0.6, to be released Any Day Now(tm).

The fix I wrote works like this:
1. When RNA wants to submit a command, he/she is sent to the command submission page (the one with the 'commit' button).
2. The command submission page generates a random token that gets included as a hidden variable in the form. The session data (apart from the random numbers) is also written to disk.
3. When the 'commit' button is pressed, the session token is looked up and cmd.cgi makes sure the session is valid (ie, belongs to the right user and is less than 15 minutes ols). If there is no valid session token, command submission fails and the user is told so.

What really kills the ability for off-site forms to circumvent this is the fact that the session token gets written to disk. Even if someone manages to guess the pseudo-random SHA1 session token (which is 2^160 to 1 against) they still can't make that session valid by writing it to the nagios-server's disk.

The CSRF issue is still in Nagios 3.0.5, but can no longer trigger execution of arbitrary programs by the Nagios process due to the changes made to prevent malicious exploitation of CVE-2008-5027. Its impact is thereby reduced to disabling monitoring of the network and similar actions that can validly be requested from the Nagios process through the GUI. Bad enough, but no longer a vulnerability that allows a remote attacker to run arbitrary programs on the one server in your network that can bypass every firewall one way or another.

I'm withholding the CVE details until Steven has had time to update the information with that contained in the above paragraph. In case I forget to update this blog-post, the CVE candidate id is CVE-2008-5028.

A fixed version of Nagios is available at http://www.op5.org/src/nagios-3.0.5p1.tar.gz. This fixed version is the base of op5 Monitor 4.0.1 which no longer suffers from the vulnerability discussed here.

The vulnerability can be proven like this:
A user without full privileges creates an off-site form to submit a comment to Nagios. In the custom webform, the comment_data field is altered to be a "textarea" rather than "text", so the user can put newlines in there (note that this can easily be done with browser addons too).

The evil user then creates the comment so that the textarea contains a newline, and lets the second line contain a completely different command. cmd.cgi only verifies that the user is allowed to submit the first command but sends the entire input to Nagios without checking it for newlines. Nagios reads its command-pipe line-by-line and has no way of picking up the username of the person that submitted the command, so it happily runs all the commands fed to it.

For Nagios 2, this wouldn't have been such a big deal. The evil user could stop Nagios entirely, which is ofcourse (very!) bad, but that's where it ends.

However, in Nagios 3, the ability to change checkcommands and their arguments was added. Authenticated users can exploit this vulnerability to cause the Nagios process to run arbitrary commands, such as emailing the Nagios configurations (with its accurate map of the network and whatever passwords are stored there) to themselves, or open up remote shell sessions originating from inside the firewall. Bad stuff indeed.

I wrote a couple of patches that completely fixes this. Those patches were included in Nagios 3.0.5 and op5 Monitor 4.0.1. All users are urged to upgrade as soon as possible.

This vulnerability has been assigned the candidate name CVE-2008-5027 by Steven M. Christey of Mitre. The CVE details are below.

======================================================
Name: CVE-2008-5027
Status: Candidate
URL: http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2008-5027
Reference: MLIST:[nagios-devel] 20081107 Security fixes completed
Reference: URL:http://sourceforge.net/mailarchive/forum.php?thread_name=4914396D.5010009%40op5.se&forum_name=nagios-devel
Reference: MLIST:[oss-security] 20081106 CVE request: Nagios (two issues)
Reference: URL:http://www.openwall.com/lists/oss-security/2008/11/06/2
Reference: MISC:http://www.nagios.org/development/history/nagios-3x.php
Reference: CONFIRM:http://www.op5.com/support/news/389-important-security-fix-available-for-op5-monitor
Reference: BID:32156
Reference: URL:http://www.securityfocus.com/bid/32156

The Nagios process in (1) Nagios before 3.0.5 and (2) op5 Monitor
before 4.0.1 allows remote authenticated users to bypass authorization
checks, and trigger execution of arbitrary programs by this process,
via an (a) custom form or a (b) browser addon.