Igor Minar's Blog

Change of Status

2010-08-08T23:10:00.000-07:00

$ sqlplus -s
SQL> connect hr@oracle.com/hr
SQL> UPDATE employees SET current = false WHERE email = "Igor.Minar@oracle.com";
SQL> COMMIT;
SQL> disconnect
SQL> exit

$ curl -X POST -H "Content-Type: application/json" \
   -d '{ "firstName":"Igor", "lastName":"Minar"}' \
   http://google.com/employee/

Thanks for All the Fish

2010-08-03T15:48:00.000-07:00

Hi guys,

Most of you don't know, but today is the 3rd birthday of wikis.sun.com. In 2007 a bunch of us decided that it was worth it to boldly go where no man has gone before and on August 3, 2007 we launched wikis.sun.com.

At that time very few corporations were actively using some kind of wiki internally and there was no known significant public wiki deployment run by a corporation. We were astonished to see the uptake and user interest and watched the project grow from a few power users and a few dozens of wiki pages to tens of thousands of users and tens of thousands of wiki pages.

Thank you all for contributing, providing feedback and helping us to make the project successful.

Despite being a small team (did you know that officially there wasn't a single person working on wikis full-time?), we managed to get a lot done and I'm very proud of our accomplishments.

I do believe that there is still room for improvements, but I made a decision that these improvements will have to be implemented by someone else. I have found a new challenge that I'm going to pursue and unfortunately it's time for me to hand the wikis project over to a new group that will oversee the operations and development of the site. I did my best to make this transition as smooth as possible and I'm hopeful that the site will be in good hands.

Just by chance my last day at Oracle coincides with wikis' 3rd birth day, I'll take that as a good sign. I wish you all the best and I hope to see you around. Internet is a small place.

Good luck to you all.

Cheers,
Igor

PS: If you want to stay in touch, you can find me at linkedin at: http://www.linkedin.com/in/igorminar

DGC VI: Wiki Organization and Working with the Community

2010-08-03T01:53:00.000-07:00

This blog post is part of the DevOps Guide to Confluence series. In this chapter of the guide, we’ll have a look at wiki organization and working with the user community. This post is going to be more subjective than the others, because the recommendation I'm going to make apply to a wiki site with similar goals and purpose as ours. I'm just going to share our experience and hopefully some of it will be useful for others.

The Purpose

First thing that should be clear for you when building a wiki site is what is the purpose that it's going to serve. Confluence has been successfully used for many purposes ranging from team collaboration, documentation writing, to website CMS system just to mention a few. When our team set out to build a wiki site, the goal was to create a wiki platform that could be used by anyone in our company to publicly collaborate with external parties without having to deploy and maintain their own wiki.

It was a pleasant surprise when one of the first groups of users who joined our pilot three years ago were technical writers eager to drop their heavy-weight tools with lots of fancy features in exchange for lightweight and more importantly inclusive collaboration tool. The main issue they were facing was that their processes and tools were very exclusive, and next to impossible for a non-writer to quickly join in order to make small edits. This resulted in lots of proxying of engineering feedback, and inevitable delays. With a wiki, the barrier for entry is very low for almost everyone. There is nothing to install or configure, a browser is all one needs. A wiki allowed a relatively small and overloaded team of technical writers to more efficiently gather and more importantly incorporate feedback from subject matter experts into the documentation. Of course there were trade-offs, mainly in the area of post processing the content for printable documentation (i.e. generating PDFs), but I'm hopeful that as the wiki system matures, more attention will be paid to make this area stronger (Atlassian: hint hint).

Anyway, with the tech writers on board, the purpose, goals and evolution of our site got heavily influenced by their feedback. In exchange we received a lot of high quality content that attracted new users who started using the wiki. This kind of bootstrap of the site greatly helped to speed up the viral adoption across our thirty-thousand-employee company.

Wiki Organization

When we launched our site three years ago, there were no other big corporations with a public facing wiki site (many corporations didn't even have an internal wiki yet, boy that has all changed since then), this put us into a position where we had to be the first explorers in search of best practices as well as things that didn't work at all.

Fortunately, since our team successfully pioneered the area of corporate blogging before the wikis launch, we had some experience with building communities that we could leverage.

Some of the main principles that we reused from our blogs site were:

Make the rules and policies as simple as possible
It is a goal shared by all employees to create a good image of the company and make the company succesful. We should trust their judgement and empower them to be able to do the right thing.
The team running the site is small, so the employees should be able to do as much as possible on their own (self-provisioning FTW!)
Since we trust our employees, we should delegate as much decision making and as many responsibilities as possible, and let them delegate some to others, otherwise we won't be able to scale.
There should be very little (close to none) policing or content organization done by the core team. We don't have the man-power for that. Besides, the Internet is not being policed by anyone and things tend to just work out. The popular, well organized and valuable content bubbles up, in one way or another.

Implemented Actions

With our principles laid out, we took these actions:

We integrated Confluence with our single sign-on and user provisioning system, which made it super easy for employees and external users to log in using their existing accounts.
Based on the information in our identity systems, we enrolled accounts of all of our employees into an employee-specific Confluence group, which we utilized when setting up global permissions.
The global permissions were set up so that employees (and only employees) could create new wiki spaces on their own, whenever they had the need, for whatever purpose
We also opened up all wiki spaces to be viewable by anyone on the Internet, but we left it up to the space admins to restrict permissions if they felt like it was necessary.
In order to mitigate spam issues, we made it impossible for anonymous users to obtain any write permissions either on the global or space level
We created a single Confluence theme that was applied to all wiki spaces by default and disabled all the other themes. This was done mainly for technical reasons — the Confluence UI has been changing dramatically over the last few years and these changes often resulted in a need to modify a custom theme to make it compatible with these changes. If we allowed anyone to create their own theme, we'd never be able to upgrade, because of a fear that we'd break someone's theme or alternatively we would have to coordinate our updates with all the maintainers of custom themes
We created an internal mailing list where space admins and other wiki users could share their experience, ask questions, and report issues.

Things We Need to Work on

Nobody's perfect and neither are we, let's look at what could we improve.

I know I just said that popular content always bubbles up, but considering how hideous the default Confluence front page is, I'd much prefer to utilize that real estate better and highlight popular or interesting content there.

I also think that we could do a better job at highlighting hardworking community members. There were some elaborate attempts to do this, but in my opinion a more lightweight approach could be more suitable for most of the sites.

Lastly, I think that staying in touch with our community is very important, and we could have done a better job at it if we had e.g. quarterly internal mini-conferences on various topics during which we could better gather their feedback. Also some better organized training sessions for our novice users could help boost our growth even further.

Conclusion

The recommendations and practices that worked for us might not be suitable for all Confluence deployments, but in our case things have worked out. There are still many areas where we could have done a better job, but I guess it's good to always have some space for improvements.

In the next chapter of my guide, we'll discuss issues and solutions that are specific for Internet-facing Confluence deployments.

DGC V: Customizing and Patching Confluence

2010-08-02T07:29:00.000-07:00

This blog post is part of the DevOps Guide to Confluence series. In this chapter of the guide, we’ll have a look at how to customize and patch Confluence.

Customizing Confluence

Before we talk about any customization at all, I need to warn you. Any kind of customization of Confluence (or any other software) comes with a maintenance and support cost. The problems usually arise during or after a Confluence upgrade, and if they catch you unprepared, you might get yourself in a lot of trouble. Keep this in mind and before you customize anything, justify your intent.

There are several ways how to customize Confluence. For some the maintenance and support cost is low, others give you lots of flexibility at a higher cost. So depending on your needs and requirements you can pick one of the following.

Confluence User Macros

I already mentioned these in the Confluence Configuration chapter — they are easy to create and usually don't break during upgrades, but they are a nightmare to maintain. Avoid them.

Confluence Themes, HTML Headers and Footers

You can easily inject html code in the header and footer by editing the appropriate sections of the Admin UI (described in the config chapter). If this html code contains visual elements, then it's possible that your code will break during upgrades. In general I would avoid editing headers and footers in this way as much as I could unless I was doing something very simple.

Confluence themes are the way to go. You can either pick a theme that was already built and published by someone else, or you can build our own. Building your own theme will give you the most flexibility, but the cost of maintaining and supporting it will be the highest. You can do some things to cut corners, but be prepared to do some Confluence plugin development (a Confluence theme, is really just a type of Confluence plugin).

What worked well for me and our heavily customized theme, is to create our theme as a patch for the Confluence default theme. I simply symlink all the relevant files from Confluence source code into a directory structure that can be built as a Confluence theme/plugin, add my atlassian-plugin.xml and patch the files with changes I need no matter how complex they are. The advantage of this approach is that my theme will always be compatible with my Confluence version (after rebase) and I get all the new features introduced in the new version. The downside is that I often need to rebase my patches during Confluence upgrades, but with a good patch management solution (see below) this headache can be greatly minimized.

Lastly there is Theme Builder from Adaptavist. I haven't personally used this Confluence plugin because it was not popular when we initially created our theme and it was not desirable for us to depend on yet another (unknown at that time) vendor during our Confluence upgrades. If I were about to start creating a theme from scratch I would compare it with my patching method and see what gives me the most benefits. The main concern with Theme Builder I have, is my ability to version control the theme, which if not easily possible might be the deal breaker for me and many others.

Confluence Plugins

I mentioned Confluence Plugins already in the previous chapter, so I'm not going to repeat myself here.

What I'm going to add is that you really can extend and customize Confluence in crazy ways via the plugins. You can either discover the existing plugins at Atlassian Plugin Exchange or you can build your own with Maven (or the Plugin SDK), Java (or another Java compatible language) and Atlassian Plugin Framework.

The nice thing about plugins is that they are encapsulated pieces of code that interact with the rest of Confluence via public API and additionally they are hot plugable. This means that in theory they should work after a Confluence upgrade and that you can install and uninstall them on the fly without a need for restart. While the latter is true in practice, the former is not always the case. Confluence's public apis sometimes change, plugins rely on behavior that was not considered to be part of the public api and the UI changes all the time, so any CSS/javascript code that relies on absolute or relative positioning or fixed DOM structure will need ocassional fixes during upgrades.

Patching Confluence (source) files

Lastly I'm going to mention that one can modify Confluence's behavior by modifying the Confluence core files, this is a large topic and deserves its own section. ;-)

Patching Confluence

Patching Confluence is definitely the most advanced way to customize Confluence, especially if you start changing the Java source code, recompiling and creating your own war files. On the other hand, this way you get the most flexibility and will be able to change anything you want, even those things that plugins can't, all at your own risk.

Issues to Be Aware of

There are several potential issues that you should be aware of before you head down this route:

you might break something unintentionally — you can mitigate this with testing
you might have a hard time preserving your changes during an upgrade — you can minimize the problems by using good patch management strategy (discussed below)
you might have a problem getting support — this was never a problem for me mainly because most of my changes have been very isolated, so I could quickly tell if an issue is caused by my patch or if there is a bug in Confluence.

Reasons for Patching

The reasons for which you might want to patch Confluence fall generally into these four categories:

config change - as I mentioned in the Confluence Configuration chapter, some of the configuration is done by modifying files that are part of the Confluence standalone or war distribution (usually those in WEB-INF/classes directory). This is quite unfortunate because it adds a significant overhead to upgrades. I would much prefer if this configuration could be done via files in Confluence Home directory, but until that happens, the best way to manage these changes is by treating them as patches.
security fix - Atlassian often releases patches that fix security vulnerabilities in older versions of Confluence. What they actually release is a binary or textual file that represents the fixed version of the affected code. This file can then be just dropped into the appropriate location in (typically) WEB-INF/classes directory and the issue is fixed. This is a nice quick hack, but if your site is bigger or you plan to be on an older version for an extended period, your situation will be a lot more maintainable if you transform the fix into a patch against your version of Confluence.
temporary bug fix - occasionally Atlassian releases a temporary fix for an issue in a form similar to a security fix that will later on be properly fixed. In the meantime, the temporary fix can be used to work around the problem. Again, for a bigger site things will be a lot more maintainable if you manage this change as a patch.
a ui/behavior change - and lastly if you run a bigger site with lots of requirements coming from different groups of users, you might need to add a feature or disable an existing feature, add or remove a UI element, or change some behavior of Confluence in a way that is not possible via a plugin or a theme, in this case you definitively want to maintain every single such a change as an isolated patch. If you don't then you'll be in a big trouble when a time to upgrade Confluence comes.

Patching Methods

Now that we know why we would be interested in patching Confluence, let's look at how to do it. Again, there are several ways, depending on what do you need to patch.

patch the non-compilable files - these typically include config files as well as, javascript, css and template files. If you are patching only these types of files, then you can just create patches against the Confluence standalone or war distributions (since no compilation for these is needed). More likely than not once you get down the patching path, you'll want or need to do a lot more though.
patch files that require re-compilation - the Java source code. Soon after I realized that I needed to patch Confluence, I ended up modifying the java source code in order to fix bug or modify behavior. Atlassian makes this relatively easy to do, because along with the binary releases they also offer source code releases which can be used to build Confluence from sources on your own. This is an huge benefit for their customers, especially those who are willing to get their hands dirty to get the most out of Confluence. Once you have access to buildable source code, you can patch it and create your own builds with relatively small effort. The benefit of creating patches against the source release is that you can patch anything and everything (though I'm not saying that you should), starting from config files, js, css and template files all the way to core java class files; and all of that in a consistent and reliable way.

Patch Management

As I mentioned already, whenever you modify the source code you want to create an isolated patch that is a logical grouping of changes needed for one bug fix, config change or feature. Once you have many smaller patches like these you can apply or omit them in a build or update them one at a time when needed.

If you were to use the standard command line tools like diff and patch to work with these patches, you would probably go nuts quickly. There are far better, higher level solutions that can be used. Distributed source code management tools that are becoming an unstoppable force in the SCM arena of software development offer features that make patch management a piece of cake.

Git, for example, offer's a feature called Stash which allows you to create and maintain patches against your git repository. I don't have a personal experience with git-stash, but from the docs it looks like it should do what we want.

The solution that I've been using and loving for the past 3 years is Mercurial and it's core plugin Mercurial Queues. Working with this plugin is also well documented here and here.

Here are some main points for how I do my patching:

I store Confluence sources in my main Mercurial repository. I simply grab the source zip from Atlassian's website, unzip it, rename the root directory to "confluence" and put it to my repository.
When a new version of Confluence is released, I delete the confluence directory in the working copy of my repo and replace it with the files from the new zip file and commit the files with --addremove flag, which will automatically add all the new files and remove all the deleted files to/from the repository. This allows me to track diffs between Confluence versions, which is very handy when I'm debugging a new issue and want to find out in which release it was introduced.
In addition to this main repository I have a versioned (stored as a real hg repo) Mercurial Queue associated with it. The patch repo is very easy to create, just by issuing hg qinit -c command.
Now every time I want to change something, I create a new patch with hg qnew mypatchname.patch, modify the confluence source and then just do hg qrefresh to move my changes to mypatchaname.patch. This doesn't commit the changes in the patch repo, you have to do that explicitly via hg qcommit or by changing your current directory to .hg/patches and issuing a regular hg commit there.
Once you have patches in your queue, you can now easily apply and unapply patches with commands like hg qpush, hg qpop and hg qgoto.
Additionally you can set "guards" on patches, so you can create collections of patches that should be applied only for certain builds. For example, if you have some patches that should be applied only in development environment, you can set guards on them via hg qguard and then switch between these collections via hg qselect followed by hg qpop -a and hg qpush -a.
If you have a need to modify an existing patch, just hg qgoto to it, modify the confluence source code and run hg qrefresh and finallyhg qcommit.
In order to store binary files in your patches (e.g. images), you'll need to tell MQ to use Git's extended diff format. This is done by adding the following lines into your ~/.hgrc:
```
[defaults]
diff = --git
qrefresh = --git
email = --git
```
When upgrading Confluence, you first need to hg qpop all the patches, replace and commit the new confluence sources as discussed above and then reapply your patches with hg qpush -a. If you are lucky all changes will apply smoothly, however sooner or later, especially once your patch collection grows into decent size you'll need to rebase your patches. The conflicts are resolved via a 3-way merge, which can be either done by hand, or by using a fancy 3-way merge tool. Once all the patches are applied, be sure to test that everything still works as originally intended.
To make conflict resolution less frequent, I strive to create patches that do as little as possible to get things done. I avoid any major refactorings, api changes and "forget" about some best practices, especially in those cases when I know that Atlassian won't be interested in accepting my patch upstream. For these patches the main focus should be on getting things done, robustness and maintainability.
Lastly, if there is a patch that is generally useful for all Confluence users, I usually attach it to relevant RFE/bug report on Confluence's bug tracker. The fewer patches I have to maintain the better. When a patch is accepted upstream, I simply remove it with hg qrm.

I could go on and on about what a life-saver Mercurial Queues are but the best way to get to know them is to do some experimentation on your own. I strongly encourage you to do that, it's a good tool to have in your toolbox.

Just to give you some inspiration, here is a list of some of patches that I created for our build:

bundle jdbc driver with the war (by specifying new maven dependency in confluence's pom file)
configure Seraph login/security framework
replace Confluence's favicon with ours
modify the default log4j config
customize error pages
turn Australian English language pack into US English
remove all lower() function calls from Hibernate mapping files and Java classes to get major boost in db performance (see CONF-10030 and this doc)
disable mail archiving UI
enforce that our custom theme is the default and only available theme
remote api security enhancements
allow only members of our employee group to become space admins
as I already mentioned, our whole theme plugin is implemented as a bunch of patches against the default Confluence theme
and the list goes on and on...

Conclusion

In this chapter we went through several possible ways to customize Confluence. Plugins and themes are definitely the safest and most manageable way to go, however patching if done right, will give you the most flexibility. If you use the right tools for patch management (like Mercurial Queues), you'll be able to manage big collections of patches with a very little maintenance overhead.

Next time we'll have a look at a non-technical aspect of running a large Confluence wiki site.

DGC IV: Confluence Upgrades

2010-07-30T21:40:00.000-07:00

This blog post is part of the DevOps Guide to Confluence series. In this chapter of the guide, we’ll have a look at Confluence upgrades.

Confluence Release History and Track Record

I started using Confluence at around version 2.4.4 (released March 2007). A lot has changed since then, mostly for better. In my early days, Atlassian was spitting out one release after another — typically 3 weeks or less apart — followed by a major release every 3 months. You can check out the full release history on their wiki.

This changed later on and recently there have been fewer minor releases and bigger major releases delivered 3.5-4 months. Depending on your point of view this is good or bad. It now takes longer to get awaited features and fixes, but on the other hand the releases are more solid and better tested.

For major releases, Atlassian now usually offers Early Access Program, which gives you access to milestone builds so that you can see and mold the new stuff before it ships.

Contrary to the past, the minor versions have been very stable lately and have contained only bugfixes, so it is generally safe to upgrade without a lot of hesitation.

The same can't be said about major releases. Even though the stability of x.y.0 releases has been dramatically improving lately, I still consider it risky for a big site to upgrade soon after a major release is announced. Wait for the first bugfix release (x.y.1), monitor the bug tracker, knowledge base and forums, and then consider the upgrade.

Having gone through many upgrades myself, I think that it is a good practice to stay up to date with your Confluence site. We have usually been at most one major version behind and frequently on the latest version, but as I mentioned avoiding the x.y.0 releases. This has been working well for us.

Staying in Touch and Getting Support

In order to know what's going on with Confluence releases, it is a good idea to subscribe to the Confluence Announcements mailing list. This is a very low traffic mailing list used for release and security announcements only.

Atlassian's tech writers usually do a good job at creating informative release notes, upgrade notes and security advisories, so be sure to read those for each release (even if you are skipping some).

There are several other channels through which people working on Confluence (plugin) development can communicate and support each other, these include:

official Confluence Development forum
official Atlassian Development IRC
community Confluence Development Skype chat - a place where some of us hang out and discuss issues or share Confluence related news.

Despite Atlassian's claims about their legendary support, I found the official support channel rarely useful. Being a DIY guy and having a reasonable knowledge about Confluence internals, I usually found myself in need of a more qualified support than what the support channel was created for. For this reason my occasional support tickets usually ended up being escalated to the development team, instead of handled by the support team.

On the other hand the public issue tracker has been an invaluable source of information and a great communication tool. I wish that more of my bug reports had been addressed, but for the most part I have been receiving reasonable amount of attention even though sometimes I had to request escalation to have someone look at and fix issues that were critical for us.

The biggest hurdle I've been experiencing with bug fixes and support was that sites of our size are not the main focus for Atlassian and they are not hesitant to be open about it. I often shake my head when I see features of little value (for us that is - because they target small deployments and have little to do with core wiki functionality) being implemented and promoted, but major architectural issues, bugs and highly anticipated features go without attention for years. Just browser the issue tracker and you'll get the idea.

Confluence Upgrades

The core of the upgrade procedure will depend on the build distribution type you use (standalone, war, building from source), but fundamentally in all cases, you need to shut down your Confluence, replace your app (standalone or war) with the new version and then start it again. An automated upgrade process will take care of updating the database schema, rebuilding the search index and other tasks required for a successful upgrade.

That was the good news, the bad news is that there is a lot more work to be done in order to successfully upgrade a site with as little downtime as possible.

Dev and Test Deployments and Testing

Before you upgrade the real thing, you should at first get familiar with the release by upgrading your dev and test environments.

It's often handy to invite your users to do a brief UAT (user acceptance testing) on your test instance as they might catch something that you or your automated tests haven't.

Picking the Outage Window

Based on your users' usage patterns (as easily identified by web analytics solutions like Google Analytics), you should pick a time when the usage is low. For our global site this has been early mornings at around 4:30 or 5am PT.

When it comes to picking a day, we usually stuck with Tuesdays, Wednesday or Thursdays. Nobody wants to be dealing with an issue during a weekend when internal (infrastructure) or external (Atlassian) support is harder to get hold of.

You also want to communicate the planned outage to your users, so that they are not caught by surprise when you announce an outage on a day when they are releasing important documents on the wiki.

As far as outage duration goes, we usually plan for a 30min outage during a 1 hour window and most of the time have been able to bring the site back online within 30min or less.

Ready, Set, Go!

The actual deployment consists of several steps, which in our case are:

disabling load balancing for both nodes (which automatically triggers redirection of all requests to a maintenance pages hosted elsewhere)
shutting down both nodes
disabling MySQL replication between the master and slave db
taking ZFS snapshot of the Confluence Home directory
taking ZFS snapshot of the MySQL db filesystem on the master
deploying the new war file
starting one node (while the loadbalancer still ignores it)
watching container and Confluence logs for any signs of problems

At this point, we have one of our nodes up and running (hopefully :-)). We can log in with an admin account and check if everything works as expected. The next tasks include:

upgrading installed plugins
upgrading custom theme (if there is one)
running a bunch of automated or manual tests, just to verify that everything is ok

If things are looking good, we can allow the load balancer to start sending requests to our upgraded node. Continue watching logs and eventually deploy the war on the second node and re-enable the MySQL replication.

If any issues occur during the deployment, we can simply:

shut down the upgraded node
revert to the latest Confluence Home snapshot
revert to the latest MySQL db snapshot
redeploy the older version of war file
either retry the deployment or re-enable load balancer and deal work on resolving the issues outside of production environment

In my experience from all the dev, test and prod deployments, we've had to roll back and redo an upgrade from scratch only once or twice. It's very unlikely that you'll have to do it, but it's better to be ready than sorry.

If you are building Confluence from patched sources and deploy your own builds frequently, then you might want to consider automating your deployments with tools like Capistrano. This will save you a lot of time and make the deployments more reliable and consistent.

Conclusion

If you do your homework, Confluence is quite easy to upgrade. It's unfortunate that the entire cluster must be shut down for an upgrade even between minor releases, but if you plan your deployment well, you will be able to minimize the downtime to just a few minutes outside of peak hours.

In the next chapter of this guide, we'll take a look at customizing and patching Confluence.

DGC III: Confluence Configuration and Tuning

2010-07-30T08:12:00.000-07:00

This blog post is part of the DevOps Guide to Confluence series. In this chapter of the guide, we’ll have a look at Confluence configuration and tuning.

There are four ways how one can modify Confluence's runtime behavior:

Config Files in Confluence Home directory
Config Files in WEB-INF/classes
JVM Options
Admin UI

Config Files in Confluence Home directory

Confluence Home directory contains one or more config files that control runtime behavior of Confluence. The most important file is confluence.cfg.xml that must be present in order for Confluence to start. This file can be modified by hand while confluence is shut down, but also gets modified by Confluence occasionally (mostly during upgrades). Your changes will be preserved, as long as you made them while Confluence was offline.

Another relevant file is tangosol-coherence-override.xml which must unfortunately be used to override Confluence’s lame multicast configuration needed for cluster configuration (see below).

Lastly there is config/confluence-coherence-cache-config-clustered.xml which contains configuration of the Confluence cache. Generally you don't want to modify this file by hand. I’ll come back to talk about cache configuration later in the Admin UI section of this chapter.

In general it is advisable to be very consistent about your environment, so that you can then just have a single version of these files that you can distribute on all servers when needed. This includes the directory layout, network interface names, and so on.

A combination of the first two files will allow you to configure the following:

Clustering

As I mentioned, this configuration is split between two config files. confluence.cfg.xml contains confluence.cluster.* properties, which allow you to set multicast IP, interface and TTL, but not the port. Only tangosol-coherence-override.xml can do that.

The cluster IP is by default derived from a "cluster name" specified via the Admin UI or installation wizard. For some reason Atlassian believes that in an enterprise environment one can just let a software pick a random IP and port to run multicast on. I don’t know about any serious datacenter where things work this way. You’ll likely want to explicitly set IP, port, interface name and TTL and the only way to do that is by modifying these files by hand and ignoring the "cluster name" setting in the UI. Make sure that settings are consistent in both files.

DB Connection Pool

Confluence comes with an embedded connection pool. I believe that you can use your own too (if it comes with your servlet container), but I’d suggest sticking with the embedded one since it is widely used and Atlassian runs their tests with it also. The pool is configured via confluence.cfg.xml and its hibernate.c3p0.* properties. The most important property is pool max_size which will prevent the pool from opening more than a defined number of connections at a time. You want this number to be higher than your typical peak concurrent request count (are you monitoring that?), but not higher than what your db can handle. We have ours set to 300, which is double of our occasional peaks. Don’t forget that in order to take advantage of these connections, you’ll likely need to also increase the worker thread count in your servlet container.

DB Connection

The connection is configured via hibernate.connection.* properties in confluence.cfg.xml. Depending on your db, you might need to specify several settings for the connection to work well and grok UTF-8. For our MySQL db, we need to set the connection url to something like

jdbc:mysql://server:3306/wikisdb?autoReconnect=true&useUnicode=true&characterEncoding=utf8

Note that if you are editing this file by hand, you must escape illegal xml characters. More info about db connection can be found in the Confluence documentation.

Config Files in `WEB-INF/classes`

Just a side note: if you are building confluence from source then these files can be found at confluence/confluence-project/conf-webapp/src/main/resources/.

These files are the most cumbersome to work with because you need to apply your changes to them after each upgrade. I'll describe how we use our automated patching machinery to do this in the future chapter of this guide. For now let's just go over the available config files and what you can change here.

atlassian-user.xml - used to configure user provisioning, e.g. LDAP. For more info read the docs.

confluence-init.properties - this file allows you to specify the path to Confluence Home directory. There is a better way to set this; see the JVM Options section below.

log4j.properties - modify logging preferences, this can also be done via the UI, but AFAIK the changes are not preserved after restart or upgrade.

seraph-config.xml - controls authentication framework. You'll likely need to modify this file if you have a custom authenticator and login page.

I should note that there are many other (usually xml) configuration files bundled with individual jars in WEB-INF/lib, but those rarely need to be modified.

JVM Options

Another way to configure certain settings is via JVM options. From the complete list of recognized options these are the ones we use:

-Dcom.atlassian.user.experimentalMapping=true - this is a critically important setting for us with 180k users. Without it, our cluster panics due to data overload (CONF-12319), unfortunately despite Atlassian’s claims that this experimental feature is production ready, it got broken soon after release, and then again recently, so you’ll have to patch atlassian-user module to get it to work.

-Dconfluence.disable.peopledirectory.anonymous=true - for big public deployments the people directory is a privacy risk and generally useless for anonymous users, we have it disabled for anonymous users.

-Dconfluence.disable.mailpolling=true - early on we decided that we don’t want people to build up mail archives on our site. While the feature is useful for small internal wikis, it’s too much of a risk with little reward to provide it on a public wiki. Unfortunately, this option only disables mail fetching. The UI for setting up mail archives will still be present in the wiki; you'll have to patch Confluence to remove it.

I didn't learn about -Dconfluence.home until recently. I would much prefer to use it than to mess with confluence-init.properties file in WEB-INF/classes.

Admin UI

Most of the Confluence settings can be configured via Confluence admin interface. The downside is that the configuration is not being versioned, and there is no easy way see diffs and to roll back unless you want to hack the db and replace data from backups. With that in mind lets look at the most important settings.

General Configuration

Server Base Url - make sure this is set up correctly, otherwise confluence and its plugins won’t work properly.

Users see Rich Text Editor by default - we have this set to off. In the past many RTE bugs were causing headaches to our writers especially those who did lots of editing. In Confluence 3.2 and 3.3 the editor has improved a lot and it might be the time for us to reconsider this decision.

CamelCase Links - this used to be one of THE wiki features in general a few years ago, but as wikis have matured and people started creating more and more content, the automatic linking started to cause more problems than help. We have it off.

Threaded Comments - very useful; make sure it’s on.

Remote API (XML-RPC & SOAP) - we have ours on, but I patched the remote api code to restrict access to it.

Compress HTTP Responses - OMG please turn this on if is isn't already. It’s a major performance booster. Alternatively you might want to do the compression in your webserver as Tim pointed out in comments below.

JavaScript served in header - we have this on, but for better performance it should be off. Unfortunately that breaks many plugins and legacy code that uses obtrusive javascript. Since this option has been around for a while, it might be worth it to just set it to off and deal with the remaining broken things as they are identified.

User email visibility - we have this set to visible to admins only, but our power users found it too be a collaboration barrier so I patched the code and made emails visible to our global employees group in addition to the admin group. It would be nice if confluence allowed such a configuration out the of box.

Anonymous Access to Remote API - No sane person will leave this on. If I were in charge, I would go as far as removing it from Confluence product.

Anti XSS Mode - This is a very handy feature. Not 100% bulletproof, but it helped to significantly decrease the number of XSS exploits in Confluence since its introduction.

Attachment Maximum Size (B) - I mentioned this one already in the first chapter when discussing the db configuration. If you are running a cluster (or think that you will eventually run it), set this to some low value. Ours is 5MB.

Connection Timeouts - these options are pretty handy when you have lots of feed macros, gadgets and other plugins that pull contet from remote sites. In order to prevent worker thread pileup in your servlet container don’t go beyond the default 10sec (which is already pretty high).

Daily Backup Administration

As I previously mentioned, this backup feature is useless for anything but tiny sites. Disable it.

Manage Referrers

Collecting referrers is ok, but don’t display them publicly if you run a site on the Internet. Otherwise you run a risk of exposing some internal only URIs that might contain confidential information.

Languages

Most of our documentation and content is written in American English, but unfortunately Atlassian doesn’t provide such a language pack. I just patch the default Australian English pack to get a US English pack. It works great and is almost no hassle to maintain.

User macros

I discourage their use in enterprise environement. The lack of versioning, automated testing and documentation makes them a nightmare to maintain. Just create Confluence plugins for everything you need.

PDF Export Language Support

This is a tricky one. It took us quite a while to find the right single font that could be used to generate PDFs in almost all languages. Finally we found soui_zhs.ttf, which is distributed with OpenOffice. It’s a huge file, but it works like charm for all kinds of non-wester languages.

Themes

For reasons I’ll discuss later, we disabled all the themes except for our custom one, which is the global and default space theme. To disable a theme you have to go to plugins view and disable the appropriate theme plugins.

Cache Statistics

The name of this section in the UI is misleading, because not only can you view cache statistics here, but more importantly you can fully control the cache size via the UI. And in this case, I’m really glad that there is a UI to manage the cache config xml file, which due to its size is really hard to work with by hand. The changes you make via the UI are persisted in the Confluence Home directory and propagated thought the cluster.

Out of all the things you can tune via the admin UI, the cache tuning will have the biggest impact on your site’s performance. Confluence ships with cache settings optimized for smaller sites, so increasing the cache size is unavoidable for larger deployments.

Tuning the cache settings is a time-consuming process because you need to balance the memory consumption with performance improvements. Usually I revisit the cache stats once a month and look for caches that are performing badly because the number of objects allowed in that particular cache is low. Confluence caching system is composed of many caches that are controlled via this UI.

The best indicator of an overflowing cache is when the "Effectiveness" value is low (under 70-80%) AND “Percent Used” value is high (over 80%) AND usually the “Expired” value will be relatively high compared to “Hit” value in the same cell. This means that Confluence needs to go to the DB too often, even though it could cache the data in memory if the cache was bigger.

If you don’t understand what all the cache names and numbers mean, don’t worry about that too much. As long as you don’t make any dramatic changes too quickly and you monitor your JVM heap usage, you can’t break anything.

As you increase the cache sized, you’ll eventually start running out of heap space. That’s why you need to monitor the JVM and increase the -Xmx value as needed. If the number of concurrent users increases, you might also need to slightly increase the -Xmn value (see the JVM Tuning chapter for more info).

I wish Atlassian would provide better descriptions for all the available caches, because unless you know Confluence internals well, you won’t know what you are doing and that doesn’t feel good. Additionally, I’d like to see a way to limit memory usage, not the number of objects, because their size varies. Ideally, I'd really like to be able to just say "Use 3GB of memory for cache and distribute it in the most efficient way. Oh and let me know if you need more or less memory to work effectively". It would be better if Atlassian moved away from an in-process cache which in my opinion is not a good fit for Confluence. Maybe we'll get there one day.

Plugins

This section of the Admin UI is where you can install, uninstall, enable and disable plugins and their modules. There is also a Plugin Repository which additionally allows you to install plugins from Altassian’s remote servers or user specified URIs. The recently released Atlassian Universal Plugin Manager will eventually replace the latter one (or both?), I’m glad to see that happening.

I suggest that you disable plugins that you don’t use or don’t want your users to use as soon as possible. We disabled all the bundled themes because we wanted to provide users with only one custom theme developed and maintained by us (I’ll explain the reasoning in a future chapter). For security reasons thehtml and html-include macros should in my opinion be disabled on all but family Confluence deployments. And for performance reasons Confluence Usage Stats plugin is not suitable for any bigger deployments.

Plugin installation is very easy to do. That’s both good and bad. The plugin framework provided by Confluence is a very sophisticated piece of software which allows you to install and uninstall plugins on the fly without any need to restart the server. Need to quickly install a fixed version of a buggy plugin without disturbing hundreds or thousands of users that are currently using your site? Done. That’s how easy it is.

On the other hand, it is tempting to install plugins just because they have cool names or promise great features. You can do that in your dev or test environment, but in production you should only install plugins that you picked after some serious consideration.

This is what I look for when deciding whether to install a plugin or not:

was the functionality provided by the plugin requested by larger group of users or is the plugin needed for site administration purposes?
was the plugin developed and tested in-house, if no is it supported by Atlassian, if no can we or some respectable Atlassian partner support it should there be some problems?
is the plugin compatible with our confluence version? does it have a track record of being compatible or was it made compatible with new Confluence versions as they were released?
are there no major unresolved bugs in the areas of performance, scalability, data integrity and security?
does the plugin have an automated test suite with good test coverage?

If you answer “yes” to all of these questions, then you may go ahead do a trial before installing the plugin in production. Otherwise, you might provide your feedback to the plugin authors and wait if the pending issues get resolved before proceeding.

I don’t want to be harsh, but especially 2-3 years ago most of the plugins created for Confluence were crap. But as the platform matures, and Atlassian partners get involved more, the quality of available plugins has been slowly increasing. The main issue that I see is that the existing plugins are not developed and tested with large scale deployments in mind. Hopefully things will change as more and more deployments grow beyond small and medium sites. It’s unfortunate that even some commercial plugins, suffer from the very same issues that plague plugins created by bunch of volunteers and enthusiast. So pick your plugins carefully, do a trial, check for unresolved bugs and existing user complaints, and then decide.

I've been reasonably active in the Atlassian development community and from these interactions, I'd like to highlight the work done by Dan Hardiker (Adaptavist) and Roberto Dominguez (Comalatech). And though I haven't worked with guys from CustomWare, they are also considered to be pretty sharp.

Be especially careful with plugins that provide new macros for the wiki content. Once you install such a plugin you won't be able to uninstall it without breaking wiki pages until all the references to that macro are removed (with tens of thousands of pages and no ability to track the references this might be a big challenge).

In general however, try to keep the number of plugins low. It’s better for performance and you won’t get in trouble as often when you need to upgrade Confluence but some of the plugins you use are not compatible with the new Confluence version.

Conclusion

You should now have a good idea about how to configure Confluence and where this configuration is done. In the next chapters we'll look at upgrading Confluence, patching and more.

DGC II: The JVM Tuning

2010-07-27T00:13:00.000-07:00

This blog post is part of the DevOps Guide to Confluence series. In this chapter of the guide, I’ll be focusing on JVM tuning with the aim to make our Confluence perform well and operate reliably.

JDK Version

First things first: use a recent JDK. Java 5 (1.5) has been EOLed 1.5 years ago, there is absolutely no reason for you to use it with Confluence. As George pointed out in his presentation, there are some significant performance gains to be made just by switching to Java 6 and you can get another performance boost if you upgrade from an older JDK 6 release to a recent one. JDK 6u21 is currently the latest release and that’s what I would pick if I were to set up a production Confluence server today.

If you are wondering about which Java VM to use, I suggest that you stick with Sun’s HotSpot (also known as Sun JDK). It’s the only VM supported by Atlassian and I really don’t see any point in using anything else at the moment.

Lastly it goes without saying that you should use -server JVM option to enable the server VM. This usually happens automatically on server grade hardware, but it's safer to set it explicitly.

VM Observability

For me using JDK 6 is not just about performance, but also about observability of the VM. Java 6 contains many enhancements in the monitoring, debugging and probing arena that make JDK 5 and its VM look like an obsolete black box.

Just to mention some enhancements, the amount of interesting VM telemetry data exposed via JMX is amazing, just point a VisualVM to a local Java VM to see for yourself (no restart or configuration needed). Be sure to install VisualGC plugin for VisualVM. In order to allow remote connections you’ll need to start the JVM with these flags:

-Dcom.sun.management.jmxremote.port=some_port
-Dcom.sun.management.jmxremote.password.file=/path/to/jmx_pw_file
-Djavax.net.ssl.keyStore=/path/to/your/keystore
-Djavax.net.ssl.keyStorePassword=your_pw

Unless you make the port available only on some special admin-only network, you should password protect the JMX endpoint as well as use SSL. The JMX interface is very powerful and in the wrong hands could result in security issues or outages caused by inappropriate actions.

For more info about all the options available read this document.

In addition to JMX, on some platforms there is also good DTrace integration which helped me troubleshoot some Confluence issues in production without disrupting our users.
And lastly there is BTrace that allowed me to troubleshoot a nasty hibernate issue once. It's a very handy tool that as opposed to DTrace, works on all OSes.

I can’t stress enough how important continuous monitoring of your Confluence JVMs is. Only if you know how your JVMs and app are doing, you can tell if your tuning has any effect. George Barnett has also a set of automated performance tests which are handy to load test your test instance and compare results before and after you make some tweaks.

Heap and Garbage Collection Must Haves

After upgrading the JDK version, the next best thing you can do is to give Confluence lots of memory. In the infrastructure chapter of the guide, I mentioned that you should prepare your HW for this, so let’s put this memory to use.

Before we set the heap size, we should decide between 32-bit JVM and 64-bit JVM. 64-bit VM is theoretically a bit slower, but allows you to create huge heaps. 32-bit JVM has heap size limited by the available 32-bit address space and other factors. 32bit OSes will allow you to create heaps up to only 1.6-2.0 GB. 64bit Solaris will allow you to create 32bit JVMs with up to 4GB heap (more info). For anything bigger than that you have to go 64bit. It’s not a big deal, if your OS is 64bit already. The option to start the VM in 64bit mode is -d64. On almost all platforms the default is -d32.

Before I go into any detail, I should explain what are the main objectives of heap and garbage collection tuning for Confluence. The objectives are:

heap size - we need to tell JVM how much memory to use
garbage collector latency - garbage collection often requires that the JVM stops your application, this is GC pauses are often invisible, but with large heaps and under certain conditions might become very significant (30-60+ seconds)

Additionally we should also know a thing or two about how Confluence uses the heap. The main points are:

Objects created by Confluence and stored on the heap generally fall into three categories:
- short-lived objects - life-cycle of these is bound to a http request
- medium-lived objects - usually represent cache entries with shorter TTL
- long-lived objects - represent cache entries with big TTL, settings and infrastructure objects (plugin framework, rendering engine, etc), cache entries taking most of the space.
Confluence creates lots of short-lived objects per request
Half or more of the heap will be used by long-lived cache objects

By combining our objectives with our knowledge of Confluences heap profile, our tuning should focus on providing enough heap space for the application to have space for the cache, short-lived objects, as well as some extra buffer. Given that long-lived objects will (eventually) reside in the old generation of the heap, we want to avoid promoting short-lived objects there, because otherwise we’ll then need to do massive garbage collections of the old generation unnecessarily. Instead we should try to limit the promotion from young generation only to those objects, that will likely belong to the long-lived category.

We’ll also need to figure out how much heap you need to use. Unfortunately there isn’t an easy way to find this out, except for some educated guessing and trial & error. You can also read this HW Requirements document from Atlassian that can give you an idea about some starting points. I believe we started at 1GB, but over time went through 2GB, 3GB, 3.5GB, 4GB, 5GB all the way to 6GB.

The Confluence heap size depends on the number of concurrent users and the amount of content you have. This is mainly because Confluence uses a massive (well, in our case it is) in-process cache that is stored on the heap. We’ll get to Confluence and cache tuning in a later chapter of this guide.

So let’s set the max heap size. This is done via -Xmx JVM option:

-Xmx6144m 
-Xms6144m

The additional -Xms parameter says that the JVM should reserve all 6GB at startup — this is to avoid heap resizing which can be slow, especially when dealing with large heaps.

The rest of the heap settings in this post are based on 6GB heap size, you might need to make appropriate changes to adjust for your total heap size.

The next JVM option is -Xmn, which specifies how much of the heap should be dedicated to young generation (you should read up on generational gc if you don’t know what I’m talking about). The default is something like 25% or 33%, I set the young generation to ~45% of the entire heap:

 -Xmn2818m

Increasing the permanent generation size is also usually required given the number of classes that Confluence loads. This is done via -XX:MaxPermSize option:

-XX:MaxPermSize=512m

Given that determining the right heap size for your environment is non-trivial task for larger instances, especially if occasional memory leaks start consuming the precious memory, you always want to have as much data as possible to debug memory exhaustion issues. Aside from good monitoring (which I mentioned in the previous chapter) you should also configure your JVM to dump the heap, when an OutOfMemoryException occurs. You can then analyze this heap dump for potential memory leaks.

Since we are dealing with relatively big heaps, make sure you have enough space on the disk (heap dumps for 6GB heap usually take 2-4GB). I’ve had a very good experience using Eclipse Memory Analyzer to analyze these large heaps (VisuaVM or jhat are not up for analyzing heaps of this size). The relevant JVM options are:

-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/some/relative/or/absolute/dir/path

While trying to minimize gc latency in order to avoid situations when users have to wait several seconds for the stop-the-world (STW) gc to finish before their pages render is a commendable thing to do, the main reason why you want to do this is to avoid Confluence cluster panics.

Confluence has this “wonderful” cluster safety mechanism that is sensitive to any latency bigger than a few tens of seconds. In case a major STW gc occurs, the cluster safety code might announce cluster panic and shut down all the nodes (that’s right, all the nodes, not just the one that is misbehaving).

In order to be informed of any latencies caused by gc, you need to turn on gc logging. This is the magic combination of switches that works well for me:

-Xloggc:/some/relative/or/absolute/path/wikis-gc.log 
-XX:+PrintGCDetails 
-XX:+PrintGCTimeStamps 
-XX:+PrintGCDateStamps 
-XX:+PrintTenuringDistribution

Unfortunately the file specified via -Xloggc will get overwritten during a jvm restart, so make sure you preserve it either manually before a restart or automatically via some restart script. Additionally reading the gc log is a tough job that requires some practice and since the format varies a lot depending on your JDK version and garbage collector, I’m not going to describe it here.

Performance tweaks

The first performance boosting JVM option I'd like to mention is -XX:+AggressiveOpts, which will turn on performance enhancements that are expected to be on by default in the future JVM versions (more info).

If you are using 64bit JVM then -XX:+UseCompressedOops will make a big difference and will virtually eliminate the performance penalty you pay for switching from 32bit to 64bit JVM.

And lastly there is -XX:+DoEscapeAnalysis which will boost the performance by another few percents.

Optional Heap and GC tweaks

To slow down object promotion into the old generation, you might want to tune the sizes of the survivor space (a heap generation within the young generation). To achieve this, we want the survivor space to be slightly bigger than the default. Additionally I also want to keep the promotion rate down (objects that survive a specific number of collections in the survivor space will be be promoted to the older generation), so I use these options:

-XX:SurvivorRatio=6
-XX:TargetSurvivorRatio=90

I also found that by using parallel gc for the young generation and concurrent mark and sweep gc for the older generation I can practically eliminate any significant SWT gc pauses. Your mileage might vary on this one, so do some testing before you use it in production. These are the settings I use:

-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC
-XX:CMSInitiatingOccupancyFraction=68
-XX:MaxTenuringThreshold=31
-XX:+CMSParallelRemarkEnabled

Resources

The information above was gather from years of experience as well as various sources, including the following:

Running Multiple Web Apps in one VM

Don't do that. Really. Don't. Bad things will happen if you do (OOME, classloading issues etc).

Conclusion

Your JVM should now be in a good shape to host Confluence and serve your clients. In the next chapter of this guide I'll write about Confluence configuration, tuning, upgrades and more.

DGC I: The Infrastructure

2010-07-25T15:00:00.000-07:00

In the introductory post, I mentioned that a Confluence cluster is the way to go big. Let's go through some of the main things to consider when you start preparing your infrastructure.

Confluence cluster

To build a Confluence site, you need Confluence :-). Well, make it two... as in a two-node cluster license. I recommend this for any bigger site with relatively high uptime expectations, even if you know that your amount of traffic won't require load balancing between two nodes. I often find my self in a need of a restart (e.g. during a patch deployment) and with a cluster, you can restart one node at a time and your users won't even know about it.

Network

My team operates other big sites, and from all of them we expect some level of redundancy. Typically we split everything between "odd" (composed of hosts with hostnames ending with an odd number) and "even" strings, and this applies to Confluence nodes as well (that's why you need two-node license). Each string is composed of a border firewall, load balancer, switches and the actual servers (web/application/database/whathaveyou) and both strings can either share the load or work as primary&standby depending on your application needs and network configuration.

This kind of splitting, allows us to take half of our datacenter offline for maintenance when needed or allows us to absorb potential failure of any hardware or software within one string without any perceivable interruption of service.

Sure, you can make things even more redundant by adding a third or forth string, but none of our apps requires that level of redundancy and the cost and complexity of getting there is therefore hard to justify.

There are two important things that matter when it comes to setting up the network, and both can make or break you Confluence clustering.

The latency between the two nodes should be minimal. Ideally they should be just one hop apart and on a fast network (1GBit). There will be a lot of communication going on between your Confluence nodes, and you want it to happen as quickly as possible, otherwise the cluster synchronization will drag down your overall cluster performance. Don't even think about putting the two nodes into different datacenters, let alone on different continents. Confluence clustering was not built for that type of scenario.
Make absolutely sure that your network (mainly switches, OS, firewall) supports multicast.

The best way to check that the multicast works reliably is to use the multicast test tool that is bundled with Coherence (a library that is bundled with Confluence). To run it just run the following command on all nodes and check if all packes are being delivered and no duplicates are present:

java -cp $CONFLUENCE_PATH/WEB-INF/lib/coherence-x.y.jar:$CONFLUENCE_PATH/WEB-INF/lib/tangosol-x.y.jar \
com.tangosol.net.MulticastTest \
-group $YOUR_MULTICAST_IP:$YOUR_MULTICAST_PORT \
-ttl 1 \
-local $NODE_IP

In our environment, it took us months of waiting for the right patch from our network gear vendor and some OS patching to make things totally stable. Fortunately, our ops guys eventually found the magic combination of patches and settings, and then we were good to go.

Our site uses both http and https protocols for content delivery and since we already had an SSL accelerator available in our datacenter we utilized it for Confluence, but I don't think that with current hardware, hw acceleration is not very important these days.

Another noteworthy suggestion I have for your network is the load balancer configuration. We started off with a session-affinity-based load-balancing, but at one point people started to notice that sometimes they see different content than their colleagues. This was due to delay in propagation of changes throughout the cluster. Usually the delay is unnoticeable, but for some reasons it's not always the case. I haven't investigated this issue further and just switched to primary&slave load balancing, which has been working great for us since. This of course will work only if each of your nodes can handle all the traffic on its own, but you can trust me that it solves all the issues with users that don't believe in eventual consistency :-).

Hopefully your load balancer will perform healthchecks against your nodes. The /errors.jsp path is the ideal target for these healthchecks, because it returns HTTP 200 only if everything is ok with the node.

When it comes to firewall rules (you have a firewall right?), you shouldn't allow incoming connection from public networks directly to your servers, all the public traffic should go through loadbalancer only. As for outbound connections, you should allow your servers to connect to any public server on ports 80 (HTTP) and 443 (HTTPS); these connections are needed for feed retrieval, open social gadgets and plugin installation.

Hardware (cpu, memory, disk)

Update: I came across this HW requirements document from Atlassian, which is helpful especially for smaller instances.

When you are making your hardware choices, I suggest you stick with a server that is relatively recent and has decent single-threaded performance, yet offers multicore parallelism. Confluence does relatively a lot of number crunching per http request, so both single-threaded and multi-threaded horse-power are needed to get good results. Additionally Confluence's boot process is not the best one, so with poor single-threaded throughput you'll end up waiting minutes for the app to start (at one point I did!).

Confluence loves memory! So don't be stingy. RAM is cheap these days, so get a few gigs that will be dedicated just to Confluence. My instance uses 6 GB JVM heap and with additional non-heap memory consumption, OS overhead and an extra buffer. I allocated 10GB of RAM for each Confluence node. You will likely start with much lower memory requirements, but as your instance grows, so will the memory requirements - keep that in mind.

When it comes to disk and disk space, you have to realize three things.

Confluence stores all of its persistent data in a (hopefully remote) database.
Confluence relies on fetching data from its Lucene index, stored on the local file system (each node has its own copy). This index is built from the db contents and can be rebuilt at any time.
Attachments, which can represent a huge chunk of your persistent data, will be stored in the database. Confluence won't let you use e.g. shared filesystem when you are running a cluster.

All of this means that you will need a few (dozen) gigabytes of local disk space that can be accessed reasonably quickly. SSD will likely not buy you much, use it for your DB! Server grade hard drives configured in redundant software or hardware RAID should be sufficient for your web/application server (you can skip the RAID if you can rebuild the server really quickly after a disk failure).

OS & Filesystem

The choice of OS is often a religious one. But I think that it's more important that you are comfortable administering your OS than anything else. We use Solaris 10, or more recently OpenSolaris everywhere. Especially OpenSolaris is superior to most (all?) of the OSes out there (heh, now I'm being religious), but it will be worth cat's pee to you if you have no clue about how to work with it and don’t have time or willingness to learn a lot of cool stuff about the OS. In general, I'd say that any 64bit *nix OS should be suitable as long as you know how to use it. You'll want 64bit OS so that you can load it with loads of RAM and create big JVM heap once you need it.

One nice thing that comes with Solaris and OpenSolaris (and BSD) is ZFS file system. If you don't know much about it, I suggest that you read a bit about it. ZFS can make your backup strategy a lot simpler and allows you to revert from a failed upgrade in a matter of seconds. I'm not exaggerating, it happened to me several times. Hopefully Btrfs will soon be production ready for Linux distros and will offer comparable conveniences. If you can't use either of these, you'll have to suck it up and deal with it. I don't envy you...

Virtualization

During the last 3 years, we tried several combinations of deployment configurations for our Confluence site. These include Solaris 10 servers shared by several apps, Solaris 10 Zones (one zone per app) and OpenSolaris with Xen virtualization. Xen and OpenSolaris is what we currently use. It works well, but if I were to make a decision today, I would probably go with OpenSolaris and Zones. This combination gives you the best stability, performance, resource virtualization and application isolation.

In any case, many people ask what is the performance penalty for going virtualized. My answer is that it depends on your application, but for a webapp, more likely than not, it isn't going be the main reason of your performance problems. Decent hardware is going to make the virtualization penalty almost invisible and at the same time will give you flexibility when allocating resources in your data center. Just to give you a rough idea, the overhead for Xen is 10-30%; for Solaris Zones it's a lot less.

Web Container

Atlassian recommends using Tomcat as the web container for Confluence. We could again spend a lot of time fighting a religious battle here, but I'm going to avoid that. If Tomcat works for you and you don't find it lacking features that make enterprise deployments and operation easier, then good for you. You will most likely want it fronted it with Apache webserver or something similar though.

I've been using Sun Web Server 7 in my production environment and was quite happy with it. Another excellent choice is GlassFish v2.1 or v3, which I've been using for Confluence on my Mac. Unfortunately, Confluence doesn't adhere to the Servlet spec in some places, so you'll have to patch it to get it to run with GF v3. Glassfish v2.1 is not affected, but suffers from Xalan classes clashing, so to fix that you need to put Confluence's xalan-x.y.z.jar into $GLASSFISH_HOME/domains/$YOURDOMAN/lib/. Otherwise everything works as expected.

For a bigger site, you'll likely need to increase the worker thread count in your servlet container. Check your container's documentation to see what's the default and how to increase it. You should also know what your peak concurrent request rate is (monitor it!) and in combination with your infrastructure capabilities (load test it!) choose the right value for you. Ours is 256 which is higher than our usual peak traffic, but lower than what we could handle if we had to.

Logs & Monitoring

Paraphrasing my friends daughter: "More data, more better!". Log as much as you can and archive logs. You'll never know when you'll need to search for an exception log and confirm that it started to appear 7 months ago, right after that particular confluence upgrade.

What helped me on several occasions, was having detailed access logs. I use Apache combined log format with an extra attribute - request duration in microseconds. This format will not only give you a good idea about your app's performance, but will also help you track various issues by logging http referal[sic] and user-agent headers. This can often be invaluable info!

Here is a list of different types of logs you should be gathering: confluence log, web container log, jvm gc log and http access log.

In order to get an in-depth information about your visitors, usage patterns and content, I suggest that you integrate your Confluence with web analytics services like Google Analytics or Omniture. From their reports you can learn more about how, when and from where your users use your site.

When it comes to monitoring, your strategy should most certainly include JVM and JMX monitoring. Confluence as well as JVM exposes quite some interesting metrics via JMX. You should know what these values look like throughout the day or week. Only then you'll be able to efficiently troubleshoot issues when they occur (and they will occur!). Bare minimum include: heap space usage, cpu usage, requests per 10 second, errors per 10 seconds, avg request duration

We have a custom monitoring app that allow us to gather, archive and analyze these JVM/JMX metrics, but there are also some open source tools available of various quality (e.g. Munin looks promising).

The second part of our monitoring strategy is implemented as a local agent (we use Satan), that closely monitors the JVM process and the app itself by checking if it's not running out of heap space, as well as by performing http health checks. In the case that multiple failures are registered, the agent restarts the app and emails out an alert with the description of the failure. This allows us to sleep through the night without worrying that a pesky memory leak is going to take down our site at night. Fortunately, we haven’t seen any stability issues for a while now, but things were different in the past.

The last part of our monitoring strategy is implemented as remote http agents. These periodically perform http health checks from various locations on the Internet and send out alerts when an issue is detected. This gives us a good visibility into potential networking issues that wouldn't be caught by a local agent. There are several third party solution that you could use, or you can build your own (and host it cross the globe on EC2).

DB

The choice is up to you. Pick something supported by Atlassian or else you'll likely regret it. We use MySQL5 and for the most part we've been quite happy with it. Our db currently takes ~26GB, so be sure to account for gigabytes of db files and several times that for db backups. The biggest space sucker are attachments. Since a Confluence cluster can currently store attachments only in the database, you have to limit the attachment size, or else you'll likely end up with performance problems due to overloaded db.

We limit attachment size to 5MB. There are several users that are not happy about that, but on the other hand, it helps people to realize that often a simple wiki page is a much better distribution medium than an OpenOffice document attached to a blank wiki page. I'd bet that our users would stick huge ISO images into our db if we allowed them to. My suggestion is to start with a low limit and increase it if there is a business justification for it. Maybe one day Confluence will support S3 or Google storage as the backend for attachments, until then, keep the size limit low.

The db should be hosted on a dedicated server with lots of RAM. I'm fortunate enough to have DBAs that take care of running the DB for me, so I don't have to worry about that part. A good DBA , MANY FAST disks (possibly SSD) and lots of RAM are the key ingredients to well performing db. Of course, make sure the latency between both Confluence nodes and the db server is minimal. You shouldn't think of doing anything worse than 1GBit network and locate the db within the same datacenter.

I mentioned ZFS before and I'll mention it again. If you put the db files that contain your Confluence database on a dedicated ZFS dataset (think volume), you'll be able to take snapshots of your db during upgrades or on the fly (you'll have to momentarily lock the db to do that) and then revert from these snapshots instantly when you need it. This is just awesome. :-)

If you are using MySQL5, your minimal my.cnf should look like this:

[mysqld]
default-storage-engine=innodb
default-table-type=innodb
default-character-set=utf8
default-collation=utf8_general_ci
max_allowed_packet=32M

The last setting will allow you to upload up to 32MB large files (attachments, plugins, etc) into the db.

Backups

Your users will hate you if you lose any of their precious data, so don’t do it! The best way to avoid any data loss is to have a backup strategy in place. Ours is composed of several parts.

Config files are stored in our version control system, which is, surprise surprise, being backed up.

Confluence home directory on our Confluence nodes is being backed up only just before the upgrade via a ZFS snapshot. All the files in there (except for the config files) can be rebuilt from the database, so I don’t worry about them.

The database is being backed up nightly via a SQL dump, which is then backed up on a tape. Additionally, just before an upgrade, we take a ZFS snapshot of the filesystem the db files reside on. This allows us to do instant rollbacks in case the upgrade fails. I experienced a situation where it took us hours to roll back from a SQL dump. It’s slooow. Since then we switched to ZFS snapshots.

The database is really the master storage of all the Confluence data, so in addition to all the backups, we also run a redundant (remember “odd” and “even”?) db server, that the master database is being replicated to on the fly via MySQL master/slave replication. During an upgrade we now also stop the replication, so that we can use the slave right away if something happened to the master during an upgrade and we couldn't use ZFS to rollback.

As if that was not enough, there is one more layer that allows users to recover from user errors in a fine-grained manner. It’s Confluence wiki page versioning and wiki space trash. The combination of these two features, enables users to undo most of the editing mistakes on their own, without bothering site administrators (I’ll talk more about delegation in chapter IV of the guide).

There is also a Confluence built-in backup mechanism, but it works well only for small instances. This backup process is resource intensive, generates lots of data and if I remember correctly breaks ones you reach certain size. Don't use it. You'll have to explicitly disable it via the Confluence Admin UI.

Prod, Test, Dev Environments

The ability to experiment in the production environment will decrease with the increase of users using the site. For this reason, you'll need to build a Test environment that closely matches your production environment. Here you can practice your Confluence upgrade, or run automated tests just before a release. If you are doing Confluence core or plugin development, you'll also need a dev environment. This one can be a simplified and scaled down version of production (e.g. you can forgo clustering) and should be conveniently located on your dev machine or server.

Conclusion

If you follow my advice, you should now have an infrastructure that is will help you run your Confluence site in a performant, scalable and reliably way. If you found something important missing, feel free to post your suggestions as comments.

In the next chapter of this guide we'll look at the JVM tuning.

DevOps Guide to Confluence (DGC)

2010-07-25T11:12:00.000-07:00

After working with Atlassian Confluence for 3 years, running one of the bigger public Confluence installations, I realized that there is a major lack of information about how to run Confluence on a larger scale and outside of the intranet firewalls. I'm hoping that I can improve this situation with a blog series that will describe some of the (best?) practices that I implemented while running, tweaking, patching and supporting our Confluence-based site.

Just to throw out some numbers to give context of what I mean by "relatively large":

# registered users: 180k+
# contributing users: 7k+
# wiki spaces: 1.5k+
# wiki pages: 65k+
# page revisions: 570k+
# comments: 10k+
# visits per month: ~300k
# page views per month: ~800k
# http requests per day: ~1m+ (includes crawlers and users with disable javascript)

So I'm not talking about a huge site like amazon, twitter, etc, but still bigger than most of the public facing confluence instances out there.

Some of the practices described in this guide might be an overkill for smaller deployments, so I’ll leave it up to you to pick the right ones for you and your environment.

There are many aspects that need careful consideration if you want to go relatively big, and there are even more of them when you run your site on the Internet as opposed to doing it internally within an organization. In my blog series I'm going to focus on these areas that I consider important:

The infrastructure
- Confluence cluster
- hardware (cpu,memory,disk)
- os, filesystem
- web container
- logs & monitoring
- network
- db
- backups
The JVM tuning
- heap
- garbage collector
- fancy switches
Confluence configuration and tuning
- where and how to configure Confluence
Confluence upgrades
- release history and track record
- following announcements and getting support
- the upgrade procedure
Customizing and patching Confluence
- customization options
- reasons for patching
- Mercurial Queues as patch management tool
Wiki Organization and Working with the Community
- main principles
- global and default permissions
- delegating decision-making to space admins
Internet-facing deployment and operation
- Varnish or other caching reverse proxy
- robots.txt

I'm not going to go into details about why to pick Confluence or why not to pick it. I really just want to focus on how to make it run smoothly and reliably while serving a relatively large audience of users (and robots).

Given that we want to run a site on the Internet, we are lucky to have well defined maintenance windows, that we can work with. Meaning that any downtime will be perceived by at least a portion of your users as your failure, and the only way how you can avoid looking like an idiot is to keep the downtime to the absolute minimum.

You are now probably thinking that a Confluence cluster will solve all your problems with scalability and reliability.

Right, that's what the marketing people tell you. Anyone who knows a thing or two about software engineering, knows that there is no such a thing as "unlimited scalability" and ironically a Confluence cluster can hit several bottlenecks quite quickly in certain situations. That said, a Confluence cluster with all its pros and cons is really the way to go big with Confluence, but you should have realistic expectations about its scalability and reliability.

The fact that makes things even more difficult is that if you do things right, your wiki is going to take off. More users, more content, more traffic, more spam, more crawlers, more users unhappy about any kind of downtime... Growth is what you need to take into account from day one. I'm not saying that you have to start big, you just shouldn't paint your self into a corner and I'm going to mention some tips on how to avoid just that.

I was inspired to write up this guide after watching George Barnett’s presentation from this year’s Atlassian Summit. George made some really good points and I encourage you to watch his talk. My guide will not focus just on performance and scalability, but also on reliability, smooth day-to-day operation and more.

Continue reading: DGC I: The Infrastructure

Improving Satan and Solaris SMF

2010-05-31T18:24:00.000-07:00

One of the features of Solaris that we heavily rely on in our production environment at work is Service Management Facility or SMF for short. SMF can start/stop/restart services, track dependencies between services and use that to optimize the boot process and lots more. Often handy in production environment is that SMF keeps track of processes that a particular service started and if a process dies, SMF restarts its services.

One gripe I have with SMF is that its process monitoring capabilities are rather simple. A process associated with a contract (service) must die in order for SMF to get the idea that something is wrong and that the service should be restarted. In practice, more often than not a process gets into a weird state that prevents it from working properly, yet it doesn't die. Failures might include excessive cpu or memory usage or even application level failures that can be detected only by interacting with the application (e.g. http health check). SMF in its current implementation is incapable of detecting these failures. And this is where Satan comes into the play.

Satan a small ruby script that monitors a process and following the Crash-only Software philosophy, kills it when a problem is detected. It then relies on SMF to detect the process death(s) and restart the given service. I fell in love with the simplicity of Satan (which was inspired by God) and started exploring the feasibility of using it to improve the reliability of SMF on our production servers.

Upon a code review of the script, I noticed several things that I wished were implemented differently. Here are some:

Satan watches processes rather than services as defined via SMF
One Satan instance is designed to watch many different processes for different services, which adds unnecessary complexity and lacks isolation
Satan is merciless (what a surprise! :-) ) and uses kill -9 without a warning
Satan has no test suite!!! :-( (i.e. I must presume that it doesn't work)

Thankfully the source code was out there on GitHub and licensed under BSD license so it was just a matter of a few keystrokes to fork it (open source FTW!). By the time I was done with my changes, there wasn't much of the original source code left, but oh well :-)

I'm happy to present to you http://github.com/IgorMinar/satan for review and comments. The main changes I made are the following:

One Satan instance watches single SMF service and its one or more processes
The single service to monitor design allows for automatic monitoring suspension via SMF dependencies while the monitored service is being started, restarted or disabled
Several bugfixes around how rule failures and recoveries are counted before a service is deemed unhealthy
At first Satan tries to invoke svcadm restart and only if that doesn't occur within a specified grace period, it uses kill -9 to kill all processes for the given contract (service)
Satan now has decent RSpec test suite (more on that in my previous post)
Improved HTTP condition with a timeout setting
New JVM free heap space condition to monitor those pesky JVM memory leaks
Extensible design now allows for new monitoring conditions (rules) to be defined outside of the main Satan source code

As always there are more things to improve and extend but, I'm hoping that my Satan fork will be a decent version that will allow us to keep our services running more reliably. If you have suggestions, or comments feel free to leave feedback.

Testing matters, even with shell scripts

2010-05-31T11:24:00.000-07:00

A few months ago, we migrated out production environment at work from Solaris 10 to OpenSolaris. We loved the change because it allowed us to take advantage of the latest inventions in Solaris land. All was good and dandy until one day one of our servers ran out of disk space and died. WTH? We have monitoring scripts that alert us long before we get even close to running out of space, yet no alert was issued this time. While investigating the cause of this incident, we found out that our monitoring scripts that work well on Solaris 10, didn't monitor the disk space correctly on OpenSolaris. When I asked our sysadmins if they didn't have any tests for their scripts that could validate their functionality, they laughed at me.

Fast forward a few months. A few days ago I started looking at Satan, to augment the self healing capabilities of Solaris SMF (think initd or launchd on stereoids). At first sight I loved the simplicity of the solution, but one thing that startled me during the code review was that there were no tests for the code, except for some helper scripts that made manual testing a bit less painful. At the same time, I spotted several bugs that would have resulted in an unwanted behavior.

Satan relies on invoking solaris commands from ruby and parsing the output and acting upon it. Thanks to its no BS nature, ruby makes for an excellent choice when it comes to writing programs that interact with the OS by executing commands. There are several ways to do this, but the most popular looks like this:

ps_output = `ps -o pid,pcpu,rss,args -p #{pid}`

All you need to do is to stick the command into backticks and optionally use #{variable} for variable expansion. To get a hold of the output, just assign the return value to a variable.

Now if you stick a piece of code like this in the middle of the ruby script you get something next to untestable:

module PsParser
 def ps(pid)
   out_raw = `ps -o pid,pcpu,rss,args -p #{pid}`
   out = out_raw.split(/\n/)[1].split(/ /).delete_if {|arg| arg == "" or arg.nil? }
   { :pid=>out[0].to_i,
     :cpu=>out[1].to_i,
     :rss=>out[2].to_i*1024,
     :command=>out[3..out.size].join(' ') }
 end
end

With the code structured (or unstructured) like this, you'll never be able to test if the code can parse the output correctly. However if you extract the command execution into a separate method call:

module PsParser
 def ps(pid)
   out = ps_for_pid(pid).split(/\n/)[1].split(/ /).delete_if {|arg| arg == "" or arg.nil? }
   { :pid=>out[0].to_i,
     :cpu=>out[1].to_i,
     :rss=>out[2].to_i*1024,
     :command=>out[3..out.size].join(' ') }
 end

 private
 def ps_for_pid(pid)
   `ps -o pid,pcpu,rss,args -p #{pid}`
 end
end

You can now open the module and redefine the ps_for_pid in your tests like this:

require 'ps_parser'

PS_OUT = {
 1 => "  PID %CPU    RSS ARGS
12790   2.7 707020 java",
 2 => "  PID %CPU    RSS ARGS
12791  92.7 107020 httpd"
}

module PsParser
 def ps_for_pid(pid)
   PS_OUT[pid]
 end
end

And now you can simply call the pid method and check if the fake output stored in PS_OUT is being parsed correctly. The concept is the same as when mocking webservices or other complex classes, but applied to running system command and programs.

To conclude, what makes you more confident about a software you want to rely on. An empty test folder: Or all green results from a test/spec suite?

Configuring Common Access Log Format in GlassFish v2 and v3

2009-12-21T12:30:00.000-08:00

For a long time Matthew and I had a dilemma about changing the non-standard access log format used by GlassFish v2 and v3, to the commonly used common or combined format used by Apache.

GlassFish does allow one to specify the access log format, but how this works is not obvious. If one tries to create a formatting string, which should result in one of the Apache access log formats, the resulting output does contain all the specified fields and in the right order, but the field delimiters are not preserved from the formatting string and instead all the fields are quoted and separated by spaces. That's not quite what we want, especially if you plan to feed the logs into a log analyzer that expect the usual Apache syntax.

While getting Confluence wiki to run on GlassFish v3, I fetched the GF source code and since I already had it, I thought that it should be trivial to find out how the Access log format gets processed in GF.

To my big surprise, I found out that there are classes with very suspicious names: CommonAccessLogFormatterImpl, CombinedAccessLogFormatterImpl and DefaultAccessLogFormatterImpl. A minute later I also found this piece of code "hidden" in PEAccessLogValve:

    // Predefined patterns
   private static final String COMMON_PATTERN = "common";
   private static final String COMBINED_PATTERN = "combined";


   ...
   ...


   /**
    * Set the format pattern, first translating any recognized alias.
    *
    * @param p The new pattern
    */
   public void setPattern(String p) {
       if (COMMON_PATTERN.equalsIgnoreCase(p)) {
           formatter = new CommonAccessLogFormatterImpl();
       } else if (COMBINED_PATTERN.equalsIgnoreCase(p)) {
           formatter = new CombinedAccessLogFormatterImpl();
       } else {
           formatter = new DefaultAccessLogFormatterImpl(p, getContainer());
       }
   }

Whoa! So both Apache formats are implemented already and one just needs to know how to "unlock" them. The "common" and "combined" constants looked like the magic keywords to do just that, and sure enough, when one sets either of them as the formatting string, the log will contain the expected output.

You can also use asadmin to make this config change:

asadmin set server.http-service.access-log.format="combined"

After a restart the log now uses the requested format:

0:0:0:0:0:0:0:1%0 - - [21/Dec/2009:07:42:45 -0800] "GET /s/1722/3/_/images/icons/star_grey.gif HTTP/1.1" 304 0
0:0:0:0:0:0:0:1%0 - - [21/Dec/2009:07:42:45 -0800] "GET /images/icons/add_space_32.gif HTTP/1.1" 304 0
0:0:0:0:0:0:0:1%0 - - [21/Dec/2009:07:42:45 -0800] "GET /images/icons/feed_wizard.gif HTTP/1.1" 304 0
0:0:0:0:0:0:0:1%0 - - [21/Dec/2009:07:42:45 -0800] "GET /images/icons/people_directory_32.gif HTTP/1.1" 304 0
0:0:0:0:0:0:0:1%0 - - [21/Dec/2009:07:42:45 -0800] "GET /s/1722/3/_/images/icons/add_12.gif HTTP/1.1" 304 0

Believe it or not, this information is not documented anywhere in the official documentation, and even folks Matthew chatted with on Sun's internal support mailing lists had no clue about it. Ideally the GF documentation and UI should be updated to make changing the access log format as simple as it should be.

Oh btw, open source FTW, when documentation is lacking one can at least read the sources!

Running Confluence on GlassFish v3

2009-12-20T19:31:00.000-08:00

Even though Atlassian considers GlassFish to be an unsupported servlet container for Confluence, it is quite easy to use Confluence with GlassFish v2.1. In fact that's the container that I've been using for a long time during my Confluence and Confluence plugin development.

I've been monitoring progress of GlassFish v3 development for several months and noticed that at some point Confluence 2.x and 3.0.x stopped working due to conflicts between different versions of Apache Felix used by both GFv3 and Confluence.

Fortunately Confluence 3.1 now contains Felix v2.x (an upgrade from 1.x), which solves the previously mentioned issues. Excited about the change, I tried to deploy Confluence 3.1 to GFv3 (final) and observed that there are a few more issues that one needs to deal with. I filed these two bugs and one RFE against Confluence and provided patches that anyone can use to get Confluence to run with GFv3:

Once the patches are applied to the Confluence source code, build a war file in the usual way, deploy it to GFv3 and you should be good to go. (There is a harmless exception thrown when Confluence starts, more info, just ignore it)

Oh, and be sure to vote for CONF-6603 to get Atlassian to officially support GlassFish.

Using Mercurial Bisect to Find Bugs

2009-11-14T14:05:00.000-08:00

Yesterday I tried to find a regression bug in Grizzly that was preventing grizzly-sendfile from using blocking IO. I knew that the bug was not present in grizzly 1.9.15, but somewhere between that release and the current head someone introduced a changeset that broke things for me. Here is how I found out who that person was.

Grizzly is unfortunately still stuck with subversion, so the only thing (besides complaining) that I can do to make my life easier, is to convert the grizzly svn repo to some sane SCM, such as mercurial. I used hgsvn to convert the svn repo.

Once I had a mercurial repo, I wrote BlockingIoAsyncTest - a JUnit test for the bug. And that was all I needed to run bisect:

$ echo '#!/bin/bash                                                                          
mvn clean test -Dtest=com.sun.grizzly.http.BlockingIoAsyncTest' > test.sh
$ chmod +x test.sh
$ hg bisect --reset #clean repo from any previous bisect run
$ hg bisect --good 375 #specify the last good revision
$ hg bisect --bad tip #specify a known bad revision
Testing changeset 604:82e43b848ae7 (458 changesets remaining, ~8 tests)
517 files updated, 0 files merged, 158 files removed, 0 files unresolved
$ hg bisect --command ./test.sh #run the automated bisect
...
(output from the test)
...
...
...

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 9 seconds
[INFO] Finished at: Sat Nov 14 11:41:07 PST 2009
[INFO] Final Memory: 25M/79M
[INFO] ------------------------------------------------------------------------
Changeset 500:983a3fc2debe: good
The first good revision is:
changeset:   501:b5239bf9427b
branch:      code
tag:         svn.3343
user:        jfarcand
date:        Wed Jun 17 17:20:32 2009 -0700
summary:     [svn r3343] Fix for https://grizzly.dev.java.net/issues/show_bug.cgi?id=672

In under two minutes I found out who and with which revision caused all the trouble! Bisect is very useful for large projects, developed by multiple users, where the amount of code and changes is not trivial. Finding regressions in this way can save a lot of time otherwise spent by debugging.

The only caveat I noticed is that you need to create a shell script that is passed in as an argument to the bisect command. It would be a lot easier if I could just specify the maven command directly without the intermediate shell script.

grizzly-sendfile to Become an Official Grizzly Module

2009-05-28T19:30:00.000-07:00

After a chat with JFA about grizzly-sendfile's future, I'm pleased to announce today that grizzly-sendfile 0.4 will be the first version of grizzly-sendfile released as an official module of grizzly. This is a huge news for grizzly-sendfile and I believe an equally important news for grizzly and its community.

What this "merger" means for grizzly-sendfile:

great opportunity to extend the reach
opportunity to become the default static file handler in Grizzly
aspiration to become the default static file handler in GlassFish v3
more testing and QA
easier and faster access to grizzly developers and contributors

What this "merger" means for grizzly:

contribution of 1 year of research, development and testing time in the area of static http downloads
several times better performance and scalability of http static file downloads
built-in X-Sendfile functionality
better JMX instrumentation for http downloads
and more

If you can't wait for 0.4, go and get recently released version 0.3.

This is a great day for both projects! :-)

Project site: http://grizzly-sendfile.kenai.com/

grizzly-sendfile 0.3 is out!

2009-05-14T00:50:00.000-07:00

After a few months of late night hacking, grizzly-sendfile 0.3 is finally ready for prime time!

New features include:

grizzly 1.9.15 compatibility
glassfish v3b48 compatibility (via an OSGi bundle)
kick-ass performance and scalability
grizzly-sendfile-server
autosendfile mode
Sendfile Algorithms: new EqualBlockingAlgorithm + improved EqualNonBlockingAlgorithm
lots of improvements to grizzly-sendfile-benchmark
maven build system
lots of smaller improvements and bug fixes

I also started using kenai's JIRA for issue tracking. So feel free to file bugs or RFE's there.

Benchmark & Enjoy!

Project Website The source code The binaries

grizzly-sendfile and Comparison of Blocking and NonBlocking IO

2009-05-10T19:00:00.000-07:00

From the very early beginnings of my work on grizzly-sendfile (intro) I was curious to compare blocking and non-blocking IO side to side. Since I didn't have any practical experience to understand which one would be more suitable when, I designed grizzly-sendfile to be flexible so that I could try different strategies and come to conclusions based on some real testing rather than theorizing or based on the words of others. In this post I'd like to compare blocking and nonblocking IO, benchmark them, and draw some conclusions as to which one is more suitable for specific situations.

grizzly-sendfile has a notion of algorithms that control the IO operations responsible for writing data to a SocketChannel (grizzly-sendfile is based on NIO and leverages lots of great work put into grizzly). Different algorithms can do this in different ways, and this is explained in depth on the project wiki. The point is that this allows me to create algorithms that use blocking or nonblocking IO in different ways, and easily swap them and compare their performance (in a very isolated manner).

Two algorithms I implemented right away were SimpleBlockingAlgorithm (SBA) and EqualNonBlockingAlgorithm (ENBA), and only recently followed by EqualBlockingAlgorithm (EBA). The first one employs the traditional approach of sending a file via the network (while not EOF write data to a SocketChannel using blocking writes), while ENBA uses non-blocking writes and Selector re-registration (in place of blocking) to achieve the same task. This means that a download is split into smaller parts, each sequentially streamed by an assigned worker thread to the client. EBA works very similarly to ENBA, but uses blocking writes.

I ran two variations of my faban benchmark ^[0] against these three algorithms. At first I made my simulated clients hit the server as often as possible and download files of different sizes as quickly as possible. Afterward I throttled the file download speed to 1MB/s per client (throttling was done by clients). While the first benchmark simulates traffic close to the one on the private network in a datacenter, the second benchmarks better represents client/server traffic on the Internet.

grizzly-sendfile delegates the execution of the selected algorithm to a pool of worker threads, so the maximum number of the treads in the pool, along with the selected algorithm, are one of the major factors that affects the performance^[1] and scalability^[2] of the server. In my tests I kept the pool size relatively small (50 threads), in order to easily simulate situations when there are more concurrent requests than the number of workers, which is common during traffic spikes.

Conc. Clients	Download limit	Algorithm	Avg init time^[3] (sec)	Avg speed^[4] (MB/s)	Avg total throughput (MB/s)
50	none	SBA	0.019	4.36	208.76
50	none	ENBA	0.021	4.15	198.79
50	none	EBA	0.018	4.23	202.29

100	none	SBA	4.666	4.32	212.79
100	none	ENBA	0.048	1.84	168.15
100	none	EBA	0.140	1.96	175.71

200	none	SBA	14.288	4.31	208.59
200	none	ENBA	0.108	0.87	144.69
200	none	EBA	0.264	0.97	158.83

Conc. Clients	Download limit	Algorithm	Avg init time^[3] (sec)	Avg speed^[4] (MB/s)	Avg total throughput (MB/s)
50	1MB/s	SBA	0.003	1.0	42.9
50	1MB/s	ENBA	0.002	0.98	41.82
50	1MB/s	EBA (81k)^[5]	0.003	0.98	41.91
50	1MB/s	EBA (40k)^[6]	0.002	0.98	42.85

100	1MB/s	SBA	20.5	1.0	40.4
100	1MB/s	ENBA	0.003	0.99	85.14
100	1MB/s	EBA (81k)	0.018	1.0	84.19
100	1MB/s	EBA (40k)	0.013	0.99	84.34

200	1MB/s	SBA	64.8	1.0	37.12
200	1MB/s	ENBA	0.112	0.86	141.8
200	1MB/s	EBA (81k)	0.2	0.95	156.59
200	1MB/s	EBA (40k)	0.159	0.96	154.2

300	1MB/s	SBA	113.9	1.0	34.2
300	1MB/s	ENBA	0.185	0.58	127.53
300	1MB/s	EBA (81k)	0.31	0.61	133.66
300	1MB/s	EBA (40k)	0.239	0.63	132.75

The interpretation of the results is that with SBA the individual download speeds are slightly higher than when ENBA or EBA is used (this is mainly due to some extra scheduling related to the Selector reregistration). However, EBA and ENBA can utilize the available server bandwidth significantly better than SBA. This is especially true when there is a large number of slow clients downloading larger files. One big downside of SBA is that if the number of concurrent downloads is higher than the number of worker threads, the time to initiate download easily increases into extreme heights.

The conclusion is that SBA is well suited for controlled environments where there is a small number of fast clients (server-to-server communication on an internal network), while EBA shines in environments where there is very little control over the number and speed of clients (file hosting on the Internet). While EBA performs and scales better than ENBA, two advantages that ENBA has over EBA are smaller latency and higher resiliency to DoS or DDoS attacks when malicious clients open connections and block.

The results above do not represent well the performance and throughput of grizzly-sendfile (the benchmarks were run on my laptop!), but they certainly provide some evidence that can be used to determine characteristics of different algorithms. I'll do some more thorough testing later when I feel that grizzly-sendfile has enough features and it is time for some fine tuning (there is still a lot that can be done to make things more efficient).

I love explaining things visually, so I drew the following diagrams that describe the differences between the two algorithms much better than many words.

SimpleBlockingAlgorithm

The server can concurrently handle only the same number of requests as the number of worker threads in the pool. All the extra downloads have to be queued until one of some workers completes a download. The downloads take shorter to process, but the worker is blocked while a write is blocked. The slower the client the more blocking occurs and the worker utilization (and server throughput) goes down.

EqualNonBlockingAlgorithm

The number of downloads a server can concurrently handle is not limited by the number of workers. The downloads take slightly longer to process, but the workers are much better utilized. The increase in utilization is due to the fact that no blocking occurs and workers can multiplex - "pause" downloads for which clients are processing the data (stored in OS/network buffers). In the meantime workers can serve whichever client is ready to receive more data. This causes the download to be split into several parts, each possibly served by a different worker thread.

EqualBlockingAlgorithm

This algorithm is a combination of the previous two. It uses blocking IO and multiplexing. I think that thanks to multiplexing the client has more time to process data in OS/network buffers and thus deplete the buffer more than without multiplexing, which results in less blocking. The blocked writes are also more efficient because they decrease the number of parts a download is split into and thus decrease the amount of overhead associated with re-registration.

Based on these benchmarks, I'm going to call EqualBlockingAlgorithm the winner and make it the default grizzly-sendfile algorithm for now. It is quite easy to override this via the grizzly-sendfile configuration, so one will still be able to pick the algorithm that fits their deployment environment the best. To be honest the results of EBA benchmarks surprised me a bit because I expected ENBA to be the winner in throttled tests, so I'm really glad that I went into the trouble of creating and benchmarking it.

All this work will be part of the 0.3 release, which is due any day now. Follow the project on kenai and subscribe to the announce mailing list if you are interested to hear more news.

[0] a few notes on how I tested - both clients and the grizzly-sendfile-server were running on my mac laptop using Java 6 and server vm. I set the ramp up period to 1min so that the server can warm up and then I started counting downloads for 10min and calculated the result. Files used for the tests were 1KB, 200KB, 500KB, 1MB 20MB and 100MB large and equaly represented. All the tests passed with 0 errors. A download is successful when the length, md5 checksum and http headers match expected values.
[1] performance - ability to process a single download quickly
[2] scalability - ability to process large number of concurrent downloads at acceptable performance
[3] init time - duration from the time a request is made, until the first bytes of the http response body (not headers) are received
[4] avg speed - for 100MB files. The smaller the file the lower the effective download speed because of the overhead associated with a download execution.
[5] EBA (81k) - EqualBlockingAlgorithm with buffer size 81KB (the default socket buffer size on mac)
[6] EBA (40k) - EqualBlockingAlgorithm with buffer size 40KB

Identifying ThreadLocal Memory Leaks in JavaEE Web Apps

2009-03-31T17:05:00.000-07:00

A few weeks ago wikis.sun.com powered by Confluence "Enterprise" Wiki grew beyond yet another invisible line that triggered intermittent instabilities. Oh boy, how I love these moments. This time the issue was that Confluence just kept on running out of memory. Increasing the heap didn't help, even breaking the 32bit barrier and using a 64bit JVM was not good enough to keep the app running for more than 24 hours.

The Xmx size of the heap suggested that something was out of order. It was time to take a heap dump using jmap and check what was consuming so much memory. I tried jhat to analyze the heap dump, but 3.5GB dump was just too much for it. The next tool I used was IBM's Heap Analyzer - a decent tool, which was able to read the dump, but consumed a lot of memory in order to do so (~8GB), and was pretty hard to use once the dump was processed.

While looking for more heap analyzing tools, I found SAP Memory Analyzer, now known as Eclipse Memory Analyzer, a.k.a MAT. I thought "What the heck does SAP know about JVM?" and reluctantly gave it a try, only to find out how prejudiced I was. MAT is a really wonderful tool, which was able to process the heap really quickly, visualize the heap in a easy-to-navigate way, use special algorithms to find suspicious memory regions, and all of that while using only ~2GB of memory. An excellent preso that walks through MAT features and how heap and memory leaks work, can be found here.

Thanks to MAT I was able to create two bug reports for folks at Atlassian (CONF-14988, CONF-14989). The only feature I missed was some kind of PDF or HTML export, but I did quite well with using Skitch to take screenshots and annotate them.

One of the leaks was confirmed right away, while it wasn't clear what was causing the other one. All we knew was that significant amounts of memory were retained via ThreadLocal variables. More debugging was in order.

I got this idea to create a servlet filter, that would inspect the thread-local store for the thread currently processing the request and log any thread-local references that exist before the request is dispatched down the chain and also when it comes back. Such a servlet could be packaged as a Confluence Servlet Filter Plugin, so that it is convenient to develop and deploy it.

There was only one problem with this idea, the thread-local store is a private field of the Thread class and is in fact implemented as an inner class with a package default access - kinda hard to get your hands on to. Thankfully private stuff is not necessarily private in Java, if you get your hands dirty with reflection code:

Thread thread = Thread.currentThread();

Field threadLocalsField = Thread.class.getDeclaredField("threadLocals");
threadLocalsField.setAccessible(true);

Class threadLocalMapKlazz = Class.forName("java.lang.ThreadLocal$ThreadLocalMap");
Field tableField = threadLocalMapKlazz.getDeclaredField("table");
tableField.setAccessible(true);

Object table = tableField.get(threadLocalsField.get(thread));

int threadLocalCount = Array.getLength(table);
StringBuilder sb = new StringBuilder();
StringBuilder classSb = new StringBuilder();


int leakCount = 0;

for (int i=0; i < threadLocalCount; i++) {
    Object entry = Array.get(table, i);
    if (entry != null) {
        Field valueField = entry.getClass().getDeclaredField("value");
        valueField.setAccessible(true);
        Object value = valueField.get(entry);
        if (value != null) {
            classSb.append(value.getClass().getName()).append(", ");
        } else {
            classSb.append("null, ");
        }
        leakCount++;
    }
}

sb.append("possible ThreadLocal leaks: ")
        .append(leakCount)
        .append(" of ")
        .append(threadLocalCount)
        .append(" = [")
        .append(classSb.substring(0, classSb.length() - 2))
        .append("] ");

logger.warn(sb);

A simple plugin like this, was able to confirm that the leaked SAXParser instances are created and stored as thread-local variables somewhere within the code that exports content as PDF. That is good enough info to pinpoint the exact line of code that creates the thread-local instance by BTrace (or code review), but that's a story for a separate blog post.

The morale of the story: ThreadLocal variables are a very powerful feature, which as is common for powerful stuff can result in a lot of nasty things when not used properly. Hopefully all the info I provided to Atlassian will be enough to get a speedy fix for the issue and bring stability to wikis.sun.com - at least until we step over the next "invisible line".

Announcing grizzly-sendfile!

2009-02-05T20:52:00.000-08:00

It's my pleasure to finally announce grizzly-sendfile v0.2 - the first stable version of a project that I started after I got one of those "Sudden Burst of Ideas" last summer.

For people who follow the grizzly development or mediacast.sun.com, this is not exactly hot news. grizzly-sendfile has been used by mediacast since last September and mentioned on the grizzly mailing list several times since then, but I haven't had time to promote it and explain what it does and how it works, so here I go.

If you don't care about my diary notes, skip down to "What is grizzly-sendfile".

A bit of background: the whole story goes back to the end of 2007 when a bunch of us where finishing up the rewrite of mediacast.sun.com in JRuby on Rails. At that time we realized that one of the most painful parts of the rewrite would be implementing the file streaming functionality. Back then Rails was single-threaded (not any more, yay!), so sending the data from rails was not an option. Fortunately, my then-colleague Peter, came up with an idea to use a servlet filter to intercept empty download responses from rails and stream the files from this filter. That did the trick for us, but it was a pretty ugly solution that was unreliable from time to time and was PITA to extend and maintain.

At around this time, I learned about X-Sendfile - a not well known http header - that some webservers (e.g. apache and ligttpd) support. This header could be used to offload file transfers from an application to the web server. Rails supports it natively via the :x_sendfile option of send_file method.

I started looking for the X-Sendfile support in GlassFish, which we have been using at mediacast, but it was missing. After some emails with glassfish and grizzly folks, mainly Jean-Francois, I learned that the core component of glassfish called grizzly could be extended via custom filters, which could implement this functionality.

The idea stuck in my head for a few weeks. I looked up some info on grizzly and NIO and then during one overnight drive to San Diego, I designed grizzly-sendfile in my head. It took many nights and a few weekends to get it into reasonable shape and test it under load with some custom faban benchmarks that I had to write, but in late August I had version 0.1 and was able to "sell" it to Rama as a replacement of the servlet filter madness that we were using at mediacast.

Except for a few initial bugs that showed up under some unusual circumstances, the 0.1 version was very stable. A few minor 0.1.x releases were followed by 0.2 version, which was installed on mediacast servers some time in November. Since then I've worked on docs and setting up the project at kenai.com.

What is grizzly-sendfile?

From the wiki: grizzly-sendfile is an extension for grizzly - a NIO framework that among other things powers GlassFish application server.

The goal of this extension is to facilitate an efficient file transfer functionality, which would allow applications to delegate file transfers to the application server, while retaining control over which file to send, access control or any other application specific logic.

How does it work?

By mixing some NIO "magic" and leveraging code of the hard working grizzly team, I was able to come up with an ARP (asynchronous request processing) filter for grizzly. This filter can be easily plugged in to grizzly (and glassfish v2) and will intercept all the responses that contain X-Sendfile header. The value of this header is the path of the file that the application that processed the request wants to send to the client.

All that an application needs to do is to set the header. In Java a simple example of such a code looks like this:

 response.setHeader("X-Sendfile", "/path/to/file.avi");

In Rails, it looks even nicer:

send_file '/path/to.png', :x_sendfile => true

That's it, grizzly-sendfile will take care of the rest.

Why should you care?

For me it was all about keeping my code clean and solving problems at layers where they made the most sense. Then it was also about performance and scalability - the kind of stuff that one can do with NIO, can't be now done in JavaEE because of its synchronous nature. And then of course it was about having full control over downloads (like successful download notification and other customizations that are possible via grizzly-sendfile plugins). Oh, and I must mention JMX monitoring:

What's next?

There is a lot of stuff on my roadmap. Two of the main missing features are partial downloads and glassfish v3 (grizzly 1.9.x) support. Then there is better monitoring and tons of performance and scalability tuning, which I haven't really focus on yet. A lot of the API still needs to be polished and cleaned-up. Also needed is a solid test suite that is more fine grained than the system/integration tests that I created with faban.

Can you use grizzly-sendfile?

Yeah, go for it. This is my pet project that I developed in my free time. The project is licensed under GPL2, so you can even grab the code if you want.

Can you help?

Sure. Code reviews, patches, suggestions and help with testing and documentation are more than welcome!

Benchmarking JRuby on Rails

2009-01-31T13:39:00.000-08:00

Last night, while working on a project I found a really neat use of Rails Components, but I also noticed that this part of Rails is deprecated, among other reasons because it's slow.

Well, how slow? During my quest to find out, I collected some interesting data, and even more importantly put JRuby and MRI Ruby face to face.

Disclaimer: the benchmarks were not done on a well isolated and specially configured test harness, but I did my best to gather data with informational value. All the components were used with OOB settings.

Setup

ruby 1.8.6 (2008-03-03 patchlevel 114) [universal-darwin9.0] + Mongrel Web Server 1.1.4
jruby 1.1.6 (ruby 1.8.6 patchlevel 114) (2008-12-17 rev 8388) [x86_64-java] + GlassFish gem version: 0.9.2
common backend: mysql5 5.0.75 Source distribution (InnoDB table engine, Rails pool set to 30)

Benchmarks

I used an excellent high quality benchmarking framework Faban for my tests. I was lazy, so I only used fhb (very similar to ab, but without its flaws) to invoke simple benchmarks:

simple request benchmark: bin/fhb -r 60/120/5 -c 10 http://localhost:3000/buckets/1
component request benchmark: bin/fhb -r 60/120/5 -c 10 http://localhost:3000/bucket1/object1

Both tests were run with JRuby as well as with RMI Ruby and in addition to that I ran the tests with Rails in single-threaded as well as multi-threaded modes. I didn't use mongler clusters or glassfish pooled instances - there was always only one Ruby instance serving all the requests.

Results

ruby 1.8.6 + mongrel
---------------------------------
simple action + single-threaded:
ops/sec: 210.900
% errors: 0.0
avg. time: 0.047
max time: 0.382
90th %: 0.095

simple action + multi-threaded:
ops/sec: 226.483
% errors: 0.0
avg. time: 0.044
max time: 0.180
90th %: 0.095

component action + single-threaded:
ops/sec: 132.950
% errors: 0.0
avg. time: 0.075
max time: 0.214
90th %: 0.130

component action + multi-threaded:
ops/sec: 131.775
% errors: 0.0
avg. time: 0.076
max time: 0.279
90th %: 0.125

jruby 1.2.6 + glassfish gem 0.9.2
----------------------------------
simple action + single-threaded:
ops/sec: 141.417
% errors: 0.0
avg. time: 0.070
max time: 0.259
90th %: 0.115

simple action + multi-threaded:
ops/sec: 247.333
% errors: 0.0
avg. time: 0.040
max time: 0.318
90th %: 0.065

component action + single-threaded:
ops/sec: 107.858
% errors: 0.0
avg. time: 0.092
max time: 0.595
90th %: 0.145

component action + multi-threaded:
ops/sec: 179.042
% errors: 0.0
avg. time: 0.055
max time: 0.357
90th %: 0.085

Platform/Action	Simple	+/-	Component	+/-
Ruby ST	210ops	0%	132ops	0%
Ruby MT	226ops	7.62%	131ops	-0.76%
JRuby ST	141ops	-32.86%	107ops	-18.94%
JRuby MT	247ops	17.62%	179ops	35.61%

(ST - single-threaded; MT - multi-threaded)

Conclusion

From my tests it appears that MRI is faster in single threaded mode, but JRuby makes up for the loss big time in the multi-threaded tests. It's also interesting to see that the multi-threaded mode gives MRI(green threads) a performance boost, but it's nowhere close to the boost that JRuby(native threads) can squeeze out from using multiple threads.

During the tests I noticed that rails was reporting more times spent in the db when using JRuby (2-80ms) compared to MRI (1-3ms). I don't know how reliable this data is but I wonder if this is the bottleneck that is holding JRuby back in the single threaded mode.

Using ZFS with Mac OS X 10.5

2009-01-13T20:43:00.000-08:00

A few days ago I got a new MacBook Pro. While waiting for it to be delivered, I started thinking about how I want to layout the installation of the OS. For a long long time I wanted to try to use ZFS file system on Mac and this looked like a wonderful opportunity. Getting rid of HFS+, which was causing me lots of problems (especially its case insensitive re-incarnation), sounds like a dream come true.

If you've never heard of ZFS before, check out this good 5min screencast of some of the important features.

A brief google search revealed that there are several people using and developing ZFS for Mac. There is a Mac ZFS porting project at http://zfs.macosforge.org and I found a lot of good info at AlBlue's blog.

Some noteworthy info:

The current ZFS port (build 119) is based on ZFS code that shipped with Solaris build 72
It's currently not possible to boot Mac OS X from a ZFS filesystem
Finder integration is not perfect yet - Finder lists a ZFS pool as an unmountable drive under devices
There are several reports of kernel panics, most of which appeared in connection to the use of cheap external USB disks (I haven't experienced any)
There are a bunch of minor issues, which I'm sure will eventually go away.

None of the above was a show stopper for me, so I went ahead with the installation. My plan was simple - repartition the internal hard drive to a small bootable partition and a large partition used by ZFS, which will hold my home directory and other filesystems.

Install ZFS

Even though MacOS X 10.5 comes with ZFS support, it's only a read-only support. In order to be able to really use ZFS, full ZFS implementation must be installed.

The installation is very simple and can be done by following these instructions: http://zfs.macosforge.org/trac/wiki/downloads. Alternatively, AlBlue created a fancy installer for the lazy ones out there.

Repartition Disk

Once ZFS is installed and the OS was rebooted, I could repartition the internal disk. If you are using an external hard drive, you'll most likely need to use zpool command instead.

First let's check what the disk looks like:

$ diskutil list
/dev/disk0
#:                       TYPE NAME                    SIZE       IDENTIFIER
0:      GUID_partition_scheme                        *298.1 Gi   disk0
1:                        EFI                         200.0 Mi   disk0s1
2:                  Apple_HFS boot                    297.8 Gi   disk0s2

Good, the internal disk was identified as /dev/disk0 and it currently contains an EFI (boot) slice and ~300G data slice/partition. Let's repartition the disk so that it contains two data partitions.

$ sudo diskutil resizeVolume disk0s2 40G ZFS tank 257G
Password:
Started resizing on disk disk0s2 boot
Verifying
Resizing Volume
Adjusting Partitions
Formatting new partitions
Formatting disk0s3 as ZFS File System with name tank
[ + 0%..10%..20%..30%..40%..50%..60%..70%..80%..90%..100% ]
Finished resizing on disk disk0
/dev/disk0
#:                       TYPE NAME                    SIZE       IDENTIFIER
0:      GUID_partition_scheme                        *298.1 Gi   disk0
1:                        EFI                         200.0 Mi   disk0s1
2:                  Apple_HFS boot                    39.9 Gi    disk0s2
3:                        ZFS tank                    252.0 Gi   disk0s3

Great, the disk was repartitioned and the existing data partition, which I call boot, was resized into a smaller 40GB partition and the extra space was used to create a ZFS pool called tank. Btw all the data on the boot partition was preserved.

Let's check my new pool:

$ zpool list
NAME                    SIZE    USED   AVAIL    CAP  HEALTH     ALTROOT
tank                    256G    360K    256G     0%  ONLINE     -

$ zpool status
pool: tank
state: ONLINE
status: The pool is formatted using an older on-disk format.  The pool can
 still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'.  Once this is done, the
 pool will no longer be accessible on older software versions.
scrub: none requested
config:

 NAME        STATE     READ WRITE CKSUM
 tank        ONLINE       0     0     0
   disk0s3   ONLINE       0     0     0

errors: No known data errors

The warning above just means that a new ZFS storage format is available but is not used by the current pool. As far as I could find there are no benefits for upgrading to the new format on Mac, but if I did, I would lose compatibility with Macs that have only the read-only ZFS support.

Create Filesystems

So now that the new pool exists, I can create a shiny new filesystem using a single command:

$ sudo zfs create tank/me3x
$ zfs list
NAME        USED  AVAIL  REFER  MOUNTPOINT
tank        388K   252G   270K  /Volumes/tank
tank/me3x    19K   252G    19K  /Volumes/tank/me3x

To configure this new filesystem as my home directory, I created a temporary admin account, logged in under this account and mounted the ZFS fs as /Users/me3x:

$ sudo mv /Users/me3x /Users/me3x.hfs
$ sudo zfs set mountpoint=/Users/me3x tank/me3x
$ sudo cp -rp /Users/me3x.hfs /Users/me3x

That's it. My Mac account now resides on a ZFS file system. Now I can finally enjoy all the benefits of using ZFS on my OpenSolaris box in my office as well as on my Mac. Bye bye HFS, I won't miss you!

Freezing activerecord-jdbc Gems into a Rails Project

2009-01-11T16:16:00.000-08:00

Over the Christmas break a Slovak friend of mine (Hi Martin!) asked me to build a simple book library management app for a school in the Philippines where he's been volunteering for the past year. I thought to my self that if someone can volunteer one year of his life in such an amazing way, I could spend a few hours to help him out too.

Since from his description it was obvious that he was looking for a low maintenance solution, I though that a rails application with an embedded database would be a good choice. I worked with derby (JavaDB) in the past and I knew that derby drivers were already available as an active-record adapter gem, so I thought that it would be pretty simple to set up dev environment using Rails, JRuby, and embedded derby db. Surprisingly there were a few issues along the way.

I started with defining the database config in config/database.yml:

development:
adapter: jdbcderby
database: db/library_development
pool: 5
timeout: 5000
...
...

The database files for the dev db will be stored under RAILS_ROOT/db/library_development

Secondly I specified the gem dependency in config/environment.rb (you gotta love this Rails 2.1+ feature):

Rails::Initializer.run do |config|
...
config.gem "activerecord-jdbcderby-adapter", :version => '0.9', :lib => 'active_record/connection_adapters/jdbcderby_adapter'
...
end

Note that you must specify the :lib parameter, otherwise Rails won't be able to initialize the gem and you'll end up with:

no such file to load -- activerecord-jdbcderby-adapter

So far so good. Now let's install the gems we depend on:

$ jruby -S rake gems:install
(in /Users/me3x/Development/library)
rake aborted!
Please install the jdbcderby adapter: `gem install activerecord-jdbcderby-adapter` (no such file to load -- active_record/connection_adapters/jdbcderby_adapter)

(See full trace by running task with --trace)

Huh? I asked rake to install gems and I get an error that I need to install gems first? It turns out that this error comes from ActiveRecord, which tries to initialize db according to database.yml, and only then environment.rb gets to be read.

Ok, so let's install the db dependencies manually:

$ sudo jruby -S gem install activerecord-jdbcderby-adapter
Password:
JRuby limited openssl loaded. gem install jruby-openssl for full support.
http://wiki.jruby.org/wiki/JRuby_Builtin_OpenSSL
Successfully installed activerecord-jdbc-adapter-0.9
Successfully installed jdbc-derby-10.3.2.1
Successfully installed activerecord-jdbcderby-adapter-0.9
3 gems installed
Installing ri documentation for activerecord-jdbc-adapter-0.9...
Installing ri documentation for jdbc-derby-10.3.2.1...
Installing ri documentation for activerecord-jdbcderby-adapter-0.9...
Installing RDoc documentation for activerecord-jdbc-adapter-0.9...
Installing RDoc documentation for jdbc-derby-10.3.2.1...
Installing RDoc documentation for activerecord-jdbcderby-adapter-0.9...

Cool, let's check if all the dependencies are available:

$ jruby -S rake gems
(in /Users/me3x/Development/library)
- [I] activerecord-jdbcderby-adapter = 0.9
- [I] activerecord-jdbc-adapter = 0.9
- [I] jdbc-derby = 10.3.2.1

I = Installed
F = Frozen
R = Framework (loaded before rails starts)

Yay, all dependencies are installed.

In the past when dependencies couldn't be declared in environment.rb, I found developing with frozen rails and gems much more manageable, especially when the app is being developed by more than one person. This also made for less deployment surprises. With the config.gem defined dependencies, the situation changes quite a bit, but there are situations when it still makes sense to freeze gems into the project. So let's freeze the gems:

$ jruby -S rake gems:unpack:dependencies
(in /Users/me3x/Development/library)
WARNING:  Installing to ~/.gem since /usr/local/jruby/jruby-1.1.6/lib/ruby/gems/1.8 and
   /usr/local/jruby/current/bin aren't both writable.
Unpacked gem: '/Users/me3x/Development/library/vendor/gems/activerecord-jdbcderby-adapter-0.9'
Unpacked gem: '/Users/me3x/Development/library/vendor/gems/activerecord-jdbc-adapter-0.9'
Unpacked gem: '/Users/me3x/Development/library/vendor/gems/jdbc-derby-10.3.2.1'

Looks good, let's check it:

$ jruby -S rake gems
(in /Users/me3x/Development/library)
- [F] activerecord-jdbcderby-adapter = 0.9
- [F] activerecord-jdbc-adapter = 0.9
- [F] jdbc-derby = 10.3.2.1

I = Installed
F = Frozen
R = Framework (loaded before rails starts)

Nice all the dependencies are now frozen!

How to Install a Glassfish Patch

2008-10-30T14:42:00.000-07:00

Recently I've been working quite extensively with glassfish and had a need to apply a patch for some of the core code.

To my big surprise I was not able to find the official way to apply patches to a glassfish installation. I found several patches posted on the issue tracker and paying customers have access to tested and supported patch bundles, which are released outside of the regular release cycle. But even some extensive googling didn't easily reveal how to apply them.

Then luckily I found this discussion on the glassfish mailing list, which describes the process.

To make life easier for others and google to index this information, here is a brief recap:

Create a directory <glassfishroot>/lib/patches
Copy the jar with your patch into this directory
Edit your <glassfishroot>/domain/<domainname>/config/domain.xml and add attribute classpath-prefix="${com.sun.aas.installRoot}/lib/patches/<patchname.jar>" to the <java-config ...> node
Restart your domain

I tested this with Glassfish v2, I'll have to check if it works with the upcoming v3 as well.

My Confluence 3.0 Wishlist

2008-10-13T18:50:00.000-07:00

Confluence 2.9 was released last month and I've seen references to 2.10 in the Confluence issue tracker, so I expect to see it out in 1-2 months. That makes me think about what's next.

As a part of my adventures of working on Sun's external wiki wikis.sun.com, I've been working on Confluence plugins and even the Confluence core code for a year and a half now, adding new features, enhancing the existing features and very often fixing bugs. Sometimes it was trivial to enhance the code or fix a bug, other times it was not, but what I want to write about today are things that were not possible at all without irreversibly forking the code.

Confluence 3.0 should be a version that really deserves to have the first digit incremented. Not because marketing said it's time for that, but because the changes in the application are so significant.

I'm sure that Atlassian has lots of ideas about what Confluence 3.0 should look like, but Atlassian guys, in case you start to run out of ideas, here is my wish list:

Fix the Database Schema

Confluence has been in development for years and the database schema definitely shows that. Since the database is the heart of the application, I think it deserves a lot of attention and major performance boost could be gained by doing a clean up.

Specific improvements:

Establish and in the future enforce naming conventions
Replace all the natural foreign keys with surrogate keys, e.g. user name, spacekey, group name should be replaced with ids in all the referencing tables (this would finally allow CONF-4063 to be implemented)
Add caches for the lower function (patch) and maybe counter caches

Rework the Clustering

Clustering is usually supposed to fulfill two functions: scalability and robustness. In the case of Confluence mainly the second attribute is missing. In fact, I'd go as far as saying that a Confluence cluster is less robust than a single instance of Confluence. Why? Because the way it is implemented makes the entire cluster vulnerable when one node has problems.

I personally experienced several cluster lock-ups or crashes, usually initiated by a separate Confluence bug, in which the effect was multiplied by the clustering code. One of such of these bugs: CONF-12319

Mike's presentation covers quite a few design goals behind the implementation in Confluence. Clustering can really get ugly and complicated and Mike covered it pretty well. Unfortunately the distributed share part of the clustering makes Confluence prone to problems.

One of the clustering goals that Mike emphasizes in his presentation is that clustering should be "admin-friendly" (low admin overhead and easy setup). While I agree with the low overhead part, the easiness of setup should not compromise the goals which clustering is trying to fulfill in the first place. Clustering is for people who are serious about running Confluence, and as such should be expected to be qualified for the job.

Specific improvements:

Either reevaluate the distributed share clustering so that it is super robust, or consider implementing clustering via a centralized share
Avoid shutting down the entire cluster when "cluster panic" is detected. A better solution, which avoids unnecessary downtime, would be to shut down all the nodes, except for the nodes properly clustered with the oldest node.

Clean Up the HTML and CSS Code

The html code that comes out of Confluence is horrendous. While the rendered output looks pretty pleasant, looking under the hood (browsing the source code in a browser) is not recommended for pregnant women, men with ED, high cholesterol, and generally not recommended for people over 50.

Some improvements were done in this area in the recent releases, but all of them were just minor cosmetic surgeries. Confluence really needs major surgery that will bring the html code up to current standards. The benefit of this will be much faster page loads and code that is easier to maintain and enhance.

Specific improvements:

Rewrite most of the templates and macros to make them XHTML 1.0 compliant
Minify and combine javascript and css files (CONF-8622)
Use image sprites to even further speed up page loads (especially in the rich text editor)

Redo the URI Namespace

Human friendly URIs and URLs are becoming more and more important on today's Internet. Confluence is not doing well in this area.

Specific improvements:

/display/MySpace/My+Page - is the /display part really necessary? Can't we do /MySpace/My+Page
/pages/diffpages.action?pageId=2490471&originalId=45714293 - What is this? I don't know. How about: /MyWiki/My+Page/diff/23:22. I think that actually means something. There might be a better format, this is just a thought.
I think in general redoing the URI name space using REST conventions would be interesting.

Improve Atlassian-Renderer

When I was creating come patches for the atlassian-renderer I was surprised to find that atlassian-render, the module responsible for rendering wiki markup into html is full of hardcoded html snippets. The main reason why this surprised me is that most of the Confluence code is pluggable, which allows for parts of the code to be replaced with a better version without a lot of problem. This is not the case with the render. And this presents two problems: it's not possible get Confluence to directly render anything else than html (pdf and doc are only derived from the html), and it's not possible to use anything else than Confluence markup as the input for the renderer.

The first problem makes me unable to render custom output like docbook or to improve the PDF output, which is pretty poor.

The second issue means that all the customers that use Confluence are locked-in because all the content created via Confluence is Confluence-specific and can't be easily moved to a different wiki engine when needed.

In my opinion the sooner all major wiki engine developers settle on one wiki markup standard the sooner we will all be better off. This might be especially difficult for Atlassian to swallow and implement, because they standardized on their own markup that they also use in their other products.

An interesting initiative that is gaining a lot of traction is Creole, a standardized wiki markup. Confluence is one of the few major wiki players that doesn't support this initiative.

Specific improvement:

Split the current renderer into two pluggable parts: parser and renderer
Implement Creole support (CONF-12077)

Improve Developer Documentation

spent countless hours, especially in the beginnings trying to figure out how Confluence works and how Confluence plugins should be written. I learned some new tricks and that's the good part, the bad thing is that the experience could have been much better if the the code contained more javadocs comments and if the plugin interfaces and mainly the configuration file format was better documented.

Specific improvements:

Add JavaDoc comments where missing
Finally provide complete specification and documentation for the plugin config file (JRA-12183)

Thanks! :-)

That's about as much as I can think of for now. There are probably other things that I missed and then there enhancements around security, which I know are already on the roadmap.

I understand that most of the changes above will create incompatibilities with many existing themes and plugins, but hey, Confluence 3 will happen only once EVAR and releases like this are expected to bring major incompatibilities. Data can always be migrated automatically and existing plugins and themes will be migrated when there are people interested in using them and proper migration instructions are provided.

I hope that Confluence 3 will not be a "marketing" release, but instead something really cool that all users and developers will enjoy working with.

Unintrusive but secure passwordless ssh authentication

2008-09-11T20:21:00.000-07:00

On a daily basis I need to log in to many remote servers inside or outside of Sun via SSH, often dozens of times per day. This can get pretty tiresome if you need to type in your password with every log in.

Some suggest setting up so-called "passwordless" authentication by generating ssh keys and specifying empty passphrase for the private key. This will result in passwordless authentication, but will also decreased security. Should anyone get hold of your private key, (s)he'll get access to all of your remote systems.

ssh-agent can help a lot in keeping the security level high and minimizing the number of times you need to type in the password. However, if you use a terminal with tabs or use both local and remote terminals on your workstation, you'll end up running many ssh-agent processes and having to authenticate every time you start such a process, which diminishes most of the conveniences of using ssh-agent.

Frustrated with this situation and with a bit of help from Martin, I created a shell script, which I added to my .bash_profile startup script. All I have to do now is to authenticate when my first terminal session starts and I'm good until the next time I restart my OS. sweeeet...

Here is how you could set it up on a workstation and a remote-server:

First, if you haven't generated your private/public ssh key pair, do that now:

workstation $ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/user/.ssh/id_rsa): **********************
Enter same passphrase again: **********************
Your identification has been saved in id_rsa.
Your public key has been saved in id_rsa.pub.
The key fingerprint is:
01:a6:95:23:1c:74:53:c7:f4:87:07:a2:50:ef:99:16 user@Computer.local

Now make sure that the file system permissions are set up correctly:

workstation $ cd ~/.ssh/
workstation $ ls -l
total 56
-rw-------  1 user  staff  1743 Aug 31 00:13 id_rsa
-rw-r--r--  1 user  staff   398 Aug 31 00:13 id_rsa.pub

The id_rsa file must be readable only by owner, if this is not true, the key will be ignored.

On the remote server you need to authorize your newly generated key pair by appending it's public key to the ~/.ssh/authorized_keys file under your remote home directory:

workstation $ cat ~/.ssh/id_rsa.pub |ssh user@remote-server 'sh -c "cat - >>~/.ssh/authorized_keys"'

Now you can try to log in:

workstation $ ssh user@remote-server
Enter passphrase for /Users/user/.ssh/id_rsa: **********************
Identity added: /Users/user/.ssh/id_rsa (/Users/user/.ssh/id_rsa)
Last login: Thu Sep 11 20:19:19 2008
remote-server $

If you open a new tab in your terminal and try to log in again, you'll be asked to enter the passphrase yet again. This is where my script becomes useful. First download the script from Mediacast: ssh-agent-init.sh and store it somewhere in your home directory

workstation $ mkdir ~/bin
workstation $ cd ~/bin
workstation $ wget http://mediacast.sun.com/users/IgorMinar/media/ssh-agent-init.sh
workstation $ chmod o+x ~/bin/ssh-agent-init.sh

The next (last) step is optional if you want to start the script manually you can skip it.

I wanted to have this script automatically invoked when I start my terminal for the first time in the interactive mode. All I needed to modify were my .bash_profile (used for interactive sessions) and .bashrc (used for non-interactive sessions) startup scripts for my bash shell (modifications are in italics):

workstation $ cat ~/.bash_profile
...
...

. ~/bin/ssh-agent-init.sh

workstation $ cat ~/.bashrc
export NONINTERACTIVE=1
. ~/.bash_profile

(Note: I source the ~/.bash_profile script from the ~/.bashrc script to enable code reuse between the two scripts.)

That's it! If you now try to open a terminal tab for the first time, you'll be asked for passphrase. Once that is done, any new tab or any other shell session created under the same account will reuse the same ssh-agent process.

I've been using this script on my MacOS X laptop as well as OpenSolaris workstation for a few weeks now and it's been working like charm.

Igor Minar's Blog

Change of Status

Thanks for All the Fish

DGC VI: Wiki Organization and Working with the Community

The Purpose

Wiki Organization

Implemented Actions

Things We Need to Work on

Conclusion

DGC V: Customizing and Patching Confluence

Customizing Confluence

Confluence User Macros

Confluence Themes, HTML Headers and Footers

Confluence Plugins

Patching Confluence (source) files

Patching Confluence

Issues to Be Aware of

Reasons for Patching

Patching Methods

Patch Management

Conclusion

DGC IV: Confluence Upgrades

Confluence Release History and Track Record

Staying in Touch and Getting Support

Confluence Upgrades

Dev and Test Deployments and Testing

Picking the Outage Window

Ready, Set, Go!

Conclusion

DGC III: Confluence Configuration and Tuning

Config Files in Confluence Home directory

Clustering

DB Connection Pool

DB Connection

Config Files in WEB-INF/classes

JVM Options

Admin UI

General Configuration

Daily Backup Administration

Manage Referrers

Languages

User macros

PDF Export Language Support

Themes

Cache Statistics

Plugins

Conclusion

DGC II: The JVM Tuning

JDK Version

VM Observability

Heap and Garbage Collection Must Haves

Performance tweaks

Optional Heap and GC tweaks

Resources

Running Multiple Web Apps in one VM

Conclusion

DGC I: The Infrastructure

Confluence cluster

Network

Hardware (cpu, memory, disk)

OS & Filesystem

Virtualization

Web Container

Logs & Monitoring

DB

Backups

Prod, Test, Dev Environments

Conclusion

DevOps Guide to Confluence (DGC)

Improving Satan and Solaris SMF

Testing matters, even with shell scripts

Configuring Common Access Log Format in GlassFish v2 and v3

Running Confluence on GlassFish v3

Using Mercurial Bisect to Find Bugs

grizzly-sendfile to Become an Official Grizzly Module

What this "merger" means for grizzly-sendfile:

What this "merger" means for grizzly:

grizzly-sendfile 0.3 is out!

grizzly-sendfile and Comparison of Blocking and NonBlocking IO

SimpleBlockingAlgorithm

Config Files in `WEB-INF/classes`