Igor Minar's Blog

grizzly-sendfile to Become an Official Grizzly Module

2009-05-28T19:30:00.000-07:00

After a chat with JFA about grizzly-sendfile's future, I'm pleased to announce today that grizzly-sendfile 0.4 will be the first version of grizzly-sendfile released as an official module of grizzly. This is a huge news for grizzly-sendfile and I believe an equally important news for grizzly and its community.

What this "merger" means for grizzly-sendfile:

great opportunity to extend the reach
opportunity to become the default static file handler in Grizzly
aspiration to become the default static file handler in GlassFish v3
more testing and QA
easier and faster access to grizzly developers and contributors

What this "merger" means for grizzly:

contribution of 1 year of research, development and testing time in the area of static http downloads
several times better performance and scalability of http static file downloads
built-in X-Sendfile functionality
better JMX instrumentation for http downloads
and more

If you can't wait for 0.4, go and get recently released version 0.3.

This is a great day for both projects! :-)

Project site: http://grizzly-sendfile.kenai.com/

grizzly-sendfile 0.3 is out!

2009-05-14T00:50:00.000-07:00

After a few months of late night hacking, grizzly-sendfile 0.3 is finally ready for prime time!

New features include:

grizzly 1.9.15 compatibility
glassfish v3b48 compatibility (via an OSGi bundle)
kick-ass performance and scalability
grizzly-sendfile-server
autosendfile mode
Sendfile Algorithms: new EqualBlockingAlgorithm + improved EqualNonBlockingAlgorithm
lots of improvements to grizzly-sendfile-benchmark
maven build system
lots of smaller improvements and bug fixes

I also started using kenai's JIRA for issue tracking. So feel free to file bugs or RFE's there.

Benchmark & Enjoy!

Project Website
The source code
The binaries

grizzly-sendfile and Comparison of Blocking and NonBlocking IO

2009-05-10T19:00:00.000-07:00

From the very early beginnings of my work on grizzly-sendfile (intro) I was curious to compare blocking and non-blocking IO side to side. Since I didn't have any practical experience to understand which one would be more suitable when, I designed grizzly-sendfile to be flexible so that I could try different strategies and come to conclusions based on some real testing rather than theorizing or based on the words of others. In this post I'd like to compare blocking and nonblocking IO, benchmark them, and draw some conclusions as to which one is more suitable for specific situations.

grizzly-sendfile has a notion of algorithms that control the IO operations responsible for writing data to a SocketChannel (grizzly-sendfile is based on NIO and leverages lots of great work put into grizzly). Different algorithms can do this in different ways, and this is explained in depth on the project wiki. The point is that this allows me to create algorithms that use blocking or nonblocking IO in different ways, and easily swap them and compare their performance (in a very isolated manner).

Two algorithms I implemented right away were SimpleBlockingAlgorithm (SBA) and EqualNonBlockingAlgorithm (ENBA), and only recently followed by EqualBlockingAlgorithm (EBA). The first one employs the traditional approach of sending a file via the network (while not EOF write data to a SocketChannel using blocking writes), while ENBA uses non-blocking writes and Selector re-registration (in place of blocking) to achieve the same task. This means that a download is split into smaller parts, each sequentially streamed by an assigned worker thread to the client. EBA works very similarly to ENBA, but uses blocking writes.

I ran two variations of my faban benchmark ^[0] against these three algorithms. At first I made my simulated clients hit the server as often as possible and download files of different sizes as quickly as possible. Afterward I throttled the file download speed to 1MB/s per client (throttling was done by clients). While the first benchmark simulates traffic close to the one on the private network in a datacenter, the second benchmarks better represents client/server traffic on the Internet.

grizzly-sendfile delegates the execution of the selected algorithm to a pool of worker threads, so the maximum number of the treads in the pool, along with the selected algorithm, are one of the major factors that affects the performance^[1] and scalability^[2] of the server. In my tests I kept the pool size relatively small (50 threads), in order to easily simulate situations when there are more concurrent requests than the number of workers, which is common during traffic spikes.

Conc. Clients	Download limit	Algorithm	Avg init time^[3] (sec)	Avg speed^[4] (MB/s)	Avg total throughput (MB/s)
50	none	SBA	0.019	4.36	208.76
50	none	ENBA	0.021	4.15	198.79
50	none	EBA	0.018	4.23	202.29

100	none	SBA	4.666	4.32	212.79
100	none	ENBA	0.048	1.84	168.15
100	none	EBA	0.140	1.96	175.71

200	none	SBA	14.288	4.31	208.59
200	none	ENBA	0.108	0.87	144.69
200	none	EBA	0.264	0.97	158.83

Conc. Clients	Download limit	Algorithm	Avg init time^[3] (sec)	Avg speed^[4] (MB/s)	Avg total throughput (MB/s)
50	1MB/s	SBA	0.003	1.0	42.9
50	1MB/s	ENBA	0.002	0.98	41.82
50	1MB/s	EBA (81k)^[5]	0.003	0.98	41.91
50	1MB/s	EBA (40k)^[6]	0.002	0.98	42.85

100	1MB/s	SBA	20.5	1.0	40.4
100	1MB/s	ENBA	0.003	0.99	85.14
100	1MB/s	EBA (81k)	0.018	1.0	84.19
100	1MB/s	EBA (40k)	0.013	0.99	84.34

200	1MB/s	SBA	64.8	1.0	37.12
200	1MB/s	ENBA	0.112	0.86	141.8
200	1MB/s	EBA (81k)	0.2	0.95	156.59
200	1MB/s	EBA (40k)	0.159	0.96	154.2

300	1MB/s	SBA	113.9	1.0	34.2
300	1MB/s	ENBA	0.185	0.58	127.53
300	1MB/s	EBA (81k)	0.31	0.61	133.66
300	1MB/s	EBA (40k)	0.239	0.63	132.75

The interpretation of the results is that with SBA the individual download speeds are slightly higher than when ENBA or EBA is used (this is mainly due to some extra scheduling related to the Selector reregistration). However, EBA and ENBA can utilize the available server bandwidth significantly better than SBA. This is especially true when there is a large number of slow clients downloading larger files. One big downside of SBA is that if the number of concurrent downloads is higher than the number of worker threads, the time to initiate download easily increases into extreme heights.

The conclusion is that SBA is well suited for controlled environments where there is a small number of fast clients (server-to-server communication on an internal network), while EBA shines in environments where there is very little control over the number and speed of clients (file hosting on the Internet). While EBA performs and scales better than ENBA, two advantages that ENBA has over EBA are smaller latency and higher resiliency to DoS or DDoS attacks when malicious clients open connections and block.

The results above do not represent well the performance and throughput of grizzly-sendfile (the benchmarks were run on my laptop!), but they certainly provide some evidence that can be used to determine characteristics of different algorithms. I'll do some more thorough testing later when I feel that grizzly-sendfile has enough features and it is time for some fine tuning (there is still a lot that can be done to make things more efficient).

I love explaining things visually, so I drew the following diagrams that describe the differences between the two algorithms much better than many words.

SimpleBlockingAlgorithm

The server can concurrently handle only the same number of requests as the number of worker threads in the pool. All the extra downloads have to be queued until one of some workers completes a download. The downloads take shorter to process, but the worker is blocked while a write is blocked. The slower the client the more blocking occurs and the worker utilization (and server throughput) goes down.

EqualNonBlockingAlgorithm

The number of downloads a server can concurrently handle is not limited by the number of workers. The downloads take slightly longer to process, but the workers are much better utilized. The increase in utilization is due to the fact that no blocking occurs and workers can multiplex - "pause" downloads for which clients are processing the data (stored in OS/network buffers). In the meantime workers can serve whichever client is ready to receive more data. This causes the download to be split into several parts, each possibly served by a different worker thread.

EqualBlockingAlgorithm

This algorithm is a combination of the previous two. It uses blocking IO and multiplexing. I think that thanks to multiplexing the client has more time to process data in OS/network buffers and thus deplete the buffer more than without multiplexing, which results in less blocking. The blocked writes are also more efficient because they decrease the number of parts a download is split into and thus decrease the amount of overhead associated with re-registration.

Based on these benchmarks, I'm going to call EqualBlockingAlgorithm the winner and make it the default grizzly-sendfile algorithm for now. It is quite easy to override this via the grizzly-sendfile configuration, so one will still be able to pick the algorithm that fits their deployment environment the best. To be honest the results of EBA benchmarks surprised me a bit because I expected ENBA to be the winner in throttled tests, so I'm really glad that I went into the trouble of creating and benchmarking it.

All this work will be part of the 0.3 release, which is due any day now. Follow the project on kenai and subscribe to the announce mailing list if you are interested to hear more news.

[0] a few notes on how I tested - both clients and the grizzly-sendfile-server were running on my mac laptop using Java 6 and server vm. I set the ramp up period to 1min so that the server can warm up and then I started counting downloads for 10min and calculated the result. Files used for the tests were 1KB, 200KB, 500KB, 1MB 20MB and 100MB large and equaly represented. All the tests passed with 0 errors. A download is successful when the length, md5 checksum and http headers match expected values.
[1] performance - ability to process a single download quickly
[2] scalability - ability to process large number of concurrent downloads at acceptable performance
[3] init time - duration from the time a request is made, until the first bytes of the http response body (not headers) are received
[4] avg speed - for 100MB files. The smaller the file the lower the effective download speed because of the overhead associated with a download execution.
[5] EBA (81k) - EqualBlockingAlgorithm with buffer size 81KB (the default socket buffer size on mac)
[6] EBA (40k) - EqualBlockingAlgorithm with buffer size 40KB

Identifying ThreadLocal Memory Leaks in JavaEE Web Apps

2009-03-31T17:05:00.000-07:00

A few weeks ago wikis.sun.com powered by Confluence "Enterprise" Wiki grew beyond yet another invisible line that triggered intermittent instabilities. Oh boy, how I love these moments. This time the issue was that Confluence just kept on running out of memory. Increasing the heap didn't help, even breaking the 32bit barrier and using a 64bit JVM was not good enough to keep the app running for more than 24 hours.

The Xmx size of the heap suggested that something was out of order. It was time to take a heap dump using jmap and check what was consuming so much memory. I tried jhat to analyze the heap dump, but 3.5GB dump was just too much for it. The next tool I used was IBM's Heap Analyzer - a decent tool, which was able to read the dump, but consumed a lot of memory in order to do so (~8GB), and was pretty hard to use once the dump was processed.

While looking for more heap analyzing tools, I found SAP Memory Analyzer, now known as Eclipse Memory Analyzer, a.k.a MAT. I thought "What the heck does SAP know about JVM?" and reluctantly gave it a try, only to find out how prejudiced I was. MAT is a really wonderful tool, which was able to process the heap really quickly, visualize the heap in a easy-to-navigate way, use special algorithms to find suspicious memory regions, and all of that while using only ~2GB of memory. An excellent preso that walks through MAT features and how heap and memory leaks work, can be found here.

Thanks to MAT I was able to create two bug reports for folks at Atlassian (CONF-14988, CONF-14989). The only feature I missed was some kind of PDF or HTML export, but I did quite well with using Skitch to take screenshots and annotate them.

One of the leaks was confirmed right away, while it wasn't clear what was causing the other one. All we knew was that significant amounts of memory were retained via ThreadLocal variables. More debugging was in order.

I got this idea to create a servlet filter, that would inspect the thread-local store for the thread currently processing the request and log any thread-local references that exist before the request is dispatched down the chain and also when it comes back. Such a servlet could be packaged as a Confluence Servlet Filter Plugin, so that it is convenient to develop and deploy it.

There was only one problem with this idea, the thread-local store is a private field of the Thread class and is in fact implemented as an inner class with a package default access - kinda hard to get your hands on to. Thankfully private stuff is not necessarily private in Java, if you get your hands dirty with reflection code:


Thread thread = Thread.currentThread();

Field threadLocalsField = Thread.class.getDeclaredField("threadLocals");
threadLocalsField.setAccessible(true);

Class threadLocalMapKlazz = Class.forName("java.lang.ThreadLocal$ThreadLocalMap");
Field tableField = threadLocalMapKlazz.getDeclaredField("table");
tableField.setAccessible(true);

Object table = tableField.get(threadLocalsField.get(thread));

int threadLocalCount = Array.getLength(table);
StringBuilder sb = new StringBuilder();
StringBuilder classSb = new StringBuilder();


int leakCount = 0;

for (int i=0; i < threadLocalCount; i++) {
    Object entry = Array.get(table, i);
    if (entry != null) {
        Field valueField = entry.getClass().getDeclaredField("value");
        valueField.setAccessible(true);
        Object value = valueField.get(entry);
        if (value != null) {
            classSb.append(value.getClass().getName()).append(", ");
        } else {
            classSb.append("null, ");
        }
        leakCount++;
    }
}

sb.append("possible ThreadLocal leaks: ")
        .append(leakCount)
        .append(" of ")
        .append(threadLocalCount)
        .append(" = [")
        .append(classSb.substring(0, classSb.length() - 2))
        .append("] ");

logger.warn(sb);

A simple plugin like this, was able to confirm that the leaked SAXParser instances are created and stored as thread-local variables somewhere within the code that exports content as PDF. That is good enough info to pinpoint the exact line of code that creates the thread-local instance by BTrace (or code review), but that's a story for a separate blog post.

The morale of the story: ThreadLocal variables are a very powerful feature, which as is common for powerful stuff can result in a lot of nasty things when not used properly. Hopefully all the info I provided to Atlassian will be enough to get a speedy fix for the issue and bring stability to wikis.sun.com - at least until we step over the next "invisible line".

Announcing grizzly-sendfile!

2009-02-05T20:52:00.000-08:00

It's my pleasure to finally announce grizzly-sendfile v0.2 - the first stable version of a project that I started after I got one of those "Sudden Burst of Ideas" last summer.

For people who follow the grizzly development or mediacast.sun.com, this is not exactly hot news. grizzly-sendfile has been used by mediacast since last September and mentioned on the grizzly mailing list several times since then, but I haven't had time to promote it and explain what it does and how it works, so here I go.

If you don't care about my diary notes, skip down to "What is grizzly-sendfile".

A bit of background: the whole story goes back to the end of 2007 when a bunch of us where finishing up the rewrite of mediacast.sun.com in JRuby on Rails. At that time we realized that one of the most painful parts of the rewrite would be implementing the file streaming functionality. Back then Rails was single-threaded (not any more, yay!), so sending the data from rails was not an option. Fortunately, my then-colleague Peter, came up with an idea to use a servlet filter to intercept empty download responses from rails and stream the files from this filter. That did the trick for us, but it was a pretty ugly solution that was unreliable from time to time and was PITA to extend and maintain.

At around this time, I learned about X-Sendfile - a not well known http header - that some webservers (e.g. apache and ligttpd) support. This header could be used to offload file transfers from an application to the web server. Rails supports it natively via the :x_sendfile option of send_file method.

I started looking for the X-Sendfile support in GlassFish, which we have been using at mediacast, but it was missing. After some emails with glassfish and grizzly folks, mainly Jean-Francois, I learned that the core component of glassfish called grizzly could be extended via custom filters, which could implement this functionality.

The idea stuck in my head for a few weeks. I looked up some info on grizzly and NIO and then during one overnight drive to San Diego, I designed grizzly-sendfile in my head. It took many nights and a few weekends to get it into reasonable shape and test it under load with some custom faban benchmarks that I had to write, but in late August I had version 0.1 and was able to "sell" it to Rama as a replacement of the servlet filter madness that we were using at mediacast.

Except for a few initial bugs that showed up under some unusual circumstances, the 0.1 version was very stable. A few minor 0.1.x releases were followed by 0.2 version, which was installed on mediacast servers some time in November. Since then I've worked on docs and setting up the project at kenai.com.

What is grizzly-sendfile?

From the wiki:
grizzly-sendfile is an extension for grizzly - a NIO framework that among other things powers GlassFish application server.

The goal of this extension is to facilitate an efficient file transfer functionality, which would allow applications to delegate file transfers to the application server, while retaining control over which file to send, access control or any other application specific logic.

How does it work?

By mixing some NIO "magic" and leveraging code of the hard working grizzly team, I was able to come up with an ARP (asynchronous request processing) filter for grizzly. This filter can be easily plugged in to grizzly (and glassfish v2) and will intercept all the responses that contain X-Sendfile header. The value of this header is the path of the file that the application that processed the request wants to send to the client.

All that an application needs to do is to set the header. In Java a simple example of such a code looks like this:

 response.setHeader("X-Sendfile", "/path/to/file.avi");

In Rails, it looks even nicer:

send_file '/path/to.png', :x_sendfile => true

That's it, grizzly-sendfile will take care of the rest.

Why should you care?

For me it was all about keeping my code clean and solving problems at layers where they made the most sense. Then it was also about performance and scalability - the kind of stuff that one can do with NIO, can't be now done in JavaEE because of its synchronous nature. And then of course it was about having full control over downloads (like successful download notification and other customizations that are possible via grizzly-sendfile plugins). Oh, and I must mention JMX monitoring:

What's next?

There is a lot of stuff on my roadmap. Two of the main missing features are partial downloads and glassfish v3 (grizzly 1.9.x) support. Then there is better monitoring and tons of performance and scalability tuning, which I haven't really focus on yet. A lot of the API still needs to be polished and cleaned-up. Also needed is a solid test suite that is more fine grained than the system/integration tests that I created with faban.

Can you use grizzly-sendfile?

Yeah, go for it. This is my pet project that I developed in my free time. The project is licensed under GPL2, so you can even grab the code if you want.

Can you help?

Sure. Code reviews, patches, suggestions and help with testing and documentation are more than welcome!

Benchmarking JRuby on Rails

2009-01-31T13:39:00.000-08:00

Last night, while working on a project I found a really neat use of Rails Components, but I also noticed that this part of Rails is deprecated, among other reasons because it's slow.

Well, how slow? During my quest to find out, I collected some interesting data, and even more importantly put JRuby and MRI Ruby face to face.

Disclaimer: the benchmarks were not done on a well isolated and specially configured test harness, but I did my best to gather data with informational value. All the components were used with OOB settings.

Setup

ruby 1.8.6 (2008-03-03 patchlevel 114) [universal-darwin9.0] + Mongrel Web Server 1.1.4
jruby 1.1.6 (ruby 1.8.6 patchlevel 114) (2008-12-17 rev 8388) [x86_64-java] + GlassFish gem version: 0.9.2
common backend: mysql5 5.0.75 Source distribution (InnoDB table engine, Rails pool set to 30)

Benchmarks

I used an excellent high quality benchmarking framework Faban for my tests. I was lazy, so I only used fhb (very similar to ab, but without its flaws) to invoke simple benchmarks:

simple request benchmark: bin/fhb -r 60/120/5 -c 10 http://localhost:3000/buckets/1
component request benchmark: bin/fhb -r 60/120/5 -c 10 http://localhost:3000/bucket1/object1

Both tests were run with JRuby as well as with RMI Ruby and in addition to that I ran the tests with Rails in single-threaded as well as multi-threaded modes. I didn't use mongler clusters or glassfish pooled instances - there was always only one Ruby instance serving all the requests.

Results

ruby 1.8.6 + mongrel
---------------------------------
simple action + single-threaded:
ops/sec: 210.900
% errors: 0.0
avg. time: 0.047
max time: 0.382
90th %: 0.095

simple action + multi-threaded:
ops/sec: 226.483
% errors: 0.0
avg. time: 0.044
max time: 0.180
90th %: 0.095

component action + single-threaded:
ops/sec: 132.950
% errors: 0.0
avg. time: 0.075
max time: 0.214
90th %: 0.130

component action + multi-threaded:
ops/sec: 131.775
% errors: 0.0
avg. time: 0.076
max time: 0.279
90th %: 0.125

jruby 1.2.6 + glassfish gem 0.9.2
----------------------------------
simple action + single-threaded:
ops/sec: 141.417
% errors: 0.0
avg. time: 0.070
max time: 0.259
90th %: 0.115

simple action + multi-threaded:
ops/sec: 247.333
% errors: 0.0
avg. time: 0.040
max time: 0.318
90th %: 0.065

component action + single-threaded:
ops/sec: 107.858
% errors: 0.0
avg. time: 0.092
max time: 0.595
90th %: 0.145

component action + multi-threaded:
ops/sec: 179.042
% errors: 0.0
avg. time: 0.055
max time: 0.357
90th %: 0.085

Platform/Action	Simple	+/-	Component	+/-
Ruby ST	210ops	0%	132ops	0%
Ruby MT	226ops	7.62%	131ops	-0.76%
JRuby ST	141ops	-32.86%	107ops	-18.94%
JRuby MT	247ops	17.62%	179ops	35.61%

(ST - single-threaded; MT - multi-threaded)

Conclusion

From my tests it appears that MRI is faster in single threaded mode, but JRuby makes up for the loss big time in the multi-threaded tests. It's also interesting to see that the multi-threaded mode gives MRI(green threads) a performance boost, but it's nowhere close to the boost that JRuby(native threads) can squeeze out from using multiple threads.

During the tests I noticed that rails was reporting more times spent in the db when using JRuby (2-80ms) compared to MRI (1-3ms). I don't know how reliable this data is but I wonder if this is the bottleneck that is holding JRuby back in the single threaded mode.

Using ZFS with Mac OS X 10.5

2009-01-13T20:43:00.000-08:00

A few days ago I got a new MacBook Pro. While waiting for it to be delivered, I started thinking about how I want to layout the installation of the OS. For a long long time I wanted to try to use ZFS file system on Mac and this looked like a wonderful opportunity. Getting rid of HFS+, which was causing me lots of problems (especially its case insensitive re-incarnation), sounds like a dream come true.

If you've never heard of ZFS before, check out this good 5min screencast of some of the important features.

A brief google search revealed that there are several people using and developing ZFS for Mac. There is a Mac ZFS porting project at http://zfs.macosforge.org and I found a lot of good info at AlBlue's blog.

Some noteworthy info:

The current ZFS port (build 119) is based on ZFS code that shipped with Solaris build 72
It's currently not possible to boot Mac OS X from a ZFS filesystem
Finder integration is not perfect yet - Finder lists a ZFS pool as an unmountable drive under devices
There are several reports of kernel panics, most of which appeared in connection to the use of cheap external USB disks (I haven't experienced any)
There are a bunch of minor issues, which I'm sure will eventually go away.

None of the above was a show stopper for me, so I went ahead with the installation. My plan was simple - repartition the internal hard drive to a small bootable partition and a large partition used by ZFS, which will hold my home directory and other filesystems.

Install ZFS

Even though MacOS X 10.5 comes with ZFS support, it's only a read-only support. In order to be able to really use ZFS, full ZFS implementation must be installed.

The installation is very simple and can be done by following these instructions: http://zfs.macosforge.org/trac/wiki/downloads. Alternatively, AlBlue created a fancy installer for the lazy ones out there.

Repartition Disk

Once ZFS is installed and the OS was rebooted, I could repartition the internal disk. If you are using an external hard drive, you'll most likely need to use zpool command instead.

First let's check what the disk looks like:

$ diskutil list
/dev/disk0
#:                       TYPE NAME                    SIZE       IDENTIFIER
0:      GUID_partition_scheme                        *298.1 Gi   disk0
1:                        EFI                         200.0 Mi   disk0s1
2:                  Apple_HFS boot                    297.8 Gi   disk0s2

Good, the internal disk was identified as /dev/disk0 and it currently contains an EFI (boot) slice and ~300G data slice/partition.

Let's repartition the disk so that it contains two data partitions.

$ sudo diskutil resizeVolume disk0s2 40G ZFS tank 257G
Password:
Started resizing on disk disk0s2 boot
Verifying
Resizing Volume
Adjusting Partitions
Formatting new partitions
Formatting disk0s3 as ZFS File System with name tank
[ + 0%..10%..20%..30%..40%..50%..60%..70%..80%..90%..100% ]
Finished resizing on disk disk0
/dev/disk0
#:                       TYPE NAME                    SIZE       IDENTIFIER
0:      GUID_partition_scheme                        *298.1 Gi   disk0
1:                        EFI                         200.0 Mi   disk0s1
2:                  Apple_HFS boot                    39.9 Gi    disk0s2
3:                        ZFS tank                    252.0 Gi   disk0s3

Great, the disk was repartitioned and the existing data partition, which I call boot, was resized into a smaller 40GB partition and the extra space was used to create a ZFS pool called tank. Btw all the data on the boot partition was preserved.

Let's check my new pool:

$ zpool list
NAME                    SIZE    USED   AVAIL    CAP  HEALTH     ALTROOT
tank                    256G    360K    256G     0%  ONLINE     -

$ zpool status
pool: tank
state: ONLINE
status: The pool is formatted using an older on-disk format.  The pool can
 still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'.  Once this is done, the
 pool will no longer be accessible on older software versions.
scrub: none requested
config:

 NAME        STATE     READ WRITE CKSUM
 tank        ONLINE       0     0     0
   disk0s3   ONLINE       0     0     0

errors: No known data errors

The warning above just means that a new ZFS storage format is available but is not used by the current pool. As far as I could find there are no benefits for upgrading to the new format on Mac, but if I did, I would lose compatibility with Macs that have only the read-only ZFS support.

Create Filesystems

So now that the new pool exists, I can create a shiny new filesystem using a single command:

$ sudo zfs create tank/me3x
$ zfs list
NAME        USED  AVAIL  REFER  MOUNTPOINT
tank        388K   252G   270K  /Volumes/tank
tank/me3x    19K   252G    19K  /Volumes/tank/me3x

To configure this new filesystem as my home directory, I created a temporary admin account, logged in under this account and mounted the ZFS fs as /Users/me3x:

$ sudo mv /Users/me3x /Users/me3x.hfs
$ sudo zfs set mountpoint=/Users/me3x tank/me3x
$ sudo cp -rp /Users/me3x.hfs /Users/me3x

That's it. My Mac account now resides on a ZFS file system. Now I can finally enjoy all the benefits of using ZFS on my OpenSolaris box in my office as well as on my Mac. Bye bye HFS, I won't miss you!

Freezing activerecord-jdbc Gems into a Rails Project

2009-01-11T16:16:00.000-08:00

Over the Christmas break a Slovak friend of mine (Hi Martin!) asked me to build a simple book library management app for a school in the Philippines where he's been volunteering for the past year. I thought to my self that if someone can volunteer one year of his life in such an amazing way, I could spend a few hours to help him out too.

Since from his description it was obvious that he was looking for a low maintenance solution, I though that a rails application with an embedded database would be a good choice. I worked with derby (JavaDB) in the past and I knew that derby drivers were already available as an active-record adapter gem, so I thought that it would be pretty simple to set up dev environment using Rails, JRuby, and embedded derby db. Surprisingly there were a few issues along the way.

I started with defining the database config in config/database.yml:

development:
adapter: jdbcderby
database: db/library_development
pool: 5
timeout: 5000
...
...

The database files for the dev db will be stored under RAILS_ROOT/db/library_development

Secondly I specified the gem dependency in config/environment.rb (you gotta love this Rails 2.1+ feature):

Rails::Initializer.run do |config|
...
config.gem "activerecord-jdbcderby-adapter", :version => '0.9', :lib => 'active_record/connection_adapters/jdbcderby_adapter'
...
end

Note that you must specify the :lib parameter, otherwise Rails won't be able to initialize the gem and you'll end up with:

no such file to load -- activerecord-jdbcderby-adapter

So far so good. Now let's install the gems we depend on:

$ jruby -S rake gems:install
(in /Users/me3x/Development/library)
rake aborted!
Please install the jdbcderby adapter: `gem install activerecord-jdbcderby-adapter` (no such file to load -- active_record/connection_adapters/jdbcderby_adapter)

(See full trace by running task with --trace)

Huh? I asked rake to install gems and I get an error that I need to install gems first? It turns out that this error comes from ActiveRecord, which tries to initialize db according to database.yml, and only then environment.rb gets to be read.

Ok, so let's install the db dependencies manually:

$ sudo jruby -S gem install activerecord-jdbcderby-adapter
Password:
JRuby limited openssl loaded. gem install jruby-openssl for full support.
http://wiki.jruby.org/wiki/JRuby_Builtin_OpenSSL
Successfully installed activerecord-jdbc-adapter-0.9
Successfully installed jdbc-derby-10.3.2.1
Successfully installed activerecord-jdbcderby-adapter-0.9
3 gems installed
Installing ri documentation for activerecord-jdbc-adapter-0.9...
Installing ri documentation for jdbc-derby-10.3.2.1...
Installing ri documentation for activerecord-jdbcderby-adapter-0.9...
Installing RDoc documentation for activerecord-jdbc-adapter-0.9...
Installing RDoc documentation for jdbc-derby-10.3.2.1...
Installing RDoc documentation for activerecord-jdbcderby-adapter-0.9...

Cool, let's check if all the dependencies are available:

$ jruby -S rake gems
(in /Users/me3x/Development/library)
- [I] activerecord-jdbcderby-adapter = 0.9
- [I] activerecord-jdbc-adapter = 0.9
- [I] jdbc-derby = 10.3.2.1

I = Installed
F = Frozen
R = Framework (loaded before rails starts)

Yay, all dependencies are installed.

In the past when dependencies couldn't be declared in environment.rb, I found developing with frozen rails and gems much more manageable, especially when the app is being developed by more than one person. This also made for less deployment surprises. With the config.gem defined dependencies, the situation changes quite a bit, but there are situations when it still makes sense to freeze gems into the project. So let's freeze the gems:

$ jruby -S rake gems:unpack:dependencies
(in /Users/me3x/Development/library)
WARNING:  Installing to ~/.gem since /usr/local/jruby/jruby-1.1.6/lib/ruby/gems/1.8 and
   /usr/local/jruby/current/bin aren't both writable.
Unpacked gem: '/Users/me3x/Development/library/vendor/gems/activerecord-jdbcderby-adapter-0.9'
Unpacked gem: '/Users/me3x/Development/library/vendor/gems/activerecord-jdbc-adapter-0.9'
Unpacked gem: '/Users/me3x/Development/library/vendor/gems/jdbc-derby-10.3.2.1'

Looks good, let's check it:

$ jruby -S rake gems
(in /Users/me3x/Development/library)
- [F] activerecord-jdbcderby-adapter = 0.9
- [F] activerecord-jdbc-adapter = 0.9
- [F] jdbc-derby = 10.3.2.1

I = Installed
F = Frozen
R = Framework (loaded before rails starts)

Nice all the dependencies are now frozen!

How to Install a Glassfish Patch

2008-10-30T14:42:00.000-07:00

Recently I've been working quite extensively with glassfish and had a need to apply a patch for some of the core code.

To my big surprise I was not able to find the official way to apply patches to a glassfish installation. I found several patches posted on the issue tracker and paying customers have access to tested and supported patch bundles, which are released outside of the regular release cycle. But even some extensive googling didn't easily reveal how to apply them.

Then luckily I found this discussion on the glassfish mailing list, which describes the process.

To make life easier for others and google to index this information, here is a brief recap:

Create a directory <glassfishroot>/lib/patches
Copy the jar with your patch into this directory
Edit your <glassfishroot>/domain/<domainname>/config/domain.xml and add attribute classpath-prefix="${com.sun.aas.installRoot}/lib/patches/<patchname.jar>" to the <java-config ...>

nodeRestart your domain


I tested this with Glassfish v2, I'll have to check if it works with the upcoming v3 as well.



My Confluence 3.0 Wishlist
2008-10-13T18:50:00.000-07:00
Confluence 2.9 was released last month and I've seen references to 2.10 in the Confluence issue tracker, so I expect to see it out in 1-2 months. That makes me think about what's next.

As a part of my adventures of working on Sun's external wiki wikis.sun.com, I've been working on Confluence plugins and even the Confluence core code for a year and a half now, adding new features, enhancing the existing features and very often fixing bugs. Sometimes it was trivial to enhance the code or fix a bug, other times it was not, but what I want to write about today are things that were not possible at all without irreversibly forking the code.

Confluence 3.0 should be a version that really deserves to have the first digit incremented. Not because marketing said it's time for that, but because the changes in the application are so significant.

I'm sure that Atlassian has lots of ideas about what Confluence 3.0 should look like,  but Atlassian guys, in case you start to run out of ideas, here is my wish list:

Fix the Database Schema
Confluence has been in development for years and the database schema definitely shows that. Since the database is the heart of the application, I think it deserves a lot of attention and major performance boost could be gained by doing a clean up.

Specific improvements:Establish and in the future enforce naming conventions
Replace all the natural foreign keys with surrogate keys, e.g. user name, spacekey, group name should be replaced with ids in all the referencing tables (this would finally allow CONF-4063 to be implemented)
Add caches for the lower function (patch) and maybe counter caches
Rework the Clustering
Clustering is usually supposed to fulfill two functions: scalability and robustness. In the case of Confluence mainly the second attribute is missing. In fact, I'd go as far as saying that a Confluence cluster is less robust than a single instance of Confluence. Why? Because the way it is implemented makes the entire cluster vulnerable when one node has problems.

I personally experienced several cluster lock-ups or crashes, usually initiated by a separate Confluence bug, in which the effect was multiplied by the clustering code. One of such of these bugs: CONF-12319

Mike's presentation covers quite a few design goals behind the implementation in Confluence. Clustering can really get ugly and complicated and Mike covered it pretty well. Unfortunately the distributed share part of the clustering makes Confluence prone to problems.

One of the clustering goals that Mike emphasizes in his presentation is that clustering should be "admin-friendly" (low admin overhead and easy setup). While I agree with the low overhead part, the easiness of setup should not compromise the goals which clustering is trying to fulfill in the first place. Clustering is for people who are serious about running Confluence, and as such should be expected to be qualified for the job.

Specific improvements:Either reevaluate the distributed share clustering so that it is super robust, or consider implementing clustering via a centralized share
Avoid shutting down the entire cluster when "cluster panic" is detected. A better solution, which avoids unnecessary downtime, would be to shut down all the nodes, except for the nodes properly clustered with the oldest node.
Clean Up the HTML and CSS Code
The html code that comes out of Confluence is horrendous. While the rendered output looks pretty pleasant, looking under the hood (browsing the source code in a browser) is not recommended for pregnant women, men with ED, high cholesterol, and generally not recommended for people over 50.

Some improvements were done in this area in the recent releases, but all of them were just minor cosmetic surgeries. Confluence really needs major surgery that will bring the html code up to current standards. The benefit of this will be much faster page loads and code that is easier to maintain and enhance.

Specific improvements:Rewrite most of the templates and macros to make them XHTML 1.0 compliant
Minify and combine javascript and css files (CONF-8622)
Use image sprites to even further speed up page loads (especially in the rich text editor)
Redo the URI Namespace
Human friendly URIs and URLs are becoming more and more important on today's Internet. Confluence is not doing well in this area.

Specific improvements:/display/MySpace/My+Page - is the /display part really necessary? Can't we do /MySpace/My+Page
/pages/diffpages.action?pageId=2490471&originalId=45714293 - What is this? I don't know. How about: /MyWiki/My+Page/diff/23:22. I think that actually means something. There might be a better format, this is just a thought.
I think in general redoing the URI name space using REST conventions would be interesting.
Improve Atlassian-Renderer
When I was creating come patches for the atlassian-renderer I was surprised to find that atlassian-render, the module responsible for rendering wiki markup into html is full of hardcoded html snippets. The main reason why this surprised me is that most of the Confluence code is pluggable, which allows for parts of the code to be replaced with a better version without a lot of problem. This is not the case with the render. And this presents two problems: it's not possible get Confluence to directly render anything else than html (pdf and doc are only derived from the html), and it's not possible to use anything else than Confluence markup as the input for the renderer.

The first problem makes me unable to render custom output like docbook or to improve the PDF output, which is pretty poor.

The second issue means that all the customers that use Confluence are locked-in because all the content created via Confluence is Confluence-specific and can't be easily moved to a different wiki engine when needed.

In my opinion the sooner all major wiki engine developers settle on one wiki markup standard the sooner we will all be better off. This might be especially difficult for Atlassian to swallow and implement, because they standardized on their own markup that they also use in their other products.

An interesting initiative that is gaining a lot of traction is Creole, a standardized wiki markup. Confluence is one of the few major wiki players that doesn't support this initiative.

Specific improvement:Split the current renderer into two pluggable parts: parser and renderer
Implement Creole support (CONF-12077)
Improve Developer Documentation
 spent countless hours, especially in the beginnings trying to figure out how Confluence works and how Confluence plugins should be written. I learned some new tricks and that's the good part, the bad thing is that the experience could have been much better if the the code contained more javadocs comments and if the plugin interfaces and mainly the configuration file format was better documented.

Specific improvements:Add JavaDoc comments where missing
Finally provide complete specification and documentation for the plugin config file (JRA-12183)
Thanks! :-)
That's about as much as I can think of for now. There are probably other things that I missed and then there enhancements around security, which I know are already on the roadmap.

I understand that most of the changes above will create incompatibilities with many existing themes and plugins, but hey, Confluence 3 will happen only once EVAR and releases like this are expected to bring major incompatibilities. Data can always be migrated automatically and existing plugins and themes will be migrated when there are people interested in using them and proper migration instructions are provided.

I hope that Confluence 3 will not be a "marketing" release, but instead something really cool that all users and developers will enjoy working with.


Unintrusive but secure passwordless ssh authentication
2008-09-11T20:21:00.000-07:00
On a daily basis I need to log in to many remote servers inside or outside of Sun via SSH, often dozens of times per day. This can get pretty tiresome if you need to type in your password with every log in.

Some suggest setting up so-called "passwordless" authentication by generating ssh keys and specifying empty passphrase for the private key. This will result in passwordless authentication, but will also decreased security. Should anyone get hold of your private key, (s)he'll get access to all of your remote systems.

ssh-agent can help a lot in keeping the security level high and minimizing the number of times you need to type in the password. However, if you use a terminal with tabs or use both local and remote terminals on your workstation, you'll end up running many ssh-agent processes and having to authenticate every time you start such a process, which diminishes most of the conveniences of using ssh-agent.

Frustrated with this situation and with a bit of help from Martin, I created a shell script, which I added to my .bash_profile startup script. All I have to do now is to authenticate when my first terminal session starts and I'm good until the next time I restart my OS. sweeeet...

Here is how you could set it up on a workstation and a remote-server:

First, if you haven't generated your private/public ssh key pair, do that now:workstation $ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/user/.ssh/id_rsa): **********************
Enter same passphrase again: **********************
Your identification has been saved in id_rsa.
Your public key has been saved in id_rsa.pub.
The key fingerprint is:
01:a6:95:23:1c:74:53:c7:f4:87:07:a2:50:ef:99:16 user@Computer.local

Now make sure that the file system permissions are set up correctly:workstation $ cd ~/.ssh/
workstation $ ls -l
total 56
-rw-------  1 user  staff  1743 Aug 31 00:13 id_rsa
-rw-r--r--  1 user  staff   398 Aug 31 00:13 id_rsa.pub
The id_rsa file must be readable only by owner, if this is not true, the key will be ignored.

On the remote server you need to authorize your newly generated key pair by appending it's public key to the ~/.ssh/authorized_keys file under your remote home directory:workstation $ cat ~/.ssh/id_rsa.pub |ssh user@remote-server 'sh -c "cat - >>~/.ssh/authorized_keys"'

Now you can try to log in:workstation $ ssh user@remote-server
Enter passphrase for /Users/user/.ssh/id_rsa: **********************
Identity added: /Users/user/.ssh/id_rsa (/Users/user/.ssh/id_rsa)
Last login: Thu Sep 11 20:19:19 2008
remote-server $

If you open a new tab in your terminal and try to log in again, you'll be asked to enter the passphrase yet again. This is where my script becomes useful. First download the script from Mediacast: ssh-agent-init.sh and store it somewhere in your home directoryworkstation $ mkdir ~/bin
workstation $ cd ~/bin
workstation $ wget http://mediacast.sun.com/users/IgorMinar/media/ssh-agent-init.sh
workstation $ chmod o+x ~/bin/ssh-agent-init.sh

The next (last) step is optional if you want to start the script manually you can skip it.

I wanted to have this script automatically invoked when I start my terminal for the first time in the interactive mode. All I needed to modify were my .bash_profile (used for interactive sessions) and .bashrc (used for non-interactive sessions) startup scripts for my bash shell (modifications are in italics):workstation $ cat ~/.bash_profile
...
...

. ~/bin/ssh-agent-init.sh

workstation $ cat ~/.bashrc
export NONINTERACTIVE=1
. ~/.bash_profile
(Note: I source the ~/.bash_profile script from the ~/.bashrc script to enable code reuse between the two scripts.)

That's it! If you now try to open a terminal tab for the first time, you'll be asked for passphrase. Once that is done, any new tab or any other shell session created under the same account will reuse the same ssh-agent process.

I've been using this script on my MacOS X laptop as well as OpenSolaris workstation for a few weeks now and it's been working like charm.


Enough with the Flash Already
2008-06-30T07:00:00.000-07:00
It often bothers me when Flash is being misused for small visual effects on home pages of websites. Not just that it slows things down, but it also usually breaks these websites terribly when I want to copy&paste stuff or when one doesn't have the Flash plugin installed.

Margarete, has recently bugged me quite a bit about making the front page of her website a bit more "visual". She started with an idea of adding some flash to her website, which I dismissed quickly.

Fortunately I was able to come up with a neat little CSS code, which is way better than Flash for what she was trying to achieve. Don't be mistaken, I didn't even need JavaScript to do all the subtle movements and eager image loading, it's all pure CSS.

The front page:


A small "visual" effect when one mouses over the "Blog" element:


I still plan to enhance the code a bit more to make it faster to load by decreasing the number of HTTP requests needed to load the asset files as well as size of the images, but I'm quite happy with what it looks and feels like already. Check out the live version at www.fgslovakia.com.

A good technical article describing a technique very similar to the one that I used can be found the A List Apart website.


BTrace == DTrace for Java
2008-06-25T07:00:00.000-07:00
Last week, I was trying to nail down a bug in SunWikis that was triggered by some kind of race condition. These kinds of issues are pretty nasty, especially if the application in question is a pretty complex beast, something that Confluence definitely qualifies as.

The debugger was no help because the timing had to be very precise for the issue to occur, and logging was totally useless because of zillions lines of logs that would be difficult to safely filter out. In many cases the information I needed was not present in logs anyway and since my guess was that the bug was coming from 3rd party code, logged data couldn't be easily expanded. DTrace, which I blogged about in my previous post, could have revealed some information, but I think that it would be very difficult to write a good D script that could give me some clues.

While waiting for my dev server to restart, I came across an interesting blog post that caught my eye. It mentioned this thing called BTrace, that I hadn't heard about before. It promised it to be a DTrace aimed at Java apps. With a fair amount of skepticism, I navigated to the BTrace website and started reading. Four things about BTrace earned my interest:It's based on same or very similar principals as DTrace, but specialized for Java apps
No restarts necessary, you can observe a running app by connecting to a JVM identified by a PID
The BTrace programs are written in a language that is a subset of Java, and heavily based on annotations. This was a bit of a turnoff for me  at first, until I found out that the programs don't need to be compiled manually but btrace takes care of that for me. Bravo!
I can call DTrace from my BTrace programs if I need to instrument native code or the OS


Armed with all this info and my eyes sparking, I wrote my first BTrace program and it worked as advertised! No restarts, no compilation, no hassle.

A few hours later, I had a program that was observing exactly those parts of Confluence that I needed. All of this while the application was running uninterrupted. I was able to snoop on how my code was interacting with Confluence and Confluence was in turn interacting with Hibernate in a way that I would have never dreamed of. All of this while concurrent events aimed at triggering the bug were happening.

Running BTrace programs is as easy as this:

$ ./bin/btrace \
  -classpath hibernate-2.1.8-atlassian.jar:spring-hibernate2-2.0.6.jar \
  6266 HibernateSessions.java

Where 6266 is the PID of my app/web server and HibernateSessions.java is my BTrace program.

Parts of the program looks like this (check out the javadoc comments for explanation of the code):

import com.sun.btrace.annotations.*;
import static com.sun.btrace.BTraceUtils.*;
import net.sf.hibernate.HibernateException;
import net.sf.hibernate.collection.PersistentCollection;
import net.sf.hibernate.engine.SessionImplementor;
import org.springframework.orm.hibernate.support.OpenSessionInViewFilter;

@BTrace
public class HibernateSessions {
  /**
   * A thread local variable used to filter out all the events that we
   * are not interested it.
   */
  @TLS private static boolean viaSso = false;
  private static boolean printStack = false;

  /** print a message when a thread enters doFilter method of SsoFilter */
  @OnMethod(
      clazz="com.sun.dse.wikis.auth.SsoFilter",
      method="doFilter"
  )
  public static void enableProbing() {
      println(str(currentThread(), " - Entered SSO Filter"));
      viaSso = true;
  }

  /** print a message when exiting the doFilter method of SsoFilter */
  @OnMethod(
      clazz="com.sun.dse.wikis.auth.SsoFilter",
      method="doFilter",
      location=@Location(Kind.RETURN)
  ) 
  public static void disableProbing() {
      println(str(currentThread(),  " - Exited SSO Filter"));
      viaSso = false;
  }

  /**
   * print an message with a HibernateException is thrown, with detailed 
   * info about the current context
   */
  @OnMethod(
      clazz="net.sf.hibernate.HibernateException",
      method=""
  )
  public static void onthrowreturn(HibernateException self, String s)  {
      println(str(currentThread(), 
              strcat(" - HibernateException was thrown: ",
              strcat(s, strcat(" | collection: ", str(collection))))));
  }

  /**
   * Print a message when Hibernate is attaching a collection to a session.
   * This was crucial info for me to get in order to resolve the issue
   */
  @OnMethod(
      clazz="net.sf.hibernate.collection.PersistentCollection",
      method="setCurrentSession",
      location = @Location(Kind.RETURN)
  )
  public static void setSessionForCollection(boolean returnValue)  {
      if (returnValue == true) {
        println(str(currentThread(),
          strcat(" - a collection was attached to a session. Collection:",
          str(collection))));
  }
}


The result of this was a workaround for my code as well as a pretty detailed bug report that Atlassian confirmed to be a very similar to a confirmed bug that was reported recently. I could hardly achieve this without BTrace or without an in-depth knowledge of how all the Confluence code works.

As great as BTrace is, it is still a beta. If you decide to use it in production environment you should keep that in mind. During the several hours I spent working with BTrace, I experienced one JVM crash/lockup caused by BTrace. I bet that this kind of issues will be ironed out soon, and often this kind of risk is worth undertaking in order to resolve otherwise untraceable bugs.

Since last week, BTrace is part of my toolkit and I already have other mysteries on my ToDo list that I want to shine some light on with BTrace. I applaud the BTrace team for giving us this amazing tool!


DTrace and Java - Observations and Docs
2008-06-24T22:30:00.000-07:00
For a long time I've been hearing about DTrace and how cool it was. I read a lot about it and saw some presentation on observing Java apps with DTrace, but only recently have I found enough free time to extensively play with it. And oh boy, it really is cool. But even with all this coolness, dtracing Java apps feels a bit awkward.

I think there are two reasons for that. The D language, which looks like C, but has a few fundamental differences, is pretty far from what Java developers are used to. And the second reason is the fact that DTrace is in reality instrumenting JVM and not the application running in the JVM. This means that dtracing Java is limited by abilities of the Java DTrace provider and also that the transition between the different layers of the stack can add some awkwardness.

Having said that, DTrace is really awesome and can be enormously helpful in many cases when developing or troubleshooting (Java) apps.

One thing I really miss in DTrace is regex support in any part of D scripts. There is a limited globbing support and some workarounds, but they ain't pretty.

Here are some DTrace resources that I found useful:DTrace Wiki Docs - the complete documentation (I love to see cool docs like this at our SunWikis)
a DTrace HowTo
DTrace for Java at Solaris Internals
DTrace Toolkit - a compilation of DTrace scripts written by DTrace pros



Catching StackOverflowError and a Bug in Regex Implementation in Java
2008-05-26T11:30:00.000-07:00
Shortly after we launched SunWikis almost a year ago, we started having an issue with StackOverflowError, which was occurring when the content of certain wiki pages was being parsed and URLs where being extracted. I documented the issue in CONF-9392.

The problem is not really a bug in Confluence (even though using an overly-complex regular expressions doesn't help the case), but it's a problem in JDK. After a brief search at bugs.sun.com I found the root cause of our StackOverflowError documented as bug 6337993.

Originally I tried to mitigate the situation by increasing the memory reserved for the stack data via the -Xss JVM parameter. This helped a little bit, but wasn't good enough in most cases.

Last week I decided to go against everything I've been taught, and wrote a patch for Confluence that wraps the part of code that results in the StackOverflowError into a try/catch block. I know that any throwables that extend from Error should not be caught by a client code because they usually indicate a failure that only JVM should try to recover from, but IMO in the case of StackOverflowError, the situation is a bit different. That is mainly because before throwing the StackOverflowError, JVM pops the stack, so by the time the code execution gets to the catch block, JVM has already recovered from the error.

I don't claim this to be a solution to the problem, it's just a workaround that works better than increasing the stack size in this particular case. The fact that Confluence doesn't find all the URLs in wiki pages (used mainly to list outgoing links in the page info view) is just a small sacrifice, compared to inability to save or copy the page.

As for the solution, it seems that reimplementing Java's regular expressions library would be the most suitable one. I tried to run a code that fails in Java in JRuby, which uses a port of Oniguruma regex engine for Java and things worked flawlessly and as I read it also gives JRuby a performance boost over java.util.regex.


Mediacast Deployment Diagram
2008-02-08T11:00:00.000-08:00
During our last meeting, Arun suggested that I publish the high-level deployment diagram of mediacast.sun.com. So here it is:



I described everything in textual form in my previous blog entry about mediacast.


JRuby on Rails Rewrite of mediacast.sun.com Launched
2008-01-27T20:16:00.000-08:00
A few days ago, we finally released Mediacast 2.0 - a complete rewrite of the old Mediacast application.



The original application was based on a few servlets, filters, and a lot of JSPs. It was all put together in hurry a few of years ago, when one of those fire-drill requests came to provide Sun employees with a site where they could publish files. The old code was hard to maintain and extend, and that's why we decided to EOL the old code base and write something better from scratch.

I'm a fan of Rails and for the past year I've been amazed by the great progress of the JRuby project. This was one of the reasons why I suggested that we could try to rewrite the app in Rails and deploy it in a regular Java web container, thanks to JRuby and Goldspike. It took some time for Rama to give us a go-ahead, but finally in late September, two other colleagues (with no Rails or JRuby experience) and I started to work on the rewrite alongside our other projects (forums.sun.com and wikis.sun.com).

Many people asked us why JRuby on Rails was picked for this project. Here is an incomplete list of reasons:we were starting from scratch, so we were not tied to any legacy code, and could pick any web framework that runs on Java
proof of concept project to evaluate the technology for other uses across our organization
verify that Rails really delivers the rapid development promise
something new/fun for the team to help balance out the not-so-fun stuff :)

Here is my experience:

Rails and JRuby Learning Curve

I worked on my first Rails project in the summer of 2006 and since then I worked on a few other (internal) Rails projects. But having to teach others and deploy an application externally  using JRuby was a new experience for me.

As far as teaching goes, I can't critique myself, but what I can say is that I had a lot of good help thanks to some Rails, Ruby and JRuby books, and many websites and blogs that have sprung up on the Internet in the last couple of years.

Learning JRuby (with previous Ruby and Java experience) is easy, because most of the time "it just works" and very rarely is the developer aware that the C-based MRI Ruby interpreter is not being used. The ability to access and seamlessly integrate with existing or new Java code is a huge plus without which we would not have been able to launch this project successfully (more on this later).

The deployment part was a different story. The Ruby on Rails application can be deployed in many different ways and JRuby offers a new alternative approach to all of them. Instead of e.g. having Apache webserver reverse-proxy requests to an army of Mongrels, JRuby and Goldspike make it possible to deploy Rails applications in a JavaEE web container. IMO this is much cleaner than anything else that is available in the non-JRuby Rails world. The downside is that this deployment method is quite new and there isn't a lot of documentation and community knowledge about it.

Application Architecture

Our app is a simple database-driven Rails application consisting of a handful of models, 4 controllers, and a bunch of view templates. The only piece of data that is not stored in the database are the actual files, which are stored on the file system.

To be on the safe side when it comes to performance, we employed quite a lot of fragment caching, which sped up our application quite a bit.

Development Environment

Our development environment is based on a self-contained¹ JRuby (1.0.3) on Rails (1.2.6) application stored in a Mercurial repository. The IDE we use is NetBeans 6 with Ruby support and Mercurial plugin. The DB of our choice is MySQL and servers we use during development are WEBrick and Glassfish v2ur1.

NetBeans makes it super easy to write the Ruby code and offers a lot of neat features for Rails application development. I have to say that Tor and the gang did a great job. In fact I switched from Eclipse to NetBeans thanks to its great Ruby support.

Because the application runs in the JVM we can use JConsole to monitor the app while load testing which comes in really handy!

So all is good here, except for one thing. I got used to using the amazing ruby-debug debugger for debugging my Ruby on Rails applications. This debugger is not yet available for JRuby on Rails application (unless you hack your way through). I read somewhere that jruby-debug should be available with NetBeans 6.1. Once that is done, the JRoR dev environment will be on par with RoR (in fact thanks to tools like JConsole it will be superior).

Production Environment

We use a pair of load-balanced T2000 with Solaris 10, JDK6 and SJSAS 9.1u1. These two servers share a nfs NAS drive used for file storage and fragment cache storage. The DB backend is a MySQL database server, which we access through a connection pool set up in the app server.

All of this was fairly easy to set up. As you can see there were no special requirements when comparing this environment to a usual JavaEE production environment.

The Good

Once my colleagues grasped some RoR basics, we got the core of the application up and running fairly fast. Sometimes it still amazes even me how much one can do with Rails in a short amount of time. Thanks to a small amount of code one needs to write, the code review process is fast as well, and fixing a bug often means changing only a few lines of code.

The Bad

Since most of the C-based gems are not compatible out-of-box with JRuby (at the moment), the jruby-extras project aims to deliver JRuby compatible versions of these libraries. One of these libraries - JRuby-OpenSSL - was needed for me to integrate our app with our authentication webservice. I soon found out that JRuby-OpenSSL was not completely implemented yet and parts of functionality that I needed were missing. I'm sure that it is just a matter of time when problems like this will go away (if it hasn't happened already) as these libraries will mature.

The Ugly

The Goldspike project provides a bridge between the JavaEE and RoR world. It does this by dispatching the incoming HTTP requests into Rails running in a JRuby runtime.

This works great for all requests that take little time to process, but if you have long-running requests like large file uploads or downloads, these requests will occupy your JRuby runtimes and you soon realize that you are running out of runtimes in your small pool, at which point your application becomes unresponsive for any new requests.

This was the biggest problem we hit, but thanks to the possibility to seamlessly integrate Java code into our JRoR app and a great idea that my colleague Peter had, we solved this issue quite elegantly.

JRuby and Java Come to the Rescue

The two main issues we faced and that consumed most of my time on this project were related to the immaturity of the libraries around JRuby and application characteristics specific to JRoR deployment. Both of these issues can be resolved thanks to years of long Java and JavaEE history, and their libraries.

The problem with reliable connection to our SOAP-over-HTTPS based authentication webservice was resolved by generating a client Java webservice stub and using that instead of SOAP4R which didn't work properly because of the problems with JRuby-OpenSSL.

The second problem with the long-running processes occupying our precious JRuby runtimes, we solved by using a servlet filter and a fake HttpServletResponse (that we sent to Goldspike instead of the original one) and streaming the data from the filter instead from the runtime. I'll write a separate blog entry on this.

Overall Impression

This project was/is fun to work on. We experimented with quite a few new (for us) technologies and learned a lot along the way. To be honest, I expected to deploy the app much sooner but at that time I was not aware of the two above mentioned issues which consumed a lot of my time.

I was a bit worried about the performance of the application, but that turned out to be a non issue once we had the download servlet filter in place, and with the performance improvements in JRuby 1.1 things will be even better in the future.

Overall I'm happy with the outcome of our project and I look forward to adding more functionality to the application.

Our Future Plans

From the infrastructure perspective:upgrade to JRuby 1.1 and Rails 2.0
start using Warbler, which looks to be superior to Goldspike's Rails plugin, for building the war file
experiment with in-memory session state replication in Glassfish

From the feature perspective:better categorization of media items
search functionality via integration with search.sun.com
previews of media items
new UI design
and the toughest one - audio and video streaming


JRuby/Goldspike/Glassfish Things I Would Like to See Improved

I'm not sure if it is us and our (mis)configuration, Goldspike, JRuby or Glassfish, but right now we need quite a big JVM heap to keep things running. With 8 JRuby instances in the pool and Http thread count set to 512, we need -Xmx set to 2-2.5GB. This is a bit too much I think. We'll have to look into this and find the culprit.
The application startup is quite slow, especially on a machine like T2000, which doesn't perform well for heavy-weighted single-threaded operations. I don't see a reason, why Goldspike couldn't initialize the JRuby runtime pool concurrently cutting down the startup time significantly.
Offload JRuby runtimes as much as possible. Currently operations like file upload or file download (via Rails' send_file) are handled by JRuby runtimes. I think that it should be possible to take care of these operations outside of the runtime, allowing the runtime to process other rails requests in the meantime.



¹ - Rails and all the other gems and jars are frozen into the project


On the Move (Away from Blogger)
2008-01-21T23:59:00.000-08:00
Today was a big day for my blog. I started a "big" move that was hanging over my mind for a few months.

I got my own domain igorminar.com and changed the blogger.com settings so that the domain name blog.igorminar.com is being used instead of net3x.blogspot.com.

Except for a small hick-up with DNS cache and a weird 404 Server not Found error (described in this blog post) everything went smoothly. My plan is to keep on using Blogger's custom domain feature for a few weeks and then migrate all the content over to my own blogging engine. I'll most likely end up using Mephisto, JRuby, Glassfish and MySQL, but things can still change.

The reason why I'm moving in two phases is to keep all the referring links that point to the old urls as well as most of the search engine ranking. The Blogger's custom domain feature now redirects everything to the new urls via 301 Permanent Redirect, so hopefully the transition will be close to painless. I'm not SEO expert, but this for sure is an interesting experiment.

And for those SEO "experts" that were claiming that getting Blogger to do proper redirects after a blog move was impossible here is a little proof:
$ wget --spider net3x.blogspot.com
--16:34:30--  http://net3x.blogspot.com/
      => `index.html.2'
Resolving net3x.blogspot.com... 72.14.207.191
Connecting to net3x.blogspot.com|72.14.207.191|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://blog.igorminar.com/ [following]
--16:34:30--  http://blog.igorminar.com/
      => `index.html.2'
Resolving blog.igorminar.com... 66.249.81.121
Connecting to blog.igorminar.com|66.249.81.121|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 0 [text/html]
200 OK



My Custom Prompt for Leopard's Terminal
2008-01-20T23:20:00.000-08:00
As I mentioned in my previous post, even though it is not perfect, I settled on using Terminal as my terminal application.

The default prompt that comes with MacOS is not very useful, so I configured my own by adding this line to my .bashrc file:
export PS1="\[\033]0;\u@\h:\w\007\]\u@\h:\W\\$ "

This results in a prompt that looks like this:

This prompt displays the prompt as username@hostname:currentdir$ and in addition to that  it sets the window title to username@hostname:pwd.


Leopard's Terminal Tabs
2008-01-20T22:55:00.000-08:00
As I shared yesterday, I love tabs. One of the few good things that MacOS 10.5 Leopard brought, was enhanced Terminal application. A basic feature - Tab support - that was missing in Terminal for way too long, was finally added, and Terminal app became finally usable for anyone who spends a great time of the day working on the command line.

Before Leopard, I used iTerm, but there was just something about it that I didn't like, but the availability of tabs made me get over it. Now with Leopard there is no need for iTerm any more.

Just to point out that Apple seems to be unable to get anything 100% right these days, there are two flaws in the Terminal's tab implementation.

The first flaw is that you can't easily switch between tabs using keyboard shortcut. All you can do is to use cmd+{ and cmd+} to switch to previous or next tab respectively. What a "wonderful" choice of keys that require use of right hand or both hands. Most importantly, one can't use the handy cmd+1, cmd+2, ..., cmd+9 shortcuts (or anything similar) to switch to one of the first nine tabs. Maybe Apple doesn't want to overload us with new features and is planning to add this to Terminal in MacOS 10.6. :-) Fortunately, the cmd+{ and cmd+} shortcuts can be remapped via System Preferences.

The second flaw is that Terminal doesn't let you name the tabs or configure how the tabs should be named. This is a useful feature that many other terminal applications have, but Apple decided that using the name of the program of the currently running process is good enough. Well, this is what it looks like:

People are doing all kinds of crazy things to work around this issue. Including using hardlinks to change names of programs or creating dummy processes that carry the tab name.

I would be very happy if Apple did something similar to what xterm does - using an environmental variable to set name of the tab dynamically. But I have a feeling that we'll see MacOSX Trash go 3D sooner than properly working Terminal tabs. :(


Mac Applications That I Use
2008-01-20T16:15:00.000-08:00
I've been meaning to write this entry on applications installed on my computer for a few months now, but never found time to do it. Hopefully none of those new Mac users that I told to check out my blog "in a few days", several months ago, got angry at me. If nothing else, my promise helped me to get more regular readers :).

Anyway, here is the list of apps installed on my Mac (all these apps work with Leopard, unless I noted otherwise):
Essentials
A Mac without these apps would never be the same:
Quicksilver - an app that helps you find things on your computer and execute actions on these things. I use it to launch apps, pause iTunes, find contacts in the AddressBook, and many many many other things. Most of the cool functionality is added via plugins, so don't forget to checkout the plugin section of preferences. This app is a must have. Here are Some docs and a good video.

Adium - a really handy IM client that can handle all the popular IM protocols. Definitely a must have. (it's much better than iChat when it comes to IM-ing).

Growl - Growl is a notification system for Mac OS X: it allows applications that support Growl to send you notifications. Notifications include Adium or Skype messages, network up/down events, volume mount/unmount events, currently played iTunes song and many others. More info - One extension that I found to be incompatible with Leopard is GrowlMail for Mail.app notifications, when I received many new messages at once this extension would cause the Mail.app to crash.

MacPorts - This is a geeky one: The MacPorts Project is an open-source community initiative to design an easy-to-use system for compiling, installing, and upgrading either command-line, X11 or Aqua based open-source software on the Mac OS X operating system.

Firefox - no description needed :)

Nice to Have
iStat menus - a nifty  app that displays stats about cpu load, cpu temperature, memory and network utilization, and a lot of other interesting data about your Mac. I use it as a replacement of MenuMeters which I couldn't get to run reliably on Leopard.

Perian - Perian is a free, open source QuickTime component that adds native support for many popular video formats.

Camino - yeah, as if the fact that I'm obsessed with tabs was not enough, I usually run 2 or 3 browsers at the same time to be able to take advantage of their unique features, or more often, while developing webapps, I need to be able to have more than one session open.

Chicken of the VNC - A handy VNC client

MacFUSE + sshfs - MacOS X implementation of FUSE, which makes it possible to painlessly mount all kinds of different filesystems e.g. sshfs.

MacFusion - GUI frontend for MacFUSE

VLC - alternative to QuickTime video player with build-in support for all kinds of video and audio codes, subtitle support and many other features

Vine Server (OSXvnc) - a good VNC server in case you need a bit more control than the Leopard's built-in server offers

Skype - I have a love-hate relationship with Skype. As soon as I find an alternative crossplatform communication tool with audio and video support and cheap calls to Slovakia, I'll switch. Maybe Gizmo will become that tool one day

NeoOffice - don't forget to install the latest Patch. I have big hopes for OpenOffice Aqua to be released soon. My initial experiments revealed some very good performance results.

iShowU - a cool app for creating screencasts. This is the only paid-for app on my list. But it's well worth it.

Caffeine - Caffeine is a tiny program that prevents your Mac from automatically going to sleep, dimming the screen or starting screen savers. Very useful during video calls, while reading PDFs or watching movies.

Twitterrific - a Twitter client

Azureus - Java BitTorrent client

SkeyCalc - An OTP (S/Key) calculator

Cisco VPN Client
Development
NetBeans - there is a lot of controversy in the community when it comes to the Eclipse vs NetBeans question. I used Eclipse for a really long time, but ever since I tried NB6 Milestone 8 or so, I started liking NB more than Eclipse. I'll give Eclipse another shot in a few weeks, but for now NetBeans is my default IDE. And as far as Ruby/JRuby development is concerned, I don't think that this is going to change for a long time.

JRuby - a pure Java-based Ruby interpreter

GlassFish - JavaEE 5.0 compliant application server

Other Stuff
Drivers for my Vydeo eSATA 34: Dual Port 34mm Express Card - More info in my older blog entry


Other Lists
Rama and Martin composed similar lists some time ago, so you might want to check those out as well.


I'm Obsessed with Tabs
2008-01-20T01:00:00.000-08:00
This is a screenshot I took during a regular development/research day:



What is your average number of browser tabs opened at once?


MultiSudokuSolver - A fun project I worked on during the Xmas break
2008-01-18T11:45:00.000-08:00
While I was visiting my family in Slovakia during the winter 2005/2006 I found a sudoku magazine in my dad's apartment. I like this kind of stuff, so I mediately started solving one puzzle after another, until I got to a "Speciality MultiSudoku" puzzle:



This puzzle is composed of five regular 9x9 puzzles that are interconnected and you can't solve either of them alone, meaning that you need to be solving all five of them at the same time and use intermediate results from one puzzle to get an intermediate result for another puzzle that this puzzle is connected with. It sounded like a good challenge so I started working on it. After two or three nights I realized that I made a mistake!! Ughhh. I erased everything I had solved and started from scratch. After another few nights I found another problem, got turned off by this puzzle and put it away.

I came across the puzzle once again when my wife and I returned to Slovakia this winter. I found the puzzle in the apartment and, being a person who likes challenges and doesn't get turned off my failures for too long, I decided to erase everything and this time be very careful and solve the puzzle once and for all.

After about two evenings I found something I didn't want to see. An error!!! It was hard to believe it, but there it was. I tried to fix it, but if you don't spot an error in a sudoku puzzle early enough, you'll spend more time fixing it than if you started from scratch.

Do you think I felt like starting from scratch? No! But I couldn't let this be. So since I had 10 or so days off during the Christmas break (thanks Sun!!!), I decided to    use my brain in a more productive way and to write a small program that would solve the puzzle for me.

Given my interest in Ruby and JRuby and the type of problem I was about to solve, the language choice was an easy one to make.

A few days later I had a script that solved the puzzle for me within 1.07 seconds. Ya! You heard right! Solved not in a few evenings but in just a little more than one second.


Top Left:
718|956|324
452|387|619
936|142|587
-----------
341|528|796
695|713|842
287|694|153
-----------
523|461|978
869|275|431
174|839|265

Top Right:
635|428|971
214|679|853
897|531|426
-----------
976|382|514
152|764|389
348|915|267
-----------
463|857|192
529|146|738
781|293|645

Bottom Left:
728|154|396
194|836|752
653|279|184
-----------
519|368|427
346|725|918
872|491|563
-----------
231|987|645
467|512|839
985|643|271

Bottom Right:
215|849|637
348|627|591
976|153|824
-----------
869|431|752
154|972|368
732|568|149
-----------
521|796|483
483|215|976
697|384|215

Center:
978|215|463
431|678|529
265|439|781
-----------
813|754|692
649|382|157
527|961|834
-----------
396|847|215
752|196|348
184|523|976


Lessons learned: "automate automate automate!" and "Don't work hard, but work smart!" :)

The algorithm is very simple. The model is based on the simplest unit - a cell which is part of a row, column and a square. Each puzzle consists of 9 columns, 9 rows and 9 squares. Cells are flexible enough to be part of more than one square, row or a column at a time, so I can have cells that are present in more than one puzzle at a time. With a flexible model like this, all that the program needs to do is to use some basic rules to eliminate candidates and determine cell values. This type of solver is often referred to as human-style solver, because it uses the same techniques used by humans.

If anyone is interested in having a look at the source code it can be downloaded from here: multi_sudoku_solver.rb. A word of warning - the code is not cleaned up nor documented. This was just a fun project that I worked on, and the fact that I achieved what I set out to achieve was a good enough milestone to call this project done.

After this experience I don't feel like solving sudoku puzzles any more. They are just a repetitious problems that are well suited for automated solution. Most importantly - I actually had more fun writing the program than I had with solving the puzzle :).


Shure SE210 Earphones (upgrade from E2c)
2007-11-23T19:20:00.000-08:00
When I bought my Shure E2c earphones two years ago, I was blown away by the sound quality. I never spend more than $50 for earphones before and after buying the E2c model I would never consider downgrading to anything less. Shure really knows how to create addictive high quality products. The quality of the sound produced by my old E2c earphones was so good that I had to get rid of some albums that I ripped into mp3s long time ago when hard drive space was measured in megabytes rather than in gigabytes.

After two years of slave-like everyday service my Shure E2c earphones started to show signs of wear and it became apparent that they were going to retire soon.

Afraid of being without a good pair of earphones during my upcoming transatlantic flight, I got a new pair of Shure SE210. The design has change slightly compared to the older model, but it still preserves the main features that I loved so much.



I've been listening to some music via these earphones for only an hour or so, but I can already tell that SE210 brings "great sound" to a brand new level. I don't consider myself to be an audiophile, but even my untrained ear can hear the big difference between the two. The new pair fits better as well as feels more comfortable in my ears.

J, lost his mind a couple of weeks ago and bought a pair of the high-end SE530. I don't even want to know what those sound like. I know that if I tried them, I would be tempted to pay an insane amount of money for them, so I better say away :)


Inflector/environment.rb Bug in Rails
2007-10-29T09:30:00.000-07:00
Today I needed to get Rails to grok an uncountable noun "media". Easy you think, right? Just add an inflection rule.

So I went to config/environment.rb, uncommented the Inflection example and modified it with my "media" noun, just like this:Rails::Initializer.run do |config|
  ...
  ...
  ...
  
  Inflector.inflections do |inflect|
    inflect.uncountable %w( media )
  end

  # See Rails::Configuration for more options
end

# Add new mime types for use in respond_to blocks:
# Mime::Type.register "text/richtext", :rtf
# Mime::Type.register "application/x-mobile", :mobile


When I tried to run rails console or tests I saw an error like this:
me@Xbook:testapp$ ./script/console 
Loading development environment.
./script/../config/../config/environment.rb:55:NameError: uninitialized constant Inflector
/opt/local/lib/ruby/gems/1.8/gems/actionpack-1.13.5/lib/action_controller/assertions/selector_assertions.rb:525:NoMethodError: undefined method `camelize' for "top":String
/Users/me3x/Development/DSE/mediacast2/apps/testapp/app/controllers/application.rb:4:NameError: uninitialized constant ActionController::Base


The problem turned out to be in the commented-out rails example in environment.rb. The Inflector.inflections block must not be nested within the Initializer.run block.

This fixes the problem:
Rails::Initializer.run do |config|
  ...
  ...
  ...

  # See Rails::Configuration for more options
end

Inflector.inflections do |inflect|
  inflect.uncountable %w( media )
end

# Add new mime types for use in respond_to blocks:
# Mime::Type.register "text/richtext", :rtf
# Mime::Type.register "application/x-mobile", :mobile