Inside Chronon 3 Recorder: How we achieved the magical speed boost

Posted by Prashant Deva on November 9, 2012

We released Chronon 3 last week with an order of magnitude performance improvement. Let’s take a look under the hood to see what makes it all tick!

The Recipe

Look at the source code of any computer program. You can pretty much predict what it is going to do, given you had some extra info like what branch will be taken and/or what the return value of a particular method call is.

In the code below, if you knew the value of ‘a’, you could predict whether ‘even’ or ‘odd’ will be printed.

Chronon 3′s recorder follows a similar principal. It instruments the bytecode of you program as its loaded and does static analysis on it to come up with an initial set of ‘predictions’ as to how your program will behave.

Once your program is running, it keeps monitoring it against the initial set of predictions. If the execution at any time differs from what was ‘predicted’:

1. The prediction data structure is modified to take into account the new information.

2. Recording data is generated to select the right ‘prediction’ from the structure. This usually takes only 1 to 4 bits.

3. If the prediction matches accurately though then no data is generated at all!

See it in Action

For example, consider the code below:

Assume there are 2 threads that call the method process().

The first thread has an object of type Foo and calls Foo.bar(), which matches the prediction made by the recorder exactly, so no data is generated at that point for that thread.

The second thread calls has an object of FooSub instead, and thus ends up calling FooSub.bar().

Now here is where things get interesting. For this thread only, the prediction structure is adjusted to account for this new method call. Then a single bit is stored to select the right method call from the prediction structure. The previous thread, on the other hand, which keeps Foo.bar() keeps generating no data as its prediction is accurately matched.

Results

As you can see that the new prediction mechanism helps in saving a ton of data that would otherwise have been generated and even when data is indeed generated, it’s only very minimal. Apart from the example given above, we have a whole bunch of predictions inside the Chronon 3 Recorder. We now even have a pluggable architecture where we can keep plugging in more of these predictions and constantly increasing the speed of the recorder, without needing to wait an entire year.

So far the current architecture has worked extremely well and almost all our previous users running into OutOfMemory issues are now able to record easily!

The Amazing Chronon 3

Posted by Prashant Deva on November 1, 2012

Its been a long time coming, but Chronon 3 is finally here!

In the next few posts, we will dive deep into what took us so long and all the technical magic that makes Chronon 3 tick. For now lets see what Chronon 3 brings us:

Features

Enormous performance upgrade

The number one issues of Chronon users has been running into OutOfMemoryErrors and all of Chronon 3 is dedicated to eradicating that. The all new, rewritten from scratch Recorder is built to conserve memory. It uses off heap memory as much as possible and even then limits the amount of total data collected by the Recorder to be many times lower than that of previous versions. This graph gives you an idea of the improvement.

5x faster unpacking

The Unpacker is much faster too! Apart from consuming much, much less resources, it also runs 5x faster!

Per Thread Time

With Chronon 3, we are introducing Per Thread Time, which gives each thread its own timeline, facilitating in must easier and faster debugging. Now the timeline view shows you the time and progess of the current thread instead of the whole system, giving you much more focused data. This is carried on in other views too, like the Stack view shows the current thread only instead of cluttering the view with all the threads in the system. The result of this is much more focused data and it also allows us to make the recorder more concurrent and faster.

On Demand Split

In case you missed it, a few months back we also introduced On Demand Split in the Recording Server, allowing you to split recordings anytime, without having to wait for a split interval.

Support for JBoss

Previously, Jboss 7 was having issues with Chronon due to OSGI classloader issues. With Chronon 3, this should be solved and you should be able to fully record your JBoss apps.

Future Updates

Now that we are done with this huge release, we will be doing much, much faster releases to keep bringing you new features. Due to time constraints for this release, we have decided not to include the ‘Recorded Console’ view. We think with Post Execution Logging you can achieve the same goal without cluttering your code with println() statements. However, if there is demand for it, we will bring the view back in the next update.

Renewed Trial

If you don’t own a Chronon subscription and your trial has expired, dont worry! You’ll get a fully renewed trial now when you download Chronon 3. Same goes for students. If your student license has expired, request your student license again and you will get a renewed one too.

Download Now

Stop waiting and go ahead and give the Chronon 3 Debugger and Recording Server a try now!

Chronon 2.5 released

Posted by Prashant Deva on June 29, 2012

While we get Chronon 3 ready, we thought of making an interim release with some goodies our users have been asking for:

Chronon Recording Server

Recording Server is the highlight of this release, and contains the most voted feature from our users:

On Demand Split

Now when you encounter a bug while testing/QA, no longer do you have to wait for an ‘automatic split interval’ or do a ‘Stop Recording’ which results in a lengthy ‘deinstrumentation’ phase and then ‘Start Recording’ again (which will result in another lengthy ‘instrumentation’ phase).

Just click the new ‘Split Recording’ button and a recording is created instantly on disk and the recorder keeps running!

Here is a demo video demonstrating this feature:

Make sure you get the latest version of Recording Server now!

Chronon Time Travelling Debugger

Some enhancements based on user feedback:

Timeline View

An Id column has been added to allow easily referring to each time bookmark by an id, instead of a complex, lengthy ‘time’ value.
All columns are now sortable.

Exceptions View

An Id column has been added here too.

Dont forget to update your debugger!

Embedded Chronon and Chronon Recorder

Plenty of bugfixes and minor performance enhancements in the Chronon Recorder for users of Embedded Chronon and Chronon Recording Server.

If you were getting a corrupt recording when using either Embedded Chronon or Chronon Recording Server, these bugfixes should now prevent that.

Also for the first time we have updated the native agents too, from version 1.0 to 1.1.0.

Chronon 3 beta: 10x performance improvement

Posted by Prashant Deva on March 23, 2012

Chronon 3 is all about performance.

We have rewritten our Recorder from the ground up to gain an order of magnitude performance improvement! The basic architecture remains the same, but under the hood its an all new engine. We turned a Yugo into a Ferrari!

No more memory issues

The biggest complaint of all users of Chronon 1 and 2 has been high memory usage and OutOfMemory errors. With Chronon 3 our main goal was to get rid of this. And I am proud to say we have blown it out of the ballpark with this one!

Chronon 3 makes absolute minimal use of your Java heap. We have shifted most of our code to native. And this time we made sure that literally every single bit of memory usage is accounted for! This means that you no longer have to fiddle with -Xmx values.

Here is a graph showing memory usage for recording entire Eclipse (namespace org.eclipse.**) with Chronon 2 and Chronon 3 (size in megabytes):

Smaller Recording file size

The Recording file size of Chronon 3 should on avg be 90-95% smaller than that of Chronon 2.

Here is a comparison of recording file size for the startup of eclipse, recording the entire org.eclipse.** namespace (size in megabytes). Note that eclipse has a lengthy, cpu intensive startup so its a good stress test.

Increased Concurrency

Chronon 3 pulls out all the stops to make sure your applications retain all of their concurrency. Highly concurrent applications will instantly see a very noticeable huge performance boost.

Logless Data Center

With the high performance of Chronon 3, you can have the Chronon recorder running all the time. This means you can have for the first time a Logless Data Center!

Consider this, with Chronon on all the time, you no longer need log files or log statements cluttering up your code. If something breaks, just pull out the Chronon recording and put in Post Execution Logging statements wherever you want.

Getting your hands on the beta

To get the beta, please fill out the form here.

We are opening the beta so you test the performance of Chronon 3 on your applications.

Since this is still beta, the recordings made from the Chronon 3 recorder will not open in the Chronon Debugger for now. You will need to use the Chronon Recording Server to record your applications. The only change would be that you would replace the recorder.jar in your Recording Server installation from v2 to v3.

If your existing Recording Server evaluation has expired, dont worry, we will give you a new one.

We would request providing some contact info so we can be in touch with you throughout the beta.

Please provide either a phone number or a skype id , or anything else (put it in the comments section). If providing a phone number, dont forget putting the country code.

Any spam/incorrect entries will be discarded.

Scaling Chronon

Posted by Prashant Deva on September 3, 2011

A nice infographic on how to scale the Chronon Recorder.

More details on our Performance Guide

Chronon Recorder saves the Chronon Recording Server

Posted by Prashant Deva on August 8, 2011

The Problem

Last week we had a big issue on our hands. A lot of people trying out the Chronon Recording Server beta were reporting they were getting corrupt files when they would download a recording from the Recording Server UI. The issue was occurring at random. Sometimes the files would be corrupt and sometimes not. And try as we might we just weren’t able to reproduce the issue on our end. We looked through our code and made a bit of change to what we thought could have been the issue and sent out the updated binaries, just hoping the issue was fixed.

But luck wasn’t on our side, our blind, shot-in-the-dark fix did not work and we were getting overwhelmed with support requests of corrupt downloads. On our end we were trying everything possible, like introducing lag, reducing bandwidth, adding more machines,etc; but nothing was able to reproduce the issue on our end.

Solution

Then finally it struck me. We would use Chronon to debug Chronon!

I asked one of our customers to take the Chronon Recorder and use it to record the Chronon Recording Server. Thus the next time he ran into the corrupt download issue, he simply took the Chronon recording of the Chronon Server and sent it over to me. Once we had the Chronon recording it was only a matter of minutes till we drilled down to the root cause and solved the issue!

Conclusion

This was one of first times we ourselves experienced the power of having a tool like Chronon. Had it not been for the Chronon Recorder we would still be pulling our hair trying to reproduce the issue and then debug it. Our customers would have continued to grow unhappier with our product, our credibility would be on the decline and it would eventually affect our revenues. If you have had to support products out in the field, you know how it is when an unhappy customer calls you about an issue and you have to make him go through all these hoops to try to figure out the root cause, making him even more unhappy in the process, till he eventually just gives up your product altogether.

With Chronon, all this just vanishes! The customer in this case just sent us the recording and no other communication was needed. We have the bug fixed now and tests in place. If you are a company that prides itself on customer support or, like Chronon Systems, pretty much depends on it to drive sales and revenue, then with Chronon you have an invaluable tool that can literally change the public perception of your company. As far reaching as this may sound, we can stand behind this statement now since last week we got to experience this for ourselves.

Recording Server now the official way to Record outside Eclipse

Posted by Prashant Deva on July 26, 2011

We have decided to deprecate the developer mode a little bit. It is now used only by the eclipse plugin while recording from within eclipse.

The Chronon Recording Server is now the official way for Recording outside Eclipse.

The reasons for doing so are:

The developer mode config files were hard to create with the many configuration options.
The server mode config file require only a ‘name’ value.
The process of creating the config, doing the recording, transferring the recording to the local machine to debug was too manual.
This is exactly what the Recording Server was designed to eliminate.
Developer mode didnt support dynamic start/stop or support for long running programs.
Turns out people who did want to record outside eclipse wanted exactly that. They wanted to skip the long initialization of their web servers and start recording after the web server had initialized and they wanted to record for long periods of time. This is exactly what the Recording Server was designed for.
No easy way to organize and view remote recordings.
With the Recording Server you can easily view all the recordings for every java application on every machine.

While the Recording Server may require a bit of an initial setup due to the need to install the Controller process on each machine, even this takes less than a minute and needs to be done only once.

Thus due to all the benefits described above we believe the Recording Server is the way to move forward for recording outside of Eclipse. It removes all the manual process which was previously required and replaces it with a nice, clean and easy to use GUI.

Server Mode in the Chronon Recorder

Posted by Prashant Deva on July 21, 2011

This week we released Chronon 1.5. The big feature of this release is the inclusion of ‘Server Mode’ in the Chronon Recorder.

What is the Server Mode?

The Server Mode is designed to allow the Chronon Recorder to be controlled through the Chronon Recording Server.

It includes features such as:

Ability to do dynamic start/stop of the recorder in a running program
The recorder can stay dormant in your program unless explicitly started from the Recording Server UI.
Ability to record long running programs.
Ability to split a recording based at a time interval or when the physical size of the recording gets too large.
Ability to dynamically modify the set of classes that are being recorded in a running program.
Thus, you can start recording with say include=com.package1.** and later decide to recorder com.package2.**. All this without the need to stop the program.

Future directions

With the addition of the Server mode in the recorder, we now have 2 distinct modes for the Recorder:

Developer Mode
Server Mode

The developer mode is the one you are probably familiar with as that is what is used when you record using the Chronon Eclipse plugin. It records the entire program from beginning to end and is meant for short running programs, as is common in development scenarios.

Moving forward we will probably have each of these 2 modes optimized for their specific use cases. There are a lot of optimizations that we want to put in the Recorder that will add a bit of overheard to the instrumentation time. While this is acceptable for long running server programs, it is not as useful if you are going to run your program for only a few minutes from within eclipse.

A good analogy is the server and client jvms. While the client jvm is optimized for quick startup and does less optimization, the server jvm is meant for long running programs and does a lot more aggressive optimization.

We will keep you posted on the specifics of how we proceed with putting optimizations/features in each of these modes of the Chronon Recorder.

Chronon Performance Guide now available

Posted by Prashant Deva on April 24, 2011

We have published a Chronon Performance Guide to help people understand and tune the various components of their system and chronon to gain the maximum performance. Of course we will keep updating this as we update Chronon.

Spoiler -

Performance tip number 1 : Use 64 bit!

Choosing what to record – Part 2: Configuration

Posted by Prashant Deva on March 4, 2011

As described in the previous post, the Chronon recorder allows you to choose what to include and exclude from recording. We really wanted to keep any sort of configuration required to use Chronon to an absolute minimum. After all who likes to go through boring manuals reading about all the various configuration parameters.

Even though the recorder and debugger allow and fully support method level selection of code, for the sake of simplicity of configuration we decided to allow only class level granularity for choosing which code to include/exclude from recording. We have seen that this suffices for almost all situations. If this does become an issue and lots of people do need method level granularity we will allow config file support for it in the future. You can read more about configuration details here.

All that said, as far the Eclipse plugin is concerned we really wanted people to just install Chronon and hit the ‘blue’ record button without worrying about any kind of configuration details. To achieve this, the eclipse plugin needs to select the right set of classes to include/exclude so that the debugger works as the user expects, right off the bat. The eclipse plugin makes the selection of classes to include/exclude by default as follows:

All classes in the src folders of the project being run and its dependent projects are included.
Classes in libraries of the projects, including JDK classes are not included.

Thus the plugin will include all the classes in ‘your’ code however it will exclude all the classes in 3rd party libraries. This way we achieve the initial goal of the include/exclude mechanism in the first place which is to record only relevant pieces of code and leave out all the code that you dont care about, which usually resides in 3rd party libraries.

However we knew that default mechanism wouldnt work for everyone and some people might want to include some libraries to record or might want to exclude some of their own code from recording. So we put in UI mechanisms while allow you to easily choose what to include/exclude from recording.

The include/exclude of classes can be configured in 2 places:

Launch Configuration
Project Configuration

Launch Configuration

You can choose the include/exclude classes per launch configuration. Note that if you choose to configure include/exclude here, it will override any similar setting in project configuration.
That is, the include/exclude config, if defined in the launch config, will be the only one used for that launch.

Project Configuration

We even allow configuration per project. The automatic tab allows you to visually select which classes to include/exclude. Note that this tab doesnt show any libraries your project uses. This is intentional since this tab really is all about ‘simplicity’. If you want to include libraries or some weird pattern, you can click on the ‘manual’ tab and specify the exact patterns to include/exclude whatever you want.