Recently in the project I am currently working on, we were experiencing some slowdowns at login time that really annoyed our users. So we decided to tackle this issue, and we quickly found out that the problem came from the computation of security privileges for a given user. But we were not able to quickly find the bottleneck because it was tricky to find.

At login time, the security stack evaluate all the authorizations of the current user based on the privileges he has access to. In the security model, one part of the authorizations is responsible to give or block access to screens. As a consequence, the security model stored in the database contains some controller class names, which are then mapped to the real controller classes at runtime. So the following piece of code seems quite understandable:

...

public String resolveControllerUrl(final String securityKey) {
  try {
    Class.forName(securityKey);
  } catch (ClassNotFoundException e) {
    // unknown controller => return null
    return null;
  }

  return buildControllerUrl(securityKey);
}

In order to avoid computations of privileges for controller classes that does not concern a current module, the code first check if the class exists in the current web application, and stops the processing if not found. Fair enough you might say.

We running a single module, the security evaluation was quite quick. But as soon as we added more modules, the loading became extremely slow (evaluation is done in parallel for all modules at the same time). And this slowdown is caused by the single class lookup method shown above.

In order to understand why, let’s first have a look at the classloader hierarchy used by Tomcat:

When trying to load a class from a web application, the class is first search in the corresponding webapp classloader. Then, if it is not found, resolution is delegated to the parent, which in turn delegate to its parent, and so on. Only if the class was not already loaded in the whole hierarchy it is then explicitly searched, top to bottom this time (that is, class is loaded at the bootstrap level, then if not found it goes down at the system level, then common, and so on…).

Loaded classes references are kept in memory which means that, even if the whole classloader hierarchy must be traversed, it is blazing fast. But bad things happens if the class cannot be found in any level. In that case, each level will explicitly try to load it. Even worse, those loading classes methods are declared as synchronized, which means that loading an nonexistent class will end up in a contention at all classloaders levels.

In our case, this small piece of code combined with the parallel security evaluation on all modules of our application (of course, each module was not running on its own Tomcat instance but we grouped a lot of modules on a small number of Tomcat instances which exhibited even more this issue).

Slightly modifying the code shown above so that it keeps in memory the list of unknown classes (we can keep it in memory as we know the list is quite short) made the login process way much faster.

Conclusion: beware when trying to manually load classes. And it seems that we are not the only ones experiencing this behavior (check out “Robust and Scalable Concurrent Programming: Lesson from the Trenches” presentation by Sangjin Lee which talks about this issue on slide 41).