Strange javamail behaviour inside tomcat

In the series: 'weird problems you'd rather not spend your valuable time on' today I present a strange javamail/tomcat related problem and its solution.
While preparing the next version of our java web app I noticed that emails sent by our app were missing the mail subject. Moreover the message appeared to get sent as plain text instead of HTML so the message body was displaying ugly html.
While debugging everything seemed OK and the javamail API (invoked via Spring) was invoked with the correct parameters and a non-null subject.
So then I wrote a jUnit test to further isolate the problem and of course the unit test, invoking the same server-side java code as before, worked like a charm: the subject was present and the message body was interpreted as HTML.
I was now faced with a configuration problem because the exact same code was working fine from a unit test but was failing when executing from within Tomcat. After some googling I found the advice to check the classpath for duplicate or conflicting javamail implementations. I use the very handy maven command:

mvn dependency:tree

which shows the full dependency tree of your referenced libraries including implicit references, i.e. a jar required for one of my own dependencies. Then I noticed that axis-2 uses a geronimo-javamail implementation; in addition to the 'standard' javax.mail javamail. Sure enough when I excluded this implicit dependency like so:

        <dependency>
            <groupId>org.apache.axis2</groupId>
            <artifactId>axis2-kernel</artifactId>
            <version>1.4.1</version>
            <exclusions>
                <exclusion>
                    <groupId>org.apache.geronimo.specs</groupId>
                    <artifactId>geronimo-activation_1.1_spec</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.apache.geronimo.specs</groupId>
                    <artifactId>geronimo-javamail_1.4_spec</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

the mail got sent correctly.

XStream and annotations: the nasty details

Recently a colleague introduced XStream into our project. The use case was to read an XML dump of a database, parse it into an object graph and use these objects to transfer the content to another database. XStream was just the right package to easily convert an XML stream into objects without having to resort to JAXB or other complex solutions. However when I tried to adapt the code to support a similar scenario I bumped into a few unexpected problems even though the documentation of this project XStream is actually pretty good.

The first step to start using XStream is to include a reference to it in the maven pom file:

<dependency>
  <groupid>com.thoughtworks.xstream</groupid>
  <artifactid>xstream</artifact>
  <version>1.3</version>
</dependency>

The idea of XStream is to annotate your domain classes with xstream annotations which map the attributes and references to XML. In our case, we were going in the other direction: from XML to objects and in that case, if the structure of the XML is reasonably small it makes sense to define the objects inside one 'mother' class. For example say we have an XML:

<?xml version="1.0" encoding="UTF-8"?>
<records>
   <book id="13">
       <author>Robert C. Martin</author>
       <title>Clean Code</title>
   </book>
</records>

This datastructure can be mapped by the following (inner) classes like so (omitting the imports)

public class XStreamDemonstrator {

  public static void main(String[] args) throws Exception {

      XStream stream = new XStream();
      stream.processAnnotations(Records.class);

      FileInputStream is = new FileInputStream("resources/books.xml");
      InputStreamReader isr = new InputStreamReader(is, "UTF-8");

      Records records = (Records) stream.fromXML(isr);

      for (Book book : records.books) {

          System.out.println("Book " + book.id + ": " + book.title + " by " + book.author);

      }
  }

  @XStreamAlias("records")
  public static class Records {

      @XStreamImplicit
      List<Book> books;
  }

  @XStreamAlias("book")
  public static class Book {
      @XStreamAlias("id")
      @XStreamAsAttribute
      String id;

      String author;

      String title;
  }
}

Now how's that for simplicity? It's fast, easy and it works. That's right; but now suppose the book gets translated and we'd like to add an extra attribute to the XML:

<?xml version="1.0" encoding="UTF-8"?>
<records>
   <book id="13">
       <author>Robert C. Martin</author>
       <title language="en">Clean Code</title>
   </book>
</records>

The problem lies in the added attribute 'language'. There is no way to map this with XStream annotations as is, this is also confirmed in the newsgroup. The solution is to use a Converter implementation class which reads out both the attribute and the child text at once:

    @XStreamAlias("book")
    public static class Book {
        @XStreamAlias("id")
        @XStreamAsAttribute
        String id;

        String author;

        @XStreamConverter(TitleLanguageConverter.class)
        TitleLanguage title;
    }

    public static class TitleLanguage {
        String title;
        String language;
        public TitleLanguage(String title, String language) {
            this.title = title;
            this.language = language;
        }
    }

    public static class TitleLanguageConverter implements Converter {
        public Object unmarshal(HierarchicalStreamReader reader, UnmarshallingContext context) {
            return new TitleLanguage(reader.getValue(), reader.getAttribute("language"));
        }
        public void marshal(Object source, HierarchicalStreamWriter writer, MarshallingContext context) {
            // not implemented
        }
        public boolean canConvert(Class type) {
            return type.equals(TitleLanguage.class);
        }
    }

That wasn't too bad...that is...there is one small problem with the above code because it throws the following stack trace upon execution:

Exception in thread "main" com.thoughtworks.xstream.converters.ConversionException: only START_TAG can have attributes END_TAG seen ...Robert C. Martin... @4:43 : only START_TAG can have attributes END_TAG seen ...Robert C. Martin... @4:43
---- Debugging information ----
message             : only START_TAG can have attributes END_TAG seen ...Robert C. Martin... @4:43
cause-exception     : java.lang.IndexOutOfBoundsException
cause-message       : only START_TAG can have attributes END_TAG seen ...Robert C. Martin... @4:43
class               : XStreamDemonstrator$Records
required-type       : XStreamDemonstrator$TitleLanguage
path                : /records/book/author
line number         : 4
-------------------------------
 at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:88)
 at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:55)
 at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:75)
 at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshallField(AbstractReflectionConverter.java:234)
 at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doUnmarshal(AbstractReflectionConverter.java:206)
 at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshal(AbstractReflectionConverter.java:150)
 at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:81)
 at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:55)
 at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:75)
 at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:59)
 at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doUnmarshal(AbstractReflectionConverter.java:213)
 at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshal(AbstractReflectionConverter.java:150)
 at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:81)
 at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:55)
 at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:75)
 at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:59)
 at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:142)
 at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:33)
 at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:931)
 at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:917)
 at com.thoughtworks.xstream.XStream.fromXML(XStream.java:861)
 at XStreamDemonstrator.main(XStreamDemonstrator.java:30)
Caused by: java.lang.IndexOutOfBoundsException: only START_TAG can have attributes END_TAG seen ...Robert C. Martin... @4:43
 at org.xmlpull.mxp1.MXParser.getAttributeValue(MXParser.java:927)
 at com.thoughtworks.xstream.io.xml.XppReader.getAttribute(XppReader.java:93)
 at com.thoughtworks.xstream.io.ReaderWrapper.getAttribute(ReaderWrapper.java:52)
 at XStreamDemonstrator$TitleLanguageConverter.unmarshal(XStreamDemonstrator.java:65)
 at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:81)
 ... 21 more

It turns out that the order is important in TitleLanguageConverter.unmarshal: when we first read the book title text, the XML cursor has already moved past the language attribute and cannot go back to read it. The solution is to reverse the order:

return new TitleLanguage(reader.getAttribute("language"), reader.getValue());

And everything works as expected!

Developing with Android - day 1

I'm not a big gadget freak but being a java developer I just could not resist the temptation to buy the new HTC G1 android phone last month when it hit the stores. The idea of being able to develop my own mobile java applications and run them on G1 appealed to me. Also it seems the Open Handset Alliance seem pretty intent om making Android a big success. The idea of publishing a home-brewn app to the Android Market and making a fortune is of course a nice idea too :-)! But before we strike it rich we have to gain some experience with Android development.
Being currently 'inbetween projects' I can now luckily devote a few days entirely to discovering this new development platform. As it turns out there is some terminology to get used to but I managed to install and run my first android application on my phone in less than one day. So how did I do that?
My main sources of information are Google's android development site but even better is a Manning early access book called "Unlocking Android" which contains very specific examples and proved to be an excellent guideline. Most of the material presented in this post was found in Manning's book.

Preparing the environment
Following the instructions it's easy to install an eclipse plugin containing the android development environment including the emulator. I also made sure to include the android tools directory in my PATH environment variable.

The application
Just to get started I decided to write a reader for the Dutch news site called nu.nl. Quite a useless idea because nu.nl already offers both a mobile-ready version of their website at http://mobiel.nu.nl/pda/ and RSS feeds but what the heck, it will be a good exercise anyway.
We want the application to start with a screen displaying all news categories in a List (displayed to the left).
When pressing one of the categories/items in this list the app should display the most recent news items in this category. That should keep us busy for a day.

The main screen
So after creating a new Android project called NuDotNLReader with a main Activity called NuReader we can start implementing. An Activity basically is the same as one screen of your application and can be composed of various Views or ViewGroups. By default the main NuReader activity is created as a direct child of the generic Activity class. We want this screen to display a List so we change its inheritance like so:
public class NuReader extends ListActivity {

This class contains a static List of category names, for starters:

static final String[] CATEGORIES = new String[] {
 "Algemeen", "Economie",
 // more categories
}

and we use this list of category names to populate the main List screen at startup. Each ListActivity has an associated ListAdapter which contains the data to be displayed in the List:

setListAdapter(new ArrayAdapter(this,
 android.R.layout.simple_list_item_1, CATEGORIES));

The second argument to the ArrayAdapter constructor got me puzzled for a while. The first part android.R refers to the main resource class named 'R' which is created automatically for each new project in the same pacakge as the main Activity. It contains references to static information such as texts and images which are stored in binary format for storage and performance reasons. When you open up the source of R.java it contains a public static class layout but nowhere can we find any reference to simple_list_item_1. It turns out that this is one of several constants defined in the R.layout section by default that are just there for us to use.
This is already enough to make the main screen of the application work.
Next we want to make something happen when one of the categories in the List is clicked by the user. Let's override the onListItemClick method whose parameter 'position' tells us exactly which category in the List was selected. This category name can be passed on to the next Activity using the shared preferences available through the application context:

getApplicationContext()
 .getSharedPreferences("nureader", Context.MODE_PRIVATE)
 .edit()
 .putString("category",
    getListAdapter().getItem(position).toString())
 .commit();
startActivity(new Intent(v.getContext(), CategoryReader.class));

Make sure to commit() the change to the shared preferences! By invoking startActivity with a new Intent for our next class CategroyReader, control will be passed on to this next Activity. It is also possible to receive some kind of return value from the next Activity by invoking startActivityForResult but we don't need this now.

Next up: the CategoryReader class, responsible for displaying the news items in a certain category. Since this is our first day we are going to make our life as easy as possible by just displaying the nu.nl category html page on a so called WebView. So the CategoryReader will just be a direct subclass of Activity with only one method, the onCreate method which is invoked each time the Activity is created:

public class CategoryReader extends Activity {
  @Override
  protected void onCreate(Bundle savedInstanceState) {
     super.onCreate(savedInstanceState);
     String category = getApplicationContext()
        .getSharedPreferences("nureader", Context.MODE_PRIVATE)
        .getString("category", "empty");
     WebView webview = new WebView(this);
     webview.loadUrl(
        "http://mobiel.nu.nl/pda/" + category.toLowerCase()+"/");
     setContentView(webview);
  }
}

In the onCreate we get the category name from the shared preferences in the application context. This category name is added to an URL which is fetched and displayed on a WebView element. A WebView element knows how to read and process HTML so it's basically a mini-browser element.

That's it!

Running the application on the G1 phone
As usual, getting an application onto the phone is astoundingly easy, once you know how to do it! First make sure you don't have the android emulator running. Connect your phone to an USB port and an USB icon appears at the top of the screen. Select 'Mount' to establish the link. Then
select Run as.. Android Application in eclipse and the application is 'automatically' installed on the phone and starts working instantly! Make sure to correctly disconnect the USB link before removing the USB cable.

Wrapup
So in just a few hours time it's possible to have at least a very small application running on your android mobile phone. Tomorrow we'll take a look at direct HTTP connections and how to asynchronously send and receive data from the nu.nl RSS feed.

Generating dynamic web apps with DSL and GWT

The idea for my presentation at Devoxx'08 originated last summer when I attended the java event organized by ProfICT. There I saw Neal Ford build the case for Domain Specific Languages (DSL) and subsequently saw Sven Efftinge demo xText and explain how to craft your own DSL with this Eclipse based plugin called openArchitectureWare. A very interesting event and I immediately saw the potential of DSLs for one of my clients.
For this client I once built an intranet application that let their internal sales reps generate quotes for their customers selling complicated power generators. These quotes consisted of different components with many dependencies between them. Back then I created an XSLT/JSP solution which described the problem domain as well as the front end representation in one huge XML file. After attending Neal Ford's relentless XML bashing I understood that this application needed a rewrite based on a DSL.
So basically the idea was to describe the problem domain (complex power generators) in an external DSL and visualize it using a nice front end application. Out of personal interest I decided to construct the front end using Google Web Toolkit (GWT) but without writing too much code for it.
Just to verify the feasability of this idea I wrote a proof of concept for a fictitious bicycle store, looking like this:

So what did I need to build this proof of concept? Well, I needed to think about how I'd like to model my problem domain. Basically, the bicycles sold in our shop are composed of a few Entities with certain values and there are relations between these entities. Also, I wanted to be able to group the potentially long list of entities over a few steps. So I came up with a model which looked like this:

So we have two entities (Type and Color) and the values of Color depend on the Type. Example: Racing bikes are only sold in silver whereas City bikes are available in all four colors. Both entities are displayed in the one and only Step (actually in the demo above there are two steps with four entities but you get the idea)
This model was constructed according to the new DSL which I had to define in xText at the same time. xText is an eclipse plugin which offers you a wide range of tools and aids to build your own DSL, making use of the familiar context sensitive code completion features in Eclipse.
The definition of the DSL is another text file which looks like this:

Model:
(elements+=Element)* (steps+=Step)*;

Element:
"elt" name=ID "{"
((option+=Option) ("," option+=Option)*)?
"}" (dependsOn+=Reference)*;

Reference:
"dependsOn" elt=[Element] "{"
((dependency+=Dependency) ("," dependency+=Dependency)*)?
"}";

Option:
description=STRING (deflt?="default")?;

Dependency:
fromOption=Option ":" "{" ((toOption+=Option) ("," toOption+=Option)*)? "}";

Step:
"step" nr=INT label=STRING "{"
(elt+=[Element] ("," elt+=[Element])*)?
"}";

This format of this grammar description is an extension of the Backus-Naur-Form. It basically says that our Model consists of one or more Elements and one or more Steps, which in turn have their own restrictions.
With a correct grammar definition you can ask openArchitectureWare to generate a new eclipse plugin editor which enables you to actually write your model. This editor automatically contains code completion compliant to your grammer, i.e. it will display a red cross when you accidentally type Elemend instead of Element. Moreover you can expand your editor with your own rules. It might, for example, not make sense to have multiple Elements with the same name. This is not enforced by the DSL grammar but can be added to the plugin editor with the following rule:

context Element ERROR "Names of all entities must be unique." :
allElements().typeSelect(Element).select(e|e.name == this.name).size == 1;

xText/openArchitectureWare is actually quite a powerful environment, backed by an active user community. Anyway, this is all nice and interesting but what if we want to do something useful with our model? This is where xPand templates come into play. The idea now is that we will write a few templates which generate java code which, in turn, is added to a skeleton GWT application. An excerpt from one of the templates:

What this template will do is generate the source code of a java enum called Entity. In the constructor of the enum we pass in an array of possible values for the entity and a String with the default value. Two more templates were needed: one to generate an enum of the Steps and the third one to define the dependencies between the entities.

The GWT application is not much more than a StackPanel with one panel for each step. Inside these steps we render the entity values in a select boxes. These select boxes contain onChange event handlers which cause the dependent entities to be recalculated, i.e. when we change the bicycle type from City bike to Racing bike, the list of possible Colors has to be reduced from four to only one Color. This logic was built in GWT and is the same for any model we generate into the GWT application using our DSL.

So in theory it would now be possible to construct a real world application with the entire list of product entities and their dependencies in the xText editor; generate the GWT classes using the xPand templates, re-generate the GWT application and we're ready! Of course the GWT application would also have to be augmented to actually DO something with all the selections, at the moment we only allow to the user to click through the different options without actually doing something (i.e. ordering a Green Citybike).

By the way, I think the power of GWT is justly demonstrated by the fact that it was ridiculously easy to include the real proof of concept into this blog post; I only had to upload the GWT generated HTMLs and other files to a directory on my hosted server and include an iframe with a reference to the app. in this post!

For those interested in the details I include the links to the presentation as well as the source code of the prototype.

Devoxx'08 a great conference

Three days at Devoxx'08 have really refueled the engine with new creative ideas for the (near) future. The conference was well organized at an excellent location: MetroPolis, just off the Antwerpen city center. We didn't make it to the opening keynote on Wednesday so we missed the JavaFX commercial but by lunch time we arrived and could listen to Brian Goetzke explain the concurrency API and how we should stop thinking in Threads and move on to multi core parallelism. The same afternoon we saw the future of rich internet applications or at least Adobe's version of it; saw a bloke from New Zealand explain the wonders of easyb and behavior driven development; were slightly disappointed by the filthy rich android clients talk because of its very dry nature and lack of cool effects and finished up with a great talk by Peter Kriens, one of the founding fathers of OSGi. As it turned out OSGi was quite hot because of the new directions SUN is taking with java and modularity in project Jigsaw.
Thursday was my personal big day with my BOF scheduled for 20:00 at night. The day started with a great keynote by Josh Bloch who showed some excerpts from his book Effective Java. We got in early so we also got to see the dancing girls, as opposed to those who watched the keynote from adjacent rooms :-). Next up Ivar Jacobson, Mr. Use Case who explained how to be smart on software projects. Later that day I came across the book The Mythical Man-Month by F.P. Brooks; quite interesting how such a relatively 'old' book can still be up to date as exemplified by Ivar Jacobson's talk. Next up more talks on JavaFX, a weird live video connection talk by javaposse.com with free beer, and a boring overview of jBoss SEAM.
My own talk on DSL and GWT went really well; the hour of day was not one to attract large audiences but those who had ventured into my room all stayed until the end. Some really good questions also during the talk and a few interested people at the end while we were gently being shoveled out of the room by the next speaker. A big sigh of relief after two months of preparations and we took the first tram back to Antwerpen city center to taste a few Belgian beers.
Friday morning three concluding talks: a pretty impressive live demo of the new Spring DM server demonstrating how one OSGi could real-time be replaced by another while the main application kept running. Java + XSLT 2.0 was a bit too high level but the final presentation was a really good presentation on EJB 3.1 and why we should reconsider the idea of never ever doing EJBs again. Good stuff.
So all in all three days well spent; I bumped into a few people I'd worked with years ago, made some new friends, scored three new (black) t-shirts, bought two new books one of them signed by The Josh Bloch himself and just generally enjoyed being immersed in Java for three days on end! I'll try to summarize my own talk on DSL and GWT in a later posting.

Talk on DSL and GWT at Devoxx'08

So this weekend the people from Devoxx sent me a mail with confirmation for me to present a Birds Of a Feather session at their next conference! Wow, that's certainly something I didn't really expect when I submitted the proposal two weeks ago. My presentation is about how I designed and used a Domain Specific Language to generate a Google Web Toolkit based user interface. I guess it must have been the combination of the two buzzwords DSL and GWT that did the trick. So the next few weeks I know what to do after I get back from the day job and putting the kids to bed! 

Since I've never before visited Devoxx (formerly known as JavaPolis) I don't really know what to expect from the conference; how many people will attend such a BOF, etc. My BOF is scheduled for 20:00 on Thursday night when everybody has started tasting the Belgium beers I suppose? We'll see.

Resource URLs in Hippo/Jetspeed

Recently we wanted to implement a few Ajax calls in our Hippo portal, a Jetspeed based portal product. Since you are quite restricted within the context of a portal application with respect to accessing or modifying the response object, it turns out that you need a special kind of strategy in order to receive the ajax calls from the client and pass back the results: resourceURLs.
Our version of Hippo is based on Jetspeed 2.1.3 which does not yet comply with JSR-286 for portal standards. This JSR specifies how to handle ajax calls within a portal. Luckily this part of the specification has already been implemented in Jetspeed 2.1.3. So how does it work?

First we have to create a resourceURL factory :

org.apache.portals.bridges.common.PortletResourceURLFactory resourceURLFactory = new org.apache.jetspeed.portlet.PortletResourceURLFactoryImpl();

I've put this code in the init(config) method of a portlet and store this factory in a private instance variable.

Now in your portlet render template you must use an URL generated by this factory. We prepare this URL in the portlet code and store it in the Freemarker model:

model.put("ajaxurl", resourceURLFactory.createResourceURL(getPortletConfig(),
request, response, null));

Refer to this url in your form:

<form id="aForm" method="post" action="${ajaxurl}">
..
</form>

Now, when this form gets submitted, you will receive a hit on your doView method and you can inspect the request to check whether it was a resource request or a normal request:

org.apache.jetspeed.request.RequestContext jrc =
(org.apache.jetspeed.request.RequestContext)
request.getAttribute(org.apache.jetspeed.PortalReservedParameters.REQUEST_CONTEXT_ATTRIBUTE);
if
(jrc.getPortalURL().getNavigationalState().getPortletWindowOfResource()!=null)
{
// it was a resource request
} else {
// it was a normal render request
}

Now you have full control over the response and can set its content type and get the output stream to pass back whatever content you like without going through the portal aggregating cycle. Pretty simple and works like a charm.