Using Web Services on the Siemens SXG-75


by Helge Timenes

March 12, 2006
Updated September 16 2006

Background

I have been waiting years and years for someone to put a reasonably priced mobile phone with integrated GPS on the market (the Garmin NavTalk was too expensive for my taste). In early 2006 the long awaited Siemens SXG-75 became available, and as it was time for renewing my cell phone provider contract (i.e. getting a new phone) I decided to get myself one of these babies.

But when the phone arrived I discovered that I had not done my homework thoroughly: The Siemens SXG-75 does not support JSR-172, the Java Micro Edition (J2ME) Web Services API. As I was currently working on a J2ME Web Services project, and wanted to use my new state of the art phone for testing, this was quite a blow. I considered my options: Returning the phone and get a JSR-172 enabled phone instead, or see if there was other ways to get things done.

The nice thing about using the JSR-172 API is that the Sun Wireless Tool Kit (WTK) does all the work for you. Provide it with a description of the web service you want to use (i.e. Web Service Description Language WSDL file) it will generate a Web Service Stub taking where all the HTTP communication and XML parsing is taken care of. There are other ways to use web services, kSoap to mention one, but these do not provide means of generating code from WSDL files. This means you have to write more code yourself, which in turn means that you have more possibilities of making mistakes, which means more debugging, cursing and yelling at innocent people. To summarize: When presented with the choice of manually writing code or having it generated for you, you have to be a suicidal idiot to choose the former.

So I decided to figure out if there was a way to use the JSR-172 Web Services API on the SXG-75. The short answer is: Yes! However, before arriving at Canterbury, I had to climb snow-covered mountains of byte code and wade through terrifying streams of data (forgive me if I’m getting too poetic about this). Hopefully this article can save other pilgrims a week or two of agony.

Solving the Problem

The SXG-75 does not support JSR-172. This was the first mountain blocking my way (and due to its size I could not see the chain of others ahead). What got me started on my first climb was an article I found while googling for information. Put shortly it claimed that all that is needed to turn your cell phone into a web service client, is to add two java archives (jar files) form the Sun JSR-172 reference implementation to your project. Perhaps this will trick other phones (apparently the Siemens CX-70 is fooled this way), but my SXG-75 remained unimpressed. What happened when I tried to start my MIDlet (containing these two jars) was this: Nothing. The program simply did not start.

I scratched my head for a while and as a desperate last act I tried removing all references to the JSR-172 classes from my project. This made a difference. I could at least start my now dysfunctional program. The presence of the two jars was not the problem. Next I tried to just reference on constant in one of the JSR-172 classes (I think it was javax.xml.rpc.Stub.ENDPOINT_ADDRESS_PROPERTY, not that it matters though…). This was enough to make my phone refuse to start the program.

Acting on a hunch, I decided to throw out the Sun jars, and instead import the corresponding source code into my project (this is also part of the zip containing the jars, and can be found here). I then renamed all the java and javax packages to jsr172.java and jsr172.javax, suspecting that my phone would scan my application for classes potentially overriding the standard java libraries. I was right. My program could now start, even when referencing the newly renamed classes. (For consistency, I renamed the entire JSR-172 implementation in this fashion, including the org.* and com.sun.* packages).

This renaming trick is a bit dirty however, as it requires some modification of the WTK generated stub classes. All the JSR-172 references must be renamed according to the new package names. Modifying generated code is never a good idea, but I considered it a small sacrifice in compared with the alternatives.

I tested the JSR-172 implementation using the generic Sun WTK emulator, and everything worked rather fine until I tested a method where one of the arguments was a double precision floating point type. When invoking this method I got a Runtime Exception with the message “Simple Type Mismatch”. This only came as a mild surprise, as the implementation I had grabbed from the sun download site is intended for CLDC 1.0, where floating point types are not supported. Rather than looking for a CLDC 1.1 implementation I decided to dive down in the JSR-172 and have a look at the origin of this exception. It is thrown in encodeSimpleType method in the com.sun.j2mews.xml.rpc.SOAPEncoder class. Here I also noticed that encoding for floating point types was implemented, but simply commented out. I uncommented it back in, and the world order was restored!

So now I had JSR-172 support on my SXG-75. I re-packaged my original MIDlet to include the JSR-172 implementation, loaded it on my phone and got disappointed again! Calling a Web Service method from the phone invariably resulted in a java.rmi.MarshallException being thrown, containing the message “Missing SOAP Body or Envelope”. A program that ran flawlessly on the Sun emulator, did not work on my phone!

I tried to figure out what the difference between the two platforms could be. The most obvious one was that I had configured the Sun WTK to run MIDlets as trusted, to get rid of the annoying network access confirmation dialogs. I changed the settings, and to my great relief the MDIlet now coughed up a MasrhallEception when running on the Sun emulator.

Far from having solved my problem, at least I now knew that trusted and untrusted MIDlets behave differently. I did at this point not know why.

My first idea for a solution was to get my MIDlet to run as trusted on my phone. To my knowledge there are two ways to do this. The proper way is to go through a long and expensive certification process. This seemed like a rather silly thing to do on an application where development had just started, and which might in the end be nothing but a stupid toy for my own entertainment. The other way is to use the dark side of the force to crack open the certificate store on your phone and add a forged certificate granting your MIDlet trusted privileges. As I did not know my way with the dark side of the force I decided that this was a bad idea, and that instead of trying to get my program more trustworthy, I should concentrate on figuring out why a trusted MIDlet behaves differently from an untrusted MIDlet.

The first thing to do was to figure out why the MarshallException was being thrown. To (for once) cut a long story short: It turned out that the server retuned an error message (HTTP 400 <h1>Bad Request (Invalid Header Name)</h1>). The JSR-172 implementation, instead of catching the HTTP 400 code and handling it ptoperly, tried to parse this response as XML, which inevitably would raise a MarshallException.

The mystery was now: Why would the server accept a request from a trusted MIDlet, but not one from an untrusted MIDlet?

Using the Sun WTK's Network Monitor I noticed that when untrusted, the user-agent property in the HTTP header was overwritten, presumably somewhere in the implementation of java.io.HttpConnection (which I had not messed with). In com.sun.j2mews.xml.rpc.OperationImpl the user agent is set to “Profile/MIDP-2.0 Configuration/CLDC-1.1”. But the in the outgoing HTTP stream it was replaced with “UNTRUSTED/1.0”. Not knowing much about web servers, my first thought was that the server was configured to deny requests from untrusted user agents. I even sent an email to my server’s administrator asking if this was the case and if he could be so kind and change it. It wasn’t and he couldn’t.

Being quite close to giving up and putting my SXG up for sale on eBay, I decided to take one last close look on the outgoing HTTP stream. This time I used a HTTP sniffer program called HTTPLook. I now discovered that the user agent property was not simply overwritten. There were two user agents properties in the header. One specifying “Profile/MIDP-2.0 Configuration/CLDC-1.1” and the other “UNTRUSTED/1.0”. The Sun Network monitor only showed the latter.

Although I would assume that a server should be able to handle redundant HTTP header properties, our Microsoft Server 2003 obviously didn’t. The last mountain to climb before reaching Canterbury was simply to remove the lines of code in the OperationImpl class where the user agent property was set, leaving only the one added in the java.io implenetation.

That was it. I had reached the end of my pilgrimage. I celebrated with several shots of Bruichladdich Single Malt Whisky. This was yesterday, possibly explaining my rather sentimental style of writing today.

Summary

One way to using Web Services on the Siemens SXG-75 (and possibly several other phones) goes as follows:

Conclusion

Using Web Services on the Siemens SXG-75, and taking advantage of the stub generator in Sun's Wireless Tool Kit is absolutely possible, even if the phone does not oficially support JSR-172. Hopefully this article may be of assistance to other pilgrims struggling to reach their web serving goals.

For comments, suggestions, corrections or questions, feel free to write me.

- h -



For comments, suggestions, corrections or questions, feel free to write me.

BTW: “Canterbury” is a reference to Geoffrey Chaucer’s “Canterbury Tales” in which a load of pilgrims travel to Canterbury and have a jolly good time about it. This reference has also, much more successfully, been used by Richard Dawkins in his excellent book on evolution “The Ancestor’s Tale”. If you haven’t read it and aren’t going to, I will fight you!


Update, Sep. 16th 2006

Lo and behold! I found another hurdle. This one took almost 6 months to find, because it is not a problem on the SXG-75 and other phones I have tested on until now. But it is a problem on Nokia phones (and possibly other phones as well)! Note that it is NOT a Nokia-bug. I was just sloppy and lucky back in March.

The problem is this: Creating a midlet using the steps outlined above will cause a Nokia phone (and possibly others) to refuse to install the midlet, giving nothing more than "Authorization Failed" as reason. This happens for many reasons, including when referencing a platform class that is not supported. As soon as I became aware that my web service MIDlets were uninstallable on Nokias, I started searching for the culprit. And sure enough: in com.sun.j2mews.xml.rpc.OperationImpl (which I had renamed jsr172.com.sun.j2mews.xml.rpc.OperationImpl as discussed above), there was a little innocent import:

import com.sun.midp.io.Base64

Now, the com.sun stuff is part of Sun's MIDP reference implementation and, frankly, should not be expected to be present in a real device. However it is in the SXG-75 (I guess they were too lazy to make their own com.siemens implementation). It is also present in the Nokia S60 emulators and, rather less surprisingly, in the Sun WTK, so you won't have any problems running your app in the emulator. But the real Nokia devices do not have this class!

Once found, the solution to the problem is easy. Substitute com.sun.midp.io.Base64
with a similar class in your workspace and change the import. As I couldn't find any available source code for Sun's class anywhere, and because I was too lazy to write my own base-64 encoder. I decided to grab one from the Bouncy Castle project (http://polydistortion.net/bc/index.html. You will need the following classes: They can be downloaded here. Thank you bouncy dudes :-)