Tuesday, July 29, 2008

Number Format Hell

I am in localised format hell at the moment.

I am currently re-working Apache Cocoon CForms.
My current task is a validating, currency field, which can properly display and edit any currency in any locale.

CForms (started long before Browsers became smart with big Ajax Libraries) allows you to use Convertor classes to map either way between an Object in the business logic and a localised String to be edited in a form.

Number Convertors are based on java.text.NumberFormat class.

Let's say that the Model (Bean, XML frgment, etc.) I am editing contains a value that represents a Currency in Pounds Sterling as a java.math.BigDecimal, I'd set up my Convertor like this :
<fd:field id="dieselprice" required="true">
<fd:label>Price for a liter diesel:</fd:label>
<fd:datatype base="decimal">
<fd:convertor variant="currency" currency="GBP"/>
</fd:datatype>
</fd:field>

When my number eg. "1000.00" goes into the Convertor, it is localised to the format for the viewer's locale, so in the UK someone should see: "£1,000.00", in France: "1 000,00 UK£" etc. (PS. Diesel will cost £1000 per litre one day, you'll see ....)

So far, so good. The user gets a number in a format they recognise, the edited number is returned to the server and the Convertor performs the reverse operation, to store it back to the java.math.BigDecimal.

Now the fun starts ........

I am changing CForms on the client-side to use all Dojo editors. In the case of currency, I use the dijit.form.CurrencyTextBox. It is a cool editor, while the editor is not focussed, it shows the formatted version "1 000,00 UK£" (fr_FR) but as soon as you click to edit it, it gives you the simple version, where it is harder to make editing mistakes "1000,00" and gives you validation feedback while you are typing.

So here comes the first impedance mis-match .... Dojo does not expect the server to format and localise the number as a String, it expects the server to send the value in the same format as a JavaScript primitive "1000.00". Dojo then uses it's vast library of localisation formatting rules to convert the primitive to a formatted String suitable for the locale of the User, allow that to be edited and post the new primitive back to the server.

So, I can hack Dojo, or I can hack CForms. I hacked Dojo, because CForms has the right behaviour, it should still work when someone has JavaScript turned off, they will see a simpler form but with properly localised values in it.

So I extended dijit.form.CurrencyTextBox <-- cocoon.forms.CurrencyField, to allow it to send and receive formatted strings instead of number primitives (as text).

Boom Boom, job done! (so I thought).

The first hint of trouble appeared when I was testing numbers represented as percentages. They would display fine in some locales but not in others. You see Dojo needs to be able to interpret the format to be able to validate it and it turned out that Java and Dojo use different formats for percentages in fr_FR and de_DE (France and Germany).

While Java formats them as "#,##0%" (123%) Dojo formats them as "#,##0 %" (123 %).

I am thinking WTF! These are supposed to be international standards! Where's this stuff coming from?

On the Java side, the closest hint I could find was that java.text.DecimalFormat is copyrighted by Taligent (IBM). It is also possible that IBM wrote the number formatting in Dojo, I believe they contributed all of the Internationalisation classes to Dojo, but specifically, Dojo's datasource for compiling it's lookup tables is Unicode.org. I have yet to ascertain where Java sources it's data, but there are clearly problems, I read that JDK7 will package the currency bundles in a way that does not require a whole JDK update when currencies change.

So there is an annoyingly large number of very detailed differences between localised currency formats in Dojo and Java, simple differences like individual locale's currency symbol can be worked around via Dojo APIs, deeper problems like group and decimal separators not matching for Arabic and some far-eastern countries are proving more wasteful of my time.

I am currently using Java 1.4.2, will upgrading change the situation? Make it better? Make it worse? My JDK is supplied and maintained by Apple. Will moving to another OS make it better or worse?

It's an abysmal situation!

One tip for the would-be user of java.util.Currency and java.text.DecimalFormat is, if you call DecimalFormat.setCurrency, make sure you also copy over the number of decimal fraction digits, DecimalFormat leaves them out, resulting in a bad format for currencies that have no decimals (Japanese Yen etc.).

Currency currency = Currency.getInstance("GBP");
int digits = currency.getDefaultFractionDigits();
DecimalFormat format = (DecimalFormat)NumberFormat.getCurrencyInstance(locale);
format.setCurrency(currency);
format.setMinimumFractionDigits(digits);
format.setMaximumFractionDigits(digits);

I did not find this in any of the tutorials online, but it is in the JavaDocs.

Friday, July 11, 2008

iPhone Day

All the net is abuzz with iPhone unboxing reports etc.

I gave my iPod touch to my 14 year old godson a few weeks ago, a bit rash, I miss it a lot, but my intention had always been to pass it on when the 3G iPhone came along.

I am bursting to get one now ....... specially after I spent half an hour looking through the iTune AppStore and already found about 10 apps I would have downloaded straight away!!

One app that caught my eye immediately was FileMagnet. One of the reasons I jailbroke my iPod was to be able to copy files to it for viewing on the device, reference material, stuff like pdf public-transport maps of places I am visiting ...... it was never an easy process ...... entering those darn file:/// urls into MobileSafari, FileMagnet looks a lot easier.

But there is one hell of a bitter pill to swallow!!!!

There is only one carrier in the UK, no competition in most countries, therefore often ludicrous prices.
O2 has a bad rep (but then I guess all carriers have a bad rep).
The deal itself in the UK is not so bad, I'd be paying only a little bit more than I do now and I'd get more service.

But the international data-roaming charges!!!!!!!!

£3 per Meg in Europe, £6 per Meg in the rest of the world!!!!!!
WTF?!?!?!?!?!?!?!?!
This is utter madness!!!!

When I am abroad in a strange city, that is just the time when I'd want data-access, GPS + Google Maps would be indispensable.

An article in ArsTechnica today, outlines the amount of data transferred per hour to use different applications.

Using Google Maps comes out at 6.5Meg or £19.50 per hour in Europe!!!!

For crying out loud!!!! And you cannot even work around it by legally de-SIM locking and using a local carrier.
This just ruins the whole purpose of the device and I think the bad feeling from this will ultimately rub-off on Apple.

The EU is currently trying to force a reduction in call-roaming charges across Europe, apparently they also have data-roaming charges in their sights, I really hope they get a move on!!!