Java 4-ever

Excellent :)

Best error message ever

All the credit goes to Microsoft.

Lisp Machine emulator

I've been having a lot of fun with virtual machines recently, and not just the JVM. I am running Windows 7 on Linux using Virtual Box, a great software (Virtual Box, not Windows 7). Today I managed to get a Texas Instrument Explorer lisp machine running in the Meroko microcode emulator on my Linux laptop. It boots a real TI Explorer machine image, so you can have fun with zmacs, Flavors and the lisp machine debugger.

Unfortunately it is not very usable for real work: it takes up one CPU core and it is still pretty slow to use. I wish I had time to fix it. The lisp machine's development environment is still better than Eclipse with SBCL . Of course today the future of Lisp is Clojure.

How to make the Citrix ICA client work on 64-bit Linux

Time to dust off this old blog...

I wasted a couple of hours yesterday to make the Citrix client work on Fedora Core 12 64-bits. Unfortunately I need to use Citrix to access my company's network. I hate Citrix with a passion. If by any chance you find this page with Google, hopefully it will save you a couple of hours.

Citrix does not provide a 64-bit version of their client for Linux, only a 32-bit version. That means you need to find the libraries it needs and install the 32-bit versions of these libraries on your system. Most modern distributions ship with a kernel that can execute both 64-bit and 32-bit applications, and unless you explicitly recompiled your kernel to disable this support you should be fine. On Fedora, the 64-bit shared libraries are in different directories than the 32-bit versions, like for instance /usr/lib64 and /usr/lib (I have been told that Debian and Ubuntu use exactly the opposite).

Note: you do NOT need to install a 32-bit version of Firefox on your system. I have Citrix working fine with the default 64-bit Firefox.

Here is how to proceed. First, download the Citrix ICA client tarball from http://www.citrix.com . Untar it and install it as root like so:

$ tar xzf linuxx86.tar.gz
$ su
# ./setupwfc

Follow the instructions of the script. If you decide to install it in a different directory than the default, remember to add "export ICAROOT="install dir" in your .bashrc file.

That was the easy part. Now you can check if it works by going to your company's web site and clicking on the link for remote connectivity, and chances are - it will not. If you start Firefox in a terminal you will see errors complaining about missing libraries, like for instance libXaw.so.7, libXmu.so.6 or libXpm.so.4.

You can actually use the ltrace tool to see what libraries it is trying to load and from where. Instead of clicking on the link in your browser, save the .ica file somewhere (right click and "save as..."). Mine was called launch.ica. Then try to open this file directly with the Citrix client prefixed by ltrace:

$ ltrace -S ./wfica -file ~/Downloads/launch.ica

It will show you all the shared libraries it is trying to load, with full paths. You can then add the missing 32-bit libraries by copying them from another machine that runs a 32-bit Linux, or you can install them with your favorite package manager. On Fedora, use "yum provides" to get the name of the package that provides a library, and "yum install" to install the missing package. Note that for 64-bit Fedora it will give you 2 versions of the same package, a 64-bit one and a 32-bit one. For example:

# yum info PackageKit-gtk-module
Loaded plugins: presto, refresh-packagekit
Installed Packages
Name : PackageKit-gtk-module
Arch : i686
Version : 0.5.7
Release : 2.fc12
Size : 228 k
Repo : installed
From repo : updates
Summary : Install fonts automatically using PackageKit
URL : http://www.packagekit.org
License : GPLv2+
Description: The PackageKit GTK+ module allows any Pango application to install
: fonts from configured repositories using PackageKit.

Name : PackageKit-gtk-module
Arch : x86_64
Version : 0.5.7
Release : 2.fc12
Size : 229 k
Repo : installed
From repo : updates
Summary : Install fonts automatically using PackageKit
URL : http://www.packagekit.org
License : GPLv2+
Description: The PackageKit GTK+ module allows any Pango application to install
: fonts from configured repositories using PackageKit.

In the example above, the first one is 32-bit (arch = i646) and the second one is 64-bit (arch = x86_64). Both are installed, but if the 32-bit version was not, you could force the 32-bit installation with "yum install PackageKit-gtk-module.i686" (notice the .i686 extension for 32-bit).

Once you have all required libraries installed (it may take a few iterations), you may still face a problem: Citrix complains about missing fonts. Again you need to start Firefox from a terminal to see this:

Warning: Missing charsets in String to FontSet Conversion
Warning: Unable to load any usable fontset
Error: Aborting: no fontset found

Of course it doesn't tell you much about what the problem really is. Some people have been able to fix it as follow:

# cd /usr/share/fonts/cjkuni-uming/
# mv fonts.dir fonts.dir.old
# mv fonts.scale fonts.scale.old

In my case that did nothing. The problem was that by default the shell LANG variable was set to "en_US.UTF-8". Doing simply this fixes the problem:

$ export LANG=C
$ # restart firefox
$ firefox&

Alternatively you can export LANG=C in your .bashrc to make Citrix work even if you don't start Firefox from a shell. Remember to log off and back on to make this take effect.

Hope this helps! Here is the Bloomberg terminal in Citrix on Fedora Core 12.

Massive Scalability

Here is an interesting article about how to scale a web site from thousands to tens of millions of users. These guys have been redesigning their application over and over many times in order to get to the next level:

• First generation: 2 web servers (ColdFusion), one database. Scaled up to .5 million users which is pretty good. The database is the bottleneck.
  
• Second generation: more web servers of course, and split the database into functional areas (i.e. one database for login, one for blogs etc.), and use a SAN instead if the machines' local disks. Scaled up to 2 millions users.
• Third generation: split the database by chunks of 1 million users, plus one last function-specific database for login. Scaled up to 10 millions. The limit was not related to the unique login database but was due to the fact that the databases were not loaded evenly. A ad-hoc approach for moving data between databases didn't work well because it became a full time job for several people.
• Fourth generation: solve the uneven load problem by a SAN from 3PAR which allows to strip a volume across thousands of disks, so a single volume/DB can deliver much more IOs. Also add a caching tier between web servers and databases, and choose to store transient user data in memory instead of in the DB - a trade-off between reliability and performance (caching could have been done earlier). And finally they rewrote the application in C#/ASP which probably allowed developers to optimize their code. Scaled up to 17 millions.
• Fifth generation: switch to 4 AMD dual core 64-bit/64 GB RAM machines for databases. They have 65 databases. Scales to 26 millions.
  

Interestingly also, this stuff runs on Windows machines (I guess security is not an issue). They discovered a few funny facts about Microsoft, for instance if you try to open more connections than the maximum that SQL server can handle, it crashes :) And that Windows has a feature which makes it shutdown if the frequency of incoming network connections exceeds a limit, because it think it is victim of a DOS attack (no pun intended).

Elegance matters

I found this photo of a Bloomberg terminal on flickr. It has the reputation of being very powerful, in fact unmatched in terms of functionality, as well as lightning fast. It works with a command-line (sounds like Emacs!). You can see it in every movie that where some scenes take place in Wall Street or in the City. The last one I saw was A Good Year with Russel Crowe and Marion Cotillard (all London scenes show these terminals in the background).

Massive Project Failures

A friend of mine who works for a Big-5 consulting firm told me that she started a really big project for a company. She said that there is an army of consultants working on writing use cases, and that they plan to write use cases for a long time because there are hundreds of them. There is no developer on the project yet, but development will take place offshore when the use cases are finished. In the mean time, an architecture group is preparing the work for the future developers.

See any problem here? I do. It reminds me of another big project I worked on when I was also in a Big-5 consulting company. There was more than a hundred people on it. I gave 3 years of my life to this project, the client spent a fortune and I heard it is still not finished. The whole system was to be released at the end when all pieces are finished, instead of component by component.

Programme management is certainly no easy task, but sometimes it just takes common sense. Here is my advice to programme and project managers:

Rule #1: Hire The Right People

Your probability of success is a direct function of the skills and the motivation of your team. No amount of repeatable/optimizing processes will ever change this. Deal with it.

If you want success, hire experienced developers, which means spend the money on them. Be aware that good programmers have no trouble finding a job: they receive job offers every week regardless of the Economy. They will need motivation anyway because they will work 10-to-12-hours days and some week-ends. Of course if you manage to hire veterans you will need much smaller teams. I worked during a year and a half in a team of 12 programmers with an average of 15 years of experience, and I can honestly say that we produced more during this time than the hundred people during 3 years in the past engagement I mentioned earlier (a lot more, and our applications are used everyday by lots of people). Smart people attract smart people, so if you manage to hire a rock-star you are already half-way there.

My past projects succeeded because I was lucky to have a few good men in my teams, sometimes people that are way smarter than me. Here is a question for all project managers working with offshore development teams: have you even met your developers?

Rule #2: Evolution, Not Intelligent Design


Design your software for the next few releases only; you will adapt the architecture later if needed. A recipe for failure is an Architecture Team responsible for all design decisions, throwing application frameworks "over the wall" to the developers who are not empowered to make their own choices. I learned this lesson the hard way.

This rule is also known as: Release Often. If your project didn't put anything in production after 4 or 5 months, you have a serious problem. User feedback is critical to ensure that your application fills business needs, and for evolving the architecture. If you have to integrate mid-project releases with legacy systems so that your application can be used by the business, so be it.

The company I now work for has been one the most successful companies in America during the last 20 years. We must do something right... I noticed two things during my first few days: first, programmers are kings - they are amongst the most respected people in the company. Second, managers push you to get the product as soon as possible in front of the client, even if it is quick and dirty, so we can sell it and refine it. If clients like it we'll spend the money fixing it.