A Java installer problem fixed by reverse engineering

Since I started running Serviio as my media server, I noticed that Java upgrades fail if I don't stop Serviio. The installer does not notice the server and warn me about it using Java. Instead, it proceeds with the install and tells me that I have to reboot. After a reboot, Java is broken and I need to reinstall it.

By now, I learned to avoid this problem. Unfortunately, when I upgraded to Secunia PSI 3.0.0.4001, it started upgrading Java even though I configured it to ask for confirmation first. When the upgrade finished, Windows restarted without any confirmation prompt. Once Windows was back up, I expected I just had to reinstall Java. Unfortunately, something was broken now, preventing any Java installation or uninstallation from succeeding.

There was no error message, so the first step was obtaining some information via Windows Installer logging. Here is the error I found:

MSI (s) (70:64) [14:09:01:503]: Invoking remote custom action. DLL: C:\Windows\Installer\MSI724C.tmp, Entrypoint: MSICheckFIUCancel
CustomAction CheckFIUCancel returned actual error code 1602 (note this may not be 100% accurate if translation happened inside sandbox)

This means some function in a DLL included with the installer is returning an error code. The DLL is temporarily written to disk, and it cannot be seen at that location afterwards.

The Java offline installers such as jre-7u9-windows-i586.exe extract themselves into corresponding subdirectory of %USERPROFILE%\AppData\LocalLow\Sun\Java\. That's where you can find Windows Installer files such as jre1.7.0_09.msi. This allows one to use msiexec from the command line. I found I could successfully install and uninstall Java with no user interface, using these commands:

msiexec /x jre1.7.0_07.msi /qn
msiexec /i jre1.7.0_09.msi /qn

However, even after a successful uninstall followed by a successful install, a normal install and uninstall still failed. To solve the problem, I would have to look at this function that keeps failing. I used Universal Extractor on jre1.7.0_09.msi, choosing the MsiX method. This created a folder with files from inside the msi file.

The MSICheckFIUCancel function was found in Binary.installerdll.dll. A quick look at the disassembly showed that the function used the string "Software\JavaSoft\FIUCancel". I searched for that in the registry, and found a key by that name at HKEY_CURRENT_USER\Software\JavaSoft\FIUCancel. It was empty. I chose to try deleting that key before analyzing the code further. It fixed the problem! There was no need to spend more time on analyzing the code. Both the installer and uninstaller worked.

The key lesson here is that reverse engineering can quickly and easily fix a problem. Yes, fully understanding a large amount of code tends to be difficult and time consuming. However, finding the problem often only requires noticing a small relevant part, and that can be fast. It can be a lot faster than attempts to fix a problem without actually looking at what's going on. Reverse engineering is like the difference between stumbling around in the dark and turning on the light and looking. Of course, free software is best, because then you can actually look at the source code.