CATEGORY {Programming}


Java and Programming29 Nov 2007 11:45 am

Minification over obfuscation and other uglifier techniques

Every mature web application enters phase when you need to optimize overall performance. Unfortunately, choosing right DB, tuning your SQL queries and using sophisticated load-balancers are not enough if a user’s browser have to wait for a 5 seconds to load your overloaded AJAX-based grid. And if it happens, you have to think a lot and try many client-side optimization solutions. Now, developers have a great analyzing YSlow tool which can give you dozens of improvement tips. Actually, you can do all this work just analyzing response headers and content with another excellent tool – Firebug.

So, when our application had been optimized against almost all possible bottlenecks, it was decided to use a sort of compression (apart from gzip, which is already employed) of resource files (js, css) based on its structure.

After a little comparison and talks with DHTML gurus, I chose using minification instead of obfuscation, because the later one could be really source of evil bugs. Nowadays, the best supported minifier I found is YUI Compressor.

YUI Compressor

I chose YUI Compressor because it is Java-based free open-sourced tool which means that I always can investigate why something goes wrong, and probable contribute something valuable. This tool can be used as a standalone command line application or used from your code instantiating necessary classes. It could parse javascript files as well as css. Also there is wide range of parameters (e.g. shorten local variables or not, preserve semicolons or not).

Ant script

Good overview of the tool usage from Ant script is here. However, it is just calls to the compressor as to command-line program, which is verbose and sometimes not appropriate. Usually for such tools, there are custom Ant tasks. And it was obvious, that since sources of the tool is available, the Ant task should exist for the YUI Compressor too. After some googling, I found what I needed – semi-abandoned page Minifying JS/CSS. Many thanks to the author – Philippe Mouawad, well done! I would like to see this patch in the release and reference on the official site.

Simple example of usage:

    <path id="yuicompressor.classpath">
        <fileset dir="${yuicompressor.dir}">
            <include name="**/yuicompressor-2.2.5.jar"/>
            <include name="**/YUIAnt.jar"/>
            <!– include name="**/rhino*.jar"/ –>
        </fileset>
    </path> 

    <target name="minify-js-css" description="Minifiy a set of files">
            <taskdef name="yuicompress" classname="com.yahoo.platform.yui.compressor.YUICompressTask">
                <classpath>
                    <path refid="yuicompressor.classpath"/>
                </classpath>
            </taskdef>

            <delete dir="${js-min.dir}"/>
            <mkdir dir="${js-min.dir}" />
            <yuicompress linebreak="300" warn="false" munge="yes" preserveallsemicolons="true"
                outputfolder="${js-min.dir}" >

                <fileset dir="${js.dir}" >
                    <include name="**/*.js" />
                </fileset>
            </yuicompress>
        </target>
 

Encountered problems

Everything works fine except one malicious trouble, I was struggling with for a day. If Rhino (which is stated as dependency library)
was included in task classpath, all returning characters “\n” was pasted as a new line, and an JS error occurred.

var a = "aaa\nbbb";
// turns after minification into:
var a = "aaa
bbb";
 

This bug was eliminated by removing Rhino jar from classpath.

Java and Perl and Programming10 Jul 2007 02:22 pm

Europa release

With a short delay after Europa release announcement, I’ve moved to it. Thanks to the fact that Eclipse doesn’t require an installation process and can be just unzipped and run, the last few times I got the tuned Eclipse from my co-workers. Those distributions were well-tuned for J2EE development, with all necessary plugins, project checkstyles, attached sources and so on. Seeing as this time nobody around me offered me such a favor, I did everything on my own.
Actually, to develop a J2EE project, you need to install a dozen of plugins (depends on which frameworks are employed) in addition to the bare distribution, so it’s better to have a cheat sheet every time you start this Eclipse tuning campaign. Since I didn’t have such a plan, I used my current Eclipse 3.1 working set as an example.
This time, I wrote some short notes during the installation process, and I’d like to share it here, to have something in the future to stick with.

Download

Firstly, you need to download an appropriate Eclipse distribution. However there is already a bundle for J2EE developer, I was interested in installing everything I want from the scratch. Therefore I went to http://www.eclipse.org/downloads/ and downloaded Eclipse Classic distribution for Windows (140 MB). It was extracted to C:\Java\eclipse3.3 directory where I store all Java stuff like IDEs, JDKs, etc. Then, as usually I wanted to create a custom shortcut with extended memory allocation for Eclipse, but suddenly noticed eclipse.ini file. To allow Eclipse allocation of more memory, just edit the eclipse.ini file (increase Xms and Xmx values):

-showsplash
org.eclipse.platform
–launcher.XXMaxPermSize
256m
-vmargs
-Xms256m
-Xmx512m
 

Web tools

Some general notes. All plugins can be installed in two common ways: through cute Help->Software Updates->Find and Install… and copying all unzipped stuff to ECLIPSE_HOME/plugins directory. Of course I prefer the first option, and every plugin I’ll be installing through this wizard, except of Sysdeo.
By “default plugins” I mean plugins which can be installed through “Europa Discovery Site” (run the mentioned wizard and find it to be accustomed, if not yet).

From the past experience I knew that I was using a WTP (stands for Web Tools Platform) plugin for web development. The problem was that there were no mentioned WTP plugin on the “Europa Discovery Site”. I carried brief investigation and dig out that the main part of that plugin is a WST subproject (the web standard tools subproject).
Steps to reproduce after Help->Software Updates->Find and Install…:
WST plugin
When I checked WST check box, the wizard apparently gave a tip to select all dependent (required by WST) things too. After I installed a WST, the link to WTP update site suddenly appeared (I guess that it was due to WST), and I decided to install it to have a full stack.
WTP plugin

Subversion integration

When I installed all useful tools from the Europa repository, I moved to things which update sites have to be added manually. The process is almost the same as when we were updating standard components, the only difference is the update site where the plugin is stored. We need to add http://subclipse.tigris.org/update_1.2.x as a New Remote Site…
Update manager

Tomcat launcher

The simplest and the most valuable plugin for Eclipse invoking I ever used is a Sysdeo Tomcat launcher. Download the archive and install according to Installation section steps. Here, the unzipped archive should be placed manually to the ECLIPSE_HOME/plugins directory.
Sysdeo butons

Useful tools

As you know how to install plugins, I’ll mention only links to update site:

  1. http://eclipse-cs.sourceforge.net/update – checkstyle plugin, helps to maintain your coding style due to different available code conventions (Sun, Eclipse, your own)
  2. http://springide.org/updatesite/SpringIDE – Spring related things like bean definitions, xml configuration auto completion
  3. http://e-p-i-c.sf.net/updates – Perl IDE, just for fun to play with RegExps or to write some admin tools
  4. http://www.fabioz.com/pydev/updates – PyDev (Python IDE) not to be restricted only by Java world and be more broadminded
  5. http://eclipse-tools.sourceforge.net/implementors/ – nice Alt+F3 short key usage, allows quick jumps to implementation from interfaces.

Stay tuned!
P.S. I know that saying this phrase is almost as popular as mentioning iPhone in the blog post :)

Code snippets and Java and Programming31 May 2007 06:29 am

File names default sort order

If you are developing an application which works with files, one day you’ll try to get list of the names of all files in a particular directory. It could be like this simplified snippet:

    public List<String> getAllImageNames() {
        List<String> names = new ArrayList<String>();
        File imagesDir = new File(IMAGE_BASE_DIRECTORY));
        if (!imagesDir.exists()) {
            return names;
        }
        for (File image : imagesDir.listFiles(COMMON_IMG_FILTER)) {
            names.add(image.getName());
        }
        return names;
    }

    private static final FileFilter COMMON_IMG_FILTER = new FileFilter() {
        public boolean accept(File pathname) {
            return pathname.isFile() && (!pathname.getName().startsWith("."));
        }
    };
 

Let’s assume that you have some image files with the same name suffix and an index number as a trailing part:

System order Natural order
IMG_1.jpg IMG_1.jpg
IMG_2.jpg IMG_2.jpg
IMG_21.jpg IMG_3.jpg
IMG_22.jpg IMG_21.jpg
IMG_3.jpg IMG_22.jpg

Everything seems to be all right, but returned list is sorted in a way Java usually sorts strings. While this results could completely satisfy your program API, it is not very useful to work with such lists for a human. Moreover, such lists are very common in our life, for example, many digital photo cameras store pictures with such names.
Let’s make the problem more general. We have a list of strings, which could contain digits, and we want to sort this array to get a natural ordered list. You’d say that it is piece of cake, because Java has good facility to perform array/list sorting : Collections.sort() and Comparator. Therefore, we need an appropriate Comparator implementation.

Known solutions for natural order sorting

Unfortunately, there are no any well-known library like Jakarta Commons to perform this sort. And if you try to google it, you will find only some posts like this one with home-grown solutions and grumbling about lack of existing proven solution. The most important thing is to know the right keywords to google against it. Most of articles on this topic are about Natural order.

I found some valuable resources:

  • Natural String Order at Stephen Friedrich’s Blog – an article of Java professional with working Java source and even JUnit tests. The implemented algorithm is very comprehensive and can work with different set of sorting rules (Ascii, case insensitive and locale specific).
  • HumaneStringComparator: Sorting Strings for People – another one Comparator implementation by Tim Fennell.
  • Implementation by Pierre-Luc Paour.
  • Natural Order String Comparison – C version of implementation and links to another languages.
  • Natural Order Numerical Sorting – overview of the problem, links to another articles and solutions, but almost all of them are broken or not Java. It seemed that the author was keen about that idea, registered a special domain and gathered essential information, and then left this site floating without any maintain.

Stress test

One of the main software development rule states: don’t invent your own wheel unless the problem is not your core business. As string sorting was a utility purpose matter, before implementing my own solution I decided to test already found. The test was easy and straightforward – every Comparator had to sort the same number of randomly generated and shuffled strings. The strings were generated by fixed pattern: [a-z][0..1000].jpg. Input data was the large array (100000) of such strings:
..............
jqazrrveqy113.jpg
wzedsgzmvo912.jpg
aayexqpfdu311.jpg
zvpzxjwkml354.jpg
nelacribtl964.jpg
rehsgmzugb244.jpg
eoptzxybtz459.jpg
ukbeogpmhe157.jpg
zgvnrzohwc176.jpg
.............. .

I was interested in only one algorithm efficiency parameter – time elapsing during full sort. As usual, the elapsing time itself doesn’t tell much about the efficiency. To have a minimum achievable value to compare with I chose a result given by the comparator based on a standard Java String.compareTo() method which should be the fastest because it doesn’t deal with string parsing.

        Comparator<String> baseComparator = new Comparator<String>() {
            public int compare(String strA, String strB) {
                // Assuming that strA always != null
                return strA.compareTo(strB);
            }
        };
 

The test results are shown in a table:

Author Version Sort Time Pros Cons
Pierre-Luc Paour NaturalOrderComparator 453ms Quite fast hard to read C-style algorithm
Stephen Friedrich NaturalComparator 4828ms Locale specific Too slow
Stephen Friedrich NaturalComparatorAscii 360ms The fastest one Only ASCII
Stephen Friedrich NaturalComparatorIgnoreCaseAscii 500ms Fast enough, case insensitive Only ASCII
Tim Fennell HumaneStringComparator 4797ms Very good example of OOP style, brief understandable algorithm, use Java facilities extensively Too slow and inefficient
Java native String.compareTo() 235ms standard standard

Conclusion

As you can see from the result sheet, the most powerful and fast is Stephen Friedrich’s package. It is written with good Java style, well commented and supplied with JUnit tests. It provides different kind of sorting depends on what are going to sort – simple ASCII strings like file names or country specific strings containing special characters.
Of course, one day I’d like to see any well-proven solution in the Jakarta Commons project and simply add it as a jar.

Blogging and Programming and Social28 May 2007 11:41 am

As I stated a few days ago, I mistakenly broke my previously used FastTrack theme. I liked it very much because of the main photo at the top. There depicted a young men with a rucksack walking along underground station. As his face was hidden and his picture was blurred, his character resembled me. I like wearing rucksacks and usually use subway if I need to get somewhere. This fact made the theme quite personal, and I didn’t need to customize it more for a long time. After I broke the theme and I had some hours to get down to WordPress Themes internal kitchen, I created my own theme.

To not take long, I’ll write my work log:

WP Theme editing on the local machine

Editing a WP theme remotely is a real pain. In this case, you have to upload/download your theme sources every time you change something. More easy way is a installing your WP copy on the local machine. There are lots of howto-s over the Internet. I’m little bit familiar with Apache+PHP+MySQL bundle, and installed it within 30 minutes. After this, I downloaded my current WP files from my hosting directory and upgraded it the latest WP version (2.2).

Tools

I took the FastTrack theme as a base for my new theme. The markup of all themes are very similar, because all of them uses the same layout: header, content, sidebar and footer. All parts are stored in few .php files with correspondent and clear names. The appearance of a theme is mostly determined by –°SS file with styles of all parts. Hence almost all time you have to work with a CSS file editor. For this purpose I chose the great FireBug Add-On which show all comprehensive information about a web page including a CSS styles and HTML components on the page. PHP files I edited with Komodo and CSS styles with TopStyle. For choosing right color schema I found out cool on-line tool for color schema generation – kuler.

Creation

I took one of my photos which looked fine as a header image and cropped it to the proper size. Then I tidied up all unnecessary elements like top navigation, odd styles, etc. The width of the post area became wider on 100px. Then I chose a color schema using kuler. It took some time to play with links appearance and font sizes. And in a few hours the new theme I called harpsichord was done.

If you want to customize your theme, I can suggest you starting from Guides to creating your own themes.

P.S.
I eventually tasted the new-brand Eclipse chewing gum :) For those who are unfamiliar with my special interest to it, I’d say that Eclipse is the most popular open-sourced IDE for Java language (not only however).

Eclipse chewing gum