After the initial learning curve, this has all been going swimmingly. I merged my first development branch into the trunk yesterday, and this branch just so happens to dovetail nicely into the whole point of this post, which is the YUI compressor, an open-source javascript and CSS minification tool developed by Yahoo’s YUI team.

The Problem

To understand quickly why one should minify production client-side code, consider only that with the upward trend of size in javascript libraries (and the necessary files for such libraries), it’s possibly to be downloading a lot of client-side code in a typical web application, especially as its scope grows.

For a long time, I was using Dean Edward’s Packer, as was everyone, because its Base62 encoding produced the very lowest file size. However, what should have been obvious to everyone is that eval(bunch_of_stuff_goes_here) is making the browser do a lot more work, and this is a problem on dinosaurs like IE6.

To make matters worse, the nature of such encoding also meant that for servers which tried to compress outgoing content like javascript (either with zlib or gzip), the compression ratio suffered. Just look at this table that Julien Lecomte posted last August.

I said to myself, “Hey, we use a lot of this stuff, and we still support a lot of users with slow computers and slow browsers.” So, I moved our project from a Packer-based compression to a YUI-based compression, and turned on server-side GZIP compression for javascript files. The only problem was that I was storing the javascript files already minified, and simply pasted them into a large a couple of large global .js files. I had to keep a separate source directory, along with any customizations.

This got to be a pain in the ass, as you might well expect, and so when it occurred to me that I might be able to use the YUI library at build-time in our Netbeans project, I immediately sprung into action.

This was around April, and one night while attending Sungard Summit in Anaheim. I did the initial work to get our Netbeans project into a state that could use the YUI compressor at build-time, creating separate source file directories and breaking our massive javascript file into modules; I did the same with our CSS, splitting it up based on what it decorated.

There are a few tutorials about using the YUI library. Some of them involve adding the YUI library to Ant’s classpath (didn’t want to go down this route); a lot of them involve invoking the library as an external executable during the build process, which is messy.

The solution I finally settled on was yui-compressor-ant-task, a small library that allows Ant to use YUI as a build task. By adding this library and the YUI compressor library to our common libraries folder, and enabling them at build only (and not for deploying in the web archive), it makes using the compressor pretty easy.

Here’s part of our build.xml:

<!--
   * minify will concatenate all of our non-TinyMCE javascripts and stylesheets
   * then use the YUI compressor library to compress them
   -->
   <target name="minify">
       <!--${libs} is path to the downloaded jars -->
       <property
           name="yui-compressor.jar"
           location="${file.reference.yuicompressor.jar}" />
       <property
           name="yui-compressor-ant-task.jar"
           location="${file.reference.yui-compressor-ant-task.jar}" />
 
       <path id="task.classpath">
           <pathelement location="${yui-compressor.jar}" />
           <pathelement location="${yui-compressor-ant-task.jar}" />
       </path>
 
       <!-- yui-compressor task definition -->
       <taskdef
           name="yui-compressor"
           classname="net.noha.tools.ant.yuicompressor.tasks.YuiCompressorTask">
 
           <classpath refid="task.classpath" />
       </taskdef>
 
       <!-- concatenation of javascript -->
       <echo message="Building global javascript" />
       <concat destfile="${build.dir}/web/common/js/global.js" force="no">
           <!-- explicitly order js concat because ordering matters here -->
           <fileset dir="${build.dir}" includes="web/common/js/jquery.js" />
           <fileset dir="${build.dir}" includes="web/common/js/jquery.bgiframe.js" />
           <fileset dir="${build.dir}" includes="web/common/js/jquery.hoverIntent.js" />
           <fileset dir="${build.dir}" includes="web/common/js/ui.core.js" />
           <fileset dir="${build.dir}" includes="web/common/js/ui.autocomplete.js" />
           <fileset dir="${build.dir}" includes="web/common/js/ui.datepicker.js" />
           <fileset dir="${build.dir}" includes="web/common/js/ui.tabs.js" />
           <fileset dir="${build.dir}" includes="web/common/js/tablesort.js" />
           <fileset dir="${build.dir}" includes="web/common/js/customsort.js" />
           <fileset dir="${build.dir}" includes="web/common/js/jquery.blockUI.js" />
           <fileset dir="${build.dir}" includes="web/common/js/jquery.form.js" />
           <fileset dir="${build.dir}" includes="web/common/js/jquery.ifixpng.js" />
           <fileset dir="${build.dir}" includes="web/common/js/jquery.superfish.js" />
           <fileset dir="${build.dir}" includes="web/common/js/jquery.cluetip.js" />
           <fileset dir="${build.dir}" includes="web/common/js/jquery.scrollTo.js" />
           <fileset dir="${build.dir}" includes="web/common/js/jquery.jqModal.js" />
           <fileset dir="${build.dir}" includes="web/common/js/validation.js" />
           <fileset dir="${build.dir}" includes="web/common/js/jquery.timeentry.js" />
       </concat>
 
       <!-- concatenation of cascading stylesheets -->
       <echo message="Building global cascading stylesheets" />
       <concat destfile="${build.dir}/web/common/css/global.css" force="no">
           <fileset dir="${build.dir}" includes="web/common/css/base.css" />
           <fileset dir="${build.dir}" includes="web/common/css/superfish.css" />
           <fileset dir="${build.dir}" includes="web/common/css/announcements.css" />
           <fileset dir="${build.dir}" includes="web/common/css/myvt.css" />
           <fileset dir="${build.dir}" includes="web/common/css/forms.css" />
           <fileset dir="${build.dir}" includes="web/common/css/cluetip.css" />   
           <fileset dir="${build.dir}" includes="web/common/css/tables.css" />
           <fileset dir="${build.dir}" includes="web/common/css/ui.tabs.css" />
           <fileset dir="${build.dir}" includes="web/common/css/ui.datepicker.css" />
           <fileset dir="${build.dir}" includes="web/common/css/ui.autocomplete.css" />
           <fileset dir="${build.dir}" includes="web/common/css/linkspan.css" />
           <fileset dir="${build.dir}" includes="web/common/css/stepMenu.css" />
           <fileset dir="${build.dir}" includes="web/common/css/print.css" />
           <fileset dir="${build.dir}" includes="web/common/css/youHaveMessages.css" />
       </concat>
 
       <!-- invoke compressor -->
       <yui-compressor warn="false" charset="UTF-8" fromdir="${build.dir}" todir="${build.dir}">
           <include name="web/common/js/global.js" />
           <include name="web/common/css/global.css" />
       </yui-compressor>
 
   </target>
 
   <!--
   * purge-src takes our compressed files, moves them to the base /common dir
   * and deletes the source js and css dirs from the build dir
   -->
   <target name="purge-src" depends="minify">
       <echo message="Purging javascript and stylesheet sources" />
 
       <move file="${build.dir}/web/common/js/global-min.js" tofile="${build.dir}/web/common/global.js"/>
       <move file="${build.dir}/web/common/css/global-min.css" tofile="${build.dir}/web/common/global.css"/>
 
       <delete dir="${build.dir}/web/common/js" />
       <delete dir="${build.dir}/web/common/css" />
 
   </target>

What you see there is essentially four steps.

1. Concatenate all the constituent source files into a global.js and a global.css
2. Compress both of these files, which creates global-min.js and global-min.css (default behavior)
3. Move these files out of the source directories and into the root of the common web directory as global.js and global.css
4. Delete the source directories in our build folder so they don’t get deployed with the web archive

Because certain browsers (IE) break without explicit ordering, we unfortunately can’t just use “*.js” and “*.css” in our concatenation step, but having to explicitly list our components in the build file certainly isn’t the end of the world. The nice thing is that the Ant task will even print out handy statistics on just how much you’ve been able to compress the files down. In our case, we have about 441.8KB of common Javascript and CSS in our source code that, by the time it gets sent to the user, has been minified and/or gzipped to about 89KB.

Our addition to the program this year was relatively recent occurrence, and so this conference came as a bit of a surprise, wedged as it is just before I leave next Monday for the Portal 2008 conference at Gettysburg College.

Conference

In the interest of diplomacy, I feel obligated here to mostly withhold my more candid observations about the conference. I will say that my university remains far ahead of other schools in terms of what we offer portal users, technologically. Say what you will about our relatively low-tech JSP + JSTL approach, but it works. There seems to be a significant struggled for consortium members (attendees of the conference) to develop useful applications: in the absence of development manpower, they turn to CampusEAI and its community resources for code, but CampusEAI’s understandable approach is to abstract much of what they write to be n-tier and properly MVC therefore applicable to any school on any platform. Thus, even with the bulk of the code written, there’s still the crucial step of writing the data access layer that hooks the business logic to whatever data store the school has. Generally speaking, it seems, schools that don’t have the technical resources or man hours to develop their own portlets also don’t have the technical resources or man hours to customize and personalize these shared resources for their environment.

Then, too, a great deal of the trouble experienced by schools is more infrastructure: we long ago abandoned Oracle Portal, which is the primary focus of CampusEAI, because it was ridiculously expensive, required a shitpile of hardware thrown at it, and was still an unreliable, clunky Frankenstein’s monster of a product, loosely cobbled together from a mix of open-source components and proprietary glue layers. It also looks like hell, allowing for very little customization. I am not afraid to dismiss this product on its lack of technical merits: Oracle Portal is utter excrement, and you’d be better served by something like uPortal or Liferay.

Since a lot of the conference, therefore, was geared toward solving problems that we don’t have, not every session was particularly helpful. I did, however, meet some very cool people. The folks from Rensselaer Polytechnic Institute come immediately to mind; we talked with them quite a bit, in part because they’re cool guys, but also because they are the lead developers for the Bedework web calendar, which we’re looking into implementing at USF. We also talked a lot with some folks from Loyola-Marymount, as well as Loyola University Chicago.

So, in that respect, the conference was a lot of fun; I’m not an immediately personable guy, so networking in the social sense isn’t always easy for me, but I think I’m coming into my own here.

Errata

I still don’t care for Cleveland. We were in the middle of downtown Cleveland, and besides being an odd mix of grand marble bank headquarters and run-down storefonts, it was also a virtual ghost town. Perhaps it was the heavy construction going on, but the streets were deserted by 6pm, leaving only the inescapable homeless people inventing sob stories as they asked you for change. The problem is so bad, apparently, that the city puts up signs advising pedestrians not to give any money to the homeless; pathetically, it’s a bit like signs at the zoo warning onlookers not to feed the animals.

I should point out that I had good beer while I was there: we ate at the Winking Lizard one night, and an Irish pub called Flannery’s the next; I had Bell’s Oberon on tap at the former and Murphy’s Stout on tap at the latter, and it was bliss. Also, Flannery’s has the best onion rings I’ve ever tasted. They were so good, I think they more or less trump everything else about my trip.

Yet, I find myself at home, with a White Russian and the urge to opine.

The Background

In the summer of 2006, my workplace (The University of St. Francis, my alma mater) went live with a portal. For the previous year and half, when I was a student worker, we’d been working desperately to implement Oracle Portal. At a conference that summer, my boss, Tim, heard about uPortal, an open-source J2EE portal. On the ride back from Ohio, we more or less decided to try it on a lark. It’s basic ease of setup let us switch to uPortal and go live in 6 weeks. That’s right, a production-ready portal in 6 weeks.

Much of the work in that time was tying CAS into our LDAP implementation (Novell’s eDirectory), customizing the view (props to Virginia Tech for their excellent Tab-Column customization), and writing some basic portlets to allow quick views into student data: schedule, personal info, billing, &c. We’re a Banner school, so we could basically just reach into our Oracle database and pull the data we needed.

The Cool Bits

Since that time, we’ve poured a tremendous amount of energy into the portal, as it’s become our go-to spot for finding information and getting things done. Actually, we’ve spent very little time on uPortal at all, as it already contains much of the quick-view functionality we want. Much of what we do is a custom extension to the portal that transparently shares session data via CAS.

We have completed our portal rollout using uPortal (running on apache/tomcat), and are integrated with Sungard Banner 7.x for much of the information we pull and display, and we are using Novell E-Directory for our directory.

We used uPortal which proved to be a much more promising and easier to use platform than Oracle portal. Within 6 weeks we had our first production portal deployment. Over the last year we have built and put the following into production:

• Student Schedule and Grades Display (pulled realtime from Banner)
• WebCT Integration/SSO
• Announcements (Targeted coming by end of summer ’07)
• SSO to Novell WebMail, Netstorage, and Banner Self Serve
• Bookstore Integration with Barnes and Noble (Shopping cart of books)
• Financial Aid Portlet/Accept Loans – Display and update banner data
• Tuition Bill Portlet – Pulled in realtime from Banner AR
• Payment of Tuition Bill/Post Directly to Banner AR
• My Profile Portlet (information pulled realtime from Banner)
• GPA/Credit Hours Portlet (Pulled realtime from banner)
• Banner Security Question/Reset Password Functionality
• Expired Password Functionality
• Faculty Schedule/Class Rosters/Email Class, Export to Excel
• Banner Survey Integration
• HR Leave Balances
• Budget Reporting/Financial Activity/View Documents, checks cut, etc
• Banner Requisition Approval
• Web Based University Income/P&L Statement
• Banner Finance Journal Voucher Upload
• PIDM Usage Utility to help identify and resolve duplicate pidms
• EZProxy/Library Databases Integration
• Advancement Constituent Profiles
• Advancement Fundraising Activity Reports
• Advancement Constituent Contacts Data Entry/Reminders
• Financial Aid targeted messages
• GUAMESG
• Residence Life View your Room/Roomates
• Bulletin Boards (phpBB)
• VT Calendar
• Various Web based Reports
• E-Directory Integration/Account Creation
• Early alert and midterm warning submission, that allows the student, advisor, registrar, athletic coaches, and other support staff know when a student is having academic difficulties.
• An online advising center. Advisors can now see all of their advisees, look at who’s registered, view issues with the students account such as registration or medical holds, etc. The goal is so that advisors can also help the students get their issues resolved.
• Our initial stab at targeted announcements is done. We can now target based on the student type, level, or college as currently stored in the Banner ERP, as well as target to only employees or faculty members. The framework is in place to target off of any data element stored in Banner. Now we just need to expand our GUI allowing users to target messages.
• Accounts Receivable Reporting Suite – Our finance office now uses the portal to do various AR reporting tasks, drilling into student details, etc.
• Enrollment/Registration Status Reports – Various members of the university can now easily look at the overall student registration status through the portal.
• A faculty attendance tracking module for faculty to electronically submit the attendance of students in their classes at the beginning of term and at the mid-term point. This helps the registrar and financial aid determine who is actually attending, and replaces an old paper based process.
• On online Student Check-In procedure. Students can now confirm their class schedules, financial aid, and pay their bill through an easy to use “wizard” interface. The system will notify when they need to perform the electronic check in process before the start of the term, and then walks them through the process. This helps to collect payments sooner and determine which students are actually going to show up to class on day 1.
• Our portal now has a targeted messaging facility which looks at business rules and notifies the students of issues or action items related to their student records. For example, if a student has holds on their account the system will tell them, show them the hold, tell them what it prevents them from doing, and who to contact to resolve the hold. For students with incomplete medical records they are notified, and shown which parts of their medical records are incomplete (and for our nursing program, which shots, etc they need to have on file to be able to attend clinicals).
• A co-curricular transcript which pulls from Banner all the clubs, organizations, and sports a student has been a member of during their time at the university, and shows them in a nicely formatted page that they can print.
• A real-time registration status dashboard for administrators, deans, and faculty members. This dashboard now shows the head counts, course registration counts, credit hours, and billing hours generated for registered students, and the number of potential students who are left to register. It breaks the summary counts down in several ways: by college, major, advisor, or level (graduate, undergrad, etc). The users can drill into this report and extract lists of students who still need to get registered, whom to call to see if they need advising help or assistance registering, etc.

As the UI architect for the portal extension, I decided this past summer to standardise on jQuery, and certainly haven’t regretted it. The only bad thing about javascript is that it becomes easy to rely on it for things which should really be controlled server-side. Important validation checks, for instance. Since our ERP system has certain business logic which it enforces on the database level, we usually do two checks for important data: one (javascript) for the user’s sake, and one (servlet) for ours.

It got to the point where we were loading 10 different jQuery javascripts and 2 stylesheets in our header. Most of the javascripts were either packed or uncompressed (if packing broke them), and totaled about 200Kb. Not only was this too much, but the eval() caused a delay for each script. Recently, after one of our rollouts to production, when I got more leeway to play, I completely upgraded our script layout. I’m not going to provide an apples-to-apples comparison table here, since the prior configuration was a hodgepodge, but after minifying and turning on server-side compression for javascripts and stylesheets, I managed to reduce the load from 200Kb in 12 requests to about 50Kb in 5 requests. According to Julien Lecomte, running mod_deflate on a minified (but no obfuscated) javascript yields the lowest file size; regardless of how often that’s true, I like the idea of avoid latency from the eval(). A lot of our users are still on IE6, and they take some pretty big performance hits.

Goodness knows there’s more I could do, but given the nature of our portal (single-image CSS layouts, anyone?), it wouldn’t be a productive use of our time); I have enough crap to do with counting pixels.

jQuery rocks my socks

I might take a moment here to say just how much I love jQuery. For those of you who don’t know, jQuery is a javascript framework which abstracts a lot of functions into a much more easily-written manner. In the past, I’ve sort of been a Copy & Paste javascript programmer, since it’s been easier to use some free script than to write one’s own code. jQuery and it’s vast array (har!) of plugins makes it easy to implement or write new code. For instance, if we want to implement a support for CSS hover effects in IE, all we do is this:

$("input:text,input:password,textarea").focus(function() {
     $(this).addClass("focused");
});
 
$("input:text,input:password,textarea").blur(function() {
     $(this).removeClass("focused");
});

Easy as pie.

Granted, there are a lot of frameworks to choose from. Originally, there was only Prototype; now there’s also jQuery, Dojo, YUI, Mootools, Rico, and any one of a number of others.

The one I wanted to talk about in this post in particular is ExtJS, which was originally an extension of the YUI framework, but which eventually split off into its own project. It recently went 2.0, and boasts some rather impressive demos. I marvel at the Vista-like UI, which, admittedly, is very slick.

But the more I look at ExtJS, and half-heartedly consider its possible deployment within our portal, the less enthused I become. ExtJS’s grid, for instance, doesn’t even function on existing HTML table markup: you mass it an array of data as a javascript variable, and it creates markup on the fly… a lot of markup, in order to create it’s slick Office 2007 interface. While this is good insofar as it manages to completely abstract data from presentation, I’m not sure how much I like the idea, since it (1) relegates users to the script default without heavy customization, and (2) doesn’t allow for any kind of fallback mechanism if the javascript should fail or the user has it disabled. Our table-sorting function, in a case like that, doesn’t at all prevent the table markup and data itself from being rendered.

Work can be fun sometimes.