PolyGlot, Inc. News

A rant about Front End Developers not knowing Modular Programming

2011-11-04T15:39:00.001-04:00

Here is a post of mine to an internal support forum at a client site, where I have to teach front end developers about basic software engineering (begging the question why people hiring them don't demand software engineers instead of self-taught programmers....rant, rant, rant....). I keep finding FEDs that think that any piece of DOM, bit of JavaScript, or chunk of CSS that they can see via Firebug is fair game to fiddle with...rant...rant...

Why does my App break when I upgrade to the new version of the Framework?

This thread is a general purpose Q&A on the topic of code breaking when a newer version of [GUI Framework] is used, and it is illustrated by a particular real-world scenario...

Example Question: "The following code was provided to us by [GUI Framework] team for auto complete in [widget] a long time ago when we were working on the [foo project] with the [GUI Framework] team. In there, you can see there are quite a few references to the private attributes. ... ... ... functionality is not working as before. Can you please provide pointers to resolve this."

Example Answer: "... A quick look at the code in the attachment makes me believe that it was a hack to replace the internal workings of the list filtering…you SHOULD have been provided an above-board way to subclass/decorate/configure [widget] (instead of that quick and dirty method)."

There are general software engineering principles of "encapsulation" and "information hiding" which come from the Modular Programming era in the mid-1970s (just before Object Oriented expanded upon it). To combat "brittleness" of a system, Components hide their (private) implementation details inside a (public) interface so that changes in one part of a system don't break other far-flung parts of that system (aka "loose coupling"). The warranty provided by the component is that as long as you use it as designed, and via the stable interface, the insides are free to be changed/fixed/enhanced without affecting you.

So, a component (where "widget" is just another word for component) is not just a collection of functionality like the collection of bits and wires below...

...but rather a collection of functionality that has had a stable interface published on the outside of a "black box" thus hiding the implementation inside that box, as shown below...

This warranty is voided however if you break into the box and connect to its private implementation (aka "hacking") as seen below...

In [GUI Framework], there are two major indicators as to whether a property or method of a widget is public interface versus private implementation:

If the name of the property or method starts with an underscore ("_") then it is definitely private (or at best "protected")
If the property or method is documented in the [GUI Framework] JsDoc API Docs then it is definitely public.

SO, IN CONCLUSION, if someone advises that non-documented or private parts of a widget be used by your app, you should get them to sign a document acknowledging that they are voiding their warranty, and they will pay the costs incurred when components are replaced or upgraded later (as they will be EVEN IF YOU REMAIN USING THE SAME VERSION OF [GUI Framework] because only the Interfaces are promised to be kept stable for a release, not the Implementations!)

Performance Tips of the RIA All-Stars

2011-02-25T10:27:00.043-05:00

CODING and LOADING

This guide will summarize the performance-related DOs and DON'Ts of JavaScript and Web Site development as espoused by well-known pundits like Douglas Crockford. The publications listed below are surveyed, but since they overlap, this guide can act as both summary and cheat sheet for their catalog of techniques. The tips fall into 2 major categories: CODING and LOADING. i.e. making JavaScript and CSS performant, and minimizing the apparent time needed to load responsive user interfaces.

[HPW] High Performance Web Sites, Steve Souders, O'Reilly, 2007
[JGP] JavaScript: The Good Parts, Douglas Crockford, O'Reilly, 2008
[EFW] Even Faster Web Sites, Steve Souders, O'Reilly, 2009
[HPJ] High Performance JavaScript, Nicholas Zakas, O'Reilly, 2010

As with any optimization effort, the number one rule is to only optimize the portions of a system that are bottlenecks as actually measured at run time. Often, optimization techniques have serious downsides that require tradeoff judgments to be made. For example, Crockford [JGP, P.65] says both that "Regular expressions usually have a significant performance advantage over equivalent string operations in JavaScript" AND "regular expressions can be very difficult to maintain and debug". Souders [EFW, P.6] says "Everything is a trade-off. When optimizing for performance, do not waste time trying to speed up code that does not consume a significant amount of the time. Measure first. Back out of any optimization that does not provide an enjoyable benefit." Galbraith [EFW, P.9] says "developers seeking to create responsive, high-performance web sites can't, and shouldn't, go about achieving that goal by optimizing every single piece of code as they write it. The opposite is true: a developer should optimize only what isn't fast enough."

On the other hand, there are basic practices (like not putting operations inside a loop that can be done once outside the loop) that can be followed right from the start. As Zakas [HPJ, P.XII] says, "JavaScript forces the developer to perform the optimizations that a compiler would normally handle in other languages". This catalog will contain a bit of each category.

And finally, in a very few cases, these authors contradict each other; for example, bitwise operators are either really fast, or really slow, depending on who is judging. Crockford [JGP, P.112], in Appendix B: Bad Parts, says bitwise operators are "very slow", whereas, Zakas [HPJ, P.156], in Use the Fast Parts, says bitwise operators are "incredibly fast"!

Caveats:

This guide assumes that the reader is already familiar with basic but modern JavaScript/Web development, and leaves to another guide the task of bringing the reader up to that level.
While the books referenced above contain a wealth of development advice, this guide will only summarize those tips that are related to improved performance.
Each tip is a summary of a high level idea. Please refer to the book sections referenced for more implementation details as well as discussion of alternate techniques for more specialized situations.

Section 1: Coding Tips

"JavaScript forces the developer to perform the optimizations that a compiler would normally handle" - N. Zakas

Arrays are not Arrays...

[JGP, P.58,105] "[Conventional] arrays can be very fast data structures. Unfortunately, JavaScript does not have anything like this kind of array. ... their performance can be considerably worse than real arrays". Arrays in JavaScript are basically objects (i.e. dynamic bags of properties) whose property names are merely the string version of the integer subscripts.

...but String Arrays are faster than String Concatenation

[JGP, P.78][EFW, P.99] "If you are assembling a string from a large number of pieces, it is usually faster to put the pieces into an array and join them than it is to concatenate the pieces with the + operator." The more pieces that are concatenated together (think very big loops here), the faster it is to use Array.join.

    var a = []; 
    a.push('a');  a.push('b');  a.push('c');  a.push('d'); 
    var c = a.join('');   // c is 'abcd' and faster than c = 'a'+'b'+'c'+'d';

[EFW, P.99] While modern browsers have improved string performance to the point that this advice may be out of date, never the less, "The performance decrease of the array technique in other browsers is typically much less than the performance increase gained in [pre-v8] Internet Explorer".

Fastest implementation of the missing String.trim function

[EFW, P.101] JavaScript is missing a string trim function, and it is usually implemented by the application. After performing research on the fastest way to execute string trimming in JavaScript, the following function consistently performs better than other variations:

    function trim( text ){ 
        text = text.replace(/^\s+/, ""); 
        for (var i=text.length-1; i>=0; --i) { 
            if (/\S/.test(text.charAt(i))) { 
                text = text.substring( 0, i+1 ); 
                break; 
            }
        }
        return text; 
    }

Faster to process an array of properties than sorting an enumeration

[JGP, P.24] "The for in statement can loop over all of the property names in an object... [but] there is no guarantee on the order of the names." So, rather than having to sort them (not to mention filtering out all of the unwanted inherited properties), use a normal for loop against an array of property names in the desired order. E.G.

    var i, pn = [ 'first-name', 'middle-name', 'last-name' ]; 
    for (i=0; i < pn.length; ++i) { 
        document.writeln( pn[i] + ': ' +  personObject[ pn[i] ] ); 
    }

Use Simple Regular Expressions instead of string manipulation

[JGP, P.65, 69] "Regular expressions usually have a significant performance advantage over equivalent string operations in JavaScript...[however] regular expressions can be very difficult to maintain and debug...[and] the part of the language that is least portable is the implementation of regular expressions. Regular expressions that are very complicated or convoluted are more likely to have portability problems. Nested regular expressions can also suffer horrible performance problems in some implementations. Simplicity is the best strategy."

Eval is Evil (and so are its aliases)

[JGP, P.110] Slow performance is just one reason to not use eval() and its equivalents. Between the two lines below, line one "will be significantly slower because it needs to run the compiler just to execute a trivial assignment statement...The Function constructor is another form of eval, and should similarly be avoided. The browser provides setTimeout and setInterval functions that can take string arguments or function arguments. When given string arguments, setTimeout and setInterval also act as eval. The string argument form also should be avoided."

    eval("myValue = myObject." + myKey + ";"); 
    myvalue = myObject[ myKey ];

Always store multi-accessed values in local variables

[HPJ, P.20-33] [EFW, P.79-88] "Generally speaking, you can improve the performance of JavaScript code by storing frequently used object members, array items, and out-of-scope variables in local variables. You can then access the local variables faster than the originals. ... A good rule of thumb is to always store out-of-scope values in local variables if they are used more than once within a function. ... Literal values and local variables can be accessed very quickly, whereas array items and object members take longer. ... Local variables are faster to access than out-of-scope variables because they exist in the first variable object of the scope chain. The further into the scope chain a variable is, the longer it takes to access. Global variables are always the slowest to access because they are always last in the scope chain. ... Nested object members incur significant performance impact and should be minimized. ... The deeper into the prototype chain that a property or method exists, the slower it is to access."

Avoid the with statement

[HPJ, P.33] "Avoid the with statement because it augments the execution context scope chain. ... Also, be careful with the catch clause of a try-catch statement because it has the same effect." The net effect of each is that it makes the previously local variables no longer local, and hence, slower.

Optimize your loops because JavaScript won't

[EFW, P.93] "loops are a frequent source of performance issues in JavaScript, and the way you write loops drastically changes its execution time. Once again, JavaScript developers don't get to rely on compiler optimizations that make loops faster" Beyond the normal refactoring of code to be outside of a loop wherever possible, there are a couple of operations of the loop itself that must be hand optimized. The first version of the loop below runs much slower than the second because it is looking up the length property of values over and over, and the comparison of i-- to zero is much faster than comparing i to length.

    //traditional unoptimized for loop
    var values = [1,2,3,4,5];
    for (var i=0; i < values.length; ++i){ process(values[i]); }
 
    //over 50% faster version
    var length = values.length;
    for (var i=length; i--;){ process(values[i]); }

Use Cooperative Multitasking for Long-Running Scripts

[EFW, P.102] "One of the critical performance issues with JavaScript is that code execution freezes a web page. Because JavaScript is a single-threaded language, only one script can be run at a time per window or tab. This means that all user interaction is necessarily halted while JavaScript code is being executed." In general, when attempting to simulate true simultaneous execution of multiple processes (like JavaScript and the user interface), each process must either volunteer to regularly yield control temporarily to other processes (cooperative multitasking), or have control regularly taken from it (preemptive multitasking). The only option with JavaScript is voluntarily yielding control to the UI thread by splitting your logic into pieces and having each piece schedule the next piece with a time delay in between. "Generally speaking, no single continuous script execution should take longer than 100 milliseconds; anything longer than that and the web page will almost certainly appear to be running slowly to the user". The setTimeout function takes a function and a delay amount as parameters. "When the delay has passed, the code to execute is placed into a queue. The JavaScript engine uses this queue to determine what to do next. When a script finishes executing, the JavaScript engine yields to allow other browser tasks to catch up. The web page display is typically updated during this time in relation to changes made via the script. Once the display has been updated, the JavaScript engine checks for more scripts to run on the queue. If another script is waiting, it is executed and the process repeats; if there are no more scripts to execute, the JavaScript engine remains idle until another script appears in the queue."

Use Memoization to cache intermediate calculations

[JGP, P.44] One of the benefits of doing true "functional programming" is that it is easy to avoid performing calculations that have already been done. True functional programming has the rule that the result of a function depends totally on its parameters; i.e. it will always give the same result given the same parameters. This means that a cache object could be kept by a function to store the result for each set of parameters as they are encountered. When a function is called, the first thing it does it try to look up a cached result for the parameters it is given, and return it if found. If the result is not found, then it is calculated and saved in the cache before returning the result.

A simple example is the Fibonacci function. The first version below is the traditional implementation and the second uses a memoization helper to prevent recalculation. Since Fibonacci calls itself recursively, the savings add up very quickly.

// traditional implementation
function fibonacci(n){ return n<2 ? n : fibonacci(n-1) + fibonacci(n-2); }; 
 
// simple memoization helper for functions-with-a-single-numeric-parameter
var memoizer = function( mementoArray, lambda )
{ 
    var memoizedLambda = function( n ) { 
        var result = mementoArray[n]; 
        if (typeof result !== 'number')
            mementoArray[n] = result = lambda( memoizedLambda, n );
        return result;
    };
 
    return memoizedLambda; 
};
 
// memoized version [ with preloaded memos for f(0)=0 and f(1)=1 ]
var fibonacci = memoizer( [0,1], function(myself,n){ return myself(n-1) + myself(n-2); } );

Avoid CSS expressions

[HPW, P.51] "CSS expressions are a powerful (and dangerous) way to set CSS properties dynamically. They're supported in Internet Explorer version 5 and later... The expression method is simply ignored by other browsers, so it is a useful tool for setting properties in Internet Explorer to create a consistent experience across browsers. ... CSS expressions are re-evaluated when the page changes, such as when it is resized... The problem with expressions is that they are evaluated more frequently than most people expect. Not only are they evaluated whenever the page is rendered and resized, but also when the page is scrolled and even when the user moves the mouse over the page. ... CSS expressions benefit from being automatically tied to events in the browser, but that's also their downfall."

The major technique used to avoid CSS expressions is to have your own JavaScript function registered as an event listener triggered by only the appropriate event(s). It should set the dynamic CSS property to the desired recalculated value.

CSS Selectors are Backwards

[EFW, P.194] "The impact of CSS selectors on performance derives from the amount of time it takes the browser to match the selectors against the elements in the document. ... Consider the following rule: #toc > LI { font-weight: bold; } Most of us, especially those who read left to right, might assume that the browser matches this rule by moving from left to right, and thus, this rule doesn't seem too expensive. In our minds, we imagine the browser working like this: find the unique toc element and apply this styling to its immediate children who are LI elements. We know that there is only one toc element, and it has only a few LI children, so this CSS selector should be pretty efficient. In reality, CSS selectors are matched by moving from right to left! With this knowledge, our rule that at first seemed efficient is revealed to be fairly expensive. The browser must iterate over every LI element in the page and determine whether its parent is toc. This descendant selector example is even worse: #toc A { color: #444; } Instead of just checking for anchor elements inside toc, as would happen if it was read left to right, the browser has to check every anchor in the entire document. And instead of just checking each anchor's parent, the browser has to climb the document tree looking for an ancestor with the ID toc. If the anchor being evaluated isn't a descendant of toc, the browser has to walk the tree of ancestors until it reaches the document root."

Section 2: Loading Tips

"Nothing else can happen while JavaScript code is being executed" - N. Zakas
"Rule 1: Make Fewer HTTP Requests" - Steve Souders
"Only 10-20% of the end user response time is spent downloading the HTML document.
The other 80-90% is spent downloading all the components in the page." - the Performance Golden Rule

What is Fast Enough

[EFW, P.9] "It's fine to say that code needs to execute ''as fast as possible'', but ... exactly what is ''fast enough'' ... Jakob Nielsen is a well-known and well-regarded expert in the field of web usability; the following quote addresses the issue of ''fast enough'':

The response time guidelines for web-based applications are the same as for all other applications. These guidelines have been the same for 37 years now, so they are also not likely to change wit whatever implementation technology comes next.
0.1 second: Limit for users feeling that they are directly manipulating objects in the UI. For example, this is the limit from the time the user selects a column in a table until that column should highlight or otherwise give feedback that it's selected. Ideally, this would also be the response time for sorting the column-if so, users would feel that they are sorting the table.
1 second: Limit for users feeling that they are freely navigating the command space without having to unduly wait for the computer. A delay of 0.2-1.0 seconds does mean that users notice the delay and thus feel the computer is ''working'' on the command, as opposed to having the command be a direct effect of the users' actions. Example: If sorting a table according to the selected column can't be done in 0.1 seconds, it certainly has to be done in 1 second, or users will feel that the UI is sluggish and will lose the sense of ''flow'' in performing their task. For delays of more than 1 second, indicate to the user that the computer is working on the problem, for example by changing the shape of the cursor.
10 seconds: Limit for users keeping their attention on the task. Anything slower than 10 seconds needs a percent-done indicator as well as a clearly signposted way for the user to interrupt the operation. Assume that users will need to reorient themselves when they return to the UI after a delay of more than 10 seconds. Delays of longer than 10 seconds are only acceptable during natural breaks in the user's work, for example when switching tasks.

In other words, if your JavaScript code takes longer than 0.1 seconds to execute, your page won't have that slick, snappy feel; if it takes longer than 1 second, the application feels sluggish; longer than 10 seconds, and the user will be extremely frustrated. These are the definitive guidelines to use for defining ''fast enough.'' "

Put Stylesheets at the Top

[HPW, P.44] "In their effort to improve one of the most visited pages on the Web, the Yahoo portal team initially made it worse by moving the [CSS] stylesheet to the bottom of the page. They found the optimal solution by following the HTML specification and leaving it at the top. Neither of the alternatives, the blank white screen or flash of unstyled content, are worth the risk. ... If you have a stylesheet that's not required to render the page, with some extra effort you can load it dynamically after the document loads."

Put JavaScript at the Bottom

[HPJ, P.3] [HPW, P.45] "Put all <script> tags at the bottom of the page, just inside of the closing </body> tag. This ensures that the page can be almost completely rendered before script execution begins." "[While modern browsers] allow parallel downloads of JavaScript files...JavaScript downloads still block downloading of other resources, such as images. And even though downloading a script doesn't block other scripts from downloading, the page must still wait for the JavaScript code to be downloaded and executed before continuing." "Since each <script> tag blocks the page from continuing to render until it has fully downloaded and executed the JavaScript code, the perceived performance of this page will suffer. Keep in mind that browsers don't start rendering anything on the page until the opening <body> tag is encountered. Putting scripts at the top of the page typically leads to a noticeable delay, often in the form of a blank white page" "This is the Yahoo Exceptional Performance team's first rule about JavaScript: put scripts at the bottom."

Use External Scripts but Group them Together...

[HPJ, P.4] [HPW, P.55] "In Raw Terms, Inline [scripts are] Faster... [but] using external files in the real world generally produces faster pages. This is due to ... the opportunity for JavaScript and CSS files to be cached by the browser." "Group scripts together. The fewer <script> tags on the page, the faster the page can be loaded and become interactive. ... when dealing with external JavaScript files, each HTTP request brings with it additional performance overhead, so downloading one single 100 KB file will be faster than downloading four 25 KB files. It's helpful to limit the number of external script files that your page references. ... Typically, a large website or web application will have several required JavaScript files. You can minimize the performance impact by concatenating these files together into a single file and then calling that single file with a single <script> tag. The catenation can happen offline using a build tool or in real-time using a tool like the Yahoo combo handler."

...ON THE OTHER HAND, don't group them All together (aka: Download Most JavaScript in a Nonblocking Fashion)

[HPJ, P.5-9] [EFW, P.22, 51, 73] "It turns out that Facebook executes only 9% of the downloaded JavaScript functions by the time the onload event is called." "Limiting yourself to downloading a single large JavaScript file will only result in locking the browser out for a long period of time, despite it being just one HTTP request. To get around this situation, you need to incrementally add more JavaScript to the page in a way that doesn't block the browser. The secret to nonblocking scripts is to load the JavaScript source code after the page has finished loading...[i.e.] after the window's load event has been fired." "Dynamic script loading is the most frequently used pattern for nonblocking JavaScript downloads due to its cross-browser compatibility and ease of use." "A new <script> element can be created very easily using standard DOM methods:

    var script = document.createElement("script"); 
    script.type = "text/javascript"; 
    script.src = "file1.js"; 
    document.getElementsByTagName("head")[0].appendChild(script);

... [file1.js] begins downloading as soon as the element is added to the page. The important thing about this technique is that the file is downloaded and executed without blocking other page processes... It's generally safer to add new <script> nodes to the <head> element instead of the <body>, especially if this code is executing during page load. ... This works well when the script is self-executing but can be problematic if the code contains only interfaces to be used by other scripts on the page. In that case, you need to track when the code has been fully downloaded and is ready for use. This is accomplished using events that are fired by the dynamic <script> node. ... You can dynamically load as many JavaScript files as necessary on a page, but make sure you consider the order in which files must be loaded. ... [Many] browsers will download and execute the various code files in the order in which they are returned from the server. You can guarantee the order by [daisy] chaining the downloads ...[or] concatenate the files into a single file where each part is in the correct order. That single file can then be downloaded ... (since this is happening asynchronously, there's no penalty for having a larger file)." "The challenge in splitting your JavaScript code is to avoid undefined symbols." "Preserving the order of JavaScript is critical, and this is true for CSS as well."

Separate Bootstrap Code from the bulk of the JavaScript

[HPJ, P.10] [EFW, P.27] "The recommended approach to loading a significant amount of JavaScript onto a page is a two-step process: first, include the code necessary to dynamically load JavaScript, and then load the rest of the JavaScript code needed for page initialization. Since the first part of the code is as small as possible, potentially containing just the loadScript() function, it downloads and executes quickly, and so shouldn't cause much interference with the page. Once the initial code is in place, use it to load the remaining JavaScript [in a nonblocking fashion with a callback invoked on load completion]. For example:

    <script type="text/javascript" src="myScriptLoader.js"></script>
    <script type="text/javascript">
      myScriptLoader( "the-rest.js", function(){ MyApplication.init(); } );
    </script>

Place the loading code just before the closing </body> tag [so that] when the second JavaScript file has finished downloading, all of the DOM necessary for the application has been created and is ready to be interacted with, avoiding the need to check for another event (such as window.onload) to know when the page is ready for initialization." "The concept of a small initial amount of code on the page followed by downloading additional functionality is at the core of the YUI 3 design."

Prevent redirects due to missing trailing URL slashes

[HPW, P.76] "A redirect is used to reroute users from one URL to another. ...the main thing to remember is that redirects make your pages slower. ... One of the most wasteful redirects happens frequently and web developers are generally not aware of it. It occurs when a trailing slash (/) is missing from a URL that should otherwise have one. For example, http://yahoo.com/astrology results in a redirect to http://yahoo.com/astrology/ . The only difference is the addition of a trailing slash. ... Note that a redirect does not happen if the trailing slash is missing after the hostname. For example, http://www.yahoo.com does not generate a redirect."

Never put an inline script after a <link> tag

[HPJ, P.4][EFW, P.75] "Steve Souders has found that an inline script placed after a <link> tag referencing an external stylesheet caused the browser to block while waiting for the stylesheet to download. This is done to ensure that the inline script will have the most correct style information with which to work. Souders recommends never putting an inline script after a <link> tag for this reason."

Minify JavaScript

[HPW, P.69] "Minification is the practice of removing unnecessary characters from code to reduce its size, thereby improving load times. When code is minified, all comments are removed, as well as unneeded whitespace characters (space, newline, and tab). In the case of JavaScript, this improves response time performance because the size of the downloaded file is reduced. ... The most popular tool for minifying JavaScript code is JSMin (http://crockford.com/javascript/jsmin) ... Gzip compression has the biggest impact [about 70%], but minification further reduces file sizes [by about 20%]." "The savings from minifying CSS are typically less than the savings from minifying JavaScript because CSS generally has fewer comments and less whitespace than JavaScript. The greatest potential for size savings comes from optimizing CSS; i.e. merging identical classes, removing unused classes, etc."

Remove Duplicate Scripts

[HPW, P.85] "It hurts performance to include the same JavaScript file twice in one page. This mistake isn't as unusual as you might think. A review of the 10 top U.S. web sites shows that two of them (CNN and YouTube) contain a duplicated script. ... The two sites that have duplicate scripts also happen to have an above-average number of scripts (CNN has 11; YouTube has 7). ... One way to avoid accidentally including the same script twice is to implement a script management module in your templating system. ... While tackling the duplicate script issue, add functionality to handle script dependencies and versioning of scripts too."

Use a Content Delivery Network

[HPW, P.18] "The average user's bandwidth increases every year, but a user's proximity to your web server still has an impact on a page's response time. ... A content delivery network (CDN) is a collection of web servers distributed across multiple locations to deliver content to users more efficiently. ... CDNs are used to deliver static content, such as images, scripts, stylesheets, and Flash. Static files are easy to host and have few dependencies. That is why a CDN is easily leveraged to improve the response times for a geographically dispersed user population." Keeping in mind the Performance Golden Rule that says "Only 10-20% of the end user response time is spent downloading the HTML document. The other 80-90% is spent downloading all the components in the page", [Tests and Yahoo experience show that] components hosted on a CDN loaded 18-20% faster than with all components hosted from a single web server."

Use Server Compression

[HPW, P.39] "If an HTTP request results in a smaller response, the transfer time decreases because fewer packets must travel from the server to the client. ... [Using server-side preprocessing] to compress HTTP responses... is the easiest technique for reducing page weight and it also has the biggest impact. Gzip is currently the most popular and effective compression method. It is a free format (i.e., unencumbered by patents or other restrictions). ... Servers choose what to gzip based on file type, but are typically too limited in what they are configured to compress. Many web sites gzip their HTML documents, but it's also worthwhile to gzip your scripts and stylesheets, and in fact, any text response including XML and JSON. Image and PDF files should not be gzipped because they are already compressed. ... Generally, it's worth gzipping any file greater than 1 or 2K."

Add "Long Shelf Life" Headers

[HPW, P.32] "Browsers (and proxies) use a cache to reduce the number of HTTP requests and decrease the size of HTTP responses, thus making web pages load faster. A web server uses the Expires header to tell the web client that it can use the current copy of a component until the specified time. ... The Cache-Control header was introduced in HTTP/1.1 to overcome limitations with the Expires header. ... [it] uses the max-age directive to specify [in seconds] how long a component may be cached. ... If the browser has a copy of the component in its cache, but isn't sure whether it's still valid, a conditional GET request is made. ... The browser is essentially saying, ''I have a version of this resource with the following last-modified date. May I just use it?'' ... If the component has not been modified since the specified date, the server ... skips sending the body of the response, resulting in a smaller and faster response. Conditional GET helps pages load faster, but they still require making a roundtrip between the client and server to perform the validity check. ... Those conditional requests add up [easily adding over 50% to the response time for subsequent views of a typical web page.]" Configuring your web servers to generate headers that specify the predicted lifespan of each component will maximize cache effectiveness. Using versioning to make each component "immutable" will allow "far future Expires" (i.e. very long lifespans) to be specified thus avoiding unnecessary Conditional GETs.

Remove ETags

[HPW, P.89] "Entity tags (ETags) are a mechanism that web servers and browsers use to validate cached components. Reducing the number of HTTP requests needed for a page is the best way to accelerate the user experience. You can achieve this by maximizing the browser's ability to cache your components, but the [default] ETag header thwarts caching when a web site is hosted on more than one server. ... The problem with ETags is that they are typically constructed using attributes that make them unique to a specific server hosting a site. ... The end result is that ETags generated by Apache and IIS for the exact same component won't match from one server to another. ... If you have components that have to be [version-stamped] based on something other than the last-modified date, [customized] ETags are a powerful way of doing that. If you don't have the need to customize ETags, it is best to simply remove them. ... removing the ETag altogether would avoid these unnecessary and inefficient downloads of data that's already in the browser's cache."

Use Iframes Sparingly

[EFW, P.191] "Even blank iframes are expensive. They are one to two orders of magnitude more expensive than other DOM elements. When used in the typical way (<iframe src="url"></ifram>), iframes block the onload event. This prolongs the browser's busy indicators, resulting in a page that is perceived to be slower. ... Although iframes don't directly block resource downloads in the main page, there are ways that the main page can block the iframe's downloads. ... The browser's limited connections per server are shared across the main page and iframes, even though an iframe is an entirely independent document. ... With all of these costs, it's often best to avoid the use of iframes, and yet a quick survey shows that they are still used frequently. ... An alternative way to [use frames] with better performance would be for the main page to create a DIV to hold the contents of the [frame]. When the main page requests the [frame's] external script asynchronously, the ID of this DIV could be included in the script's URL. The [frame's] JavaScript would then insert the [frame] in the page by setting the innerHTML of the DIV. This approach is also more compatible with [frames] that take over a large part of the window and thus cannot be constrained by an iframe. The use of iframes is declining as these other techniques for inserting." ON THE OTHER HAND, iframes have been retained and even enhanced in HTML5.

Flush the Document Chunks Early

[EFW, P.191] While the backend server is generating its response to a web page request, the browser (and the user) must sit and wait. "In most cases, the browser waits for the HTML document to arrive before it starts rendering the page and downloading the page's resources." In order for the user to get some immediate feedback, and for the browser to start downloading images and the like, the backend server logic should flush an initial chunk of HTML back to the browser before starting any "slow" logic needed on its end to marshall up the remainder of the HTML. To do this, HTML 1.1 "chunked encoding" needs to be supported by both the browser and server, and the server logic needs to "flush" its output buffer back to the browser after an appropriate HTML split point has been reached. "This is exactly what's needed to combat the two shortcomings of a slow HTML document: blocked rendering and blocked downloads." ON THE OTHER HAND, early flushing and chunked encoding has many gotchas which can make it problematic in the real-world.

Reduce DNS Lookups...

[HPW, P.73] [EFW, P.171] "the Domain Name System (DNS) maps hostnames to IP addresses, just as phonebooks map people's names to their phone numbers. ... DNS has a cost. It typically takes 20-120 milliseconds for the browser to look up the IP address for a given hostname." Minimizing how many different domain names your page references reduces DNS lookup delays. "Google is the preeminent example of this, with only one DNS lookup necessary for the entire page."

...ON THE OTHER HAND, Shard your Domain

[EFW, P.161-8] Most browsers limit the number of resources that they will simultaneously download from any particular domain name. "Some web pages have all their HTTP requests served from one domain. Other sites spread their resources across multiple domains. ... sometimes increasing the number of domains is better for performance, even at the cost of adding more DNS lookups." Splitting resource downloads between multiple domain names (e.g. foo.com and www.foo.com) is known as Domain Sharding. Having a particular resource always associated with a particular domain name is needed to maximize caching. "Research published by Yahoo shows that increasing the number of domains from one to two improves performance, but increasing it above two has a negative effect on load times. The final answer depends on the number and size of resources, but sharding across two domains is a good rule of thumb."

Make Fewer and Thinner Image Requests

[HPW, P.10] [EFW, P.133] "Images are an easy place to improve performance without removing features. Often, we can make substantial improvements in the size and number of images with little to no reduction in quality. (1) If you use multiple hyperlinked images (say a row of buttons), [client-side] image maps may be a way to reduce the number of HTTP requests without changing the page's look and feel. An image map allows you to associate multiple URLs with a single image. The destination URL is chosen based on where the user clicks on the image. (2) Like image maps, CSS sprites allow you to combine images [onto one large compound image such that each individual image can be cropped via CSS] ... One surprising benefit is reduced download size. Most people would expect the combined image to be larger than the sum of the separate images because the combined image has additional area used for spacing. In fact, the combined image tends to be smaller than the sum of the separate images as a result of reducing the amount of image overhead (color tables, formatting information, etc.). (3) Optimizing images begins with creative decisions made by the designer about the minimum number of colors, resolution, or accuracy required for a given image [availing the use of lossy optimizations.] ... Once the quality choice has been made, use [automated] nonlossy compression to squeak the last bytes out of the image. ... fantastic open source tools exist for optimizing images. ● Start by choosing the appropriate format: JPEG for photos, GIF for animations, and PNG for everything else. Strive for PNG8 whenever possible. ● Crush PNGs, optimize GIF animations, and strip JPEG metadata from the images you own. Use progressive JPEG encoding for JPEGs more than 10 KB in file size. ● Avoid AlphaImageLoader. ● Optimize CSS sprites. ● Create modular sprites if your site has more than two to three pages. ● Don't scale images in HTML. ● Generated images should be crushed, too. Once generated, they should be cached for as long as possible. Convert images to PNG8 and determine whether 256 colors is acceptable."

JavaScript: The Surprising Parts

2011-01-31T23:15:00.000-05:00

This post is a summary of the the things that surprised me as I've been reading several books to update my knowledge of JavaScript (and practices that have changed since 2004/2005). While this info would have been big news to most and worth writing articles about back then, it begs the question as to why I should blog about it given that now there are several good books available on the subject. There are two self-canceling reasons; (1) this blog will act as a cheat sheet for me, and (2) the very act of writing it will make me more likely not to need a cheat sheet. While there are many small things that I didn't happen to know (or remember knowing), I am writing here about the things that actually surprised me, and they tended to be either big or small surprises, so I will organize them that way.

Small Surprises

[JGP, p.6] undefined is not a reserved word!
[JGP, p.107] undefined is not a constant; it is a global variable which can be redefined! (so is NaN).
[JGP, p.103] Reserved words can be used as object property names (unless accessed via dot notation)! Even stranger, an empty string is a legal property name! (as in obj[""]=123;)
[JGP, p.7] integer types are really double under the hood!
[JGP, p.10, 102] Nested block statements (i.e. {...}) do not define a new variable scope!
[JGP, p.29] If a method is called as a simple function, this is not undefined but rather defined as the global object!
[JGP, p.36, 102] Local variables exist before they are declared! (which is why they may as well all be declared at the beginning of a function)
[JGP, p.105] The arguments array is not an array! (For that matter, arrays are not arrays!)
[JGP, p.109] The == operator does type coercion and the === operator does not. But the type coercion is done so poorly that it is too dangerous to use == at all in general!
[JGP, p.102] The return statement will return undefined if the return value expression does not start on the same line as the return keyword!
[HPJ, p.2] Browsers block loading anything else while the JavaScript inside a script tag executes!
[HPJ, p.7] Scripts can be dynamically loaded by creating script tags via DHTML, and those scripts don't block loading other elements (like images, etc).
[HPJ, p.20] Global variables are not the fastest to access (compared to variables in other scopes), as would be the case in compiled languages, but are in fact the slowest since all the other scopes are searched first.
[JGP, p.112, HPJ, p.156] Bitwise operators are either really fast, or really slow, depending on who is judging. Crockford, in Appendix B: Bad Parts, say bitwise operators are "very slow", whereas, Zakas, in Use the Fast Parts, say bitwise operators are "incredibly fast"!

Large Surprises

[JGP, p.37, HPJ, pp.21-26] Closures imply Cactus Stacks. I didn't really understand in a deep way what closures were, and JGP didn't help (nor any other JavaScript articles/books for that matter). After all, Pascal had nested function definitions where the inner procedures had access to the outer local variables, but that didn't make closures. What really made me understand was realizing that JavaScript was like Scheme with regard to closures. (See JavaScript is Scheme not Java below). As I wrote in a comment to someone else's blog attempting to explain JavaScript closures...
I think it would be more accurate to say right up front that our normal notion of a “stack” (as used to hold parameters and local variables) does not apply in languages like JavaScript (and Scheme, etc). Unlike Pascal, Ada, C, C++, Java, etc., JavaScript has only “heap with garbage collection"; all those “stack frames” to hold parameters and local variables are just more objects on the heap. When no more references to each exist, they individually get garbage collected. This means that there is no special “saving the stack frame” after the outer function has returned for closures to use…they all just get garbage collected like anything else does when all references to it are deleted.
BTW, the only way I ever grokked this was after learning Scheme by watching the UC-Berkeley Comp Sci 61A class on iTunes U. Except for the EARRRRLY Fortran on CDC supercomputer days, where recursion was not supported and the hardware instruction set did not implement stacks, I had not experienced a language that wasn’t fundamentally stack-oriented (well ok, there was that funny Lisp language in the exotic languages class). The 61A class video explained (with nice pictures) how the “stack frames” are instead “environments” that chain to each other (just like JavaScript objects chain via Prototypes). There is a global environment with the global vars, plus a new environment for each function (to hold params and local vars), and instead of a stack of frames on a hardware stack, it is a linked list of environments on the heap.
It probably also requires explaining right up front that unlike the Pascal, Ada, C, Java, etc languages (where functions are defined at COMPILE-TIME), JavaScript, Scheme, etc languages define functions at RUN-TIME. Because of this, there is always an "environment" that is active at the time of function creation/definition, namely, the call-stack that exists while the code creating a function is executed. [EXCEPT, of course that there is no stack... There is a great way to visualize how the stack is really a tree at http://en.wikipedia.org/wiki/Spaghetti_stack].
A reference to this “current environment” chain is saved as a part of the function definition. In Java-like languages this makes no sense because there is no “current stack or environment” at compile time when the function was defined/compiled.
SO, when a function gets called, instead of “pushing” its params/local vars on top of THE stack, it “pushes” it on top of an environment chain that was saved with the function definition. And since it isn't a real (hardware) stack, but actually sitting in the heap, the function can refer to “local variables” of its “outer” function long after that outer function has returned.
[JGP, pp.40-57] JavaScript is Scheme not Java. Having accidentally watched the entire 61A course that teaches Scheme and functional programming (mentioned above) before ever reading JGP, I immediately recognized the exotic techniques of JGP chapter 4 (currying, etc) as paralleling the topics in 61A. [BTW, I originally watched 61A because I wanted to learn about this Hadoop buzzword I kept coming across in Job Ads. When I looked it up and saw that it had something to do with Map/Reduce (whatever that was), and that this iTunes class had a section on Map/Reduce, I tried to watch just those lessons. It soon became clear that I needed to start from the beginning of the course to learn Scheme itself.] Learning the culture and toolkit of functional programming, which goes way beyond simply using function pointers or having functions be objects, I could see that the new style of programming JavaScript was not very strange Java, but fairly normal Scheme. Back in 2005, I was aware of, but avoided, many of those alternate ways that JavaScript could be twisted to define classes, objects, functions, methods, inheritance, etc, BECAUSE, why would you want to make your JavaScript code look so un-Java-like!?! If one embraces JavaScript as Scheme, it is not a weird syntax but simply a different programming paradigm: functional programming with lambdas and lexical scoping.

One of the many differences between functional and non-functional programming is the cognitive "weight" of functions. In Fortran thru Java (and JavaScript as I had been using it), methods and functions are fairly heavy weight things that get full blown API documentation (e.g. JavaDoc/jsDoc), and lots of whitespace around them, and are mostly not defined in a nested scope. In functional programming, functions definitions are like water, and are created, used, and abandoned in as casual a fashion as Strings are in "normal" languages. Function definitions within function definitions in Scheme are as common as string concatenations in Basic. SO, one has to get over the feeling that it is expensive overkill to define a function just to wrap another function so that a one variable closure can be created.

[JGP] Crockford, Douglas. JavaScript: The Good Parts, O'Reilly, 2008
[HPJ] Zakas, Nicholas. High Performance JavaScript, O'Reilly, 2010

Adding Cross-browser support to the Gravey Framework

2011-01-08T17:01:00.001-05:00

In order to get back into the Web 2.0 development world, I have recently taken on the project of upgrading the Gravey framework to run on browsers other than Internet Explorer. This post is a chronicle of what I encountered/learned getting its sample applications to also run on iPad 4.2, Safari 5.0, Chrome 8.0, Firefox 3.6, SeaMonkey 2.0, and IE 8.0 (while not breaking IE6 on Win2K). [Note that the issues listed herein may have an IE-centric viewpoint, but it is only because the original code base naively used circa-2005 IE-only techniques, and therefore the news for Gravey is how everybody else does it now.]

The Gravey framework (gravey.org) is a JavaScript-only set of code that supports building Rich-Internet-Apps (RIA) using AJAX where the logic completely resides in the browser. The server-side need only provide a REST-style API for data persistence (which can be a simple as the file:: protocol accessing local xml files). The framework includes GUI widgets, automated domain object persistence, in-browser XML/XSL processing, and complete undo/redo capability. In short, it was intended to allow RIA development in the style of "fat GUI" development in Java without requiring any particular server side technology (e.g. J2EE, .Net, LAMP, whatever). The new version 2.5 of Gravey now runs on several browsers (including Safari on the iPad) without requiring any particular client-side technology (e.g. jQuery, Prototype, whatever).

Because it was back in 2005, when I was learning DHTML/OO-JavaScript/AJAX/CSS/REST on the fly, while building Gravey for use in several internal applications for a top-5-in-the-USA bank, and because their requirements only mandated (and their schedule only allowed) Internet Explorer 5.5 as the target browser, Gravey v1.0/v2.0 required IE5+, and were only tested on IE5/IE6 in Win2K/WinXP.

Since Gravey is built in layers, I had originally intended to merely retrofit the bottom layer to use jQuery to make short work of cross-browser compatibility. However, I immediately encountered enough problems with jQuery that I was required to really learn the issues anyway, and so I ultimately fixed all of the problems in Gravey directly, and as a result, it still has no dependence on any other frameworks.

••• AJAX / XML / XSL Issues •••

Issue: jQuery doesn't cover in-browser XSL processing
Gravey and its example apps use in-browser processing of the XML returned from AJAX calls via XSL (also loaded from the server). It turns out that the XSL processing API is browser-specific, and because jQuery didn't handle it, I was going to have to learn how to fix it directly anyway...

Issue: IE doesn't do XSL processing like everyone else
In order to support older IE browsers, the existing Gravey code that uses ActiveX is required, but other browsers need to use the standard DOM APIs.

IE really wants XSL files to be loaded via Msxml2.DOMDocument even though XML files can be loaded via Msxml2.XMLHTTP. So, once the XML DOM has been loaded via AJAX, the IE XSL processing looks like...

    var xmlDom = XHR.responseXML;

    var xslDom = new ActiveXObject("Msxml2.DOMDocument");
        xslDom.async = false;
        xslDom.load( xslURL );
    if (xslDom.parseError.errorCode != 0)
        alert("Error loading["+xslURL+"]:" + xmlDom.parseError.reason);

    var resultStr = xmlDom.transformNode( xslDom );

The other browsers accomplish this via the standard API...

    var xmlDom  = XHR1.responseXML;
    var xslDom  = XHR2.responseXML;

    var xslProc = new XSLTProcessor();
        xslProc.importStylesheet( xslDOM );
    var resultStr = xslProc.transformToDocument( xmlDOM ).documentElement.textContent;

BTW, when I used an XMLSerializer to generate the result string in the above code, it gave me quotes around the result, so using the .textContent was simpler and did not include extraneous quotes.
IE also requires the XML file to be loaded via Msxml2.DOMDocument when the files are local. In other words, when the .html page is loaded from local disk via the "file::" protocol instead of via "http::" from a web server, IE fails attempting to load the xml unless it is via Msxml2.DOMDocument. Otherwise, for local files, the AJAX callback will be given readyState==4 and status==0, instead of status==200. At this point in Firefox/others, the file has loaded ok, but in IE it will have failed.

Issue: Some browsers can't handle <xsl:import>
While good ole IE6 can handle an XSL file including another one via the <xsl:import> directive, new browsers like Safari and Chrome can't. I never did find a reasonable fix, so I bit the bullet and just copied the imported XSL directly into the top level XSL file. Not pretty but it works. If I had a larger system to worry about, I would preprocess, on the server side, the XSL file to include the imports before sending it back to the browser's AJAX request.

Issue: Firefox doesn't call your AJAX callback on synchronous requests
Parts of Gravey performed AJAX requests with async=false with no problem until Firefox came along. It turns out FF doesn't call your callback function like everyone else on synchronous requests. For a fix, I simply changed to async=true because I was already processing the responses via callback instead of inline code anyway! [ Back in 2005, I had thought that synchronous requests might be faster, or higher priority, than async requests, but experience has taught me otherwise.]

••• DOM Issues •••

Issue: jQuery doesn't like funny characters in your element IDs.
This problem I ran into immediately because Gravey uses all sorts of special characters (e.g. ".", "#", "_", ":", etc), and it took a while to track down a fix. By that time, I had taken to fixing Gravey directly. But there is a work-around...
Given that ID="tar.foo.bar", replace the following...
$(ID).whatever...
with...
$( jq(ID) ).whatever...
where...

    function jq(ID){ return '#' + ID.replace(/(:|\.)/g,'\\$1'); }

Issue: Only IE looks up Names as well as IDs in getElementById()
Gravey originally created HMTL elements identified by the "name" attribute because it worked just fine in IE when doing document.getElementById("foo"). Other browsers don't, SO, I changed all those .name references to .id references.

Issue: innerHTML doesn't equal what it was just set to
Early on, I came across speculation that browser security might strip away inline JavaScript when setting the .innerHTML of some element. Being paranoid, I put in code to check if the .innerHTML I just set, was different than what I just set it to. To my surprise, it was quite often literally different, but not functionally different. In other words, the before and after strings did not match because the quotes might have been changed, the attributes might be in different orders, and the upper/lowercase of the tags might have been switched. I never caught browser security stripping off anything, but the .toString() of the innerHTML could be completely reworked compared to my original HTML string.

Issue: Saving data between page loads in window or navigator properties is no longer reliable
In the olden days of IE6 on Win2K, one could create properties on the navigator object that would allow communications between page loads. With modern browsers this isn't reliable. Gravey used this ability to save a reference to data between the time an AJAX request was made and when the callback was later invoked on the reply event. Luckily, I was only doing this because I didn't understand JavaScript "closures" enough to use one instead. The Gravey 2.5 code uses a closure to save data in the callback function to be accessed when that function is later invoked on the reply event.

If I really needed client-side storage of data in between page loads or between pages, I would hope to wait until the near future when HTML5 is supposed to have local storage to share fat data between pages. There is an article here that tests this new feature out on a the usual suspect browsers.

Issue: Some browsers change the placement of span and legend tags
In tracking down all the event handling problems, I came across a bug in some browsers (e.g. Firefox/Safari/SeaMonkey, but not Chrome/IE) where I created a <legend> tag inside of a <span> tag (via setting innerHTML), but in Firefox (via Firebug) I could see that it put the legend outside and after the span! Luckily, I could live with the <span> being inside of the <legend>, and all the browsers seem to handle that ok.

Issue: IE8 changes behavior depending on where a page is served up from
I was trying to use a simple trick to determine if the browser is IE (and which version of IE) via conditional compilation as shown here at quirksmode.org. Unfortunately, IE8 on WinXP would compile the comments and define the flags when the page was loaded from my test server on my LAN. The exact same code would NOT compile when served up from my Internet web site. Go figure. I tried adding a DOCTYPE to invoke quirks mode (which I had not specified before), but that made no difference. I wound up using the code below to distinguish between the cases that I needed to determine: pre-IE7, IE-in-general, IE-style-AJAX, Firefox-style-AJAX.
In the home web page...
    
In the javascript...
    function IE_Pre7(){ return (typeof(Pre7IE)=="undefined") ? false : Pre7IE; }
    function FF_AJAX(){ return document.implementation
                    && document.implementation.createDocument; }
    function IE_AJAX(){ return IE_Pre7() || !FF_AJAX(); }
    function Is_IE (){ return '\v'=='v'; }

••• Display and Styling Issues •••

Issue: Handling of partial escape sequences is browser-specific
A bug in Gravey went undetected because IE quietly ignored partial escape sequences like "&nb". The bug generated partial escape sequences in element text (e.g. <div>foo&nb</div>) and browsers like Safari and IE quietly display "foo". Other browsers like Firefox and Chrome display "foo&nb". I fixed the bug so that only full sequences like " " would be generated.

Issue: Special character escape sequences are browser and OS-specific
Originally, Gravey used special characters via the Microsoft "symbol" font, so I had a goal to replace them with standard HTML characters. E.G. replace...
    '<font face="Symbol">e</font>'
with...
    "ε"

The set of graphic characters like arrows, and bullets, and greek letters, etc can not be all rendered by all browsers. Unfortunately, the intuitive idea that if an older browser can render a character, its newer brethren would also, is alas not true. Most browsers, including IE6 on Win2000, can render the black star (☆) and white star (★) characters, but IE8 on WinXP can't! Some characters can be displayed when given the hex charcode, but not the name (e.g. ⇒). And some characters that you would think are part of a set are not all implemented together. E.G. Of the up/down/right/left arrow set (see shapes), IE6 can render the right arrow but not the down arrow.

Issue: Fieldset Legend rendering is browser-specific
Given the same style settings, the Legends of Fieldsets render differently on different browsers. Gravey had used the attribute "text-align:center" which didn't do anything on IE, but it shifts the entire legend display to the center (instead of the left that I want) on non-IE browsers. I removed the attribute. Unfortunately, the "vertical-align: middle" attribute DOES do something I want in IE but not the other browsers. It was needed to get the gif images (being used for expanded/collapsed icons) to line up with the title text inside the legend box. After a lot of fiddling with transparent pixels around the icons and legend padding attributes, I gave up and switched to graphic arrow characters (ala Google results). Unfortunately, I ran into all of the special character problems listed earlier, and still had to specify a different set of characters (wingdings) for IE6 than specified for the other browsers.

Issue: CSS doesn't easily replace all HTML presentational attributes
While the original Gravey used some external CSS, there was still a lot of inline style specs, and old-time HTML presentational markup and "spacer" gifs. I set out to replace them all with CSS, but since I was using IE specific attributes like cellspacing, cellpadding, bordercolordark and bordercolorlight it was not trivial. Sadly, some IE attributes like table "cellspacing" can't be replaced with CSS. Others like cellpadding, bordercolordark and bordercolorlight can but with great effort. Luckily, Gravey used a cellspacing and cellpadding of zero, some of which can be specified with "border-collapse:collapse". Also, the bordercolorlight/dark colors being used created an effect similar to { border-style: outset }.

Issue: Background color gradients are browser-specific
The gradient coloring Gravey used was IE specific. Other browser specific style settings had to be added. E.G. the following...
    body {

      filter:progid:DXImageTransform.Microsoft.Gradient
        (GradientType=0,StartColorStr='#f8f8f8',EndColorStr='#802222');
    }

gets replaced with...
    body {
      background: -webkit-gradient(linear, left top, left bottom, from(#f8f8f8), to(#802222));
      background: -moz-linear-gradient(top, #f8f8f8, #802222) no-repeat;
      filter:progid:DXImageTransform.Microsoft.Gradient
        (GradientType=0,StartColorStr='#f8f8f8',EndColorStr='#802222');
    }

Issue: Tooltip rendering is browser-specific
Tooltips are used extensively by Gravey, and for images it used the .alt text. It turns out that non-IE browsers don't render image .alt text, only .title text, so I changed all the alts to titles.

There is also a known bug in Firefox where tooltips are not displayed on disabled buttons. As a workaround, the "readOnly" attribute can be used, however, when setting the readonly attribute rather than the disabled attribute, I could never get the CSS styling to kick in, even though I tried the selectors ".fooClass[readonly]" and "input[readonly]" and "input:read-only". So, I wound up dynamically setting (via JavaScript) the style for both readonly/not-readonly explicitly via classnames (ala .foo and .fooRO).

Issue: Cursor styles are browser-specific
Gravey extensively used the "hand" and "not-allowed" cursor shapes which turn out to be IE-specific. The "hand" cursor can be changed to the standard "pointer" to get the same effect. But while many browsers implement the non-standard "not-allowed", Firefox on the Mac does not and has no real equivalent. Because there is a visual difference between the default cursor and the "pointer" on Firefox, I live with it as is.

Issue: Disabled button styles are browser-specific
Gravey expects the rendering of GUI widgets to reflect their enabled/disabled status. Some browsers stop showing disabled buttons differently if their foreground color is set to black (as Gravey was doing and IE not caring). I was able to simply stop setting the foreground color since black was the default color anyway).

Issue: Chrome displays popup menu items in a strange order
There is a strange (but known?) bug in Chrome where it displays items in a pop-up menu in a weird order that is different than the order shown in all the other browsers. I give up and document it as a known Chrome bug.

Issue: Cross-browser manipulation of element visibility is complicated
Gravey originally got away with making chunks of HTML visible or not via...

    e.style.visibility = visibleFlag ? "visible"  : "hidden";
    e.style.position   = visibleFlag ? "relative" : "absolute";
    e.style.display    = visibleFlag ? "inline"   : "none";

However, to do this with other browsers the original visible state needs to be saved and reused later when restoring visibility. Code like the following is needed...

    if (visibleFlag) showElem(e); else hideElem(e);

    function hideElem( e )
    {
      if ( e.style.display === "none" ) return;
      e.oldDisplay = e.style.display;
      e.style.display = "none";
      e.style.visibility = "hidden";
    }
    function showElem( e )
    {
      e.style.visibility = "visible";
      if ( e.oldDisplay ) e.style.display = e.oldDisplay;
      else if ( e.style.display==="none") e.style.display = "";
    }

Also, some browsers have a problem unless the HTML is rendered visible before making it hidden.

Issue: Pop-up windows are no longer reliable
In the early IE5 days, the cursor would not reliably change to the "please wait" icon when told to, so as a workaround, Gravey used a little "please wait" popup window that was launched and later killed. Unfortunately, modern browsers suppress popup windows, so, that code won't reliably work anymore. If the cursor STILL can not be reliably set in this modern age, then a replacement "please wait" popup window would have to be done via fancy CSS floating DIVs as I've seen elsewhere.

••• Event Handling Issues •••

Issue: Event object location is browser-specific

In IE, the event object may be accessed in event handler functions by the global variable "event". Most other browsers pass the event object as a parameter to the event handler function and it is not available as a global variable.

Thus, code like the following...
<body onKeyPress="onKey()">
...
function onKey(){ alert(event...); }

must be changed to...

    <script> 

      document.onkeypress = onKey;

      function onKey(e){ e = e || window.event; alert(e...); }

    </script> 

Another consequence of event handlers being passed the event as a parameter is that the calling signature has to be changed when passing your own parameters to the event handler. Even if you don't care about the event arg itself, the other parameters need to be shifted down otherwise the formal parameter list won't match the actual parameter list. For example, Gravey generates new HTML elements by building a string and setting the innerHTML.
So, change code like this...
    e.innerHTML = "<select onchange='return foo("+myArg+")'>";
   ...
   function foo( myArg ){ alert(myArg); }
to this...
    e.innerHTML = "<select onchange='return foo(event,"+myArg+")'>";
   ...
   function foo( e, myArg ){ alert(myArg); }

Issue: Not all body tag event handlers are attached to the "window" object

When moving the specification of event handlers from inline in the <body> tag to being set in JavaScript, one might think that they all go on the window event, but not so. Thus, this tag...

<body onKeyPress="onKey()" onBeforeUnload="onBUL()" onLoad="onLoad()">

gets replaced with...

    <script>
        document.onkeypress   = onKey;

        window.onbeforeunload = onBUL;
        window.onload         = onLoad;

</script>

Issue: Keyboard event objects are browser-specific

In keyboard-related events, some browsers supply the key pressed via the event's "keyCode" attribute, whereas others via the "which" attribute.

Thus code that reads keys...
function onKeyPress(){ alert( event.keyCode ); }
must be changed to...
function onKeyPress( e ){
    e = e || window.event;
    var code = e.keyCode || e.which;
    alert( code );
}
Thus code that sets keys (like keystroke filtering)...
event.keyCode = char;
must be changed to...
e = e || window.event;
if (e.keyCode) e.keyCode = char; else e.which = char;
Secondarily, some browsers supply a code that directly reflects the use of control keys, etc, whereas others supply a code for the basic key and require also looking at other attributes that flag the use of any modifier keys. Thus, to detect a control-z key, the following...
function onKeyPress(){ if ( event.keyCode==26 ) ... }
must be changed to...
function onKeyPress( e ){
    e = e || window.event;
    var code = e.keyCode || e.which;
    if ( code==26 || code==122 && e.ctrlKey ) ...
}

Even then, Chrome on the Mac will not generate an event at all when control-y is pressed, even though control-z works fine. So, after not finding any info about this Chrome quirk, I gave up and documented the use of shift-control-y as a workaround.

Issue: Modifier keys on mouse click are browser-specific
Gravey and its example apps use "control-clicks" (ie. holding down the control key while clicking) in several places. Unfortunately, on the Mac, control-click activates the context menu (ala back-click). So, for it, allow the "command" key to be used as an alternate. Unfortunately, the DOM3 standard indicator for this key is the event.metaKey flag and Internet Exploder does not even implement that attribute, so references to it will be undefined. So, this code...

function onClick(){
var specialFlag = event.ctrlKey;
}

has to be changed to...

function onClick(e){
    e = e || window.event;
    var cmdKey = (typeof e.metaKey=="undefined") ? false : e.metaKey;
    var specialFlag = e.ctrlKey || cmdKey;
}

Issue: Some "are you sure you want to leave?" techniques are browser-specific
In order to ask if the user really wants to leave the web page, only some browsers recognize setting the event object's "returnValue" attribute to the desired message. However all browsers tested (except for a known bug on the iPad) will recognize returning the string message from the onbeforeunload event handler. Thus, code like this...

    function onBeforeUnLoad(){
      if (unsavedData) event.returnValue = "Are you sure?";
    }

gets replaced with code like this...

    function onBeforeUnLoad(){

      if (unsavedData) return "Are you sure?";

}

Issue: Getting a reference to the event's target element is browser-specific

For example, to blur the element that is the target of a keyboard event, the original code...

    event.srcElement.blur();

has to be changed to (according to here in quirksmode.org)...

    var targ = e.target || e.srcElement;
    if (targ.nodeType == 3) // defeat Safari bug
        targ = targ.parentNode;
    targ.blur();

Issue: Focus and Blur and Change events are browser-specific
There is a known problem: "Safari and Chrome do not fire focus/blur events when the user uses the mouse to access checkboxes, radios or buttons. Text fields, textareas and select boxes work correctly."
Hmmm...this is difficult because the entire architecture of Gravey is based on blur events triggering datamodel updates. Lets see if I can fake it for the offending browsers with the change event (but only the offending ones since onchange is apparently buggy on IE). Actually because of what I am doing on blur is idempotent, I can just make both onchange and onblur invoke the same event handler (and also because of idempotency I dont have to worry about the unpredictable ordering of blur and change events in different browsers).

Neural Nets, Vagueness, and Mob Behavior

2010-08-28T16:52:00.004-04:00

In response to the following question on a philosophy discussion board, I replied with the short essay below and reproduce it here.

"It was then that it became apparent to me that these dilemmas – and indeed, many others – are manifestations of a more general problem that affects certain kinds of decision-making. They are all instances of the so-called ‘Sorites’ problem, or ‘the problem of the heap’. The problem is this: if you have a heap of pebbles, and you start removing pebbles one at a time, exactly at what point does the heap cease to be a heap?"

VAGUE CONCEPTS
This leads to the entire philosophy of "vagueness". i.e. are there yes/no questions that don't have a yes/no answer? Are some things like baldness vague in essence, or, is our knowledge merely incomplete? e.g. we don't know the exact number of hairs on your head, and/or, we don't know/agree on the exact number of hairs that constitutes the "bald" / "not bald" boundary?

NEURAL NETS
My personal conclusion is that there ARE many vague concepts that we have created that are tied to the way our brains learn patterns (and, as a side effect, how we put things into categories). In contrast to rational thought (i.e. being able to demonstrate logically step by step our conclusions), we "perceive" (ala Locke/Hume/Kant) many things without being able to really explain how we did it.

In Artificial Intelligence, there are "neural network" computer programs that simulate this brain-neuron style of learning. They are the programs that learn how to recognize all different variations of a hand-written letter "A" for example. They do not accumulate a list of shapes that are definitely (or are definitely not) an "A", but rather develop a "feel" for "A"-ness with very vague boundaries. They (like our brains) grade a letter as being more or less A-like. It turns out that this technique works much better than attempting to make rational true/false rules to decide. This is the situation that motivates "fuzzy logic" where instead of just true or false answers (encoded as 1 or 0), one can have any number in-between, e.g. 0.38742 (i.e. 38.7% likely to be true).

WISDOM OF THE CROWD?
Because each person has their own individually-trained "neural net" for a particular perception (e.g. baldness, redness, how many beans are in that jar?), we each come up with a different answer when asked about it. However, the answers do cluster (in a bell-curve-like fashion) around the correct answer for things like "how many beans". This is what led Galton to originally think that there was "wisdom in the crowd". This idea has been hailed as one of the inspirations for the new World Wide Web (aka Web 2.0). The old idea was that McDonalds should ask you if "you want fries with that?" to spur sales. The new Web 2.0 idea is that Amazon should ask you if you want this OTHER book based on what other people bought when they bought the book you are about to buy. I.E. the crowd of Amazon customers know what to ask you better than Amazon itself.

The problem is that there are many failures of "crowd wisdom" (as mentioned in that Wikipedia page in the link above). My conclusion is that most people advocating crowd wisdom have not realized that it is limited to "perceptions". Many Web 2.0 sites are asking the crowd instead about rational judgments, expecting them to come up with a better answer than individuals. The idea of democracy (i.e. giving you the right to vote) has been confused with voting guaranteeing the best answer, no matter the question. In fact, Kierkegaard wrote "Against The Crowd" almost 200 years ago where he recognized that individuals act like witnesses to an event, whereas people speaking to (or as a part of) a crowd, speak what we would now call "bullshit" because they are self-consciously part of a crowd. We can see this in the different results of an election primary (a collection of individuals in private voting booths) versus Caucuses where people vote in front of each other. So, Web 2.0 sites (Facebook, MySpace, blog Tag Clouds, etc) that allow people to see the effect on other people of what they are saying, are chronicling mob mentality rather than collecting reliable witness reports.

BTW, I have written several blog posts related to vagueness, for example:
http://existentialprogramming.blogspot.com/2010/03/model-entities-not-just-their-parts.html

Fuzzy Unit Testing, Performance Unit Testing

2010-06-09T15:41:00.000-04:00

In reading Philosophy 101, about Truth with a capital "T", and the non-traditional logics that use new notions of truth, we of course arrive at Fuzzy Logic with its departure from simple binary true/false values, and embrace of an arbitrarily wide range of values in between.

Contemplating this gave me a small AHA moment: Unit Testing is an area where there is an implicit assumption that "Test Passes" has either a true or false value. How about Fuzzy Unit Testing where there is some numeric value in the 0...1 range which reports a degree of pass/fail-ness? i.e. a percentage pass/fail for each test. For example, testing algorithms that predict something could be given a percentage pass/fail based on how well the prediction matched the actual value. Stock market predictions, bank customer credit default prediction, etc come to mind. This sort of testing of predictions about future defaults (i.e. credit grades) is just the sort of thing that the BASEL II accords are forcing banks to start doing.

Another great idea (if I do say so myself) that I had a few years ago was the notion that there is extra meta-data that could/should be gathered as a part of running unit test suites; specifically, the performance characteristics of each test run. The fact that a test still passes, but is 10 times slower than the previous test run, is a very important piece of information that we don't usually get. Archiving and reporting on this meta-data about each test run can give very interesting metrics on how the code changes are improving/degrading performance on various application features/behavior over time. I can now see that this comparative performance data would be a form of fuzzy testing.

<br>

Too cool for school (or, history anyway)

2010-04-08T11:40:00.001-04:00

There has been a flurry of articles and blog posts in reaction to Oracle's Jeet Kaul who, at EclipseCon 2010, said "We need to get the younger generation interested and excited [about Java] just like I was" and "I would like to see people with piercings doing Java programming". There have been some exchanges over "cool" languages and whether "cool" is a good thing. I contributed the following 2-cents on a JavaWorld discussion...

"Cool" led to most computer security problems in the world today

I have been around long enough to have seen several generations of "cool" languages overtaking established ones. Unfortunately, the new language was sometimes a big step backward.

A case in point was when C swept the world, replacing the line of languages like Pascal and the stillborn Ada. Because Unix was starting to get out of the lab and into microcomputers, C gained visibility.

BIG PROBLEM: Pascal (and others) could tell how big an array was, and hence could stop execution when someone attempted to write past the end. C was unable to do this (and because of the language design there was no way to "fix it" in the compiler). This is the security hole that is at the root of all computer virus exploits.

If developers chose new languages (or systems) based on technical merit rather than "cool", we would be decades ahead of where we are today.

As a side note/rant: I have always thought that "cool" was effected by political zeitgeist. It seems more than coincidence that European/Govt-regulated/strong-typing languages (e.g. Pascal/Ada) were replaced by get-the-compiler-regulation-off-the-programmers-back ethic of C at the same time Ronald Reagan was selling everyone on the same notion about government. Let the programmer do dangerous things (like convert an array into a pointer) because he knows better...(sounds like let the bankers do whatever they want because they know better than the regulators who after all only learn from past mistakes...wimps!)

Moore's Paradox. I'm just saying!

2010-03-30T17:45:00.001-04:00

The popular phrase "I'm just saying" has been around long enough for most people to have heard it, but not long enough for it to be well-documented as to where it originated. I heard a great stand up comic bit about it in the 1980's by Paul Reiser. There are several blog sites that muse over its origin and solicit theories:

It turns out that the most common definition of the phrase exhibits a logical paradox from Philosophy. The book "this sentence is false" is a collection of philosophical paradoxes, and it describes Moore's Paradox (as developed by G.E. Moore). I summarize it as follows:

Normally, everything that can be said about the world can be said by anyone. I can say the moon is made of green cheese, and you can say it. The state of the world described by me can equally be described by you with no logical paradox...EXCEPT... I can say that the moon is made of green cheese, and I can say that you do not believe that the moon is made of green cheese, but YOU can not say the same thing. I.E. you can not say that X is true and at the same time say that you do not believe that X is true. Note that you are not saying that you could be wrong in your belief, you are be saying that you believe both that X is, and is not, true at the same time. A logical contradiction.

However, whenever you use the phrase "I'm just saying!", you are in effect performing Moore's paradox.

Where Am I?

2008-12-18T14:21:00.009-05:00

In I am a Strange Loop[1], Doug Hofstadter ponders where one's "self" is located while being mentally absorbed by a situation that is located in a different place than one's body is currently residing. A simple example being that of reading Jane Austin while sitting in a chair. Another example being remote-controlling a robot on the moon. He asks the question "Where Am *I*" (where "I" is his shorthand for soul/self/consciousness).

It reminded me of my very first days exploring the World Wide Web in 1994. I explained to my family, as I gave them a guided tour of my new toy "Netscape", that we could "visit" places all around the world! Look, here we go to the South Pole[2] or Australia[3]! Because in those early days, the web server and the content were actually physically in those places, and because the browser was hardly more than a remote terminal program, it really was like remote logging in to computers around the world, which felt very much like being there. That made switching from one site to another feel like teleporting instantly from one continent to another.

These days, the content about a place, versus the server serving the content, versus the location of the many cached copies (e.g. Akamai), and so forth have blurred "where am I" so much as to be meaningless and not even contemplated anymore. But in the early days, there was a real sense of "I am in Antarctica now!".

[1] "I am a Strange Loop",2007, Hofstadter, Basic Books

http://www.kinderwijs.nl/artikelen.asp?postid=188

[2] http://www.usap.gov/videoClipsAndMaps/spwebcam.cfm

[3] http://www.radioaustralia.net.au/

Shakespeare is (not) Shakespeare

2008-01-05T11:21:00.001-05:00

[Ed. Note. This is part two of a series found on "Existential Programming, the blog": "A Rose is a Rose is (not) a Rose"]

In the early part of the book The Stuff of Thought by Steven Pinker, the problem of what-a-name-names, is explored with the example of Shakespeare. Pinker distinguishes between Shakespeare: the historical figure, and Shakespeare: the author of numerous plays like Hamlet attributed to Shakespeare.

In my earlier post, it was somewhat easy to see that there were multiple aspects to Superman because each aspect already had its own name; Superman vs Clark Kent. With Shakespeare however, it is much more subtle because the different aspects have the same name: Shakespeare. Additionally, we are not used to thinking that they are different aspects that can be independent of each other, any more than we think of Cher-the-person and Cher-the-singer as being independent things. But, as discussed in the book, many people over the centuries have debated whether the author of Hamlet, et al was really Francis Bacon, Christopher Marlowe, Queen Elizabeth, etc.

The interesting thing is that because Shakespeare is SO ingrained as the name of the playwright that even if Sir Francis were to be proven the author, the headline will be "Bacon is the REAL Shakespeare!" which is absurd because clearly, Shakespeare-the-historical-figure is the "real" Shakespeare. Changing the human associated with the author-of-Hamlet concept will not change the concept's name; it will remain "Shakespeare's Hamlet (written by Bacon)" and not "Bacon's Hamlet".

So, when assigning ID#(s) to putatively single entities, flexibility should be built in to allow ad-hoc collections of attributes of any entity to be grouped and named and referenced separately. Otherwise, the system would not be able to represent the statement: Shakespeare is not Shakespeare, Bacon is.

Clark Kent is (not) Superman

2007-12-22T13:45:00.002-05:00

[Ed. Note. This is part one of a series found on "Existential Programming, the blog": "A Rose is a Rose is (not) a Rose"]

It delights me to find out that what I thought had been a particular nugget of wisdom, specific to building Identity matching computer systems, actually has a deep principle at work. While working on one of these systems, I learned the strategy of NOT merging all variations of an individual identity's name/address/phone/etc into a single canonical version. It turns out that the need to keep, and assign a unique key to, every variation of identity data (as opposed to only the "canonical" one) has deep roots in language itself...

While reading the Intellectual Devotional (which I highly recommend), I came across its page about "Philosophy of Language" and it had an immediate resonance with a project at my current client. The page describes the "problem of reference" where ideas about what a name "means" have been debated and changed over time.

One theory says that "names" don't have any meaning, in and of themselves, they merely refer to some thing that has meaning. Hence, Shakespeare's quote "A rose by any other name would smell as sweet" summarizes the position that the word "rose" is not meaningful, and could be exchanged with any other word that refers to the thing "rose". That is why "gulaab" (the Urdu word for rose) can work just as well for speakers of Urdu.

Another more modern theory though, says that names not only refer to some thing, they also carry the connotation of "in what sense" is the thing being referenced. The book illustrates the example of Superman and Clark Kent both being names for the same thing (the being Superman), but they are not interchangeable. Clark Kent (mild mannered reporter) has a work address of the Daily Planet whereas Superman (superhero able to leap tall buildings) does not. It matters which name is used when talking about Superman.

So, in the same way that Clark Kent and Superman both refer to different aspects of the same entity, and are thus not interchangeable, a computer system managing legal entity identity data can not translate name/address variations into a single entity ID# when those variations actually refer to different aspects of the entity. For example, if there is data that is specific to a particular store branch, that branch needs its own well-known ID# even though it is only a portion of a single legal entity. Further, since legal entity names are not unique (I own two different corporations with the identical legal name), the entire name/address/phone/etc combination needs managing rather than separate "alternate name" lists. It is also not sufficient to support alternate name/address records merely as search aids that still ultimately result in the ID# of the entity-as-a-whole. Otherwise one would loose track of the fact that we were talking about Clark Kent, not Superman.

Odometer Game Redux

2007-11-30T19:44:00.000-05:00

Well, after 35 years of pondering what I thought was an abstract mathematical puzzle, my "odometer game" has found a real-world application!

It turns out that my notion of "remarkable" numbers [i.e. numbers that are so remarkable that if the driver saw his odometer sitting on that number he would either honk his horn or point it out to his passengers] are just the ticket for finding "fake" ID numbers.

My current contract at a major bank found me looking for suspect ID numbers, Tax IDs, phone numbers, etc. in various customer databases. The bank employees entering this information would often get around the fact that these fields were "required" via entry of syntactically legal digit strings that were none the less meaningless. After viewing a few of these it quickly became obvious that they were related to my notion of remarkableness. Actual values found included: 0, 121212121, 000000000, 999999999(9), 111111111, 111111112, 222222221, 888888889, 188888888, 0999999999, 589999999, 255511555 (?)

So, rather than an explicit list of IDs to put on a watch list (as I was asked to find), it became clear that a better answer would have been to use an evaluation function that reported the remarkableness score for each value. A cutoff point could then be established to filter out suspicious values. Alas, while I have casually pondered the mathematics involved in scoring the remarkableness of a number, I've never actually tried to program it. But, now it has become more than an obscure puzzle, and shows signs of having "real world" value!

The Odometer Game

2007-10-13T11:51:00.000-04:00

Dear Dr. Douglas Hofstadter,
Having been a fan of yours since GEB (I had you sign my copy in '83 at UC Santa Cruz), I have always wanted to write to you about an "odometer game" I concocted about 1973 which touches upon several of your favorite themes: patterns, their recognition, and "human" vs "machine" intelligence. Following Hofstadter's law, it has taken longer to write you than I ever thought it would. ;-)

The thought-provoking part is imagining how a computer would ever "play" the game. It would involve mathematically defining "remarkable" odometer numbers, where "remarkable" is defined as any number that would intuitively cause the driver to remark to the other passengers: "Hey, look at the odometer!". The more likely a number is to cause a driver to say that, the higher its "remarkableness".

I've casually pondered the math for this for years. Let me know if you have already solved it.
sincerely,
Bruce Wallace

The Odometer Game

As many people have done over the years, I honked my horn when my odometer rolled over to all zeros [000000] (back when that only took 100,000.0 miles to do so). Later, when I put on another 11,111.1 miles [111111], I decided that an odometer reading of all ones was also worthy of a honk (a bit of a geek whimsy).

Since I had many long boring drives between college and parents, I came up with a little diversion which was to honk (or otherwise take note of) any "remarkable" odometer reading [where "remarkable" was any number that would make a driver point it out to the passengers].

I even accumulated imaginary points that mirrored the amount of "remarkableness" of the number. But, soon I realized I needed a reason to keep from simply taking note of EVERY number in my quest to build up my point total. I thought that maybe some function balancing total-points vs average-points-per-honk was needed. And to make things more sporting, I should lose points if I missed any numbers in a "pattern" once I had taken points for that pattern. In other words, if I took points for [000000] and [111111], I would lose points if I missed [222222]. (I.E., don't start a pattern if you aren't going to keep it up!)

So, [000000] was definitely remarkable, and so was [111111], [222222], [333333], etc. (Hmmm... [111111] seems less remarkable than [000000], and [222222] thru [888888] all seem less remarkable than either [000000] or [111111]...should they all get the same points?). Then came [123456]. And while less remarkable, [234567], [345678], etc. all seem pretty good.

Palindromes are very good, but [123321] seems more remarkable than [394493] or [825528]. And [121212] & [123123] are very good, but less so [838383] & [378378]. While [010101] and [999999] beat out [898989] & [888888] respectively, all seem good enough to take the points.

Round numbers like [010000], [020000], [030000], etc. seem nice because the pattern is anchored with [000000]. Actually, a number like [000000] meets lots of patterns at once: [aaaaaa], [ababab], [abccba], [abcabc], etc. (in addition to being the ultimate round number), so, it gets LOTS of points.

The Puzzle

Why are some numbers (i.e. digit strings) instinctively more "remarkable" than others? How would one model this mathematically? Patterns seem part of the answer, but a readily recognizable pattern is in [192837] even though it would seem very unlikely for a driver to make a passenger take note of that number/pattern.

And why are [000000], [111111], & [999999] all more "remarkable" than [222222] thru [888888]? Why is [123456] more than [012345], but [121212] and [010101] seem more of a toss up? Is the "simplicity" of the pattern the crux of "remarkableness"? How would one describe that "simplicity" mathematically (especially when 0 and 1 and 9 seem somehow more "simple" than 2 thru 8)? What grammar "parses" this string language?

Language Plateaus in Evolution?

2007-05-21T13:10:00.000-04:00

In reading[1] about the different levels of human language competency that plateau at various ages (6, puberty, etc), it made me wonder if those capabilities mirrored those of our ancestors at various stages of evolution. Just as a human embryo looks like amphibians, etc as it is developing (in a mirror of DNA development over the ages), maybe language skill levels that jump in quantum leaps mirror primate evolution?

[1] Introducing Chomsky, John Maher, Judy Groves, Icon Books, 1996.

The reXume/GanttMagic webapps can now be embedded...

2003-11-29T13:06:00.005-05:00

The reXume and GanttMagic web apps have been modified such that they can be configured to process a hardwired reXume file rather than only uploaded files.

My hand coded Interactive Resume web site ( http://rexume.dyndns.org/resume ) is being replaced with a new version based on the std. reXume and GanttMagic web apps ( http://rexume.dyndns.org/reXume/RBWrexume.html ) that also acts as a demonstration of using reXume for personal resume sites.

The WAR files are currently deployed on JRun 3.0 but have been tested on Tomcat 4.1 as well.

Announcing RDDL file for GanttMagic XML Namespace...

2003-10-06T10:56:00.001-04:00

There is a first major stab at publishing an RDDL file documenting the XML namespace I've defined for the GanttMagic XML input data. It is at http://www.polyglotinc.com/NS/GanttMagic/.

Soon (hopefully) there will be a similar one for the reXume data XML namespace [ http://www.polyglotinc.com/NS/reXume ].

How to Pick a Science Fair Experiment

2003-09-21T22:08:00.000-04:00

Presented is a set of intuitive classroom assignments that lead students
to pick better quality science fair projects, and in the process, teaching
them the "Scientific Process" and how it is something that they already
intuitively know.

Students are often impeded with the notion that "thinking like a scientist"
is so different from the way they normally think that they don't know how
to proceed. The given approach starts students with intuitive activities
with which they are already familiar then shows them how to merely "fine tune"
their thinking rather than viewing science as "speaking a foreign language".

In a nutshell, I propose that choosing a topic for a science fair project
and designing its experiment properly are the equivalent to the intuitive
activities of making a list of “Whenever THIS then THAT” statements and then
choosing the best “bar bet” that can be constructed from the list.

One intuitively makes observations and constructs theories to explain them
whenever a “Whenever THIS then THAT” statement is made. One intuitively
evaluates the quality of a science experiment whenever one evaluates the
quality of a bet. I.E. the same things that make a good bet make good science.
Namely:
* - you think you know something about the world that the other guy
doesn’t know (otherwise he won’t take the bet if he knows it too)
* - you think you understand it well enough to explain it (otherwise it is
not a safe bet)
* - you think you can demonstrate a prediction about it (i.e. the bet itself)
in such a way that the outcome will be clear (otherwise there will be
an argument over who won the bet).
* - you have controlled the conditions of the demonstration (otherwise it
makes the outcome of the bet be affected by things you can’t predict).
* - someone else can perform the bet if need be (otherwise people will think
you’ve rigged the game).

-------------------
The Approach
-------------------

0) Preview the Scientific Process to students

a) observations about the world are made
b) theories are created that explain those observations
c) predictions are made from those theories
d) experiments to verify the predictions are made
e) the results of the experiments either confirm the
predictions and therefore the theory, or they don't,
in which case one goes back to (b) taking into account
the new observations made by this experiment.

0) Preview some factors that make “good science”

*) observations are new
*) observations are surprising
*) theories make predictions that can be tested
*) experiments produce results that are conclusive
*) experiments are repeatable by others

1) Observations and Theories

[Use the intuition students exhibit, to both make observations
and generate theories to explain them, whenever they use
sentences like "Whenever THIS then THAT".]

(a) Assignment 1: make a list of things you know/believe about the world

“Everyone make a list of things that they have noticed (or been told
or read) about the world.
(At least 5 things and extra credit for each extra item up to 20 items).
The items in the list should all be in one of the following forms:
* - Whenever THIS happens (or not),
then THAT seems to always happen (or not).
* - Whenever THIS situation exists (or not),
then THAT seems to always happen (or not).

Examples of form:
* - I've noticed that whenever I let go of something I'm holding,
it seems to always fall to the ground.
* - I've noticed that whenever I drop a rock in water,
it never floats.
* - I've noticed that whenever people are with their friends,
they are louder than when they are alone or with strangers.
* - I've been told that whenever plants get too much water,
they die.”

(b) follow up discussion to Assignment 1: review how students made both
observations *and* theories to explain them.
* - By putting beliefs (whether inspired by direct observation or
having been informed by others) into the form "when this then that",
it filtered out simple observations that had no theory attached.
E.G. "I noticed that the sky is usually blue" is an observation, but
there is no theory to explain it. In order to say "if A then B",
one had to already have enough of an idea about cause and effect to
make the statement, whether the statement itself was correct or not.
But people/students makes statements like this all the time and
therefore are producing theories whether they realized it or not.

2) Experiment Selection and Design

[Use the intuition students exhibit, to choose interesting science
experiments and to design them to produce a clear result, whenever
they use sentences like "I'll bet you!".]

(a) Assignment 2: make bets out of the theories

Part 1: Everyone take their list of observations, and for each one, make
a bet out of it.
Examples:
* - I'll bet you that if I let go of this ball,
it will fall to the floor.
* - I'll bet you that if I throw this rock in the water,
it won't float.
* - I'll bet you that if we measure the noise level of 3 friends
eating together in the lunchroom it will be louder than if we
measure 3 people eating together that don't know each other.
* - I'll bet you that if I give this houseplant way more water
than the gardening book says it should get, it will die.

Part 2: Take your list of bets and rate each one for the factors below.
Add these factors together to get the quality score for each bet.
For safety, add the practicality factor 3 times instead of once.
The higher the score, the better the quality.
* - How non-obvious is this? (i.e. will anyone take this bet?)
Rank from 1 to 10 where 10 is "nobody knows this but me"
and 1 is "everyone on the planet knows this"
* - How well do you understand your theory (i.e. how sure of
the bet are you?). Rank from 1 to 10 where 10 means "I'm
sure I'll win the bet" and 1 means "I'm just guessing
what will happen."
* - How practical is it? (i.e. is there a way to actually
make a bet out of this?) Rank from 1 to 10 where 10 means
"this is easy to perform" and 1 means "this will take
a UN resolution to actually do."
* - How obvious will the outcome be? (i.e. how obvious will
it be who won the bet?) Rank from 1 to 10 where 10 means
"obvious result" and 1 means "we'll be in an argument
all day over who won, was it fair, is it a do-over, etc."

Part 3: Make any changes you can to the description/design of each
bet to improve its quality score before settling on the final
quality score for each bet. Take the top 3 bets and rank
each for the following factors:
* - How well can I control things that might affect the result?
(i.e. will I lose the bet because of something I can't
predict or control?). Rank from 1 to 10 where 10 means
"nothing should foul up the works if I specify when/where/
how/etc" and 1 means "every time the air conditioner
comes on it blows down my house of cards".
What conditions can be added that will make it more of a
sure bet? After adding them, make a final rank for this
factor.
* - How well can I describe the procedure? (i.e. how easy will
it be for a 3rd party to perform the bet?) Rank from 1 to
10 where 10 means "even a trained monkey could do this
correctly" and 1 means "I'm the only one who can ever
make this work".
How can the procedure and description be simplified and
improved such that others can get the same results every
time? After making the improvements, make a final rank for
this factor.

Part 4: Add the factors from part 3 to those from part 2 for the top
3 bets and pick the one with the highest quality score as your
choice for an experiment.

The GanttMagic webapp works separately from reXume now....

2003-09-15T11:55:00.002-04:00

The multiple web app WAR files previously published for GanttMagic have been consolidated into a single GanttMagic.war which can be deployed on any web container and has been tested with Tomcat 4.1 (J2EE 1.3) and JRun3.0 (J2EE 1.2). For older J2EE web containers running on JDK 1.3, you may need
to install the following library jar file into the shared library area (e.g. for JRun3.0 it is at [jrun_install]/lib/ext/ ).

Also with this release, this GanttMagic webapp may be deployed and used separately from the new release of the reXume webapp. Previously the two had to be deployed within a single webapp to work properly.

This code is running and available online at http://rexume.dyndns.org/GanttMagic/
where is runs on my old Pentium II system with RedHat6.x and JRun3 (so don't expect lightning speed).
The demo installation is still found at this test page.

The reXume webapp works separately from GanttMagic now....

2003-09-15T11:49:00.002-04:00

The multiple web app WAR files previously published for reXume have been consolidated into a single reXume.war which can be deployed on any web container and has been tested with Tomcat 4.1 (J2EE 1.3) and JRun3.0 (J2EE 1.2).

Also with this release, this reXume webapp may be deployed and used separately from the new release of the GanttMagic webapp. Previously the two had to be deployed within a single webapp to work properly.

This code is running and available online at http://polyglotinc.dyndns.org/reXume/ where is runs on my old Pentium II system with RedHat6.x and JRun3 (so don't expect lightning speed).
The demo installation is found at this interactive resume page.

Things I learned today while fighting the MSBlaster worm...

2003-09-08T19:03:00.000-04:00

(1) what looks like normal problems with Charter cable modem
service being flaky can actually be caused by MSBLASTER.

(1.5) It is hard to diagnose anything over Charter these days
because they have disabled all ICMP (i.e. ping/traceroute)
messages in a vain attempt to fight viruses. Earthlink
happily does not block ICMP so you can dial out to Mindspring
to ping Charter boxes.

(2) I found that one of my Win2K systems was infected by seeing
"msblast.exe" in the Task Manager display.

(3) searching Yahoo I found a good page about the blaster worm
which told me how to fix it and had a link to the patch
to prevent getting it in the future.

(4) while I normally am immune to these problems because of
my firewall, it didnt take long for the worm to find and
infect me while I was dialed into Mindspring/Earthlink
which puts my computer directly on the Internet (only the
cable modem goes thru the router/firewall [Netgear RP614]).

(4.5) I see that when I dial directly to the net via Earthlink,
I am constantly SMURF attacked which doesn't happen when
behind the firewall when connecting via the cable modem.

(4.6) Even though the Netgear router lets you set up a static
IP address but still set it to "ask for DNS server addresses",
it doesnt work (at least with Charter) which makes sense
since DHCP which gives you a dynamic IP address, also gives
you the DNS addresses and if you dont ask for one, you wont
get the other either.

(5) searching my Linux box's various logs to see what all that
network activity was about, I saw in the Apache logs (which
I never look at) that there were lots of failed requests
via the web for "default.ida" which is the symptom of other boxes
with the Code Red virus trying to attack me. Good little
discussion of it here.

(6) Just because you see the Norton AntiVirus running its auto-
update feature every day or so to update its virus definitions,
that doesn't mean it is scanning for viruses too. That is
scheduled separately (and it hadn't scanned my system since
the last time I did it manually about 8 months ago.)

Announcing GanttMagic from PolyGlot, Inc....

2003-09-04T16:29:00.002-04:00

I've finally gotten around to making my Gantt chart software available to others as either a product or an open-source-project (haven't decided which yet).

As with the reXume project (which uses GanttMagic to produce the "experience charts"), there is lots of web-accessible documentation to still to produce and the code published today is a first baby step.

GanttMagic produces a gantt chart (often misspelled as "gannt") given data in the GanttMagic XML format. (Here is the small example data file that the test program mentioned below uses.)
It currently can produce the chart image in SVG, JPG, and PNG formats.
It is implemented in Java using the Apache Batik SVG library.

I have broken out and cleaned up my GanttMagic code and packaged it into a JavaBean that includes all of the basic functionality, a standalone application to use the bean via a command line, and a J2EE web application to make the functionality available from any JSP/Servlet engine.

I have produced the following three varieties of WAR file for loading into the appropriate J2EE servers. [Tested on Tomcat and JRun]
1) GanttMagic3.war - tested on Tomcat3.3 and JRun3.0 (both J2EE 1.2 compliant)
2) GanttMagic40.war - tested on Tomcat4.0 (J2EE 1.3 compliant)
3) GanttMagic41.war - tested on Tomcat4.1 (J2EE 1.3 compliant)

This code is running and available online at http://rexume.dyndns.org/GanttMagic/
where is runs on my old Pentium II system with RedHat6.x and JRun3 (so don't expect lightning speed).

I'll add documentation soon, but for now, try using the test page to experiment with. Choose which type of image you want (SVG,JPG,PNG) and hit the "show me" button. The other controls are not interesting for now. JPG is the most universal format. To see the SVG, you'll need either a standalone viewer or a browser plugin. I've heard new IE browsers may already have an SVG plugin installed. Otherwise follow the links to install an SVG viewer browser plug-in to see the raw SVG rendered. Without an SVG viewer, you can see the raw SVG source which is just an XML text file.

announcing reXume from PolyGlot, Inc....

2003-08-29T20:33:00.002-04:00

I've finally gotten around to making my interactive resume available to others and I grabbed a couple of domain names to lock in the product/open-source-project (haven't decided which yet) name: "reXume" (pronounced "resume" following a trend to replace 's' with 'x' in names and to be a reference to the XML nature of the product).

So, peruse www.rexume.org or www.rexume.net and see the one and only page so far. It lets one upload one's resume in my reXume XML format, and then it returns it back to you all pretty formatted and filtered a la my interactive resume.

Somebody in Hong Kong already has rexume.com but they haven't even hooked up a "parked" web page for it. I grabbed the name because a google search showed virtually zero hits for it which makes it a great brand name.

I'll add documentation soon, but for now, try uploading my rexume file to experiment with.

BTW, I got the domain names including free forwarding from goDaddy for ~$8/yr per name.

Navigation missing from MVC paradigm?

2003-02-25T17:11:00.001-05:00

After playing with Struts and seeing its notion of what Model-View-Controller means (versus traditional "fat client" MVC), I think that MVC needs to add an explicit model for Navigation. A so-called NMVC design pattern.

In traditional MVC, Views are individual pages/dialog-boxes/panes/etc. and Controllers are geared towards translating (mouse/keyboard/clock) events into transaction requests against some data Model.

The large item left out is the universal need to define Navigation between views using controllers that are geared towards "movement" i.e. transactions against a Navigation model. In other words, a data model specifically geared towards user interface control is needed that is separate from the data models managing "business/domain/application data".

Since UI navigation (especially in a web site context) usually maps well onto a finite state machine, a navigation model geared towards FSMs (with the definition of the state-transition graph specified in an external config-file maybe?) would take out some of the grunt work of UIs.

Can Psychology help Framework Designs?

2003-02-25T14:21:00.004-05:00

There is a continuum of component sets (i.e. frameworks) that range between "simple" sets containing fewer more universal components versus "complex" sets containing more specialized components. For example, compare old school Lego pieces that were generic shapes that built anything, versus new style Legos that build only one particular type of Jedi Starfighter.

These frameworks tend to not integrate very well with other frameworks. It is simpler to pick one as a standard, but how does it affect those that prefer the one not picked? What is the overall cost/benefit of picking one vs the other [vs not picking one at all, and doing the more complicated work of supporting both]?

It would be interesting to perform a series of psych experiments to determine whether people have clear preferences between simple vs complex component sets and how well they can adapt to their non-preferred choice. Also, do the choices change based on time pressure, organization, documentation of the sets, appropriateness of the components to the overall goal?

For example...

*) - Ask people to build a "plane" from the provided components.

Let people pick between a simple vs complex set of components (e.g. specialized legos vs generic tinkertoys). Measure time taken, and "quality" of result (both self evaluation and objective third party evaluation), and "how well they liked it" i.e. "how fun was it" i.e. "would they like to do another one". Compare the success and quality rates overall for simple vs complex when used by those preferring each.

1) variation: mixed pile of components

2) variation: pile of mixed simple and separate pile of mixed complex

3) variation: organized piles

4) variation: various time limits

5) variation: make complex parts clearly "plane related"

6) variation: make complex parts clearly unrelated to overall goal

*) - Ask people to build another goal object, but require them to use the opposite of their preference in the first test. Compare measurements to those of when using their preferred components. Use similar variations to above (especially for various time limits). Compare the success/quality rates of simple vs complex overall when used by those not preferring it.

Overall goals: determine whether:

1) success rate is significantly different between simple vs complex

2) quality rate is significantly different between simple vs complex

3) time taken is significantly different between simple vs complex

4) do people have strong preferences vs weak (what is the distribution each)

5) are people more able to use either simple or complex better when it is not preferred

*) i.e. can one pick simple vs complex as a standard or must both be avail and integrated with each other??

Another variation to try is having the simple and complex component sets be compatible with each other (as opposed to the previous example of special-legos/tinkertoys that can not be used together). E.G. generic legos vs specialized legos. Goal: see if the choices made between simple/complex are as strong and the effects of the choices as strong when there is less of a consequence to starting with one set or the other.

Also, add variations to previous experiments that vary the size of the "simple" set and the "complex" set and see what the size boundaries are to these categories. See what the performance curve is when graphed against set size for the various categories of people.

Abstract Unit Tests

2003-02-23T15:03:00.002-05:00

I WANT TO UNIT TEST Requirements, Specifications, Use Cases, etc, and not just fully coded Implementations!

I believe that there needs to be a language that allows sufficiently abstract logic to be specified/programmed such that tests can be written against use cases, high level specifications, etc that are very general. An example of a high level use case might be "customer can get account balance from the ATM".

In other words, I'd like to write, compile, and lock away in a test suite, a test that verifies that if joeBlow has $10 in his account, the result he gets from requesting his balance is $10, even though no user interface, extra details like logging in first, etc have been decided.

Then later refinements of the system details can be "plugged in" such that the test can still run without being rewritten. Traditionally, tests must be written at a level of detail that is very specific such that it can interact with the actual system being tested, user interface screens, UI testing tools, etc.

On the other hand, "high level tests" have traditionally been written as high level test plans in (possibly structured) English; just as the use case descriptions themselves were in English. A language is needed that can capture this logic (in either spec or test-case form) that is precise enough to be "compilable", and have a robust enough notion of "interfaces" that future detailed implementations can be passed at test runtime as "implementors of those abstract interfaces".

E.G. If test cases can take the system-being-tested itself as an explicit parameter (even though the system is normally the ultimate of implicit parameters), that explicit parameter's type is an abstract interface, and the system-being-tested is defined as implementing that interface. [Ed. note: the rest of the world will later get this idea and call it dependency-injection!]

The abstract interfaces must be able to be associated with the more specific interfaces that make up the various levels of abstraction that are captured as the system design is fleshed out.

So, at the very high level (described in the use case above), the interface provides a simple "get balance" request that returns a "number". Eventually, the actual system has a whole sequence of interaction required to login, navigate to account, request balance, receive formatted display, logout which is encapsulated in a test case defined at that level of abstraction but which none the less can "implement" the getBalance() interface and return the dollar formatted amount as a "number". So, the specific test case implementation "subclasses" the abstract test case implementation.

QUESTION: when (i.e. at what level of abstraction) does the "interface" analogy get replaced by the "subclass" analogy?

ANSWER: silly rabbit, each level of abstraction is really a framework that defines interfaces for the more detailed level(s) to implement, and defines "subclasses" to implement the interfaces defined by the framework(s) of the higher level(s) of abstraction. [Ed. note: don't confuse frameworks with implementation-via-subclassing. They can use other mechanisms than subclassing to override generic default behavior with specific behavior.]

Musing on Events

2003-02-04T13:45:00.002-05:00

"Events" are a subclass of "Data" where there is one required attribute, namely, a time-stamp. Arbitrary amounts of other data may be included in an Event but they all have a time-stamp.

The canonical use of Events is for them to act as a trigger for some action to occur. They secondarily may be logged for audit/replay/etc. Many systems do not care what the actual time-stamps are, but only that they are sorted in chronological order.

Examples of events are; keystrokes, message receipt, mouseclick, etc.

Since the only requirement for an event is that it has a time-stamp, the clock itself can generate events that consist of nothing more than the time-stamp.

"Actors" are generators of events. Having a "thread of control" means "being able to generate events" rather than only being able to react to events.

Since events only need time, people often overlook the fact that a "clock" can be an "actor" and therefore the source of events in a system.

Objects are usually described as containing both behavior and state. State is data. Behavior is a sequence of events. Control flow is the proscription of how to decide which events are to be generated. So, if objects can be modelled as state change diagrams, the states represent the possible instances of the object data, and the events represent the arcs that trigger a change from one state to another.

Programs represent a planned source of events, and "external actors" represent an unplanned (i.e. unpredictable) source of events.

Add Performance Tests to the Unit Test Paradigm

2002-12-24T14:40:00.006-05:00

Of the types of automated unit tests (such as are typically implemented in frameworks like JUnit), I think a worthy addition would be performance tests.

It is often the case that code changes affect the performance of the code. While a bug fix might fix a bug, it might slow down performance in the process. So called "optimizations" and refactorings may actually slow down performance rather than improve it. Mostly, it is not typically tested, and therefore not recognized when each chunk of code is checked in, whether it had any effect on performance whether good or bad!

So, one could extend the JUnit framework to track the performance history of each test.

This could be simple and lightweight, storing performance numbers in local text files, or it could support a database driven framework with lots of fancy historical analysis.

It could track the min/max/weighted-average/weight of the real-time and cpu seconds to execute each test. It could flag any test whose execution performance has changed dramatically from the norm. Refactorings that did not affect functionality (but did affect performance) can be noted this way.

Something like...

import java.io.*;

import java.util.*;

import java.text.*;

import junit.framework.*;

/**

* A custom JUnit fixture (following JBuilder practice)

* that times each test run and accumulates the data in

* a flat comma-separated-value file (to be easily analysed

* via a spreadsheet).

* @author Bruce Wallace

* @see "JBuilder 7 Building Applications pg 9-6"

public class TimerFixture

{

final static String kPrefix = "TestData_";

final static String kSuffix = ".csv";

final static String kSep = ",";

long fTime;

TestCase fTest;

public TimerFixture(Object obj)

{

fTest = (TestCase) obj;

}

public void setUp()

{

fTime = System.currentTimeMillis();

}

public void tearDown() throws Exception

{

// get data to log

long now = System.currentTimeMillis(); // GET THIS FIRST!!

long elapsed = now - fTime;

String date = DateFormat.getDateTimeInstance

( DateFormat.SHORT, DateFormat.SHORT )

.format( new Date(now) );

String name = fTest.getName();

//System.out.println("\nTest: "+name+" elapsed time:"+elapsed );

// append info to test data collection file

String fname = kPrefix + name + kSuffix;

PrintWriter out = new PrintWriter( new FileWriter(fname,true) );

out.println( date +kSep+ name +kSep+ elapsed );

out.close();

}

Add main() documentation to JavaDoc

2002-12-18T16:18:00.002-05:00

Open Letter to JavaDoc group at Sun...

There is a major aspect of Java programming that JavaDoc does not cover, and that is documenting standalone programs; i.e. main() methods.

The reason simply adding JavaDoc comments to main() is not sufficient, is that while the program may be implemented inside some internal package, running the program from the outside world's perspective is a public thing.

There needs to be an "Applications" section up top in the overview page along with the packages summary. In this section, all main() methods that are being "published" can document their calling sequence (aka command line parameters), the formats of any input and output files, the list of and meaning of the process exit codes, and generally anything that all programs need to document such that the outside world knows how to use them.

In the Unix world, these things were in "man" pages. Java needs a standard, platform invariant, way to publish the same info via the JavaDoc mechanism.

I would be interested in following up with you to develop specific proposals, but at this point I simply wanted to register the need with you and see if any efforts were already in the works.

There is no such thing as a Component

2000-06-08T14:59:00.006-04:00

I maintain and preach that there is no such thing as a Component in the same way that there is no such thing as a donut hole. Just as the donut hole doesn't exist without a donut to define it, a Component doesn't exist without a Framework to define it. Using a printed circuit board as a metaphor for a framework, it's the "sockets", into which IC chips are meant to be plugged, that define components. So called universal or standalone components are meaningless (and certainly useless) without some framework that expects components of the same purpose and interface.

Ok, so what's your point? The point is that too many developers (and books on the subject) think about components as standalone chunks of functionality that can be "glued together" after the fact. They don't realize that the framework has to come first and foremost in conception and design. Szyperski doesn't get around to talking about frameworks until chapter 21 of his Component Software book for heaven's sake.

Even physical components are like this. The prototypical component, the IC chip, always was designed within a family of chips that were meant to work together. They all needed the same voltage levels for zeroes and ones and tri-states, the same amperage levels, the same clock rates, etc, etc. Other families used other voltage levels. The first reusable, interchangeable parts in history were for rifles. They were meant to be easy and quick to replace (as opposed to the hand crafted muskets they were replacing) but they were meant specifically to make rifles!

You can find all sorts of "widgets" and "gizmos" that you might guess are components to something, but unless you know what framework they were meant to be a part of, they are not good for anything but door stops or paperweights. In other words, random components don't tend to fit together or work together.

Too many people are trying to make "universal" components without realizing that those components still work within some framework that allows them to be put together and communicate with each other. The problem is that other people doing the same thing have defined other "generic" frameworks that are none the less incompatible.

For example, the toys that baby boomers played with when they were young abounded with generic frameworks of universal components: Tinker Toys, Lincoln Logs, Erector Sets, LEGOs. They all had universal components within a generic framework that let you build anything. BUT, you couldn't mix Tinker Toy parts with Erector Set parts (without glue or duct tape).

Ah, you say. That's why I like duct tape, weakly typed, languages like Perl that lets me glue together parts. Also, what about Play-doh?! You could stick anything together with that! Yes, but there was a reason you made bridges out of Erector Sets instead of Play-doh...and the same reasons apply to software systems....but Strong versus Weak typing is another discussion...

Objects versus Components

Until I had this epiphany about components as donut holes, I didn't have a good answer to the question "what's the difference between an object and a component?". I now understand that all objects ARE components, but not all components are objects. The framework that defines a set of components does not have to be an object oriented framework. But all object oriented languages define an object framework. They are generic enough frameworks that any objects programmed in that language may interoperate with each other. Unfortunately though, as with Tinker Toys and Lincoln Logs, Java objects typically can't interact with Smalltalk objects.

In the Java language there are at least two levels of object framework. There are plain old Java objects (POJOs) and there are so-called JavaBeans. Whereas any property of a POJO can be accessed (assuming its not protected by the "private" keyword) via a fooObject.barProperty syntax, only special properties may be accessed via the JavaBeans framework. JavaBeans are those objects that have defined special property accessor methods of the form: getBarProperty() and setBarProperty(). "JavaBean" is the name given to any component that works within that specialized framework. To make matters confusing however, it turns out that Javasoft called more than one framework "JavaBeans" (arrgh!). There are even more specialized versions of JavaBeans that are made to work with fancy GUI toolkits. SO, WITHOUT CLEARLY FOCUSING ON FRAMEWORKS, EVEN JAVASOFT CONFUSES DIFFERENT COMPONENT TYPES WITH EACH OTHER!

The moral? Don't fret that there is no such thing as a truly "universal" component. Don't spend energy trying to build them, or building "single universal" frameworks. Focus on what is needed for your situation and design a well crafted framework first and foremost. If it needs to work with other frameworks (like whatever Microsoft builds that won't integrate with anybody else), you will understand that the Frameworks need bridges to each other, and not mistakenly think that mere component socket adapters are sufficient.

POSTSCRIPT - Nov 6th, 2002
And of course, how could I forget the further confusion Javasoft caused by calling yet another (different) "widget", from yet another (different) framework, a JavaBean: The Enterprise JavaBean!

Component Software book ideas

1999-07-17T17:20:00.000-04:00

I see that there are some ideas in my Fault Expectant Programming framework that are similar to ideas in "Component Software" by Clemens Szyperski.

On page 307, he talks about "atomic actions" that are invoked indirectly, and even queued up for execution. This rhymes with my atomic transactions that are managed by a coordinator.

On page 46, there is a section titled "Specifying Time and Space Requirements" which rhymes with my notion of contracts having enough performance details that it enables bidding on requests by would-be workers.

Catalog of Fault Mechanisms

1999-07-07T16:29:00.003-04:00

In the previous blog entry, What Could Fault Expectant Programming Mean?, I outlined a framework abstracted out of recurring themes in a series of error detection, recovery, and avoidance mechanisms that I had compiled. Here follows that list of mechanisms (in no particular order).

Distributed Systems: remove single point of failure by using multiple components
Loose Coupling: avoids failure from spreading via ripple effects or brittleness
Ack/Nak: verify results and, on failure, repeat request (e.g. protocols)
Fail Over: verify results and, on failure, repeat request but to different component
Do Over: set aside failed requests or out-of-bound results and retry them later
Auto Restart: on failure of a component, it should auto-reset/restart and continue
Leases: dead-man switch on any allocated resource of a component (including the attention of its partners a la protocol timeout)
Event Driven: handle non-deterministic order of returned results from components
Fault Tolerance: ignore failed requests or out-of-bound results and continue rather than generating errors/exceptions
Backtracking: processing that expects to hit dead-ends and so backtracks to try other approaches (e.g. Prolog logic rules, parsers that employ backtracking algorithms)
Redundant Components: issue parallel requests to multiple components and take a majority-rules vote on the result (but must know if results are deterministic or not; i.e. if more than one result can be valid then different components may return different, but still valid, results)
Evolutionary Programming: issue parallel requests to multiple components and result is taken from the most fit component i.e. chosen by the quality of the result rather than the result itself
Auctions: issue parallel requests to multiple components that compete on cost of service as a definition of most fit
Neural Nets: issue parallel requests to multiple components and take a vote biased by each component's dynamically adjusted success rating (i.e. each component votes its stock).
Fuzzy Logic: result returned by component is biased by a probability/quality rating
Transactions: state transitions are confined to successful atomic steps
Exceptions: asynchronous notification and response to problems where there is the ability to either continue or abort current operation.
Game Theory: multiple conflicting rule sets are projected via min-max trees to find a balanced result
Blackboard Systems: multiple workers on problem process as much as each is able and share common result state i.e. individual workers are not expected to produce a complete result or even any result at all. E.G. JavaSpaces
Workflow Models: combination of state-transition models and PERT dependency models to keep track of progress at a global level given multiple parallel workers and to reset to some given state(s) if results are not converging
Belt & Suspenders: multiple independent methods of verifying results or progress
Mobile Agents: workers dynamically move to more appropriate environments e.g. load balancing, fail-over, seek more reliable communications, etc.
Design by Contract: assertions to detect failure on the part of either the requestor or the worker e.g. Cleanroom techniques, Component Interfaces, Strong Types, policy driven security managers, etc.
Pattern Matching: non-trivial matching of requestor interfaces with worker interfaces to allow more flexible and dynamic establishment of contracts e.g. KQML , JATLite

What Could Fault-Expectant Programming Mean?

1999-07-07T13:41:00.015-04:00

"Fault-Expectant" is a new term I coined for an alternative to "fault tolerant". It was inspired by reading the transcript of the JAVA LANGUAGE FUTURES session at the 1999 JavaOne conference, where there was a discussion of building robust large scale systems by mimicking biological systems' strategy of actually expecting all their parts to fail. I realized that fault tolerant (aka High Availability) systems still basically expect things to go right, but provide ways to recover from exceptional failures. A completely different mind set is needed though to act as if failure is so normal that faults are expected, and expected everywhere.

A part of the JavaOne discussion centered around the difficulty of building large systems because of the inability of programmers to generate perfect code above a certain size. The heart of the problem seemed to be our general expectation to the contrary. Unlike biological systems that expect to deal with constant subsystem failure, we basically expect the operations that we invoke to work. We don't, as a rule, build in fine-grained mechanisms to carry on in the event of faults. It reminded me of a networking company I worked with that had built error detection into their link layer protocol, but no higher layers. When I asked why, they said that since there was guaranteed message delivery from machine-to-machine there was no need to recheck it at the end-to-end level. When I asked, what if there was message loss inside a machine, they replied, "that would be a bug that we would have to fix". It didn't occur to them that protocol stacks could be built to succeed in spite of "bugs" at many levels.

So, what would "fault expectant" mean? After cataloging all the error detection, handling, & prevention techniques I could think of, and taking a step back, I saw that fault-expectant must mean some amount of non-determinism. When I ask somebody to do something, I must expect that it will fail, and automatically ask again (whether the same somebody, or someone else). But this "try again" strategy can't be shallow. It can't merely be like a protocol that re-requests a packet because it got a corrupted one. It has to be more like a game playing program that looks ahead several promising moves, but backtracks multiple moves when it finds itself painted into a corner. Like a database transaction that, not only rolls back changes if an error occurs before commit time, but actually tries a different algorithm on the re-attempt.

Non-deterministic systems are basically trying to optimize a solution using some (hopefully) converging algorithm. These systems have a long heritage in Artificial Intelligence literature. There are many algorithms that attempt solutions using components (e.g. heuristic or logic rules) that may or may not succeed, and so rolling back and trying a different path is a normal and not exceptional mode of operation. Prolog is an example of a computer language having built-in goal-seeking behavior, and backtracking after "failures". I put failures in quotes because they are more like SQL queries that return zero results; not really a failure, just didn't get the info we were looking for.

A corollary to this approach is that "instructions" must be at such a high level of abstraction, that they are effectively declarative (again like SQL). I.E. I need to specify what I want, and not how to do it. Specify idealized instructions then expect & allow imperfect executions of those instructions. The instruction follower interprets the instructions in view of its own circumstances and its own bag of tricks. The book "Meme Machine" (pg x – xii) talks about successful memes having idealized instructions that avoid any details which may fail. As the meme is perpetuated, each individual attempts to carry out those instructions and may do so imperfectly, but the essence of the idea carries on.

A Conceptual Framework

After reviewing my catalog of error mechanisms (enumerated in a separate blog entry), I came up with the following framework abstracted from a few recurring ideas. Rather than using terms heavy with connotations (e.g. client, server, message), I have used the following more neutral terms: requestor, request, result, worker, contract, match, & goal.

The Request: A requestor, seeking a goal, makes requests to a set of workers where zero or more workers may return results. There must be a decision as to which, if any, of the results are acceptable. If the goal has not been met, there must be a decision as to how to proceed. In the event of detection of worker failure (e.g. lease expiration, unacceptable result, etc.), a decision must be made as to how to proceed (e.g. Ack/Nak, Tolerance, Do-Over, Fail-Over, etc.). If more than one result is acceptable, a decision must be made as to which to choose (e.g. majority-rules, most-fit, voting-stock, etc.). The most trivial case of this entire scenario is the statically compiled function call, followed by the increasingly less trivial scenarios: method calls, RPCs, ORB remote methods, web service requests, etc.
The Negotiation: A requestor and all responding workers must have a common contract between them that specifies the accepted protocol for each request. There must be a decision made as to which workers are a match for the proffered contract (e.g. compatible interfaces, domain expertise, etc.). There must be a decision made as to which compatible workers are actually chosen to be engaged (e.g. lowest cost).
The Transaction: The act of matching requestor and worker(s) for a particular request, engaging the worker(s), judging the result(s), and concluding or aborting the request. It is considered an atomic transaction that commits or rolls back system state.
The Goal: The act of "engaging a set of workers that are returning (possibly partial) results" is decoupled from the act of "determining whether the overall goal of the request has been met". This means that there may be a communal data space (e.g. JavaSpaces, blackboard systems, etc.) where (possibly partial) results are deposited and acted upon rather than results being returned directly via any individual worker's response. It also means that responses with a fuzziness factor may or may not meet goals which have quality requirements, and hence, work will continue on the original request.
The Coordinator: The determination of which transaction(s) [and hence request(s)] are generated next is made by a workflow coordinator. It may use a static state transition scheme (i.e. simple control flow), or it may use any number of dynamic search/optimization schemes (e.g. min-max trees, backtracking, etc.). Its task is to monitor that progress is being made towards its goal, and reset as needed when progress has ceased. All state transitions are transactions. The relationship between coordinators and workers is recursive, i.e. the internal logic of a worker is handled by its own coordinator. Multiple independent coordinators may monitor the same workflow (i.e. belt & suspenders). The coordinator(s) insure that failed requestors/workers are restarted or disposed of (aka garbage-collection).

Fig 1. The transaction players and their shared data space context