Open Laszlo 3.x vs Open Laszlo Legals: About that tablecloth...

Jim Grandy wrote

the magician yanks the tablecloth out from under the place settings for an elegant meal, leaving the wine glasses and candlesticks in their places, unperturbed. A neat trick, but even better would be to replace the tablecloth with a new one, all without touching the candelabra or the crystal goblets. I’m not usually given to hyperbole, but with our latest Legals snapshot, PR3, we are nearing the end of a process that feels a lot like replacing the tablecloth.

Developers have started looking at the preview releases, and they're noticing that not only is the DHTML runtime incomplete, but the swf runtime has major regressions. "What's with all these regressions? Isn't that a bad indicator of code quality? Is the next version of Open Laszlo going to be worse than the current one?"
Well... I can see how this is worrisome. Let me offer this adjustment to Jim Grandy's analogy: Legals doesn't just replace the tablecloth; Legals sends the chef to culinary academy. The chef comes back from culinary academy, and changes the sautée pans, the waiters' uniforms, the produce supplier, and the actual furniture... but he leaves the menu the same. And, yes, he keeps the same silverware and china. That nasty ceviche appetizer is made with tuna instead of snapper, and the house salad uses butter lettuce and hothouse tomatoes instead of iceberg and cherry tomatoes.
What I'm trying to say here, while perhaps pushing the analogy too far, is that Legals changes the compiler and the Laszlo Foundation Classes. A lot. We refactored the Laszlo Foundation Classes into the runtime-independent parts and the runtime-dependent parts. In 3.x, the LFC code assumed that it was executing in a flash runtime. The 3.x LFC wasn't shy about reaching down to touch the flash movieclip directly. Legals inserts a portability layer between the LFC and runtime-dependent code; we call this portability layer the kernel. Everywhere that LFC code, components, and examples assume a flash runtime needs to be refactored to call the kernel API. So, you see, just about every line of the LFC has changed, even for the swf runtime. All that change was bound to cause regressions and introduce new bugs.
Here's the good news: we're software engineers with a track record of shipping solid software. We manifest our commitment to high-quality software by fixing bugs properly not just expeditiously. This stage right now is the right time for fundamental architectural changes and thorough discussion and investigation thereof. We have a large set of unit and integration tests, which we run, and ever-improving tools for tracking, assigning, reviewing, and fixing bugs. At any given time, several team members are devoted full-time to either increasing test coverage, improving test tools, or fixing known bugs.
When you sit down to eat at the new restaurant before opening night, the table might wobble, and there might be too much cayenne in the chili. But we're fixing the table by switching to a three-legged design, not just shoving a wad of napkin under one leg. We switched to venison and buffalo chili, so we're fine-tuning the spices with every batch we cook. We're working hard on getting it all back to solid and tasty, and we won't open the kitchen until taste-testers have approved every dish.

"there's more than one way to do it"

TMTOWTDI is pronounced tim-todd-ee and it means there's more than one way to do it. I think I picked this up from Programming Perl or Learning Perl. I didn't really learn to program or learn perl, from consulting those books, but I did learn TMTOWDI... which is kind of a problem.
I'm working on a relatively simple task: measure, store and analyze benchmarks for Open Laszlo runtime performance. The problem is that there's so many ways to do it! My thinking started with Java, because most of the OL server is written in Java. Indeed, I implemented the measure-and-store part of the problem with Java Server Pages (JSP). Now it's time for log analysis, and ye gods, Java has been through major honking changes in its support for logging, regular expressions, and XML. There is a surfeit of XML API's for java; in java 1.4 or so log4j-ish behavior became part of the java platform, in java.util.logging; Xerces, Xalan, SAX, JAXB, and JAXP all seem to do pretty much the same thing (parse XML) but Xerces and Xalan were merged into another project, and JAXB and JAXP seem to be just implementations of SAX. Maybe. And how does this all relate to java.util.xml? On the way-back machine that is the WWW, projects from 1999 and Java 1.1 can be nearly indistinguishable from a project from 2006 and Java 1.5. And how can I distinguish between an abandoned project and a project which is so stable that it hasn't needed an update in two years?
...Then the java code that does seem relevant seems to require a heckuva lot of boilerplate. All I want is a bit of regular expression support and math!
That sort of thinking leads directly to perl. Perl has superpowers and was designed for just this sort of task, but I'm not a perl programmer, and I don't want to be. Sorry. Does that make me a bad person? Lazy, maybe: I'd rather spend my time learning and working with a language that I like. I'm not talking fundamental goodness here; I'm talking personal preference. I don't like the idioms of global variables with names like "$_".
Personal preference leads me to Ruby and JavaScript. Ruby, oh you tasty jewel, would be a pleasure to write this log analysis and reporting in -- but ruby is one of the few languages that isn't required by the Open Laszlo tool set yet. A disciplined software engineer wouldn't add a new language requirement to the development environment on a whim. JavaScript would be darn nice, too, especially using E4X, ECMAScript for XML. Is E4X supported by the version of Rhino we use for unit tests in Open Laszlo, rhino 1.5R3? I just spent ten minutes browsing mozilla.org trying to figure this out, and I still don't know. (This feels like what I'll end up doing, though.)
My high school fencing teacher used to say, "don't parade your ignorance" and maybe as a career web programmer I'm hurting my future prospects by confessing confusion with all these API's. So I'll skip rhapsodies on python and bash and XSLT, and just get to the point: TMTOWDI, so let's just get it done! All of these languages and libraries can get the job done, with a complex set of trade-offs between brevity, readability, maintainability, portability, external dependencies, and personal preference. It doesn't matter. Just pick one and get on with it. Stop trying to figure out which one is best; stop trying to grok new technologies while standing in an aisle at Barnes & Noble. What would get me from here to performance benchmark analysis by tomorrow? Straight-up java parsing. Forget XML, just use line-based text parsing with a string tokenizer. Forget object serialization. Forget XSLT. Forget trend graphs of benchmarks over time. Just use plain old text files, and get the job done.
(If I get to producing a simple report with single-number reports on the performance of three different builds, today, then I can play with Ruby while watching Desperate Housewives.)

UPDATE (eight hours later): I wrote the tool that I need in a few hours, using java's StringTokenizer and JUnit. I also went for a walk along the beach and took a nap. Shiny things: IntelliJ IDEA 6 integrated JUnit support. I'm in love with the green bar.

Porting to Legals

In a post on the Open Laszlo forums, rcyeager of cooqy described his experience trying his Open Laszlo applications in the latest preview release, PR4:

The demos in the PR4 Explorer work...but my "real-world" code fails miserably. I expected DHTML conversion issues, but to lose support of SWF mode compatibility is scary...and this is with Legals supposedly going to Beta soon? In my case, 100% of my V3 apps don't work in either SWF mode or DHTML w/ Legals PR4. Is anyone getting better results with their real-world V3 apps running in Legals?? So far the attempt by Legals to pull the tablecloth out from under the dishes has left my lobster dinner strewn all across the floor...

Jim Grandy's post about Legals Preview Release 3 sets the stage. My perspective on porting existing applications to the multiple runtimes infrastructure is based on converting the calendar demo. A few highlights:

• swf assets won't work in DHTML. To display a swf asset, you'd need the flash player; we're not using the flash player in DHTML, so, swf assets just won't appear. The solution here is to use bitmap assets instead of swf's.
• We haven't made all the components work yet in PR4. We've got open bugs in JIRA for all the component problems.
• drawview support was just added in the last few days; it's not in PR4. Look for it in beta 1, at least for firefox. (IE support is still in progress, I think.)
• DHTML doesn't support embedding fonts; swf does. If you try to embed a font, you'll get a compile warning, and the text will look ugly, or it won't appear at all.
• The flash runtime is permissive about derefrencing a null reference and getting properties on an undefined object. DHTML blows up if you do this.Because it was permitted in swf, lots of code doesn't check for null. All the places that don't check for null need to be tracked down and fixed, both in the LFC, the components, and application code. This was the source of at least half of the problems I had.

The overall story is that we're still getting up to feature-complete, and we know we've got a lot of optimization and bug fixing ahead of us. We really appreciate all our users, especially when they take the time to post their concerns and problems. Keep the bug reports coming, and we'll keep working.

changing values

Doing laundry instead of shopping for new clothes. A simple meal cooked and enjoyed at home, instead of another $16 meal from a thai restaurant. Getting a great deal at Sports Basement instead of overnight shipping on a plastic-within-plastic patagonia mail order. Neighbors who know my cats' names and all three of my names, instead of neighbors who genteely avert their eyes if we cross paths in a hallway. My threadbare collge sweatshirt instead of a pre-stressed sweatshirt with the name of a surf competition that never happened at a beach that doesn't exist. Spending time with my cats on the back porch, watching them bat at moths, instead of battery-powered pet toys with fluorescent feathers. $10 key-combo software instead of a new keyboard with literally buttons and whistles. Running a line from the cable box to the clock radio, instead of buying new speakers. Discovering old music in my collection instead of "Discovering New Music" at Starbucks. Self-serve locally-roasted Kona Blend instead of a six-dollar latte. Rereading my favorite books instead of buying the flavor-of-the-week at Barnes & Noble Big Box.
It's not just that these things cost less than the alternative... It's that they actually feel better. You couldn't have told me this ten years ago, or even five. I used to think that costing more money inherently made something better than any lower-cost alternative. I used to think that new meant luxurious, and old meant moldy. I used to think that bills were just greeting cards with numbers on them. And today, just today, I'd rather have a little less debt than a little more stuff.

Constraint maintenance in Open Laszlo

Sometimes someone says something so clear that part of my worldview crystallizes, creating order where before there were only vague ideas. A few weeks ago, "tomcat is a servlet container" hit me this way. Today, it's getters and setters in Open Laszlo. In recent email to the laszlo-dev mailing list, P. T. Withington wrote:Before my time, I think we had getter's that pulled values, now we have setters that push them, and getters are only there for back-compatibility.

I understand this, I think: Open Laszlo is permissive about accessing an object's attributes. Open Laszlo maintains a one-way constraint propagation network. If value x changes, then any constraints which depend on x must also change.

<attribute name="x" value="70"/>
<attribute name="hotness" value="${x * 3}" />

To maintain the desired constraints, hotness must be recalculated whenever x changes. There are (generally speaking) two ways to maintain this dependency: "push" or "pull."

In a pull system, hotness would only be recalculated when it's accessed. Between the moment when x changes and the moment that hotness is accessed, hotness is in limbo; it doesn't reflect the value of x. That doesn't matter as long as you only access hotness with an accessor method. While the value of hotness is unused, it doesn't matter. However, if you change x and then access hotness directly, you are not going to trigger the constraint recalculation, and you'll get an old dirty value of hotness:

// Pull system example
Debug.print("hotness is ", myobj.hotness); // prints "hotness is 210
myobj.x = 12; 
Debug.print("hotness is ", myobj.hotness); // error, prints "hotness is 210"

To get the right value of hotness in a pull system, you'd have to call the getter:

// Pull system example continued 
Debug.print("hotness is ", myobj.hotness); // prints "hotness is 210"
myobj.x = 12; 
var t = myobj.getAttribute("hotness");
Debug.print("hotness is ", t ); // correct, prints "hotness is 36"

In a push system, hotness would be recalculated when x changes. The runtime maintains a list of the attributes that depend on x, and when x is changed, it tries to inform interested parties of that change. (This is really handled with delegates, but that's an implementation detail.) The runtime needs a chance to push the changed value into hotness and charm. It only gets that chance if a setter is called:

// Push system example
Debug.print("hotness is ", myobj.hotness); // prints "hotness is 210"
myobj.setAttribute("x", 12);
// Behind the scenes, delegates update hotness and charm. 
Debug.print("hotness is ", myobj.hotness ); // correct, prints "hotness is 36"

Open Laszlo uses a push sytem for constraints. It is safe to get the value of an attribute with simple object access myobj.hotness but it is not safe to set the value of an attribute by direct access without calling the setter. To set an attribute which is an independent variable in constraints, you must call the setter." Setting a value directly, without the setter, can only be done if you don't need dependent constrained values to update. This is dangerous, but fast.

Understanding Tucker's quick quote explains why some code uses getters (an old idiom, no longer necessary, but valid) and some code uses setters. (They're required for constraint updates.) The correct, safe idiom is to only set attributes with setters. When this is too slow, setters can judiciously be replaced with direct assignment. Why do we use a push system? Performance. Typically the ratio of sets to gets of properties in a program is 1:4 1:5, so it is more efficient to only have to intercept the sets.

Postscript: Tucker adds, "setAttribute should be an implementation detail too, some day. The compiler really ought to intercept your attempts to set properties that constraints depend on and do the right thing."

Alternate ending for The Departed (not a spoiler)

Colin Sullivan runs into the woman's apartment, hoping he's gotten there before the bad guys. He's too late, but she's still alive, gagged and tied to a chair! They spared her life, but took her (horrors!) right hand and wrist! As soon as he removes the gag from her mouth she gasps, "the sink! the sink!" He runs to the sink where her fingers are just barely visible in the maw of the garbage disposal... which is on! In perhaps vain hope of reattachment, he scrambles to find the off switch, and simultaneously to remove the severed hand from the drain. But (again!) he's too late; he grasps at the fingertips; he can't get a grip on them, but the motion of the disposal on the tendons opens the fingers, releasing... what? A travel-size game of Clue, the classic Milton-Bradley board game! The camera zooms in on the pristine label of the game, and the sink and the desperate man and the brutalized woman are replaced with an advertisement for a new, The-Departed-branded special edition of Clue, based in not a crumbling mansion but a map of the Massachusetts State House, with butlers and professors replaced by, ahem, cops and lawyers. With the characters fate implied but unknown, the bad guys probably drinking Guinness in a bar in Southie, and the marketing deals solidly in place (the Clue special edition includes a DVD with the theatrical trailer for the movie! and a real replica "T" token!), the credits roll, and we (the self-forgotten audience) are sitting in the theatre dumbfounded, all of our passion and excitement and dread created by the ever-onrushing story replaced with a none-too-subtle feeling that our lives would be this exciting, too, if we were much worse people, but given our straightforward allegiances and Judaeo-Christian values, we can at least spend a sunday afternoon (the concluding anti-climax product placement tells us) pretending we are as vital as William Caustigan, Jr, as ethical as Captain Queenan, as powerful as Frank Costello, and as vicious as Mr. French. And yet... we are simply the bourgeoisie, resembling these brazen, sultry, consumed characters only in that we, too, are doomed to die.
FIN.

tuesday is tools day

Today is a big demo day for Open Laszlo at Ajax World. where we're showing off more progress on our multiple runtimes infrastructure. I've been heads-down crunch-mode working on demo apps for a week, so I think today might be tools day. A few months ago I rebuilt our nightly build system, and by now it's become clear that it's really a build-on-demand system. The turnaround from checking in a bug fix to running an official build on a qa machine could be about a third what it takes now, if I parallelize it. The build is already distributed onto three machines and three OS's, but it runs those three builds serially. Worse, it doesn't even parallelize pushing the distros from build machines to distribution machines. I can improve this a lot, pretty easily. It doesn't feel as <em>urgent</em> as fixing bugs in demos or tracking down bugs in the DHTML kernel, but I think it might actually be more <em>important</em>. Saving thirty minutes each time we're waiting for an official build adds up to real time, especially with some big releases coming up. Time is more precious when preparing for a release, so saving time in the release process saves extra-special time. I've convinced myself, so, it looks like tuesday is tools day for me.