Truer Words - A Journal

Working On Tools

seth@macrobyte.net — Tue, 31 Jul 2007 21:17:03 GMT

Immediately after "retiring" from the Prototype Core Team, I became active (for the first time!) on the group and finally did what I was there to do in the first place. The next version of Prototype (1.6) will have custom events. The custom events code in 1.6 doesn't look much like the code I described in my essay a year ago, but it's built on the same idea: piggyback custom events on one of the browser's built-in events. (The custom events code in 1.6 was written by a number of people, not just me.)

Anyway, the real point here is that I use Prototype for nearly all of my web projects now, and I contribute to its development. That's working on my own tools.

Plus, immediately after finishing my side of Prototype's new events code, I realized that the next version of Prototype didn't look quite right in BBEdit's function popup. (Some objects were listed as [anonymous] when they should have had names, and some class methods were listed as though they weren't contained by anything.)

So, I updated BBEdit's JavaScript module to fix that problem.

I'm rather proud of the JavaScript support in BBEdit, but (again) the real point here is that I love being able to work on my own tools! (See the BBEdit Disclaimer...)

The same is true for Conversant, which currently runs on Frontier, and which runs my site (and lots of others).

Being a tool-builder makes me feel like a real craftsman.

MySQL in Frontier, and Conversant!

seth@macrobyte.net — Fri, 15 Jun 2007 20:04:04 GMT

MySQL is now supported by Frontier (the open source version). Took a little longer to settle down than was first expected, but it looks like it's ready for testing in teh real world now.

What's that mean for, uh, me? Well, Conversant is still written on top of Frontier. We'd like to change that. In fact, there's a third party that has expressed some interest in Conversant, but that absolutely requires a migration plan. In plain (but still somewhat techie) English: the first step of any migration plan for moving Conversant from Frontier to another scripting language (Python or Ruby) is to move all of the data to MySQL.

Conversant already has support for non-Frontier databases, but there hasn't been a good connection between Frontier and MySQL until now so there hasn't been any good way to store message and user data in a relational database. That has now changed.

So my next step is to merge the MySQL code from the main Frontier branch into the Conversant branch. Then I'll finish two Twine MySQL modules: one for users, one for messages. When those are done... well, then it's time to start testing Conversant running with MySQL databases. That's exciting!!!

(To be clear: I had nothing to do with Frontier gaining MySQL support. That was done by David Gewirtz.)

How to Serve Compressed Data with mod_gzip and Apache 1.3 on Mac OS X

seth@macrobyte.net — Fri, 17 Nov 2006 01:04:09 GMT

Some of us have recently been discussing the size of the Prototype library, my preferred library for DHTML/AJAX). Proponents of some of the other libraries play up their smaller file sizes, and it's true that this is a real issue for some people.

This little essay/how-to explains the basic ideas (the what, how, and why), and then walks you through setting up Apache on Mac OS X, to enable mod_gzip and serve compressed content. If you skip the editorial content and just follow the steps I've outlined, you should have everything up and running in fifteen minutes or less.

Frontier Developer Preview Release 2

seth@macrobyte.net — Fri, 26 May 2006 19:20:34 GMT

The Frontier developer's group has produced the next Developer Preview release of Frontier. Actually, Thomas Creedon deserves most of the credit for the release itself: he assembled everything and put an awful lot of time into the browser-based installer.

Lots of new features and bug fixes in this version, and it's even a Universal Binary for the Mac OS X!

(The Conversant branch of Frontier is kept in lock-step with the main 'trunk' of Frontier. I believe most of the Conversant servers 'out there' are running the Conversant-equivalent of this Developer Preview Release 2.)

An Old New Member Helping with the Frontier Kernel

seth@macrobyte.net — Wed, 10 May 2006 21:47:49 GMT

Doug Baron just announced his presence on the Frontier Kernel mailing list.

Andre just stepped in to help close the loop that Seth opened last week when he dropped me an email out of nowhere. I fished his [that is, Seth's --ed] message out of my Junk mail folder, and a flurry of email exchanges later I find myself here, introducing myself to the list. :)

Reading the names of recent senders makes me feel as though I'm already amongst friends: Henri Asseily, David Gewirtz, David Gewirtz, Scott Lawton, Seth Dillingham, Andre Radke, Matt Neuburg. Greetings to all of you. To those who don't know me, I was one of the lead developers of the Frontier kernel, but have not been involved with it since before it went open source.

My intention in joining this list is primarily to make myself available to developers who need help working with the kernel. Time permitting, I may contribute to the build. In any case, I look forward to re-connecting with Frontier and this community.

Don't know who Doug is? He's the person second-most responsible for the existence of Frontier, after Dave Winer. Doug programmed the kernel from 1990 through sometime in the early 2000's. (I did a search in the source code: his initials appear 4,314 times!)

He was a huge part of the community, and universally liked.

I find that I'm still quite attached to Frontier, in spite of my long term intention to the contrary. Having realized that, and that I was learning quite a bit about lower-level programming by working on the kernel, it was clear that what I really want is for the kernel to keep improving. For that, we need more volunteers, which would have to come from the pool of (old) Frontier users.

That happens to be a group of people I understand quite well. Among other things, it's a group that has a collective (if small) emotional scar over the perceived "loss" of Frontier and the amazing community that had built up around it in the 90's. Virtually everyone in the community has moved on... and bringing any of them back would be difficult.

Doug is likely to be a great resource for the current, small group of kernel developers, even if he never gets very involved in coding. (He'll be the "Oracle at Austin.") Anything beyond that — like actual coding time or drawing in more of the "old ranch hands" — is gravy.

Blah blah blah... my point is that I've accepted a personal mission with the Frontier kernel. I want to build up a bright new community like we had in the late-mid 90's, around a modern, ever-improving Frontier. Doug's joining the group is a big, important first step.

What Was His Name?

seth@macrobyte.net — Wed, 03 May 2006 21:13:06 GMT

To all of the old Frontier users who happen by here in the near future:

Back in the good old days (early ScriptMeridian days, perhaps even the Frontier-Users / Frontier-Talk lists), there was an attorney using Frontier who used to participate on the lists with us. He wasn't really a guru, but he was smart and he participated fairly regularly.

Anybody else remember that? (And something is tickling my brain with the idea that he may even have been running for some local office at one point or another during that time period.)

What was his name!? This is really bugging me. I've recently been in touch with another old friend from those days — one of the oldest of the old timers, in fact — but neither of us have managed to come up with this gentleman's name.

"Issues" with Conversion of Character Sets

seth@macrobyte.net — Tue, 25 Apr 2006 23:50:16 GMT

(Is there any geekier topic..? Ahem.)

A couple people have been testing Frontier's new character set conversion features, as described here. One of the testers said that it works, but the conversion from "macintosh" to "iso-8859-1" has a problem: some single characters are replaced with multiple characters in the output.

Here's an easy example:

	- text is in "macintosh" encoding
	- text contains character 183: ∑
	- text is run through the new convertCharset verb
		string.convertCharset( "macintosh", "iso-8859-1", '∑' )
	- the result is: <sum>

I responded that the iso-8859-1 character set doesn't contain that particular character, so Apple had two options: they could put in a bogus character like Microsoft's converter does, or they could put in a series of characters which actually mean the same thing as the original character. They chose the latter: <sum> for ∑, >= for ≥, etc.

Yes, this sacrifices round-trip fidelity, which at first blush seems like a big mistake, a real issue. One can't convert mac->ansi->mac and end up with the original string.

But really… who cares? Is it that important, in the real world, that you be able to convert a string through a chain of character sets and end up back where you started? (Note that Frontier actually converts a *copy* of the original string… it's up to the scripter to replace it if so desired.)

Isn't it more likely that you're converting a string to another character set because you actually *need* the string in that other character set, probably because someone needs to be able to read the text on another platform?

New Character Set Features in Frontier (Done!)

seth@macrobyte.net — Tue, 25 Apr 2006 23:46:07 GMT

As I've hinted at a few times, I spent a lot of time in the last few weeks working on text encoding issues. Specifically, in the Frontier kernel, I hooked up the OS API's for converting text between character sets (macintosh, iso-8859-1, utf-8, utf-16, shift_jis, etc., etc., etc.)

Thanks to some help from a few people — especially Jim Correia, someone with some Hard Core experience in this area — everything is working on both platforms. Excellent!

On Windows, I had to actually learn a little about COM programming. (Really, I'm not kidding.) Frontier is now a real COM client (at the kernel level), just so that it can ask the OS for a list of the character sets available on the machine.

I don't really enjoy windows programming, it makes me feel greasy and out of my element, but I must admit that I *did* enjoy seeing it work the first time. It's especially gratifying to see the feature working on both platforms. Also in this case I didn't really have a choice: we needed this for Conversant, and most of the Conversant servers run on Windows.

A New Conversant Branch?

seth@macrobyte.net — Sat, 01 Apr 2006 19:55:23 GMT

The following is a copy of a letter I just sent to the Frontier-kernel developers' mailing list.

(This is no April Fools Day joke, I promise!)

Hi all,

I've been maintaining my own local branch of the kernel with the changes we need for Conversant.

The only significant difference between the trunk and my copy is one verb group (builtins.conv) with a single verb (compileString). The other differences are all to resources, either to support the extra verb group or to change various static strings in the app (like the name and version number).

More parts of Conversant can/should/will be kernelized, so the number of verbs in that group will grow with time, but that's all there is for now.

It's been Macrobyte's plan, from the very beginning, to release Conversant as open source. Now I'd like to follow through with that by setting up a conversant branch on the sourceforge-hosted svn repository, and maintaining the source code there rather than locally. This code would be available under the same license as the rest of the Frontier kernel, of course.

Does anyone have any significant objections, questions, or concerns before I proceed with this plan?

Now the FAR bigger problem for me is actually the Conversant-related .root files. This is a totally HUGE amount of data, much bigger than everything in Frontier.root and MainResponder.root. These should all go into the repository also (probably as another branch, like Thomas did with Frontier's .root files), but I haven't yet figured out how to handle root updates with a cvsSync-based system.

(Perhaps I'll actually do everything twice: check all changes into a root updates server, AND check the changes in with cvsSync. Honestly, I haven't spent much time investigating this yet.)

Anyway, I hope and assume that my plan is ok with everybody, but please speak up if I'm mistaken.

Thanks,

Seth

We'll see what happens. I'm not anticipating any problems, but who knows? Not knowing is why I asked in the first place (and is why I probably shouldn't hav ended the letter by saying that I "assume" the plan is ok with everybody, since asking shows I wasn't assuming that at all!)

Updated the string-wrapping functions

seth@macrobyte.net — Tue, 21 Mar 2006 20:07:35 GMT

I'll keep this one short: I've updated both of the usertalk-based string wrapping functions today. Thomas Creedon found a bug which affected both versions, so I had to update them both.

If no more bugs are found, then these are the versions which I'll based the kernel code on in a couple of weeks.

(Oh, they're here: future replacement for string.wrap and the brand new string.wrapQuoted.)

Update: Grr. Fixed the link.

Wrapping Quoted Text in UserTalk: string.wrapQuoted()

seth@macrobyte.net — Tue, 21 Mar 2006 00:48:38 GMT

Yesterday I mentioned Frontier's string.wrap, and asked for help testing the replacement I've written for it.

A related problem is wrapping email-quoted text: you know, the text that begins with ">". It's not the same as wrapping regular text because you don't want the quote delimiters to appear anywhere but at the beginning of the line.

I've created a new Frontier (UserTalk) script which currently lives at workspace.stringWrapQuoted(). I'd like some help testing it, just like the last one. Once I'm confident that it works correctly, I'll kernelize it (rewrite it in C) and release it to the open source project. It's final location will probably be string.wrapQuoted.

The full script and testing instructions are on their own page: Wrapping Quoted Text with UserTalk.

I'm looking for testers. If you have a copy of Frontier, Conversant, Manila, Radio, or the OPML Editor, please give it a whirl!

Update 3/21/2006: Fixed the link.

Replacement for Frontier's string.wrap function

seth@macrobyte.net — Mon, 20 Mar 2006 03:46:19 GMT

Frontier's string.wrap is a nearly useless verb, and is especially weird because it does the opposite of what its name suggests: it actually unwraps a wrapped string. In other words, you pass it some text, like an email with hard returns at the end of every line, and it strips out the hard returns within the paragraphs and leaves the double-spaced paragraphs alone.

I'm working on a new version of this function that attempts to mimic the string-wrapping features in the better text editors like BBEdit or Mailsmith.

Rather than waste a ton of space on my home page and the RSS feed with this, I've put the full story on it's own page, Replacement for Frontier's string.wrap function.

I'm looking for testers. If you have a copy of Frontier, Conversant, Manila, Radio, or the OPML Editor, please take a look at that page and let me know how it works for you, ok?

VeriSign Buys Weblogs.com for $2.3 Million

seth@macrobyte.net — Fri, 07 Oct 2005 15:30:06 GMT

Weblogs.com -- which is essentially a "weblog update pinging services" -- has been purchased by VeriSign, Inc., for $2.3 Million from Scripting News, Inc (ie, Dave Winer).

In my opinion, this is mostly -- well, at least half -- about the domain name. That's a very, very good domain name, obviously.

The service it runs is important, that's true. There are lots of secondary services that check the very long list of recently updated weblogs on an hourly or daily basis. As the number of weblogs has grown over the years, the load on that server has grown by several orders of magnitude, to the point that it's almost unusable by people sitting in front of their browsers: the only 'clients' patient enough to use it are the services that ping it and the automated scripts that check it.

So, Verisign paid a couple million dollars for one of the most valuable domain names imaginable. The domain name comes with a service they'll have to run themselves which can't generate any cash on its own. They claim that they're going to maintain the service as "open and free," which is actually the only way it could be done (*nobody* would pay to ping). With a relatively small investment (less than $100K), they could/should/will set up a new ping server that can handle the load and not require constant babying.

My Prediction

Unfortunately, knowing Verisign -- part of one of the most hated companies on the internet -- they will screw this up completely. Perhaps they'll require free registration before your weblog can submit pings, or they'll change the ping API. Maybe they'll come up with some even more clever way to totally annoy the weblog-publishing masses. People will react in horror that they've broken something so important, but eventually Verisign will reveal just how they plan to make money with this domain name and eventually people will shut up.

Congrats

Congratulations to Dave, though!

Now he has enough money (again) to do whatever it is he's planning with his outliner, or to go back to driving around the country, or to start pushing the next XML dialect that everybody will use and most of the digerati will hate. :-) Maybe he has something else up his sleeve! As he's made clear over the years (today being a good example), the man can talk an awful lot without ever telling us what's really going on in his head.

Setting Attributes on Elements in Frontier's Compiled XML

seth@macrobyte.net — Fri, 03 Jun 2005 18:33:12 GMT

Ugh, the subject is a mouthful, but the topic is very simple if you know Frontier at all.

To any Frontier developers who read my site but may not be paying attention to the kernel developer's mailing list: could you take a look at this message and tell me if you see any problems with my little suggestion?

Please, no ranting about... well, anything. It's a given that this is a feature we should have had yesterday.

Dave Winer's Software, Users, and Fiftieth Birthday

seth@macrobyte.net — Sat, 30 Apr 2005 23:13:17 GMT

Years ago, some of us on the old Frontier mailing lists would complain that Dave Winer wrote his software to work for him, specifically. It did exactly what he wanted, how he wanted to do it. Frontier. The Website Framework (Frontier's original static web CMS). Manila. Radio. Some feature requests would make it into the software, but anything more than an inch to either side of Dave's focus had no chance. You had to write it yourself.

Support for using the existing features in ways he didn't anticipate usually got the same lack of interest, or confusion, or even, occasionally, hostility. ("Why would you want to use it that way!?")

That drove me nuts, at the time. It's like we were all being allowed to play with Dave's toys, but only so long as we played with them his way.

Some of you are nodding your heads, remembering those days. Some of you are probably also shaking your heads (at the same time?), thinking this is an attack on Dave. It's not. Keep reading, you'll see this is about users, and even my own education.

Over the years, I've come to see things in a slightly different way. Dave loves the users -- I truly believe that -- just not necessarily *his* users. When Dave has an idea, a vision, he pushes his software in that direction with single-minded ferocity and the force of a hurricane. His users -- meaning, those using his software -- either go with him or blow away. (Consider the transition from Frontier as a Mac scripting system to Frontier as a web development platform.)

His software is just that: his. It's how he expresses his ideas. Dave is -- or at the very least, he thinks of himself as -- the software industry's Uncle Dave. His users aren't those people using his software. His users are those people using his ideas. XML-RPC. Weblogs for everybody. RSS everywhere.

He isn't always right, his ideas aren't always the best, and his implementations almost never are. They are often inspiring, though. Programmers look at what he's done and are inspired to write something better, faster, more scalable, more thorough. But they don't get it. *I didn't get it.* Dave's output isn't his software. His software is just an example, the implementation of whatever idea he was focused on.

The Atom folks think that with Google behind them, they'll beat Dave because his format (RSS) isn't as good. The spec isn't as well written, the format isn't as flexible, and there is no tightly-coupled API (atom is both a format and a publication API). They're probably right about the facts.

But! If, ten years from now, there are one hundred million or a billion Atom feeds and RSS has gone the way of HTML 3.2 (not likely), who has "won?" Would there be an Atom if Dave hadn't worked with Netscape on RSS, or pushed back so hard against the insanity that was RSS 1.0?

Certainly not.

Nor would there be a Conversant without Frontier's original Mainresponder-based discussion groups. "They suck," said I, and Macrobyte (at the time, Brian Andresen, Art Peña and myself) believed we could do better. Conversant has been evolving ever since. (That's a terrible over-simplification, but this isn't a history of Conversant.)

Seen that way, we look like nothing more than a swirly little storm that spun off the side of Hurricane Dave. (As does the current Userland, actually, and a few other companies and applications.) There would be no Conversant without Dave and his annoying, unscalable, made-to-run-his-way software. The expression of Dave's needs, Dave's ideas.

Dave turns fifty on Monday. This is Dave the man, not Dave the hurricane. Dave the man that's missing his Uncle Vavavoom, and is surely thinking about his own mortality.

So, I just want to say, "Thank you, Dave!", and let you know that while others may appreciate you for your abrasive clarity, I prefer the hurricane of ideas and the constant focus on us, your idea-users.

Oh, and Happy Birthday, Dave.

A Better Way to Bring Frontier to the Front...

seth@macrobyte.net — Mon, 18 Apr 2005 11:14:29 GMT

Andre had a better way to bring Frontier to the front: Frontier.showApplication()

It's a noop on Mac (which is strange), but was added to the Windows open source code last year, at Dave's request, to allow the startup script to decide whether or not the app should be shown when it launches. In other words, Dave already asked for this solution once, and got it.

I don't have a glue script in my Frontier.root for this verb, so I didn't know about it. (Dave has said basically the same thing.)

How to Bring Frontier or Radio to the Front on Startup, on Windows

seth@macrobyte.net — Sun, 17 Apr 2005 23:22:12 GMT

Dave Winer wants to know how to bring Frontier to the front on startup, on Windows:

In the spirit of documenting our programming work openly whenever possible, here's a question for Dave Luebbert who is working with me on the outliner, and anyone else who's inclined to poke around the Frontier source.

When the program launches it's only accessible through the Windows system tray. This behavior makes sense for Frontier and Radio, but not for the OPML editor.

It should come to the front when it's launched. I tried adding a call to Frontier.bringToFront to the startup script, thinking that would do it, but it doesn't.

One possible solution, totally script based, is to just ask the OS to open the application for you a second time. I figured this out (years ago) when I noticed that double-clicking Frontier's icon on the desktop will reliably open Frontier's MDI window if the application is already running. (The root of the problem Dave's trying to work around is that launching Frontier on Windows doesn't result in a window, you just get the little mini Frontier icon in the system tray.)

launch.application( Frontier.getProgramPath() )

That's all it takes. Just put that in a startup script (you can actually call it whenever and however you want.)

I'd really like to know if this doesn't work for someone. Please let me know if you have problems with it.

Unfortunately, it looks like he's going with a modification to the kernel. I didn't think that's what he wanted, but maybe this ultra-mini how-to will be useful for someone else, someday.

dbSpy: Database Analysis for Frontier

seth@macrobyte.net — Tue, 29 Mar 2005 18:12:40 GMT

dbSpy is a new Guest Database analyzer for Frontier and Radio, by Andre Radke and Philippe Martin.

dbSpy is a low-level database analysis tool for Frontier. It provides information about the structure and the contents of a GDB, in the form of a Frontier hierarchy of tables (detailing used, free or orphaned nodes and other similar information) and optionally in the form of a graphical representation of the GDB.

Unfortunately, the web page doesn't yet say why you'd use it.

In late December, 2004, the Frontier-Kernel list was discussing problems with root-file (database) corruption and bloat. Dave Winer explained what he had been seeing on one of his machines. Andre responded with a description of a tool he had partially completed:

I wrote a Frontier tool some time ago that understands the low-level database format and knows how to scan through a root file by visiting all blocks either in sequence and also according to the table hierarchy.

I basically started out with re-implementing the window.dbstats verb in UserTalk. The idea was to be able to analyse corrupted roots and to eventually develop strategies to fix them even if doing a Save A Copy from within Frontier was no longer able to do so, but I never got that far.

dbSpy is a more complete version of Andre's tool, but now it includes a very pretty (and useful), html-based, graphical overview of the database. (It reminds me a little of the disk-overview image provided defrag tools like Norton Utilities.)

To paraphrase Andre, its purpose is to analyze the low-level format of corrupted or bloated databases, to help the operator (programmer) learn what went wrong and what might be fixed (or fixable) in the kernel to prevent it from happening again.

Dave Apologizes, Seth Considers, Nothing Changes

seth@macrobyte.net — Tue, 15 Mar 2005 22:44:44 GMT

Dave Winer has apologized for -- essentially -- "being mean."

Here's the story. Early this year we had a little -- very little -- community of developers just barely starting to coalesce around the newly opened (open source) Frontier kernel. Unfortunately, nobody from Userland was participating in that effort: they have their own license, and chose to pursue their own development efforts rather than join the open source project. Our changes would exist only in the open source version, and their changes would only be found in their commercial version.

Brian Ablaza asked about bug fixes in the two different versions, and which one he should use. His question was asked on Userland's own message board. Scott Young (the boss at UserLand) responded with a non-answer.

David Gerwitz called him on it. Scott responded to David, but still without really answering the question.

Finally, Scott stopped beating around the bush:

"UserLand will continue to release its own version of Frontier bundled with its products."

David obviously didn't like that answer. Besides his response there, he posted a pointer on the kernel developers mailing list. That's when I involved myself, posting one message to the yahoo group, and two messages to Userland's board.

Dave Winer's response? He called us names. I didn't like that very much, but at least I made my point without losing my temper.

Unfortunately, I believe Dave's insulting tone (there were more messages, as you can see if you follow that thread on the developer's list) squashed the list. Many of us went through this back in the late 90's, and just don't want to do it again. At least, I know I don't.

Now look at the messages posted in the last six weeks. Go back even further if you want. There are spaces, here and there, with very little activity, but generally you can see that things are getting done. Flip was organizing the bug reports, some others (myself include) were fixing bugs and adding some small features. We were starting to self-organize. As you can see, though, things died almost completely after Dave's last messages on February 9 and 10. One very brief discussion of tcp.ftp limitations, and then nothing.

Now he apparently wants us to start all over again. After his apology, he justifies his behavior by saying that we weren't doing what he wanted:

First, I'd like to apologize for using the term "drama queens" to describe the people who were lobbying UserLand to do something related to this project. It was over the top. I'm sorry I said it.

I really want to get something going here, and going back to UserLand was not going to help that. The way to go is to start slowly and get some releases out, improve the kernel in small ways, get some successes under our belt, learn from them, and then do it again.

(Emphasis added.)

Now, it's no secret that Conversant is (still, presently) built on Frontier. I have good reasons for wanting to see something happen with the open source project, so I have good reasons for thinking about what Dave said. He isn't hiding the fact that he wants us to help him, for his sake. He wants his community built around his outliner, which specializes in his file format (OPML) and serves a market he's helping to pioneer (podcasting). Though much of what 'he' wants to do has nothing to do with Conversant, any improvements made to the kernel could also help with Conversant.

On the other hand, there's a lot of history to be considered. If I'm volunteering my time, I definitely don't want to feel like I'm working for Dave.

I haven't decided what to do yet, besides think out loud right here.

For now, I'll continue on the path I've been on for months: working on my own version, privately. Use our fixes and conversant-only features in our non-distributed versions of Conversant. When I do something worth sharing, like these items, I'll certainly continue to share them with the community.

However, it's going to take more than a half-hearted apology (immediately followed by a justfication) to really get me involved again. I can't speak for others, of course... but then, perhaps their silence is speaking for them.

Scripting for Production Automation - Seybold Seminars '99

seth@macrobyte.net — Thu, 24 Feb 2005 16:53:06 GMT

In March of 1999, I was on a panel at Seybold Seminars in Boston on the subject of Prepress Production Automation. It was really a Mac-specific panel, and I was there mainly as the Frontier guy.

I went into that panel with a fully-prepared presentation, only to find that they wanted to be informal... that's why my little "speech" section has such a light, semi-goofy overtone.

The best thing to come from that experience was a fantastic metal coffee mug, given to all speakers that year. I've never seen its equal!

(I just found that transcript, that's the only reason for the link.)

Frontier and the New Userland

seth@macrobyte.net — Wed, 09 Feb 2005 12:31:31 GMT

David Gewirtz has posted a most excellent message on Userland's message board, regarding the state of Userland's support for the open source project.

He's blunt but not too harsh, and backs up his words with the money his company spends on Frontier and Manila.

We discussed this a little on the Yahoo Groups list that's used by the project developers. Here are my thoughts.

Updated Frontier's Script Profiler

seth@macrobyte.net — Mon, 10 Jan 2005 16:23:15 GMT

This weekend, I updated Frontier's script profiler to track script performance in milliseconds (thousandths of a second) instead of ticks (sixtieth's of a second).

While I was in there, I added a clock.milliseconds() verb.

The combination of all the optimizations we're doing and the speed of modern computers has rendered ticks a lot less useful than they used to be. For an example, consider the following code:

local ( secs = 10 )
local ( targTicks = clock.ticks() + ( secs * 60 ) )




loop
	if ( clock.ticks() >= targTicks )
		break
	i++




msg( i + " calls in " + secs + " seconds." )

After this runs on my Mac G4/450, the value of i is approximately 140,000. On my Athlon, the value of i is about 1,400,000.

My point is that, like clock.ticks itself, most of the verbs in Frontier run in less than one tick. Since the profiler was only measuring ticks, it was sometimes difficult to see where the performance bottlenecks were. If one of the verbs in your profile table always runs in less than half of a tick, odds are good that more than half of the calls to that verb will never even increment the profiler's tick counter. This makes it very easy to completely miss real opportunities for script optimization.

Improving the profiler's granularity from ticks to milliseconds makes the profiles more useful and trustworthy because they're more precise.

Andre and I agreed that we'd like to see it taken all the way to microseconds. Unfortunately, at 1,000,000 microseconds per second, some of the slower verbs (like string verbs operating on very large strings, or verbs which must visit very large tables) would run a significant risk of over-running the limits of Frontier's maximimum long value. Being told that a verb run in -1,874,936,265 microseconds isn't particularly helpful. ;-)

Showdown: string.patternMatch vs. re.match

seth@macrobyte.net — Thu, 30 Dec 2004 12:55:48 GMT

Certainly not for the geekless among my readers: I've just posted a short review of three methods for seeking for a simple pattern in a string in usertalk.

This leads directly to my next post, coming soon...

Frontier 10: Being Involved (Long and Rambly)

seth@macrobyte.net — Tue, 14 Dec 2004 13:42:25 GMT

After Frontier's source was released, I wrote about my thoughts and feelings on the subject. I couldn't afford to get involved with it but I was happy it was done, my huge (emotional and professional) investments in Frontier, etc., etc., etc.

"Blah blah blah." That's what it amounted to.

In late September (edit: this said December. Duh.), I decided to join (and lurk on) the developer mailing list. The list is quiet, and for awhile there wasn't much to catch my attention. That changed when they started discussing tests and optimizations. (I'm freaky-addicted to both of those things. I love writing test scripts, and I can lose myself in code optimizations for days at a time.) Andre wants to optimize Frontier's handling of large tables, and needed a test script to look at the current performance, so he'd have some hard numbers for comparing with whatever he does to the kernel. I wrote the test code, and a few revisions based on group feedback (especially his).

Don't worry, I wasn't just feeding my addiction. Any significant improvement to performance could and should have a direct affect on Conversant.

In November, Andre asked if anyone was interested in working on some of Frontier's string verbs. He described how many of them could be significantly optimized. Shoot, there's that word again. I resisted at first, but eventually the itch needed to be scratched, and I took a stab at it.

This was the first C coding I'd done in a couple years. Andre helped a lot with my first task, string.mid(). Next up was string.delete(), and I needed a little help from him and a little from Brian Andresen because I was confused over operator precedence. (Which comes first: pointer dereferencing or pointer addition? Ack! Solution: use parens.) Finally, I optimized string.nthField(), string.nthWord(), and string.firstWord(), and by then I didn't need any help at all. (Note: links point to messages describing the before-and-after test results.)

A day of testing Conversant with these improvements (plus a switch to the newer PCRE verbs instead of the old regex DLL) revealed a very nice boost in Conversant's performance. Depending on a number of factors like server load, page size, and page complexity, I saw improvements of 10% to 50%! In other words, if rendering and serving a page took 1.0 seconds before the optimization, it would take somewhere between 0.5 and 0.9 seconds afterwards.

This was too good to pass up, so after some more stability tests I upgraded all of Macrobyte's production servers. Lots of people have commented on the speed boost, so I'm glad I did it.

A bug in Frontier's dialog boxes on the Mac (any dialog with a text field, like the Find/Replace window, or the window opened by dialog.ask()) finally annoyed me enough that I fixed it. Now, when a dialog is opened, the contents of the text field are pre-selected. (This was a very small bug.)

The upgrade on the servers introduced a strange bug that went unnoticed for days. Eventually, I realized that there was something wrong with the PCRE verbs: they were unable to insert just a single backslash into the result via the replacement pattern.

What, I'm speaking greek? Maybe this will help. This worked:

local ( s = "ab" );
regex.subst( "(a)(b)", "\\\\1\\\\\\\\\\\\2", @s );
dialog.notify( s )

The notification window would show "a\\b". However, this didn't work:

local ( s = "ab" );
s = re.replace( re.compile( "(a)(b)", "\\\\1\\\\\\\\\\\\2", s );
dialog.notify( s )

That way, the dialog would show "a\\\\b" (two backslashes).

I surprised myself by finding and fixing this one in just a few hours, too. The function in the kernel that scans the replacement string is supposed to treat two backslashes as a single escaped (literal) backslash. Instead, if the backslash wasn't escaping a named or numbered pattern group reference (like \\1 or \\g<foo>), each backslash was treated as a literal. With that fixed, the weird bug "in Conversant" was gone, too.

Anyway, I've decided that contributing to the kernel in ways that will help Conversant (or any of my clients) is a very good use of my time. (Just need to remember to keep it balanced! It would be easy to "play" with the kernel all day...)

How Does TLS Affect the Open Source Frontier?

seth@macrobyte.net — Wed, 20 Oct 2004 07:38:49 GMT

In response to my story about the transfer of TLS from Macrobyte to UserLand, Daniel Berlinger asks, Is there any impact on the open source release of Frontier?

My answer is, "probably not, but I don't really know."

UserLand didn't release Frontier as open source, Dave Winer did. My understanding (which may not be accurate) is that UserLand simply has a special license to sell Frontier as Manila. If they chose to contribute TLS to the open source project I'm sure they'd get a lot of thanks and appreciation, but by NOT doing that they have an advantage over the open source version to offer their paying clients.

Appreciation doesn't pay Jake's or Lawrence's salary, and UserLand is a new company struggling through it's first year of "startup."

TLS is Now UserLand's

seth@macrobyte.net — Tue, 19 Oct 2004 19:08:18 GMT

Way back in the early history of the world (late 2002? early 2003?), one of Macrobyte's clients needed to serve secure web pages from Frontier. We tried -- in vain -- to make it work correctly with IIS, but it just wasn't stable enough.

So, we implemented a tool known generally as TLS, but which was officially called "Macrobyte Resources TLS," for Frontier and Radio. By "we," in this case, I mean Brian Andresen. He did all the hard work. I acted like a client, just testing, providing feedback, and asking questions.

TLS itself wasn't our creation. The IETF did that, it stands for Transport Layer Security. We 'simply' implemented a solution for Radio and Frontier, so they could act as TLS clients and servers.

In the late Spring of this year, UserLand Software purchased TLS from Macrobyte. Today they finally announced it, and opened up their TLS site to the world.

OK, with all that background out of the way... how's this for a lame-o quote?

"I'm quite pleased that UserLand has taken over TLS," said Seth Dillingham, President of Macrobyte Resources. "This shows a commitment to enhancing the tools available to their developer community, and a genuine interest in offering and promoting secure web sites and services. With TLS now being managed, distributed, and updated by UserLand, it's sure to become a key component in the UserTalk developer's toolbox."

Whoo-ee, the things we'll say for the sake of business! It's all true, and I really did write that myself, but... who talks like that!? Makes me sound like a marketing person trying to talk like a geek. (Instead of the other way around?)