Sunday, May 11, 2008

My Parts Per Million

My company did a website for a water testing device manufacture (I realize, not our usual political fare... not every client can be running for President). The client was so pleased with the new site they sent us a nice gift basket and a few of their products. Once quick USPS shipment and I am the proud owner of an HMDigital TDS-4.

So, I drew myself a glass of tap water (I don't filter my water, but Sarah does... I'll test hers once she gets a new filter) and gave it a go. My Santa Cruz tap water measures in at 216 TDS PPM. That's right, 216 Total Dissolved Solids parts per million. The back of the product says the EPA's Maximum Contaminant Levels of TDS for human consumption is 500 ppm. I'm not sure if 216 is good, great, acceptable, below average, I just know it's not the maximum contaminant level... huzzah?!

Great thing is, this device is portable, so I can start taking it to restaurants and provide reviews on water quality. It's a whole new world of eating out metrics. Oh sure, we could go out to there, but the TDS was a little high last time...

Tuesday, May 06, 2008

AJAX File Upload: The Cake is a Lie

For a long time I have been smitten with the idea of AJAX. By now everyone has experienced AJAX, even if they don't know it yet. AJAX powers web 2.0 sites like Flickr and Gmail. Allowing the user to interact with a website without a page-refresh is a strangely liberating technology... finally my applications have state! But the true holy grail of AJAX lies with the mysterious mechanism of file uploading. No doubt you've done this before, in a non-ajax fashion. While filling out some innocuous HTML form you are presented by a seemingly innocent file selection dialog box, perhaps selecting the latest photo of you kitty, to send along with the other information. This basic file uploading capability is made possible by creating a special HTML form, like so:
<form action='upload.cgi' method='post' enctype='multipart/form-data'>
...HTML form fields go here...
<input type='file' name='my_picture'>
...maybe some more HTML form fields go here...
<input type='submit' value='Down the Tubes!'>
</form>
That enctype business there tells your browser to send a special sort of HTTP request that can contain binary data. Generally requests just send text, but by enabling binary data transmission, we can send photos, mp3s, pdfs, anything within the size limit of the protocol. Trouble is, ajax requests are built such that you cannot change the enctype to multipart/form-data! Even with the cross-browser prowess of Prototype (my preferred javascript framework), there is just no way to change the nature of the HTTP request. It's either text or bust.

So, how do internet giants like Flickr and Facebook do it? What is the secret ingredient? A little googling reveals the answer as satisfactory, yet unsatisfying. Allow me to explain. To start, we need to redefine our objectives... since we can't "use AJAX to upload a file" our objective needs to be "make it appear like we are using AJAX to upload a file." When we say, "use AJAX" what we really mean is communicate with the server without a page reload. But we must remember the earlier lesson, you can only upload a file use a multipart/form-data form. Put another way, we have to call submit() on that form... there is no other path to the promised land.

HTML Forms are a tricky thing. Left to their own devices, when you call submit(), the entire page reloads. So that's out. But, we can set a target for the form, such that calling submit() causes the form to load in the target window. Setting target='_new' will create an entirely new window where the form will be processed. This is sort of cool, in that the underlying window remains unchanged. But we certainly don't want new windows popup up all the time. Yuck.

We could set the target to an embedded iframe in the main window itself. This is a lot closer, because there is no messy popup business. But now you've got this iframe reloading, which isn't exactly the seamless experience we are shooting for. The final piece to our puzzle then is to make the iframe hidden with style='display: none;' attribute.

So, now are form from above looks like this.
<form action='upload.cgi' method='post' target='empty_iframe' 
enctype='multipart/form-data'>
...HTML form fields go here...
<input type='file' name='my_picture'>
...maybe some more HTML form fields go here...
<input type='submit' value='Down the Tubes!'>
</form>
<iframe src='about:blank' name='empty_iframe'
style='display: none;'></iframe>
Now, when you hit the submit button the form sends the data, including the file, off to the server and the response comes back to the invisible iframe. To the user, nothing seems to have changed. You can add a little pre-process magic with javascript, like hiding the form, but what if you want to do post-process magic? With a traditional AJAX request you could get an XML payload back, or javascript if you use a framework like Prototype. Turns out you can do something similar with the iframe trick. You can call methods on the parent window from within the iframe by sending javascript inside of a <script> tag.

Your output to the iframe will looking something like this:
<script type='text/javascript'>
window.top.window.function();
</script>
You can call as many functions from within the script tags as you like, just remember that the iframe has no sense of the variables available from within the parent window, so that can complicate things. But a little forth-thought can go a long way to making the magic happen. You can also do some cool things like insert and remove the iframe on the fly so that it's only there during the form processing bit.

Now that you know how it works, it should be obvious how this is all a lie... a horrible, horrible lie. There isn't anything the least bit AJAX-Y about this. In fact, if you accept this as a valid method of asynchronous server communication, then you can pretty much never use the XMLHttpRequest object ever again... just communicate via hidden iframes! I realize that file uploading is a serious security concern (we don't want malicious coders to be able to upload files from your harddrive without your knowledge), and I know that AJAX presents it's own security concerns... but there has got to be a better way. I hope that future revisions to the XMLHttpRequest object provide a way to send multipart/form-data responses so we can ditch this awful, messy hack.

Tuesday, April 29, 2008

A Fantastic Waste of Time

Following a series of random links today, I discovered a wonderful invention of the internet, known as the AutoMotivator.

Here's a small taste of what is possible:


Monday, April 21, 2008

The Danger of Abitrary Strings

A friend of mine recently pointed out a rather unique "feature" that cnn.com appears to be introducing: the ability to print headlines directly to a t-shirt for, I don't really know... live blogging purposes? Maybe.

Anyway, whether by his own brilliance or act of God, Tom noted that the webpage used to generate these print-on-demand t-shirts is nothing more than your average GET string. So, when you click on the headline "Nail polish color may tip off politics" the browser sends that string to the t-shirt generator as the URL "headline=Nail%20polish%20color%20may%20tip%20off%20politics". But, GET strings have significant security implications, in that it is trivial for the end user to alter those strings, resulting in rather humorous results.

For example, you could request a t-shirt that says "CNN is stupid". Tom had other ideas. I figured, if you can say something silly, what's to stop you from saying something newsworthy.


Just in case CNN fix this little glitch, here's a snapshot of the last link's output


As Tom noted, it seems all very familiar. While CNN was stupid enough to allow their tool to be used for subversive ends, they learned Nike's lesson and prohibit you from actually purchasing t-shirts with unauthorized headlines. Which gets to the point of this post. Developers of applications are often looking for flexibility in the constant drive to make code do whatever the client requests. My own company is certainly not immune to this siren's call. But sometimes that flexibility can lead to real issues..

The right way to build such a t-shirt tool is to pass a unique ID to that t-shirt application that corresponds to the particular story in the CNN database. The tool then goes and fetches the headline and publication date and prepares a delightful, non-embarrassing, headline t-shirt. Of course, that leads to the dreaded coupling, where two seemingly unrelated pieces of technology become reliant on one another... a significant violation of the Agile principles. But you know what, I think I prefer heresy over being fired.

Thursday, March 27, 2008

Newest Family Member

Last night Sarah and I adopted a two year old cat from our local animal shelter. She came with the name Ginny, but we are fairly certain we are going to change that... just not sure to what.


Ginny?

She's very playful... and doesn't seem inclined to sleep at night, thus ensuring both Sarah and got very little sleep. Hopefully tomorrow she will fall into a more regular sleeping pattern.

Update

Ginny turned out to be name Sage... who knew?!

Saturday, March 22, 2008

The Tagging Paradigm

Welcome to my first installment on technical rumination... let's all hope by the time it's posted I've actually had something valuable to say that is more than just jargon filed ramblings. I've actually been thinking about this topic for a while now, ever since a conversation I had with David (sorry Mr. Morgan, you are no longer the only David in my life). He (David) is in the process of laying the initial foundations for a our company's new CMS system. It's actual a very interesting process, building a CMS system, because you are making decisions with significant longterm impact that are not easily changed--regardless of how agile one might be--because once the system is in use with clients, there's no easy way to retrain. So David and I spend several hours every week discussing long term implications, naming strategies, and general paradigms. Most recently we discussed the implications of a tagging based system.

For those not immediately familiar with tagging, here's a brief intro as I see it (which means get out your salt shakers, because I'm not going to fall into the boosters club on this one). Tagging is part of the the semantic web (SW) movement. For purposes fair-play, here is a direct quote from the W3C's FAQ on the main goals of SW:
The Semantic Web allows two things.

1. It allows data to be surfaced in the form of real data, so that a program doesn’t have to strip the formatting and pictures and ads off a Web page and guess where the data on it is.
2. it allows people to write (or generate) files which explain—to a machine—the relationship between different sets of data. For example, one is able to make a “semantic link” between a database with a “zip-code” column and a form with a “zip” field that they actually mean the same – they are the same abstract concept. This allows machines to follow links and hence automatically integrate data from many different sources.
So, the idea is to take the human-readable content and convert it into machine readable content in a way that makes sense and allows for all sorts of cool functionality. The best example of this in today's world is an RSS feed, which takes blog posts (just like this one) and converts it into data that an RSS reader can understand and synthesize into a format that you, as the reader, desire. So far, so good.

Tagging fits into the semantic web concept as the chief mechanism for aggregating data. So, imagine we have three bits of data: bit one [1] is tagged as "blue" and "red"; bit two [2] is tagged as "red" and "yellow"; and the final bit [3] is tagged as "yellow" and "blue". Now we can ask for all the bits tagged as blue [1,3], all the bills tagged as blue and red [1], the bits tagged as blue or red [1,2,3], or even all the bits tagged blue and not red [3]. The power of this approach is obvious... the problem with tagging is that in the rush to embrace the approach no one seems to be talking about the weaknesses.

Before I get into that, let me say something about my general philosophy when it comes to Computer Science. My general belief is that the original computer science folks were really smart and figured out nearly everything truly neat that was to be discovered based on the hardware available at the time. Linus Torvalds said it best, when commenting on Microsoft's claim that they held patents on technology in the Linux Kernel: pretty much everything about operating systems was figured out in the 60s and 70s. If you believe this--as I do--then you begin to see that the only true innovations left are a combination of hardware and software... here I'm thinking parallel programming made possible by dual-core CPUs.

Which brings us back to tags... why, if they have such power, are they really only now coming into vogue? Several possibly reasons to be suspicious come to mind.

1) Tags are unstructured data.

When talking about this with David, he said he sees tags as lists, but I don't think that's right. A list implies an order, with a start, an end, and clearly defined order between the two. But going back to our bits, if I ask for all the blue bits, the results could be [1,3] or it could be [3,1]. There is nothing about a list of tags that says either is right or wrong. It's better to say a list is a collection. Of course, I can impose an order on the collection by sorting the elements, but then the ordering is coming from within the data and is not part of the collection itself. This means (a) I have to order the collection every time I want to work with it, (b) I cannot apply arbitrary order to the collection unless I start storing meta data about the order in the items themselves.

2) Tags will never be as efficient as an actual list.

In the world of web development, the easiest way to create a list is to just use a table in your relational database. This is handy because it's just one table and you don't have to mess with JOIN statements that are inherently slower than a single table SELECT statement. Tagging, however, requires three tables. 1) a table of the data to be tagged, 2) a table of the tags, 3) a table pairing the first and second tables together. So, now you've got three tables and two joins, which is just not going to be as fast as a simple SELECT.

3) Over reliance on tags could replace good design.

Of course, developers already know this even if they are rushing to embrace the tagging concept. They know it because they are still using foreign keys in their tables to provide the usual one-to-one and one-to-many relationships. To demonstrate this, consider a recent project involving an online election tool I wrote. For this I had a table of polls, a table of choices, and a table of votes. The choices table had a poll_id field that linked the poll together with the choices, thus allowing me to ask for all the choices associated with the particular poll, or find the poll of the particular choice. The votes then had a choice_id and a poll_id, so I could do the same lookups. I did this because it was efficient and easy to understand... but one could envision doing the same thing with tags. Come up with a tag name for the poll itself and then just tag all the choices and votes with the poll tag. We'd get the same result, but it would require a lot more joining and be overall less-efficient.

4) In a collaborative web setting, tagging makes too many assumptions about users.

While researching for this post (that's right, research!) I discovered an excellent essay by internet luminary Cory Doctorow entitled Metacrap: Putting the torch to seven straw-men of the meta-utopia. He has seven reasons why the meta-data system championed by the semantic web is doomed to failure. It's short and worth a read, but the items I want to draw attention to are: (2) people are lazy and (7) there's more than one way to describe something.

4.a) People are lazy.

This blog has a tagging system, provided at no cost by the fine folks a blogger.com. If you scroll down you'll see a GIANT list of tags I've used over the years on various stories. It's a mess, and as you'll see, very few tags have more than one or two posts. I don't consider myself a lazy person, but the truth is that I haven't put in the time to organize and categorize each story... and since the tagging options available to me are limited only by my imaginative vocabulary, chances are the list of tags is only going to continue to grow... making an increasingly useless tag system for my readers.

4.b) There's more than one way to describe something.

This is the kicker, and Cory hits it right on the head when he says, "No, I'm not watching cartoons! It's cultural anthropology." Both are fair ways of describing something like The Simpsons or The Family Guy. Unless we want to go around tagging everything with all the possible synonyms (whether agreed to or not) those left with the dubious task of aggregating data based on the tags are going to have a difficult time find everything they seek.

5) Namespace Collision

The final argument is especially geeky, so hold on tight. In the real world, when I'm talking with Sarah about "David" she knows I'm talking about David at work, because our namespace has "David" mapped to "David Chelimsky who works at Articulated Man," but when I talk to Sheridan about "David" he knows I'm talking about "David Morgan who works at Ernst & Young" because that is our shared namespace. Two people, both tagged with "David", but they point to different people depending on the context in which I say the name. You run into the same problem whenever you search on google and the results are not 100% what you are looking for. Humans don't really think in terms of namespace, but for computers it's a necessity. Without a clear context, you end up with namespace collision. So, how do we solve the two Davids dilemma? We could provide more exhaustive tagging... so, now I tag them as "David Morgan" and "David Chelimsky"... but what if I know two David Morgans? Do I provide a tag that is really long and includes date of birth and social security number? Thing is, developers already use the social security number approach, by assigning every object a unique identification number. It's not especially human readable, but it does resolve the namespace collision.

Just like Cory, I don't walk away from tagging with a sense of abandonment, just healthy skepticism. I think there are two primary lessons to pull out of all of this that govern when tags make sense. 1) When the aggregation of data is truly arbitrary. This is how we avoid a tagging regime for the polling system I described above. There was a clear object hierarchy there and using the tables to describe that hierarchy was far more concise than tags. 2) when the data has a clear enough context to avoid namespace collisions. You don't want to be tagging by a persons name... there are just too few names in the world. But, if you are working with a theatre company, tagging by the particular performance ("Romeo & Juliet", "Midsummer Night's Dream", etc) might do the trick 3) When the act of tagging itself is centrally controlled. This is to say, allowing the public and large to apply tags leads to pretty useless tags (see slashdot.org as an excellent example of useless tags... not to say they aren't funny). But if the tagging regime is centrally controlled, then you can rely on a uniform scheme of description and prevent the laziness problem that I face. If I had come up with a uniform system at the beginning, it would be useful today.

I will wait to see how tags unfold in the larger internet. Done in limited situations and with the proper structure, there is real power... but just beyond that ridge is a desert of poorly organized and inefficiently accessible data.

Sunday, March 16, 2008

Changing Course

I started this blog a while ago now... back when I was still in law school. I've stuck with Blogger as a platform for longer than any other, so I feel like I have something invested here. Since it's inception, Pro Bono Geek has been a decidedly political blog, emphasizing the Pro Bono over the Geek. My law school and student government experience provided a lot of good fodder for blogging... and while the online comments I received were never great in number, I was always surprised at how many people at least read what I posted and told me in the "real" world.

But since I've moved to Santa Cruz, I've been posting a lot less. I think being out-of-sync from actual political discourse has been a big part. My political world is pretty much Sarah and the Washington Post. On top of that, I'm not entirely certain I have anything new to contribute to the larger discussion. For the most part I'm just recycling what you can read elsewhere tempered by a bit of market-idealism. Yes, I've got the occasional intellectual property rant, but I think those were more for me than anyone else.

The question remains: if the blog is to continue, what will it be about? I think, perhaps, the answer to this question lies where I started this post: Pro Bono Geek. Except, this time, maybe we need a bit more geek. I feel like I'm got an interesting viewpoint on the larger technical universe right now. Author of the world's formost legislative simulation website, a sprawling Perl CGI application in desperate need of love (mod_perl anyone?); system administrator for one of the largest Zope deployments around hosting some of the biggest names in Democratic politics; a lieutenant of David Chelimsky world renowned Agile evangelist and core contributor to R-Spec with whom I will developing some kick-ass stuff in the next couple of years; power-user of Debian GNU/Linux and KDE; and javascript wizard for a leader in political web design. For better or worse, my life has taken a decided turn towards the geeky end of my personality, and it's time I embrace that way of life.

To that end, I'm going to start posting again (trying for once a week) about the interesting technical/geeky stuff I'm doing... tragically, not everything I do can be disclosed, but it's not like a Google-NDA, so I'm freer than most to discuss my latest thoughts on all manner of topics. For some of you, this may be that final excuse to kick me off your RSS reader... and I understand that. For others, you might finally be interested in what I write here. Others may be interested in just continued life updates, and I'm hoping to get back into the swing of that too, but no promises.

Oh, and if you were hoping I would stop posting political content all together, I have some bad news for you.