The relational database is an extremely powerful tool. But sometimes data isn’t very relational, and sometimes transactional, relational, integrity is not as important as it is for, say, a bank. This is one reason why so many sites can get away with mySQL backed by myISAM tables — they’re fine if you’re read-heavy and data integrity is not mission-critical.

Some new projects have sprung up which provide key-value stores or simpler kinds of databases without all the overhead and inflexibility of a relational database.

On the other hand, sometimes data is way more interrelated than a traditional relational database is prepared to handle. Sometimes different kinds of items (i.e. rows) in a database can be related to many other kinds of items in that database, and sometimes end users can create not just new items or new relationships, but new kinds of relationships between items. This type of database is called a graph database, and there are also projects pushing the boundaries of relational in this completely opposite direction.

Pretty much everywhere I interviewed back in February 2008 was either building their own graph database, working on an existing one, or repurposing a relational database (or, in one case, a search backend), to kinda, sorta behave like one. The w3c, not one to be left behind when there’s a specification to be written, is even working on a SQL-inspired query language intended to search them¹.

Most applications have some combination of totally un-relational data that can go in a key-value store, some strictly relational data that belongs in a SQL database, and some flexible, highly relational data that belongs in a graph database.

What will happen when these alternative databases start giving traditional relational databases a run for their money? Well, sharding, caching, and normalization all start to sound a lot more complex when the data is in a few different kinds of databases — but then again, maybe optimization won’t be as necessary if a single SQL database isn’t doing all the heavy lifting. Object-relational mappers (and the web frameworks that use them) might need to talk to, and abstract away from, different kinds of databases².

And the different types of data won’t always be easily separated along table boundaries. Maybe these different types of databases will talk to each other, or maybe they will mature into über-databases that understand lots of different types of data relationships.

But the monolithic, strictly relational, master SQL database is eventually going to go the way of Cobol³.

1. Of course, if it’s anything like other technologies designed by the w3c, it’s a steaming pile.
2. Some can already handle talking to multiple SQL databases, and of course there’s two-phase commit.
3. Or Kobol.

A web application is just a stateless¹ application that responds to various requests by performing actions and providing resources. There’s no fundamental reason an application must only communicate over HTTP. Web applications are going to start adding alternative methods of interaction, and I think the first common one will be email.

Perhaps an example will best illustrate this:

Like many web forums, posts to Mosuki’s discussion forums get mailed out in email. But, unlike any other web forums I know of, they also behave like mailing lists. All the emails have a reply-to header with an email address that identifies the message, the recipient, and the action to be taken if that email address is used. In this case, the contents of a reply email are posted to the forum exactly as if a reply had been posted via the website.

In other words, the action “post a message” can be accessed via a web page and a browser or via a reply-to header and your mail client.

There are other examples of this separation between input/output channels and the application logic. The most obvious is Twitter, which of course can be interacted with via HTTP or SMS². And the Son of Sam project intends to let you “use modern concepts like handlers, requests, responses, state machines” to interact with email.

Confirm a Facebook friend request, RSVP to an Evite, revert a Wikipedia edit, or reassign a bug report, just by replying to an email or sending an SMS.

There are a number of technical issues inherent a system like this.  An application’s framework has to handle multiple input channels, and massage email bodies, HTTP requests, and other input into a least common denominator “request.” Authenticating a user via email, an intrinsically forgeable medium, and protecting against spam, are non-trivial challenges. And a suite of templates suddenly gets a lot more complex when it has to provide views for multiple types of interfaces³ .

This blurring of the line between email, HTTP, SMS, and other communications is not new, strictly speaking. But I think it will become commonplace and even expected. Rather than writing a modern (MVC, stateless, REST-ful, &c.) web application, people will be writing modern (MVC, stateless, REST-ful, blah, blah, blah) applications that have web interfaces, email interfaces, and whatever other interfaces they need.

Stay tuned for the next installment of The next big thing: Taking the relational out of relational databases.

1. More or less stateless, that is, authentication tokens like cookies notwithstanding.
2. As well as more standalone apps than you can shake a stick at.
3. Generating text and HTML responses for email that look good and work well in the top 75% of desktop and web email clients is a lot harder than testing a site’s HTML in Firefox, IE, Safari and Opera.

This is the first part of what will become an ongoing series.

If you’ve built a website in the last few years, most likely you’ve adopted an architecture similar to Model-View-Controller, or MVC. If not, well, either your website is terribly simple, you haven’t had to modify it yet, or your code is spaghetti and you should be fired. Just kidding. (Or maybe you’ve come up with an even better architecture, in which case you should share your insights with us mere mortals.)

In MVC architecture, the model reads and writes data to and from a back-end data-store, and organizes the relational data in a nice, hierarchical fashion to be used by the controller. The view accepts input from the controller and generates output HTML, XML, RSS, JavaScript, SVG, PDF, or whatever you want to send to the user’s browser. And the controller accepts browser input, figures out what to query the model for, and picks which view to use and what data to send it.

This separation is nice because changes to the database engine or relational structure are isolated from the output generation, and adding new output formats for the same data or radically modifying existing output generation is similarly isolated from the database queries. Although controllers sometimes simply pass database results through to a view, the controller stage is important because there is very often idiosyncratic processing that has to happen between user input and database access, and between results from the database and output generation.

There you have the state-of-the art in website architecture, circa 2002. But what happens when you take an existing MVC website and turn it into a responsive AJAX site by adding a bunch of JavaScript to update the page, via XMLHTTPRequest¹, without refreshing the page? You have to add views that generate XML as well as HTML, often for the same queries. You have to generate or modify HTML browser-side by accessing the DOM with JavaScript, and often this HTML must exactly match the HTML generated for similar data by your server-side view.

Suddenly half of the benefit of using MVC is gone; your output HTML generation is no longer nicely encapsulated in a single set of views on the server side; it’s spread out across two separate parts of the code, which use two different languages (JavaScript and whatever you wrote your web app in), two different kinds of input (XML from the XML HTTP response or native language objects from your controller), and two different ways of generating HTML (DOM or whatever templating language you use), and you have more JavaScript to test on different browsers. Yuck.

With an architecture like this, you might as well go back to the good old days when web apps mixed SQL generation, SQL queries, and HTML fragment generation in a single function. Yee-haw!

How can you clean this architecture up? First, focus on what you can’t avoid. If you’re building a responsive AJAX web app that uses XMLHTTPRequest, you cannot avoid generating HTML with JavaScript. What would happen if you moved all of your HTML generation to JavaScript? The views on the server-side would generate only XML responses for XML HTTP requests from JavaScript. In the browser, your app would consist of static HTML and static JavaScript that queries the server for data when the page loads, and whenever user action requires it.

Suddenly your HTML generation is nicely encapsulated in a single place again. And the “view” in your MVC web app generates only XML. From the browser’s point of view, the JavaScript is a kind of model, querying a back-end engine (the web server) for data and massaging it into a form that the controller (the HTML) can display.

Now, this architecture is a bit idealistic. You’re not going to want to give up your powerful server-side templating language and do all your HTML generation in JavaScript with the woefully inadequate DOM interface (or worse, with document.write and .innerHTML). You’re still going to have to generate formats other than HTML server side, like PDFs or RSS feeds. You might want to use the JavaScript HTML generation pattern for some of the more dynamic features of your site, and server-side HTML generation for other, more static, features. But MVC is an idealistic goal too — even the best-compartmentalized MVC apps probably have bits of controller code sprinkled in views and models. Once you start down this road, you’ll find it harder and harder to justify server-side HTML generation at all.

When I stumbled across Google’s AJAXSLT a few years ago, I wondered why on earth anyone would take the time to implement the (bizarre) XSL templating language in JavaScript (also not my favorite language). Now I know. It lets you generate HTML in JavaScript based on XML responses from a web server. You could even use AJAXSLT to fetch XSL templates from a server, and if your server-side views were in XSLT too, you’d be able to use the same XSLT to generate HTML on the browser and server sides.

Another solution would be to keep generating xHTML server side, and write JavaScript to copy that xHTML verbatim, from the XML HTTP response, into the DOM. Either way, you still have JavaScript updating static HTML.

If you’re still doubting that this is the way to go, go to Gmail and view the source. It’s almost all JavaScript. None of the buttons or links that you see in the page are actually in the source HTML — they are all generated on the fly by JavaScript. This is not to say that Google’s always right, or that they are doing exactly this; it just shows that it is possible to write a complex web app where the HTML is generated by JavaScript.

There’s one final missing part to this parallel between MVC on the server side and JavaScript model and HTML controller on the browser side. What’s the analogue of view on the browser? It’s supposed to be CSS. The HTML is supposed to represent the abstract, hierarchical nature of the information being displayed. And just as the controller feeds hierarchical objects to the view, the HTML provides hierarchical data for the CSS to style.

Separating CSS into layout/spacing CSS, and color/background CSS makes it easy to allow users to choose, or make their own color themes without royally screwing up layout.

Unfortunately, CSS is still woefully inadequate for this job, but it’s nice to dream, isn’t it?

In the end, this isn’t so much of an AJAX vs. MVC deathmatch as it is a JavaScript-CSS-HTML meets Model-View-Controller hippie love-fest. It is possible to retain the modularization of the Model-View-Controller pattern and still write a modern, responsive web app, by extending the MVC pattern into the code running on the browser.

1. For simplicity, I’m going to ignore JSON, <iframe>, and other ways of communicating with the server without refreshing the page. However, the same problem exists: generating some HTML in JavaScript, and some on the server, is messy. Also, JSON is ugly and insecure.