Going All-the-Way with the Client-Server App Concept
So you're going to build a big web app. A long-lived, single-page app with lots of data and Ajax updates. Maybe something like Gmail.
the traditional server-side web framework
Many server-side web frameworks were primarily designed to build traditional web sites. A request for a page arrives to the server. The server gets the necessary data from the database, assembles the HTML presentation of that data and sends back that page. The response page may even have a form or two. That is great if you want to build a traditional web site. It is the only way, in fact. The server has to do all the work.
The traditional server-side web framework must combine data (e.g. in files or databases), presentation (e.g. HTML), and application (e.g. links and forms) in pages. This combination tightly couples the work of the server-side and client-side programmers. This can be a development problem for a big single-page application as the server-side and client-side programmers are working on the same files but primarily concerned with different aspects of the program.
The server-side programmer is mostly worried about security and data access and persistence. HTML, JavaScript, CSS, image files and other static resources in a modern web app are all part of the client-side application and not the server-side programmer's main interest. To a certain degree these files are clutter that may use a different development and release cycle. If the server's responses are full of HTML then testing is complicated by the fact that the response data is buried in the response HTML. Tools have been developed to help deal with this but generally it is still a mess.
The client-side programmer is worried about the wow-factor and marketability of the user interface presentation. Also the client-side programmer is interested in the API that the server provides for Ajax interactions. Security is a much smaller concern for the client-side programmer as there isn't too much he can do in that department anyway.
The worst example I know of for tight client-server coupling in a rich, single page web app has to be from Ruby on Rails. Last I checked, Rails' EJS templates return JavaScript code filled with HTML in strings. The JavaScript code is evaluated when it arrives to the client. That JavaScript then inserts the HTML into the various parts of the page as necessary. How would any other type of client (think non-browser) use this type of API? In short, other clients cannot use this API. A Rails app must provide various response formats for the same data. You can request GET /todos/2009/04/21 and get the EJS version or you can ask for pure data GET /todos/2009/04/21.json. The server has to know how to make both types of responses and test both types of responses. If something in the client-side JavaScript changes then the EJS responses may need adjustment so the HTML in those responses makes sense even though the data in those responses hasn't changed.
If the large amount of coupling that a traditional server-side framework might lead a programmer to add between the server and client programs could be reduced, both sides would benefit and development time would be reduced.
the non-web client-server app
There is a model for client-server applications that is older than the web. In this model, the client has applications installed first. These applications are concerned with the user-interface and getting and sending data to a server. The user interface application doesn't have the data. The server knows about and stores data. The server doesn't know anything about the user interface. The two sides talk only through the server's data API. It is a very small and can be a very clean interface between two large parts. Such a data API is easy to document, understand and test. If the client-side presentation changes the server doesn't need to change at all. If the server code is updated for speed improvements, bug fixes, etc, the client-side doesn't necessarily need to change.
A key part of this architecture is the client has the application installed before it requests the data. They don't arrive together like they do in many web pages. The answer for cleaner web application implementation is separating the application and the data.
going all the way on the web
To go all the way on the web we must completely separate application and data. We can start with a completely static application page. The page can contain HTML elements including forms, and links to CSS, JavaScript and image files. The important part is the whole page is completely static. The server doesn't have to fill any template variables with user-specific data when the page is served.
Taken to the extreme the static page can be as short as the following.
<!DOCTYPE HTML PUBLIC
"-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<title>The App</title>
</head>
<body>
<script src="/static/js/the-app/bootstrap.js"
type="text/javascript"></script>
</body>
</html>When the application page loads, the client asks for necessary data though the server's generic data API (e.g. JSON, XML, CSV response formats.) Assuming valid auth information is supplied in the request header, the server returns the data. The client application uses this data to build the user interface. As the user continues to interact with the user interface, the client application can continue to communicate with the server though the server's data API.
What I like so much about this system is the server-side programmer is only concerned with creating a secure data API. Removing the complexity of the client-side application means less code in the server-side programmer's repository. It means tests are easier to write as they are only testing data structures that don't change when the presentation changes.
What I like so much about this system is all the client-side application files can live outside the server-side application's code base. Even in a different repository. There is no connection between the two except for data requests. A mock server can be built to return dummy data specifically designed to help user interface development. The entire client-side app is in the hands of the client-side programmers.
What I like so much about this system is the server and client apps can have different release cycles with less time spent coordinating between server and client programmers.
an extra request
If the application and data are mixed, as they are in many web sites, then they arrive in the same first request.
Separating the client-side application from the data might mean an extra request. The client first requests the application. That application then requests the data. This cost is likely small and may already be in place in many web pages that don't quite go all the way.
securing access to the client-side application page
In a traditional framework with a dynamically generated application page, the application page is not available unless the correct auth information is sent with the request for that page. Often times, however, the other resources for the page (CSS, JavaScript, images, etc) are not protected and anyone can view them.
If the client-side application page is completely static and goes on "the static server" then what is protecting anyone from viewing it? Nothing. Well nothing if nothing is protecting your other static files. If your application requires protection for the application page then it probably requires protection for the JavaScript files. Use the same method to protect both so that only authorized people can see any of them.
there is no silver bullet
I'm not suggesting every page on every site would be well architected if designed to work as described in this article.
summary
One of my long-term work projects over the past couple years has been a long-lived, single-page web application. The way we built it is quite close to what I've described above. The application page is very small. Most of the application's data is retrieved through the server's generic data API. Unfortunately we didn't go all the way. The page does contain a small amount of user specific data and it is still dynamically generated. Many times I do wish we had gone all the way. I know complete client-server separation would have made development faster and easier for both sides. I'm looking forward to going all the way some time in the future.
Comments
Have something to write? Comment on this article.
Andrew,
I agree this is not the right approach for all pages and would be a very tough sell for a page available to the general web. Most complex tools that I have encountered that might benefit from the design I discuss are likely to be behind login where JavaScript is a necessity anyway.
MV,
Thanks for the link. What I'm suggesting is not new but hopefully my article sparks some more thought.
How do you manage the UI design of the application? Via templating in JavaScript? Isn't it hard to incorporate changes from the web designers into the JavaScript?
Johan,
One of the files that the client gets from the server is a templates.xml file. It looks roughly like the following and is easy for anyone to edit.
<?xml version="1.0"?>
<templates>
<t id="greet"><![CDATA[
hello, {{name}}
]]></t>
</templates>A couple small JavaScript functions (~40 lines total) load the template file
loadTemplates('/static/templates/the-app/templates.xml',
{success: function(xhr) {/* ... */},
failure: function(xhr) {/* ... */}});and can render a template and data as an HTML string
var html = renderTemplate('greet', {name: 'world'});The above is just one approach. It works because the project I'm on has simple templating needs and so the JavaScript to run the templating system is small. If the templating system grew to hundreds of lines of JavaScript something would need to change...
Another approach would be to compile template files (no XML involved) to JavaScript functions and send those compiled functions to the browser. I like the idea of this compiled approach better but there wasn't a practical need in my current project.
nice article. this is how web apps should be built in my mind.
i've been lucky enough to build a few apps just like this for some corporate intranets. what made the separation of concerns possible was client-side XSLT.
we had a similar empty html page with the JS bootstrap link. all we ever pull from the server are XML services, XSL templates, Javascript, and CSS. So clean and nice and organized.
XSL was great for allowing us to do things like sorting, filtering, in place editing - all client side. There was surprisingly little javascript code, most of it concerned only with pulling the right resources to generate the view.
nowadays it seems clear that JSON is usurping XML for these type of activities. What bothers me though, is that I've not seen anything that comes close to XSLT for JSON.
i get sick when i see all these embedded HTML fragments inside of Javascript code. it's just not right. hopefully one of these JSON Templating projects really gets it right
I very much agree. I had a similar epiphany over a year ago, which made me write these two articles;
and
The End of Web Frameworks - part II
The realization that the client should be put on the client, and the server become 'headless' is described at greater length in a tech-talk I did on the subject at Google last year.
A good starting point is the effort Thin Server Architecture Working Group.
There are a lot of people investigating these ideas at the moment, and my hope is that a more sane approach to building web apps will emerge the coming years where people in the teams can do what they're really good at instead of having to deal with everyone else's business :)
Cheers,
PS
Eight of the ten major web projects I've worked on since 2005 have all employed this model. The client-server architecture approach is better for several reasons (some of which you mention):
- it scales better
- it is easier to maintain
- it is easier to build on non-web clients
- it makes "offline mode" not just possible, but simple
There are, of course, downsides:
- load performance becomes critical because client-side code is larger
- many UI developers don't understand MVC or PAC design patterns
- i18n, a11y, and customization all require new thought paradigms
dojo toolkit helps tremendously with this type of app. dojo fully supports RPC and REST style server APIs. It helps trim load times. It also comes complete with i18n and a11y solutions. Of those eight web apps I mentioned above, six now use dojo (some started out using other frameworks and migrated to dojo).
Thanks for the great article, Peter!
Yeah, I can speak pretty extensively about this approach. I've been working for 2 years on an application employing this technique, and I wrote all of the infrastructure code to make it work. Some day I hope to make the framework open source, but I think we should probably finish the app first ;)
In order to pull it off, I actually did go the template compiler route. Basically, all of our view templates are written in valid xml which is a superset of xhtml. Basically, you can start with xhtml, but there is a lot of additional abstractions for code reuse like custom tags/attributes. The custom tags and attributes are all easily defined using other views and javascript. Basically if it will ever result in html on the screen, it is specified in an xml file. If it controls business logic, it is written in javascript.
I built data binding into the view compiler, so you basically never need to write a single line of dom code or even jquery style code. In the end, it is similar to working with Flex or Silverlight, but obviously with its own unique flavor.
That said, I totally agree that this is the direction that many application developers will be going in the future. It doesn't work for a large class of sites/applications out there, but for us, its definitely the right way to go.
I've been thinking of doing something like this ever since I read the Sitepen article about the Client side framework which uses Dojo's client side DTL template interpreter. Which is great for 'Web Application' but bad for web content (no SEO, browser history is complicated, etc).
The thing about using DTL on the client is that you can also back it up with a server version as well if you need to, which is good to content presentation. Most web-apps have to do both at some point.
I think having a server as a pure data model is the way forward. It should spits out something like XML which can then be transformed via XSLT into whatever format the client wants.
Given the number of platforms/clients these days what other alternative is there!
I very much agree in a separation between the UI/application and the data/application.
However, I don't think that Javascript is particularly well suited (compared to a simple fast server-side language) in terms of efficiency/performance at compiling large amounts of HTML with data on the fly. Can it be done? Sure. But it's often times going to be overkill or poor performant.
I also have seen MANY examples of how the presentation logic (Javascript) being mixed with the markup is much harder to debug, and it unfortunately couples Javascript CODE with UI presentation, which is almost as bad as coupling the HTML inside your PHP or Python scripts.
So we need separation, agreed.
Why can't there be a simple set of HTML files, with a simple template engine (take your pick!) wrapped around them, on the server, which is ITSELF a *client* of your Data API.
The template engine can request of the data application the data, in JSON format (I hate ugly XML), and do simple combinations into the mostly pure HTML templates, and deliver these to the server.
So, there's 3 pieces... the CLIENT (browser, javascript, ux logic, etc), the UI presentation template engine (something lean and mean and with NO business logic), and the data application.
That way there's no strong coupling but we still let the server do things it's better/faster at doing.
And, as a benefit, if necessary, the Javascript in the client *CAN* bypass the templating engine and make Ajax calls to the same Data API as necessary.
All this concept is actually the central thesis to the topic of proper UI architecture, which I'm very passionate about of late. I'm hoping to talk about it at SXSW (hint, go vote: http://bit.ly/LWyig) and maybe even write more about it in book or magazine format.
In any case, I applaud you for calling this out to the community. It's so important!
Kyle,
At the end of Peter Svensson's Google Tech Talk that he linked above, someone made the same suggestion you are making about having a client rendering process on the server that talks to the data process also on the server. It seems to me that traditional MVC server-side frameworks do this already and probably more efficiently: The model and controller layers prepare data for the view layer. There is no need for serialization and de-serialization of JSON or some other data interchange format as the data is not being moved between server processes.
This is how I've been developing large internal web applications for a long time now. I've used client side XSLT on some projects and on a couple recent projects used JST from Google's Trimpath. I haven't tried using the full Trimpath project, but with a couple hundred lines of javascript I get the JST templating facility which is much easier for other developers to work with than XSLT. In the current app we are doing less templating as we are using jQuery plugins so we more or less hand over JSON data to the plugins. In my opinion, all client code and markup should be received as static files and the only dynamic content should be the data. This way your markup can be compressed statically, cached and not sent to the client over and over again. A good approach for web front ends to databases in order to get very responsive applications.
I really do like this kind of architecture, but doesn't it make your web app virtually un-indexable by google or other search engines? Or is the idea to have a more traditional, page oriented parent site from which you launch such an app?
rationalthug,
Your point is a good one. The system described above can have traditional content in a static HTML page. The extreme example of a page with just a script tag would not be the only way to make one of the static HTML pages. Also, not all pages are intended to deliver indexable content. Some pages are behind login. Some pages are applications rather than information resources.
@Peter-
Actually, most server-side MVC frameworks send a rich M, a model, which is a data structure object representation of the data, to the view. This data structure usually has not only data but methods for accessing/setting the values of the data.
The difference between this and passing a simple JSON object to the view is clearly efficiency. If all you provide to the view is simple object representation without all the overhead of accessor functions, you reduce by quite a bit the amount of serialization effort the M has to go through before handing off the the V.
Almost all server side languages have native support for JSON serialization/deserialization, but almost none will have native support for whatever custom data M structure your application defines. So, in general, the JSON will win out in speed and efficiency.
Also, a lot of server side frameworks are interpreted languages like python or php, which means the amount of object instantiation and abstraction *does* directly affect the memory and speed performance.
The other HUGE benefit to using JSON as the interchange is that JSON is, nearly universally, considered the best representation for data destined for (or coming from!) the browser, since it's natively interpreted by the JSVM and transfers over the wire with little overhead.
So, if your data API might, as I suggested, also have some Ajax end points on it, then it doesn't need to first pass through the custom M data structure and then to JSON to go to the browser... it's straight to the JSON and ready to ship out.
Or, if you need to include the JSON data from the server API along with your view code, such as embedded in the HEAD of a HTML file for instance, you can do that easily, again without any conversions of data.
Hi Peter,
I made a frmework based on the concept a couple of years ago, and it proved to be a valid approach. Check the link above to see a basic page of it at work. BTW, the code of it has not changed for more than a year, and still all browsers (to my knowledge) can handle the asynchronous technique.
BTW sorry my blog and page are down, I have to repair a server here. So unfortunately a glimpse of the working code is all I can provide at this time.
Greetings, Frankie
As to the google indexing: the concept allows for even better (e.g. faster and more precise) indexing, provided you code all relevant visible parts of a page server side also and leave out the irrelevant parts which are then loaded by the JS bootstrap procedure. That is, all indexable visible objects must have a serverside creation algorithm. This is only possible if you use an absolutely strict MCV pattern that works server side also using the same templates as the client side otherwise you completely lose control of the project. In effect the web client user gets a fullAJAX app - google sees whats necessary, not a bit more.
Hi Peter
I agree about not using textual HTML to transport your app from the server into the client.
That's just a "picture" of an application. It requires a side order of javascript as the "description" of the application to then get into it, and liven it up.
IMNSHO, it's pointless to send a picture AND a description.
The application can be fully described in javascript. The best of the latest javascript UI frameworks allow this. A full semantic description of the application can be encapsulated in JSON. The framework updates the UI dynamically based on this.
There's no place for sprawling, unpredictable, textual HTML in modern web apps.
I don't get Ajaxian's point about View Source. What do application consumers care about that? Nothing whatsoever.
And the transported JSON IS semantic markup. It's just more COMPLETE than HTML.
Peter - I couldn't agree more!
check out APPLEBOX (applebox.com.au .. hit the browse DVDs button) I went the full in-browser javascript mvc with a stateless soap back end. This is much more than a SPA - the controller manages multiple iframes and it switches the z-Order to display the current ui. State is maintained between UIs .. no reloading extensive js libs and ui widgets per page navigation. 2 years later it's proved itself in terms of performance, maintenance and usability. This paradigm is rock solid.
Complete separation of concerns .. I can only encourage people to experiment with it. What's missing? A uniform implementation of j-mvc for the browser. IMO the major toolkits (dojo/yui/ext/jquery) need to add this to their offerings for people to experiment with. Try and develop from scratch and you loose productivity as you get bogged down with mvc mechanics. Try and do it separately with sproutcore, javascriptMVC and you loose the penetration and visibility the major javascript frameworks already have.
Forget the architecture discussions .. a vs b vs c can be simply a matter of taste. But create an app that gives a superior user experience ... there is no argument. With JavaScript MVC we can take browser apps to the next level.
It is great to read thoughts and experiences by developers who have also tried the architecture.
This is a very interesting idea. The standardistas would lose it completely of course, but as you mentioned this is for an application rather than just a web page.
You are right, of course, that the application ought not to be rendering out its view, then sending it down the wire, then manipulating that view. In a perfect world I would agree. In the world we actually live in, however, I think I hybrid approach is sensible. In makes me think of how apple has figured out making apps launch "quickly". The first thing you see when you relaunch an app on the iphone is actually a static image of the apps last state, which is displayed while the app loads. It gives a perception of performance. Which is why sending a full page to be rendered natively by the client is still a good idea. Sure the app is going to rip it apart and replace most of it anyway, but rather than look at a loading bar the user might be happier looking at the app.
Or maybe what I am trying to say is that web browsers are very good at downloading html and rendering it, it seems a shame to get rid of that. Maybe ultimately we would prefer the browser to work the way you described, but today I don't think it does.
Also have a look at the Open Source widget platform we're working on at DreamFace Interactive.
As other readers have mentioned, this is not a viable solution if you want to be SEO. However, still using this model and putting a server-side rendering engine for the client code I think would still make this work perfectly. For example, let's say you have a Java backend, you could easily put JSP, Tea, CF, or Server-side JavaScript. That way you are still using a frontend technology and keeping it separate from the backend business logic, while also allowing the page to render text into the HTML. Of course you would still have normal Ajax and other DHTML stuff to make it look nice.
I don't really understand this SEO thing.
We're talking about totally interactive, data-driven applications here.
How is your accounting application, or your shipping application searched and indexed? It's not!
The idea of directories of static HTML pages sitting there waiting to be rendered which may be searched for "keywords" is old hat. It's for text based "web sites" and that's not what we are talking about.
I don't work on "web sites".
Nige,
I agree. There are multiple kinds of web pages now and they don't need to be built the same way. Some pages are information to be indexed. Other pages are genuine applications that a search engine cannot even access.
I find it surprising that after all of this talk about the server strictly being a restful json api, no one has mentioned CouchDB!
And all the views you create in Couch are written in JavaScript. One language to rule them all.
The only drawbacks that I see at this point are the lack of any authentication, and that it is not relational (which could go either way depending on who you are talking to).
As far as authentication is concerned, you could write a simple server-side gateway between the client and Couch.
Have something to write? Comment on this article.
Interesting idea. I started down this route for an embedded iPhone web app, but we eventually opted for a somewhat more traditional model.
I started with a thin HTML shell, not much different from your extreme example (though I used the shorter HTML5 doctype!) I wrote a whole bunch of JavaScript to make the proper requests, wrote heaps of jsonTemplate code to massage JSON responses into HTML, etc.
In the end, we decided it was simpler to request snippets of HTML to insert into the DOM. This does couple the server- and client-sides more tightly, and it probably makes the server-side guy's life more complicated, but it removed a layer of complexity from my client-side work.
I think it was the right choice for our application. Wrapping up HTML in JavaScript templates didn't smell like the proper separation of concerns. And, that's in the best case scenario (an iPhone-only web application).
While a cool ideal, I think this concept would be an even tougher sell for the open web, mostly because of accessibility concerns. Progressive enhancement is still a good model for traditional websites and most web applications.