The WebApp Wizard Web development made magical


Web development process, from end to end

Just a little post I wanted to write for a long time. Even if it is mainly aimed at beginner web developers, I think it could be interesting for more experienced ones to read on too. I don't pretend to be one of these kick-ass developers we see out there, but just an average one who loves his job, and do it as well as I can, yet not perfectly. So here is a quick honest overview of an average web development process and the tools that come with it, with its advantages and drawbacks.


Maybe one of the most important tools, as we are using it a great part of our time. I used a few of them, from the most simple ones to the most advanced ones. After having used notepad, a few fancy visual editors, a few awesome and simple editors like TextMate and Zend Studio, I think I kind of made up my mind with Aptana.

First, the default theme looks great and has been a real relief for my eyes. At first, I wondered why this dark background was used, I felt it like a regression: we used to type light characters on dark backgrounds a long time ago, and then we started writing with black characters on white background. Just like we do with a pen and paper, that seemed kind of logical to me. But when we think about it, this is just stupid. A dark background doesn't emit as much light as a light one, so it puts really less strain on the eyes. So I'm definitely convinced by this default theme.

Then, even if they are not always perfect, available bundles to help with auto-completion or documentation work quite well, and suit my needs. Plus, there are some little bonuses like Capistrano integration that made me adopt it.

The only thing I regret from Zend Studio is the PHPUnit / Code coverage / Debug integration. That was really great. But Aptana will reintegrate debug support someday, I hope.

And last but not least, it is freely available.

There are also really great editors like TextMate, but honestly, I didn't have the courage yet to learn how to use these properly. Just know that it embeds tons of shortcuts and features that saves you time and effort though.


A key aspect of web development, as many of us know, consists in reducing the number of HTTP request our pages make, and in reducing the weight we transfer over the wire. I became aware of this relatively recently, and since I have spent quite a lot of time trying to resolve this issue. Reducing the number and weight of our requests helps a lot in producing fast responding pages, which is crucial for users with a slow Internet connection, but also counts for users with a good broadband connection.

Combining and minifying our files is good, but we don't want it to slow down or clutter up our development process. Ideally, this should be taken care of automatically. And after trying a few ways to do that minification process more or less manually, I recently stumbled upon the ideal solution for me, which I recently talked about. Assetic is so good not only because it takes care of combining and minifying your files, but also because it allows you to use all sorts of filters like Less, Sass, or virtually anything that could come to your mind. Moreover, it facilitates caching AND leaves no room for out of date cache. But I'll talk about this in the next paragraph.


Caching is also very important. I can't make my mind about this question: what is the most important part? Minifying or caching? After all, if our caching is right, our non-minification will only hurt us the first time the user hits the page. Once all the files are in the cache, magnification doesn't matter anymore.

So I consider caching at the same level of importance as minification. Once again, Assetic helped me a lot with this. But whatever the tool you use, the key is the key. Nope, this sentence has no mistake in it.

The most basic cache associates a key with a content, and updates the content related to a key, when necessary. The problem with this is that the client can't know directly if its cache is up to date or not. It has to ask the server "hey, is this cache still good or do I need to update it?". If it doesn't do this, it can lead to out of date cache entries, which can be more or less of an issue, depending on the context. So we can't have the best performance and best reliability with this kind of cache.

Another approach to this is to update not the value of a cache entry, but rather the key. When the client finds a key (filename) it doesn't know about, it will be forced to ask the content to the server. As long as the content doesn't change, the key won't change. But as soon as the content is modified, a new key/value pair is created, leaving the old key/value pair or deleting it. One one hand, we have a single key with multiple values over time, and on the other hand, we have a multiple keys over time, with only one value each. So the evolution of the cache is not a problem anymore. In other words, you have to version your filenames. You can do it manually, or you can use a tool like Assetic take care of this for you. It allows you to always serve fresh content, with maximum caching capabilities as the client doesn't have to ask if the cached entry is ok or not. Be careful though, adding a version number in the query string isn't always a good idea, as some proxies rely only on the filename to determine whether to download the file or serve it from its cache. So the best option is to change the filename itself.


The last important step is how you deploy your app. Like many people, I started with some basic FTP upload, but as soon as you start working on more serious applications, you probably need something more reliable and more automated. That's why I began writing deployment scripts to help me out. The main problem when we do this manually is we are humans, so we are prone to error and to forget things. How many times did I have to put the server configuration file back in place, as it was erased by my development one which I just committed by mistake? I don't know, nut what I know for sure is "too many times". A script is more reliable for this, but, when this script is written by one human, it is also prone to error. The difference is that once the error is spotted and corrected, it is for good.

But we can do better: a deployment script that is written and used by many humans, therefore reducing the error risk. Not to mention this script will also probably have more features, which can be good too. That's the case with Capistrano, a really great deployment tool I started using last year. It not only takes care of deploying an app from a repository, it also versions it and supports a nice rollback feature in case something went wrong. Another really nice thing is that it allows you to store your users files outside of your code, and create symlinks automatically to ensure everything works as if the files where right under your code's tree.

To sum up

This was a quick post despite its length, and it covers a few topics just on the surface, but the aim was not to dive too deeply in these. It is more meant to give a few leads to follow and make your own opinion about what tools to use or not to use. I could also have talked about testing, but I am still not using this as a real part of my development process, sadly. I tend to use them more and more, but I don't think I am ready to really talk about this right now. A lot of other people will do this way better than me out there.

I just wanted to share the principle I run with, hoping it will help somebody, as I know I've searched for a long time before finding my way of doing these things.

Happy coding!


Web performance : further optimization

If you use tools like YSlow, PageSpeed and WebPageTest, you have probably already gone a long way about web performance.

The problem

Working on a website which had already good YSlow / PageSpeed ratings, I just wanted to push a little further: can I go up to 100/100 or get very very close to it? That may seem a bit pointless, I mean : who is going to be able to tell the difference? Will it make any difference for the server, too? Well, I don't know, but I wanted to try this for fun. Yeah, strange kind of fun.

So I looked at the metrics of my favorite tools, and PageSpeed told me something: maybe you should try to inline these scripts. What? Am I not supposed to make my scripts external (rule 8 of my bible)? In fact, not always. Making an extra HTTP request is contrary to rule 1, after all. A small file is often not worth a request. So we'd better make it inline, right into the page, to avoid unnecessary HTTP overhead.

But hey, I don't want to sacrifice my cleanly organized JS folder just for the sake of performance. So I had to come up with something that would inline my scripts/css when necessary, without me having to copy and paste the contents of said resources. More importantly, I want it to be dynamic: maybe my files will grow large enough to be worth an extra HTTP request again. So there is no way I manage this by hand.

The solution

Working with Smarty on this project, I decided to make a little Smarty plugin that would help me doing this. The idea is, based on a file size limit, to include scripts the "normal" way or to inline them.

I came up with two little plugins, one for JS files, the other for CSS files.

The results

Using these plugins resulted in one tiny script (a few hundred bytes) and one CSS, on some pages, to be included inline. To be perfectly honest, I didn't measure if there were any "real" performance improvement, and I don't know if it had such a big impact on performance from the user point of view. But it is obvious that this really tiny javascript file generated more HTTP overhead than its content, which is ridiculous. So inlining it can't be bad for performance either.

I was quite surprised by the results from the metrics point of view, though. My YSlow score jumped from 92 - 93 up to 99! Now, that's what I'm talking about: a pretty solid A-grade score :-) . I didn't expect much gain on the YSlow side, as it doesn't mention anything about inlining your scripts. I was even expecting a slightly lower score as YSlow tells you to make your scripts external. But it seems that it doesn't only rely on some stupid rules of thumb, but rather also on real performance.

99 YSlow A-grade score

The PageSpeed score also jumped from something around 91 to 98, which is less of a surprise, as I just applied its recommendations.

What about the server?

That's nice, but I still have a doubt about overall performance, or, more accurately, server charge. That's not really a problem in my case, as I don't have thousands of simultaneous users, so my server can take a little extra charge, but: looking at my plugins implementation, I wonder if this couldn't be optimized. Each time it is used, it checks the file size to decide whether to inline it or not. I don't know if it's a heavy operation, if there is some kind of cache somewhere in the system that avoids to make disk access each time, etc. And when it decides to inline it, it reads the file contents and writes it in the page. And neither do I know precisely how heavy this is.

Anyway, as I told, that's not much of an issue for me, so the overall performance isn't affected. And that's easy to understand: it's quite easy to shave off 100 from the front end (any HTTP request easily takes that much), but what is 100 ms on the back end? That's a whole lot of time. 100ms of PHP execution (or Ruby, or Python, or Java, or C) is huge: most operations won't take more than a few ms. So I think it's pretty safe to say that avoiding unnecessary HTTP traffic is worth a little extra work on the server. And that's the whole thing! I see people working hard optimizing their server code, just to save 3ms here and there. On the server side, that may be important if you have tons of simultaneous connections, but the user won't even notice. When you start working just a little on front end optimization, you save milliseconds by packs of 500!


So, how could I get up to 100/100 on YSlow (and maybe PageSpeed)? Well, if I look at YSlow output, I see this:

Google Analytics preventing 100 YSlow A-grade score

Google Analytics script is not allowing me to get the holy 100, just because it doesn't set a far-future expiration date, thus making it difficult to cache for the browser. I don't know if there is any way to fix this, and I would be glad to hear there is one. I'm pretty sure that would be a great enhancement for the user, as this script download is not that fast.


Assets management with Assetic (and Smarty)

Being quite a web performance geek, I came across various solutions concerning JS and CSS resources management. But this time, I think I found the right one.

At the very beginning, like everyone else, I didn't care much about having 10 javascript and 10 css files included in my web pages. But as my users met performance and strong bandwidth constraints, I started digging into how to improve loading speeds. And then, I fell into web perfs, reading this book. I started applying advice by hand, and as I saw and felt a real improvement, even on my high speed Internet connection, I wanted more. I began making my own automation solutions, and it did quite well the job, even if it was still a bit raw and not so easy to use. But at this time, I used to work alone on my project. As another developer now joins me from time to time, I needed to make minification, compression, caching, and all the rest as painless as possible. My main concern was, at first, to provide an easy way to switch between production and development mode (minified or not), without having to launch any particular command or generating any file. A simple boolean in my configuration had to do the trick. That was an absolute requirement for me.

Then I discovered Assetic thanks to @tijuan_. Assetic is a Symfony2 component, which I tried to use on my non-Symfony project.

And there it is. The thing that I always dreamed about. It allows you to combine and minify the files you want in one single file, and it has several filters you can apply to your assets, like YUI compressor, LESS/SASS parser, Google Closure compiler, among others.

The only problem was: how to integrate this thing neatly? As I use Smarty, some custom block plugin would have been nice. So I made it. The idea is that you tell which files you want to include, which filters you want to apply to them, and whether you're in debug configuration or not. The plugin then takes care of combining and minifying your files if not in debug mode, generating a single file which will be re-generated with a different name if any change is made on the source files. That means you can choose a very aggressive caching strategy without any trouble. The first modification you will make in your source code will result in generating a new file, so the browser will have no other choice than downloading it. You already use some fake parameter in your URLs to avoid caching issues? Not a bad idea, but still not perfect: some proxies will ignore URL query strings. The only 100% reliable option is to generate a totally new file name, which Assetic does.

Just a little example of usage:

    <link rel="stylesheet" href="{$asset_url}">

Cherry on the cake: it allows you to keep track of dependencies between your files. You don't have to explicitly include the libs you need, assetic-smarty takes care of it, keeping your "files to include" list clean and uncluttered. Don't worry, it won't include the same file twice.

Enough chat, I strongly encourage you to discover Assetic and the associated Smarty plugin.