The WebApp Wizard Web development made magical


Distributed, transparent NodeJS architecture in hostile environment

Hi everyone.

As we're trying to redesign an application at ORU-MiP, we're wondering if something already exists.

Let's settle the context first: it is an application designed for disater, emergency situations. It allows users (health professionals) to simply input victims basic data. There are three main concerns about this application:

  1. Anybody must be able to use it under difficult, extreme circumstances. Just imagine you have a ton of victims coming at you, and you must ask their names, age, etc. and type this as fast as possible in an unknown piece of software...
  2. It must just work. You have absolutely no time to configure anything, just plug your tablet PC / iPad / anything you want on the local network, type an URL in your browser, and you're ready to go.
  3. Maybe the most difficult part: we have absolutely no idea of what we're going to encounter. We cannot rely on one medium for the network part. 3G might not be available (imagine a subway crash). The hardware parts to connect could be too far apart to connect via RJ45. There might be some wave interferences, making any wireless attempt fail. A disaster can happen anywhere, anytime, so we must consider we are going to be in a really unfriendly environment. We're not talking about users in an office, sitting on a chair, in front of a 24 inches screen here.

The last 2 points are very important in this post.

So, what do we want to achieve? It's quite simple:

  1. Any client must be able to connect to the system and use it just by typing an URL in a browser
  2. The clients will have everything stored locally after the first step (hello localStorage, hello cache), and must be able to work independently of the network state.
  3. The server actually just broadcasts the messages it receives to the other connected clients (synchronization messages: create, update and delete).
  4. The server must be hosted on one of the clients (pre-installed). Less hardware to deploy is better.
  5. There might be more than one server one the network, just in case. In fact, every client will also be a server.
  6. If one server falls, another must be able to replace it instantly, with no action from the user.
  7. Everything must be absolutely transparent for the user.
  8. The user must have nothing to do (did I already say that?)

It seems like an HTML5 web app is a perfect fit for our needs. It allows us to combine ease of deployment (just type in an URL) and client-side storage and processing.

It could be really simple: one of the clients also hosts the server, and everything is synchronized by this server. But what if this server fails? What if the tablet PC on which it will be installed runs out of battery? What if the system crashes? What if the tablet PC is destroyed (unlikely, but still possible)? And believe me, these considerations are not here just for fun. It happens, and when it does, the system must continue with no action from the user.

So the idea is to duplicate the server on each client. We can pre-install the machines, unless it's some complicated, time-consuming stuff. The application is currently coded in, and it is a real pain to install/update the framework, install system updates, etc. We want to get rid of this.

Ok, so we have a few machines with the same lightweight server on it. NodeJS and Socket.IO are good choices as it allows to code fast and responsive web apps. The clients would be single-page web apps, so this is OK.

Here is how I plan to deal with the complicated stuff: each client, once connected to the server, would establish a socket connection with every server on the network (we would use some network detection protocol to achieve this). It would promote one of these sockets to the state of "pub" socket. This socket would be used to publish (pub) messages (add, update, delete of victims for instance). The other sockets would be only used to subscribe (sub) to other clients modifications.

So what happens when a client publishes a message on his "pub" socket? The server will broadcast the message to every connected client, which means every client, as every client is connected to every server via their "sub" sockets. Nice!

What happens if the server is down or unreachable? The client has a list of every other server on the network (remember the "sub" sockets). So it is possible to promote a "sub" socket to state of "pub" socket, and use this one instead. The client will then publish his message to another sever, chosen at random. The server will be able to broadcast the message to every other client, as every client is connected to every server.

In the end, after a few network connection issues, or unresponsive server for whatever reason, we might end up with every client connected to a different server, but it's no big deal, as the servers are stateless and only used to brodcast client messages.

Moreover, we could imagine that any client that was not intended to be used in the system could come in and help. Just connect to one of the servers via your browser, and you're ready. It just won't be able to be used as a server, but that's not that important. Somebody is walking by, we could ask him for help, and in a few seconds, he would be able to welcome victims and type in their names, ages, etc.

A little simplified diagram to sum things up (only two client sets of socket connections represented for the sake of readability...):

Distributed NodeJS architecture with automatic failover

In fact, I dream about the client to client communication protocol the W3C is working on. But in the meantime, we need a workaround. And, for the moment, this is the best one I've found. Every other solution I imagined would fail someday under certain circumstances (and remember, we're not talking about traditional web here, we have to consider that everything out there is trying to kill your system, and if it happens, it stacks up another disaster on top of the first one), and if so, would require some user action to continue working.

Technically, it might be pretty simple to implement. But I'm wondering:

  • Is there already some NodeJS framework or module that does the same kind of thing?
  • If not, would you be interested in one?

Thanks for reading this long post.