Designing a message processing system

I have been asked to create a message processing system as following. As I am not sure if this is the right place to post this, feel free to move it to any other appropriate SC group.

Problem

Server have about 100 to 500 clients connected at every moment. When a client connects to server, server loads part of their data and cache it in memory for faster access. Server will receive between 200~1000 messages per second for all clients. These messages are relatively small (about 500 bytes). Any changes to data in cache should be saved to disk as soon as possible. When client disconnects all their data is saved to disk and removed from cache. each message contains some instruction and a text message which will be saved as file. Instructions should be executed as fast as possible (near instant) and all clients using that file should get the update. Only writing the modified message to disk can be delayed.

Here is my solution in a diagram

My solution consists of a web server (http or socket) a message queue and two or more instances of file server and instruction server.

• Web server grabs client messages and if there is message available for client in message queue, pushes it back to client.


• Instruction processor grabs instructions from queue and creates necessary message to be processed by file server (Get/set file) and waits for the file to be available in queue and more process to create another message for client.


• File server only provides the files, either from cache or physical file depending on type of file.

Concerns:

• There are peak times that total connected clients might go over 10000 at once and total messages received from clients increase to 10~15K. I should be able to clear the queue and go back to normal state as soon as possible (with processing requests obviously).


• I should be able to add extra instruction processors and file servers on the fly without having to shut down the other instances.


• In case file server crashes it shouldn’t lose files so it has to write files to disk as soon as there are any changes and process time is available.


• File system should be in b+ tree format so some applications (local reporting apps) could easily access files without having to go through queue server

My Solution

I am thinking of using node.js for socket/web server. And may be a NoSQL database for file server and a queue server such as rabbitMQ or Node_Redis and Redis.

Questions: - Is there a better way of structuring this system? - What are my other options for components of this system? - is it possible to run all the instances in same server machine or even in same application (in different thread)?