proper way to monitor/control a server remotely over http in realtime

On my client (a phone with a browser) i want to see the stats of the server CPU,RAM & HDD and gather info from various logs.

Atm i'm using ajax polling.

On the client every 5 sec (setInterval) i call a php file:

1. 
   

   scan a folder containing N logs

   
   


2. 
   

   read the last line of each log

   
   


3. 
   

   convert that to json

   
   

problems:

1. open new connection every 5 sec


2. multiple ajax calls


3. request headers (they are also data and so conume bandwich)


4. response headers (^)


5. use php to read files every 5 sec even if nothing changed...

the final json data is prolly less than 5kb, but i send it every 5 sec, and there are the headers and new connection everytime. so basically every 5 i have to send 5-10kb to get 5kb wich are 10-20kb...

those are 60sec / 5sec = 12 new connections per minute and about 15mb per hour of traffic if i leave the app open.

lets say i have 100 users that i let monitor / control my server that would be araund 1.5gb outgoing traffic in one hour.

not to mention that the php server is reading multiple files 100 times every 5 sec.

I need something that on the server reads the last lines of those logs every 5 sec and maybe writes them to a file.Then i want to push this data to the client only if it's changed.

---

SSE (server sent events) with php

header('Content-Type: text/event-stream');
header('Cache-Control: no-cache');
while(true){
 echo "id: ".time()."\ndata: ".ReadTheLogs()."\n\n";
 ob_flush();
 flush();
 sleep(1);
}

in this case after the connection is established with the first user the connection keeps open (php is not made for that) and so i save some space (request headers,response headers). this work on my server bu most server don't allow to keep the connection open for long time. Also with multiple users i read the log multiple times.(slowing down my old server) And i can't control the server ... i would need to use ajax to send a command...

i need WebSockets!!!

---

node.js and websockets

using node.js, from what i understand, i can do all this without consuming alot of resources and bandwich. The connection keeps open so no unnecessary headers, i can recieve and send data.it handles multiple users very well.

And this is where i need your help.

1. 
   

   the node.js server should in background update, and store the logs data every 5 sec if the files are modified.OR should that do the operating system with (iwatch,dnotify...)

   
   


2. 
   

   the data should be pushed only if changed.

   
   


3. 
   

   the reading of the logs should be happen only one time after 5 sec ... so not triggered by each user.

   
   

---

this is the first example i have found.....and modified..

var ws=require("nodejs-websocket");
var server=ws.createServer(function(conn){

    var data=read(whereToStoreTheLogs);
    conn.sendText(data)// send the logs data to the user 
    //on first connection.

    setTimeout(checkLogs,5000);
    /*
     here i need to continuosly check if the logs are changed.
     but if i use setInterval(checkLogs,5000) or setTimeout
     every user invokes a new timer and so having lots of timers on the server
     can i do that in background?         
    */
    conn.on("text",function(str){
     doStuff(str); // various commands to control the server.
    })
    conn.on("close",function(code,reason){
     console.log("Connection closed")
    })
}).listen(8001);

var checkLogs=function(){
 var data=read(whereToStoreTheLogs);
 if(data!=oldData){
  conn.sendText(data)
 }
 setTimeout(checkLogs,5000);
}

the above script would be the notification server, but i also need to find a solution to store somwhere the info of those multiple logs and do that everytime something is changed, in the background.

How would you do to keep the bandwich low but also the server resources.

How would you do?

EDIT Btw. is there a way to stream this data simultaneosly to all the clinets?

EDIT

About the logs: i also want to be able to scale the time dilatation between updates... i mean if i read the logs of ffmpeg i ned the update every sec if possible... but when no conversion is active.. i need to get the basic machine info every 5min maybe ... and so on...

GOALS: 1. performant way to read & store somewhere the logs data (only if clinets connected...[mysql,file, it's possible to store this info inside the ram(with node.js??)]). 2. performant way to stream the data to the various clients (simultanously). 3. be able to send commands to the server.. (bidirectional) 4. using web languages (js,php...), lunix commands( something that is easy to implement on multiple machines).. free software if needed.

best approach would be:

read the logs, based on current activity, to the system memory and stream simultaneously and continuosly, with an already open connection, to the various clients with webSockets.

i'don't know anything that could be faster.