[NODEJS] Socket.io event triggered

[ste2425]

Im adding an interpretable file transfer between two NodeJS instances for our application at work.

The laptops connects to the VPN. Currently the trouble were having is it tries to sync up a rather large zip in one go and if the laptop looses WiFi the sync fails. Both the Server and laptop Node instances use socket.io for their portals so im going to chop the file up into chunks, use socket.io to send the chunks and re build it on the server. Then if the WiFi drops the server just drops the chunk if its not complete and asks for it again when the laptop reconnects.

Now my question is do the socket.io events fire only when all data is received? Im using it at the moment to send very small JSON objects the data that will be sent will be up-to 5 meg so much larger. How does socket.io handle situations where the connection is lost before all the data is received on a pending event? Will it drop it and clean up or will it hold on to it?

Hope that all makes sense. Thanks guys n gals.

 

[Ross211]

According to some info on stackoverflow the data will be lossed if the connection is dropped (
NodeJS + Socket.io connections dropping/reconnecting? &
SocketIO + Node - Recovering Lost Connections)

I'm betting socket.io provides a way to fire callbacks during a transfer, indicating how many bytes have been transferred and how many are left. Check out this post here on stackoverflow, it should give a good starting point.

 

[W1zzard]

I would send a small (=quick) packet for the beginning and end of the transfer, including byte count to send and byte count received, so you can check on both ends.

Not sure if you even need socket io (yes i know what it is and have used it before). just regular http post requests fired from js should be fine.

How do you plan on splitting the file? JS has no filesystem access

http://www.w3.org/TR/FileAPI/ with limited browser support

 

[ste2425]

Awesome links thanks.

I'm using socket.io as both the server and client have it. By client I mean the small NodeJS server on the laptop, not a browser so I handle chopping the files up and re-building them on both ends using NodeJS's FileSystem module. I've currently got a working prototype. The Client keeps track of how much has been uploaded. It will hit the server when connected with details on the file to upload and the data-chunk. The server inserts this data into the file specified at the location in the file specified then hits the client back to tell the client it has done it, with info on whether it was successful. Once the client receives this complete message it updates its database on how much has been successfully sent and repeats with the next chunk. This means that if the connection is lost anywhere in that process before the complete message is sent it doesn't matter. Even if that data has been successfully added to the file as it will be resent the next time the client connects as it didn't receive a successful complete message back. This works quite well with me rather violently pulling my Ethernet cable during the process and at the end i still get a complete, uncorrupted zip file.

 

[W1zzard]

So you run nodeJS on both ends of your connection? Which means you could run any other custom software for the transfer?

rsync --partial + automatic restart when rsync dies wont work for you ?

 

[ste2425]

Hard for something to work for you when you don't realise it exists.

Judging by https://github.com/mattijs/node-rsync rsync is some built in command(Assuming the linked module is a wrapper for it)? Ill have to look into it a bit more. Well that was a day wasted.

EDIT: One system in windows based the other linux, will rsync still be a possible solution? Still have to look into it. What if the laptop goes to sleep half way through? The current solution will successfully continue, could rsync?

 

[W1zzard]

There is rsync for win32. rsync will continue when returning from sleep. Yes it's a command line tool on Linux