Streaming File Uploads to Storage Server with Node.js

In this tutorial, We will show how to stream a file uploaded by a user to a file storage server such as Amazon S3 or any other custom server without storing the file in a temporary directory. The advantage of this is that in case there is a lot of parallel uploads of large files then it prevents your filesystem from hanging. We also won’t be storing the file in primary memory, therefore, we are also preventing memory getting hanged.

We don’t let the user to directly upload files to storage server when we want to verify user authentication and also modify or scan the files.

For this tutorial, we are going to use the latest stable version of Express framework. Since express version 4, express doesn’t inherit connect module anymore. In earlier versions, express used connect for middleware and route definition functionalities and was only responsible for rendering views and some other functionalities.

Now Express 4 has its own middleware and route definition functionality. Express 4 now directly inherits from built-in HTTP module of Node.js. Earlier Express inherited connect and connect inherited the HTTP module.

Parsing “multipart/form-data” Content-Type

Express doesn’t parse the body of POST requests. Earlier versions of express used the body-parser module that came built-in with connect to parse the body of POST requests. But now body-parser needs to be installed separately because express doesn’t use connect anymore.

But the problem is body-parser doesn’t parse HTTP POST request body if its type is multipart/form-data. Therefore body-parser is useless when you are using multipart/form-data to upload files.

So we need to use multiparty or connect-multiparty modules to parse POST request body if Content-Type is multipart/form-data.

connect-multiparty stores the uploaded file in a temporary directory. Whereas multiparty also provides readable stream objects for every uploaded file to read their content. Actually, connect-multiparty is built on top of multiparty. So for our use, we need the multiparty module.

Streaming Uploaded File

Here is the Node.js code for our use:

var app = require("express")();
var multiparty = require("multiparty");"/submit", function(httpRequest, httpResponse, next){
var form = new multiparty.Form();
form.on("part", function(part){
var FormData = require("form-data");
var request = require("request")
var form = new FormData();
form.append("thumbnail", part, {filename: part.filename,contentType: part["content-type"]});
var r ="http://localhost:7070/store", { "headers": {"transfer-encoding": "chunked"} }, function(err, res, body){
r._form = form
form.on("error", function(error){
app.get("/", function(httpRequest, httpResponse, next){
httpResponse.send("<form action="http://localhost:9090/submit" enctype="multipart/form-data" method="post"><input name="thumbnail" type="file" /><input type="submit" value="Submit" /></form>");});

In the above code, the part event is triggered when a file is encountered in the body of the request. the part parameter of the callback is a Readable Stream that reads the content of that particular file for which the part event was triggered.

We are then constructing an HTTP POST request using request module to send the file to the storage server. The second parameter of the form.append method takes a Readable Stream and pipes the data to the writable stream, which continuously sends data to the storage server without saving the data anywhere. Whenever a chunk of data arrives from the client for the file then data event of the readable stream is fired. The buffer that holds the chunk is removed automatically when the data event’s callback is finished executing,

As we don’t know the size of the files, therefore, we are setting transfer-encoding:chunked’ header instead of the content-length header.

To run this code make sure you are running a storage server at URL http://localhost:7070/store.

Note: This is not my own work. I took reference from Narayan Prusty’s article on Qnimate which is licensed under MIT. Shared it with my readers because it might be useful for them.

What do you think?

0 points
Upvote Downvote

Written by Snehasish Nayak

Google Top Contributor (Allo, Duo), a Local Guide, and Founder of TechIncludes. Follow me on twitter @Th3Snehasish.


Leave a Reply

Your email address will not be published. Required fields are marked *





Machine Learning with Google AI

Learn AI with Google, ML education available for everyone

What is Web Crypto API and where can you use it