Upload to S3 Directly or Through Webserver

• Language Support
• Node.js
• Direct to S3 File Uploads in Node.js

This article was contributed past Will Webberley

Will is a computer scientist and is enthused past virtually all aspects of the technology domain. He is specifically interested in mobile and social computing and is currently a researcher in this surface area at Cardiff University.

Direct to S3 File Uploads in Node.js

English language — 日本語に切り替える

Final updated March 09, 2022

Table of Contents

• Uploading straight to S3
• Overview
• Prerequisites
• Initial setup
• Direct uploading
• Running the app
• Summary

Web applications often require the power to allow users to upload files such every bit images, movies and archives. Amazon S3 is a pop and reliable storage choice for these files.

This commodity demonstrates how to create a Node.js application that uploads files directly to S3 instead of via a web awarding, utilising S3'due south Cantankerous-Origin Resources Sharing (CORS) support. The Express web framework is used to facilitate request-treatment in the examples beneath, simply the procedure should be near identical in any Node.js application.

Uploading straight to S3

A complete example of the lawmaking discussed in this article is available for directly utilise in this GitHub repository.

The chief advantage of direct uploading is that the load on your application's dynos would exist considerably reduced. Using app-side processes for receiving files and transferring to S3 can needlessly tie up your dynos and will mean that they will not be able to respond to simultaneous spider web requests as efficiently.

The application uses customer-side and app-side JavaScript for signing the requests. The actual upload is carried out asynchronously so that you can decide how to handle your application's catamenia after the upload has completed (for example, you may wish to redirect users to another page upon successful upload rather than a full folio refresh).

An example unproblematic account-editing scenario is used as a guide for completing the various steps required to accomplish the direct upload and to chronicle the application of this to a wider range of use-cases. More data on this scenario is provided subsequently.

Overview

S3 is comprised of a set of buckets, each with a globally unique proper name, in which individual files (known as objects) and directories, tin be stored.

For uploading files to S3, yous will need an Access Key ID and a Secret Access Key, which act as a username and password. The access key business relationship will demand to have sufficient access privileges to the target saucepan in society for the upload to be successful.

Please see the S3 Article for more data on this, creating buckets and handling your authentication keys.

In full general, the method described in this article follows these simple steps:

• A file is selected for upload by the user in their web browser;
• The user'south browser makes a request to your web application on Heroku, which produces a temporary signature with which to sign the upload request;
• The temporary signed request is returned to the browser in JSON format;
• The browser then uploads the file directly to Amazon S3 using the signed request supplied by your Node.js application.

This guide includes information on how to implement the client-side and app-side code to form the complete organisation. After post-obit the guide, y'all should accept a working barebones system, allowing your users to upload files to S3. Still, it is usually worth adding extra functionality to assist improve the security of the system and to tailor information technology for your own particular uses. Pointers for this are mentioned in the appropriate parts of the guide.

The signature generation on the server uses AWS's official SDK, as explained later. Please see their documentation for information on the features of this SDK.

Prerequisites

• The Heroku CLI has been installed;
• Node.js has been installed;
• A Heroku application has been created for the current project;
• An AWS S3 bucket has been created;
• You have AWS authentication keys that have write admission to the bucket.

Initial setup

S3 setup

You will now demand to edit some of the permissions backdrop of the target S3 bucket and so that the final asking has sufficient privileges to write to the bucket. In a web-browser, sign in to the AWS console and select the S3 section. Select the appropriate bucket and click the Permissions tab. A few options are now provided on this page (including Block public access, Access Control List, Saucepan Policy, and CORS configuration).

Firstly, ensure that "Block all public access" is turned off, and in item turn off "Cake public access to buckets and objects granted through new access control lists" and "Cake public access to buckets and objects granted through whatever access command lists" for the purposes of this project. Setting up the saucepan in this way allows u.s. to read its contents without signed URLs, but this may not be suitable for services running in production.

Next, you will need to configure the bucket's CORS (Cross-Origin Resource Sharing) settings, which will permit your application to access content in the S3 bucket. Each rule should specify a set up of domains from which access to the bucket is granted and also the methods and headers permitted from those domains.

For this to work in your application, click 'Edit' and enter the following JSON for the saucepan'south CORS settings:

            [     {         "AllowedHeaders": [             "*"         ],         "AllowedMethods": [             "Go",             "HEAD",             "POST",             "PUT"         ],         "AllowedOrigins": [             "*"         ],         "ExposeHeaders": []     } ]                      

Click 'Salve changes' and close the editor.

This tells S3 to let any domain access to the bucket and that requests tin can contain whatsoever headers, which is mostly fine for testing. When deploying, you should change the 'AllowedOrigin' to but take requests from your domain.

If y'all wish to use S3 credentials specifically for this application, and then more than keys can be generated in the AWS business relationship pages. This provides further security, since yous tin can designate a very specific set of requests that this fix of keys are able to perform. If this is preferable to y'all, and so you will demand to configure your IAM users in the Edit bucket policy selection in your S3 bucket. There are various guides on AWS's web pages detailing how this can be achieved.

App setup

If your app hasn't withal been setup, and then it is useful to exercise and so at this stage. To go started, create a directory somewhere on your local machine:

            $ mkdir NodeDirectUploader                      

At present create two further subdirectories of NodeDirectUploader/ to respectively contain your HTML pages and support files:

            $ cd NodeDirectUploader $ mkdir views $ mkdir public                      

Node's parcel manager, npm, should have been installed past default along with Node and can be used to handle the installation and updates of the required packages for your app. To begin this, run Node'due south interactive package setup tool in the root of your app directory:

            $ npm init                      

The tool will enquire some questions near your app, including its name, clarification, licensing, and version-control, and create a file called parcel.json in the app's root. This file uses your responses to maintain information about your app, which you tin can edit freehand as you develop farther.

The same file can be used to easily declare your app'southward dependencies, which volition facilitate the deployment and share-ability of your app. To do and then, edit parcel.json and add a "dependencies" JSON object to comprise the following package dependencies:

            {   "name": "NodeDirectUploader",   "version": "0.0.1",    ...    "dependencies": {     "aws-sdk": "2.10",     "ejs": "ii.x",     "express": "four.10"   } }                      

These dependencies can and so be installed using npm:

            $ npm install                      

Use of these packages volition become clear subsequently, and installation of them in this way allows for greater command of your per-app dependencies as your apps abound.

Heroku setup

In club for your awarding to admission the AWS credentials for signing upload requests, they will need to be added as configuration variables in Heroku:

            $ heroku config:gear up AWS_ACCESS_KEY_ID=xxx AWS_SECRET_ACCESS_KEY=yyy Adding config vars and restarting app... done, v21     AWS_ACCESS_KEY_ID => xxx     AWS_SECRET_ACCESS_KEY => yyy                      

In addition to the AWS access credentials, set your target S3 bucket's name:

            $ heroku config:set S3_BUCKET=zzz Adding config vars and restarting app... washed, v21     S3_BUCKET     => zzz                      

Using config vars is preferable over configuration files for security reasons. Avert placing passwords and access keys directly in your awarding's lawmaking or in configuration files. Please run into the article Configuration and Config Vars for more data.

Setting upwardly local environment variables for your app is useful for running and testing your app locally. For more information, see the Prepare up your local environment variables section of the Heroku Local article. Information on launching your app locally is provided later in this article.

Remember to add the .env file to your .gitignore, since this file should just exist used for local testing.

Direct uploading

The processes and steps required to accomplish a straight upload to S3 will be demonstrated through the use of a elementary profile-editing scenario for the purposes of this commodity. This instance will involve the user being permitted to select an avatar image to upload and enter some basic information to be stored as part of their account.

In this scenario, the following procedure will take place:

• The user is presented with a web page, containing elements encouraging the user to cull an prototype to upload every bit their avatar and to enter a username and their own proper name.
• An element is responsible for maintaining a preview of the called prototype past the user. By default, and if no image is chosen for upload, a default avatar prototype is used instead (making the image-upload effectively optional to the user in this scenario).
• When a user selects an image to be uploaded, the upload to S3 is handled automatically and asynchronously with the process described earlier in this article. The image preview is and then updated with the selected paradigm once the upload is complete and successful.
• The user is and so gratuitous to move on to filling in the rest of the information.
• The user then clicks the "submit" push, which posts the username, proper name and the URL of the uploaded image to the Node awarding to be checked and/or stored. If no image was uploaded by the user earlier the default avatar image URL is posted instead.

Setting up the client-side code

No third-party code is required to complete the implementation on the customer-side.

The HTML and JavaScript tin now be created to handle the file option, obtain the request and signature from your Node application, so finally make the upload request.

Firstly, create a file called account.html in your application's views/ directory and populate the head and other necessary HTML tags accordingly for your application. In the trunk of this HTML file, include a file input and an element that will comprise status updates on the upload progress. In addition to this, create a form to allow the user to enter their username and full name and a hidden input element to hold the URL of the chosen avatar paradigm:

To run into the completed HTML file, delight come across the appropriate code in the companion repository.

            <input type="file" id="file-input"> <p id="status">Please select a file</p> <img id="preview" src="/images/default.png">  <form method="Mail service" action="/salve-details">   <input type="hidden" id="avatar-url" name="avatar-url" value="/images/default.png">   <input blazon="text" name="username" placeholder="Username"><br>   <input type="text" name="full-proper noun" placeholder="Full name"><br><br>   <input type="submit" value="Update profile"> </class>                      

The #preview element initially holds a default avatar image (which would go the user's avatar if a new prototype is non chosen), and the #avatar-url input maintains the electric current URL of the user's chosen avatar image. Both of these are updated past the JavaScript, discussed below, when the user selects a new avatar.

Thus when the user finally clicks the submit button, the URL of the avatar is submitted, along with the username and total name of the user, to your desired endpoint for server-side handling.

The client-side code is responsible for achieving two things:

1. Retrieve a signed request from the app with which the image can be PUT to S3
2. Actually PUT the image to S3 using the signed asking

JavaScript's XMLHttpRequest objects tin be created and used for making asynchronous HTTP requests.

To accomplish this, start create a <script> block and write some code that listens for changes in the file input, once the document has loaded, and starts the upload process.

            (() => {   certificate.getElementById("file-input").onchange = () => {     const files = document.getElementById('file-input').files;     const file = files[0];     if(file == zip){       return alarm('No file selected.');     }     getSignedRequest(file);   }; })();                      

The code also determines the file object itself to be uploaded. If 1 has been selected properly, it gain to call a part to obtain a signed PUT request for the file. Next, therefore, write a function that accepts the file object and retrieves an appropriate signed request for it from the app.

            function getSignedRequest(file){   const xhr = new XMLHttpRequest();   xhr.open('Get', `/sign-s3?file-proper name=${file.proper name}&file-type=${file.blazon}`);   xhr.onreadystatechange = () => {     if(xhr.readyState === 4){       if(xhr.status === 200){         const response = JSON.parse(xhr.responseText);         uploadFile(file, response.signedRequest, response.url);       }       else{         alarm('Could non get signed URL.');       }     }   };   xhr.transport(); }                      

If the name (file.name) and/or mime type (file.blazon) of the file y'all upload contains special characters (such as spaces), then they should be encoded first (e.thousand. encodeURIComponent(file.name)).

The in a higher place function passes the file'south name and mime type every bit parameters to the Get request since these are needed in the construction of the signed request, as volition exist covered later in this article. If the retrieval of the signed asking was successful, the office continues by calling a function to upload the bodily file:

            function uploadFile(file, signedRequest, url){   const xhr = new XMLHttpRequest();   xhr.open('PUT', signedRequest);   xhr.onreadystatechange = () => {     if(xhr.readyState === 4){       if(xhr.status === 200){         certificate.getElementById('preview').src = url;         document.getElementById('avatar-url').value = url;       }       else{         alert('Could not upload file.');       }     }   };   xhr.send(file); }                      

This function accepts the file to be uploaded, the signed request, and generated URL representing the eventual retrieval URL of the avatar image. The latter two arguments will be returned equally part of the response from the app. The part, if the request to S3 is successful, then updates the preview element to the new avatar epitome and stores the URL in the hidden input so that information technology can be submitted for storage in the app.

Now, one time the user has completed the rest of the form and clicked submit, the name, username, and avatar image tin can all exist posted to the same endpoint.

If yous find that the page isn't working as you intend after implementing the system, then consider using console.log() to record whatsoever errors that are revealed inside the onreadystatechange function and employ your browser'southward error console to assist diagnose the problem.

Information technology is adept practice to inform the user of any prolonged action in any form of awarding (web- or device-based) and to display updates on changes. Therefore a loading indicator could be displayed between selecting a file and the upload being completed. Without this sort of information, users may doubtable that the page has crashed, and could try to refresh the folio or otherwise disrupt the upload process.

Setting up the app-side Node code

This section discusses the use of Node.js for generating a temporary signature with which the upload asking can be signed. This temporary signature uses AWS authentication credentials (the admission fundamental and secret fundamental) as a basis for the signature, but users will not have direct admission to this data. Afterwards the signature has expired, then upload requests with the same signature will not be successful.

To encounter the completed Node file, please see the appropriate code in the companion repository.

Start by creating your main application file, app.js, in the root of your application directory and set up your skeleton application appropriately:

            const express = crave('express'); const aws = require('aws-sdk');  const app = limited(); app.gear up('views', './views'); app.use(express.static('./public')); app.engine('html', require('ejs').renderFile); app.listen(process.env.PORT || 3000);  const S3_BUCKET = process.env.S3_BUCKET;                      

In some scenarios, it may be necessary to check that the environment'due south PORT var is a number by using Number(process.env.PORT).

The packages installed with npm are imported at the top of the awarding. The Express app is then prepare-up and finally the bucket name is loaded from the environment.

You lot should now configure your AWS region. To do and so, update the imported aws object. For example:

            aws.config.region = 'eu-west-ane';                      

Remember to use the region that your target bucket resides in. If you need it, employ this folio to discover your region.

Next, in the same file, you will demand to create the views responsible for returning the right information back to the user's browser when requests are made to diverse URLs. Inside the app.js file, define the view for requests to /account to return the page account.html, which contains the form for the user to consummate:

            app.become('/account', (req, res) => res.render('account.html'));                      

Now create the view, in the same JavaScript file, that is responsible for generating and returning the signature with which the client-side JavaScript can upload the image. This is the commencement request made by the client before attempting an upload to S3. This view responds with requests to /sign-s3:

            app.get('/sign-s3', (req, res) => {   const s3 = new aws.S3();   const fileName = req.query['file-name'];   const fileType = req.query['file-type'];   const s3Params = {     Saucepan: S3_BUCKET,     Key: fileName,     Expires: sixty,     ContentType: fileType,     ACL: 'public-read'   };    s3.getSignedUrl('putObject', s3Params, (err, information) => {     if(err){       console.log(err);       render res.end();     }     const returnData = {       signedRequest: data,       url: `https://${S3_BUCKET}.s3.amazonaws.com/${fileName}`     };     res.write(JSON.stringify(returnData));     res.end();   }); });                      

This code uses the aws-sdk module to create a signed URL that the browser can employ to brand a PUT request to S3. In addition, the prospective URL of the object to be uploaded is produced as a combination of the S3 saucepan name and the object proper name. This URL and the signed asking are then returned to the browser in JSON format.

The Expires parameter describes the number of seconds for which the signed URL volition be valid for. In some circumstances, such as when uploading big files, a larger value may exist more advisable in order to extend the validity of the signed URL.

Initialising the s3 object automatically loads the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY variables that were set into the surroundings earlier.

You may wish to assign another, customised proper name to the object instead of using the i that the file is already named with, which is useful for preventing accidental overwrites in the S3 saucepan. This name could be related to the ID of the user's business relationship, for instance. If not, you should provide some method for properly quoting the name in case there are spaces or other awkward characters present. In add-on, this is the stage at which you lot could provide checks on the uploaded file in guild to restrict access to certain file types. For example, a elementary check could be implemented to let simply .png files to keep beyond this point.

Finally, in app.js, create the view responsible for receiving the account information after the user has uploaded an avatar, filled in the grade, and clicked submit:

            app.post('/save-details', (req, res) => {   // TODO: Read POSTed form data and do something useful });                      

This function is currently only a stub that you'll need to complete in order to permit the app to read and shop the submitted profile data and to correctly acquaintance it with the residual of the user's business relationship details.

Running the app

Everything should now exist in place to perform the direct uploads to S3. To exam the upload, save any changes and use heroku local to start the application:

You will need a Procfile for this to be successful. See Getting Started with Node.js on Heroku for more information. Also remember to correctly set your environment variables on your ain machine before running the application locally.

            $ heroku local fifteen:44:36 spider web.1  | started with pid 12417                      

Press Ctrl+C to return to the prompt. If your application is returning 500 errors (or other server-based issues), then first your server in debug way and view the output in the Last emulator to help set up your problem:

            $ DEBUG=express:* node app.js                      

Summary

This article covers uploading to Amazon S3 directly from the browser using Node.js to temporarily sign the upload request. Although the guide and companion code focuses on the Express framework, the idea should easily bear over to other Node applications.

pringleselven.blogspot.com

Source: https://devcenter.heroku.com/articles/s3-upload-node

Share this post