Misc scribbles

Making a serverless website for photo and video upload pt. 2

2020-12-26

This post follows onhttps://cmdcolin.github.io/2020-12-24.html

It is possible I zoomed ahead too fast to make this a continuous tutorial, but overall I just wanted to post an update

In pt. 1 I learned how to use the aws-sam CLI tool. This was a great insight for me about automating deployments. I can now simply run sam deploy and it will create new dynamodb tables, lambda functions, etc.

After writing pt 1. I converted the existing vue-js app that was in the aws tutorial and converted it to react. Then I extended the app to allow

It will be hard to summarize all the changes since now the app has taken off a little bit but it looks like this:

Repo structure

./frontend
./frontend/src/App.tsx
./lambdas/
./lambdas/postFile
./lambdas/getFiles
./lambdas/postComment
./lambdas/getComments

Here is a detailed code for uploading the file. We upload one file at a time, but the client code post to the lambda endpoint individually for each file

This generates a pre-signed URL to allow the client-side JS (not the lambda itself) to directly upload to S3, and also posts a row in the S3 to the filename that will. It is very similar code in to https://cmdcolin.github.io/2020-12-24.html

./lambdas/postFile/app.js

'use strict'
 
const AWS = require('aws-sdk')
const multipart = require('./multipart')
AWS.config.update({ region: process.env.AWS_REGION })
const s3 = new AWS.S3()
 
// Change this value to adjust the signed URL's expiration
const URL_EXPIRATION_SECONDS = 300
 
// Main Lambda entry point
exports.handler = async event => {
  return await getUploadURL(event)
}
 
const { AWS_REGION: region } = process.env
 
const dynamodb = new AWS.DynamoDB({ apiVersion: '2012-08-10', region })
 
async function uploadPic({
  timestamp,
  filename,
  message,
  user,
  date,
  contentType,
}) {
  const params = {
    Item: {
      timestamp: {
        N: `${timestamp}`,
      },
      filename: {
        S: filename,
      },
      message: {
        S: message,
      },
      user: {
        S: user,
      },
      date: {
        S: date,
      },
      contentType: {
        S: contentType,
      },
    },
    TableName: 'files',
  }
  return dynamodb.putItem(params).promise()
}
 
const getUploadURL = async function (event) {
  try {
    const data = multipart.parse(event)
    const { filename, contentType, user, message, date } = data
    const timestamp = +Date.now()
    const Key = `${timestamp}-${filename}` // Get signed URL from S3
 
    const s3Params = {
      Bucket: process.env.UploadBucket,
      Key,
      Expires: URL_EXPIRATION_SECONDS,
      ContentType: contentType,
      // This ACL makes the uploaded object publicly readable. You must also
      // uncomment the extra permission for the Lambda function in the SAM
      // template.
 
      ACL: 'public-read',
    }
 
    const uploadURL = await s3.getSignedUrlPromise('putObject', s3Params)
 
    await uploadPic({
      timestamp,
      filename: Key,
      message,
      user,
      date,
      contentType,
    })
 
    return JSON.stringify({
      uploadURL,
      Key,
    })
  } catch (e) {
    const response = {
      statusCode: 500,
      body: JSON.stringify({ message: `${e}` }),
    }
    return response
  }
}

./lambdas/getFiles/app.js

// eslint-disable-next-line import/no-unresolved
const AWS = require('aws-sdk')
 
const { AWS_REGION: region } = process.env
 
const docClient = new AWS.DynamoDB.DocumentClient()
 
const getItems = function () {
  const params = {
    TableName: 'files',
  }
 
  return docClient.scan(params).promise()
}
 
exports.handler = async event => {
  try {
    const result = await getItems()
    return {
      statusCode: 200,
      body: JSON.stringify(result),
    }
  } catch (e) {
    return {
      statusCode: 400,
      body: JSON.stringify({ message: `${e}` }),
    }
  }
}

./frontend/src/App.tsx (excerpt)

async function myfetch(params: string, opts?: any) {
  const response = await fetch(params, opts)
  if (!response.ok) {
    throw new Error(`HTTP ${response.status} ${response.statusText}`)
  }
  return response.json()
}
 
function UploadDialog({
  open,
  onClose,
}: {
  open: boolean
  onClose: () => void
}) {
  const [images, setImages] = useState<FileList>()
  const [error, setError] = useState<Error>()
  const [loading, setLoading] = useState(false)
  const [total, setTotal] = useState(0)
  const [completed, setCompleted] = useState(0)
  const [user, setUser] = useState('')
  const [message, setMessage] = useState('')
  const classes = useStyles()
 
  const handleClose = () => {
    setError(undefined)
    setLoading(false)
    setImages(undefined)
    setCompleted(0)
    setTotal(0)
    setMessage('')
    onClose()
  }
 
  return (
    <Dialog onClose={handleClose} open={open}>
      <DialogTitle>upload a file (supports picture or video)</DialogTitle>
      <DialogContent>
        <label htmlFor="user">name (optional) </label>
        <input
          type="text"
          value={user}
          onChange={event => setUser(event.target.value)}
          id="user"
        />
        <br /> <label htmlFor="user">message (optional) </label>
        <input
          type="text"
          value={message}
          onChange={event => setMessage(event.target.value)}
          id="message"
        />
        <br />
        <input
          multiple
          type="file"
          onChange={e => {
            let files = e.target.files
            if (files && files.length) {
              setImages(files)
            }
          }}
        /> {error ? (
          <div className={classes.error}>{`${error}`}</div>
        ) : loading ? (
          `Uploading...${completed}/${total}`
        ) : completed ? (
          <h2>Uploaded </h2>
        ) : null} <DialogActions>
          <Button
            style={{ textTransform: 'none' }}
            onClick={async () => {
              try {
                if (images) {
                  setLoading(true)
                  setError(undefined)
                  setCompleted(0)
                  setTotal(images.length)
                  await Promise.all(
                    Array.from(images).map(async image => {
                      const data = new FormData()
                      data.append('message', message)
                      data.append('user', user)
                      data.append('date', new Date().toLocaleString())
                      data.append('filename', image.name)
                      data.append('contentType', image.type)
                      const res = await myfetch(API_ENDPOINT + '/postFile', {
                        method: 'POST',
                        body: data,
                      })
 
                      await myfetch(res.uploadURL, {
                        method: 'PUT',
                        body: image,
                      })
 
                      setCompleted(completed => completed + 1)
                    }),
                  )
                  setTimeout(() => {
                    handleClose()
                  }, 500)
                }
              } catch (e) {
                setError(e)
              }
            }}
            color="primary"
          >
            upload
          </Button>
          <Button
            onClick={handleClose}
            color="primary"
            style={{ textTransform: 'none' }}
          >
            cancel
          </Button>
        </DialogActions>
      </DialogContent>
    </Dialog>
  )
}

template.yaml for AWS

AWSTemplateFormatVersion: 2010-09-09
Transform: AWS::Serverless-2016-10-31
Description: S3 Uploader
 
Resources:
  filesDynamoDBTable:
    Type: AWS::DynamoDB::Table
    Properties:
      AttributeDefinitions:
        - AttributeName: 'timestamp'
          AttributeType: 'N'
      KeySchema:
        - AttributeName: 'timestamp'
          KeyType: 'HASH'
      ProvisionedThroughput:
        ReadCapacityUnits: '5'
        WriteCapacityUnits: '5'
      TableName: 'files'
 
  # HTTP API
  MyApi:
    Type: AWS::Serverless::HttpApi
    Properties:
      # CORS configuration - this is open for development only and should be restricted in prod.
      # See https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-property-httpapi-httpapicorsconfiguration.html
      CorsConfiguration:
        AllowMethods:
          - GET
          - POST
          - DELETE
          - OPTIONS
        AllowHeaders:
          - '*'
        AllowOrigins:
          - '*'
 
  UploadRequestFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: lambdas/postFile/
      Handler: app.handler
      Runtime: nodejs12.x
      Timeout: 3
      MemorySize: 128
      Environment:
        Variables:
          UploadBucket: !Ref S3UploadBucket
      Policies:
        - AmazonDynamoDBFullAccess
        - S3WritePolicy:
            BucketName: !Ref S3UploadBucket
        - Statement:
            - Effect: Allow
              Resource: !Sub 'arn:aws:s3:::${S3UploadBucket}/'
              Action:
                - s3:putObjectAcl
      Events:
        UploadAssetAPI:
          Type: HttpApi
          Properties:
            Path: /postFile
            Method: post
            ApiId: !Ref MyApi
 
  FileReadFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: lambdas/getFiles/
      Handler: app.handler
      Runtime: nodejs12.x
      Timeout: 3
      MemorySize: 128
      Policies:
        - AmazonDynamoDBFullAccess
      Events:
        UploadAssetAPI:
          Type: HttpApi
          Properties:
            Path: /getFiles
            Method: get
            ApiId: !Ref MyApi
 
  ## S3 bucket
  S3UploadBucket:
    Type: AWS::S3::Bucket
    Properties:
      CorsConfiguration:
        CorsRules:
          - AllowedHeaders:
              - '*'
            AllowedMethods:
              - GET
              - PUT
              - HEAD
            AllowedOrigins:
              - '*'
 
## Take a note of the outputs for deploying the workflow templates in this sample application
Outputs:
  APIendpoint:
    Description: 'HTTP API endpoint URL'
    Value: !Sub 'https://${MyApi}.execute-api.${AWS::Region}.amazonaws.com'
  S3UploadBucketName:
    Description: 'S3 bucket for application uploads'
    Value: !Ref 'S3UploadBucket'

To display all the pictures I use a switch from video or img tag based on contentType.startsWith('video'). I also use the "figcaption" HTML tag to have a little caption on the pics/videos

./frontend/src/App.tsx

function Media({
  file,
  style,
  onClick,
  children,
}: {
  file: File
  onClick?: Function
  style?: React.CSSProperties
  children?: React.ReactNode
}) {
  const { filename, contentType } = file
  const src = `${BUCKET}/${filename}`
  return (
    <figure style={{ display: 'inline-block' }}>
      <picture>
        {contentType.startsWith('video') ? (
          <video style={style} src={src} controls onClick={onClick as any} />
        ) : (
          <img style={style} src={src} onClick={onClick as any} />
        )}
      </picture>
      <figcaption>{children}</figcaption>
    </figure>
  )
}

Now the really fun part: if you get an image of a picture frame like https://www.amazon.com/Paintings-Frames-Antique-Shatterproof-Osafs2-Gld-A3/dp/B06XNQ8W9T

You can make it a border for any image or video using border-image CSS

style = {
  border: '30px solid',
  borderImage: `url(borders/${border}) 30 round`,
}

#Summary

The template.yaml automatically deploys the lambdas for postFile/getFile and the files table in dynamoDB

The React app uses postFile for each file in an <input type="file"/>, the code uses React hooks and functional components but is hopefully not too complex

I also added commenting on photos. The code is not shown here but you can look in the source code for details

Overall this has been a good experience learning to develop this app and learning to automate the cloud deployment is really good for ensuring reliability and fast iteration.

Also quick note on serverless CLI vs aws-sam. I had tried a serverless CLI tutorial from another user but it didn't click with me, while the aws-sam tutorial from https://searchvoidstar.tumblr.com/post/638408397901987840/making-a-serverless-website-for-photo-upload-pt-1 was a great kick start for me. I am sure the serverless CLI is great too and it ensures a bit less vendor lock in, but then is also a little bit removed from the native aws config schemas. Probably fine though

#Source code

https://github.com/cmdcolin/aws_photo_gallery/