AWS Startup Workshop
Hamburg 2019

https://talks.superluminar.io

Workshop: Serverless URL Shortener with AWS

Jan & Soenke

Topics

Example how to use AWS for serverless
Deploy an HTTP API service
Manage Storage with DynamoDB
Use AWS Lambda for business logic

Serverless

Only write business logic
Orchestrate everything else
Architecture written in code
Production-Ready product: URL Shortener

URL Shortener

Familiar Topic
Public Examples available: bit.ly or tinyurl.com

Requirements

~~Editor~~
~~Terminal~~
~~Go environment~~
AWS account

Setup your IDE

Click https://eu-central-1.console.aws.amazon.com/cloud9/home/create
Enter your name
Click Next Steps, Next Steps, Create Environment
Open the IDE via https://eu-central-1.console.aws.amazon.com/cloud9/home
Click Open IDE

Let's go!

URL Shortener

Create shortened URLs (Lambda)
Resolve shortened URLs (Lambda)
Store URL mapping (DynamoDB)
HTTP Interface (API Gateway)

$ > git clone \
    https://github.com/superluminar-io/boilerplate-go.git
$ > cd boilerplate-go
$ > tree .

.
├── Makefile
├── src
│   └── example
│       ├── handle.go
│       ├── handle_test.go
│       └── main.go
└── template.yml

Bootstrap Cloud9

Configure cloud9

$ > make cloud9

Workflow

Compile source code
Bundle config with binaries
Upload deployable artifacts
Deploy service

Prepare AWS

Create S3 Bucket for artifacts

$ > make configure

Build it

Compile Go code

$ > make build

Create Artifact

Bundle configuration with binaries

$ > make package

Deploy

Use CloudFormation to deploy service

$ > make deploy

More Details

Prepare AWS

$ > make configure

aws s3api create-bucket \
    --bucket example-url-shortener-artifacts \
    --region eu-west-1 \
    --create-bucket-configuration LocationConstraint=eu-west-1

Create Artifact

$ > make package

aws cloudformation package \
    --template-file ./template.yml \
    --s3-bucket example-url-shortener-artifacts \
    --output-template-file ./dist/stack.yml \
    --region eu-west-1

Deploy

$ > make deploy

aws cloudformation deploy \
    --template-file ./dist/stack.yml \
    --region eu-west-1 \
    --capabilities CAPABILITY_IAM \
    --stack-name example-url-shortener-stack \
    --force-upload \
    --parameter-overrides \
      PREFIX=example \
      PROJECT=url-shortener

What Next?

You deployed a serverless project. 🎉
You know the basic workflow. 👍
You don't know what you deployed. 🤷‍

Remember this?

$ > tree .

.
├── Makefile
├── src
│   └── example
│       ├── handle.go
│       ├── handle_test.go
│       └── main.go
└── template.yml

# template.yml

[…]

Resources:
  Function:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: go1.x
      Handler: dist/handler/example

[…]

// handle.go

type event struct {
    ShouldFail bool `json:"should_fail"`
}

func handle(ctx context.Context, e event) (string, error) {
    if e.ShouldFail == true {
        return '', errors.New("Error")
    }

    return "Done", nil
}

URL Shortener

Next Steps

Configure HTTP access
Create your own functions
Use DynamoDB for storage

Change signature for handler

import (
  "github.com/aws/aws-lambda-go/events"
)

func handle(
  request events.APIGatewayProxyRequest
) (events.APIGatewayProxyResponse, error) {

}

Configure event trigger for handler

Resources:
  Function:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: go1.x
      Handler: dist/handler/example
      Events:
        CustomEventName:
          Type: Api
          Properties:
            Path: /example
            Method: get

Events:
  CustomEventName:
    Type: Api
    Properties:
      Path: /example
      Method: get

Events:
  CustomEventName:
    Type: Api
    Properties:
      Path: /example/{foo}
      Method: post

Update response for HTTP Interface

return events.APIGatewayProxyResponse{
  StatusCode: 200, 
  Body: "Example String",
}, nil

Build & Deploy again

$ > make build package deploy

How to access the API?

HTTP integration uses Amazon API Gateway
HTTP endpoint is configured automatically (SAM)
How to retrieve the HTTP URL?

CloudFormation Outputs

# template.yml

Outputs:
  PREFIX:
    Value: !Ref PREFIX
  PROJECT:
    Value: !Ref PROJECT

CloudFormation Outputs

# template.yml

Outputs:
  Endpoint:
    Value: !Sub https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod

$ > make build package deploy
$ > make describe

[
    {
        "OutputKey": "Endpoint",
        "OutputValue": "https://lrrcoo1jn6[…]com/Prod"
    }
]

$ > curl https://lrrcoo1jn6[…]com/Prod/example

Example String

Back to Topic

URL Shortener has two functions
URL Shortener has two HTTP Endpoints
Handle POST requests to store new URL
Handle GET request to resolve URL

$ > curl \
    -XPOST -d '{"url":"https://talks.superluminar.io"}' \
    https://$ENDPOINT

> POST / HTTP/1.1
< HTTP/1.1 Created 201

Created short url: https://://$ENDPOINT/$ID

$ > curl -v https://$ENDPOINT/$ID

> GET /$ID HTTP/1.1
< HTTP/1.1 302 Found
< Location: https://talks.superluminar.io

Next Steps

Create two new folders & files in ./src
Copy & Paste is fine.
Create two new functions in template.yml
Copy & Paste is fine.

$ > tree .

.
├── Makefile
├── src
│   ├── create
│   │   ├── handle.go
│   │   └── main.go
│   ├── example
│   │   ├── handle.go
│   │   └── main.go
│   └── resolve
│       ├── handle.go
│       └── main.go
└── template.yml

Resources:

  ResolveFunction:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: go1.x
      Handler: dist/handler/resolve

  CreateFunction:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: go1.x
      Handler: dist/handler/create

ResolveFunction:
  [...]
  Events:
    ResolveEvent:
      Type: Api
      Properties:
        Path: /{id}
        Method: get

CreateFunction:
  [...]
  Events:
    CreateEvent:
      Type: Api
      Properties:
        Path: /
        Method: post

Deploy again

$ > make build package deploy

Checkpoint

Two functions in ./src
Two HTTP routes / and /{id}
Static response Example String

Next: Storage

template.yml contains all resources for the API
Add DynamoDB configuration to the resource list

Resources:

  DynamoDBTable:
    Type: AWS::Serverless::SimpleTable
    Properties:
      TableName: !Sub ${PREFIX}-${PROJECT}

Checkpoint

Basic Architecture is done
Functions for resolve and create
API Gateway with HTTP routes
DynamoDB Table for Storage created

Access DynamoDB

DynamoDB tables have names
Remember? TableName: !Sub ${PREFIX}-${PROJECT}
DynamoDB table name as environment variable
DynamoDB has API to putItem and getItem
Of Course: Grant Access to DynamoDB for Lambda

DynamoDB Access Policies

AWS Policies for most common use cases
Policy for CRUD access DynamoDBCrudPolicy
Policy for Read access DynamoDBReadPolicy

Properties:
  Handler: dist/handler/create
  Runtime: go1.x
  Policies:
    - DynamoDBCrudPolicy:
        TableName: !Ref DynamoDBTable

Properties:
  Handler: dist/handler/resolve
  Runtime: go1.x
  Policies:
    - DynamoDBReadPolicy:
        TableName: !Ref DynamoDBTable

Environment Variables

[…]

  Handler: dist/handler/create
  Runtime: go1.x
  Environment:
    Variables:
      DYNAMODB_TABLE_NAME: !Ref DynamoDBTable

Access Environment variable

body := fmt.Sprintf(
  "env variable DYNAMODB_TABLE_NAME is %s",
  os.Getenv("DYNAMODB_TABLE_NAME"),
)

return events.APIGatewayProxyResponse{
  StatusCode: 200, 
  Body: body,
}, nil

DynamoDB: GetItem

// resolve/handle.go
import (
  "github.com/aws/aws-sdk-go/aws"
  "github.com/aws/aws-sdk-go/aws/session"
  "github.com/aws/aws-sdk-go/service/dynamodb"
)

func handler(request events.APIGatewayProxyRequest) {
  table := os.Getenv("DYNAMODB_TABLE_NAME")
  id := request.PathParameters["id"]

  s := session.Must(session.NewSession())
  client := dynamodb.New(s)

  […]
}

result, err := client.GetItem(
  &dynamodb.GetItemInput{
    TableName: aws.String(table),
    Key: map[string]*dynamodb.AttributeValue{
      "id": {S: aws.String(id)},
    },
  },
)

if err != nil {
  return events.APIGatewayProxyResponse{
    StatusCode: 500, 
    Body: "Failed to access data",
  }, nil
}

if result.Item == nil {
  return events.APIGatewayProxyResponse{
    StatusCode: 404, 
    Body: "Unknown ID",
  }, nil
}

return events.APIGatewayProxyResponse{
  StatusCode: 302, 
  Headers: map[string]string{
    "Location": *result.Item["url"].S,
  },
}, nil

Store data in DynamoDB

Ensure create has a configured environment variable
Access HTTP request body
Create shortened id for submitted URL

import (
    "encoding/json"
    "fmt"
    "hash/fnv"
    "net/url"
    "os"
    "strconv"

    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/dynamodb"
)

func handler(request events.APIGatewayProxyRequest) {
  table := os.Getenv("DYNAMODB_TABLE_NAME")

  s := session.Must(session.NewSession())
  client := dynamodb.New(s)
}

var data map[string]string
err := json.Unmarshal([]byte(request.Body), &data)

// Error Handling

url, ok := data["url"]

// Error Handling

id, err := shorten(url)

// Error Handling

func shorten(u string) (string, error) {
    if _, err := url.ParseRequestURI(u); err != nil {
        return "", err
    }

    hash := fnv.New64a()
    if _, err := hash.Write([]byte(u)); err != nil {
        return "", err
    }

    return strconv.FormatUint(hash.Sum64(), 36), nil
}

DynamoDB: putItem

_, err = client.PutItem(
  &dynamodb.PutItemInput{
    TableName: aws.String(table),
    Item: map[string]*dynamodb.AttributeValue{
      "id":  {S: aws.String(id)},
      "url": {S: aws.String(url)},
    },
  },
)

body := fmt.Sprintf("Created short url: %s/%s/%s", request.Headers["Host"], "Prod", id)
return events.APIGatewayProxyResponse{
    StatusCode: 201,
    Body:       body,
}, nil

$ > make build package deploy

$ > # Remember how to get the Endpoint URL ?

$ > curl \
    -XPOST -d '{"url":"https://talks.superluminar.io"}' \
    https://$ENDPOINT

> POST / HTTP/1.1
< HTTP/1.1 Created 201

Created short url: http://$ENDPOINT/$ID

$ > curl -v http://$ENDPOINT/$ID

> GET /$ID HTTP/1.1
< HTTP/1.1 302 Found
< Location: https://talks.superluminar.io

Thanks.

Cheat sheet

https://github.com/superluminar-io/boilerplate-go/tree/cheatsheet/

AWS Startup WorkshopHamburg 2019

Jan & Soenke

Topics

Serverless

URL Shortener

Requirements

Setup your IDE

Let's go!

URL Shortener

Bootstrap Cloud9

Workflow

Prepare AWS

Build it

Create Artifact

Deploy

More Details

Prepare AWS

Create Artifact

Deploy

What Next?

Remember this?

URL Shortener

Next Steps

Build & Deploy again

How to access the API?

CloudFormation Outputs

CloudFormation Outputs

Back to Topic

Next Steps

Deploy again

Checkpoint

Next: Storage

Checkpoint

Access DynamoDB

DynamoDB Access Policies

Environment Variables

Access Environment variable

DynamoDB: GetItem

Store data in DynamoDB

DynamoDB: putItem

Thanks.

Cheat sheet

AWS Startup Workshop
Hamburg 2019