Custom Go module URLs with serverless AWS and Terraform

• infra
Back

Go, as a language and as an ecosystem, is known for its minimalism. One example of this is the package management system in the go get command. Import paths are as simple as the URL to the source code repository that holds the package's code.

This is nice because there is no need for another centrally managed package server (a la RubyGems or NPM) to host distributable packages distinct from source code, but has some limitations compared to central repository model. One limitation is the direct coupling between all import statements that depend on a certain package, and the location where that package's author has decided to host its code.

Go's core team introduced a solution to this problem: the ability to mask the import path of packages with so-called "vanity URLs". For example, I could have my code hosted at github.com/example/my-go-module, but everyone who wanted to use my package could go get go.example.com/my-go-module. That way, if I ever decided to move to BitBucket or GitLab, I could do so without breaking the builds of everyone who depends on my package.

This works by setting up some sort of web server to respond to GET requests to the vanity URL and respond with a custom HTML <meta> tag, pointing clients (go get) to the current source code repository. This behavior is documented here. Google Cloud Platform released a HTTP server that you can run yourself to translate go get requests for you, but personally I'd rather not have to manage another server in my infrastructure, so I decided to build a serverless request translator using AWS's serverless offerings. Going serverless truly allows you to "set and forget" production systems like nothing else.

Serverless Go Vanity URLs on AWS

These are the components that make up this system.

  1. S3 bucket, as a simple web host to serve up <meta> tags in response to package requests.
  2. CloudFront distribution, to sit in front of the S3 bucket to serve requests via HTTPS (which go get expects).
  3. A free AWS Certificate Manager TLS certificate which we can easily attach to our CloudFront distribution to support HTTPs traffic.
    • This certificate must be set up in the us-east-1 region because of limitations within CloudFront. I address this in more detail further down the page.
  4. Two Route53 DNS records to
    1. Point go.example.com to the CloudFront distribution.
    2. Validate ownership of this domain to allow AWS Certificate Manager to issue a certificate.

You must define the variables defined at the top of the following Terraform config.

Overall, the Terraform code should be plug-and-play. I didn't include the Terraform state backend and provider configuration, which I assume almost everybody has. If you don't have that set up, I recommend following the Terraform "Getting Started" tutorial.

Certificate Setup

One important thing to note is that the AWS Certificate Manager certificate must be set up in the us-east-1 (N. Virginia) region for it to be compatible with CloudFront distributions. If the rest of your infrastructure is in a different region, don't fret. Terraform handles multi-region setups very nicely, so all you need to do is configure a new provider named aws with the us-east-1 alias, and use that to create just the TLS cert. It will link up with the rest of your resources in your primary region very easily.

# Primary, implicitly selected provider
provider "aws" {
  version = "~> 1.33"
  region  = "us-west-2"
}

# Secondary provider, accessible as `aws.us-east-1`
provider "aws" {
  alias   = "us-east-1"
  version = "~> 1.33"
  region  = "us-east-1"
}

Once this provider has been setup, you can instruct certain resources to be created using that provider, which I have already done in the provided Terraform config.

resource "aws_acm_certificate_validation" "certificate" {
  provider = "aws.us-east-1"
  ...
}

go-modules.tf

Here is the example code for your use.

# Zone ID of the Route53 hosted zone where your DNS records will be created. This hosted zone must be a parent of the `var.go_hostname`.
variable "zone_id"  { type = "string" }
# FQDN under which you want to serve `go get` requests. (i.e. go.example.com)
variable "go_hostname"  { type = "string" }
# URL under which your packages are currently hosted. (i.e. github.com/example)
variable "go_vcs_base" { type = "string" }
# List of modules that you want to "proxy" to your current VCS
variable "go_modules"   { type = "list" }

resource "aws_route53_record" "go-modules" {
  zone_id = "${var.zone_id}"

  name = "${var.go_hostname}"
  type = "A"

  alias {
    zone_id = "${aws_cloudfront_distribution.go-modules.hosted_zone_id}"
    name    = "${aws_cloudfront_distribution.go-modules.domain_name}"
    evaluate_target_health = true
  }
}

# Create S3 bucket and module objects
resource "aws_s3_bucket" "go-modules" {
  bucket  = "${var.go_hostname}"
  acl     = "public-read"

  tags {
    name = "${var.go_hostname}"
  }

  website {
    index_document = "index.html" # This file doesn't exist, but it must be specified.
  }
}

resource "aws_s3_bucket_object" "go-modules-redirect" {
  count = "${length(var.go_modules)}"

  bucket  = "${aws_s3_bucket.go-modules.bucket}"
  acl     = "public-read"

  key     = "${element(var.go_modules, count.index)}"
  content = <<EOF
<meta name=go-import content="${var.go_hostname}/${element(var.go_modules, count.index)} git ${var.go_vcs_base}/${element(var.go_modules, count.index)}.git">
EOF
}

locals {
  modules_s3_origin_id = "GoModulesS3Origin"
}

# Create CloudFront distribution to allow TLS access.
resource "aws_cloudfront_distribution" "go-modules" {
  origin {
    domain_name = "${aws_s3_bucket.go-modules.bucket_regional_domain_name}"
    origin_id   = "${local.modules_s3_origin_id}"
  }

  enabled = true
  is_ipv6_enabled = true
  default_root_object = "index.html"
  comment = "Go Modules"

  aliases = ["${var.go_hostname}"]

  default_cache_behavior {
    allowed_methods = ["GET", "HEAD", "OPTIONS"]
    cached_methods  = ["GET", "HEAD"]
    target_origin_id  = "${local.modules_s3_origin_id}"

    forwarded_values {
      query_string = false
      cookies {
        forward = "none"
      }
    }

    viewer_protocol_policy = "allow-all"
    min_ttl     = 0
    default_ttl = 3600
    max_ttl     = 86400
  }

  restrictions {
    geo_restriction {
      restriction_type = "none"
    }
  }

  viewer_certificate {
    acm_certificate_arn = "${aws_acm_certificate_validation.go-modules-dist-cert.certificate_arn}"
    ssl_support_method  = "sni-only"
  }
}

# Create new TLS certificate and automatically validate it with a DNS record.
resource "aws_acm_certificate" "go-modules-dist-cert" {
  provider = "aws.us-east-1" # this certificate must live in the us-east-1 region

  domain_name = "${var.go_hostname}"
  validation_method = "DNS"

  lifecycle { create_before_destroy = true }
}

resource "aws_acm_certificate_validation" "go-modules-dist-cert" {
  provider = "aws.us-east-1" # this certificate must live in the us-east-1 region
  certificate_arn = "${aws_acm_certificate.go-modules-dist-cert.arn}"
  validation_record_fqdns = []
}

resource "aws_route53_record" "go-modules-dist-cert-validation" {
  zone_id = "${var.zone_id}"

  name    = "${aws_acm_certificate.go-modules-dist-cert.domain_validation_options.0.resource_record_name}"
  type    = "${aws_acm_certificate.go-modules-dist-cert.domain_validation_options.0.resource_record_type}"
  records = ["${aws_acm_certificate.go-modules-dist-cert.domain_validation_options.0.resource_record_value}"]
  ttl     = 60
}

Validation

Once you have integrated this configuration into your infrastructure, its time to test it to make sure it works. I generally tend to test things piece by piece.

First, lets test the DNS record.

$ host go.example.com
go.example.com has address 54.239.xxx.xxx
go.example.com has address 54.239.xxx.xxx
go.example.com has address 54.239.xxx.xxx
go.example.com has address 54.239.xxx.xxx

Looks good. We created a Route53 alias record, which under the hood is represented by an A record. Unlike a CNAME, we don't see any textual reference to another server.

Now, lets use curl to test that our HTTPS request is returning the proper response, namely a 200 OK with our <meta> tag.

$ curl -i https://go.example.com/package
HTTP/2 200
content-type: binary/octet-stream
content-length: 118
last-modified: Tue, 12 Feb 2019 19:57:56 GMT
accept-ranges: bytes
server: AmazonS3
date: Thu, 14 Feb 2019 02:43:21 GMT
etag: "xxxxxxxxxxxxxxxxxx"
age: 50
x-cache: Hit from cloudfront
via: 1.1 xxxxxxxxxxxxxxxxx.cloudfront.net (CloudFront)
x-amz-cf-id: xxxxxxxxxxxxxxxxx

<meta name=go-import content="go.example.com/package git https://github.com/example/package.git">

Finally, lets test this using a real world scenario -- a real go get.

$ go get go.example.com/package && echo "Exit code $?"
Exit code 0

It works!

© Andrew PageRSS