Today I Learned

6 posts about #aws

Handling missing data points in AWS metrics

Sometimes you might want to treat missing data as good (not breaching threshold) to avoid keeping insufficient state. AWS allows to specify few options:

  • notBreaching – Missing data points are treated as “good” and within the threshold
  • breaching – Missing data points are treated as “bad” and breaching the threshold
  • ignore – The current alarm state is maintained
  • missing – If all data points in the alarm evaluation range are missing, the alarm transitions to INSUFFICIENT_DATA.

For more details go here: AWS Docs

Terraform config:

  resource "aws_cloudwatch_metric_alarm" "some_resource_usage" {
  ...
    treat_missing_data = "notBreaching"
  }

Creating DNS record for current origin in AWS Route53

Raw DNS zone:

@ 3600   IN   MX  1 aspmx.l.google.com.
@ 3600   IN   MX  10  aspmx2.googlemail.com.
@ 3600   IN   MX  10  aspmx3.googlemail.com.
@ 3600   IN   MX  5 alt1.aspmx.l.google.com.
@ 3600   IN   MX  5 alt2.aspmx.l.google.com.

From rfc1035:

A free standing @ is used to denote the current origin.

when using Terraform just leave the name blank:

resource "aws_route53_record" "google_mail_mx" {
  zone_id = aws_route53_zone.primary.zone_id
  name    = "" # 
  type    = "MX"
  ttl     = 3600

  records = [
    "1 aspmx.l.google.com.",
    "5 alt1.aspmx.l.google.com.",
    "5 alt2.aspmx.l.google.com.",
    "10 aspmx2.googlemail.com.",
    "10 aspmx3.googlemail.com."
  ]
}

AWS Route 53, prefer ALIAS over CNAME if you can

Let’s assume you have a zone defined:

resource "aws_route53_zone" "primary" {
  name = "example.com"
}

Now you want to add subdomain api and point it to your Application Load Balancer, normally you would think to add CNAME to point to your ALB:

resource "aws_route53_record" "api" {
  zone_id = aws_route53_zone.primary.zone_id
  name    = "api"
  type    = "CNAME"
  ttl     = "5"
  records = ["your-alb-dns-name"]
}

but you will be charged extra for CNAME queries: Route53 Pricing

You incur charges for every DNS query answered by the Amazon Route 53 service, except for queries to Alias A records that are mapped to Elastic Load Balancing instances, CloudFront distributions, AWS Elastic Beanstalk environments, API Gateways, VPC endpoints, or Amazon S3 website buckets, which are provided at no additional charge.

To help you decide please read documentation: Choosing alias and non-alias

Route 53 charges for CNAME queries. vs. Route 53 doesn’t charge for alias queries to AWS resources.

So it’s better to use aliases to AWS resources if you can:

resource "aws_route53_record" "api" {
  zone_id = aws_route53_zone.primary.zone_id
  name    = "api"
  type    = "A"

  alias {
    name                   = "your-api-alb-dns-name"
    zone_id                = "your-api-alb-zone-id"
    evaluate_target_health = true
  }
}

Use `pluck` to fetch paginated results from S3 client

Some of AWS client calls provide responses with the limited amount of data (typically 1.000 items per response).

Example response may look as follows:

aws_client.list_objects_v2(bucket: bucket)

=> #<struct Aws::S3::Types::ListObjectsV2Output
 is_truncated=true,
 contents=
 [#<struct Aws::S3::Types::Object
    key="reports/report_2.csv",
    last_modified=2019-03-13 14:25:04 UTC,
    etag="\"5a7c05eb47dcd13a27a26d34eb13b0ec\"",
    size=466,
    storage_class="STANDARD",
    owner=nil>,
    ...
 ]
 name="awesome-bucket",
 prefix="",
 delimiter=nil,
 max_keys=1000,
 common_prefixes=[],
 encoding_type=nil,
 key_count=1000,
 continuation_token=nil,
 next_continuation_token="1wEBwtqJOGmZF5DXgu5UhTMv386wdtND0EQzkkOUEGPPeF8tC58BEbfBvfsVHKGnxNgHxvFARrcWdCPJXXgiMzUtpedrxZP2G9wu/0but8ALLHDGdZVD4OHb41DWQKocGGAOwr0wfOeN4hUoCzimKeA==",
 start_after=nil>

Because list_objects_v2 method takes continuation_token as an argument, one of the solutions to fetch all the records may be to loop through the responses using next_continuation_token until the next_continuation_token field is empty.

Instead, you can use the built-in enumerator in the response object, which will return results from all the pages (next pages will be fetched automatically by SDK):

aws_client.list_objects_v2(bucket: bucket).map { |page| page[:contents] }

=> [[#<struct Aws::S3::Types::Object
   key="reports/report_2.csv",
   last_modified=2019-03-13 14:25:04 UTC,
   etag="\"5a7c05eb47dcd13a27a26d34eb13b0ec\"",
   size=466,
   storage_class="STANDARD",
   owner=nil>,
  #<struct Aws::S3::Types::Object
   key="reports/report_1.csv",
   last_modified=2019-03-13 13:43:30 UTC,
   etag="\"dc7215c066f62c7ddedef78e123dbc7c\"",
   size=191722,
   storage_class="STANDARD",
   owner=nil>,
   ... ]

However, there is even simpler solution to achieve the same result. You can use pluck method as follows:

aws_client.list_objects_v2(bucket: bucket).pluck(:contents)

Terraform AWS - moving state to another module

If your infrastructure grows and you find that certain resources should be moved to its own module because they need to be shared with others (or you made a mistake by putting them in the wrong module in the first place), you can move the state using CLI rather than recreating resources from scratch.

let’s say you have:

module "s3" {
  source = "./modules/s3"
  ...
}

and inside you defined user with access policy:

resource "aws_iam_user" "portal" {...}

resource "aws_iam_user_policy" "portal" {...}

Use:

terraform state mv module.s3.aws_iam_user.portal  module.iam
terraform state mv module.s3.aws_iam_user_policy.portal  module.iam

After that you can move your resource definitions from s3 to iam module. At the end, run terraform plan - terraform shouldn’t detect any changes.

Documentation here.