I say all this to say that in the roughly nine years that I have been using terraform to manage large infrastructure projects I have banged my shins on many sharp corners and the bruises have left strong opinions.

If you already have strong opinions about terraform, this post is not for you. Your approach is great, keep using it. My opinion doesn't matter. Don't read another word. Go see what's happening on mastodon or something.

If you do not have strong opinions about terraform and are just getting started, you are my target audience.

Assumptions: everything here is based on the assumption that you're using terraform for something long term and important. If you just want to get something done quickly, or the thing you're working on can be burned down and recreated without causing problems, none of this applies to you.

(aside: all this applies to opentofu as well).

Rule 1. Complexity is the enemy

In a regular software project one can optimize for any number of things; memory or CPU efficiency, ability to easily refactor and add features, correctness of the implementation (eg. rigorous tests) and so on. With a nontrivial terraform project you should optimize for one thing: ability to reason about the project.

It is incredibly easy to build a terraform project which is extremely difficult to reason about.

Yes the plan helps but do NOT believe the plan! Apply is all that matters and plan != apply. If you have not yet seen a plan that looked good which turned into an apply that went bad don't worry, you will.

But before you can even see the plan you have to implement your change. You have to look at the existing codebase, decide how to structure your change and what needs to be modified, and execute. In a nontrivial terraform codebase this can be a daunting task!

Sources of complexity are numerous:

• layers of indirection (eg. nested modules)
  • Speaking of which, do not nest modules.
• disparate inputs (eg. env vars, tfvars files)
• complex logic

At every turn, with every PR, you must push back on anything that increases complexity. When you inevitably cave and add another layer of indirection or piece of logic, leave a comment that explains why.

The comment audience is your future self, who will have long forgotten what you were thinking.

The fewer places you have to look to figure out what's happening, the better.

Don't think of it as "IaC"

The problem with the phrase "Infrastructure as Code" is in the word "code". As soon as you call it code people start to cargo cult in all these software engineering principles that have NO BUSINESS in a terraform project (see rule 1). If you want to control your infrastructure with actual code go use Pulumi or AWS CDK and implement an AbstractLoadBalancerFactoryBaseClass() or whatever.

Terraform is "infrastructure as config files".

Sure, go ahead and write reusable modules. Add loops. Use conditionals. But do so sparingly. It is much easier to reason about 27 github_repo resources with slightly different configs than one module called 27 times or worse, looping over one module using a list that contains 27 data structures representing each repo config.

Don't use community modules

A terraform module is an opinion about how to do things with respect to some group of resources. The problem with community modules is that they need to support the many opinions present in the community, making them more complex by necessity (see rule 1).

Adam Jacob brilliantly explains this using what he calls the "200% knowledge problem" in this talk (deep link to the specific moment, just watch for 40 seconds).

Read them for inspiration and then write your own, with fewer resources and using fewer variables and conditions. Instead of swallowing community opinions wholesale, capture the opinions of your organization within your own modules. A module should contain opinions like "This ALB configuration suits the needs of our app" or "replicating data across two zones in a single region is sufficiently robust for our needs".

Disregard anyone who argues this point by invoking phrases like "reinventing the wheel". Those people are not on the hook for maintaining your infrastructure, you are.

Don't add tools until you're forced to

• TF Cloud is fine.
• Spacelift is fine.
• Atlantis is fine.
• Terragrunt is fine I guess, I've never used it.
• I do not understand the point of terratest at all.
  • Please do not try to explain it to me, I don't care.
• I'm sure the others I've forgotten are cool too.

Don't use them unless you ABSOLUTELY NEED THEM (see rule 1). For some reason everyone is happy to say "Kubernetes is overkill for most teams" but nobody wants to say "TF Cloud is overkill for most teams".

Well I'm sayin it!

Many platform / SRE teams are 1-4 people and you can get most of the value of collaboration tools like Atlantis from a little team communication and good PR hygeine.

Iterating on a broken plan/apply cycle with a tool like Atlantis in the developer loop sucks. Just tell the team you're planning / applying from your laptop while you iterate, trust the state lock to avoid collisions, and communicate the results when you're finished.

With respect to the tools that don't focus on collaboration; you can get very far using just terraform and some project structure. If you do not have a rock solid reason to adopt them, don't.

Pursue consistency when "threading"

A big terraform project involves repeatedly passing information from one place to the next, and then when terraform is complete it is often necessary to pass outputs into something else like an ansible playbook or a helm chart.

eg.

variable "domain" {  
  type = string
  default = "foo.com"
}

module "dns" {  
  root = var.domain
}

module "loadbalancer" {  
  hostname = module.dns.registered_domain
}

output "monitoring_endpoint" {  
  value = module.loadbalancer.external_name
}

Here the domain for a project gets "threaded" from a variable called domain into the DNS module as a parameter called root which outputs a value called registered domain (maybe having prefixed www or something) which is passed to the LB module as a parameter called hostname, then output again as external_name (maybe having added https://) and finally output from terraform into some monitoring tool as external_name. This (not as contrived as you might think) example passes the same data around with seven different labels.

Could you make the argument that each use of this data happens in its own domain and has its own internal data model that makes sense for that use case? Sure.

Fight back against that argument.

Try to use a consistent name across the entire lifecycle of the data as it gets passed around. Reduce the number of things you have to look at to understand what's happening (see rule 1).

Don't rely on memory

Avoid anything that requires you to remember to do something before you apply. Don't set it up so you have to export the correct AWS_SECRET_ACCESS_KEY. Instead hard code a named profile or a specific IAM role in the provider config.

Don't require your team to switch to the correct workspace before applying, in fact don't use workspaces at all.

It's too easy to accidentally bulldoze production this way. Make it impossible to fuck up.

Style

This is just a random smattering of preferences which help improve maintainability.

Do not put everything into one huge pile, separate things into separate terraform states. There are lots of guides on how to structure this. This one is fine but there are plenty of others. Choose one.

Never use count to create more than one of something. count should only ever be used when you want to create something if some condition is true, and not create it otherwise. If you want to create more than one of something, use for_each.

I searched for a better source to explain why but this was the most succinct thing I could find.

Arguments to resources are the domain of the provider and you are beholden to them. AWS tags can only contain certain characters while K8s labels can only contain others. Don't fret about what goes in there, but terraform resource names are yours. You decide how they look and feel.

• Never capitalize them.
• Never use anything but underscores as separators.
• Never put the resource type in the name.
  • The type is right there next to it. This tip brought to you by the department of redundancy department.

🤮

resource "aws_ec2_instance" "Nginx-Instance"

🤩

resource "aws_ec2_instance" "nginx"

I recently got nerd-sniped by a blog post that explored an old istio vulnerability. The vulnerability meant that istio users whose access to Kubernetes secrets was restricted by RBAC policies could circumvent those policies in order to access kubernetes secrets in any namespace that istio had access to (which is all of them by default).

I remember when this bugfix shipped and I didn't think much of it at the time. I got confused trying to follow the steps and then realized there were some key bits of information missing. To be fair to the author, as someone who teaches people to use istio for a living I appreciate how hard it is to explain this stuff. This is not intended as a critique of the author but an addition to the conversation.

The repro environment involves deploying istio with two custom ingress gateways into separate namespaces called ns-a and ns-b. The author advises readers to do this by the use of two different customized installation profiles however in the browsers I tried the YAML rendering in the blog post seemed mangled somehow which made it hard to read. It is unnecessary to use two separate profiles to achieve the desired configuration of a gateway in each namespace so I merged them into a single profile.

Also the provided installation profile installs istio without any of the CRD's or the istiod controller which provides the gateway injection webhook functionality. As it is impossible to reproduce this issue as described without both of those things, my modified profile installs them as well.

The subsequent steps outlined in the post involve modifying and apply more of those hard to read manifests. This was a bit fiddly so I have checked-in formatted versions in here.

After applying these manifests the cluster will have a toy workload (I used nginx) in each namespace, a virtualservice in each namespace to route traffic to the toy workload, and a gateway configuration in each namespace to terminate TLS. TLS termination is performed using a self-signed certificate and key found in a kubernetes secret called foo-dot-com. Crucially however, the secret only exists in the ns-a namespace.

If the istio version being tested is vulnerable to the security issue, then it will be possible to connect to the gateway in namespace ns-b using TLS, despite the fact that the secret necessary for terminating TLS is not there.

If the istio version being tested is not vulnerable, then it will not be possible and the gateway will reset the connection. I have included a test script to validate this.

1. In the pixel watch app on your phone, under Notifications, deselect all the watch apps and all the phone apps. In the General section disable "auto turn on notifications for new apps". Under the "mute notifications" section, enable all the "mute ..." options in the Watch section.

2. In the watch itself open the settings (cog icon), Apps & Notifications, Do Not Disturb, enable DND.

3. In the fitbit app on your phone, click the profile picture icon in the upper left corner of the app. Then click the "Google Pixel Watch" item in the list (surprise, that's a button!), then disable "Reminders to Move". While you're in there make sure that "Main Goal" is set to "Steps". Back out of there and click the "Activity & Wellness" item under the SETTINGS heading, then click "Daily Activity", and set the "Steps Goal" to 999,999 (the highest it will accept. Now never take more than a million steps and you won't be interrupted again.

Nice enough watch I guess but it is extremely one point oh.

One thing I've seen come up several times is folks asking about best practices for Istio Gateways. Specifically, should you have one Gateway or many? What about Ingress Gateways?

In keeping with tradition, a standard issue "it depends" is the correct answer to these questions, but if you just need a quick suggestion to get you started here you go.

• Gateways: you should probably have as few as you can get away with, but also it probably doesn't matter.
• Ingress Gateways: you should probably have one.

Ingress Gateways

An Istio Ingress Gateway is a Kubernetes Deployment that consists of just an Envoy proxy. Unlike the other proxies in the mesh which are bolted on to your workloads in sidecar fashion, the Ingress Gateway proxies sit on their own at the edge of the mesh accepting outside traffic and directing it inwards, like a load balancer in your cluster.

It may seem at first that having multiple Ingress Gateways can be a strategy for high availability, but a single Ingress Gateway Deployment can be as highly available as any other Deployment by way of multiple replicas across multiple zones and by scaling in response to traffic by way of its Horizontal Pod Autoscaler.

If your traffic scale is truly large or your isolation needs are extreme, then like with traditional load balancers those could be reasons to have more than one Ingress Gateway, but if you can't think of any specific reasons right now then start with one.

Gateways

Unlike an Ingress Gateway an Istio Gateway does not map to a Kubernetes Deployment. It is purely an Istio configuration object which Istio consumes and uses to program the lower level Envoy config on your behalf.

Specifically, a Gateway will inform the Envoy Listener configuration. But unlike Istio Virtual Services which also control Envoy Listeners throughout the mesh, a Gateway object will only apply to those Envoy Listeners on the Ingress Gateway Deployment.

Ugh, this stuff is terminology word salad.

Basically you can think of an Istio Gateway object kind of like an nginx server block with fewer options. It tells the Ingress Gateway Deployment:

• which port(s) to listen on
• which protocol(s) to listen for
• which hostname(s) to handle requests for
• which TLS certificate(s) to use

Where I once configured this blog with an nginx server block like this.

server {  
  listen 443 ssl;
  server_name rob.salmond.ca;

  ssl_certificate /etc/letsencrypt/live/rob.salmond.ca/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/rob.salmond.ca/privkey.key;

  location / {
    proxy_pass http://localhost:2370;
    proxy_redirect off;
  }
}

I now use an Istio Gateway config object like this.

apiVersion: networking.istio.io/v1beta1  
kind: Gateway  
metadata:  
  name: my-gateway
spec:  
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - 'rob.salmond.ca'
    port:
      name: http
      number: 80
      protocol: HTTP
  - hosts:
    - 'rob.salmond.ca'
    port:
      name: https-default
      number: 443
      protocol: HTTPS
    tls:
      credentialName: ingress-cert
      mode: SIMPLE

Note that routing traffic to the NodeJS app (localhost:2370) is not handled here. Istio uses Virtual Services to configure the matching of requests to their corresponding apps. Also rather than a full path to a file the TLS certificate is loaded from a Kubernetes Secret called ingress-cert.

Great but how many Gateways do I need?

If you're just serving traffic for a single domain, or even a wildcard domain, you only need one Gateway. By separating the routing configuration into Virtual Services Istio just needs your Gateway to indicate that it is hosting *.sweetdomain.com while you use Virtual Services to route traffic for www.sweetdomain.com, api.sweetdomain.com, signup.sweetdomain.com, etc.

If you're serving traffic for a lot of different domains you can either pack them all onto a single Gateway object by appending more hosts entries onto the servers list, or create a separate (and more succinct) Gateway object for each and every domain. Which way you go will be a combination of taste and the limitations of whatever god awful YAML templating tool you find yourself chained to.

Why is it a matter of taste? (or why does it "probably not matter"?)

Because with both of these approaches, under the hood Istio is going to turn these configurations into Envoy Listeners for you, and there will only be one Listener for each unique host:port pair. For your average web app this means that no matter how many Gateways you create, or hosts you add to one Gateway, you will end up with just one 0.0.0.0:8080 Listener and one 0.0.0.0:8443 Listener (these port numbers in Istio are += 8000 to avoid running the proxy as root).

For every host you add Istio will append its configuration ² to the corresponding Listener. When requests come in on that Listener, Envoy will either look at the Host or Authority headers or use SNI inspection to figure out which domain the connection is destined for then pass it off to the corresponding Envoy Route (where Istio has sent your Virtual Service configurations to handle request matching) for forwarding to the correct place.

I suppose that regardless of which approach you use to add all your different domains you could wind up with a Listener config so large it blows Envoy out in some interesting way, but if you get to that point then I'm afraid you will need more insight than this blog post can offer.

For some years now I have been saying that (pre 0.12) Terraform is better thought of more like XML than a DSL. Originally the project founder wanted to keep HCL "as simple as possible" and the more familiar programming language like constructs of looping and conditionals were conceded to later in the projects life. However since these were addons and the project was not designed with them in mind they lead to some scary issues. That one in particular bit me hard in a way that deleted IAM accounts for 2/3rds of our engineering team.

While working with 0.12 though I have managed to add three dramatic improvements to some of our modules by leveraging some awesome new features.

First some background. We have a module which creates GKE clusters and due to the varying needs of our projects sometimes we want one with a default node pool and sometimes we don't. (It doesn't matter if none of that means anything to you, we'll go with simple examples).

Like many Terraform resources, the config for a GKE cluster is complex and requires both arguments and config blocks.

resource "my_resource" "foo" {  
  argument = "value"
  config_block {
    config_a = 1
    config_b = 2
  }
}

It has always been straightforward with Terraform to pass a variable in and assign it to a value, but until 0.12 it was not possible to optionally specify a config block. It's either on the resource or it's not. In our case we wanted to support the creation of resources which had mutually exclusive combinations of arguments and config blocks. So the workaround looked like this.

resource "my_resource" "variant_a" {  
  count    = "${var.enable_variant_a ? 1 : 0}"
  argument = "value"
}

resource "my_resource" "variant_b" {  
  count    = "${var.enable_variant_a ? 0 : 1}"
  config_block {
    config_a = 1
    config_b = 2
  }
}

In this way we can achieve the desired result, but it's kinda a gross workaround that results in code duplication and is generally the sort of thing you find all over a complex legacy Terraform code base.

We also needed this module to output the name of the resource regardless of which variant was created so we defined an output.

output "resource_name" {  
  value = "${var.enable_variant_a ? join("", my_resource.variant_a.*.name) : join("", my_resource.variant_b.*.name) }"
}

Hideous.

This is where we introduce our first dramatic improvement with a new Terraform 0.12 feature, string templates.

The addition of conditional directives in strings allowed us to take the above monstrosity and change it to this.

output "resource_name" {  
  value = "%{ if var.enable_variant_a }${my_resource.variant_a[0].name}%{ else }${my_resource.variant_b[0].name}%{ endif }
}

Still burly but infinitely more readable.

Our second dramatic improvement is enabled by combining three new features, complex types, the new null type, and dynamic blocks. Using these we eliminated the need for duplicate resource variants.

First we constructed a complex type to describe all the configuration required for our resource.

variable "my_resource_config" {  
  type = object({
    name                = string
    location            = string
    enable_variant_a    = bool
    config_block = list(object({
      config_a = number
      config_b = number
    }))
  })
}

Note the inclusion of the mutually exclusive options. Also note that this variable definition needs to exist both in the module as a definition of the interface to the module and at the calling site so a correctly typed instance of the variable can be populated. I opted to separate this complex type from the other variables being passed to the module and put it in it's own file so I could symlink it into the calling Terraform space to avoid duplication (we use symlinks as Terraform "include" statements a fair bit).

Next we populate the values of this variable type in our terraform.tfvars file.

my_resource_config = {  
  "name"                = "sweet_resource"
  "location"            = "us-central1"
  "enable_variant_a"    = true
  "config_block"        = []
}

Finally we add the dynamic block to the resource itself.

resource "my_resource" "resource" {  
  name     = var.my_resource_config.name
  location = var.my_resource_config.location
  argument = var.my_resource_config.enable_variant_a
  dynamic "config_block" {
  for_each = var.my_resource_config.config_block
    content {
      config_a = config_block.value.config_a
      config_b = config_block.value.config_b
    }
  }
}

Note that to access the elements of the config_block on our complex type we reference the name of the dynamic block (also config_block) followed by .value followed by the element name. If this is confusing you can add the line iterator = some_other_name to use in place of the name of the dynamic block.

With this approach the values we've set in my_resource_config will cause the boolean value to be passed to our argument to enable whatever feature we want and the empty list in the config_block field will mean the for_each loop in the dynamic block never runs and the block is not added.

To flip this around we define the config this way.

my_resource_config = {  
  "name"                = "sweet_resource"
  "location"            = "us-central1"
  "enable_variant_a"    = null
  "config_block"        = [
    "config_a" = 1
    "config_b" = 2
  ]
}

The presence of the null type means that Terraform will not actually set the value of the argument and will not complain about a type mismatch, while the presence of a list of objects in config_block means the loop will now execute and the dynamic block is populated. Note that there are some resources which support multiple copies of a dynamic block (eg. ingress rules on a security group) and in those cases you can just add more elements to the list.

This second improvement also means that the first improvement with the string templates actually gets tossed out because we no longer have multiple variants of our resource we can just output my_resource.resource.name, another win!

The third improvement happened on another module that combines the elements of the second improvement with the addition of a for_each construct on a resource itself as well as on a dynamic block.

For this resource our config object looks like this.

variable "my_resource_configs" {  
  type = map(object({
    name             = string
    location         = string
    enable_variant_a = bool
    config_block = list(object({
      config_a = number
      config_b = number
    }))
  }))
}

Note that in contrast to our first config object the root begins with a map of objects rather than an object. This allows us to pass multiple configs in and iterate over them.

my_resource_configs = {  
  "default" = {
    "name"             = "default"
    "location"         = "us-central1"
    "enable_variant_a  = null
    "config_block" = [
      {
        "config_a" = 1
        "config_b" = 2
      }
    ]
  }
  "variant" = {
    "name"              = "variant"
    "location"          = "us-central1"
    "enable_variant_a   = true
    "config_block"      = []
  }
}

And in our module we define our resource like so.

resource "my_resource" "resource" {  
  for_each = var.my_resource_config
  name     = each.value.name
  location = each.value.location
  argument = each.value.argument

  dynamic "config_block" {
    for_each = each.value.config_block
    content {
      config_a = config_block.value.config_a
      config_b = config_block.value.config_b
    }
  }
}

Now we iterate over each of the config items safely (we could use this approach to create IAM accounts and not nuke them when somebody quits and we remove an username from a list of users passed in) and when we come to the dynamic block we further iterate over the elements of that list.

These improvements only touched two Terraform modules in our codebase but resulted in the elimination of over 150 lines of legacy boilerplate and workarounds. Probably my favourite pull request to date

I recently discovered an rtl-sdr project of mine had stopped receiving data. After a bit of investigation it turned out the ppm error value I had previously calculated seems to have drifted over time.

PPM is a unit used to measure the variance in a crystal oscillator between the frequency it is meant to operate at and the frequency it actually operates at. When using an rtl-sdr you need to calculate this variance and account for it in your tuning software, eg. rtl_fm -p <ppm goes here>.

Apparently temperature is a factor and it has been chilly lately so I'm guessing that's the culprit.

The first time around I had followed these instructions and computed the PPM for my tuner manually. Since the project I'm operating doesn't need to operate at night (because the boats aren't running) I started thinking that with some scripty goodness I could just recalibrate the tuner nightly to avoid future downtime.

So I hacked something up to do just that, again using the kalibrate-rtl program which computes PPM by checking known base frequencies of cell phone channels. My script checks the three strongest nearby cellular channels three times each and then compute the average.

The problem is it doesn't work. Each cellular channel it checks corresponding to a separate cell tower was producing a wildly different PPM value from the others. It turns out I'm not the first person to notice this. Basically the reason you can't use these supposedly "known good" base signals to calculate error from is that they also contain a small amount of error and the errors are different from tower to tower.

I found references in various HAM radio forums about using NOAA weather signals to manually compute PPM as they're known to be reliably well tuned, however it wasn't something I could easily script so I abandoned that approach.

As it turns out with some further reading I discovered that the RTL-SDR library itself ships with a program to compute the PPM error using a completely different approach. rtl_test computes the PPM error by comparing the the oscillator in the radio to the oscillator in your computers clock simply by checking the time repeatedly. Much simpler, faster, and reliable!

For this valuable contribution the author received only a paltry six upvotes on his post in the RTL-SDR subreddit. Turns out the same guy also wrote the rtl_fm program I'm using to tune my project as well as a program called rtl_power which, among other things, can be used to do passive radar tracking.

Bad ass!

So for helping me fix my project and for all your awesome contribution to the RTL-SDR community thank you Kyle Keen!

Then I came to a challenge that required implementing and mining a blockchain. Given a particular genesis block, difficulty requirements, and a hash algorithm you had to provide a chain of at least four valid blocks to retrieve the flag.

Specifically, the genesis block looked like this.

{
"identifier": "000102030405060708090A0B0C0D0E0F", 
"nonce": 3754873684,
"data": "Genesis Block for CTF contest, all block chains must start with this block. This is equivalent to Big Bang, time didn't exist before this",
"previous_hash": null
}

The hash algorithm looked like this.

def hash_block(block):  
    message = hashlib.sha256()
    message.update(str(block['identifier']).encode('utf-8'))
    message.update(str(block['nonce']).encode('utf-8'))
    message.update(str(block['data']).encode('utf-8'))
    message.update(str(block['previous_hash']).encode('utf-8'))
    return message.hexdigest()

And the difficulty for each block was pre-assigned in the following order: 8 for the genesis block, then 4, 4, 5, 6, 7, 9, 11, 13, and 16 for the remaining blocks. In this case difficulty is defined as the number of leading zeros in the resulting hash.

The content of the data field was dealers choice, so I populated with a poem I'm fond of and implemented the miner over my lunch break. The result produced about 100k hashes per second on my macbook, randomly hashing the block with a different nonce value until hitting upon one which produced a hash that satisfied the difficulty requirement. It completed the required four blocks before I had finished eating. I submitted the chain and collected my flag and went back to work.

But one thing kept bothering me. I kept thinking about the fact that even though it took only four blocks to collect the flag the instructions provided difficulty values for a ten block chain. What would happen if I submitted all ten blocks? Bonus points? A hidden challenge? My weight in dogecoins?

I stopped working on all the remaining challenges and focused on completing the chain. Since the new gig is a Go heavy shop and my Go game is weak I decided to port the miner to Go for a compiled language speed up. First step was to see if I could replicate the hashing algorithm which turned out to be pretty straightforward with the standard library crypto/sha256 package.

func (b *Block) hash() []byte {  
    h := sha256.New()
    h.Write([]byte(b.Identifier))
    h.Write([]byte(strconv.Itoa(b.Nonce)))
    h.Write([]byte(b.Data))
    h.Write([]byte(b.Previous_hash))
    return h.Sum(nil)
}

I can see why Python people like Go, it's pretty intuitive. This bought me about an 8x speed up. My macbook was churning out 800k hashes per second which ripped through the next few blocks rapidly up until it hit block 7 with a difficulty rating of 11, and there it stayed for quite a while.

Recalling back to the early days of bitcoin mining I decided that a mining pool seemed like a good approach. I have nerdy friends, they have computers, by our powers combined we can do a thing right?

So I implemented a quick and dirty web app to loosely coordinate a distributed pool of miners. I made sure the miners would generate the same identifier field for a given block where previously it had also been random, so the fleet would be working on the same problem. Then I had them poll the server every thirty seconds to see if anyone else had mined a block. Stumbling about with my new Go legs it took about a day and a half to get the server and the miner working in tandem.

On Friday the 3rd I started soliciting friends to run the miner.

I was testing the pool with my work macbook, my personal laptop, and my desktop. All told these were combining to produce about 2.4 million hashes per second. There seemed to be some upper limit at the 800k mark that didn't seem to vary much with CPU speed. As friends came on board the pool hashrate began to climb. Slowly.

The ones with beefy gaming PC's complained that their many cores remained idle while mining so I set about learning how to use Go's concurrency to make use of them. Eventually I got it working after faffing about with channels and worker pools and whatnot. The code is here and it's hideous and no doubt rife with bugs. I haven't had to deal with pointers since the 90's and my approach is roughly on par with one that a friend once half jokingly suggested, "Keep adding asterisks and ampersands until it works".

Anyway it's ugly but functional, the concurrent miner spawned a dedicated goroutine for each available core on the system and did a great job pinning the entire CPU to 100% usage.

We went from this.

To this.

Somewhere around this point on the evening of the Nov 4th I decided to start graphing the hashrate of the mining pool.

We were doing better but for the next two days the entire pool churned and churned on block 7 getting nowhere. I decided to stand up a bunch of cheap google cloud compute instances to help the effort. My friend Adam had a bunch of cores sitting around from a screw up with a cloud provider ages ago so he fired up a dozen miners in there. At its peak the pool hit nearly 50 million hashes per second. By then I'd had to fortify the web app with more workers and a proper MySQL rather than just SQLite.

Some time around 8 am on Sunday the 5th a google instance in South America with miner id 662F4F146E1504EA mined the elusive block 7 and the whole fleet rolled over to working on block 8 with difficulty level 13. At this point we had three days to go till the end of the CTF contest. People had been suggesting it for a while and I'd been hoping to avoid it but I finally decided to cave and take a serious look at implementing a GPU accelerated miner.

I spent all day Sunday and most of Monday evening tinkering with a variety of GPU based sha256 implementations, even getting a little twitter help from one author.

I managed to get a hash actually computed on my GPU but writing an algorithm to run at GPU scale parallelism is pretty far from the sort of software I normally write. I was just beginning to see how I could divide the work up and was just skirting the outlines of how I might get meaningful answers back out, but with time running short and brain cells in short supply I sputtered to a stop a day before the contest ended.

Feeling pretty burnt out I threw up my hands in defeat. What I did learn was a fair bit about Go, who my most competitive friends are, and way more than I ever needed to know about how sha256 works. For the curious, here's my tl;dr.

Remember these puzzles?

That's pretty much how sha256 works. When you fire it up the innards are arranged in a standard configuration hand crafted from the finest artisanal entropy. Then the input data is padded to ensure its size is a multiple of 512 bits and it is shoved through the system 512 bits at a time. Each block of bits turns the crank and slides the configuration around in a consistent but input dependent way ensuring that if even one input bit is altered the sliders wind up in wildly varied final states.

And that's it. See? Cryptographically secure hashes aren't that hard. Now whatever you do don't go read FIPS-180. That way lies total madness.

I hope the CTF is this much fun next year.

What I found however is that one of the pages I was monitoring had a dynamically generated <script> tag in it which was triggering spurious notifications I wanted to suppress. There didn't seem to be an obvious way to ignore particular tags so I created a simple hook to do this.

from urlwatch import filters  
from urlwatch import jobs  
from urlwatch import reporters  
from bs4 import BeautifulSoup

class IgnoreFilter(filters.FilterBase):  
    __kind__ = 'ignore'

    def filter(self, data, subfilter=None):
        if subfilter is None:
            return data

        soup = BeautifulSoup(data, 'html.parser')
        for element in soup.select(subfilter):
            element.extract()
        return soup

This adds a new filter type called ignore which accepts a CSS selector as parameter. It then uses the magical BeautifulSoup HTML parser to find all the elements which match the selector and remove them before returning the remaining HTML.

Urlwatch then does its normal comparison against the previous run to see if anything has changed and carries as usual.

To use the filter update your config like so altering the CSS selector to suit your needs.

$ urlwatch --edit

---
name: "some site"  
url: "https://something.com/"  
filter: "ignore:body > script:nth-of-type(2)"  
---

This ignores the second <script> tag beneath the <body>.

In the weeks since submitting this project I've had a couple friends mention that they would like to learn more about working in docker so I've extracted the good bits and put them in a public repo. Here I'll describe how it works and how to use it.

The Flask App and Configuration

I reached for Flask to build the web app as that's my go to framework and I wanted to build a strong submission, go with what you know as they say. I have replaced the business logic from the actual assignment so as not to provide reference material for future applicants but the structure is the same.

If you're looking to learn Flask there are better projects out there. I recommend Overholt an oldie but a goody. Cookiecutter Flask which is more geared towards full web apps than APIs. Or my favourite Flusk, a clean, fairly modern, and well organized Flask boilerplate.

The only thing worth mentioning with respect to docker is the way configuration is handled. There is a bit of extra logic in there to deal with the MySQL replicas (more on that below) but basically it just grabs the value of any environment variables prefixed with HELLO_ and hangs them off the Flask config object. I took this approach because environment variable injection is the baseline approach for passing config into a running container both in docker-compose and practically every container orchestration system. This gives us an easy on ramp to move from dev to prod.

If you find yourself baking config files into your containers, or having some script in your container fetch a config file from somewhere you're gonna have a bad time. In that case just go straight to making your app Consul or etcd aware and be done with it.

To make use of the read-only replica I used this flask-replicated extension which is a bit naive in that it uses the HTTP method rather than the database operation to decide which database to execute the query on. For example if you had some user.last_accessed_on datetime field that got updated on every page view this wouldn't cut it but it gets the job done for this simple app.

The Dockerfile

One thing of note regarding the dockerfile is the separation of the requirements.txt file (the python version of a Gemfile or a package.json file) from the rest of the app in terms of layers. Since each ADD statement creates a new layer in the image and minimizing the number of layers is best practice this may seem counter intuitive.

The idea here is to speed up build times. The build process will only rebuild those layers which have been modified since the last build, however it must then rebuild any layers built upon the modified layer. By placing the requirements.txt layer above the layer for the rest of the app code we ensure that rebuilding that layer (and the subsequent apt-get install ... pip install ... layer) only happens when the requirements change. Without this separation every single line of code we changed would mean a tedious rebuild of those layers.

The good news is that you don't need to rebuild the container every time you change the code though, next we'll look at how to hack in this environment.

The docker-compose and override files

This is where much of the development magic happens, using docker-compose we can stand up all the dependencies our app(s) rely on. In this case two MySQL containers in a master/replica configuration (courtesy of Tao Wang) and a memcached container.

There are a few things worth highlighting here. First is the use of healthcheck and restart directives. These will both let you know if your services become unreachable for some reason and try to restart them in case they stop. Useful in dev when trying weird stuff is a common occurrence.

Next and more important is the use of explicit app level configuration for connecting to the services in the supporting containers, in this specific case by providing environment variables for memcached host and MySQL URI strings but this could be any app level config.

When linking containers via the docker-compose depends-on mechanism the Hello app could simply default to looking for the hostname master or memcached which would resolve to the correct container. However the pattern of using code level dev-default values, be they service dependencies or feature flags or basically anything that might be different in production, creates a minefield of unknown unknowns when it comes time to ship your containers.

By explicitly specifying these configurations during development we have a roadmap to follow when we deploy to Kubernetes or ECS or whatever else down the road. Believe me when I say that reverse engineering this sort of config without a guide sucks.

Finally we should look at the docker-compose override file. By default docker-compose will parse the main docker-compose.yml file and then update the config it finds there with any additions or changes it finds in docker-compose.override.yml. We can leverage this mechanism to provide a nice developer experience by setting up the primary docker-compose file with the assumption that every container in the stack will behave normally (that is start running the app it hosts) when it comes up. Then we can use the override file to knock out any container we care to hack on by replacing the command directive so that rather than running the app it just keeps the container running indefinitely, and adding a volume directive so that rather than using the source baked into the container it reads our local copy on the host OS so we can hack using our preferred editor.

This turns the container into our dev system, hooked to all the dependency containers, isolated from our host OS, and fully loaded with all the libraries our app depends on.

We can then hop into the running container to interact with our code as we update it by using the docker exec -it <container_id> /bin/bash command. Or in this case we can use the make target for just this purpose and instead run: make dev.

The Makefile

This provides a lot of convenience tools for interacting with the dev environment. This could be done with any similar tool like rake or grunt or yarn or whatever the cool kids are using now.

Some useful patterns are things like the DB migration target make setup-db. This starts up the db containers then manually runs the app container with the necessary parameters to link with the databases and execute the initial migrations. This could be done with yet another docker-compose override file but those grow numerous quite quickly. Note that this pattern is the reason the docker network is created externally (by make setup) rather than implicitly by docker-compose, so that we can link stand alone containers to those running in docker-compose.

Other handy dev targets are make testdata for generating canned API calls to our app and make nuke to completely blow the environment away when we inevitably screw it all up.

Final Thoughts

As usual this is mostly an exercise in capturing my thoughts for future reference but hopefully someone besides future me will find this helpful. I intend to use this repo as boilerplate for new projects so it should see at least a bit of upkeep here and there as I hack on stuff. I have also been tinkering with redeploying some of my personal projects in containers so I will likely have a follow up post sooner or later about the trip from dev to prod.

Also, I got the job so I guess I must have done something right!

The mobile app has some basic functionality built in, set the colour based on the temperature or the price of your favourite stock symbol and that sort of thing. It also has IFTTT integration which is a handy service I've used for many years. Unfortunately it's not quite as useful as I'd like. For example if you trigger a "blink the light" event by some action the light just keeps blinking until you intervene which I'm not thrilled about.

But then if you got a problem with a hackable light you go ahead and you hack it.

I noticed that the response time between my pressing a button on the mobile app and the light updating itself was extremely fast, so fast that I assumed the app had to be communicating with the light directly over wifi. I fired up ettercap and ran a MITM attack between my phone and the light (MITM on a light, this is the world we live in?) but found no traffic going between them.

Next I ran it between the light and my router looking for it's control channel. I found that it established a TCP connection to imp02b.boxen.electricimp.com:31314 but couldn't make heads or tails of the binary protocol in use so I moved on to the app. Unfortunately the mobile app was communicating via HTTPS so I switched over to mitmproxy so I could see inside the encrypted traffic.

Here's what I found.

The app was doing a nice simple POST of some straightforward form data to control the light. During setup each unit produces a unique device code which is used to identify it, this is the URI being addressed.

It seemed like it'd be easy to replicate so I set about curling to see if I could do it but it ended up getting late and I found myself frustrated by countless 404 responses. I replicated every header, the user agent, even went so far as to script up a request that would copy the lower case 'h' in the host: header with no success.

My request looked absolutely identical and still wouldn't work. I gave up and went to bed.

As is often the case when I looked at the problem with fresh eyes today the answer jumped out at me. The Content-Length in my spoofed request was way off, lots more data than the app was sending. I compared the payloads in hex and found the problem.

The trick it turns out is to send the payload with an application/x-www-form-urlencoded header but not actually URL encode it. I couldn't figure out how to make cURL or my HTTP client library of choice do something so stupid but of course urllib2 has so such qualms.

import urllib2  
import json

YOUR_DEVICE_CODE = '<thing goes here>'

URL = 'https://agent.electricimp.com/{}'.format(YOUR_DEVICE_CODE)

def set_color(color):  
    data = {"program":"Demo","color":"#{}".format(color)}
    req = urllib2.Request(URL, json.dumps(data))
    response = urllib2.urlopen(req)
    foo = response.read()

if __name__ == '__main__':  
    set_color('FFFFFF')

And since it works from anywhere on the internet now I can activate disco mode when I'm not even home!

Since I'm often mentioning interesting things I've watched I've had a few friends recently ask me for channel suggestions so here we go. My first listicle! In no particular order here are a bunch of youtube channels that I find interesting.

The Economist

Better than you'd think, even if you are interested in economics already.

Talks at Google

Google hosts a dizzying array of experts to give talks on a wide variety of subjects.

Crash Course

Learn everything.

Philosophy Tube

Just what it says on the tin.

School of Life

Just what it says on the tin.

Vice and Vice News

Journalism with f bombs, shit you won't see on MSNBC.

Singularity Lectures

A mix of scientists and dreamers speculating about how current state of the art might play out in the future.

Healthcare Triage

A pracicing pediatrician, researcher, columnist, and author sets you straight about health care.

Science Magazine

For real. The actual Science magazine.

The Royal Institution

For real. The actual Royal Institution.

Vox

Context on current events.

Caspian Report

This guy explains history and geopolitics. If the news tells you what happens, he can tell you why.

Channel Criswell

Digging deep into cinema.

Now You See It

Digging deep into cinema.

Every Frame a Painting

Digging deep into cinematography.

History Buffs

Comparing historical cinema to historical facts.

Cold Fusion

Sorta tech history focused, back stories on tech companies, trends, press releases, etc.

Complexity Academy

Complexity itself is a thing.

Stated Clearly

Evolution and the latest ideas on the origin of life laid out in simple terms.

Deep Learning TV

Machine learning explained clearly.

Derren Brown

A mentalist and skeptic screws around with peoples heads.

Motherboard TV

A tech blog for people who are wary of tech.

Nerdwriter

Thoughtful video essays about current events and popular culture.

Samy Kamkar

Hacking explained.

SciShow Space

Space is dope.

The Brain Scoop

Behind the scenes at one of the worlds best natural history museums.

Tom Scott

He goes to cool places and talks about things you might not know.

Wired UK

For some reason better than the main Wired channel.

PBS Space Time

The most hardcore space and physics show I've ever seen.

After experimenting with using nothing but chromebooks for a while to see how well I could operate with just a shell, browser, and cloud, I treated myself to a top of the line thinkpad last year and decided to give a tiling window manager a try and installed i3wm.

I've become fairly proficient with it and am quite happy with the user experience, but with two external monitors at home for easily moving workspaces around I really miss them when I'm out and about.

I also recently spent the holidays visiting family and as usual performed a bit of tech support. Couldn't figure out what was up with the iPad I gave my grandad last Christmas so this week I went out and got him a new one. Having it sitting around today and having plans to meet a friend for an afternoon of hacking at a local coffee shop I wondered if I could somehow use the iPad as an external monitor.

Turns out you can!

The trick is to use xrandr to define a virtual display device and then xvnc with the -clip flag restricting the shared viewport to the size of the virtual display to make it remotely visible. Then any old vnc client on the tablet will do the rest.

This is my config for an iPad 4.

phro@shard:~/.screenlayout$ cat tablet.sh  
#!/bin/sh
xrandr  --output VIRTUAL1 --mode 848x1080_60.00 --right-of eDP1 --output eDP1 --mode  
 1920x1080 --primary --pos 0x0 --rotate normal
x11vnc -clip 848x1080+1921+0  

Here's how it looks.

It works pretty great at home but out and about there are a few problems to figure out. First, many public wifi hotspots will dynamically create a small /31 network for each client which joins to prevent hostile users sniffing / spoofing / whatever the other folks. In that situation the tablet won't be able to connect to the VNC server, so I had to do a little network hopping before I could use it.

Another issue is latency. One network I tried out despite having ping times in the 10-20ms range to Google's DNS servers it was producing numbers > 1 second between devices on the network. Using the bloated uncompressed VNC protocol it took 15-20 seconds for a window moved onto the tablet to appear.

Also configuration is a pain in the ass. The VNC client I installed on the tablet has a bookmarking feature that let's you save hostnames and credentials for commonly accessed servers but of course if you're on a strange network the IP of the laptop will change meaning it's a bit less seamless to get started.

To work around all this I'm planning to grab a low profile USB wireless adapter to set up an ad-hoc network between the tablet and the laptop. And of course, my own tablet since I'll be shipping this one off to grandpa this week.

I wanted a simple tool for a simple task so I wrote one and called it datahoser.

It's built atop the SQLAlchemy reflection system, creates databases, tables, and inserts rows, and has a simple but thorough verification step after data has been copied.

It's not quite ready for prime time as it relies on an unreleased SQLAlchemy bugfix and could use a bit of tidying up but it functions as advertised. It also has some examples of how to convert between non-native data types in case your source database uses a type not available in the target DB. In theory it should copy data to or from any RDBS supported by SQLalchemy though I've only tried it on sqlite and mysql so far.

I'll throw it up on pypi when I'm able. If you need to use it lemme know how it goes.

In unrelated news my blog turned ten years old yesterday. I still agree with my original assessment. Accelerando is a hell of a book.

Last week I visited San Francisco for the first time. I was there for work and those obligations consumed the bulk of my time but I arranged to spend a couple extra days in town to play tourist. Friday was the first day I had free so I did the typical stuff. Walked the Embarcadero. Stuck my nose in touristy shops. Ate greasy food. Checked out Pier 39. Took a photo of the golden gate bridge.

Walked up and took a look at Lombard street, said "fuck that" and went elsewhere.

That evening I met an old colleague for drinks at the Adler Museum cafe in North Beach, a fantastic dive. I also drank some wretched stuff the locals drink called Fernet, it was awful. Stay away from it.

After he went on his way and another coworker who had spent the afternoon wandering about with me headed to the airport to fly home I set about tackling one of my favourite things to do in any city, but specifically a new city. Finding a bar to knock back some drinks and make friends.

I wandered into Buddha Bar in Chinatown, ordered a cocktail and a shot, and struck up a conversation with a couple visiting from Florida. We tried to make sense of a crazy game called "Liars Dice" the bartender was teaching anyone who seemed interested. After they left another group of folks sat down next to me and started playing. We chatted. I ordered more shots and cocktails. They invited me to join them in crashing some party at a nearby hotel.

I accepted. The night was starting to get interesting!

I recall walking some distance to this party. It turned out to be a fancy halloween party at a really nice hotel. I was very underdressed but it didn't seem to matter. The details of this party are somewhat obscured at this point by the copious rounds I'd been sharing with my new friends. I recall talking to a woman dressed as a bird and a guy in a gorilla suit buying me a glass of some scotch that seemed ludicrously expensive.

At some point I decided to say goodnight and head back to my AirBNB. Outside the hotel was a queue of cabs, I grabbed one and left.

Without my wallet.

In a brilliant stroke of luck, at some point in the evening I had bought a drink with my credit card and slid it into my hip pocket instead of back into my wallet. When the cabbie realized I had no cash to pay him he took me to an ATM at a grocery store to try to get a cash advance. My Canadian card wouldn't play ball with the American ATM.

The cabbie left me there.

I'm not exactly sure why I went back downtown at that point, maybe I was hoping to find the hotel and try to locate my wallet. Maybe I could get into the office and crash on a couch. At any rate I had no idea where I was or where my AirBNB was and for some reason I aimed for the biggest buildings I could see and started walking.

For two hours (more on how I know that later).

Somehow I wound up on Russian Hill back in North Beach, then I wandered downtown. My phone was dead by this point. I was sobering up and exhausted, I'd walked almost 20km that day. I started to try to decide by what criteria I should select a doorway to pass out in.

I spotted a guy standing around outside a fast food joint with some friends looking at his phone and in a moment of desperation I approached him and asked if he could look up directions to where I thought my AirBNB was. He graciously did so and informed me it would be a fifty minute walk back to Potrero hill.

He then did an amazing thing and hailed me an Uber and sent me safely back to sleep the night off. I recall only that his name was Sean, I owe you big time man, thank you! I reached out to Uber to see if they can figure out who this generous soul was so I can throw him a few bucks for saving my tail. If they work it out I'll update this post.

Hungover the next morning I set about calling and cancelling all my cards. I also cracked open my laptop to check in for my flight home that evening but found I wasn't able to. I double checked the booking.

I'd missed my flight home. It was actually booked for the night before. No idea how I bungled that one.

In the middle of calling the bank back to desperately ask them to un-freeze my now locked accounts so that I could try to get a new flight, the host of the AirBNB knocked on the door to tell me I'd stayed far past check out and she needed to clean up for the next guests.

It is fair to say that at this point I was freaking out.

I apologized and threw on some clothes, grabbed my bags and was out the door in minutes. I wandered to a nearby diner to sit down and make a phone call and ask somebody for a really big favour.

I called my mother. "Hi Mom, I'm in trouble. I'm trapped in a foreign country with no money. Want to buy me a flight home?".

The conversation evolved from there but we hit a snag trying to purchase the flight, some anti-fraud thing was mucking things up.

I called my brother. "Hey bro, I'm in trouble ..."

He and his wife sorted me out and I made it home later that day. Thanks you guys, I owe you big time!

After a quick change and a shower at home I headed out to a Halloween party to tell the story of the unfortunate and mysterious night out. Lots of questions came about that I didn't have answers to. Who were the people who'd brought me to the party? What hotel was it in? Where had the cab dumped me?

I realized today that I might have some photos in my phone from that night, possibly even geotagged. I checked but all I had were a few blurry shots of the Buddha Bar.

It occurred to me though that Google might know where I'd been, and it turns out it did. Google location services was turned on in my phone so there's a detailed log of my movements.

Here I am walking from Buddha Bar to the fancy party, as it happens my memory of that was pretty accurate. The Fairmont San Francisco is a beautiful hotel!

Here's me about an hour later in a cab, turns out he did take me to where I was staying and then mere blocks away to try to get some cash.

Once he abandoned me there (I do feel bad about burning him on the ride) if I'd known where I was I could have walked back in minutes. Instead here's what I did for the next two hours.

Yes I did wander mindlessly through the infamous Tenderloin in the wee hours of the night despite being repeatedly told to stay out of it. Nothing interesting happened though, another stroke of good luck.

It turns out I actually fairly faithfully retraced my steps. Never did find my wallet, I called the Fairmont today. No luck there either. Ah well. I got a wild story to tell at least.

If you want to see what google knows about where you've been you can try it out for yourself.

FLOSS is everywhere."