Contaminate AWS instances on ssh login

One of the principles of running large numbers of instances is that consistency is key. Config deviations cause oddities that’ll drain your time with investigations and nothing causes entropy on your hosts like an admin investigating an issue. In this post we’ll configure our instances to mark themselves as contaminated when someone logs in. We can then use other tooling to query, collate and probably reap, machines corrupted by the keystrokes of humans.

While the example here is step-by-step and interactive, you’d normally bake this in to your AMI or deploy it very early in your config stage, possibly using cloud-init. For our test we’ll spin up an instance and grant it an ec2 instance profile so it can alter its own tags.

In terms of moving parts we’ll install the awscli package, add a short script that’ll tag the instance when run and configure PAM to invoke the script when an ssh session opens to the machine.

# install required dependency
sudo apt-get install awscli
cat > /usr/local/bin/add-dirty-tag <<EOF
#!/bin/bash

[ "$PAM_TYPE" = "open_session" ] || exit 0

INSTANCE_ID=$(ec2metadata --instance-id)

REGION=$(ec2metadata --availability-zone)
REGION=${REGION%?} # remove the last letter

aws ec2 --region $REGION create-tags --resources $INSTANCE_ID --tags Key=contaminated,Value=true
EOF
sudo chmod a+rx /usr/local/bin/add-dirty-tag

Now we have a script to add the ‘contaminated’ tag to our instance we’ll configure PAM to run it when a new ssh session starts. On a Ubuntu system the config should be placed in /etc/pam.d/sshd.

# tag the machine as contaminated if anyone sshs in.
session    optional     pam_exec.so /usr/local/bin/add-dirty-tag

It’s worth opening another ssh session and logging in to confirm this works. That will leave you with an established connection in case you misconfigure PAM in some way. Once you’ve successfully logged in and caused the new tag to be added to the instance you can run a cli filter from outside the instance to show all hosts that have been interactively connected to:

aws --region eu-west-1 ec2 describe-instances             \
    --filters "Name=tag:contaminated,Values=true"         \
    --query 'Reservations[].Instances[].{id: InstanceId}'
[
    {
        "id": "i-x134x34x"
    }
]

If you decide to adopt an approach like this you can expand the values stored in the tag using the values PAM exposes, such as $PAM_USER or $PAM_RUSER and a time stamp. There’s also nothing stopping you from adding something more structured. A concise JSON dict maybe. Just be careful that you don’t overwrite the details on each successive login.