A bastion with no open ports

I needed to reach a private Postgres database from my laptop, so I built a little bastion to tunnel through. The interesting thing about it is that it has no open ports at all. Nothing listens for inbound connections. You can't ssh to it. You can't reach it from the internet in any way. And when I'd finished, I felt safe.

That feeling turned out to be mostly an illusion. Not entirely. The thing is better than the alternative. But the part of it I was proud of was not the part keeping anyone out, and the way that happened seems worth writing down. I suspect a lot of people have built the same thing and feel the same misplaced pride.

Here's how it works. A tiny instance sits in the same network as the database. Its security group, which is the firewall more or less, has no inbound rules. The agent on the box dials out to AWS's Systems Manager, and Session Manager uses that outbound connection to open a tunnel from my laptop, through the bastion, to the database. The security group looks like this:

  BastionSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Bastion host, SSM, no inbound
      VpcId: !Ref VpcId
      SecurityGroupEgress:
        - IpProtocol: tcp           # outbound to SSM
          FromPort: 443
          ToPort: 443
          CidrIp: 0.0.0.0/0
        - IpProtocol: tcp           # to the database only
          FromPort: 5432
          ToPort: 5432
          DestinationSecurityGroupId: !Ref RdsSecurityGroupId
      # and no SecurityGroupIngress block at all

Notice there's no ingress block. That's the whole trick, and it's a real one. A bastion with an open ssh port is a thing attackers find and poke at. There's a daemon to brute force, and a new ssh hole to patch every few months. This has none of that. There are no keys to hand out or take back, and nothing to scan. If you'd asked me whether it was secure, I'd have pointed at the empty list of inbound rules and said yes.

But secure against what? That's the question I didn't ask at first, and it's the one that matters.

Nobody is going to break into this box over the network, because there's no way in over the network. So an attacker won't come that way. He'll come the way I come. To open the tunnel I call an AWS API, authenticated as myself. Anyone who can make that same call can open the same tunnel. And whether someone can make the call has nothing to do with ports. It's a question of who holds the right credentials, and what those credentials are allowed to do.

Which means the empty firewall, the thing I was proud of, wasn't the lock on the door. The lock was somewhere else, in a place I hadn't looked.

What had happened, I realized, was not that I'd removed the risk but that I'd moved it. It used to live on the network, where it's easy to see and reason about, since a port is either open or closed. Now it lived in the permission system, where it's easy to get wrong in ways you won't notice for a long time. That can be a good trade. A risk you've thought carefully about beats one you haven't. But it's only a good trade if you do the thinking, and the satisfying click of closing the ports tempts you to stop right before the part that counts.

When I went and looked at the permissions, I found something that surprised me. I'd assumed that letting someone open a tunnel let them open a tunnel. It doesn't. The same API call, with its default options, drops you into a shell on the bastion: a command prompt on a machine sitting inside the network, next to the database. So a permission I'd have described as "they can reach the database through the tunnel" in fact meant "they can get a shell on a box next to the database." Those are not the same sentence, and the gap between them is the kind of thing that stays invisible until someone shows it to you.

The fix is to narrow the permission so it allows the tunnel and nothing else, and to tie it to this one machine. Roughly:

{
  "Effect": "Allow",
  "Action": "ssm:StartSession",
  "Resource": "arn:aws:ec2:ap-south-2:*:instance/*",
  "Condition": {
    "StringEquals": { "ssm:resourceTag/Name": "app-bastion" }
  }
}

paired with a second statement that allows only the port-forwarding document, not the one that opens a shell. I say roughly because the rules about which documents a permission covers have enough corners that I wouldn't paste this into a real account without checking it against the current docs. But the shape is the point. A bastion whose only job is to forward a port should be allowed to forward a port, and not to do the more general and more dangerous thing the same API will happily do if you let it.

There's a second place where the feeling of safety outran the reality. Look again at the one outbound rule I kept. The box is allowed to make connections out to anywhere, on the port HTTPS uses. It has to reach AWS's servers somehow, and that's how. But anywhere is a lot of places. If someone did get onto the box, that open road out is exactly what they'd use to carry data off it. The instance also has a public address, which is one more thing for someone to find.

You can close both at once, and it's worth doing for anything that sits next to data you care about. Put the bastion on a private network with no public address, and give it private doorways to the few AWS services it actually needs. Then it has no route to the open internet at all. Nothing public to find, and no open road out. As a side effect it saves a few dollars a month, which is not the reason to do it but is pleasant.

People also like this design because it keeps a record, and that's true, with a caveat worth stating plainly. AWS logs the fact that I opened a session: who I was, which machine, when. That's more than an ssh bastion gives you for free. But it does not, by itself, record what happens inside the session. If you want that you have to turn it on, and it's off until you do. For a bastion that can only forward a port this matters less, because there's no shell to record, which is one more reason to make sure it really can only forward a port.

So is the thing secure? More than the alternative, yes. But not for the reason I first thought. The closed ports, the part I could point at and feel good about, were never doing the work. The work was being done, or not done, by a set of permissions I hadn't looked at, in a system where mistakes are quiet.

I think this is a general hazard and not just an AWS one. The security measures that feel the most satisfying tend to be the visible ones: the closed port, the locked door, the deleted account. They're satisfying precisely because you can see them. But the things that actually let people in are usually the ones you can't see at a glance, and the good feeling from the visible measure is what stops you looking for the rest. The empty firewall was a fact about my network. And my network, it turned out, was never the way in.

---

If you want to build the thing: the relevant AWS docs are Session Manager port forwarding, restricting session access by tag and document, VPC endpoints for Systems Manager, and session logging.