We have bastion server B.
We need to SSH from A through B to C, using private key.

What is the better option:

• 
  

  Put the private SSH key on server B. We read that it's a bad idea to do that in a production environment.

  
  
  

  From here:

  
  
  

  
  

  Never place your SSH private keys on the bastion instance. Instead,
  use SSH agent forwarding to connect first to the bastion and from
  there to other instances in private subnets. This lets you keep your
  SSH private key just on your computer.

  
  

  
  


• 
  

  Use SSH agent forwarding. For setting up agent forwarding, I need to allow TCP Forwarding. When setting up agent forwarding, a socket file is created on the forwarding host, which is the mechanism by which the key can be forwarded to the destination. In the Bastion settings at AWS:

  
  
  

  
  

  TCP forward: Setting this value to true will enable TCP forwarding
  (SSH tunneling). This can be very useful but it is also a security
  risk, so we recommend that you keep the default (disabled) setting
  unless required

  
  

  
  
  

  Also from here:

  
  
  

  
  

  SSH Agent Forwarding considered harmful

  
  

  
  

What is better? What about the alternative from the second link: ProxyCommand, I understand it helps with the socket file issue, but still I think I have to enable TCP forwarding, so is it secure enough?

Use ProxyCommand or ProxyJump

I would recommend to use ProxyCommand (or even better ProxyJump as the syntax is easier but requires openssh 7.3+ I think on the client side), and you do not need to deploy private key on the Bastion, everything stays local.

Example with ProxyJump

On your client computer you write a file under ~/.ssh/config with a similar content to bellow:

Host bastion
 HostName bastion.example.com
 User bastion-user
 Port 22
 IdentityFile ~/.ssh/id_bastion

Host srvC
 HostName srvC.local
 User server-user
 IdentityFile ~/.ssh/id_protected_lan
 ProxyJump bastion

Then doing ssh srvC will connect you to C via B (bastion) without Agent Forwarding nor deploying the private key to the bastion.

In the above example, "bastion" is an alias to your Bastion host and srvC is an alias to your server C. In the HostName you need to put either IPs or real fully qualified domain name for your hosts. For the users, you need to update the User for the correct login name on the Bastion and server C. Finally the IdentityFile is optional if you use a local agent (e.g. KeeAgent or ssh-agent), but if it is not running then it will also work and ask you for each key passphrases.

Deploying the public keys

Of course you need to deploy the public keys to both bastion and srvC. You can use (the $ sign is just to illustrate the prompt, do not type it):

$ ssh-copy-id -i ~/.ssh/id_bastion.pub 
 -o PreferredAuthentications=password 
 -o PubkeyAuthentication=no 
 bastion
$ ssh-copy-id -i ~/.ssh/id_protected_lan.pub 
 -o PreferredAuthentications=password 
 -o PubkeyAuthentication=no 
 srvC

Note: the above will work only if password authentication is still allowed. After the above deployment and verifying that everything work as intended, you should disallow password authentication on the 2 servers.

Example with ProxyCommand instead of ProxyJump

If you have an older version of OpenSSH which does not support ProxyJump (on the client side), then replace:

ProxyJump bastion

by

ProxyCommand ssh -q -W %h:%p bastion

As far as I understood, this is similar.

I saw the answer about ProxyJump. Let's talk about ProxyCommand.

But wait, wait! I can write to you how to hack the server that uses Agent forwarding, that would be much easier to understand the difference!

Let's hack!

For the basic steps: you can read my post here

Basic steps are the following:

1. Create bastion users


2. Disable root login


3. Block hacking attempts


4. Change port


5. Configure firewall


6. Configure SELinux

How to use AgentForwarding

-Create config in ~/.ssh/config

 Host bast
 Hostname BASTION_IP
 ForwardAgent yes
 User bastion

-Add your authentification key to ssh-agent

ssh-add ~/.ssh/name_rsa

-Connect to bastion hos

ssh bast

-Connect application server from the bastion

 ssh app@IP -p PORT

Hacking!

You may, well, ask me the question:

• 
  

  Is my server secure? And the answer is quite simple:

  
  
  
  • NO!
  
  


• 
  

  Why?

  
  
  
  • Because you are using SSH Agent forwarding!
  
  


• 
  

  And where is the problem?

  
  
  
  • Because Agent forwarding is dangerous and it's consider considered harmful.
  
  


• 
  

  Why?

  
  
  
  • Let's explain everything inside out: When you connect bastion host your glorious ssh-agent is forwarded. It means that the socket will be set up so that someone may use this socket data to access your servers. Imagine that your bastion server is compromised, If someone has sufficient permissions on your Linux server he/she will just use your socket info. As a result, all your server can be accessed. I know the window of compromise is very small because it depends on how much time you are connected to the bastion host. But do you really want to the risk when you have other options like ProxyCommand? Hence, just use ProxyCommand!
  
  

How to hack servers if you compromised bastion host?

Track Target

In /tmp directory you may see something like that:

[root@localhost tmp]# ll
total 12
drwx------ 2 bastion bastion 4096 Sep 7 17:35 ssh-mKX88v0Vlo

Let's open the temporary file

[root@localhost tmp]# cd ssh-mKX88v0Vlo/
[root@localhost ssh-mKX88v0Vlo]# ll
total 0
srwxr-xr-x 1 bastion bastion 0 Sep 7 17:35 agent.10507

Let's see connections to this process id.

netstat -nxp | grep 10507

result:

unix [ ] STREAM CONNECTED 501384 10507/sshd: bastion

and who is connected?

lsof -i -a -p 10507

result:

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sshd 10507 bastion 3u IPv4 501301 0t0 TCP *IP*:ssh->*IP*:8279 (ESTABLISHED)

We can also see socket files:

cd /proc/10507/fd/
ls

result:

lrwx------ 1 root root 64 Sep 7 17:46 0 -> /dev/null
lrwx------ 1 root root 64 Sep 7 17:46 1 -> /dev/null
lrwx------ 1 root root 64 Sep 7 17:46 10 -> /dev/ptmx
lrwx------ 1 root root 64 Sep 7 17:46 14 -> /dev/ptmx
lrwx------ 1 root root 64 Sep 7 17:46 15 -> /dev/ptmx
lrwx------ 1 root root 64 Sep 7 17:46 2 -> /dev/null
lrwx------ 1 root root 64 Sep 7 17:46 3 -> socket:[501994]
lrwx------ 1 root root 64 Sep 7 17:46 4 -> socket:[502069]
lrwx------ 1 root root 64 Sep 7 17:46 5 -> socket:[502072]
l-wx------ 1 root root 64 Sep 7 17:46 6 -> /run/systemd/sessions/1836.ref
lr-x------ 1 root root 64 Sep 7 17:46 7 -> pipe:[502079]
l-wx------ 1 root root 64 Sep 7 17:46 8 -> pipe:[502079]
lrwx------ 1 root root 64 Sep 7 17:46 9 -> socket:[502080]

And what happens when client will be connected to remote server? let's see:

lrwx------ 1 root root 64 Sep 7 17:46 0 -> /dev/null
lrwx------ 1 root root 64 Sep 7 17:46 1 -> /dev/null
lrwx------ 1 root root 64 Sep 7 17:46 10 -> /dev/ptmx
lrwx------ 1 root root 64 Sep 7 17:48 11 -> socket:[502267]
lrwx------ 1 root root 64 Sep 7 17:46 14 -> /dev/ptmx
lrwx------ 1 root root 64 Sep 7 17:46 15 -> /dev/ptmx
lrwx------ 1 root root 64 Sep 7 17:46 2 -> /dev/null
lrwx------ 1 root root 64 Sep 7 17:46 3 -> socket:[501994]
lrwx------ 1 root root 64 Sep 7 17:46 4 -> socket:[502069]
lrwx------ 1 root root 64 Sep 7 17:46 5 -> socket:[502072]
l-wx------ 1 root root 64 Sep 7 17:46 6 -> /run/systemd/sessions/1836.ref
lr-x------ 1 root root 64 Sep 7 17:46 7 -> pipe:[502079]
l-wx------ 1 root root 64 Sep 7 17:46 8 -> pipe:[502079]
lrwx------ 1 root root 64 Sep 7 17:46 9 -> socket:[502080]

We can even see if socket file is used using netstat:

unix 3 [ ] STREAM CONNECTED 502267 10561/sshd: 
 bastion /tmp/ssh-oVoMXC6vb8/agent.10561
unix 3 [ ] STREAM CONNECTED 502072 10561/sshd: bastion 

Steal Socket info and IP address

Now we need to steal the socket information while the session of bastion host is open. Oh, we also need destination server IP, so just use netstat:

netstat -tn

The final step to use the forwarded socket file

eval "$(ssh-agent -s)"
SSH_AUTH_SOCK=/tmp/ssh-EAKxOdL4fl/agent.10507

Check if the key is loaded.

ssh-add -l

result should be something like that:

2048 SHA256:2Psdl..B5KQ /home/usr/.ssh/name_rsa (RSA)

Server is hacked, how to fix the security problem?

Proxy command

Host app
 Hostname *.*.*.*
 IdentityFile ~/.ssh/your_rsa
 User *******
 Port ****
 ProxyCommand ssh -W %h:%p bast

Host bast
 Hostname *.*.*.*
 ForwardAgent no
 User ******

For basic operations: how to transfer files via the servers (from client to server, server to client), you can read on my post here

Conclusion

• If you use bastion host, don't use AgentForwarding but use
  ProxyCommand


• Always use non-root user for authentification


• Use a firewall and block all unnecessary connections.


• Use SELinux (In general)


• Block the IP address who tries to log in several times with incorrect credentials


• If it's not necessary don't give sudo permission to the user


• Monitor your server


• Update your server for security patches

More information, see my blog. Additionally I have some screeenshots, so it may be helpfull for you.