mjhjmtu

I installed libvirt out of the box on Ubuntu 16.04, it creates a virbr0 virtual bridge, at 192.168.122.0/24, a dnsmasq instance to serve that range and sets up basic NAT masquerading.

I created a tap interface beneath that for my virtualization software to use. The software itself is able to access the internet and remainder of the network perfectly, so masquerading is working properly. I can also access the virtualization software ssh port from the host, using the IP address claimed by that software. So I think the routing tables "out of the box" are generally correct.

What I want to do is port forward an inbound ssh connection from outside to this software, and the host system in general. Now the host system of course has ssh, and I need that to work. So I'm trying to use another port, such as port 2022.

I've created a DNAT rule in iptables to direct inbound traffic on port 2022 to the IP address of the software at port 22. I get destination unreachable when I attempt to ssh in from the outside (ex. computer A below). It looks like the address translation is happening properly, but immediately after traversing to the "FORWARD" step in iptables, I start getting outbound unreachable ICMP packets.

I'm stumped on how to proceed. I've tried some SNAT rules, but I either botched it or that's not the problem. Does anyone have ideas on this?

UPDATE:
Wireshark on tap0 does not show having received the forwarded packet. Whatever is returning the packet is not related to the virtualization software.

Here's a picture of my network:

 ----------
 | Internet |
 ----------
 |
 |
 --------
 | Router | 10.211.255.1
 --------
 |
 | 
 ____________|_______________
 | |
 ---------- ---------
| | | enp0s5 | 10.211.255.4
| Computer | 10.211.255.2 | Host | 192.168.122.1
| A | | virbr0 |
 ---------- ---------
 |
 | LAN NAT 192.168.122.0/24 
 |
 ----------
 | tap0 |
 | Guest OS | 192.168.122.118
 | |
 ----------

Here are my IP tables rules:

Chain INPUT (policy ACCEPT)
target prot opt source destination
LOG all -- anywhere anywhere LOG level debug prefix "INTRACE: "
ACCEPT udp -- anywhere anywhere udp dpt:domain
ACCEPT tcp -- anywhere anywhere tcp dpt:domain
ACCEPT udp -- anywhere anywhere udp dpt:bootps
ACCEPT tcp -- anywhere anywhere tcp dpt:bootps

Chain FORWARD (policy ACCEPT)
target prot opt source destination
LOG all -- anywhere anywhere LOG level debug prefix "FORTRACE: "
ACCEPT all -- anywhere 192.168.122.0/24 ctstate RELATED,ESTABLISHED
ACCEPT all -- 192.168.122.0/24 anywhere
ACCEPT all -- anywhere anywhere
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
LOG all -- anywhere anywhere LOG level debug prefix "OUTTRACE: "
ACCEPT udp -- anywhere anywhere udp dpt:bootpc

NAT tables:
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
LOG tcp -- anywhere anywhere tcp dpt:2022 LOG level debug prefix "NATPTRACE: "
DNAT tcp -- anywhere anywhere tcp dpt:2022 to:192.168.122.118:22

Chain INPUT (policy ACCEPT)
target prot opt source destination
LOG all -- anywhere anywhere LOG level debug prefix "NATITRACE: "

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
LOG all -- anywhere anywhere LOG level debug prefix "NATOTRACE: "

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
LOG all -- anywhere anywhere LOG level debug prefix "NATQTRACE: "
SNAT tcp -- anywhere anywhere tcp spt:ssh to::2022
RETURN all -- 192.168.122.0/24 base-address.mcast.net/24
RETURN all -- 192.168.122.0/24 broadcasthost.localdomain
MASQUERADE tcp -- 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535
MASQUERADE udp -- 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535
MASQUERADE all -- 192.168.122.0/24 !192.168.122.0/24

Finally this is an example of what I see in syslog with the above rules:

Oct 20 09:49:15 ubuntu kernel: [85149.812291] RAWPTRACE: IN=enp0s5 OUT= MAC=00:1c:42:3a:00:df:00:1c:42:00:00:08:08:00 SRC=10.211.55.2 DST=10.211.55.4 LEN=64 TOS=0x00 PREC=0x00 TTL=64 ID=40132 DF PROTO=TCP SPT=53245 DPT=2022 WINDOW=65535 RES=0x00 SYN URGP=0
Oct 20 09:49:15 ubuntu kernel: [85149.812318] MGPTRACE: IN=enp0s5 OUT= MAC=00:1c:42:3a:00:df:00:1c:42:00:00:08:08:00 SRC=10.211.55.2 DST=10.211.55.4 LEN=64 TOS=0x00 PREC=0x00 TTL=64 ID=40132 DF PROTO=TCP SPT=53245 DPT=2022 WINDOW=65535 RES=0x00 SYN URGP=0
Oct 20 09:49:15 ubuntu kernel: [85149.812326] NATPTRACE: IN=enp0s5 OUT= MAC=00:1c:42:3a:00:df:00:1c:42:00:00:08:08:00 SRC=10.211.55.2 DST=10.211.55.4 LEN=64 TOS=0x00 PREC=0x00 TTL=64 ID=40132 DF PROTO=TCP SPT=53245 DPT=2022 WINDOW=65535 RES=0x00 SYN URGP=0
Oct 20 09:49:15 ubuntu kernel: [85149.812336] MGFTRACE: IN=enp0s5 OUT=virbr0 MAC=00:1c:42:3a:00:df:00:1c:42:00:00:08:08:00 SRC=10.211.55.2 DST=192.168.122.118 LEN=64 TOS=0x00 PREC=0x00 TTL=63 ID=40132 DF PROTO=TCP SPT=53245 DPT=22 WINDOW=65535 RES=0x00 SYN URGP=0
Oct 20 09:49:15 ubuntu kernel: [85149.812340] FORTRACE: IN=enp0s5 OUT=virbr0 MAC=00:1c:42:3a:00:df:00:1c:42:00:00:08:08:00 SRC=10.211.55.2 DST=192.168.122.118 LEN=64 TOS=0x00 PREC=0x00 TTL=63 ID=40132 DF PROTO=TCP SPT=53245 DPT=22 WINDOW=65535 RES=0x00 SYN URGP=0
Oct 20 09:49:15 ubuntu kernel: [85149.812354] RAWOTRACE: IN= OUT=enp0s5 SRC=10.211.55.4 DST=10.211.55.2 LEN=92 TOS=0x00 PREC=0xC0 TTL=64 ID=50608 PROTO=ICMP TYPE=3 CODE=3 [SRC=10.211.55.2 DST=192.168.122.118 LEN=64 TOS=0x00 PREC=0x00 TTL=63 ID=40132 DF PROTO=TCP SPT=53245 DPT=22 WINDOW=65535 RES=0x00 SYN URGP=0 ]
Oct 20 09:49:15 ubuntu kernel: [85149.812357] MGOTRACE: IN= OUT=enp0s5 SRC=10.211.55.4 DST=10.211.55.2 LEN=92 TOS=0x00 PREC=0xC0 TTL=64 ID=50608 PROTO=ICMP TYPE=3 CODE=3 [SRC=10.211.55.2 DST=192.168.122.118 LEN=64 TOS=0x00 PREC=0x00 TTL=63 ID=40132 DF PROTO=TCP SPT=53245 DPT=22 WINDOW=65535 RES=0x00 SYN URGP=0 ]
Oct 20 09:49:15 ubuntu kernel: [85149.812361] OUTTRACE: IN= OUT=enp0s5 SRC=10.211.55.4 DST=10.211.55.2 LEN=92 TOS=0x00 PREC=0xC0 TTL=64 ID=50608 PROTO=ICMP TYPE=3 CODE=3 [SRC=10.211.55.2 DST=192.168.122.118 LEN=64 TOS=0x00 PREC=0x00 TTL=63 ID=40132 DF PROTO=TCP SPT=53245 DPT=22 WINDOW=65535 RES=0x00 SYN URGP=0 ]
Oct 20 09:49:15 ubuntu kernel: [85149.812364] MGQTRACE: IN= OUT=enp0s5 SRC=10.211.55.4 DST=10.211.55.2 LEN=92 TOS=0x00 PREC=0xC0 TTL=64 ID=50608 PROTO=ICMP TYPE=3 CODE=3 [SRC=10.211.55.2 DST=192.168.122.118 LEN=64 TOS=0x00 PREC=0x00 TTL=63 ID=40132 DF PROTO=TCP SPT=53245 DPT=22 WINDOW=65535 RES=0x00 SYN URGP=0 ]

The problem was in the FORWARD rules. As I did not set them up, I was misinterpreting output from iptables --list.

In that list rule 4 reads:

ACCEPT all -- anywhere anywhere

This appears to accept all incoming packets from anywhere bound to anywhere.

However if I look at iptables -S rule 4 is actually:

-A FORWARD -i virbr0 -o virbr0 -j ACCEPT

Which is not going to be matched by a packet coming in from enp0s5 destined for vibr0. So it was hitting the REJECT rule 5, which was actually a reject for all packets destined for vibr0.

The lesson I have learned is not to trust the output of iptables --list but rather to check iptables -S output when something seems wrong.