Proxmox NAT & CSF

AlwaysSkint

I have the above setup, at long last nearing the point of using it, rather than idling. I can connect to a VM from the 'net via redirected ports and the VM is able to connect to the world.
When I ssh in to the VM, it is being logged as coming from the host server, not my home IP, so I'm obviously missing something in the setup. Any pointers for me?
Ta.

InceptionHosting

The NAT les servers just use simple SNAT per external IP e.g.
iptables -t nat -A POSTROUTING -s 192.168.0.0/24 -o br0 -j SNAT --to 104.105.106.107
Then port forwarding with DNAT e.g for ssh access
iptables -t nat -A PREROUTING -p tcp -d 104.105.106.107 --dport 321 -j DNAT --to 192.168.20.3:22
and then a port range through DNAT for the rest:
iptables -t nat -A PREROUTING -p tcp -d 104.105.106.107 --dport 301:320 -j DNAT --to-destination 192.168.20.3

But that assumes a few things, its been an age since I used proxmox properly but I remember it had a number of different options on how to pass traffic through when adding an interface type for the VM's the above would only work in a bridged network, I do seem to remember (going back maybe 7 - 8 years) that the default NAT setup in proxmox was to masquerade which may be the cause of your problem, I think a bond would do the same so you need bridged or a seperate VLAN for guest networks using a manually configured bridge would be my guess.

Could be wrong, just guessing without seeing.

AlwaysSkint

I'm using a bridged network on an OVH host. I have the following if that makes things clearer.
Part of my iptables-set.sh/csfpost.sh

• echo 1 >/proc/sys/net/ipv4/ip_forward
• echo 1 > /proc/sys/net/ipv4/conf/vmbr1/proxy_arp
• /sbin/iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 8000 -j DNAT --to 10.0.0.100:80
• ...
• iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j MASQUERADE
• /sbin/iptables -F FORWARD
• /sbin/iptables -P FORWARD ACCEPT

In /etc/network/interfaces

• post-up iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j MASQUERADE
• post-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 8000 -j DNAT --to 10.0.0.100:80

There appears to be a duplication above but it's taken me ages to get packets traversing the host. Seems like I'm missing SNAT entries .. I'm sure I spotted them someplace. :-S

InceptionHosting

I think the issue is the MASQUERADE, so it is well, masquerading if i were you I would strip it all back and follow the examples above, its actually a lot more simple than it seems to achieve what you want.

AlwaysSkint

I was following a mashup of 'instructions' on the web and got pretty confused along the way. I also have nginx reverse proxy and webserver VMs on this host, so I'll take my time over the reconfiguration. Thanks!

mikho

I was about to write something useful, @AnthonySmith beat me to it.
SNAT is the way to go unless you either want to MASQUERADE the original IP or in some cases uses a dynamic IP.

AlwaysSkint

Latest configuration:
/etc/network/interfaces (snippet)

• post-up echo 1 > /proc/sys/net/ipv4/ip_forward
• post-up iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j SNAT --to 178.32.xxx.xxx
• post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j SNAT --to 178.32.xxx.xxx
• post-up iptables -t nat -A PREROUTING -d 178.32.xxx.xxx -p tcp --dport 8000 -j DNAT --to 10.0.0.100:80
• post-down iptables -t nat -D PREROUTING -d 178.32.xxx.xxx -p tcp --dport 8000 -j DNAT --to 10.0.0.100:80

I couldn't get outbound (eg. apt update) from the VM when the host had CSF active, until I placed this in csfpost.sh

• #!/bin/bash
• echo 1 >/proc/sys/net/ipv4/ip_forward
• echo 1 > /proc/sys/net/ipv4/conf/vmbr1/proxy_arp
• /sbin/iptables -F FORWARD
• /sbin/iptables -P FORWARD ACCEPT
• /sbin/iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j SNAT --to 178.32.xxx.xxx

I dunno the consequences of those iptables FORWARD directives, however.. will need to do a search.

EDIT: VM still shows host IP for my laptop doing an ssh to it. In other words the originating IP is being stripped during the forwarding.

kmmm

I routed all traffic to the bare metal nginx instance before passing traffic to corresponding virtual network interfaces (created and assigned by Proxmox, ofc). This is dead simple, and I do not need to mess with iptables since I am also using block IPs based on ipset. My setup is for personal use, so no idea how it really effects the performance of Proxmox host under load.

vimalware

What I do is simpler
Access to the NAT vm's is via a http(s) load balancer KVM(dedicated IPv4)

The individual KVMs serial terminals can be 'accessed' by ' qm terminal 101' in a tmux ssh/mosh session to the proxmox host.

My vm templates don't have openssh-server. I install it when required

AlwaysSkint

A proxmox console is one way around it but doesn't address other services. I used ssh as an example because it is relatively simple to monitor activity to it. You also assume that an additional IP is available (I know I didn't cover that aspect) but that's not an option on Kimsufi servers, unfortunately. I have another proxmox instance elsewhere with /29, so that this subject is never considered, except for IPv6 trials.

TLDR; Originating IP is replaced by host IP at the VM

terrorgen

Use IPv6

AlwaysSkint

Sorry to drag up this old thread again.. I'm back trying to get NAT working, though I am a bit further forward than previously.
On a different server, again with only one public IP, with Proxmox handling VMs; it's nginx that I've been tearing my hair out with. I'm trying to use it on the main server to redirect to the appropriate VMs.
I can access VMs from the outside world by using the server defined ports but I can't access using the VMs internal ports. There's also the issue of how to handle SSL certs.
Example:
http://tx200.vmdomain.com:80 <-- 502 Bad Gateway
http://tx200.vmdomain.com:42080 <-- Testing 123 (nginx default page)
https://tx200.vmdoamin.com:443 <-- PR_END_OF_FILE_ERROR
https://tx200.vmdomain.com:42443 <-- (Firefox:) Unable to connect
https://tx200.vmdomain.com:10000 <-- Error code: SSL_ERROR_RX_RECORD_TOO_LONG
https://tx200.vmdomain.com:42000 <-- webmin loads, with self-certified, correct FQDN SSL warning

This is what I've cobbled together for the nginx tx200.vmdomain.com.conf

• server {
• server_name tx200.vmdomain.com www.tx200.vmdomain.com;
• listen 80;
• location / {
• proxy_set_header Host $host;
• proxy_set_header X-Real-IP $remote_addr;
• proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
• proxy_set_header X-Forwarded-Proto $scheme;
• proxy_pass http://tx200.vmdomain.com:42080;
• proxy_read_timeout 90;
• proxy_redirect http://tx200.vmdomain.com:42080 http://tx200.vmdomain.com:80;
• }
• }
• server {
• server_name tx200.vmdomain.com www.tx200.vmdomain.com;
• listen 443;
• location / {
• proxy_set_header Host $host;
• proxy_set_header X-Real-IP $remote_addr;
• proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
• proxy_set_header X-Forwarded-Proto $scheme;
• proxy_pass https://tx200.vmdomain.com:42443;
• proxy_read_timeout 90;
• proxy_redirect https://tx200.vmdomain.com:42443 https://tx200.vmdomain.com:443;
• }
• }
• server {
• server_name tx200.vmdomain.com www.tx200.vmdomain.com;
• listen 10000;
• location / {
• proxy_set_header Host $host;
• proxy_set_header X-Real-IP $remote_addr;
• proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
• proxy_set_header X-Forwarded-Proto $scheme;
• proxy_pass https://tx200.vmdomain.com:42000;
• proxy_read_timeout 90;
• proxy_redirect https://tx200.vmdomain.com:42000 https://tx200.vmdomain.com:10000;
• }
• }

I've tried without proxy_redirect but made no odds.
Obviously I have the IP addresses defined in the server hosts file and the firewall has a block of ports opened.
VMs can access the outside world and I can ssh in using the appropriate redirected port.
As well as the KVM above, I have a LXC that unsurprisingly displays similar symptoms.
Next thing I'm about to try is copy the self-cert SSL from webmin over to the default Apache location to see if it resolves the 443 issue.
I feel that I'm close but no cigar.

Neoon

HAProxy takes care of TLS/Plain inspection with SNI and forwards it to the right machine:
https://gist.github.com/Ne00n/b52e112afa5d587178920e0164691083
Maybe helpful.
https://github.com/Ne00n/NanoKVM-Tools

AlwaysSkint

@Neoon Thanks! Will take a look.

Edit: I'm definitely missing something there.. The 'tools' appear to use DHCP which I specifically don't want use - will only have about 4 VMs max.
The HAProxy SNI TLS + HTTP looks promising, assuming I can extend to other services. Goes off to read about yet another prog.

Neoon

@AlwaysSkint said:
@Neoon Thanks! Will take a look.

Edit: I'm definitely missing something there.. The 'tools' appear to use DHCP which I specifically don't want use - will only have about 4 VMs max.
The HAProxy SNI TLS + HTTP looks promising, assuming I can extend to other services. Goes off to read about yet another prog.

The HAProxy config is just copy & paste, also DHCP is just optional for every network, its not mandatory.
Same goes for radvd for IPv6.

AlwaysSkint

Think I'm too stupid for this game. :-(
Same errors as above, apart from port 10000 which now has PR_END_OF_FILE_ERROR.
(that Testing 123 page is Apache on the VM, not nginx.)

• frontend http
• bind *:80
• mode http
• #Define hosts
• acl host_tx100.domain.com hdr(host) -i tx100.domain.com
• use_backend tx100.domain.com if host_tx100.domain.com
• acl host_tx200.domain.com hdr(host) -i tx200.domain.com
• use_backend tx200.domain.com if host_tx200.domain.com
• backend tx100.domain.com
• server tx100.domain.com 10.0.0.100:80
• mode http
• backend tx200.domain.com
• server tx200.domain.com 10.0.0.120:80
• mode http
• frontend https
• bind *:443
• mode tcp
• acl tls req.ssl_hello_type 1
• tcp-request inspect-delay 5s
• tcp-request content accept if tls
• #Define hosts
• acl host_tx100.domain.com_https req.ssl_sni -i tx100.domain.com
• use_backend tx100.domain.com_https if host_tx100.domain.com_https
• acl host_tx200.domain.com_https req.ssl_sni -i tx200.domain.com
• use_backend tx200.domain.com_https if host_tx200.domain.com_https
• backend tx100.domain.com_https
• server tx100.domain.com 10.0.0.100:443
• mode tcp
• backend tx200.domain.com_https
• server tx200.domain.com 10.0.0.120:443
• mode tcp
• frontend webmin
• bind *:10000
• mode tcp
• acl tls req.ssl_hello_type 1
• tcp-request inspect-delay 5s
• tcp-request content accept if tls
• #Define hosts
• acl host_tx100.domain.com_webmin req.ssl_sni -i tx100.domain.com
• use_backend tx100.domain.com_webmin if host_tx100.domain.com_webmin
• acl host_tx200.domain.com_webmin req.ssl_sni -i tx200.domain.com
• use_backend tx200.domain.com_webmin if host_tx200.domain.com_webmin
• backend tx100.domain.com_webmin
• server tx200.domain.com 10.0.0.100:12321
• mode tcp
• backend tx200.domain.com_webmin
• server tx200.domain.com 10.0.0.120:10000
• mode tcp

Am I right in thinking there's no need to have a httpd service running now, as haproxy takes over?

Neoon

HAProxy does forward HTTP/HTTPS to a specific IP:PORT, you need to have a webserver running at that destination.
If you have no webserver running, then you get errors.

Try plain, if plain works, setup LetsEncrypt, then https will work also.
You can setup https like on any normal webserver since HAProxy forward encrypted https traffic.

Use pastebin if you post any configs, I wont read that otherwise.
The formating is painful.

AlwaysSkint

Yes, of course the VMs have a httpd running, I meant at the server.
Yep the code flag on the forum goes mental - I'll try a pastebin.
https://pastebin.com/gDhXQZ7W

Neoon

I see no error does plain work?

AlwaysSkint

@Neoon
Thanks for bearing with me. Turnkey NextCloud/LAMP confused issues a bit due to redirections at the VM.
Current situation: port 80 requests work fine. 433 works only after I access webmin via the NAT 41000/42000 ports and request LE SSL certificates, plus add them manually to the webmin defined virtual hosts. A very convoluted way of doing things but at least I'm getting there.

Neoon

Yes, SSL won't work until you setup valid certificates.
You just need a for example and nginx with LetsEncrypt and HAProxy forwards the requests fine.

I do not know your exact setup, you need to debug and check what's wrong. eg log files.
As far as I know it should work as expected.