Why

The gear I've got for firewall is stupidly overkill (optane drive & 16GB RAM) for something that is 99% idle so I want to stick a couple other things on the same hardware via proxmox.

Assumed knowledge and requirements

Proxmox, linux cli, basic networking, (very basic) firewall knowledge. Nothing here is super hardcore technically as such...just unforgiving in sequence & details if unfamiliar hence notes. i.e. when it doesn't work it's not obvious where you fkd up.

Bring backups & loads of time. Frankly...assume that you'll fu.ck this up if you haven't done this before. So this is a Saturday morning project not a Sunday night one.

Before you start set up a ventoy USB stick adding both proxmox and opnsense ISO on there (plus perhaps a PDF copy of this post). You may not have internet access while doing this so come prepared...

I'm using my own IP ranges etc in post - adjust as needed. Key pieces you need to know to follow my descriptions:

vmbr0 is LAN side
vmbr1 is WAN (internet) side. (Proxmox sticks VMs on 0 by default so you want LAN side being 0)
10.32.0.1 is the opnsense VM [LAN side], 10.31.0.5 is WAN side but that doesn't really matter key part is 31 makes it separate from our main 32 LAN
10.32.0.2 is the LAN side IP of proxmox
My device has 5 ports with enp2s0 being internet facing [aka vmbr1], rest of the NICs are bridge to vmbr0
The key networking file on proxmox is at /etc/network/interfaces

Assumes upstream ISP is Eth based and just DHCPs whatever is connected. If you've got something else on ISP side you're on your own (sorry)

Virtualized FW

There are two basic ways to do this. Either pass through the NICs to the FW VM, or do two sets of bridges with one being WAN (internet) side and one LAN side. Passthrough is tricky to get working on a basic linux/bios/hardware level but solves some security issues. I'm doing the bridge approach because my gear (oddly) doesn't seem to support pass through. This guide is bridge only...so abandon ship now if you're going passthrough.

Bridge has one significant security implication. Proxmox gets the external traffic and bridges it to the firewall VM. Grand. Except Proxmox is fond of serving the proxmox management interface on ALL interfaces. So it's serving the damn management GUI before anything gets to our FW. Meaning it is potentially exposed BEFORE our fancy opnsense security gets to it. That's a problem we're trying to get around with below section on messing with the Proxmox firewall...despite setting up a opnsense firewall. [In reality the WAN side has a pvt non-routable IP so should be fine even without rules but I don't 100% trust that]

Keep this in mind when doing say incoming wireguard...you're effectively double-firewalled...and would need to pass through wireguard on both.

Security

The bulk of this guide happens BEFORE anything is connected to the internet, mostly because above "I don't trust this" issue. Just the FW device and a laptop directly connected to it.

GO

Step 1: Install proxmox & ensure you can access the GUI

On the network section do:
IP 10.32.0.2
Gateway 10.32.0.1
Set DNS to 1.1.1.1
Select a NIC that is LAN side (enp7s0 for me) - this should be whatever your laptop is plug in to

The proxmox interface is on https://10.32.0.2:8006

Step 2: Set up a second bridge in /etc/network/interfaces with WAN

auto vmbr1
iface vmbr1 inet static
        address 10.31.0.5/24
        bridge-ports enp2s0
        bridge-stp off
        bridge-fd 0

iface enp2s0 inet manual

Note how it is set to a static unroutable IP despite us aiming for DHCP upstream ISP side. That's because we want the firewall VM to grab the DHCP from ISP, not the proxmox level interface.

Restart to make the 2nd bridge show in GUI.

Step 3: Set up a opnsense VM

Copy over ISO off USB stick like so:

mkdir /media/usbstick
fdisk -l  [[[to get the /dev/ location for the stick]]]
mount /dev/sda1 /media/usbstick
cp /media/usbstick/OPNSense.sio /var/lib/vz/template/iso/

If the iso is in that location then proxmox GUI should pick it up.

Set up a VM that has 12GB, 5120mb RAM and 3 cores. Going lower on space & ram works but has consequences downstream on how zfs plays out so if you can do 12 & 5 or more.

Before starting VM add the second bridge (vmbr1). Ideally add the LAN side during VM creation so that the naming 0 & 1 line up consistently. Memorize the MACs on both (sorta...just which is which).

Step 4: Install opnsense

Where it asks you for a login, use installer as user and opnsense as password

Key point here is you need to line up what opnsense thinks is LAN side and WAN side...with the right MAC per above...which in turn maps to the right bridge...which in turn maps to a physical port.

Finish install, change root pass & unmount the ISO. Reboot.

Step 5: Initial config of opnsense

Log into console of opnsense (the proxmox built in one). Option 2 - set interface IP. Configure LAN side. This part is a little confusing. We DONT want it to use DHCP as in where it gets it's IP, but we want it to SERVE DHCP...so that other LAN devices can get an IP from it. So in this menu the first time it asks you about DHCP the answer is no, the second time the answer is yes. Set IP to 10.32.0.1 and 24 as subnet bit count. Enable DHCP server. Start clients at 10.32.0.51 and end at 10.32.0.254. That way we've got ~50 IPs for statics. Change GUI to http. Reset to defaults - yes.

...now try to access GUI on http://10.32.0.1

On the GUI you should have an initial setup. Set hostname, domain, and DNS. Go with a public one for now e.g. 1.1.1.1 Leave the WAN side as DHCP - no need to change anything here.

Step 6: Firewall

Go back to proxmox & navigate to datacenter on left then to firewall. This stuff cascades so the rules you set on datacenter level apply to the device and to VMs below.

The firewall rules use first match principle, starting with zero i.e. top of list. So in general we want a pattern like so

[0] - Allow SSH on LAN side
[1] - Disallow SSH on everything

So lan side incoming SSH hits rule 0 matches that and gets approved, while everything else fails rule 0 then goes to rule 1 and gets dropped.

Direction: In
Action: Accept
Interface: vmbr0 [[[[this is LAN side]]]]
Protocol: TCP
Dest Port: 8006
Enable: Tick

Direction: In
Action: Drop
Interface: [[[[leave empty]]]]
Protocol: TCP
Dest Port: 8006
Enable: Tick

Do same for 22 TCP and ICMP (protocol, no port). I'd suggest messing with ICMP to experiment and get a better grasp of what blocks when.

Triple check that you've got this right. Next step is enable firewall...and if you fkd this (8006 specifically) up then you're locked out and will need a physically screen & keyboard to sort it out.

You should also block these ports on vmbr1 (internet/wan) for good measure:
TCP 111, UDP 111, UDP 323, TCP 25, TCP 3128. No idea what they all do...but proxmox is listening on them & I don't want them open internet facing. I used command "lsof -i -P -n" to work out what ports are being listened to.

Step 7: Connect device to internet

If all went well then a device plugged into LAN side can now access the internet. First thing you want to do is head over to

https://www.yougetsignal.com/tools/open-ports/

And confirm they can't see your port 8006

Trouble shooting

1) You'll need to set a static IP in the right range on your laptop when connecting directly to a device. So when you can't reach the interfaces (proxmox or opnsense) checking that you're on the right net is helpful. Also windows is really shtty on this so sometimes you need to disable and reenable the network adapter to make it pick up things right. Try that before assuming something else is broken. You may also need to restart the laptop entirely...windows tends to get confused easily when rapidly switching networks.

2) Very likely that you'll f up DNS somewhere along the way...so when trying to work out whether a path through the firewall works try to use IPs like 1.1.1.1 with ping rather than google.com.

3) If genuinely stuck do ping device by device. i.e. my laptop is connected to a proxmox device...can i ping that by IP. Yes, ok can I ping opnsense by IP. Yes, can I ping 1.1.1.1. Yes. Can I ping google.com.

Devices

You should point all devices at the opnsense instance for both gateway and DNS [10.32.0.1 for me]. Never point a device at the pihole IP directly...that would work yes but it'll get you into trouble later in unexpected & hard to troubleshoot ways. You want the devices asking opnsense for DNS and opnsense in turn asking the pihole/adguardhome, and that asking 1.1.1.1 in turn. It's under Unbound in opnsense...section called Override...that's where the pihole goes. Also I suggest using Adguard home...it's superior to pihole.

YAY & Thoughts

Once it all sorta works, export the config in opnsense and save a copy of the proxmox /etc/network/interfaces - opnsense has a really nice "one config backup file everything is in it" config thing going...use it.

If anyone finds holes in this post please do tell...I'm running this live so if I'm wrong about something I'd like to find out from you rather than an evil hacker

Full /etc/network/interfaces for completeness

root@firewall:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

iface enp2s0 inet manual

iface enp3s0 inet manual
iface enp4s0 inet manual
iface enp5s0 inet manual
iface enp6s0 inet manual
iface enp7s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 10.32.0.2/24
        gateway 10.32.0.1
        bridge-ports enp7s0 enp6s0 enp5s0 enp4s0 enp3s0
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet static
        address 10.31.0.5/24
        bridge-ports enp2s0
        bridge-stp off
        bridge-fd 0

And firewall view: