Proxmox 7 node on OVH unreachable after reboot

miu · July 2021

Hello, can somebody help me resolve following maddness pls?

Proxmox host node is OVH network dedicated (SoYouStart)

/etc/network/interfaces:

auto lo
iface lo inet loopback

iface eno3 inet manual

auto vmbr0
iface vmbr0 inet static

  address 54.38.192.11/24
  gateway 54.38.192.254
  bridge-ports eno3
  bridge-stp off
  bridge-fd 0

1) After node server full reboot is network UNREACHABLE

2) But when i run SSH command:

service networking restart

network is reachable and work fine:

# ip route
default via 54.38.192.254 dev vmbr0
54.38.192.0/24 dev vmbr0 proto kernel scope link src 54.38.192.11

I do not know resolve this madness WHY after reboot mentioned config does not work = network unreachable, but after additional manual networking restart the same config works fine....

BTW: i do not want use DHCP config = default (automatic OVH install) config:

auto lo
iface lo inet loopback

iface eno3 inet manual

auto vmbr0
iface vmbr0 inet dhcp
  bridge-ports eno3
  bridge-stp off
  bridge-fd 0

I want use STATIC config, not DHCP

many thanks in advance for each attempt to help resolve this madness

Abdullah · July 2021

is that the only network config file?

dirty fix - just add an init script in /etc/init.d/ to restart networking at every reboot.

miu · July 2021

update 2: after searching on net i found that several ppl report the same problem after upgrade on PROXMOX 7 (unreachable network). So probably version 7 is still pretty bugged. (maybe reason why OVH still not offers Proxmox 7 template?) I tried several changes in config but not resolved this (not found exact reason and clean solution), without success

Not_Oles · July 2021

Hi @miu!

Please answer a few questions from a clueless guy:

(A) In your original post,

@miu said: 1) After node server full reboot is network UNREACHABLE

2) But when i run SSH command:

service networking restart

network is reachable and work fine:

# ip route
default via 54.38.192.254 dev vmbr0
54.38.192.0/24 dev vmbr0 proto kernel scope link src 54.38.192.11

How do you run the ssh command when the network is unreachable?

(B) What do the logs say about bringing the network up as part of the reboot? Is there something that failed?

(C) In your

@miu said: /etc/network/interfaces:

auto lo
iface lo inet loopback

iface eno3 inet manual

auto vmbr0
iface vmbr0 inet static

address 54.38.192.11/24
gateway 54.38.192.254
bridge-ports eno3
bridge-stp off
bridge-fd 0

interface eno3 is set to "manual."

In man (5) interfaces, it says,

"The manual Method
       This method may be used to define interfaces for which  no  configuration  is
       done  by  default.  Such interfaces can be configured manually by means of up
       and down commands or /etc/network/if-*.d scripts.

Further down in the vmbr0 section, your /etc/network interfaces also has the line, bridge-ports eno3. What is supposed to happen when ifup reaches that bridge-ports eno3line following reboot? Could it be that there is a failure at that bridge-ports eno3 line because eno3 is called but eno3 is not present because manual configuration was specified for eno3?

I have seen other similar configurations which specify bridge-ports none instead of bridge-ports eno3. Proxmox' Routed Configuration example has bridge-ports none:

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet static
        address  198.51.100.5/24
        gateway  198.51.100.1
        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up echo 1 > /proc/sys/net/ipv4/conf/eno1/proxy_arp


auto vmbr0
iface vmbr0 inet static
        address  203.0.113.17/29
        bridge-ports none
        bridge-stp off
        bridge-fd 0

Maybe your issue could be resolved by changing bridge-ports eno3 to bridge-ports none?

In summary:

How can you run an ssh command when the network is unreachable?
What do the logs say?
What is supposed to happen when ifup reaches the "bridge-ports xxx" line in /etc/network/interfaces, and could having the interface xxx non-functioning cause ifup to fail?

Could your issue be resolved by changing bridge-ports eno3 to bridge-ports none?

Hope this is helpful! Best wishes from Mexico!

miu · July 2021

Hello Oles, firstly thank you for your response!

@Not_Oles said:
Hi @miu!

Please answer a few questions from a clueless guy:

How do you run the ssh command when the network is unreachable?

ANSWER @ A) Remotely Through IPMI console

(B) What do the logs say about bringing the network up as part of the reboot? Is there something that failed?

(C) In your

@miu said: /etc/network/interfaces:

auto lo
iface lo inet loopback

iface eno3 inet manual

auto vmbr0
iface vmbr0 inet static

address 54.38.192.11/24
gateway 54.38.192.254
bridge-ports eno3
bridge-stp off
bridge-fd 0

interface eno3 is set to "manual."

In man (5) interfaces, it says,
"The manual Method
       This method may be used to define interfaces for which  no  configuration  is
       done  by  default.  Such interfaces can be configured manually by means of up
       and down commands or /etc/network/if-*.d scripts.
Further down in the vmbr0 section, your /etc/network interfaces also has the line, bridge-ports eno3. What is supposed to happen when ifup reaches that bridge-ports eno3line following reboot? Could it be that there is a failure at that bridge-ports eno3 line because eno3 is called but eno3 is not present because manual configuration was specified for eno3?

I have seen other similar configurations which specify bridge-ports none instead of bridge-ports eno3. Proxmox' Routed Configuration example has bridge-ports none:
auto lo
iface lo inet loopback

auto eno1
iface eno1 inet static
        address  198.51.100.5/24
        gateway  198.51.100.1
        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up echo 1 > /proc/sys/net/ipv4/conf/eno1/proxy_arp


auto vmbr0
iface vmbr0 inet static
        address  203.0.113.17/29
        bridge-ports none
        bridge-stp off
        bridge-fd 0
Maybe your issue could be resolved by changing bridge-ports eno3 to bridge-ports none?

ANSWER @ B & C:

Thanks for your suggestion - this i not tried (change bridge-ports from ens 3 to none)
From time reasons after several hours of frustration i reinstalled whole node from ISO 6.4 - it works FINE (network) with my above mentioned origin static config
Maybe i will try your suggested change next time when install another Proxmox node from v7 ISO, then i will can update this

thanks! regards miu

miu · July 2021

So just now i can conclude only:

auto lo
iface lo inet loopback

iface eno3 inet manual

auto vmbr0
iface vmbr0 inet static

  address 54.38.192.11/24
  gateway 54.38.192.254
  bridge-ports eno3
  bridge-stp off
  bridge-fd 0

works on Proxmox 6.4 + SYS well (network works without any troubles)

but on new Proxmox v7 + SYS not at all (network UNREACHABLE)

ehab · July 2021

@miu said:
but on new Proxmox v7 + SYS not at all (network UNREACHABLE)

ehab · July 2021

@miu i would look at the networking service if its started at reboot.
look at the kernel logs in the console etc ...
but if its a bug in v.7 then you have to live with it until update.

Not_Oles · July 2021

@ehab said:
@miu i would look at the networking service if its started at reboot.
look at the kernel logs in the console etc ...
but if its a bug in v.7 then you have to live with it until update.

@ehab Yeah. for sure, but @miu can't look at the logs after the machine is reinstalled.

miu · July 2021

@ehab said:

@miu said:
but on new Proxmox v7 + SYS not at all (network UNREACHABLE)

u r right - exactly ma face after suffering from this issue! :-D

imok · July 2021

I'm using a similar network config with Proxmox VE 7.0-9 and no problems at all.

But I didn't use the ISO. Installed on top of Debian.

AlwaysSkint · July 2021

Somehow I missed that 7 was available - still running 6.4 on my boxes. Thanks for the heads-up.

havoc · July 2021

Reddit is full of people bombing their boxes with 7 upgrade. I've decided I'll wait a month or 5. There isn't really anything I need in 7...

AlwaysSkint · July 2021

I'll run the tests and make sure that I have done some updates on my Proxmox Backup Server.
@miu Re. networking issue; I wonder if this is the cause:

Linux Bridge MAC-Address Change

With Proxmox VE 7 / Debian Bullseye, a new systemd version is used, that changes how the MAC addresses of Linux network bridge devices are calculated:
MACAddressPolicy=persistent was extended to set MAC addresses based on the device name. Previously addresses were only based on the ID_NET_NAME_* attributes, which meant that interface names would never be generated for virtual devices. Now a persistent address will be generated for most devices, including in particular bridges.

-- https://www.freedesktop.org/software/systemd/man/systemd.net-naming-scheme.html#v241
A unique and persistent MAC address is now calculated using the bridge name and the unique machine-id (/etc/machine-id), which is generated at install time.

Please either ensure that any ebtable or similar rules that use the previous bridge MAC-Address are updated or configure the desired bridge MAC-Address explicitly, by switching to ifupdown2 and adding hwaddress to the respective entry in /etc/network/interfaces.

Not_Oles · July 2021

@miu said: i not tried (change bridge-ports from ens 3 to none)

@miu I ran a little test.

My server (running Debian sid, not Proxmox) had bridge-ports none.

Just for fun I tried changing the "none" to "enp7s0", which is the name of my server's primary interface, corresponding to "eno3" on your server.

systemctl restart networking following this change broke networking, which was the hoped-for result.

I reverted via the rescue system.

Conceivably, my little test supports the possibility that changing "eno3" to "none" on your system might have fixed the network problem?

Does anybody know how likely it is for my clueless guess to be right?

Friendly greetings from Mexico!

Falzo · July 2021

@Not_Oles said:

@miu said: i not tried (change bridge-ports from ens 3 to none)

I ran a little test.

My server (running Debian sid, not Proxmox) had bridge-ports none.

Just for fun I tried changing the "none" to "enp7s0", which is the name of my server's primary interface, corresponding to "eno3" on your server.

systemctl restart networking following this change broke networking, which was the hoped-for result.

I reverted via the rescue system.

Conceivably, my little test supports the possibility that changing "eno3" to "none" on your system might have fixed the network problem?

Does anybody know how likely it is for my clueless guess to be right?

Friendly greetings from Mexico!

your mixing things up here old friend.

he does not try to actually route (additional) IPs yet. he sets the initial bridge vmbr0 on the main IP (therefore ens3 config is empty/manual) and with that he MUST have something bridging to that interface - because it does not get an IP and therefore nothing can be routed to it.

if you'd later on add an additional bridge for a routed config you also do not route towards ens3 then but vmbr0. small but important difference to what you are suggesting.

TL;DR; I suggest to try and not mix up 'bridged config' and 'routed config' just yet ;-)

PS: with soyoustart/OVH the routed config does not work well anyway. but you'll get virtual macs for all additional IPs individually (even if bought in subnets). so the bridged config is correct and the better choice.

mine looks exactly the same, but I haven't looked at proxmox 7 yet and also always only install on top of debian 🤷‍♂️

bdl · July 2021

I had this exact issue with 7 on a Hosthatch VPS.

MaxKVM VPSes upgraded without issue.

I ended up rolling back to 6.4, definitely was a bug in pve 7. Didn't seem related to the MAC address changes in bullseye.

ehab · July 2021

hey @Not_Oles why not upgrade now and worry later

miu · July 2021

Good day good people!

UPDATE from another forum (credit @ polluxtroy3758) = what is in journalctl log on OVH-brands PVE7 server:

bridge port does not exist

-- Boot 1d7b66b4b10e4870b9c19a354be86dcf --
Jul 28 10:00:14 pve systemd[1]: Starting Network initialization...
Jul 28 10:00:14 pve networking[617]: networking: Configuring network interfaces
Jul 28 10:00:16 pve networking[675]: error: vmbr0: bridge port enp1s0 does not exist
Jul 28 10:00:16 pve systemd[1]: Finished Network initialization.
-- Boot fd6fb88a3e094183baf8a14c81f98d36 --
Jul 28 10:48:46 pve systemd[1]: Starting Network initialization...
Jul 28 10:48:46 pve networking[622]: networking: Configuring network interfaces
Jul 28 10:48:48 pve networking[718]: error: vmbr0: bridge port enp1s0 does not exist
Jul 28 10:48:48 pve systemd[1]: Finished Network initialization.

miu · July 2021

and here is (mine) working network log from SYS with PVE6.4 (installed from ISO) with bridged (above mentioned) config, bridge port found and works as should:

-- Logs begin at Tue 2021-07-27 20:23:58 CEST, end at Wed 2021-07-28 13:49:12 CEST. --
Jul 27 20:24:03 pve systemd[1]: Starting Raise network interfaces...
Jul 27 20:24:04 pve ifup[1059]: Waiting for vmbr0 to get ready (MAXWAIT is 2 seconds).
Jul 27 20:24:04 pve systemd[1]: Started Raise network interfaces.

miu · July 2021

no response from Proxmox staff on it yet (on their forum where i reported issue), when will be then i will update this thread also

Not_Oles · July 2021

@miu Thanks for the updates! Especially with the logs. Could you please post or PM me your Proxmox forum post link?

miu · July 2021

@Not_Oles said:
@miu Thanks for the updates! Especially with the logs. Could you please post or PM me your Proxmox forum post link?

Hello Oles (Tom)
SURE (hopefully i do not break any rules when post here URL from another forum)
https://forum.proxmox.com/threads/proxmox-node-on-ovh-soyoustart-unreachable-after-each-reboot.93470/

I will be glad when this may help to someone in future

For now as i have mentioned i have resolved it so that i have reinstalled machine from ISO 6.4 (no time, not own ability fix issue)

Not_Oles · July 2021

@miu Thanks again! Watching the thread on the Proxmox forum.

miu · July 2021

Nothing for thanking ;-) Unfortunately non-paid (customer) support there is not very well (slow and sometime w/o response of staff ever).. But i understand that for complete and good support i should purchase paid licence

Falzo · July 2021

@miu said:
Good day good people!

UPDATE from another forum (credit @ polluxtroy3758) = what is in journalctl log on OVH-brands PVE7 server:

bridge port does not exist

-- Boot 1d7b66b4b10e4870b9c19a354be86dcf --
Jul 28 10:00:14 pve systemd[1]: Starting Network initialization...
Jul 28 10:00:14 pve networking[617]: networking: Configuring network interfaces
Jul 28 10:00:16 pve networking[675]: error: vmbr0: bridge port enp1s0 does not exist
Jul 28 10:00:16 pve systemd[1]: Finished Network initialization.
-- Boot fd6fb88a3e094183baf8a14c81f98d36 --
Jul 28 10:48:46 pve systemd[1]: Starting Network initialization...
Jul 28 10:48:46 pve networking[622]: networking: Configuring network interfaces
Jul 28 10:48:48 pve networking[718]: error: vmbr0: bridge port enp1s0 does not exist
Jul 28 10:48:48 pve systemd[1]: Finished Network initialization.

from that the network device simply is not ready when the networkis tried to be brought up. could be that when you use dhcp it only fires at later point in time because of different dependencies and therefore does not see the same issues.

might even be unrelated to OVH/SYS specifics, but could also be related to drivers and whatnot... interesting. a workaround could probably be to fiddle with systemd and have it bring up networking only later.

miu · July 2021

@Falzo said:
@miu said:
Good day good people!

UPDATE from another forum (credit @ polluxtroy3758) = what is in journalctl log on OVH-brands PVE7 server:

bridge port does not exist
-- Boot 1d7b66b4b10e4870b9c19a354be86dcf --
Jul 28 10:00:14 pve systemd[1]: Starting Network initialization...
Jul 28 10:00:14 pve networking[617]: networking: Configuring network interfaces
Jul 28 10:00:16 pve networking[675]: error: vmbr0: bridge port enp1s0 does not exist
Jul 28 10:00:16 pve systemd[1]: Finished Network initialization.
-- Boot fd6fb88a3e094183baf8a14c81f98d36 --
Jul 28 10:48:46 pve systemd[1]: Starting Network initialization...
Jul 28 10:48:46 pve networking[622]: networking: Configuring network interfaces
Jul 28 10:48:48 pve networking[718]: error: vmbr0: bridge port enp1s0 does not exist
Jul 28 10:48:48 pve systemd[1]: Finished Network initialization.
from that the network device simply is not ready when the networkis tried to be brought up. could be that when you use dhcp it only fires at later point in time because of different dependencies and therefore does not see the same issues.

might even be unrelated to OVH/SYS specifics, but could also be related to drivers and whatnot... interesting. a workaround could probably be to fiddle with systemd and have it bring up networking only later.

Additionally - bridged config with DHCP also does not work on OVH with PVE7 (ISO installation)

auto lo
iface lo inet loopback

iface eno3 inet manual

auto vmbr0
iface vmbr0 inet dhcp
  bridge-ports eno3
  bridge-stp off
  bridge-fd 0

(on 6.4 ISO works)

Falzo · July 2021

@miu said: Additionally - bridged config with DHCP also does not work on OVH with PVE7 (ISO installation)

oh, okay, I misread that and thought this at least would work. however, still thinking systemd simply tries to start networking to early then while the real device is not ready.

miu · July 2021

Thank you for all your contribution, time and suggestions, appreciated by me!

ialexpw · July 2021

@miu said:
Thank you for all your contribution, time and suggestions, appreciated by me!

Had this exact problem when I moved to PVE7 (upgraded from 6.4). I messed with my interfaces for a while as the network appeared to be up after a restart for ~5 minutes before dying.

I'll probably have a note somewhere as to what happened on my side as I cannot remember what I changed, my interfaces file is as below now though;

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 91.xx.xx.xx/xx
        gateway 91.xx.xx.xx
        bridge-ports eth0
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet static
        address 10.10.10.1/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0

Which seems to be working - will update if I can find the logs from before! This is a dedi with Skynode.

ehab · July 2021

@ialexpw said:
I'll probably have a note somewhere as to what happened on my side as I cannot remember what I changed

i can help you remember

Proxmox 7 node on OVH unreachable after reboot

Comments

This Site is currently in maintenance mode.
Please check back here later.

Proxmox 7 node on OVH unreachable after reboot

Comments

This Site is currently in maintenance mode.Please check back here later.

This Site is currently in maintenance mode.
Please check back here later.