Proxmox 7 node on OVH unreachable after reboot
Hello, can somebody help me resolve following maddness pls?
Proxmox host node is OVH network dedicated (SoYouStart)
/etc/network/interfaces:
auto lo
iface lo inet loopback
iface eno3 inet manual
auto vmbr0
iface vmbr0 inet static
address 54.38.192.11/24
gateway 54.38.192.254
bridge-ports eno3
bridge-stp off
bridge-fd 0
1) After node server full reboot is network UNREACHABLE
2) But when i run SSH command:
service networking restart
network is reachable and work fine:
# ip route
default via 54.38.192.254 dev vmbr0
54.38.192.0/24 dev vmbr0 proto kernel scope link src 54.38.192.11
I do not know resolve this madness WHY after reboot mentioned config does not work = network unreachable, but after additional manual networking restart the same config works fine....
BTW: i do not want use DHCP config = default (automatic OVH install) config:
auto lo
iface lo inet loopback
iface eno3 inet manual
auto vmbr0
iface vmbr0 inet dhcp
bridge-ports eno3
bridge-stp off
bridge-fd 0
I want use STATIC config, not DHCP
many thanks in advance for each attempt to help resolve this madness
Good day and Goodbye
Comments
is that the only network config file?
dirty fix - just add an init script in /etc/init.d/ to restart networking at every reboot.
https://webhorizon.net
update 2: after searching on net i found that several ppl report the same problem after upgrade on PROXMOX 7 (unreachable network). So probably version 7 is still pretty bugged. (maybe reason why OVH still not offers Proxmox 7 template?) I tried several changes in config but not resolved this (not found exact reason and clean solution), without success
Good day and Goodbye
Hi @miu!
Please answer a few questions from a clueless guy:
(A) In your original post,
How do you run the ssh command when the network is unreachable?
(B) What do the logs say about bringing the network up as part of the reboot? Is there something that failed?
(C) In your
interface eno3 is set to "manual."
In man (5) interfaces, it says,
Further down in the vmbr0 section, your /etc/network interfaces also has the line,
bridge-ports eno3. What is supposed to happen when ifup reaches thatbridge-ports eno3line following reboot? Could it be that there is a failure at thatbridge-ports eno3line because eno3 is called but eno3 is not present because manual configuration was specified for eno3?I have seen other similar configurations which specify
bridge-ports noneinstead ofbridge-ports eno3. Proxmox' Routed Configuration example hasbridge-ports none:Maybe your issue could be resolved by changing
bridge-ports eno3tobridge-ports none?In summary:
Could your issue be resolved by changing
bridge-ports eno3tobridge-ports none?Hope this is helpful! Best wishes from Mexico!
Hello Oles, firstly thank you for your response!
ANSWER @ A) Remotely Through IPMI console
ANSWER @ B & C:
Thanks for your suggestion - this i not tried (change bridge-ports from ens 3 to none)
From time reasons after several hours of frustration i reinstalled whole node from ISO 6.4 - it works FINE (network) with my above mentioned origin static config
Maybe i will try your suggested change next time when install another Proxmox node from v7 ISO, then i will can update this
thanks! regards miu
Good day and Goodbye
So just now i can conclude only:
works on Proxmox 6.4 + SYS well (network works without any troubles)
but on new Proxmox v7 + SYS not at all (network UNREACHABLE)
Good day and Goodbye
@miu i would look at the networking service if its started at reboot.
look at the kernel logs in the console etc ...
but if its a bug in v.7 then you have to live with it until update.
@ehab Yeah. for sure, but @miu can't look at the logs after the machine is reinstalled.
u r right - exactly ma face after suffering from this issue! :-D
Good day and Goodbye
I'm using a similar network config with Proxmox VE 7.0-9 and no problems at all.
But I didn't use the ISO. Installed on top of Debian.
Somehow I missed that 7 was available - still running 6.4 on my boxes. Thanks for the heads-up.
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
Reddit is full of people bombing their boxes with 7 upgrade. I've decided I'll wait a month or 5. There isn't really anything I need in 7...
I'll run the tests and make sure that I have done some updates on my Proxmox Backup Server.
@miu Re. networking issue; I wonder if this is the cause:
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
@miu I ran a little test.
My server (running Debian sid, not Proxmox) had
bridge-ports none.Just for fun I tried changing the "none" to "enp7s0", which is the name of my server's primary interface, corresponding to "eno3" on your server.
systemctl restart networkingfollowing this change broke networking, which was the hoped-for result.I reverted via the rescue system.
Conceivably, my little test supports the possibility that changing "eno3" to "none" on your system might have fixed the network problem?
Does anybody know how likely it is for my clueless guess to be right?
Friendly greetings from Mexico!
your mixing things up here old friend.
he does not try to actually route (additional) IPs yet. he sets the initial bridge vmbr0 on the main IP (therefore ens3 config is empty/manual) and with that he MUST have something bridging to that interface - because it does not get an IP and therefore nothing can be routed to it.
if you'd later on add an additional bridge for a routed config you also do not route towards ens3 then but vmbr0. small but important difference to what you are suggesting.
TL;DR; I suggest to try and not mix up 'bridged config' and 'routed config' just yet ;-)
PS: with soyoustart/OVH the routed config does not work well anyway. but you'll get virtual macs for all additional IPs individually (even if bought in subnets). so the bridged config is correct and the better choice.
mine looks exactly the same, but I haven't looked at proxmox 7 yet and also always only install on top of debian 🤷♂️
I had this exact issue with 7 on a Hosthatch VPS.
MaxKVM VPSes upgraded without issue.
I ended up rolling back to 6.4, definitely was a bug in pve 7. Didn't seem related to the MAC address changes in bullseye.
hey @Not_Oles why not upgrade now and worry later
Good day good people!
UPDATE from another forum (credit @ polluxtroy3758) = what is in journalctl log on OVH-brands PVE7 server:
bridge port does not exist
Good day and Goodbye
and here is (mine) working network log from SYS with PVE6.4 (installed from ISO) with bridged (above mentioned) config, bridge port found and works as should:
-- Logs begin at Tue 2021-07-27 20:23:58 CEST, end at Wed 2021-07-28 13:49:12 CEST. --
Jul 27 20:24:03 pve systemd[1]: Starting Raise network interfaces...
Jul 27 20:24:04 pve ifup[1059]: Waiting for vmbr0 to get ready (MAXWAIT is 2 seconds).
Jul 27 20:24:04 pve systemd[1]: Started Raise network interfaces.
Good day and Goodbye
no response from Proxmox staff on it yet (on their forum where i reported issue), when will be then i will update this thread also
Good day and Goodbye
@miu Thanks for the updates! Especially with the logs. Could you please post or PM me your Proxmox forum post link?
Hello Oles (Tom)
SURE (hopefully i do not break any rules when post here URL from another forum)
https://forum.proxmox.com/threads/proxmox-node-on-ovh-soyoustart-unreachable-after-each-reboot.93470/
I will be glad when this may help to someone in future
For now as i have mentioned i have resolved it so that i have reinstalled machine from ISO 6.4 (no time, not own ability fix issue)
Good day and Goodbye
@miu Thanks again! Watching the thread on the Proxmox forum.
Nothing for thanking ;-) Unfortunately non-paid (customer) support there is not very well (slow and sometime w/o response of staff ever).. But i understand that for complete and good support i should purchase paid licence
Good day and Goodbye
from that the network device simply is not ready when the networkis tried to be brought up. could be that when you use dhcp it only fires at later point in time because of different dependencies and therefore does not see the same issues.
might even be unrelated to OVH/SYS specifics, but could also be related to drivers and whatnot... interesting. a workaround could probably be to fiddle with systemd and have it bring up networking only later.
Additionally - bridged config with DHCP also does not work on OVH with PVE7 (ISO installation)
(on 6.4 ISO works)
Good day and Goodbye
oh, okay, I misread that and thought this at least would work. however, still thinking systemd simply tries to start networking to early then while the real device is not ready.
Thank you for all your contribution, time and suggestions, appreciated by me!
Good day and Goodbye
Had this exact problem when I moved to PVE7 (upgraded from 6.4). I messed with my interfaces for a while as the network appeared to be up after a restart for ~5 minutes before dying.
I'll probably have a note somewhere as to what happened on my side as I cannot remember what I changed, my interfaces file is as below now though;
Which seems to be working - will update if I can find the logs from before! This is a dedi with Skynode.
piximg image and gallery hosting · fa.to client-side encrypted storage
i can help you remember
@miu
I checked your post on the Proxmox Forum to see whether Proxmox replied. I found no reply.
Then I remembered this 2020 Proxmox forum thread where Wolfgang from the Proxmox staff explains that "network configuration is out of scope" so that even buying a Proxmox subscription "would help nothing" for a network configuration question.
Further up in the same thread Wolfgang said "We do no configuration."
Here is a link to the 2020 Proxmox forum thread as archived on archive.org.
Maybe network configuration being "out of scope" might explain why Proxmox seems not to have replied to your post on their forum?
Friendly greetings from the Sonoran Desert! ☀️
Hi Tom, Check proxmox thread
Is possible that dude who have updated it found solution (add hwaddress param into config file)
When i will have any time i will try/test it then i'll be back with result
Have a great weekend everyone!
Good day and Goodbye
I already mentioned that, on page one:
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
Truth.
I assume seems be solution. And apologies for my blindness.
Good day and Goodbye
interesting still, that this was not needed before Proxmox 7 and maybe related to sticking to ifupdown even with the newer kernel? also according to your findings simply restarting the networking later in the process seemed to work even without providing hwaddress in that config, so...
Can confirm that adding the hwaddress definitely fixed this issue. Just did a bunch of new installs as well as test migrations from pve 6.4 to 7 to confirm. No idea why that didn't work for me the first time!
1) use 'ip -c link' to determine the mac address of the physical interface (eg. ens3)
2) edit /etc/network/interfaces and specify this mac (using hwaddress ) on the bridge (eg. vmbr0)
Installing 'ifupdown2' on an existing pve 6.4 install also worked without having to muck around with the hwaddress
only so I undestand this right (as it will be very helpful): so you had the same problems with networking after upgrading from 6.4 to pve7 but just installing this package before the migration fixed it as well, without having to touch any configs?
that'd be the preferred upgrade path then I'd say.
can you recall if your pve6 has been initially installed from iso or on top of debian?
hey @Falzo,
Hope this helps:
When 7 was first released, tried two 6.4->7 upgrades on boxes from MaxKVM. All single IP machines, installed with PVE 6 .iso, with a bridged nic to the outside world. From memory I think I'd installed ifupdown2 on these, but can't remember for sure. Had no problems whatsoever so thought I'd try it on one of my Chicago Hosthatch box. Chicago Hosthatch box had also been installed from a Pve 6.x iso, so I did the 6 to 7 Proxmox upgrade test like I'd done for the MaxKVM boxes, it passed, so I went ahead.
After the upgraded completed successfully, I lost network access on reboot. From memory, I tried renaming nics and re-establishing the bridges, nothing worked. After reinstalling 6.4 from iso and having networking working, and then also repeating the process a second time with the same result, I just wrote it off as PVE 7 teething problems and was going to wait for it to be patched out. The Proxmox documentation describing the 'macaddress' option wasn't as detailed as it is now and (I think) I did gloss over it but it didn't seem applicable. I had private networking set up on that box too, so wasn't sure if that was causing more problems.
I purchased a new Hosthatch service (in a different location) which was provisioned today, and I was going to install PVE 6.x on it given the issues I'd had with 7 previously (again, waiting for 7 to be updated and migrating at a later date), but thought I'd do more testing as some of the comments in this thread seemed to be inconclusive. Tried an install from .iso for 7, no networking. Reinstalled with 6.x .iso, networking worked perfectly.
I then installed ifupdown2 on the 6.x .iso install, ran the upgrade script and it upgraded no problem.
Then, I thought, "well, that works but I can't be stuffed installing 6.x and then upgrading for future installs", so tried a 7 .iso installation. Again, no networking post upgrade. I checked the mac's with 'ip -c link' and noticed that my bridge had a different macaddress to the physical interface. I then edited /etc/network/interfaces and defined 'hwaddress' for the bridge, then reloaded the network using 'ifreload -a' and could verify I had Internet connectivity. Tested and it survived a reboot.
Was curious to see ifupdown2 had been installed by default on 7.x and it had, I definitely had to add the macaddress for my bridge to come up.
I'm going to do a fresh install on the Hosthatch Chicago box I referred to above. Going to install 6.4 from iso, update, install ifupdown2 and then do the inplace upgrade to 7. Will report back if it works OK.
Apologies for the waffle, hope it helps. Let me know if you have any other questions.
Argh! Don't know why i typed "macaddress" instead of "hwaddress" above..
Anyway, did some more testing with fresh installs of PVE 6.4 from iso.
First install, installed 6.4, apt-upgraded, ran the update script - then lost connectivity after I rebooted into PVE 7. Running 'ip -c link' verified that the bridge's MAC was different to that of the physical interface it was meant to bridge. I edited /etc/network/interfaces and added 'hwaddress' for the bridge and set the MAC to be the same as the physical interface. Then ran '/etc/init.d/networking' to restart networking and the network came back.
Second install, installed 6.4, apt-upgraded, installed 'ifupdown2', then ran the update script - didn't lose connectivity after rebooting into PVE 7. Verified that MAC addresses for bridge and physical interfaces were the same using 'ip -c link'.
Final test was a clean install with PVE 7 .iso. No networking after install and reboot. Confirmed bridge and and physical interface MAC addresses were different using 'ip -c link'. Modified /etc/network/interfaces accordingly, and after a 'service networking restart' networking was good to go!
@bdl thanks a lot for all the information and tests, that helps a lot. could well be that with (old) ifupdown the bridge does not get a new/seperate hwaddress and that this is only introduced with the new version. that said, if one updates to ifupdown2 via apt it might run some post-inst script that detects and pins the hwaddress, while the pve udpate script somehow seems to miss that...
I am still waiting with the updating as for now I don't see the need... but at some point obviously need to do that and will make sure to switch to ifupdown2 beforehand then. though, my installation are all not from ISO but on top of debian.
again thanks a lot for the efforts and report!
@bdl Well done, for the time/effort/perseverance.
In stasis until the shitposting stops/abates.
Than=compare;then=sequence:brought=bring;bought=buy:staffs=pile of sticks:informations/infos=no plural.
It wisnae me! A big boy done it and ran away. || NVMe2G for life! until death (the end is nigh).
I thought I'd just add my random $0.02 - one of the funky things with (I think) newer kernels and/or systemd (or some crazy lunar configuration between them) is that the there's some dynamic sorcery going on for automatic bridge devices (at least at startup). What seems to happen is that the mac address is dynamically picked to be the lowest (sorted) MAC address of all the devices that are part of the bridge. This is where there seems to be some shenanigans going on. I generally avoid systemd but with Proxmox, that is (so far) not possible.
Effectively by forcing the hw address to be set, you are changing this unpredictability and everything works automagically.
This is all theory of course
@Falzo I'm actually surprised the upgrade checking script doesn't check for the presence of ifupdown2, that would make more sense to me. I'm like you, I didn't really want to be pfutzing around with hwaddress entries if the installation of a package would resolve the issue, hence my testing to try and figure it out once and for all
Until 7.1 
My Hosthatch Chicago machine has two ens devices too which further complicated things when I first attempted an upgrade, from memory I think there there was funny buggers with the bridge grabbing the mac address of the physical interface not in use. I ended up cracking it and just re-installing 6.4 at the time
You're all very welcome, glad I could contribute something small back to the community. I am still going to hold back upgrading most of my infrastructure, however the newer Hosthatch service is a 10Tb storage server and I didn't really want to have to try and juggle 10Tb around for a failed upgrade in the future so that's staying at 7.x.
As I upgrade my other boxen, if I notice anything not in line with what I posted above, I'll post again to the thread.
Just tried another server:
Existing 6.4 iso install, verified that ifupdown2 was installed before commencing upgrade check and upgrade itself.
Proxmox upgraded while maintaining network connectivity - no manual tweaking of hwaddress required!
I installed PVE 6.4 on top of Debian 10, tried to reboot, but I unable to add any additional network, it was solved with install
ifupdown2and restart the vps⭕ A simple uptime dashboard using UptimeRobot API https://upy.duo.ovh
⭕ Currently using VPS from BuyVM, GreenCloudVPS, Gullo's, Hetzner, HostHatch, InceptionHosting, LetBox, MaxKVM, MrVM, VirMach.