I got a reply on my second ticket, this time it wasn't auto closed.
Apparently its fixed but the VM is still down, I can't access their website right now, so I can't check anything.
According to them, they lost track of the ticket.
Anyway, I keep you updated.
"Sorry, too busy bombing Afghanistan and lost track of your ticket".
(probably) the support team
I speak fluent sarcasm and broken logic. | I would agree with you, but thæn we’d both be wrong.
PK is back online since about 2 days.
I missed the email from hetrix, however I got a Ticket response yesterday.
Someone or something fixed PK, but I don't know who, the Support that replied yesterday denies any allegations.
Basically they just replied that the VM is working and Online, as far as I know last time I checked I couldn't even boot it.
Anyway, enjoy your PK containers everyone.
I am unsure at this point if I gonna enable deployments on PK again.
The Console is currently still broken, I dislike most "fixes", I don't have a propper non hacky solution yet.
As stated in previously sent emails, as part of migration of Japan services to our new infrastructure at Equinix TY8 location, the following changes will be applied with regard to IPv6 addresses assigned to existing services:
• All of the services will automatically receive one dedicated routed /48 IPv6 prefix (once the migration is completed on 22nd April 2026).
Also
Maintenance Window:
Start Time: 1 AM 22nd April 2026 (Japan time)
Completion Time: 7 AM 22nd April (Japan time)
Impact: Service will be inaccessible during the migration phase (depending on disk size it can take 5 to 30 minutes per service)
@Neoon my Islamabad/PK container cannot be rebooted regardless of me trying from the Dashboard or from the Shell can you check? It is still online & accessible though!
Ah: stopping and starting the container worked!
@jcn50 said: @Neoon my Islamabad/PK container cannot be rebooted regardless of me trying from the Dashboard or from the Shell can you check? It is still online & accessible though!
Ah: stopping and starting the container worked!
You have been rebooting it successfully, according to logs, neither do I have issues rebooting.
What is the issue?
I have a service that starts @ boot time, and whatever I tried it did not start:
Dashboard Restart button
Dashboard Stop + Start buttons
Shell console reboot command
The uptime command gives the uptime of the host~ not of the container.
Still, I don't understand what your issue is, your container rebooted fine.
Yes some tools might give you different info depending on what and what os you are using.
@Neoon said: The Subnet in JP is ready and configured.
Thank you, I'm glad it has IPv6 now.
Here is a minor problem in Tokyo Equinix node however, when I try to install Arch Linux OS to a LXC VPS, the control panel prompted success and the IPv6 is reachable via ping, but SSH failed, and the port seems not open (I tried nmap -6 -p 22 IPV6-ADDRESS).
In the past, I found sshd in Arch Linux is not enabled via systemd, but since console or shell worked in the control panel, I could manually enable it there, but now console or shell seem not working on my side either. By the way, Debian OS works perfectly. Thank you.
@somik said: @Neoon can help to check on the alpine config when you are free?
Running free -h reports the host's memory in alpine, not the guest alpine OS, while cat /sys/fs/cgroup/memory.max shows the memory correctly. Probably some config issue or some weirdness with Alpine itself...
lxc5d7a7b88:~# free -h
total used free shared buff/cache available
Mem: 7.7G 5.8G 535.7M 361.3M 1.4G 246.5M
Swap: 0 0 0
lxc5d7a7b88:~# cat /sys/fs/cgroup/memory.max
268435456
lxc5d7a7b88:~# cat /sys/fs/cgroup/memory.current
27455488
@jcn50 said: The uptime command gives the uptime of the host~ not of the container.
@Neoon said: The Subnet in JP is ready and configured.
Thank you, I'm glad it has IPv6 now.
Here is a minor problem in Tokyo Equinix node however, when I try to install Arch Linux OS to a LXC VPS, the control panel prompted success and the IPv6 is reachable via ping, but SSH failed, and the port seems not open (I tried nmap -6 -p 22 IPV6-ADDRESS).
In the past, I found sshd in Arch Linux is not enabled via systemd, but since console or shell worked in the control panel, I could manually enable it there, but now console or shell seem not working on my side either. By the way, Debian OS works perfectly. Thank you.
its always DNS, the IPv6 of the node changed but nginx cached the DNS response, fixed.
@somik said: @Neoon can help to check on the alpine config when you are free?
Running free -h reports the host's memory in alpine, not the guest alpine OS, while cat /sys/fs/cgroup/memory.max shows the memory correctly. Probably some config issue or some weirdness with Alpine itself...
lxc5d7a7b88:~# free -h
total used free shared buff/cache available
Mem: 7.7G 5.8G 535.7M 361.3M 1.4G 246.5M
Swap: 0 0 0
lxc5d7a7b88:~# cat /sys/fs/cgroup/memory.max
268435456
lxc5d7a7b88:~# cat /sys/fs/cgroup/memory.current
27455488
@jcn50 said: The uptime command gives the uptime of the host~ not of the container.
Yea, nah, its disabled by default for a reason.
Its a nice feature to allow a container to display the correct memory usage.
However it requires a elevated process on the node to respond, which is limited of course but, nah.
Maintenance Announcement
We still got 4 nodes running LXD, that are yet not migrated.
The plan is for the next weekend to migrate them to Incus plus a distro upgrade with a new kernel.
This includes: Singapore, Tokyo Equinix, Oradea and Islamabad.
Expect up to 30 minutes downtime per Node, usually less.
Relocation Announcement
Our backend currently runs in AMS with Gigahost, Gigahost is gonna discontinue AMS on Wednesday morning.
Hence our backend gets physically moved to Norway, this should not take more than 24 hours.
During this time, the Panel including any operations will be unavailable.
However, containers on the nodes will be running as usual.
Backups are done, in case something goes wrong on the side of Gigahost, we are good.
I am gonna take the opportunity, to also containerize and compartmentalize our backend.
Which means the Panel might be unavailable for a short time today/tomorrow, during the migration.
Just a heads up.
@Neoon said: Relocation Announcement
Our backend currently runs in AMS with Gigahost, Gigahost is gonna discontinue AMS on Wednesday morning.
Hence our backend gets physically moved to Norway, this should not take more than 24 hours.
During this time, the Panel including any operations will be unavailable.
However, containers on the nodes will be running as usual.
Backups are done, in case something goes wrong on the side of Gigahost, we are good.
I am gonna take the opportunity, to also containerize and compartmentalize our backend.
Which means the Panel might be unavailable for a short time today/tomorrow, during the migration.
Just a heads up.
@Neoon said: Maintenance Announcement
We still got 4 nodes running LXD, that are yet not migrated.
The plan is for the next weekend to migrate them to Incus plus a distro upgrade with a new kernel.
This includes: Singapore, Tokyo Equinix, Oradea and Islamabad.
Expect up to 30 minutes downtime per Node, usually less.
@Neoon said: Maintenance Announcement
We still got 4 nodes running LXD, that are yet not migrated.
The plan is for the next weekend to migrate them to Incus plus a distro upgrade with a new kernel.
This includes: Singapore, Tokyo Equinix, Oradea and Islamabad.
Expect up to 30 minutes downtime per Node, usually less.
I will start at about 14:00 CET next Sunday.
I am targeting to upgrade this week Mumbai and Sandefjord, same time.
Just one reboot will do, about 5 minutes downtime.
I set Sandefjord, Helsinki and Meppel for new deployments, to KVM only.
A Buildserver for KVM templates will happen soonTM, so we can hopefully also offer 128MB and 64MB KVM's.
@Neoon said: Maintenance Announcement
We still got 4 nodes running LXD, that are yet not migrated.
The plan is for the next weekend to migrate them to Incus plus a distro upgrade with a new kernel.
This includes: Singapore, Tokyo Equinix, Oradea and Islamabad.
Expect up to 30 minutes downtime per Node, usually less.
I will start at about 14:00 CET next Sunday.
I am targeting to upgrade this week Mumbai and Sandefjord, same time.
Just one reboot will do, about 5 minutes downtime.
Regarding the failed deployments today and the unavailability of the console / shell.
A internal cert expired, it has been renewed but this caused deployments to fail and the console / shell to be unavailable for a few hours today.
I set Sandefjord, Helsinki and Meppel for new deployments, to KVM only.
A Buildserver for KVM templates will happen soonTM, so we can hopefully also offer 128MB and 64MB KVM's.
Kernel module loading for containers has been disabled on nodes where its possible.
The rest still have to be upgraded to 6.x until we can go and disable it.
As soon as news hits, these modules will be blacklisted asap anyway by me.
If you need a certain module loaded, you can send a DM.
Containers on Sandefjord have been enabled again (with custom packages).
Meppel will follow once upgraded to 6.x.
Livepatches for the Kernel are applied as usual, when published.
A few days ago, I got a email saying WH is gonna discontinue their old IPv4 subnet.
I just read this email today, I wasn't send to the correct mailbox, my bad.
Hence we have to change the IPv4 today, since its gonna stop working tomorrow.
I wish we had more time but seems to be urgent, its just a week.
New IP is displayed in the Panel, so you can update your DNS.
Will be swapped out later today.
We have sent you this email to let you know that we have network card failure on one of our Singapore nodes which has caused servers on that node become inaccessible.
Maintenance has been enabled for that node and we are investigating the cause to resolve the issue.
The Node in India had an outage today due to a fiber cut, both redundant path went down due to unforeseen circumstances, this has been resolved though.
Bump justification: This is an ongoing thread with updates from time to time, now maybe the HAProxy on Tokyo Equinix doesn't work as expected, so I'm asking @Neoon for help.
@bliss said:
Hello @Neoon,
The HAProxy on Tokyo Equinix node seems not working, I'd appreciate it if you could check out when you are free.
Thank you.
I see no problem, please provide a bit more info.
Yes, you are right, it works as expected. I'm sorry for this misreport. I'll check twice in the future, and thank you very much.
Details are:
1. I've set up the hostname, pointing to the public IPv4, then opened a test python script on port 80 like this sudo python -m http.server 80, and disabled the firewall for test. But I couldn't visit the hostname with Firefox browser with error code "NS_ERROR_NET_EMPTY_RESPONSE";
2. Then you replied the service was fine, so I tried with curl -4 http://www.my-domain.com, and it did work fine.
So I guess I've tuned the config of Firefox too strictly, and I'm sorry for this misunderstanding. It's great the redirect rule on Cloudflare can be removed, and use default 443 instead.
Comments
"Sorry, too busy bombing Afghanistan and lost track of your ticket".
Restocked: Tokyo, Singapore, Bratislava and Valdivia
Free NAT KVM | Free NAT LXC
FYI, alpine linux does not "shutdown" with the "Stop" button in panel, probably cause alpine uses
poweroffinstead of the standardshutdown -h 0PK is back online since about 2 days.
I missed the email from hetrix, however I got a Ticket response yesterday.
Someone or something fixed PK, but I don't know who, the Support that replied yesterday denies any allegations.
Basically they just replied that the VM is working and Online, as far as I know last time I checked I couldn't even boot it.
Anyway, enjoy your PK containers everyone.
I am unsure at this point if I gonna enable deployments on PK again.
The Console is currently still broken, I dislike most "fixes", I don't have a propper non hacky solution yet.
Free NAT KVM | Free NAT LXC
The console issue has been fixed, sorry that it took so long.
The patch included security fixes, so I could not hesitate updating incus/lxd.
Free NAT KVM | Free NAT LXC
Tokyo is getting its prefix, tomorraw.
As stated in previously sent emails, as part of migration of Japan services to our new infrastructure at Equinix TY8 location, the following changes will be applied with regard to IPv6 addresses assigned to existing services:
• All of the services will automatically receive one dedicated routed /48 IPv6 prefix (once the migration is completed on 22nd April 2026).
Also
Maintenance Window:
Start Time: 1 AM 22nd April 2026 (Japan time)
Completion Time: 7 AM 22nd April (Japan time)
Impact: Service will be inaccessible during the migration phase (depending on disk size it can take 5 to 30 minutes per service)
Free NAT KVM | Free NAT LXC
And the SG dedicated server I got from oneprovider just gives a /128 IPv6 and they say they can't even add additional IPv6 IPs to the server...
Oh well, IPv6 adoption amongst the provider is still very poor...
@Neoon my Islamabad/PK container cannot be rebooted regardless of me trying from the Dashboard or from the Shell
can you check? It is still online & accessible though!
Ah: stopping and starting the container worked!
💩 VirCrap 💩
You have been rebooting it successfully, according to logs, neither do I have issues rebooting.
What is the issue?
Free NAT KVM | Free NAT LXC
I have a service that starts @ boot time, and whatever I tried it did not start:
rebootcommandThe
uptimecommand gives the uptime of the host~ not of the container.💩 VirCrap 💩
Nevermind: everything seems to be working fine now! Sorry for the bother!
💩 VirCrap 💩
Still, I don't understand what your issue is, your container rebooted fine.
Yes some tools might give you different info depending on what and what os you are using.
Free NAT KVM | Free NAT LXC
I also added PK back in stock, since it has been stable the last 3 weeks.
Free NAT KVM | Free NAT LXC
The Subnet in JP is ready and configured.
I will do a quick reboot tomorrow and enable IPv6 afterwards.
As soon its available, just press reconfigure in the panel.
Free NAT KVM | Free NAT LXC
Done.
Free NAT KVM | Free NAT LXC
Thank you, I'm glad it has IPv6 now.
Here is a minor problem in Tokyo Equinix node however, when I try to install Arch Linux OS to a LXC VPS, the control panel prompted success and the IPv6 is reachable via ping, but SSH failed, and the port seems not open (I tried nmap -6 -p 22 IPV6-ADDRESS).
In the past, I found sshd in Arch Linux is not enabled via systemd, but since console or shell worked in the control panel, I could manually enable it there, but now console or shell seem not working on my side either. By the way, Debian OS works perfectly. Thank you.
MicroLXC is lovable.
This is because Alpine relies on sysinfo for that information.
To fix it you have to set
security.syscalls.intercept.sysinfo=trueunder the instance setting for each Alpine container and restart itreference:
Instance options - Security policies - security.syscalls.intercept.sysinfo
its always DNS, the IPv6 of the node changed but nginx cached the DNS response, fixed.
Free NAT KVM | Free NAT LXC
Yea, nah, its disabled by default for a reason.
Its a nice feature to allow a container to display the correct memory usage.
However it requires a elevated process on the node to respond, which is limited of course but, nah.
Free NAT KVM | Free NAT LXC
Lads and Gents, we got a new Node in Ho Chi Minh.
Courtesy of Onidel, thanks @oloke
/64 routed iPv6, HAPorxy, DNS Proxy the usual.
Free NAT KVM | Free NAT LXC
Richard Nixon back again?
"It's a hard life- to be a stick insect." - Karl Pilkington
Is that a China node? Wow!
Free NAT KVM | Free NAT LXC
Maintenance Announcement
We still got 4 nodes running LXD, that are yet not migrated.
The plan is for the next weekend to migrate them to Incus plus a distro upgrade with a new kernel.
This includes: Singapore, Tokyo Equinix, Oradea and Islamabad.
Expect up to 30 minutes downtime per Node, usually less.
I will start at about 14:00 CET next Sunday.
Free NAT KVM | Free NAT LXC
Relocation Announcement
Our backend currently runs in AMS with Gigahost, Gigahost is gonna discontinue AMS on Wednesday morning.
Hence our backend gets physically moved to Norway, this should not take more than 24 hours.
During this time, the Panel including any operations will be unavailable.
However, containers on the nodes will be running as usual.
Backups are done, in case something goes wrong on the side of Gigahost, we are good.
I am gonna take the opportunity, to also containerize and compartmentalize our backend.
Which means the Panel might be unavailable for a short time today/tomorrow, during the migration.
Just a heads up.
Free NAT KVM | Free NAT LXC
Done, its up again.
Free NAT KVM | Free NAT LXC
Erm, ok... so either wait for a while or do it again so it goes down?
If it stays up for more then 4 hours, see a doctor!
Done.
Free NAT KVM | Free NAT LXC
I am targeting to upgrade this week Mumbai and Sandefjord, same time.
Just one reboot will do, about 5 minutes downtime.
Free NAT KVM | Free NAT LXC
Just a heads up.
I set Sandefjord, Helsinki and Meppel for new deployments, to KVM only.
A Buildserver for KVM templates will happen soonTM, so we can hopefully also offer 128MB and 64MB KVM's.
Free NAT KVM | Free NAT LXC
Done.
Free NAT KVM | Free NAT LXC
Regarding the failed deployments today and the unavailability of the console / shell.
A internal cert expired, it has been renewed but this caused deployments to fail and the console / shell to be unavailable for a few hours today.
Free NAT KVM | Free NAT LXC
Kernel module loading for containers has been disabled on nodes where its possible.
The rest still have to be upgraded to 6.x until we can go and disable it.
As soon as news hits, these modules will be blacklisted asap anyway by me.
If you need a certain module loaded, you can send a DM.
Containers on Sandefjord have been enabled again (with custom packages).
Meppel will follow once upgraded to 6.x.
Livepatches for the Kernel are applied as usual, when published.
Free NAT KVM | Free NAT LXC
Heads up for the India Node.
A few days ago, I got a email saying WH is gonna discontinue their old IPv4 subnet.
I just read this email today, I wasn't send to the correct mailbox, my bad.
Hence we have to change the IPv4 today, since its gonna stop working tomorrow.
I wish we had more time but seems to be urgent, its just a week.
New IP is displayed in the Panel, so you can update your DNS.
Will be swapped out later today.
Free NAT KVM | Free NAT LXC
WebHorizon: Emergency Maintenance (Singapore)
We have sent you this email to let you know that we have network card failure on one of our Singapore nodes which has caused servers on that node become inaccessible.
Maintenance has been enabled for that node and we are investigating the cause to resolve the issue.
Free NAT KVM | Free NAT LXC
Heads up for the Romania Node
There are planned IP address changes, but I don't know when to be precise.
I will update you when I know more.
Free NAT KVM | Free NAT LXC
Nevermind, doesn't affect the microLXC node.
Free NAT KVM | Free NAT LXC
NL got rebooted and migrated to another Node.
Live migration wasn't possible due to the cpu passthrough we got enabled.
Hence the 30 minutes downtime.
Enjoy the CPU upgrade.
Free NAT KVM | Free NAT LXC
Rebooted Helsinki, since the bookworm patch just got available.
Free NAT KVM | Free NAT LXC
The Node in India had an outage today due to a fiber cut, both redundant path went down due to unforeseen circumstances, this has been resolved though.
Free NAT KVM | Free NAT LXC
Hello @Neoon,
The HAProxy on Tokyo Equinix node seems not working, I'd appreciate it if you could check out when you are free.
Thank you.
MicroLXC is lovable.
I see no problem, please provide a bit more info.
Free NAT KVM | Free NAT LXC
Yes, you are right, it works as expected. I'm sorry for this misreport. I'll check twice in the future, and thank you very much.
Details are:
1. I've set up the hostname, pointing to the public IPv4, then opened a test python script on port 80 like this
sudo python -m http.server 80, and disabled the firewall for test. But I couldn't visit the hostname with Firefox browser with error code "NS_ERROR_NET_EMPTY_RESPONSE";2. Then you replied the service was fine, so I tried with curl -4 http://www.my-domain.com, and it did work fine.
So I guess I've tuned the config of Firefox too strictly, and I'm sorry for this misunderstanding. It's great the redirect rule on Cloudflare can be removed, and use default 443 instead.
Thank you and have a nice day, Neoon.
MicroLXC is lovable.