Welcome to LES! Congrats on your first post! I am delighted to be the first to thank you!
Want to share a little about who and where you are, your Linux experience, and what you want to do on the server? I think everyone will be glad to meet you!
@Not_Oles said: @Not_Oles said: LXC still is not expected to work yet partly because lxcbr0 isn't present because /etc/network/interfaces hasn't been updated yet. There is no bridge in the current setup. I sent the possible new interfaces configuration to Cloudie, so we will see what he says.
Haven't yet heard back from @Cloudie. He is a great guy! Sometimes he gets busy, and that's okay.
@Cloudie is very helpful and kind. But he still seems busy. Maybe my message about the network configuration went astray. I took the liberty of bumping that message
@Not_Oles said: @Not_Oles said: Hi Guys! Heard from Cloudie. He said he is checking. Best! Tom
Heard again from @Cloudie, who said one of the routers is misbehaving.
Let's pause adding new accounts until the situation with the router is resolved. Hopefully soon!
Thanks everyone!
Even though there doesn't yet seem to be more news, it's beyond doubt that @Cloudie still is working hard on the router situation.
Welcome to LES! Congrats on your first post! I am delighted to be the first to thank you!
Want to share a little about who and where you are, your Linux experience, and what you want to do on the server? I think everyone will be glad to meet you!
I have been using Windows for a long time since my school, but after I choose development path I find difficult to work in it.
One of my friend recommended linux.I tried it out, and it has an amazing development environment.
I searched for some cheap linux servers, but all were too costly for my usage and I find this forum.
I like to have a linux server to improve my coding skills and linux knowledge. I will use it to learn more in linux and run/test some of my codes in linux environment.
Welcome to LES! Congrats on your first post! I am delighted to be the first to thank you!
Want to share a little about who and where you are, your Linux experience, and what you want to do on the server? I think everyone will be glad to meet you!
I have been using Windows for a long time since my school, but after I choose development path I find difficult to work in it.
One of my friend recommended linux.I tried it out, and it has an amazing development environment.
I searched for some cheap linux servers, but all were too costly for my usage and I find this forum.
I like to have a linux server to improve my coding skills and linux knowledge. I will use it to learn more in linux and run/test some of my codes in linux environment.
Thanks.
Hi Melvik!
Thanks for your message!
Sorry, does TJC mean "Tyler Junior College?"
Do you have any code or anything else about you online anywhere?
Have you tried any of the free accounts available for students from Google, Oracle, Amazon, Azure, etc?
@tme said:
Hi, here is my public SSH key. ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDntE+N38zRGvpDB3eKffGBoUrTFR+4wylNevShXAXlZZA3ihJm9dHagj3+XT/cAdIParvZbJiqtjXPNOaRsKORClOjQLKqEnhtXlIHEFZnfjBCMVpYc5+Jnh/1MtsHDnD/ZkE3OWS1eGN+Ge3IU0aHELDJ6SzqGOUyI8qkpMfMuWK7kQzsoVBJJdsFSoQ4T4HtaqolR9HAIPxni6Ayo4ekZCm2WaRufTVI1aRCUGSdpWjAlE/S+HHajQaZ5LuhEDBQkGV7NJejwcuEM8nJIyQif0aB85eFHdVF1VQFn7wA9hFIHchvsVsjZSI4azrxenIWblrdKcgXI7gZR3qcGpzr8Pflgtb0P9rB2VShO/tj30soLvtkL+sZLHanFTnDPrqofA1S2R9pPsAFa6y1YUdwDnJvJptFuaJgme2qSmU6AdfFqbJjJDE5XxIIk7j6YbSdkyJJPZEeM5G1Sj0NkAhcssabL9CejP3vR4EJXyHfNuIBh4IEo1jKNCL1GJPeTHM= user@TME-PC
Thank you for asking! Sorry we haven't been going faster!
We are waiting on two issues, (1) a router sometimes is not working, and (2) we are looking for approval from @Cloudie on the proposed, new /etc/network/interfaces. With (2), the questions include whether exposing the container MAC addresses on the physical network is a problem.
You are more than welcome to compile on one of my other servers. The other guys here also are more than welcome on my other servers.
If you want to take over the network configuration here or provide even more help with how it should be done, then we can go faster on the configuration. But there also are the router issues.
January 4 -- @yoursunny says "Bridging containers directly on the physical port may result in the containers' MAC addresses becoming visible on the physical network. I don't know about Cloudie, but doing this in KVM would get filtered in Virtualizor, and doing this on Hetzner would trigger infraction warning letter." @Not_Oles decides he ought to check with @Cloudie to see whether the proposed /etc/network/interfaces configuration is okay.
January 4 -- Revised /etc/network/interfaces proposed
January 7 -- Multiple downtime/uptime cycles previous night. @Not_Oles says, ". . . when he has time, I'm sure i will hear from @Cloudie on the server /etc/network/interfaces configuration. . . ."
January 7 -- @Not_Oles reports, "Heard from Cloudie. He said he is checking."
January 8 -- @Cloudie says one of the routers is misbehaving. @Not_Oles pauses new accounts until the router issue is resolved.
January 14 -- @Not_Oles says, "@Cloudie is very helpful and kind. But he still seems busy. Maybe my message about the network configuration went astray. I took the liberty of bumping that message."
January 14 -- @Not_Oles says, "Even though there doesn't yet seem to be more news, it's beyond doubt that @Cloudie still is working hard on the router situation."
I believe you can create lxcbr0 not joined with the uplink, so that no container MAC address would show up on the hardware side.
I did some trials on an ephemeral Alpine 3.17 system, created by netboot.xyz and running in tmpfs.
The IP addresses in the sample are:
The host has a primary IPv4 (not shown), used for host originated traffic.
192.168.5.188 is the host's secondary IPv4, used for outgoing NAT traffic from containers.
192.168.5.188 is marked preferred_lft 0 so that it would not be auto-selected for host originated traffic.
192.168.188.0/24 is assigned to containers.
Traffic from this subnet is NAT'ed.
2600:4040:2ca4:a5bc::/64 is assigned to containers.
This prefix must be routed to the server.
If there's only on-link prefix, NDP responder would be required to convert it into a routed prefix.
These are typed by root into Alpine console:
: install necessary packages
echo http://dl-cdn.alpinelinux.org/alpine/v3.17/community | tee -a /etc/apk/repositories
apk update
apk add iptables lxc lxc-download lxcfs shadow-subids xz
: enable cgroups
rc-update add cgroups
rc-service cgroups start
: create LXC bridge and assign addresses
ip link add lxcbr0 type bridge
ip addr add 192.168.188.1/24 dev lxcbr0
ip addr add 2600:4040:2ca4:a5bc::/64 dev lxcbr0
: enable IPv4 and IPv6 forwarding
sysctl net.ipv4.ip_forward=1
sysctl net.ipv6.conf.all.forwarding=1
: setup IPv4 NAT, with a secondary public IPv4 address for outgoing traffic
ip addr add 192.168.5.188/24 dev eth0 preferred_lft 0
iptables -t nat -A POSTROUTING --src 192.168.188.0/24 -o eth0 -j SNAT --to 192.168.5.188
: create a user and grant permission for subids and LXC bridge
adduser user
echo 'user:100000:65536' | tee /etc/subuid /etc/subgid
echo 'user veth lxcbr0 100' | tee /etc/lxc/lxc-usernet
These are typed by user into Alpine console:
: write LXC config defaults
mkdir -p ~/.config/lxc
cat > ~/.config/lxc/default.conf <<EOT
lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.idmap = u 0 100000 65536
lxc.idmap = g 0 100000 65536
EOT
: create Ubuntu 22.04 container and assign IP addresses
lxc-create -n ubuntu -t download -- -d ubuntu -r jammy -a amd64
tee -a ~/.local/share/lxc/ubuntu/config <<EOT
lxc.net.0.ipv4.address = 192.168.188.2/24
lxc.net.0.ipv4.gateway = 192.168.188.1
lxc.net.0.ipv6.address = 2600:4040:2ca4:a5bc::2/64
lxc.net.0.ipv6.gateway = 2600:4040:2ca4:a5bc::
lxc.init.cmd = /bin/bash
EOT
: start the container
lxc-start -n ubuntu
: attach to the container console
lxc-attach -n ubuntu
: kill the container
lxc-stop -k -n ubuntu
These are typed into container console:
: set DNS server
rm /etc/resolv.conf
echo 'nameserver 2600:4700:4700::1111' | tee /etc/resolv.conf
: network will work after this
Caveats:
All the network configs on the host are ephemeral.
They must be re-typed after the host reboots.
IPv6 into containers is on-link, not routed.
It's possible to add routed IPv6 subnet, by running one ip route add command for each container on the host.
IP address conflicts between containers are possible, if two users assign the same address.
If a conflict occurs, the container that starts later will typically show the address as dadfailed.
systemd in the container will not work, so the entry process is changed to bash.
By changing entry process to bash, the container will not shutdown, and can only be killed.
We accept Karma donations for the last flan. 🍮 affbrr
@Not_Oles said:
ip: RTNETLINK answers: Not supported
modprobe: FATAL: Module ip_tables not found in directory /lib/modules/6.1.1-0-lts
If you installed the kernel from packages, you need to modprobe some modules, including but not limited to: bridge, ip_tables.
If you have upgraded the kernel but has not rebooted, now is the time to do so, otherwise modprobe will not work because the modules directory for the running kernel is already deleted.
If you compiled the kernel from source, you need to re-compile with some options enabled, including but not limited to: CONFIG_BRIDGE, CONFIG_NETFILTER.
We accept Karma donations for the last flan. 🍮 affbrr
@Not_Oles said:
ip: RTNETLINK answers: Not supported
modprobe: FATAL: Module ip_tables not found in directory /lib/modules/6.1.1-0-lts
If you installed the kernel from packages, you need to modprobe some modules, including but not limited to: bridge, ip_tables.
If you have upgraded the kernel but has not rebooted, now is the time to do so, otherwise modprobe will not work because the modules directory for the running kernel is already deleted.
If you compiled the kernel from source, you need to re-compile with some options enabled, including but not limited to: CONFIG_BRIDGE, CONFIG_NETFILTER.
@yoursunny Thanks very much for your helpful comment! Now that there have been updates and upgrades and a reboot, it's time to take a look at the errors related to the kernel. 👀 More before too long. . . . Thanks again!
@yoursunny Seem to be no errors upon repeating ./lxc-up.sh. No further changes beyond updating, upgrading, and rebooting.
Please check whether you now have everything you need. If there is anything more, please tell me.
Other guys listed in /etc/subuid and /etc/subgid, please also check. If you need anything, please tell me by posting here in the thread.
Thank you all so much!
fmt:~# date -u
Mon Jan 23 22:44:33 UTC 2023
fmt:~# # Retry @yoursunny lxc-up commands after updates and reboot
fmt:~# cat lxc-up.sh
ip link add lxcbr0 type bridge
ip addr add 192.168.188.1/24 dev lxcbr0
ip addr add [$REDACTED] dev lxcbr0
sysctl net.ipv4.ip_forward=1
sysctl net.ipv6.conf.all.forwarding=1
ip addr add 192.168.5.188/24 dev eth1 preferred_lft 0
iptables -t nat -A POSTROUTING --src 192.168.188.0/24 -o eth1 -j SNAT --to 192.168.5.188
fmt:~# ./lxc-up.sh
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
fmt:~# exit
Connection to fmt.metalvps.com closed.
Hi @Not_Oles ,
Is the server ready for running a LXC container now?
I`m fail to start a LXC container.
Maybe I do it in a wrong way.
~/pri/pf/lxc_test $ cat $HOME/.config/lxc/default.conf
lxc.include = /etc/lxc/default.conf
lxc.idmap = u 0 1022000000 65536
lxc.idmap = g 0 1022000000 65536
~/pri/pf/lxc_test $ CONFIG=$HOME/.config/lxc/default.conf lxc-checkconfig
LXC version 5.0.2
WARNING: Unable to detect version from configuration, assuming latest
--- Namespaces ---
Namespaces: required
Utsname namespace: missing
Ipc namespace: required
Pid namespace: required
User namespace: missing
Network namespace: missing
--- Control groups ---
Cgroups: missing
Cgroup namespace: enabled
Cgroup v1 mount points:
- /sys/fs/cgroup/openrc
- /sys/fs/cgroup/cpuset
- /sys/fs/cgroup/cpu
- /sys/fs/cgroup/cpuacct
- /sys/fs/cgroup/blkio
- /sys/fs/cgroup/memory
- /sys/fs/cgroup/devices
- /sys/fs/cgroup/freezer
- /sys/fs/cgroup/net_cls
- /sys/fs/cgroup/perf_event
- /sys/fs/cgroup/net_prio
- /sys/fs/cgroup/hugetlb
- /sys/fs/cgroup/pids
Cgroup v2 mount points:
- /sys/fs/cgroup/unified
Cgroup v1 systemd controller: missing
Cgroup v1 clone_children flag: enabled
Cgroup device: missing
Cgroup sched: missing
Cgroup cpu account: missing
Cgroup memory controller: missing
--- Misc ---
Veth pair device: missing
Macvlan: missing
Vlan: missing
Bridges: missing
Advanced netfilter: missing
CONFIG_IP_NF_TARGET_MASQUERADE: missing
CONFIG_IP6_NF_TARGET_MASQUERADE: missing
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: missing
CONFIG_NETFILTER_XT_MATCH_COMMENT: missing
FUSE (for use with lxcfs): missing
--- Checkpoint/Restore ---
checkpoint restore: missing
CONFIG_FHANDLE: missing
CONFIG_EVENTFD: missing
CONFIG_EPOLL: missing
CONFIG_UNIX_DIAG: missing
CONFIG_INET_DIAG: missing
CONFIG_PACKET_DIAG: missing
CONFIG_NETLINK_DIAG: missing
File capabilities: enabled
Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig
~/pri/pf/lxc_test $ lxc-create -n guest1 -f $HOME/.config/lxc/default.conf -t download
Downloading the image index
---
DIST RELEASE ARCH VARIANT BUILD
---
almalinux 8 amd64 default 20230123_23:10
...
ubuntu focal arm64 default 20230123_07:46
...
---
Distribution:
ubuntu
Release:
focal
Architecture:
amd64
Downloading the image index
Downloading the rootfs
Downloading the metadata
The image cache is now ready
Unpacking the rootfs
---
You just created an Ubuntu focal amd64 (20230123_07:42) container.
To enable SSH, run: apt install openssh-server
No default root or user password are set by LXC.
~/pri/pf/lxc_test $ lxc-start -n guest1 -f $HOME/.config/lxc/default.conf
lxc-start: guest1: ../src/lxc/lxccontainer.c: wait_on_daemonized_start: 878 Received container state "ABORTING" instead of "RUNNING"
lxc-start: guest1: ../src/lxc/tools/lxc_start.c: main: 306 The container failed to start
lxc-start: guest1: ../src/lxc/tools/lxc_start.c: main: 309 To get more details, run the container in foreground mode
lxc-start: guest1: ../src/lxc/tools/lxc_start.c: main: 311 Additional information can be obtained by setting the --logfile and --logpriority options
@Not_Oles said:
ip addr add 192.168.188.1/24 dev lxcbr0
iptables -t nat -A POSTROUTING --src 192.168.188.0/24 -o eth1 -j SNAT --to 192.168.5.188
The 192.168.5.188 is a sample value in my test box.
It's meant to be a public IPv4 address routed to the host machine, that is used for outgoing NAT traffic.
IPv6 assignment is wrong too.
Currently eth1 has 2602:fba1:999::2/48 and lxcbr0 has 2602:fba1:999::/48, so that the kernel thinks both interfaces are on the same subnet and would not forward traffic between them.
You should reduce the subnet size on each, so that they don't overlap.
For example, change eth1 to 2602:fba1:999::2/64 and change lxcbr0 to 2602:fba1:999:1c00::/56.
@yoursunny
I run the start command with -F,it seems to be a file permission issue.
The owner of the rootfs is 10220000 which is different from 1022000000.
Is the number 1022000000 out of range to be used as suid or guid?
~/pri/pf/lxc_test $ lxc-start -n guest1 -f $HOME/.config/lxc/default.conf -F -P $(pwd)/container
lxc-start: guest1: ../src/lxc/conf.c: lxc_storage_prepare: 496 Operation not permitted - Failed to recursively turn root mount tree into dependent mount
lxc-start: guest1: ../src/lxc/conf.c: lxc_rootfs_init: 542 Invalid argument - Failed to prepare rootfs storage
lxc-start: guest1: ../src/lxc/start.c: __lxc_start: 2079 Failed to handle rootfs pinning for container "guest1"
lxc-start: guest1: ../src/lxc/tools/lxc_start.c: main: 306 The container failed to start
lxc-start: guest1: ../src/lxc/tools/lxc_start.c: main: 311 Additional information can be obtained by setting the --logfile and --logpriority options
~/pri/pf/lxc_test $ ls container/guest1/ -al
total 16
drwxrwx--- 3 10220000 subenhon 4096 Jan 24 11:28 .
drwxr-xr-x 3 subenhon subenhon 4096 Jan 24 11:27 ..
-rw-r----- 1 subenhon subenhon 752 Jan 24 11:28 config
drwxr-xr-x 17 10220000 10220000 4096 Jan 23 07:47 rootfs
~/pri/pf/lxc_test $ cat /etc/subuid
root:100000:65536
notoles:1000000000:65536
localhost:1002000000:65536
Fritz:1005000000:65536
yoursunny:1018000000:65536
subenhon:1022000000:65536
From simple free Alpine shell accounts to the mind-boggling complexities of networking with LXC containers
"A single swap file or partition may be up to 128 MB in size. [...] [I]f you need 256 MB of swap, you can create two 128-MB swap partitions." (M. Welsh & L. Kaufman, Running Linux, 2e, 1996, p. 49)
I make a mistake.I should remove "-f $HOME/.config/lxc/default.conf" from the command
~/pri/pf/lxc_test $ lxc-start -n guest1 -F -P $(pwd)/container
Failed to look up module alias 'autofs4': Function not implemented
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!!!!] Failed to mount API filesystems.
Exiting PID 1...
The container guest1(ubuntu) fail to start up ,but Devuan container works fine !
The network seems not ready yet !
Going to find out how to make it work.
~/pri/pf/lxc_test $ lxc-start -n Devuan -P $(pwd)/container
~/pri/pf/lxc_test $ lxc-attach -n Devuan -P $(pwd)/container
root@Devuan:/# uname -a
Linux Devuan 6.1.7-0-lts #1-Alpine SMP PREEMPT_DYNAMIC Thu, 19 Jan 2023 08:05:57 +0000 x86_64 GNU/Linux
root@Devuan:/# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0@if23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:5e:fe:c4 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::216:3eff:fe5e:fec4/64 scope link
valid_lft forever preferred_lft forever
@yoursunny said: @Not_Oles said:
ip addr add 192.168.188.1/24 dev lxcbr0
iptables -t nat -A POSTROUTING --src 192.168.188.0/24 -o eth1 -j SNAT --to 192.168.5.188
The 192.168.5.188 is a sample value in my test box.
It's meant to be a public IPv4 address routed to the host machine, that is used for outgoing NAT traffic.
IPv6 assignment is wrong too.
Currently eth1 has 2602:fba1:999::2/48 and lxcbr0 has 2602:fba1:999::/48, so that the kernel thinks both interfaces are on the same subnet and would not forward traffic between them.
You should reduce the subnet size on each, so that they don't overlap.
For example, change eth1 to 2602:fba1:999::2/64 and change lxcbr0 to 2602:fba1:999:1c00::/56.
Hi @yoursunny! Thanks for your comments and corrections! Please let me try again on the configuration. I haven't touched the server yet. If you say the newly revised configuration looks right, then I will reboot the server and run the revised lxc-up.sh. Other guys also welcome to comment.
Let's start revising with the current eth1 and lxcbr0 IPv4 address assignments in mind:
fmt:~# ip -4 addr show eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
inet 23.134.88.226/28 scope global eth1
valid_lft forever preferred_lft forever
inet 192.168.5.188/24 scope global deprecated eth1
valid_lft forever preferred_lft forever
fmt:~# ip -4 addr show dev lxcbr0
6: lxcbr0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
inet 192.168.188.1/24 scope global lxcbr0
valid_lft forever preferred_lft forever
fmt:~#
And also starting with the current lxc-up.sh:
fmt:~# cat lxc-up.sh
ip link add lxcbr0 type bridge
ip addr add 192.168.188.1/24 dev lxcbr0
ip addr add 2602:fba1:999::/48 dev lxcbr0
sysctl net.ipv4.ip_forward=1
sysctl net.ipv6.conf.all.forwarding=1
ip addr add 192.168.5.188/24 dev eth1 preferred_lft 0
iptables -t nat -A POSTROUTING --src 192.168.188.0/24 -o eth1 -j SNAT --to 192.168.5.188
fmt:~#
IPv4 Changes
Change 192.168.5.188 to "a public IPv4 address routed to the host machine"
fmt:~# cat lxc-up.sh
ip link add lxcbr0 type bridge
~~ip addr add 192.168.188.1/24 dev lxcbr0~~ # Will be removed in reboot prior to rerunning lxc-up.sh or could be removed manually with `ip addr del`
ip addr add 23.134.88.227/28 dev lxcbr0 # Add a public IPv4 address routed (?) to host machine
ip addr add 2602:fba1:999::/48 dev lxcbr0
sysctl net.ipv4.ip_forward=1
sysctl net.ipv6.conf.all.forwarding=1
~~ip addr add 192.168.5.188/24 dev eth1 preferred_lft 0~~
ip addr add 23.134.88.227/28 dev eth1 preferred_lft 0
~~iptables -t nat -A POSTROUTING --src 192.168.188.0/24 -o eth1 -j SNAT --to 192.168.5.188~~
iptables -t nat -A POSTROUTING --src 192.168.188.0/24 -o eth1 -j SNAT --to 23.134.88.227
fmt:~#
IPv6 Changes
Current IPv6 address assignments:
fmt:~# ip -6 addr show eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
inet6 2602:fba1:999::2/48 scope global
valid_lft forever preferred_lft forever
inet6 fe80::21e:67ff:fed4:dbbc/64 scope link
valid_lft forever preferred_lft forever
fmt:~# ip -6 addr show dev lxcbr0
6: lxcbr0: <BROADCAST,MULTICAST> mtu 1500 state DOWN qlen 1000
inet6 2602:fba1:999::/48 scope global tentative
valid_lft forever preferred_lft forever
fmt:~#
Change eth1 to 2602:fba1:999::2/64 and change lxcbr0 to 2602:fba1:999:1c00::/56.
fmt:~# cat lxc-up.sh
ip link add lxcbr0 type bridge
~~ip addr add 192.168.188.1/24 dev lxcbr0~~ # Will be removed in reboot prior to rerunning lxc-up.sh or could be removed manually with `ip addr del`
ip addr add 23.134.88.227/28 dev lxcbr0 # Add a public IPv4 address routed (?) to host machine
~~ip addr add 2602:fba1:999::/48 dev lxcbr0~~
ip addr add 2602:fba1:999:1c00::/56 dev lxcbr0 # Are this and the following 2 ip addr commands in the right order?
ip addr add 2602:fba1:999::2/64 dev eth1
ip addr del 2602:fba1:999::2/48 dev eth1 # Needs manual removal unless /etc/network/interfaces is changed.
sysctl net.ipv4.ip_forward=1
sysctl net.ipv6.conf.all.forwarding=1
~~ip addr add 192.168.5.188/24 dev eth1 preferred_lft 0~~
ip addr add 23.134.88.227/28 dev eth1 preferred_lft 0
~~iptables -t nat -A POSTROUTING --src 192.168.188.0/24 -o eth1 -j SNAT --to 192.168.5.188~~
iptables -t nat -A POSTROUTING --src 192.168.188.0/24 -o eth1 -j SNAT --to 23.134.88.227
fmt:~#
Proposed change
So lxc-up.sh would become:
fmt:~# cat lxc-up.sh
ip link add lxcbr0 type bridge
ip addr add 23.134.88.227/28 dev lxcbr0
ip addr add 2602:fba1:999:1c00::/56 dev lxcbr0
ip addr add 2602:fba1:999::2/64 dev eth1
ip addr del 2602:fba1:999::2/48 dev eth1
sysctl net.ipv4.ip_forward=1
sysctl net.ipv6.conf.all.forwarding=1
ip addr add 23.134.88.227/28 dev eth1 preferred_lft 0
iptables -t nat -A POSTROUTING --src 192.168.188.0/24 -o eth1 -j SNAT --to 23.134.88.227
fmt:~#
You may need ip link set lxcbr0 up, just after this line.
ip addr add 2602:fba1:999::2/64 dev eth1
ip addr del 2602:fba1:999::2/48 dev eth1
These two lines must be swapped.
Otherwise you get RTNETLINK answers: File exists because the same address 2602:fba1:999::2 already exists on the netif even if the prefix length differs.
We accept Karma donations for the last flan. 🍮 affbrr
You may need ip link set lxcbr0 up, just after this line.
ip addr add 2602:fba1:999::2/64 dev eth1
ip addr del 2602:fba1:999::2/48 dev eth1
These two lines must be swapped.
Otherwise you get RTNETLINK answers: File exists because the same address 2602:fba1:999::2 already exists on the netif even if the prefix length differs.
Thanks @yoursunny! I put the new lxc-up.sh on the server. Right now @subenhon is running lots of stuff, so I wanna let him know the reboot is coming. Except that I'm going to sleep soon, so, if @subenhon isn't around, maybe the reboot and test will wait until the morning.
chronos@penguin:~$ ssh f
Welcome to Alpine!
fmt:~# date -u
Wed Jan 25 05:02:06 UTC 2023
fmt:~# reboot
fmt:~# Connection to fmt.metalvps.com closed by remote host.
Connection to fmt.metalvps.com closed.
chronos@penguin:~
@yoursunny After running all the commands except ip link set lxcbr0 up the ip link show command says lxcbr0 is DOWN. After running the ip link set lxcbr0 up lxcbr0 is UNKNOWN. Also 227 address on eth1 is "deprecated." Thanks!
Wed Jan 25 05:06:35 UTC 2023
fmt:~# # Test of revised lxc-up.sh
fmt:~# # After reboot, before running any commands from lxc-up.sh
fmt:~# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 00:1e:67:d4:db:bb brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:1e:67:d4:db:bc brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 00:1e:67:d4:db:bd brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 00:1e:67:d4:db:be brd ff:ff:ff:ff:ff:ff
fmt:~# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 00:1e:67:d4:db:bb brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:1e:67:d4:db:bc brd ff:ff:ff:ff:ff:ff
inet 23.134.88.226/28 scope global eth1
valid_lft forever preferred_lft forever
inet6 2602:fba1:999::2/48 scope global
valid_lft forever preferred_lft forever
inet6 fe80::21e:67ff:fed4:dbbc/64 scope link
valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 00:1e:67:d4:db:bd brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 00:1e:67:d4:db:be brd ff:ff:ff:ff:ff:ff
fmt:~# ip route show
default via 23.134.88.225 dev eth1 metric 1 onlink
23.134.88.224/28 dev eth1 scope link src 23.134.88.226
fmt:~# ip -6 route show
2602:fba1:999::/48 dev eth1 metric 256
fe80::/64 dev eth1 metric 256
default via 2602:fba1:999::1 dev eth1 metric 1 onlink
multicast ff00::/8 dev eth1 metric 256
fmt:~# cat -n lxc-up.sh
1 ip link add lxcbr0 type bridge
2 # ip link set lxcbr0 up # Perhaps not needed
3 ip addr add 23.134.88.227/28 dev lxcbr0
4 ip addr add 2602:fba1:999:1c00::/56 dev lxcbr0
5 ip addr del 2602:fba1:999::2/48 dev eth1
6 ip addr add 2602:fba1:999::2/64 dev eth1
7 sysctl net.ipv4.ip_forward=1
8 sysctl net.ipv6.conf.all.forwarding=1
9 ip addr add 23.134.88.227/28 dev eth1 preferred_lft 0
10 iptables -t nat -A POSTROUTING --src 192.168.188.0/24 -o eth1 -j SNAT --to 23.134.88.227
fmt:~# ip link add lxcbr0 type bridge
fmt:~# ip addr add 23.134.88.227/28 dev lxcbr0
fmt:~# ip addr add 2602:fba1:999:1c00::/56 dev lxcbr0
fmt:~# ip addr del 2602:fba1:999::2/48 dev eth1
fmt:~# ip addr add 2602:fba1:999::2/64 dev eth1
fmt:~# sysctl net.ipv4.ip_forward=1
net.ipv4.ip_forward = 1
fmt:~# sysctl net.ipv6.conf.all.forwarding=1
net.ipv6.conf.all.forwarding = 1
fmt:~# ip addr add 23.134.88.227/28 dev eth1 preferred_lft 0
fmt:~# iptables -t nat -A POSTROUTING --src 192.168.188.0/24 -o eth1 -j SNAT --to 23.134.88.227
fmt:~# # After
fmt:~# ip link show
[ . . . ]
6: lxcbr0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether e2:0b:8d:5a:0c:d6 brd ff:ff:ff:ff:ff:ff
fmt:~# # Need to set lxcbr0 up
fmt:~# ip link set lxcbr0 up
fmt:~# ip link show
[ . . . ]
6: lxcbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
link/ether e2:0b:8d:5a:0c:d6 brd ff:ff:ff:ff:ff:ff
fmt:~# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
[ . . . ]
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:1e:67:d4:db:bc brd ff:ff:ff:ff:ff:ff
inet 23.134.88.226/28 scope global eth1
valid_lft forever preferred_lft forever
inet 23.134.88.227/28 scope global secondary deprecated eth1
valid_lft forever preferred_lft forever
inet6 2602:fba1:999::2/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::21e:67ff:fed4:dbbc/64 scope link
valid_lft forever preferred_lft forever
[ . . . ]
6: lxcbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
link/ether e2:0b:8d:5a:0c:d6 brd ff:ff:ff:ff:ff:ff
inet 23.134.88.227/28 scope global lxcbr0
valid_lft forever preferred_lft forever
inet6 2602:fba1:999:1c00::/56 scope global
valid_lft forever preferred_lft forever
inet6 fe80::e00b:8dff:fe5a:cd6/64 scope link
valid_lft forever preferred_lft forever
fmt:~# ip route show
default via 23.134.88.225 dev eth1 metric 1 onlink
23.134.88.224/28 dev eth1 scope link src 23.134.88.226
23.134.88.224/28 dev lxcbr0 scope link src 23.134.88.227
fmt:~# ip -6 route show
2602:fba1:999::/64 dev eth1 metric 256
2602:fba1:999:1c00::/56 dev lxcbr0 metric 256
fe80::/64 dev eth1 metric 256
fe80::/64 dev lxcbr0 metric 256
default via 2602:fba1:999::1 dev eth1 metric 1 onlink
anycast 2602:fba1:999:: dev eth1 metric 0
anycast fe80:: dev eth1 metric 0
anycast fe80:: dev lxcbr0 metric 0
multicast ff00::/8 dev eth1 metric 256
multicast ff00::/8 dev lxcbr0 metric 256
fmt:~#
ip addr add 23.134.88.227/28 dev lxcbr0
ip addr add 23.134.88.227/28 dev eth1 preferred_lft 0
You accidentally assigned the same IPv4 address on both netifs.
lxcbr0 is meant to have an RFC1918 address such as 192.168.188.1.
IPv6 assignments and routing are correct now.
However, the server doesn't seem to have routed IPv6, but it is on-link IPv6.
Hence, you would need to have ndpresponder listen on eth1 and respond to the IPv6 subnet of lxcbr0.
We accept Karma donations for the last flan. 🍮 affbrr
@yoursunny said: You accidentally assigned the same IPv4 address on both netifs.
lxcbr0 is meant to have an RFC1918 address such as 192.168.188.1.
fmt:~# cat lxc-up.sh
ip link add lxcbr0 type bridge
# ip link set lxcbr0 up
ip addr add 23.134.88.227/28 dev lxcbr0
ip addr add 2602:fba1:999:1c00::/56 dev lxcbr0
ip addr del 2602:fba1:999::2/48 dev eth1
ip addr add 2602:fba1:999::2/64 dev eth1
sysctl net.ipv4.ip_forward=1
sysctl net.ipv6.conf.all.forwarding=1
ip addr add 23.134.88.227/28 dev eth1 preferred_lft 0
iptables -t nat -A POSTROUTING --src 192.168.188.0/24 -o eth1 -j SNAT --to 23.134.88.227
fmt:~# cp -p lxc-up.sh lxc-up.sh~
fmt:~# sed -i 's/23.134.88.227\/28 dev lxcbr0/192.168.188.1\/24 dev lxcbr0/' lxc-up.sh
fmt:~# sed -i 's/\# //' lxc-up.sh
fmt:~# cat lxc-up.sh
ip link add lxcbr0 type bridge
ip link set lxcbr0 up
ip addr add 192.168.188.1/24 dev lxcbr0
ip addr add 2602:fba1:999:1c00::/56 dev lxcbr0
ip addr del 2602:fba1:999::2/48 dev eth1
ip addr add 2602:fba1:999::2/64 dev eth1
sysctl net.ipv4.ip_forward=1
sysctl net.ipv6.conf.all.forwarding=1
ip addr add 23.134.88.227/28 dev eth1 preferred_lft 0
iptables -t nat -A POSTROUTING --src 192.168.188.0/24 -o eth1 -j SNAT --to 23.134.88.227
fmt:~#
I am going to reboot the server and run ipv6-up.sh just to see if anything interesting seems to happen.
@yoursunny said: IPv6 assignments and routing are correct now.
However, the server doesn't seem to have routed IPv6, but it is on-link IPv6.
Hence, you would need to have ndpresponder listen on eth1 and respond to the IPv6 subnet of lxcbr0.
From inside the server how does one easily determine whether the IPv4 and IPv6 allocations are on-link or routed?
For ndpresponder, I remember installing on fmt Alpine some glibc compatability layer. Perhaps we might grab the ndpresponder binary that was compiled on the i9 Proxmox server and use it on Alpine via the glibc compatability layer. Alternatively, we perhaps could, on fmt Alpine, install go and again compile ndpresponder from sources against musl libc. Which route is grassy and in want of wear?
As always, thanks very much @yoursunny and other guys on the server.
NAT IPv4 is fully working.
IPv6 is not working due to missing NDP responder.
@Not_Oles said:
For ndpresponder, I remember installing on fmt Alpine some glibc compatability layer. Perhaps we might grab the ndpresponder binary that was compiled on the i9 Proxmox server and use it on Alpine via the glibc compatability layer. Alternatively, we perhaps could, on fmt Alpine, install go and again compile ndpresponder from sources against musl libc. Which route is grassy and in want of wear?
You should recompile ndpresponder binary in a container that runs the same OS as the host, or on the host itself. fmt already has gcc and go compilers, but it lacks linux-headers package that supplies the kernel headers used by GoPacket library.
We accept Karma donations for the last flan. 🍮 affbrr
fmt:~# uname -r
6.1.7-0-lts
fmt:~# who
root pts/0 00:00 Jan 27 00:30:58 187.189.238.1 # Nobody here but me!
fmt:~# date -u
Fri Jan 27 00:42:04 UTC 2023
fmt:~# reboot
fmt:~# Connection to fmt.metalvps.com closed by remote host.
Connection to fmt.metalvps.com closed.
chronos@penguin:~$
New fmt accounts are on pause because of router and network configuration issues. However, considering that you are one of the most important people in the ♒︎ Free VPS Movement ♒︎ how can I say no?
It should work on both IPv4 and IPv6 with your ssh key. Password login has been disabled, but your account password is in a file in your home directory. You might need the password if/when we give you sudo. Please feel free to change your password.
Please let us know what you plan to do with your shell account. Please let us know whether you get in.
New fmt accounts are on pause because of router and network configuration issues. However, considering that you are one of the most important people in the ♒︎ Free VPS Movement ♒︎ how can I say no?
It should work on both IPv4 and IPv6 with your ssh key. Password login has been disabled, but your account password is in a file in your home directory. You might need the password if/when we give you sudo. Please feel free to change your password.
Please let us know what you plan to do with your shell account. Please let us know whether you get in.
Again, welcome aboard!
Tom
I'll get back home and I'll definitely try it out! Thanks!
Maybe we should extend our project to LES too if administrators approve our offer?
We'll discuss this later tonight/ early morning for you.
I forgot to run lxc-up.sh when I last rebooted. So, I ran it. I also added it to /etc/local.d.
fmt:/etc/local.d# cat README
This directory should contain programs or scripts which are to be run
when the local service is started or stopped.
If a file in this directory is executable and it has a .start extension,
it will be run when the local service is started. If a file is
executable and it has a .stop extension, it will be run when the local
service is stopped.
All files are processed in lexical order.
Keep in mind that files in this directory are processed sequentially,
and the local service is not considered started or stopped until
everything is processed, so if you have a process which takes a long
time to run, it can delay your boot or shutdown processing.
fmt:/etc/local.d# cp -p /root/lxc-up.sh lxc.start
fmt:/etc/local.d# ls -l
total 8
-rw-r--r-- 1 root root 652 Jan 7 07:34 README
-rwx------ 1 root root 436 Jan 25 20:55 lxc.start
fmt:/etc/local.d#
If it's a good idea, I probably could add a line something like
@yoursunny@Not_Oles
NAT IPv4 and IPv6 is working .
Now, I tring to run docker in LXC alpine container,but some error occurs.
Is it able to run docker in the unprivileged LXC container?
alpine:~# docker run --rm hello-world
time="2023-02-03T01:46:28.788841772Z" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/b14c8946789ce7eef82508628d5af341b8e5176e3cfe44904053a8639f20c988 pid=615
INFO[2023-02-03T01:46:28.800773054Z] shim disconnected id=b14c8946789ce7eef82508628d5af341b8e5176e3cfe44904053a8639f20c988
ERRO[2023-02-03T01:46:28.800847291Z] copy shim log error="read /proc/self/fd/14: file already closed"
ERRO[2023-02-03T01:46:28.801334879Z] stream copy error: reading from a closed fifo
ERRO[2023-02-03T01:46:28.801334871Z] stream copy error: reading from a closed fifo
ERRO[2023-02-03T01:46:28.892741764Z] b14c8946789ce7eef82508628d5af341b8e5176e3cfe44904053a8639f20c988 cleanup: failed to delete container from containerd: no such container
ERRO[2023-02-03T01:46:28.898990888Z] Handler for POST /v1.41/containers/b14c8946789ce7eef82508628d5af341b8e5176e3cfe44904053a8639f20c988/start returned error: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: no cgroup mount found in mountinfo: unknown
docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: no cgroup mount found in mountinfo: unknown.
Thanks for your report! Congrats on getting your container working! I don't know about the Docker error you mentioned, but I think @yoursunny uses Docker inside containers, so I bet he can tell us about the error.
When you have time, could you please post the commands you used to configure, create, and start your Alpine LXC container?
@Not_Oles ,Here is how I create the Lxc container.
~ $ cat $HOME/.config/lxc/default.conf
lxc.include = /etc/lxc/default.conf
lxc.idmap = u 0 1022000000 65536
lxc.idmap = g 0 1022000000 65536
~ $ lxc-create -n alpine -f $HOME/.config/lxc/default.conf -t download -- --dist alpine --release 3.17 --arch amd64
Using image from local cache
Unpacking the rootfs
---
You just created an Alpinelinux 3.17 x86_64 (20230123_20:03) container.
~ $# change container root password and network setting frome dhcp to manual
~ $ lxc-attach -n alpine
/ # passwd
Changing password for root
New password:
Retype password:
passwd: password for root changed by root
/ # exit
~ $ # container ip config
~ $ echo -e "lxc.net.0.ipv4.address = 192.168.188.58/24\nlxc.net.0.ipv4.gateway = auto" >> $HOME/.local/share/lxc/alpine/config
~ $ echo -e "lxc.net.0.ipv6.address = 2602:fba1:999:1c00:58::/64\nlxc.net.0.ipv6.gateway = auto" >> $HOME/.local/share/lxc/alpine/config
~ $ #restart container
~ $ lxc-stop -n alpine
~ $ lxc-start -n alpine
~ $ lxc-console -n alpine
Connected to tty 1
Type <Ctrl+a q> to exit the console, <Ctrl+a Ctrl+a> to enter Ctrl+a itself
Welcome to Alpine Linux 3.17
Kernel 6.1.8-0-lts on an x86_64 (/dev/tty1)
alpine login: root
Password:
Welcome to Alpine!
The Alpine Wiki contains a large amount of how-to guides and general
information about administrating Alpine systems.
See <https://wiki.alpinelinux.org/>.
You can setup the system with the command: setup-alpine
You may change this message by editing /etc/motd.
alpine:~# #add nameserver
alpine:~# echo -e "nameserver 1.1.1.1\nnameserver 2001:470:20::2\n" >> /etc/resolv.conf
alpine:~# ping -4 google.com
PING google.com (142.251.46.238): 56 data bytes
64 bytes from 142.251.46.238: seq=0 ttl=119 time=1.691 ms
64 bytes from 142.251.46.238: seq=1 ttl=119 time=1.604 ms
--- google.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 1.604/1.647/1.691 ms
alpine:~# ping -6 google.com
PING google.com (2607:f8b0:4005:80b::200e): 56 data bytes
64 bytes from 2607:f8b0:4005:80b::200e: seq=0 ttl=117 time=50.880 ms
64 bytes from 2607:f8b0:4005:80b::200e: seq=1 ttl=117 time=1.773 ms
Hi guys! Sorry, but I just received an email from Hetrix saying that fmt ping monitor is down on IPv4 and on IPv6. Reported to @Cloudie. Will update when I know more. Thanks very much! Tom
Comments
Hi @QuantumBackdoor!
Welcome to LES! Congrats on your first post! I am delighted to be the first to thank you!
Want to share a little about who and where you are, your Linux experience, and what you want to do on the server? I think everyone will be glad to meet you!
Best wishes and kindest regards,
Tom
@Cloudie is very helpful and kind. But he still seems busy. Maybe my message about the network configuration went astray. I took the liberty of bumping that message
Even though there doesn't yet seem to be more news, it's beyond doubt that @Cloudie still is working hard on the router situation.
Best wishes and kindest regards,
Tom
Hi @Not_Oles,
I am Melvik, a student of TJC.
I have been using Windows for a long time since my school, but after I choose development path I find difficult to work in it.
One of my friend recommended linux.I tried it out, and it has an amazing development environment.
I searched for some cheap linux servers, but all were too costly for my usage and I find this forum.
I like to have a linux server to improve my coding skills and linux knowledge. I will use it to learn more in linux and run/test some of my codes in linux environment.
Thanks.
Hi Melvik!
Thanks for your message!
Sorry, does TJC mean "Tyler Junior College?"
Do you have any code or anything else about you online anywhere?
Have you tried any of the free accounts available for students from Google, Oracle, Amazon, Azure, etc?
Thanks again!
Tom
Do google, amazon, oracle have any free student offers?
Almost all the big companies have free offers. Some offers for everyone, and, often, bigger offers for students. Here's a link to Oracle's Free Tier
https://www.oracle.com/cloud/free/
If you look around, you will find many free and introductory offers from the bigger companies.
Best wishes!
Tom
Thanks
Haha thanks for it, am trying to figure out how to use it
Was on vacation, so couldn't respond earlier
lxcbr0 when?
Currently compiling OpenWrt on a 4-core Skylake machine.
It would be 4x faster if it's compiling on an LXC container in fmt.MetalVPS.
We accept Karma donations for the last flan. 🍮 affbrr
TL;DR
Hi @yoursunny!
Thank you for asking! Sorry we haven't been going faster!
We are waiting on two issues, (1) a router sometimes is not working, and (2) we are looking for approval from @Cloudie on the proposed, new /etc/network/interfaces. With (2), the questions include whether exposing the container MAC addresses on the physical network is a problem.
You are more than welcome to compile on one of my other servers. The other guys here also are more than welcome on my other servers.
If you want to take over the network configuration here or provide even more help with how it should be done, then we can go faster on the configuration. But there also are the router issues.
Always best wishes and kindest regards,
Tom
Links
January 3 -- New /etc/network/interfaces proposed
January 4 -- @yoursunny says "Bridging containers directly on the physical port may result in the containers' MAC addresses becoming visible on the physical network. I don't know about Cloudie, but doing this in KVM would get filtered in Virtualizor, and doing this on Hetzner would trigger infraction warning letter." @Not_Oles decides he ought to check with @Cloudie to see whether the proposed /etc/network/interfaces configuration is okay.
January 4 -- Revised /etc/network/interfaces proposed
January 4 -- Revised /etc/network/interfaces sent to @Cloudie
January 6 -- Waiting to hear from @Cloudie about /etc/network/interfaces. @Not_Oles says he might try something.
January 6 -- Multiple users experience downtime.
January 7 -- Multiple downtime/uptime cycles previous night. @Not_Oles says, ". . . when he has time, I'm sure i will hear from @Cloudie on the server /etc/network/interfaces configuration. . . ."
January 7 -- @Not_Oles reports, "Heard from Cloudie. He said he is checking."
January 8 -- @Cloudie says one of the routers is misbehaving. @Not_Oles pauses new accounts until the router issue is resolved.
January 14 -- @Not_Oles says, "@Cloudie is very helpful and kind. But he still seems busy. Maybe my message about the network configuration went astray. I took the liberty of bumping that message."
January 14 -- @Not_Oles says, "Even though there doesn't yet seem to be more news, it's beyond doubt that @Cloudie still is working hard on the router situation."
I believe you can create lxcbr0 not joined with the uplink, so that no container MAC address would show up on the hardware side.
I did some trials on an ephemeral Alpine 3.17 system, created by netboot.xyz and running in tmpfs.
The IP addresses in the sample are:
192.168.5.188 is the host's secondary IPv4, used for outgoing NAT traffic from containers.
192.168.5.188 is marked
preferred_lft 0so that it would not be auto-selected for host originated traffic.192.168.188.0/24 is assigned to containers.
Traffic from this subnet is NAT'ed.
2600:4040:2ca4:a5bc::/64 is assigned to containers.
This prefix must be routed to the server.
If there's only on-link prefix, NDP responder would be required to convert it into a routed prefix.
These are typed by root into Alpine console:
These are typed by user into Alpine console:
These are typed into container console:
Caveats:
All the network configs on the host are ephemeral.
They must be re-typed after the host reboots.
IPv6 into containers is on-link, not routed.
It's possible to add routed IPv6 subnet, by running one
ip route addcommand for each container on the host.IP address conflicts between containers are possible, if two users assign the same address.
If a conflict occurs, the container that starts later will typically show the address as
dadfailed.systemd in the container will not work, so the entry process is changed to bash.
We accept Karma donations for the last flan. 🍮 affbrr
Thanks @yoursunny!
Here are the results of the first try, from last night.
There are a few mistakes I made and maybe a few tweaks still might be needed.
This is fun for me! Thanks again! 💖
Tom
If you installed the kernel from packages, you need to modprobe some modules, including but not limited to: bridge, ip_tables.
If you have upgraded the kernel but has not rebooted, now is the time to do so, otherwise modprobe will not work because the modules directory for the running kernel is already deleted.
If you compiled the kernel from source, you need to re-compile with some options enabled, including but not limited to: CONFIG_BRIDGE, CONFIG_NETFILTER.
We accept Karma donations for the last flan. 🍮 affbrr
Today's upgrades.
Reboot coming very soon! Currently:
Reboot successful.
@yoursunny Thanks very much for your helpful comment! Now that there have been updates and upgrades and a reboot, it's time to take a look at the errors related to the kernel. 👀 More before too long. . . . Thanks again!
@yoursunny Seem to be no errors upon repeating ./lxc-up.sh. No further changes beyond updating, upgrading, and rebooting.
Please check whether you now have everything you need. If there is anything more, please tell me.
Other guys listed in /etc/subuid and /etc/subgid, please also check. If you need anything, please tell me by posting here in the thread.
Thank you all so much!
Hi @Not_Oles ,
Is the server ready for running a LXC container now?
I`m fail to start a LXC container.
Maybe I do it in a wrong way.
The
192.168.5.188is a sample value in my test box.It's meant to be a public IPv4 address routed to the host machine, that is used for outgoing NAT traffic.
IPv6 assignment is wrong too.
Currently eth1 has
2602:fba1:999::2/48and lxcbr0 has2602:fba1:999::/48, so that the kernel thinks both interfaces are on the same subnet and would not forward traffic between them.You should reduce the subnet size on each, so that they don't overlap.
For example, change eth1 to
2602:fba1:999::2/64and change lxcbr0 to2602:fba1:999:1c00::/56.CONFIG environ is meant to be kernel config file, not LXC config file.
Try
CONFIG=/boot/config-lts lxc-checkconfig.Did you follow the prompt?
We accept Karma donations for the last flan. 🍮 affbrr
@yoursunny
I run the start command with -F,it seems to be a file permission issue.
The owner of the rootfs is 10220000 which is different from 1022000000.
Is the number 1022000000 out of range to be used as suid or guid?
From simple free Alpine shell accounts to the mind-boggling complexities of networking with LXC containers
"A single swap file or partition may be up to 128 MB in size. [...] [I]f you need 256 MB of swap, you can create two 128-MB swap partitions." (M. Welsh & L. Kaufman, Running Linux, 2e, 1996, p. 49)
I make a mistake.I should remove "-f $HOME/.config/lxc/default.conf" from the command
The container guest1(ubuntu) fail to start up ,but Devuan container works fine
!
The network seems not ready yet !
Going to find out how to make it work.
Hi @yoursunny! Thanks for your comments and corrections! Please let me try again on the configuration. I haven't touched the server yet. If you say the newly revised configuration looks right, then I will reboot the server and run the revised lxc-up.sh. Other guys also welcome to comment.
Let's start revising with the current eth1 and lxcbr0 IPv4 address assignments in mind:
And also starting with the current lxc-up.sh:
IPv4 Changes
Change 192.168.5.188 to "a public IPv4 address routed to the host machine"
IPv6 Changes
Current IPv6 address assignments:
Change eth1 to 2602:fba1:999::2/64 and change lxcbr0 to 2602:fba1:999:1c00::/56.
Proposed change
So lxc-up.sh would become:
That's right!
Yaaay! That's the true Low End Spirit!
You may need
ip link set lxcbr0 up, just after this line.These two lines must be swapped.
Otherwise you get
RTNETLINK answers: File existsbecause the same address2602:fba1:999::2already exists on the netif even if the prefix length differs.We accept Karma donations for the last flan. 🍮 affbrr
Thanks @yoursunny! I put the new lxc-up.sh on the server. Right now @subenhon is running lots of stuff, so I wanna let him know the reboot is coming. Except that I'm going to sleep soon, so, if @subenhon isn't around, maybe the reboot and test will wait until the morning.
@Not_Oles
Thanks,I had saved my work,It is ok to reboot now
Wow! LES is faster than the wall(1) command, from util-linux, which isn't installed on fmt.
Thanks @subenhon! I'm off to reboot!
@yoursunny After running all the commands except
ip link set lxcbr0 uptheip link showcommand says lxcbr0 is DOWN. After running theip link set lxcbr0 uplxcbr0 is UNKNOWN. Also 227 address on eth1 is "deprecated." Thanks!You accidentally assigned the same IPv4 address on both netifs.
lxcbr0 is meant to have an RFC1918 address such as 192.168.188.1.
IPv6 assignments and routing are correct now.
However, the server doesn't seem to have routed IPv6, but it is on-link IPv6.
Hence, you would need to have ndpresponder listen on eth1 and respond to the IPv6 subnet of lxcbr0.
We accept Karma donations for the last flan. 🍮 affbrr
I am going to reboot the server and run ipv6-up.sh just to see if anything interesting seems to happen.
From inside the server how does one easily determine whether the IPv4 and IPv6 allocations are on-link or routed?
For ndpresponder, I remember installing on fmt Alpine some glibc compatability layer. Perhaps we might grab the ndpresponder binary that was compiled on the i9 Proxmox server and use it on Alpine via the glibc compatability layer. Alternatively, we perhaps could, on fmt Alpine, install go and again compile ndpresponder from sources against musl libc. Which route is grassy and in want of wear?
As always, thanks very much @yoursunny and other guys on the server.
NAT IPv4 is fully working.
IPv6 is not working due to missing NDP responder.
You should recompile
ndpresponderbinary in a container that runs the same OS as the host, or on the host itself.fmt already has gcc and go compilers, but it lacks
linux-headerspackage that supplies the kernel headers used by GoPacket library.We accept Karma donations for the last flan. 🍮 affbrr
More soon. Thanks again @yoursunny!
Compile and install ndpresponder
Might there be some
gopath or environment variable settings which were not made automagically whengowas installed byapk?It's unclear to me whether the current location of the ndpresponder binary is okay.
Also ndpresponder probably needs to be configured.
Friendly greetings!
Location doesn't matter, if you manually invokes it in a SCREEN session.
Everything is on the command line.
-i eth1 -n 2602:fba1:999:1c00::/56We accept Karma donations for the last flan. 🍮 affbrr
Can I get a shell account?
I assume you already have my key, and it looks I was missing all these while I was in LET
Hi @FatGrizzly!
Lovely to see you here!
New fmt accounts are on pause because of router and network configuration issues. However, considering that you are one of the most important people in the ♒︎ Free VPS Movement ♒︎ how can I say no?
Please try something like
ssh [email protected] -p 42365It should work on both IPv4 and IPv6 with your ssh key. Password login has been disabled, but your account password is in a file in your home directory. You might need the password if/when we give you
sudo. Please feel free to change your password.Please let us know what you plan to do with your shell account. Please let us know whether you get in.
Again, welcome aboard!
Tom
I'll get back home and I'll definitely try it out! Thanks!
Maybe we should extend our project to LES too if administrators approve our offer?
We'll discuss this later tonight/ early morning for you.
lxcbr0 bridge disappeared…
We accept Karma donations for the last flan. 🍮 affbrr
I forgot to run lxc-up.sh when I last rebooted.
So, I ran it. I also added it to
/etc/local.d.If it's a good idea, I probably could add a line something like
nohup /root/go/bin/ndpresponder -i eth1 -n 2602:fba1:xxxx:xxxx::/56 &to
lxc.start.@yoursunny @Not_Oles
NAT IPv4 and IPv6 is working .
Now, I tring to run docker in LXC alpine container,but some error occurs.
Is it able to run docker in the unprivileged LXC container?
@subenhon
Thanks for your report! Congrats on getting your container working! I don't know about the Docker error you mentioned, but I think @yoursunny uses Docker inside containers, so I bet he can tell us about the error.
When you have time, could you please post the commands you used to configure, create, and start your Alpine LXC container?
Thanks again!
@Not_Oles ,Here is how I create the Lxc container.
@subenhon Very nice! Thanks so much!
@terrorgen Did you try it too?
Hi guys! Sorry, but I just received an email from Hetrix saying that fmt ping monitor is down on IPv4 and on IPv6. Reported to @Cloudie. Will update when I know more. Thanks very much! Tom