@Not_Oles said:
Hi guys! Sorry, but I just received an email from Hetrix saying that fmt ping monitor is down on IPv4 and on IPv6. Reported to @Cloudie. Will update when I know more. Thanks very much! Tom
Hetrix says its monitors are again receiving ping responses from fmt as of eight minutes ago.
If you did everything the same as @subenhon, are you seeing any errors? It can be helpful for seeing errors to use the "-F" a/k/a "--foreground" option to lxc-start.
I've been busy, so, unfortunately, I haven't tried this myself yet. But I will, eventually.
It seems that the LXC containers might be working on this server! Thanks to @yoursunny for helping a lot to prepare the way! Thanks to @subenhon for blazing the trail!
~ $ cat $HOME/.config/lxc/default.conf
lxc.include = /etc/lxc/default.conf
lxc.idmap = u 0 1022000000 65536
lxc.idmap = g 0 1022000000 65536
~ $ lxc-create -n alpine -f $HOME/.config/lxc/default.conf -t download -- --dist alpine --release 3.17 --arch amd64
Using image from local cache
Unpacking the rootfs
---
You just created an Alpinelinux 3.17 x86_64 (20230123_20:03) container.
~ $# change container root password and network setting frome dhcp to manual
~ $ lxc-attach -n alpine
/ # passwd
Changing password for root
New password:
Retype password:
passwd: password for root changed by root
/ # exit
~ $ # container ip config
~ $ echo -e "lxc.net.0.ipv4.address = 192.168.188.58/24\nlxc.net.0.ipv4.gateway = auto" >> $HOME/.local/share/lxc/alpine/config
~ $ echo -e "lxc.net.0.ipv6.address = 2602:fba1:999:1c00:58::/64\nlxc.net.0.ipv6.gateway = auto" >> $HOME/.local/share/lxc/alpine/config
~ $ #restart container
~ $ lxc-stop -n alpine
~ $ lxc-start -n alpine
~ $ lxc-console -n alpine
Connected to tty 1
Type <Ctrl+a q> to exit the console, <Ctrl+a Ctrl+a> to enter Ctrl+a itself
Welcome to Alpine Linux 3.17
Kernel 6.1.8-0-lts on an x86_64 (/dev/tty1)
alpine login: root
Password:
Welcome to Alpine!
The Alpine Wiki contains a large amount of how-to guides and general
information about administrating Alpine systems.
See <https://wiki.alpinelinux.org/>.
You can setup the system with the command: setup-alpine
You may change this message by editing /etc/motd.
alpine:~# #add nameserver
alpine:~# echo -e "nameserver 1.1.1.1\nnameserver 2001:470:20::2\n" >> /etc/resolv.conf
alpine:~# ping -4 google.com
PING google.com (142.251.46.238): 56 data bytes
64 bytes from 142.251.46.238: seq=0 ttl=119 time=1.691 ms
64 bytes from 142.251.46.238: seq=1 ttl=119 time=1.604 ms
--- google.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 1.604/1.647/1.691 ms
alpine:~# ping -6 google.com
PING google.com (2607:f8b0:4005:80b::200e): 56 data bytes
64 bytes from 2607:f8b0:4005:80b::200e: seq=0 ttl=117 time=50.880 ms
64 bytes from 2607:f8b0:4005:80b::200e: seq=1 ttl=117 time=1.773 ms
There are some small differences between my procedure and @subenhon's procedure. In particular, I don't understand why @subenhon seemed to get working autoconfigured IPv4 out of the box and I perhaps didn't.
Here is my transcript.
# See @subenhon's post at
# https://lowendspirit.com/discussion/comment/127589/#Comment_127589
fmt:~/.config/lxc$ cat default.conf
lxc.include = /etc/lxc/default.conf
lxc.idmap = u 0 1000000000 65536
lxc.idmap = g 0 1000000000 65536
fmt:~/.config/lxc$ cd
fmt:~$ lxc-create -n alpine -f $HOME/.config/lxc/default.conf -t download -- --dist alpin
e --release 3.17 --arch amd64
../src/lxc/cmd/lxc_usernsexec.c: 64: opentty - Permission denied - Failed to open tty
../src/lxc/cmd/lxc_usernsexec.c: 64: opentty - Permission denied - Failed to open tty
../src/lxc/cmd/lxc_usernsexec.c: 64: opentty - Permission denied - Failed to open tty
Downloading the image index
Downloading the rootfs
Downloading the metadata
The image cache is now ready
Unpacking the rootfs
---
You just created an Alpinelinux 3.17 x86_64 (20230220_13:11) container.
fmt:~$ lxc-ls
alpine
fmt:~$ lxc-attach -n alpine
lxc-attach: alpine: ../src/lxc/attach.c: get_attach_context: 406 Connection refused - Failed to get init pid
lxc-attach: alpine: ../src/lxc/attach.c: lxc_attach: 1470 Connection refused - Failed to get attach context
fmt:~$ lxc-start -n alpine
fmt:~$ lxc-attach -n alpine
/ # cd
/bin/ash: cd: can't cd to /home/notoles: No such file or directory
/ # whoami
root
/ # cd root
~ # pwd
/root
~ # passwd
Changing password for root
New password:
Retype password:
passwd: password for root changed by root
~ # echo -e "nameserver 1.1.1.1\nnameserver 2001:470:20::2\n" >> /etc/resolv.conf
~ # ping -4 -c 2 google.com
ping: bad address 'google.com'
~ # exit
fmt:~$ echo -e "lxc.net.0.ipv4.address = 192.168.188.10/24\nlxc.net.0.ipv4.gateway = auto
" >> $HOME/.local/share/lxc/alpine/config
fmt:~$ echo -e "lxc.net.0.ipv6.address = 2602:fba1:999:1c00:10::/64\nlxc.net.0.ipv6.gatew
ay = auto" >> $HOME/.local/share/lxc/alpine/config
fmt:~$ cat -n $HOME/.local/share/lxc/alpine/config
1 # Template used to create this container: /usr/share/lxc/templates/lxc-download
2 # Parameters passed to the template: --dist alpine --release 3.17 --arch amd64
3 # For additional config options, please look at lxc.container.conf(5)
4
5 # Uncomment the following line to support nesting containers:
6 #lxc.include = /usr/share/lxc/config/nesting.conf
7 # (Be aware this has security implications)
8
9
10 # Distribution configuration
11 lxc.include = /usr/share/lxc/config/common.conf
12 lxc.include = /usr/share/lxc/config/userns.conf
13 lxc.arch = linux64
14
15 # Container specific configuration
16 lxc.include = /etc/lxc/default.conf
17 lxc.idmap = u 0 1000000000 65536
18 lxc.idmap = g 0 1000000000 65536
19 lxc.rootfs.path = dir:/home/notoles/.local/share/lxc/alpine/rootfs
20 lxc.uts.name = alpine
21
22 # Network configuration
23 lxc.net.0.ipv4.address = 192.168.188.10/24
24 lxc.net.0.ipv4.gateway = auto
25 lxc.net.0.ipv6.address = 2602:fba1:999:1c00:10::/64
26 lxc.net.0.ipv6.gateway = auto
fmt:~$ lxc-stop -n alpine
fmt:~$ lxc-start -n alpine
fmt:~$ lxc-attach -n alpine
/ # ping -4 -c 2 google.com
PING google.com (142.251.32.46): 56 data bytes
ping: sendto: Network unreachable
/ # ping -6 -c 2 google.com
PING google.com (2607:f8b0:4005:811::200e): 56 data bytes
64 bytes from 2607:f8b0:4005:811::200e: seq=0 ttl=119 time=1.602 ms
64 bytes from 2607:f8b0:4005:811::200e: seq=1 ttl=119 time=1.561 ms
--- google.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 1.561/1.581/1.602 ms
/ # / # ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0@if35: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP qlen 1000
link/ether 00:16:3e:24:60:3e brd ff:ff:ff:ff:ff:ff
/ # ip address show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0@if35: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP qlen 1000
link/ether 00:16:3e:24:60:3e brd ff:ff:ff:ff:ff:ff
inet6 2602:fba1:999:1c00:10::/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::216:3eff:fe24:603e/64 scope link
valid_lft forever preferred_lft forever
/ # ip address add 192.168.188.10/24 dev eth0
/ # ip route add 192.168.188.1 dev eth0
/ # ip route add default via 192.168.188.1 dev eth0
/ # ping -4 -c 2 google.com
PING google.com (142.251.46.238): 56 data bytes
64 bytes from 142.251.46.238: seq=0 ttl=119 time=1.552 ms
64 bytes from 142.251.46.238: seq=1 ttl=119 time=1.534 ms
--- google.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 1.534/1.543/1.552 ms
/ # wget -qO- ipv4.icanhazip.com
23.134.88.227
/ # vi ipv4-up.sh
/ # cat -n ipv4-up.sh
1 ip address add 192.168.188.10/24 dev eth0
2 ip route add 192.168.188.1 dev eth0
3 ip route add default via 192.168.188.1 dev eth0
/ #
Hi @yoursunny! We were talking about setting your container quantity in /etc/lxc/lxc-usernet. Is it time to raise your quantity now? I forget how many you wanted, so can you remind me, please? Thanks!
@Not_Oles said: There are some small differences between my procedure and @subenhon's procedure. In particular, I don't understand why @subenhon seemed to get working autoconfigured IPv4 out of the box and I perhaps didn't.
@Not_Oles I change the network type from dhcp to manual when attach to the container , maybe you missed this part.
~ $# change container root password and network setting from dhcp to manual
~ $ lxc-attach -n alpine
/ # passwd
Changing password for root
New password:
Retype password:
passwd: password for root changed by root
@Not_Oles said:
Hi @yoursunny! We were talking about setting your container quantity in /etc/lxc/lxc-usernet. Is it time to raise your quantity now? I forget how many you wanted, so can you remind me, please? Thanks!
I typically configure "100" in lxc-usernet so it's never a limitation.
There's really no need to place a limit here, because a user can have practically any number of containers by making creating nested containers that isn't subject to this limit.
We accept Karma donations for the last flan. 🍮 affbrr
I'm not sure this is right, because we don't even have installed either the LXD package or the lxc command (which is part of LXD and not part of LXC). We do have the lxc and lxcfs packages installed, but they do not include the lxc command.
fmt:~# apk info lxd
lxd-5.0.2-r3 description:
A container hypervisor and a new user experience for LXC - 'LTS' release channel
lxd-5.0.2-r3 webpage:
https://linuxcontainers.org/lxd/
lxd-5.0.2-r3 installed size:
43 MiB
fmt:~# apk -e info lxd
fmt:~# # No output here means it's not installed.
I tried commenting out bash and adding a line about systemd to my /home/notoles Debian LXC config.
~/.local/share/lxc/debian $ cat config
# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template: --dist debian --release sid --arch amd64
# For additional config options, please look at lxc.container.conf(5)
# Uncomment the following line to support nesting containers:
#lxc.include = /usr/share/lxc/config/nesting.conf
# (Be aware this has security implications)
# Distribution configuration
lxc.include = /usr/share/lxc/config/common.conf
lxc.include = /usr/share/lxc/config/userns.conf
lxc.arch = linux64
# Container specific configuration
lxc.include = /etc/lxc/default.conf
lxc.idmap = u 0 1000000000 65536
lxc.idmap = g 0 1000000000 65536
lxc.rootfs.path = dir:/home/notoles/.local/share/lxc/debian/rootfs
lxc.uts.name = debian
# Network configuration
lxc.net.0.ipv4.address = 192.168.188.11/24
lxc.net.0.ipv4.gateway = 192.168.188.1
lxc.net.0.ipv6.address = 2602:fba1:999:1c00:11::/64
lxc.net.0.ipv6.gateway = 2602:fba1:999:1c00::
# lxc.init.cmd = /bin/bash
systemd_container=yes
~/.local/share/lxc/debian $
The result was these errors:
fmt:~$ lxc-start -F -n debian
Failed to find module 'autofs4'
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!!!!] Failed to mount API filesystems.
Exiting PID 1...
fmt:~$
If there is no way to get Alpine's LXC without LXD to run an unprivileged container for a systemd OS, then, yes, maybe we should install LXD.
I will check to see if starting the unprivileged container with bash and then stopping/disabling systemd-networkd is enough to get an unprivileged Debian container to start in the usual way, without setting PID 1 as bash.
@yoursunny seems pretty sure that systemd won't work inside the LXC containers:
@yoursunny said: systemd in the container will not work, so the entry process is changed to bash.
@terrorgen Did you try changing the entry process to bash for your nix container?
Do you guys understand how/why making unprivileged containers with LXD as explained in https://wiki.alpinelinux.org/wiki/LXD could enable systemd to work inside those containers when getting systemd working might not be possible inside unprivileged containers made with lxc-create? What's the difference between LXD's lxc command and LXC's lxc-create command that allows systemd to work with unprivileged LXD containers but not with unprivileged LXC containers?
@Not_Oles said: I'm not sure this is right, because we don't even have installed either the LXD package or the lxc command (which is part of LXD and not part of LXC). We do have the lxc and lxcfs packages installed, but they do not include the lxc command.
But /etc/conf.d/lxc does exist, so it may not have to do with LXD.
@Not_Oles said: @terrorgen Did you try changing the entry process to bash for your nix container?
That won't work. NixOS is heavily dependant on systemd to get things working. Also, LXC "boots" the container by starting /sbin/init inside the container. In a NixOS container this is actually a generated bash script, which among other things contains the system configuration that will be activated at "boot". My init script is throwing errors because it cannot find a proper sys mount.
@Not_Oles said: What's the difference between LXD's lxc command and LXC's lxc-create
My understanding is, LXD's lxc command is a higher level management tool that abstracts away the lower level configuration of lxc-*. It can also manage VMs. I don't claim to be an expert for either, though.
@Not_Oles said: By the way, where / how did you get your nix LXC image?
Found a guide that downloads the rootfs tarball from NixOS's own repository. NixOS developers decided against publishing them to linuxcontainers.org because it seems counterintuitive.
I personally have a NixOS LXC container running in my homelab Proxmox server, so I know it can be done.
So it just clicked that most of my problems (besides cgroup delegation) is permissions:
❯ lxc-start nixos -F
lxc-start: nixos: ../src/lxc/utils.c: safe_mount: 1220 Resource busy - Failed to mount "sys" onto "/usr/lib/lxc/rootfs/dev/.lxc/sys"
<<< NixOS Stage 2 >>>
install: cannot change permissions of '/tmp': Operation not permitted
running activation script...
ln: failed to create symbolic link '/bin/.sh.tmp': Permission denied
mv: cannot stat '/bin/.sh.tmp': No such file or directory
Activation script snippet 'binsh' failed (1)
install: cannot change permissions of '/root': Operation not permitted
mkdir /var/lib: Permission denied at /nix/store/snb4523ghvw9917q15j401fz26d5plh3-update-users-groups.pl line 17.
Activation script snippet 'users' failed (13)
setting up /etc...
Died at /nix/store/rg5rf512szdxmnj9qal3wfdnpfsx38qi-setup-etc.pl line 27.
Activation script snippet 'etc' failed (13)
/nix/store/8ndxpvlgfjbbas506vqrad69rzjzxwsp-nixos-system-nixos-23.05pre452927.6ccc4a59c3f/activate: line 129: /etc/shadow: No such file or directory
Activation script snippet 'hashes' failed (1)
ln: failed to create symbolic link '/sbin/init': Permission denied
Activation script snippet 'installInitScript' failed (1)
install: cannot create directory '/nix/var': Permission denied
install: cannot create directory '/nix/var': Permission denied
/nix/store/8ndxpvlgfjbbas506vqrad69rzjzxwsp-nixos-system-nixos-23.05pre452927.6ccc4a59c3f/activate: line 167: /root/.nix-channels: Permission denied
Activation script snippet 'nix' failed (1)
mkdir: cannot create directory '/usr/bin': Permission denied
ln: failed to create symbolic link '/usr/bin/.env.tmp': No such file or directory
mv: cannot stat '/usr/bin/.env.tmp': No such file or directory
Activation script snippet 'usrbinenv' failed (1)
mkdir: cannot create directory '/var/tmp': Permission denied
mkdir: cannot create directory '/var/empty': Permission denied
find: '/var/empty': No such file or directory
chmod: cannot access '/var/empty': No such file or directory
chown: invalid user: 'root:root'
Activation script snippet 'var' failed (1)
chown: invalid user: 'root:root'
chown: invalid user: 'root:messagebus'
chown: invalid user: 'root:root'
chown: invalid user: 'root:root'
chown: invalid user: 'root:root'
chown: invalid user: 'root:root'
chown: invalid user: 'root:root'
chown: invalid user: 'root:root'
chown: invalid user: 'root:root'
chown: invalid user: 'root:root'
chown: invalid user: 'root:root'
chown: invalid user: 'root:root'
chown: invalid user: 'root:root'
chown: invalid user: 'root:root'
chown: invalid user: 'root:root'
chown: invalid user: 'root:root'
Activation script snippet 'wrappers' failed (1)
mkdir: cannot create directory '/nix/var': Permission denied
ln: failed to create symbolic link '/nix/var/nix/gcroots/current-system': No such file or directory
cp: cannot create regular file '/etc/nixos/configuration.nix': Permission denied
terminate called after throwing an instance of 'nix::Error'
what(): error: cannot determine user's home directory
/nix/store/l411104qj58cq7f1gg2wiryi0lzly5jk-local-cmds: line 17: 132 Aborted /nix/store/nnznavnhyli08264apz6lanbjza48si1-nix-2.13.2/bin/nix-store --load-db < /nix-path-registration
terminate called after throwing an instance of 'nix::Error'
what(): error: cannot determine user's home directory
/nix/store/l411104qj58cq7f1gg2wiryi0lzly5jk-local-cmds: line 20: 133 Aborted /nix/store/nnznavnhyli08264apz6lanbjza48si1-nix-2.13.2/bin/nix-env -p /nix/var/nix/profiles/system --set /run/current-system
unpacking the NixOS/Nixpkgs sources...
mkdir: cannot create directory '/nix/var': Permission denied
terminate called after throwing an instance of 'nix::Error'
what(): error: cannot determine user's home directory
/nix/store/l411104qj58cq7f1gg2wiryi0lzly5jk-local-cmds: line 32: 135 Aborted /nix/store/nnznavnhyli08264apz6lanbjza48si1-nix-2.13.2/bin/nix-env -p /nix/var/nix/profiles/per-user/root/channels -i /nix/store/70hcm36cm9v6wwvl224w2zvxvshrh1ff-nixos-23.05pre452927.6ccc4a59c3f --quiet --option build-use-substitutes false
mkdir: cannot create directory '/root/.nix-defexpr': Permission denied
ln: failed to create symbolic link '/root/.nix-defexpr/channels': No such file or directory
mkdir: cannot create directory '/var/lib': Permission denied
touch: cannot touch '/var/lib/nixos/did-channel-init': No such file or directory
/sbin/init: line 130: /etc/machine-id: Permission denied
starting systemd...
Failed to find module 'autofs4'
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!!!!] Failed to mount API filesystems.
Exiting PID 1...
Comparing notes between my homelab NixOS container vs MetalVPS's...
rootfs permission in my homelab is set to uid:100000 and gid:100000, which maps to root:root in the container.
Whereas my rootfs' permission in MetalVPS is set to my uid:gid.
so I did a tweak in my config file:
lxc.idmap = u 0 1015 1
lxc.idmap = g 0 1015 1
lxc.idmap = u 1 1015000000 65535
lxc.idmap = g 1 1015000000 65535
so the container root is myself in MetalVPS.
solved most of the permission issues above.
now this is how it looks like:
❯ lxc-start nixos -F
lxc-start: nixos: ../src/lxc/utils.c: safe_mount: 1220 Resource busy - Failed to mount "sys" onto "/usr/lib/lxc/rootfs/dev/.lxc/sys"
<<< NixOS Stage 2 >>>
running activation script...
setting up /etc...
starting systemd...
Failed to find module 'autofs4'
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!!!!] Failed to mount API filesystems.
Exiting PID 1...
@terrorgen said: So it just clicked that most of my problems (besides cgroup delegation) is permissions
I did a tweak in my config file:
lxc.idmap = u 0 1015 1
lxc.idmap = g 0 1015 1
lxc.idmap = u 1 1015000000 65535
lxc.idmap = g 1 1015000000 65535
so the container root is myself in MetalVPS.
Congrats on figuring out the permissions issues!
@terrorgen said: Failed to find module 'autofs4'
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!!!!] Failed to mount API filesystems.
Exiting PID 1...
Looks just like the errors I am seeing when starting a Debian container:
@Not_Oles said:
fmt:~$ lxc-start -F -n debian
Failed to find module 'autofs4'
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!!!!] Failed to mount API filesystems.
Exiting PID 1...
fmt:~$
@terrorgen I will look at this some more, including your suggested changes to the node configuration and raising the number of containers permission limit.
Now that you fixed your user permissions, might starting with bash as PID 1 work? And be helpful?
rc-update add dbus
Reboot and lxd should be working.
fmt:~# date
Thu Feb 23 00:35:42 UTC 2023
fmt:~# cat /etc/conf.d/lxc
# Configuration for /etc/init.d/lxc[.*]
# Enable cgroup for systemd-based containers.
#systemd_container=no
systemd_container=yes
# autostart groups (comma separated)
#lxc_group="onboot"
# Directory for containers' logs (used for symlinked runscripts lxc.*).
#logdir="/var/log/lxc"
fmt:~#
fmt:~# rc-update add lxc
* service lxc added to runlevel default
fmt:~# rc-update add lxd
* rc-update: service `lxd' does not exist
fmt:~# rc-update add lxcfs
* service lxcfs added to runlevel default
fmt:~# rc-update add dbus
* rc-update: service `dbus' does not exist
fmt:~#
Do we need to apk add lxd and apk add dbus plus enable both before rebooting, or is it worth while to try just adding lxc and lxcfs to runlevel default?
Basically you’d need root to crate you a /sys/fs/cgroup/user.doskanoness cgroup or something similar, then chown it over to you and move your shell’s PID into it.
At that point, lxc-start should be able to detect that and since you now own that cgroup, will be able to create its own entries in there for the container.
I'd want to read about making cgroups, but it looks like we could do it. Do you want to go ahead with cgroups, or stay awhile longer on the present path? Should we (1) go ahead and reboot, (2) add lxd and dbus and then reboot, (3) revert the changes I just made to /etc/conf.d/lxc and the rc scripts, (4) work on the cgroups, or (5) some combination?
Failed to create /init.scope control group: Permission denied
Failed to allocate manager object: Permission denied
[!!!!!!] Failed to allocate manager object.
Exiting PID 1...
Solution proposed as:
OK - I figured it out after stepping away from it and trying fresh this morning. I had an issue in my /etc/pam.d/system-login file. I fixed the cgfs line to read like this:
Comments
Hetrix says its monitors are again receiving ping responses from fmt as of eight minutes ago.
Hey Tom, just trying it and looks like you need to add me to /etc/sub{u,g}id
DM us for private tracker invite.
Hi! Okay, added to /etc/sub*id and also to /etc/lxc/lxc-usernet. Please let me know if you need anything else. Have fun!
Thanks! However, I can't start my LXC container. Maybe it has something to do with lxcbr0?
DM us for private tracker invite.
Hi again @terrorgen!
Can you please take another look at how @subenhon got his container started?
If you did everything the same as @subenhon, are you seeing any errors? It can be helpful for seeing errors to use the "-F" a/k/a "--foreground" option to
lxc-start.I've been busy, so, unfortunately, I haven't tried this myself yet. But I will, eventually.
Thanks!
Tom
I did, and it is stored at
~terrorgen/logif you want to take a peek.To save everyone's time, I think it is because of this:
Does the lack of
systemdin Alpine contribute to this? Cause I also saw some TRACE messages related to systemd not being found.DM us for private tracker invite.
There are many interesting entries in those logs!
I forgot to initialize the bridge when I last rebooted. I have to set the bridge so it starts automagically.
Want to try starting your container again, please?
And it's still not working for me to get NixOS run on LXC.
I tried downloading a standard debian and Ubuntu templates but was unsuccessful as well.
Must be the configuration.
DM us for private tracker invite.
It seems that the LXC containers might be working on this server! Thanks to @yoursunny for helping a lot to prepare the way! Thanks to @subenhon for blazing the trail!
There are some small differences between my procedure and @subenhon's procedure. In particular, I don't understand why @subenhon seemed to get working autoconfigured IPv4 out of the box and I perhaps didn't.
Here is my transcript.
Yes, systemd won't start in LXC container due to lack of cgroups.
See my test report:
https://lowendspirit.com/discussion/comment/125271/#Comment_125271
Look at
lxc.init.cmdline under "these are typed by user".We accept Karma donations for the last flan. 🍮 affbrr
Hi @yoursunny! We were talking about setting your container quantity in /etc/lxc/lxc-usernet. Is it time to raise your quantity now? I forget how many you wanted, so can you remind me, please? Thanks!
Sorry to hear. I should try these as well. Do you have any info about what went wrong? Any error messages? Thanks!
@Not_Oles I change the network type from dhcp to manual when attach to the container , maybe you missed this part.
According to https://wiki.alpinelinux.org/wiki/LXC, cgroup can be enabled with
rc-update add cgroupsWorth a try, @Not_Oles?
DM us for private tracker invite.
Hi again! Here are some shenanigans with Debian.
Hi @terrorgen!
Already done, I think. Please take a look at the seond to last entry under Default runlevel.
It looks like I can get a persistent Debian container by using tmux.
Inside the container, enter Ctrl-b followed by d to detach from the container and return to the host while leaving the container running.
On the host, enter "tmux attach" to reattach to the container which is still running inside tmux.
To stop the container from inside the container (which is inside tmux), enter "exit." To close tmux, enter "exit" again.
Looks like bash is PID1 in your container?
DM us for private tracker invite.
Btw, FMT not pingable from my end.
DM us for private tracker invite.
Yup. Thanks for the reminder! Much appreciated!
Yeah, work in the datacenter. Downtime was not expected. But it nevertheless seems to have arrived.
You can check
IPv4
IPv6
Yeah, I think so, as a way to fix certain errors noted above which are, for easy reference, repeated here:
Now the questions are (1) why did the errors happen, and (2) can we find a configuration that prevents them from happening?
I typically configure "100" in lxc-usernet so it's never a limitation.
There's really no need to place a limit here, because a user can have practically any number of containers by making creating nested containers that isn't subject to this limit.
We accept Karma donations for the last flan. 🍮 affbrr
I got similar errors on my NixOS container about autofs and cgroup and PID1 refusing to proceed further. Looks like a systemd issue now.
DM us for private tracker invite.
Oh just found this from https://wiki.alpinelinux.org/wiki/LXD:
If you plan to run systemd based Linux distributions (Debian, Ubuntu, etc.), add this to /etc/conf.d/lxc:
systemd_container=yesand enable both lxc and lxd to start at boot:
If you have problems, try to enable dbus:
rc-update add dbusReboot and lxd should be working.
DM us for private tracker invite.
HI! The server seems to be back up, and services are restarted. If anybody has any issues, please let me know. Thanks!
Inside LXC Container Described Above
On The Bare Metal Server Node
I would also love to try one of the accounts. Thank you!!
Hi @superyuluo!
Welcome to LES! I hope you enjoy your time here!
Since you have a new account, I am sure that everyone here would enjoy meeting you. Would you please introduce yourself? Please tell us
who and where you are,
about your Linux experience, and
what you plan to do on the server.
Also, to give you an account, we need your ed25519 ssh public key. Would you please post your key or a link to it?
I am looking forward to having you join us on the server!
Best!
Tom
Looks like I am still not able to start my container.
@Not_Oles you'll have to do this ☺️
DM us for private tracker invite.
Edited. . . .
@terrorgen Please note 12, 13, and 14.
I'm not sure this is right, because we don't even have installed either the LXD package or the
lxccommand (which is part of LXD and not part of LXC). We do have the lxc and lxcfs packages installed, but they do not include thelxccommand.I tried commenting out bash and adding a line about systemd to my /home/notoles Debian LXC config.
The result was these errors:
If there is no way to get Alpine's LXC without LXD to run an unprivileged container for a systemd OS, then, yes, maybe we should install LXD.
I looked at https://wiki.alpinelinux.org/wiki/LXC. This page suggests stopping and disabling systemd-networkd inside a privileged container:
I will check to see if starting the unprivileged container with bash and then stopping/disabling systemd-networkd is enough to get an unprivileged Debian container to start in the usual way, without setting PID 1 as bash.
By the way, where / how did you get your nix LXC image? Nix doesn't seem to be on the linuxcontainers.org image server.
@yoursunny seems pretty sure that systemd won't work inside the LXC containers:
@terrorgen Did you try changing the entry process to bash for your nix container?
Do you guys understand how/why making unprivileged containers with LXD as explained in https://wiki.alpinelinux.org/wiki/LXD could enable systemd to work inside those containers when getting systemd working might not be possible inside unprivileged containers made with
lxc-create? What's the difference between LXD'slxccommand and LXC'slxc-createcommand that allows systemd to work with unprivileged LXD containers but not with unprivileged LXC containers?But
/etc/conf.d/lxcdoes exist, so it may not have to do with LXD.That won't work. NixOS is heavily dependant on systemd to get things working. Also, LXC "boots" the container by starting
/sbin/initinside the container. In a NixOS container this is actually a generated bash script, which among other things contains the system configuration that will be activated at "boot". My init script is throwing errors because it cannot find a propersysmount.My understanding is, LXD's
lxccommand is a higher level management tool that abstracts away the lower level configuration oflxc-*. It can also manage VMs. I don't claim to be an expert for either, though.Found a guide that downloads the rootfs tarball from NixOS's own repository. NixOS developers decided against publishing them to linuxcontainers.org because it seems counterintuitive.
I personally have a NixOS LXC container running in my homelab Proxmox server, so I know it can be done.
DM us for private tracker invite.
So it just clicked that most of my problems (besides cgroup delegation) is permissions:
Comparing notes between my homelab NixOS container vs MetalVPS's...
rootfs permission in my homelab is set to uid:100000 and gid:100000, which maps to root:root in the container.
Whereas my rootfs' permission in MetalVPS is set to my uid:gid.
so I did a tweak in my
configfile:so the container
rootis myself in MetalVPS.solved most of the permission issues above.
now this is how it looks like:
DM us for private tracker invite.
Congrats on figuring out the permissions issues!
Looks just like the errors I am seeing when starting a Debian container:
@terrorgen I will look at this some more, including your suggested changes to the node configuration and raising the number of containers permission limit.
Now that you fixed your user permissions, might starting with bash as PID 1 work? And be helpful?
it may boot successfully but because systemd is heavily relied upon by NixOS, it won't be any useful.
I am sorry if I am not being helpful.
DM us for private tracker invite.
You are always very helpful!
I meant: "Now that you fixed your user permissions, might starting with bash as PID 1 work? And be helpful to you?"
seems like there is another solution:
https://discuss.linuxcontainers.org/t/failed-to-mount-cgroup-at-sys-fs-cgroup-systemd-operation-not-permitted/13646
DM us for private tracker invite.
Do we need to
apk add lxdandapk add dbusplus enable both before rebooting, or is it worth while to try just adding lxc and lxcfs to runlevel default?Okay, at the link you posted @stgraber says:
I'd want to read about making cgroups, but it looks like we could do it. Do you want to go ahead with cgroups, or stay awhile longer on the present path? Should we (1) go ahead and reboot, (2) add lxd and dbus and then reboot, (3) revert the changes I just made to /etc/conf.d/lxc and the rc scripts, (4) work on the cgroups, or (5) some combination?
Cgroup comparison between fmt (Alpine) and ex100 (Debian sid):
Let's go ahead and reboot and see if it works! If it doesn't after the reboot, we'll try something else
DM us for private tracker invite.
Hi @terrorgen!
Reboot:
Following the reboot:
We got progress!
same error message here.
DM us for private tracker invite.
Welp, for whatever it's worth:
Unpriviliged container wont start - Failed to allocate manager object
Same errors in Arch Linux:
Solution proposed as:
Here is our /etc/pam.d/system-login:
We don't have a pam_cgfs line in our /etc/pam.d/system-login.
I found a Debian libpam-cgfs package and a Github repo for pam_cgfs.c which seems to be part of LXC. Apparently, Alpine might not have the separate package.
Maybe I could try adding the suggested pam_cgfs line, but I don't understand it enough yet. Ideas?
Worth a shot.
DM us for private tracker invite.
I tried adding the suggested line to /etc/pam.d/system-login and then rebooting.
That addition produced these errors, which seem to be the same:
So I reverted the change and rebooted. Restarted the networking and ndpresponder.
I logged in and nobody was around. So. . . .