What do you use to manage & update multiple servers?

terrorgen

Nix works on Macs too! Check out nix-darwin.

AuroraZero

@terrorgen said:
Nix works on Macs too! Check out nix-darwin.

Son of a they got another one

somik

@AuroraZero said:

@terrorgen said:
Nix works on Macs too! Check out nix-darwin.

Son of a they got another one

Well, on my office mac, we are already using "uv" to run python as we need to switch between different envs. So why not give nix a try. If i like it (and the corporate IT overlords allows it) all good. Otherwise, just try something else.

bingobangobongo

@somik said: Now where should I set it up? A raspberry pi?

Definitely in the same space that the old VM you deleted took up on your laptop. It's full circle - she's home again!

cochon

@somik said:

@cochon said:
Many years ago I used to set remote servers to refresh configs via a simple overnight cron job that would pull a PGP signed tarball, verify and unpack it before running the update script it contained.

These days I tend to use ansible, but still set up that cron job ‘just in case’. No one’s infallible or immune from sawing off the branch they’re sat on.

You mean like deleting the only VM that stored all your ssh keys that you used to access all of your other servers and not having any backups?

No I don’t mean that at all, I was thinking more of making an accidental config change (maybe a typo) that locks you out of remote admin completely. I then still have a pull update mechanism to fall back on.

No, that never happened to me. I am not that stupid.

To be honest that’s not a great attitude for a sysadmin. Where I come from we have a saying: ‘pride comes before a fall‘. I also think I’m not that stupid, but I always factor in that one day I might be

somik

@cochon said:

@somik said:

@cochon said:
Many years ago I used to set remote servers to refresh configs via a simple overnight cron job that would pull a PGP signed tarball, verify and unpack it before running the update script it contained.

These days I tend to use ansible, but still set up that cron job ‘just in case’. No one’s infallible or immune from sawing off the branch they’re sat on.

You mean like deleting the only VM that stored all your ssh keys that you used to access all of your other servers and not having any backups?

No I don’t mean that at all, I was thinking more of making an accidental config change (maybe a typo) that locks you out of remote admin completely. I thæn still have a pull update mechanism to fall back on.

No, that never happened to me. I am not that stupid.

To be honest that’s not a great attitude for a sysadmin. Where I come from we have a saying: ‘pride comes before a fall‘. I also think I’m not that stupid, but I always factor in that one day I might be

Ah, I see that went over your head...

Some background info for you.

In 2024, i lost access to all my servers cause the only ssh key was in a VM on my laptop that I deleted when I reformatted my laptop. My post was meant to be a sarcastic reminder to myself for that mistake...

hornet

@GeekWanderer said: Ansible

This is the way.

AuroraZero

I believe it would be a dereliction of my duty if I did not mention Ansible.

somik

Fought with ansible for a while. It wants python module called "six" to be installed on ALL of my remote nodes/vms...

Anyone know what I am doing wrong?

fatal: [dev]: FAILED! => 
{
  "ansible_facts": {},
  "changed": false,
  "failed_modules": {
    "ansible.legacy.setup": {
      "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
      },
      "exception": "Traceback (most recent call last):\r\n File \"/root/.ansible/tmp/ansible-tmp-1762609549.911138-3919-137150221656244/AnsiballZ_setup.py\", line 102, in <module>\r\n _ansiballz_main()\r\n File \"/root/.ansible/tmp/ansible-tmp-1762609549.911138-3919-137150221656244/AnsiballZ_setup.py\", line 94, in _ansiballz_main\r\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\r\n File \"/root/.ansible/tmp/ansible-tmp-1762609549.911138-3919-137150221656244/AnsiballZ_setup.py\", line 37, in invoke_module\r\n from ansible.module_utils import basic\r\n File \"/tmp/ansible_ansible.legacy.setup_payload_vbni3qiv/ansible_ansible.legacy.setup_payload.zip/ansible/module_utils/basic.py\", line 176, in <module>\r\nModuleNotFoundError: No module named 'ansible.module_utils.six.moves'\r\n",
      "failed": true,
      "module_stderr": "Shared connection to 10.10.0.50 closed.\r\n",
      "module_stdout": "Traceback (most recent call last):\r\n File \"/root/.ansible/tmp/ansible-tmp-1762609549.911138-3919-137150221656244/AnsiballZ_setup.py\", line 102, in <module>\r\n _ansiballz_main()\r\n File \"/root/.ansible/tmp/ansible-tmp-1762609549.911138-3919-137150221656244/AnsiballZ_setup.py\", line 94, in _ansiballz_main\r\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\r\n File \"/root/.ansible/tmp/ansible-tmp-1762609549.911138-3919-137150221656244/AnsiballZ_setup.py\", line 37, in invoke_module\r\n from ansible.module_utils import basic\r\n File \"/tmp/ansible_ansible.legacy.setup_payload_vbni3qiv/ansible_ansible.legacy.setup_payload.zip/ansible/module_utils/basic.py\", line 176, in <module>\r\nModuleNotFoundError: No module named 'ansible.module_utils.six.moves'\r\n",
      "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
      "rc": 1
    }
  },
  "msg": "The following modules failed to execute: ansible.legacy.setup\n"
}

My ansible file:

- name: Update VMs
  hosts: vms
  gather_facts: yes
  become: yes
  tasks:
    - name: Ping host
      ping:
      register: ping_result
      ignore_errors: yes

    - name: Update APT cache
      apt:
        update_cache: yes
        cache_valid_time: 3600

    - name: Upgrade packages
      apt:
        upgrade: dist
        autoremove: yes
        autoclean: yes

    - name: Reboot if required
      reboot:
        msg: "Rebooting VM after upgrade"
        pre_reboot_delay: 5
        post_reboot_delay: 10
        test_command: uptime

- name: Wait for VMs to be back online
  hosts: localhost
  gather_facts: no
  tasks:
    - name: Pause 2 minutes for VM reboot
      pause:
        minutes: 2

    - name: Wait for each VM SSH
      wait_for:
        host: "{{ item }}"
        port: "{{ hostvars[item].ansible_port | default(22) }}"
        timeout: 300
      loop: "{{ groups['vms'] }}"

IAmNix

@somik said:

Fought with ansible for a while. It wants python module called "six" to be installed on ALL of my remote nodes/vms...

Anyone know what I am doing wrong?

This smells like mismatched ansible version to me. Ubuntu repos especially are very behind - maybe try finding a PPA to install ansible from. Or maybe install it from pip.

@somik said:

Wont that expose my server's IP and ssh port? Or do you mean self hosted git repository or private github repo?

Not unless you want to share the git repo with other people - I was assuming this was 100% for personal use, and that you already have SSH access to some server to store the git repo somehow.
To be clear this git shenanigans is only useful for you to sync your ansible code between different personal machines, and as a simple backup. All it does is copy the code to a server somewhere.

For shared usage I would just use a public git host with a private repository, like codeberg.org.

(This SSH trick works because a git repository is technically just any folder with a .git subfolder in it. So we can use any protocol to sync the changes.)

How to setup an ansible git repo over SSH:

On your remote machine:

mkdir my-ansible; cd my-ansible
git init
# git commit something just to test with

On your laptop:

git clone ssh://user@remote-machine/path/to/my-ansible
cd my-ansible
git log # You should see your test commit
# git commit something
git push # <-- this will automatically push over SSH and changes will show up in the remote folder
git pull # <-- this will automatically fetch any changes from the remote folder

bingobangobongo

@IAmNix said:

@somik said:

Fought with ansible for a while. It wants python module called "six" to be installed on ALL of my remote nodes/vms...

Anyone know what I am doing wrong?

This smells like mismatched ansible version to me. Ubuntu repos especially are very behind - maybe try finding a PPA to install ansible from. Or maybe install it from pip.

Yep definitely something between versions here… I actually don’t install anything on nodes and have a Debian prep script that runs everything needed, so any blank/base install just werks.

Will sanitize and post here shortly!

localhost

Currently have my laptop and phone (juice ssh) with keys to most vms

Realized it hard way a few days ago... if I wanted to transfer files from one vm to another it is next to impossible as all are key based auth and none have keys to login to other

Curious what is the way to go about...

Have a decent bunch of idlers so having n idlers keys on m machines seems too much?

bingobangobongo

@localhost said:
Currently have my laptop and phone (juice ssh) with keys to most vms

Realized it hard way a few days ago... if I wanted to transfer files from one vm to another it is next to impossible as all are key based auth and none have keys to login to other

Curious what is the way to go about...

Have a decent bunch of idlers so having n idlers keys on m machines seems too much?

Maybe have a secondary layer of auth/keys for stuff between nodes? Then as part of setup, deploy the key used for inter node work?

Or Ansible, since noone has mentioned it yet…

localhost

@bingobangobongo said:

@localhost said:
Currently have my laptop and phone (juice ssh) with keys to most vms

Realized it hard way a few days ago... if I wanted to transfer files from one vm to another it is next to impossible as all are key based auth and none have keys to login to other

Curious what is the way to go about...

Have a decent bunch of idlers so having n idlers keys on m machines seems too much?

Maybe have a secondary layer of auth/keys for stuff between nodes? Then as part of setup, deploy the key used for inter node work?

Or Ansible, since noone has mentioned it yet…

Yeah.
Secondary keys is needed. But wanna keep overhead of number of keys limited.
How to securely set same key on all nodes?

I started tinkering with ansible and got my laptop and phone keys added to authorized keys. So there's that. But actual private key sharing over the wire seems trippy?

WSS

@localhost said:
Curious what is the way to go about...

python3 -m HTTPServer

localhost

@WSS said:

@localhost said:
Curious what is the way to go about...

python3 -m HTTPServer

Thats one way I was thinking too.
Curious if having one key across all makes sense or to do the painful way for x keys on y servers

AuroraZero

@localhost said: Curious if having one key across all makes sense or to do the painful way for x keys on y servers

2 keys total, one main one and a backup. If you are the only admin there is no sense making it difficult to manage or prone to human error.

somik

@localhost said:

@WSS said:

@localhost said:
Curious what is the way to go about...

python3 -m HTTPServer

Thats one way I was thinking too.
Curious if having one key across all makes sense or to do the painful way for x keys on y servers

No, it's never a good thing to share ssh keys.

I usually go with 3 ways.

If I want to automate file transfers, I usually create a low level user on the remote server and setup a ssh key between the 2 servers (like from my main server to my backup server).

If I only need to transfer a lot of files, I use SCP to download it to my desktop and reupload it back.

If it's just 1 or 2 files, I use: https://github.com/somik123/python3_http_upload_progress

@IAmNix said:

@somik said:

Fought with ansible for a while. It wants python module called "six" to be installed on ALL of my remote nodes/vms...

Anyone know what I am doing wrong?

This smells like mismatched ansible version to me. Ubuntu repos especially are very behind - maybe try finding a PPA to install ansible from. Or maybe install it from pip.

I installed it from ubuntu's repo, when that didn't work, I reinstalled it from python3 pip, but that didn't help either. I guess it's time to either try a proper PPA or install centos 8 and try from there. Thanks for the recommendations!

@bingobangobongo said:

@IAmNix said:

@somik said:

Fought with ansible for a while. It wants python module called "six" to be installed on ALL of my remote nodes/vms...

Anyone know what I am doing wrong?

This smells like mismatched ansible version to me. Ubuntu repos especially are very behind - maybe try finding a PPA to install ansible from. Or maybe install it from pip.

Yep definitely something between versions here… I actually don’t install anything on nodes and have a Debian prep script that runs everything needed, so any blank/base install just werks.

Will sanitize and post here shortly!

Thanks! Looking forward to it!

localhost

@somik said:

@localhost said:

@WSS said:

@localhost said:
Curious what is the way to go about...

python3 -m HTTPServer

Thats one way I was thinking too.
Curious if having one key across all makes sense or to do the painful way for x keys on y servers

No, it's never a good thing to share ssh keys.

I usually go with 3 ways.

If I want to automate file transfers, I usually create a low level user on the remote server and setup a ssh key between the 2 servers (like from my main server to my backup server).

If I only need to transfer a lot of files, I use SCP to download it to my desktop and reupload it back.

If it's just 1 or 2 files, I use: https://github.com/somik123/python3_http_upload_progress

Interesting
Well file transfer is definitely not automated.

And its not just transfer. I may need to quickly ssh into another box right from there... I know I can connect organically from my laptop. But since inam already on server 1 so if I need to check something in server 2 it should be easy, right?

Thanks for the quick http file server. Will bookmark it

nullnothere

I've not been following this fully, but a few quick hints:

1. Assume that your SSH connections are white listed (i.e. A->B, A->C and B<->C is allowed)
2. You have an ssh key for logins to connect from A->B and A->C
3. You want to transfer some file(s) from B->C
4. Use ssh-agent forwarding to provide access to your key ("resident" on A) to B (or C) thereby allowing B->C (or C->B) connections via this agent.
5. Since B->C (or vice-versa) is possible via agent forwarding, you can now directly rsync/scp files from B->C (or vice-versa).
6. man ssh-agent and ssh-add for options/security etc. RTFM.

Profit?

localhost

@nullnothere said:
I've not been following this fully, but a few quick hints:

1. Assume that your SSH connections are white listed (i.e. A->B, A->C and B<->C is allowed)
2. You have an ssh key for logins to connect from A->B and A->C
3. You want to transfer some file(s) from B->C
4. Use ssh-agent forwarding to provide access to your key ("resident" on A) to B (or C) thereby allowing B->C (or C->B) connections via this agent.
5. Since B->C (or vice-versa) is possible via agent forwarding, you can now directly rsync/scp files from B->C (or vice-versa).
6. man ssh-agent and ssh-add for options/security etc. RTFM.

Profit?

I think ssh-agent forwarding may work. Thanks

somik

@localhost said:

@nullnothere said:
I've not been following this fully, but a few quick hints:

1. Assume that your SSH connections are white listed (i.e. A->B, A->C and B<->C is allowed)
2. You have an ssh key for logins to connect from A->B and A->C
3. You want to transfer some file(s) from B->C
4. Use ssh-agent forwarding to provide access to your key ("resident" on A) to B (or C) thereby allowing B->C (or C->B) connections via this agent.
5. Since B->C (or vice-versa) is possible via agent forwarding, you can now directly rsync/scp files from B->C (or vice-versa).
6. man ssh-agent and ssh-add for options/security etc. RTFM.

Profit?

I think ssh-agent forwarding may work. Thanks

ssh bastion host?

localhost

@somik said:

@localhost said:

@nullnothere said:
I've not been following this fully, but a few quick hints:

1. Assume that your SSH connections are white listed (i.e. A->B, A->C and B<->C is allowed)
2. You have an ssh key for logins to connect from A->B and A->C
3. You want to transfer some file(s) from B->C
4. Use ssh-agent forwarding to provide access to your key ("resident" on A) to B (or C) thereby allowing B->C (or C->B) connections via this agent.
5. Since B->C (or vice-versa) is possible via agent forwarding, you can now directly rsync/scp files from B->C (or vice-versa).
6. man ssh-agent and ssh-add for options/security etc. RTFM.

Profit?

I think ssh-agent forwarding may work. Thanks

ssh bastion host?

That too...
Any recommendations?

somik

@localhost said:

@somik said:
ssh bastion host?

That too...
Any recommendations?

DIY solution without any software bloats or extra security risks?
https://smallstep.com/blog/diy-ssh-bastion-host/

Or you can go with one of the more popular software solutions:
https://goteleport.com/blog/ssh-bastion-host/

Or go with a proper zero trust network with bastionXP:
https://www.bastionxp.com/docs/guide/

IAmNix

@nullnothere said:
I've not been following this fully, but a few quick hints:

1. Assume that your SSH connections are white listed (i.e. A->B, A->C and B<->C is allowed)
2. You have an ssh key for logins to connect from A->B and A->C
3. You want to transfer some file(s) from B->C
4. Use ssh-agent forwarding to provide access to your key ("resident" on A) to B (or C) thereby allowing B->C (or C->B) connections via this agent.
5. Since B->C (or vice-versa) is possible via agent forwarding, you can now directly rsync/scp files from B->C (or vice-versa).
6. man ssh-agent and ssh-add for options/security etc. RTFM.

The ssh agent forwarding option is -A.

So the commands would look like this:

user@laptop:~$ ssh -A server-1
user@server-1:~$ scp my-files/ server-2:

Only do this if you trust server-1. It essentially shares access to all ssh keys on your laptop. As the man ssh says:

Agent forwarding should be enabled with caution. Users with the ability to bypass file permissions on the remote host (for the agent's Unix-domain socket) can access the local agent through the forwarded connection. An attacker cannot obtain key material from the agent, however they can perform operations on the keys that enable them to authenticate using the identities loaded into the agent. A safer alternative may be to use a jump host (see -J).

bingobangobongo

Finally posting my Ansible template for Debian setup... Sorry for the delay!

Codie mc Coderson

---
- name: Initial Debian Server Setup
  hosts: all
  gather_facts: no

  # Prompt for credentials at the very start.
  vars:
    # Automatically read the public key from the control node's /.ssh/id_ecdsa.pub
    authorized_pub_key: "{{ lookup('file', lookup('env', 'HOME') + '/.ssh/id_ecdsa.pub') }}"

  vars_prompt:
    - name: "username"
      prompt: "Enter the username to create"
      private: no
    - name: "password"
      prompt: "Enter the password for the new user"
      private: yes
      confirm: yes

  pre_tasks:
    - name: Install Python (required for all other Ansible modules)
      raw: |
        if ! command -v python3 &> /dev/null; then
          apt update && apt install -y python3
        fi
      become: yes
      changed_when: false

    - name: Update apt cache
      apt:
        update_cache: yes

    - name: Upgrade all installed packages (dist-upgrade)
      apt:
        upgrade: dist

    - name: Check if sudo is installed
      command: which sudo
      register: sudo_check
      ignore_errors: yes

    - name: Install sudo if not present
      apt:
        name: sudo
        state: present
        update_cache: yes
      when: sudo_check.rc != 0

    - name: Install community.general collection
      ansible.builtin.command: ansible-galaxy collection install community.general
      delegate_to: localhost
      run_once: true
      changed_when: false

    - name: Gather facts now that everything is set up
      setup:

  tasks:
    - block:
        - name: Install generally required packages
          apt:
            name:
              - curl
              - zip
              - unzip
              - git
              - smartmontools
              - nvme-cli
              - htop
              - iperf3
              - dnsutils
              - net-tools
              - whois
            state: present
            update_cache: yes

        - name: Hash the password using mkpasswd
          command: "mkpasswd --method=SHA-512 {{ password | quote }}"
          register: hashed_password
          no_log: true

        - name: Create the user with a hashed password
          user:
            name: "{{ username }}"
            password: "{{ hashed_password.stdout }}"
            shell: /bin/bash
            state: present
            groups: sudo
            append: yes
            createhome: yes
          become: yes

        - name: Pause for 3 seconds to ensure user is registered
          pause:
            seconds: 3

        - name: Ensure the user's home directory exists and set ownership
          file:
            path: "/home/{{ username }}"
            state: directory
            owner: "{{ username }}"
            group: "{{ username }}"
            mode: '0755'

        - name: Ensure .ssh directory exists with correct permissions
          file:
            path: "/home/{{ username }}/.ssh"
            state: directory
            owner: "{{ username }}"
            group: "{{ username }}"
            mode: '0700'

        - name: Add authorized SSH key for the new user
          authorized_key:
            user: "{{ username }}"
            key: "{{ authorized_pub_key }}"
            state: present
            path: "/home/{{ username }}/.ssh/authorized_keys"

        - name: Set correct permissions on the authorized_keys file
          file:
            path: "/home/{{ username }}/.ssh/authorized_keys"
            owner: "{{ username }}"
            group: "{{ username }}"
            mode: '0600'

        - name: Allow the user passwordless sudo in sudoers.d
          copy:
            dest: "/etc/sudoers.d/{{ username }}"
            content: "{{ username }} ALL=(ALL) NOPASSWD: ALL\n"
            mode: '0440'

        - name: Remove all files in sshd_config.d directory
          command: find /etc/ssh/sshd_config.d/ -type f -delete
          changed_when: true

        - name: Configure SSH to disable password authentication for sudo/admin users
          copy:
            dest: /etc/ssh/sshd_config.d/10-disable_password_for_admins.conf
            content: |
              Match Group sudo,admin
                  PasswordAuthentication no
            mode: '0644'
          notify: restart ssh

        - name: Ensure PermitRootLogin is set to prohibit-password in sshd_config
          lineinfile:
            path: /etc/ssh/sshd_config
            regexp: '^#?PermitRootLogin\s+.*$'
            line: 'PermitRootLogin prohibit-password'
            state: present
            backup: yes
          notify: restart ssh

        - name: Install fail2ban package
          apt:
            name: fail2ban
            state: present
            update_cache: yes

        - name: Configure fail2ban with custom settings and SSH protection
          copy:
            dest: /etc/fail2ban/jail.d/custom.conf
            content: |
              [DEFAULT]
              ignoreip = 127.0.0.1
              bantime = 1h
              findtime = 10m
              maxretry = 3
              banaction = ufw

              [sshd]
              enabled = true
              port    = ssh
              filter  = sshd
              logpath = /var/log/auth.log
            mode: '0644'
          notify: restart fail2ban

        - name: Install ufw package
          apt:
            name: ufw
            state: present
            update_cache: yes

        - name: Allow SSH through UFW
          community.general.ufw:
            rule: allow
            name: OpenSSH
          notify: restart ufw

        - name: Deny RPC through UFW
          community.general.ufw:
            rule: deny   
            port: '111'
          notify: restart ufw

        - name: Enable UFW on startup
          community.general.ufw:
            state: enabled
          notify: restart ufw

      rescue:
        - debug:
            msg: "An error occurred during provisioning. Please check the logs above."
        - fail:
            msg: "Provisioning failed."

    - debug:
        msg: "Provisioning complete. All tasks executed successfully."

    - name: Flush all pending handlers
      meta: flush_handlers

    - name: Prompt for reboot confirmation
      pause:
        prompt: "Server setup is complete. Do you want to reboot the server now? (y/n)"
      register: reboot_response

    - name: Reboot the server
      reboot:
        reboot_timeout: 300
      when: reboot_response.user_input | lower in ['y', 'yes']

    - name: Display message if reboot was skipped
      debug:
        msg: "Reboot skipped. You may want to reboot manually later to ensure all changes take effect."
      when: reboot_response.user_input | lower not in ['y', 'yes']

  handlers:
    - name: restart ssh
      service:
        name: ssh
        state: restarted

    - name: restart fail2ban
      service:
        name: fail2ban
        state: restarted

    - name: restart ufw
      service:
        name: ufw
        state: restarted

somik

@bingobangobongo said:
Finally posting my Ansible template for Debian setup... Sorry for the delay!

Codie mc Coderson

---
- name: Initial Debian Server Setup
  hosts: all
  gather_facts: no

  # Prompt for credentials at the very start.
  vars:
    # Automatically read the public key from the control node's /.ssh/id_ecdsa.pub
    authorized_pub_key: "{{ lookup('file', lookup('env', 'HOME') + '/.ssh/id_ecdsa.pub') }}"

  vars_prompt:
    - name: "username"
      prompt: "Enter the username to create"
      private: no
    - name: "password"
      prompt: "Enter the password for the new user"
      private: yes
      confirm: yes

  pre_tasks:
    - name: Install Python (required for all other Ansible modules)
      raw: |
        if ! command -v python3 &> /dev/null; thæn
          apt update && apt install -y python3
        fi
      become: yes
      changed_when: false

    - name: Update apt cache
      apt:
        update_cache: yes

    - name: Upgrade all installed packages (dist-upgrade)
      apt:
        upgrade: dist

    - name: Check if sudo is installed
      command: which sudo
      register: sudo_check
      ignore_errors: yes

    - name: Install sudo if not present
      apt:
        name: sudo
        state: present
        update_cache: yes
      when: sudo_check.rc != 0

    - name: Install community.general collection
      ansible.builtin.command: ansible-galaxy collection install community.general
      delegate_to: localhost
      run_once: true
      changed_when: false

    - name: Gather facts now that everything is set up
      setup:

  tasks:
    - block:
        - name: Install generally required packages
          apt:
            name:
              - curl
              - zip
              - unzip
              - git
              - smartmontools
            - nvme-cli
              - htop
            - iperf3
            - dnsutils
            - net-tools
              - whois
            state: present
            update_cache: yes

        - name: Hash the password using mkpasswd
          command: "mkpasswd --method=SHA-512 {{ password | quote }}"
          register: hashed_password
          no_log: true

        - name: Create the user with a hashed password
          user:
            name: "{{ username }}"
            password: "{{ hashed_password.stdout }}"
            shell: /bin/bash
            state: present
            groups: sudo
            append: yes
            createhome: yes
          become: yes

        - name: Pause for 3 seconds to ensure user is registered
          pause:
            seconds: 3

        - name: Ensure the user's home directory exists and set ownership
          file:
            path: "/home/{{ username }}"
            state: directory
            owner: "{{ username }}"
            group: "{{ username }}"
            mode: '0755'

        - name: Ensure .ssh directory exists with correct permissions
          file:
            path: "/home/{{ username }}/.ssh"
            state: directory
            owner: "{{ username }}"
            group: "{{ username }}"
            mode: '0700'

        - name: Add authorized SSH key for the new user
          authorized_key:
            user: "{{ username }}"
            key: "{{ authorized_pub_key }}"
            state: present
            path: "/home/{{ username }}/.ssh/authorized_keys"

        - name: Set correct permissions on the authorized_keys file
          file:
            path: "/home/{{ username }}/.ssh/authorized_keys"
            owner: "{{ username }}"
            group: "{{ username }}"
            mode: '0600'

        - name: Allow the user passwordless sudo in sudoers.d
          copy:
            dest: "/etc/sudoers.d/{{ username }}"
            content: "{{ username }} ALL=(ALL) NOPASSWD: ALL\n"
            mode: '0440'

        - name: Remove all files in sshd_config.d directory
          command: find /etc/ssh/sshd_config.d/ -type f -delete
          changed_when: true

        - name: Configure SSH to disable password authentication for sudo/admin users
          copy:
            dest: /etc/ssh/sshd_config.d/10-disable_password_for_admins.conf
            content: |
              Match Group sudo,admin
                  PasswordAuthentication no
            mode: '0644'
          notify: restart ssh

        - name: Ensure PermitRootLogin is set to prohibit-password in sshd_config
          lineinfile:
            path: /etc/ssh/sshd_config
            regexp: '^#?PermitRootLogin\s+.*$'
            line: 'PermitRootLogin prohibit-password'
            state: present
            backup: yes
          notify: restart ssh

        - name: Install fail2ban package
          apt:
            name: fail2ban
            state: present
            update_cache: yes

        - name: Configure fail2ban with custom settings and SSH protection
          copy:
            dest: /etc/fail2ban/jail.d/custom.conf
            content: |
              [DEFAULT]
              ignoreip = 127.0.0.1
              bantime = 1h
              findtime = 10m
              maxretry = 3
              banaction = ufw

              [sshd]
              enabled = true
              port    = ssh
              filter  = sshd
              logpath = /var/log/auth.log
            mode: '0644'
          notify: restart fail2ban

        - name: Install ufw package
          apt:
            name: ufw
            state: present
            update_cache: yes

        - name: Allow SSH through UFW
          community.general.ufw:
            rule: allow
            name: OpenSSH
          notify: restart ufw

        - name: Deny RPC through UFW
          community.general.ufw:
            rule: deny   
            port: '111'
          notify: restart ufw

        - name: Enable UFW on startup
          community.general.ufw:
            state: enabled
          notify: restart ufw

      rescue:
        - debug:
            msg: "An error occurred during provisioning. Please check the logs above."
        - fail:
            msg: "Provisioning failed."

    - debug:
        msg: "Provisioning complete. All tasks executed successfully."

    - name: Flush all pending handlers
      meta: flush_handlers

    - name: Prompt for reboot confirmation
      pause:
        prompt: "Server setup is complete. Do you want to reboot the server now? (y/n)"
      register: reboot_response

    - name: Reboot the server
      reboot:
        reboot_timeout: 300
      when: reboot_response.user_input | lower in ['y', 'yes']

    - name: Display message if reboot was skipped
      debug:
        msg: "Reboot skipped. You may want to reboot manually later to ensure all changes take effect."
      when: reboot_response.user_input | lower not in ['y', 'yes']

  handlers:
    - name: restart ssh
      service:
        name: ssh
        state: restarted

    - name: restart fail2ban
      service:
        name: fail2ban
        state: restarted

    - name: restart ufw
      service:
        name: ufw
        state: restarted

Wow... I'll need this weekend to process this...

Thanks for the detailed ansible template!

SpeedBus

Found this on reddit a while ago,
https://www.reddit.com/r/selfhosted/comments/1nlth9k/finally_built_patchmon_my_linux_updates

GitHub: https://github.com/PatchMon/PatchMon

GeekWanderer

@bingobangobongo said:
Finally posting my Ansible template for Debian setup... Sorry for the delay!

Codie mc Coderson

Nice! A couple of nice nuggets in that config. Consider them stolen. 😝

emperor

IDK if its already said, but you can install Komari Monitor and you can use remote exec commands on all servers at once.

AuroraZero

I don't think I mentioned Ansible in a minute or two, sorry I am slacking

bingobangobongo

@somik said:

@bingobangobongo said:
Finally posting my Ansible template for Debian setup... Sorry for the delay!

Codie mc Coderson

---
- name: Initial Debian Server Setup
  hosts: all
  gather_facts: no

  # Prompt for credentials at the very start.
  vars:
    # Automatically read the public key from the control node's /.ssh/id_ecdsa.pub
    authorized_pub_key: "{{ lookup('file', lookup('env', 'HOME') + '/.ssh/id_ecdsa.pub') }}"

  vars_prompt:
    - name: "username"
      prompt: "Enter the username to create"
      private: no
    - name: "password"
      prompt: "Enter the password for the new user"
      private: yes
      confirm: yes

  pre_tasks:
    - name: Install Python (required for all other Ansible modules)
      raw: |
        if ! command -v python3 &> /dev/null; thæn
          apt update && apt install -y python3
        fi
      become: yes
      changed_when: false

    - name: Update apt cache
      apt:
        update_cache: yes

    - name: Upgrade all installed packages (dist-upgrade)
      apt:
        upgrade: dist

    - name: Check if sudo is installed
      command: which sudo
      register: sudo_check
      ignore_errors: yes

    - name: Install sudo if not present
      apt:
        name: sudo
        state: present
        update_cache: yes
      when: sudo_check.rc != 0

    - name: Install community.general collection
      ansible.builtin.command: ansible-galaxy collection install community.general
      delegate_to: localhost
      run_once: true
      changed_when: false

    - name: Gather facts now that everything is set up
      setup:

  tasks:
    - block:
        - name: Install generally required packages
          apt:
            name:
              - curl
              - zip
              - unzip
              - git
              - smartmontools
              - nvme-cli
              - htop
              - iperf3
              - dnsutils
              - net-tools
              - whois
            state: present
            update_cache: yes

        - name: Hash the password using mkpasswd
          command: "mkpasswd --method=SHA-512 {{ password | quote }}"
          register: hashed_password
          no_log: true

        - name: Create the user with a hashed password
          user:
            name: "{{ username }}"
            password: "{{ hashed_password.stdout }}"
            shell: /bin/bash
            state: present
            groups: sudo
            append: yes
            createhome: yes
          become: yes

        - name: Pause for 3 seconds to ensure user is registered
          pause:
            seconds: 3

        - name: Ensure the user's home directory exists and set ownership
          file:
            path: "/home/{{ username }}"
            state: directory
            owner: "{{ username }}"
            group: "{{ username }}"
            mode: '0755'

        - name: Ensure .ssh directory exists with correct permissions
          file:
            path: "/home/{{ username }}/.ssh"
            state: directory
            owner: "{{ username }}"
            group: "{{ username }}"
            mode: '0700'

        - name: Add authorized SSH key for the new user
          authorized_key:
            user: "{{ username }}"
            key: "{{ authorized_pub_key }}"
            state: present
            path: "/home/{{ username }}/.ssh/authorized_keys"

        - name: Set correct permissions on the authorized_keys file
          file:
            path: "/home/{{ username }}/.ssh/authorized_keys"
            owner: "{{ username }}"
            group: "{{ username }}"
            mode: '0600'

        - name: Allow the user passwordless sudo in sudoers.d
          copy:
            dest: "/etc/sudoers.d/{{ username }}"
            content: "{{ username }} ALL=(ALL) NOPASSWD: ALL\n"
            mode: '0440'

        - name: Remove all files in sshd_config.d directory
          command: find /etc/ssh/sshd_config.d/ -type f -delete
          changed_when: true

        - name: Configure SSH to disable password authentication for sudo/admin users
          copy:
            dest: /etc/ssh/sshd_config.d/10-disable_password_for_admins.conf
            content: |
              Match Group sudo,admin
                  PasswordAuthentication no
            mode: '0644'
          notify: restart ssh

        - name: Ensure PermitRootLogin is set to prohibit-password in sshd_config
          lineinfile:
            path: /etc/ssh/sshd_config
            regexp: '^#?PermitRootLogin\s+.*$'
            line: 'PermitRootLogin prohibit-password'
            state: present
            backup: yes
          notify: restart ssh

        - name: Install fail2ban package
          apt:
            name: fail2ban
            state: present
            update_cache: yes

        - name: Configure fail2ban with custom settings and SSH protection
          copy:
            dest: /etc/fail2ban/jail.d/custom.conf
            content: |
              [DEFAULT]
              ignoreip = 127.0.0.1
              bantime = 1h
              findtime = 10m
              maxretry = 3
              banaction = ufw

              [sshd]
              enabled = true
              port    = ssh
              filter  = sshd
              logpath = /var/log/auth.log
            mode: '0644'
          notify: restart fail2ban

        - name: Install ufw package
          apt:
            name: ufw
            state: present
            update_cache: yes

        - name: Allow SSH through UFW
          community.general.ufw:
            rule: allow
            name: OpenSSH
          notify: restart ufw

        - name: Deny RPC through UFW
          community.general.ufw:
            rule: deny   
            port: '111'
          notify: restart ufw

        - name: Enable UFW on startup
          community.general.ufw:
            state: enabled
          notify: restart ufw

      rescue:
        - debug:
            msg: "An error occurred during provisioning. Please check the logs above."
        - fail:
            msg: "Provisioning failed."

    - debug:
        msg: "Provisioning complete. All tasks executed successfully."

    - name: Flush all pending handlers
      meta: flush_handlers

    - name: Prompt for reboot confirmation
      pause:
        prompt: "Server setup is complete. Do you want to reboot the server now? (y/n)"
      register: reboot_response

    - name: Reboot the server
      reboot:
        reboot_timeout: 300
      when: reboot_response.user_input | lower in ['y', 'yes']

    - name: Display message if reboot was skipped
      debug:
        msg: "Reboot skipped. You may want to reboot manually later to ensure all changes take effect."
      when: reboot_response.user_input | lower not in ['y', 'yes']

  handlers:
    - name: restart ssh
      service:
        name: ssh
        state: restarted

    - name: restart fail2ban
      service:
        name: fail2ban
        state: restarted

    - name: restart ufw
      service:
        name: ufw
        state: restarted

Wow... I'll need this weekend to process this...

Thanks for the detailed ansible template!

So have you got all your fancy new BF deal VEE PEE ESSES built via Ansible yet or what?!?

AuroraZero

Hey anyone mention Ansible yet?

dosai

Ansible

somik

@bingobangobongo said:

@somik said:
Wow... I'll need this weekend to process this...

Thanks for the detailed ansible template!

So have you got all your fancy new BF deal VEE PEE ESSES built via Ansible yet or what?!?

No, got just 1 dedi from "he who shall not be named (on LES)" and set it up with docker. Did not get the bandwidth to work on ansible yet...

Maybe i'll give it a try this coming weekend!

@AuroraZero said:
I don't think I mentioned Ansible in a minute or two, sorry I am slacking

Ansible? Is that like a new TV show?

@AuroraZero said:
Hey anyone mention Ansible yet?

What's that? Can eat?

@dosai said:
Ansible

No, it's A-sible. We do not put "an" before "S"

AuroraZero

@somik said:

@bingobangobongo said:

@somik said:
Wow... I'll need this weekend to process this...

Thanks for the detailed ansible template!

So have you got all your fancy new BF deal VEE PEE ESSES built via Ansible yet or what?!?

No, got just 1 dedi from "he who shall not be named (on LES)" and set it up with docker. Did not get the bandwidth to work on ansible yet...

Maybe i'll give it a try this coming weekend!

@AuroraZero said:
I don't think I mentioned Ansible in a minute or two, sorry I am slacking

Ansible? Is that like a new TV show?

@AuroraZero said:
Hey anyone mention Ansible yet?

What's that? Can eat?

@dosai said:
Ansible

No, it's A-sible. We do not put "an" before "S"

Ansible? Never heard of it

AaronSS

https://patchmon.net

WSS

Emacs

legendary

@AuroraZero said:
Unattended upgrades and kernel patching with notifications.

That sounds like ubuntu cheats. How about debian?

AuroraZero

@legendary said:

@AuroraZero said:
Unattended upgrades and kernel patching with notifications.

That sounds like ubuntu cheats. How about debian?

Same use it on debian 12 daily.

somik

@AuroraZero said:

@legendary said:

@AuroraZero said:
Unattended upgrades and kernel patching with notifications.

That sounds like ubuntu cheats. How about debian?

Same use it on debian 12 daily.

Yea, this seems like a better option if you want a fully hands off experience.

AuroraZero

@somik said:

@AuroraZero said:

@legendary said:

@AuroraZero said:
Unattended upgrades and kernel patching with notifications.

That sounds like ubuntu cheats. How about debian?

Same use it on debian 12 daily.

Yea, this seems like a better option if you want a fully hands off experience.

Keeps me free to debug python code backends.

WSS

Your mom

somik

@WSS said:
Your mom

I wish she was tech savy enough to manage my servers!

kako1talk

Use Ansible for the "Heavy Lifting"

somik

@kako1talk said:
Use Ansible for the "Heavy Lifting"

After fighting with Ansible for over 3 weekends, I regret to inform you that I have given up on getting it to work with my servers. Since most of my servers are LXC or minimum installs, most python modules ansible needs are not installed. So unattended updates it is. I can just go in and manually upgrade kernel for the full OS VM or dedis.

AuroraZero

@somik said:

@kako1talk said:
Use Ansible for the "Heavy Lifting"

After fighting with Ansible for over 3 weekends, I regret to inform you that I have given up on getting it to work with my servers. Since most of my servers are LXC or minimum installs, most python modules ansible needs are not installed. So unattended updates it is. I can just go in and manually upgrade kernel for the full OS VM or dedis.

Whahaha I beat Ansible!!!