What do you use to manage & update multiple servers?

somik

So I have a few servers on ubuntu and debian that I need to ssh into and run the following commands:

sudo apt update
sudo apt upgrade -y
sudo reboot

Previously I created a shell script that I ran from my linux box to ssh into the servers one by one and update them. I also configured the servers sudoers file to allow to run the above 3 commands for my user without password.

So I am wondering if there is a better way to do this.

Current ideas:
1. Continue to use shell scripts
2. Setup unattended-upgrades and run all updates through that
3. Use ansible from a control node
4. Give root access to chatgpt to run updates for me
5. Maybe a mixture of options 1 to 3?

Would love to hear how you are handling the updates and what you recommend is the best way.

AuroraZero

Unattended upgrades and kernel patching with notifications.

somik

@AuroraZero said:
Unattended upgrades and kernel patching with notifications.

Unattended upgrades, sure.

How to do kernel patching with notifications?

AuroraZero

I see I need to write a blog post on it

SharedGrid

@somik said:
4. Give root access to chatgpt to run updates for me

The professionals setup n8n to run updates for them.

For Ubuntu we use unattended upgrades for security updates. Anything else is done via Ansible.

terrorgen

NixOS.

Discovered it in 2021 and never looked back.

Now, We manage over 20 VPS, some with a provider, some self hosted.

GeekWanderer

Ansible for this and many other tasks.

sh97

Try Ansible - I think that is the way.

bugabuga

I use this: https://jimradford.github.io/superputty/

While its not automated, it allows you to issue a command to multiple server at once. Currently have 12 server setup in the manager. Saves time over having to putty to each server individually..

somik

Hmm, for some reason, I feel like I should try to do this with Ansible. Not sure why I feel that way. Just a gut feeling I guess...

.

@SharedGrid said:
Anything else is done via Ansible.

@GeekWanderer said:
Ansible for this and many other tasks.

@sh97 said:
Try Ansible - I think that is the way.

jaden

@somik said:
Hmm, for some reason, I feel like I should try to do this with Ansible. Not sure why I feel that way. Just a gut feeling I guess...

Surprised no one mentioned using Ansible...

yoursunny

1. CTRL+ALT+T opens the terminal.
2. Type ssh vps0 to connect to server.
3. Type sudo apt update && sudo apt full-upgrade -y && sudo reboot to run the upgrade, and copy this command to clipboard.
4. CTRL+SHIFT+T opens a new tab.
5. Type ssh vps1 to connect to server.
6. Paste the copied command.
7. Repeat steps 4~6 for the remaining 28 servers.
8. If SSH says hostname not found, it means the hostname is available for the next $6/year dealz.

somik

@yoursunny said:
1. CTRL+ALT+T opens the terminal.
2. Type ssh vps0 to connect to server.
3. Type sudo apt update && sudo apt full-upgrade -y && sudo reboot to run the upgrade, and copy this command to clipboard.
4. CTRL+SHIFT+T opens a new tab.
5. Type ssh vps1 to connect to server.
6. Paste the copied command.
7. Repeat steps 4~6 for the remaining 28 servers.
8. If SSH says hostname not found, it means the hostname is available for the next $6/year dealz.

If that's how you want to do it, you can do it much easily using a shell script (like i used to)...

somik

@jaden said:

@somik said:
Hmm, for some reason, I feel like I should try to do this with Ansible. Not sure why I feel that way. Just a gut feeling I guess...

Surprised no one mentioned using Ansible...

AuroraZero

@somik said:

@jaden said:

@somik said:
Hmm, for some reason, I feel like I should try to do this with Ansible. Not sure why I feel that way. Just a gut feeling I guess...

Surprised no one mentioned using Ansible...

Did anyone mention Ansible?

thagoat

Xpanes with an alias in my .zshrc to all the VM's IP's. Then I drop the yay bomb.

somik

@AuroraZero said:

@somik said:

@jaden said:

@somik said:
Hmm, for some reason, I feel like I should try to do this with Ansible. Not sure why I feel that way. Just a gut feeling I guess...

Surprised no one mentioned using Ansible...

Did anyone mention Ansible?

somik

@thagoat said:
Xpanes with an alias in my .zshrc to all the VM's IP's. thæn I drop the yay bomb.

zshrc = mac?

I see they have it for debian/ubuntu and centos as well... But i dont really use linux as a daily driver anymore. Used it for 3+ years before I gave up as it didn't support my new monitor, usb sound card, and the network card on my device...

quicksilver03

I use Ansible and Puppet: Ansible to run commands on all the hosts in my inventory, Puppet to get each host to a certain state depending on what the host is supposed to do.

Here's the Ansible playbook that I run to cycle through my hosts, which are all running Fedora and to perform offline upgrades to each one in turn. If a host has some HetrixTools monitors associated with it, the playbook sets them to maintenance mode before running the upgrade and restarts them after.

---

- name: "offline-upgrade"
  hosts: all
  user: operator
  become: true
  serial: 1
  tasks:
    - name: Pause Uptime Monitors
      uri:
        url: "https://api.hetrixtools.com/v2/{{ hetrixtools_api_token }}/maintenance/{{ item }}/3/"
      with_items: "{{ hostvars[inventory_hostname]['hetrixtools_monitors'] }}"
      delegate_to: "127.0.0.1"
    - name: Download Upgrades
      shell:
        cmd: dnf5 upgrade -y --refresh --offline
      register: dnf_download
    - name: Pause Before Reboot
      pause:
        seconds: 30
    - name: Rebooting Host
      shell:
        cmd: sleep 5 && dnf5 -y offline reboot
      async: 1
      poll: 0
      ignore_errors: yes
    - name: Host Is Installing Upgrades
      pause:
        seconds: "{{ wait_after_reboot_secs }}"
    - name: Waiting For Host Reboot
      wait_for_connection:
      args:
        delay: 15
        timeout: 300
    - name: Execute Puppet
      shell:
        cmd: /usr/local/sbin/puppet
    - name: Restart Uptime Monitors
      uri:
        url: "https://api.hetrixtools.com/v2/{{ hetrixtools_api_token }}/maintenance/{{ item }}/1/"
      with_items: "{{ hostvars[inventory_hostname]['hetrixtools_monitors'] }}"
      delegate_to: "127.0.0.1"

somik

@quicksilver03 said:
I use Ansible and Puppet: Ansible to run commands on all the hosts in my inventory, Puppet to get each host to a certain state depending on what the host is supposed to do.

Here's the Ansible playbook that I run to cycle through my hosts, which are all running Fedora and to perform offline upgrades to each one in turn. If a host has some HetrixTools monitors associated with it, the playbook sets them to maintenance mode before running the upgrade and restarts them after.

Thank you for the ansible playbook! Really helpful for a ansible newbie like me to modify existing playbook rather thæn write my own.

IAmNix

Outside ansible, another option is to use a terminal multiplexer like Terminator to type commands in many windows at once

If you do go with ansible, I'd propose the way we do it at work:

• Store ansible playbook in git repository
• git commit + push any change you actually deploy
• Run ansible from your laptop

netrix

telegram bot
my server pick up a command & run it every 1 min (if exist)

AuroraZero

@netrix said:
telegram bot
my server pick up a command & run it every 1 min (if exist)

That is kind of a novel way to do it.

terrorgen

Shameless Nix plug here.
You don't need to switch distros to get the benefit of nix.

AuroraZero

@terrorgen said:
Shameless Nix plug here.
You don't need to switch distros to get the benefit of nix.

Why did you do it man? Wy did you have to do it?

terrorgen

@AuroraZero said:

@terrorgen said:
Shameless Nix plug here.
You don't need to switch distros to get the benefit of nix.

Why did you do it man? Wy did you have to do it?

why not? Nix is great but lacking marketing resources like Ansible!!

localhost

What is Nix?

AuroraZero

@localhost said:
What is Nix?

SSSSSSHHHHHH man you will awaken the weirdos!!!

localhost

@AuroraZero said:

@localhost said:
What is Nix?

SSSSSSHHHHHH man you will awaken the weirdos!!!

Why you think I posted this ?

Its too quiet here

AuroraZero

@localhost said:

@AuroraZero said:

@localhost said:
What is Nix?

SSSSSSHHHHHH man you will awaken the weirdos!!!

Why you think I posted this ?

Its too quiet here

You don't want that kind of weirdness believe me man.

brauni

@netrix said:
telegram bot
my server pick up a command & run it every 1 min (if exist)

do you run a bot per server?
I tried something similar, but multiple instances of the same bot were no bueno
Maybe you can give some context, would love to know where i went wrong

localhost

@AuroraZero said:

@localhost said:

@AuroraZero said:

@localhost said:
What is Nix?

SSSSSSHHHHHH man you will awaken the weirdos!!!

Why you think I posted this ?

Its too quiet here

You don't want that kind of weirdness believe me man.

So what kind of things will calm you down?

AuroraZero

@localhost said:

@AuroraZero said:

@localhost said:

@AuroraZero said:

@localhost said:
What is Nix?

SSSSSSHHHHHH man you will awaken the weirdos!!!

Why you think I posted this ?

Its too quiet here

You don't want that kind of weirdness believe me man.

So what kind of things will calm you down?

I am calm man I have not banned anyone since my second term that level of calm is extraordinary.

terrorgen

@AuroraZero said:

@localhost said:

@AuroraZero said:

@localhost said:
What is Nix?

SSSSSSHHHHHH man you will awaken the weirdos!!!

Why you think I posted this ?

Its too quiet here

You don't want that kind of weirdness believe me man.

Why is nix weird?

AuroraZero

@terrorgen said:

@AuroraZero said:

@localhost said:

@AuroraZero said:

@localhost said:
What is Nix?

SSSSSSHHHHHH man you will awaken the weirdos!!!

Why you think I posted this ?

Its too quiet here

You don't want that kind of weirdness believe me man.

Why is nix weird?

Real Q is why isn't it?

terrorgen

@AuroraZero said:

@terrorgen said:

@AuroraZero said:

@localhost said:

@AuroraZero said:

@localhost said:
What is Nix?

SSSSSSHHHHHH man you will awaken the weirdos!!!

Why you think I posted this ?

Its too quiet here

You don't want that kind of weirdness believe me man.

Why is nix weird?

Real Q is why isn't it?

Speaking about nix as a package-manager-as-code, it is like ansible with its own repo, and more. Nix is a programming language tailored for reproducibility of system environments.

Nix was already out there before Ansible was even a thing. But Ansible has marketing budget, so...

bingobangobongo

You should definitely check out Ansible! It's super powerful and very amazing - much gud.

somik

@IAmNix said:
Outside ansible, another option is to use a terminal multiplexer like Terminator to type commands in many windows at once

If you do go with ansible, I'd propose the way we do it at work:

• Store ansible playbook in git repository
• git commit + push any change you actually deploy
• Run ansible from your laptop

Wont that expose my server's IP and ssh port? Or do you mean self hosted git repository or private github repo?

@netrix said:
telegram bot
my server pick up a command & run it every 1 min (if exist)

Ya, tried that, but gave up quickly. I would rather run it via cron thæn use telegram bot message polling.

@brauni said:

@netrix said:
telegram bot
my server pick up a command & run it every 1 min (if exist)

do you run a bot per server?
I tried something similar, but multiple instances of the same bot were no bueno
Maybe you can give some context, would love to know where i went wrong

No, you have 1 centralized bot and poll the bot for messages (not webhook). So basically you request this url every 1 min:
https://api.telegram.org//getUpdates

Note that you NEED to have webhooks disabled for the bot for this to work, so go ahead and create a new bot using telegram botfather for this. Once you send any message to the telegram bot, it'll show up if you open the api getUpdates as json. You can parse it to use the same bot to send commands to all servers.

@bingobangobongo said:
You should definitely check out Ansible! It's super powerful and very amazing - much gud.

Weird... I dont think anyone mentioned Ansible yet. Is it something new?

somik

And yes, I am completely skipping over everything related to nix

AuroraZero

@somik said:
And yes, I am completely skipping over everything related to nix

Chicken!!!!

terrorgen

@somik said:
And yes, I am completely skipping over everything related to nix

bo-hood

no balls

somik

@AuroraZero said:

@somik said:
And yes, I am completely skipping over everything related to nix

Chicken!!!!

@terrorgen said:

@somik said:
And yes, I am completely skipping over everything related to nix

bo-hood

no balls

terrorgen

It's really not that bad...

somik

@terrorgen said:
It's really not that bad...

You sure? People complained about snaps when ubuntu introduced it. Now you want to discuss NixOS!!!

I mean jJust look at the screenshots...

Regular linux (Linux Mint) vs NixOS:

terrorgen

You're judging a distro by its screenshots?

And I was talking about Nix, not NixOS. You can run nix in every major distros.

somik

@terrorgen said:
You're judging a distro by its screenshots?

And I was talking about Nix, not NixOS. You can run nix in every major distros.

Ofcourse! Haven't you heard about judging a book by it's cover? It's where the title and author are printed.

Nix, I'm guessing, it's what nixos uses? Like a snap or flatpac lookalike?

cochon

Many years ago I used to set remote servers to refresh configs via a simple overnight cron job that would pull a PGP signed tarball, verify and unpack it before running the update script it contained.

These days I tend to use ansible, but still set up that cron job ‘just in case’. No one’s infallible or immune from sawing off the branch they’re sat on.

AuroraZero

You all know I am just picking on nix right? It is a good tool that does it job. No need for the hate I am just trying to have fun and make people laugh.

On that note did anyone mention Ansible?

WSS

I used ansible when working for providers. My own set are so diverse that it makes no sense to automate anything except for my cluster of nameservers, because everything else is completely purpose-built for that specific task.

terrorgen

@somik said:

@terrorgen said:
You're judging a distro by its screenshots?

And I was talking about Nix, not NixOS. You can run nix in every major distros.

Ofcourse! Haven't you heard about judging a book by it's cover? It's where the title and author are printed.

Nix, I'm guessing, it's what nixos uses? Like a snap or flatpac lookalike?

it is like snap and ansible had a baby that was born before both existed.

somik

@cochon said:
Many years ago I used to set remote servers to refresh configs via a simple overnight cron job that would pull a PGP signed tarball, verify and unpack it before running the update script it contained.

These days I tend to use ansible, but still set up that cron job ‘just in case’. No one’s infallible or immune from sawing off the branch they’re sat on.

You mean like deleting the only VM that stored all your ssh keys that you used to access all of your other servers and not having any backups?

No, that never happened to me. I am not that stupid.

Oh, you are asking why I had to recreate and reinstall OS on all of my servers in 2024? That's because I felt it was time for a refresh; and since I had backups of my data in my external drive, why not!

Definitely not because I am stupid and deleted the only VM that ran on my windows laptop that had the only pair of ssh keys that I used to access all the other servers...

@AuroraZero said:
On that note did anyone mention Ansible?

No, I dont believe anyone had mentioned that...

@WSS said:
I used ansible when working for providers. My own set are so diverse that it makes no sense to automate anything except for my cluster of nameservers, because everything else is completely purpose-built for that specific task.

Sounds like I have what I need to work on this weekend... Ansible...

Now where should I set it up? A raspberry pi?

@terrorgen said:

@somik said:

@terrorgen said:
You're judging a distro by its screenshots?

And I was talking about Nix, not NixOS. You can run nix in every major distros.

Ofcourse! Haven't you heard about judging a book by it's cover? It's where the title and author are printed.

Nix, I'm guessing, it's what nixos uses? Like a snap or flatpac lookalike?

it is like snap and ansible had a baby that was born before both existed.

Sounds like fun! I'll look into it when I have more time. If it works on mac, i might be able to use it to install and switch between all the Java versions I need for work...