Rubin Shrestha
Zend certified PHP/Magento developer
Rubin Shrestha
Blog

GitHub Actions Workflow – Remote Server Upgrade / Reboot and Continue to the Next Step

by |Published

I’m in the beginning of developing a workflow yml script. The main idea is to have a manually triggered “remote server” build file, to set up all the necessary applications in a “Host Server” that will later handle docker containers and other workflows will be handles by merge triggers.

Problem I’m facing is with the first step of the server configuration. Manually, I do:

apt update
apt apt upgrade -y
sudo reboot

Wait for the server to reboot (come back online) so it loads the new kernel and continue with the installation.

I’m trying to automate the setup in GitHub Actions Workflow. Up till now, this is how my file looks like:

name: Host Server Build V1.0.0

on:
  workflow_dispatch:

jobs:
  build:
      runs-on: ubuntu-latest
      
      env:
        CONFIG_SERVER_SSH_KEY: '${{ secrets.CONFIG_SERVER_SSH_KEY }}'
        CONFIG_SERVER_BACKEND_IP: '${{ secrets.CONFIG_SERVER_BACKEND_IP }}'

      steps:
        - name: Ubuntu Update / Upgrade / Reboot (back-end)
          shell: bash
          run: |
            BACKEND_HOST_SERVER_SH_SCRIPT=' 
            sudo DEBIAN_FRONTEND=noninteractive apt update && sudo DEBIAN_FRONTEND=noninteractive apt upgrade -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold"; 
            sleep 5; 
            sudo reboot || true; 
            ';
            echo "${{ env.CONFIG_SERVER_SSH_KEY }}" > id_rsa_server_private_key_temp.pem;
            chmod 600 id_rsa_server_private_key_temp.pem;
            ssh -v -t -t -i id_rsa_server_private_key_temp.pem -o ConnectTimeout=300 -o StrictHostKeyChecking=no ubuntu@${{ env.CONFIG_SERVER_BACKEND_IP }} "$BACKEND_HOST_SERVER_SH_SCRIPT"
            # rm id_rsa_server_private_key_temp.pem;
            echo "Status check: Reboot initiated.";
            sleep 60;
            # Verify if the server is back online
            # until ping -c 1 ${{ env.CONFIG_SERVER_BACKEND_IP }} & > /dev/null; do
            #   echo "Server not yet available, retrying in 15 seconds...";
            #   sleep 15;
            # done;
            # echo "Status check: Server is back online.";
            echo 'Waiting for server to come back online...';
            until ssh -o StrictHostKeyChecking=no -i id_rsa_server_private_key_temp.pem ubuntu@${{ env.CONFIG_SERVER_BACKEND_IP }} 'echo "Server is back online"'; do
              echo "Server not yet available, retrying in 30 seconds...";
              sleep 30;
            done
            echo 'Status check: Server is back online, proceeding with the workflow.';
            rm id_rsa_server_private_key_temp.pem;
            
        - name: Docker - Install
          shell: bash
          run: |
            BACKEND_HOST_SERVER_SH_SCRIPT=' 
            sudo apt install -y apt-transport-https ca-certificates curl software-properties-common; 
            curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg; 
            echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null; 
            sudo apt install -y docker-ce docker-ce-cli containerd.io; 
            ';
            echo "${{ secrets.CONFIG_SERVER_SSH_KEY }}" > id_rsa_server_private_key_temp.pem;
            chmod 600 id_rsa_server_private_key_temp.pem;
            ssh -v -t -t -i id_rsa_server_private_key_temp.pem -o StrictHostKeyChecking=no ubuntu@${{ env.CONFIG_SERVER_BACKEND_IP }} "$BACKEND_HOST_SERVER_SH_SCRIPT"
            rm id_rsa_server_private_key_temp.pem;
            echo "Status check: Docker and dependencies installed successfully.";

I had to modify apt upgrade -y part because of the prompts it would be stuck on.

However, seems like the runner is encountering an error after reboot and can’t continue to the next steps. The error looks something like this:

NEEDRESTART-VER: 3.5
NEEDRESTART-KCUR: 6.***.***-1***17-aws
NEEDRESTART-KEXP: 6.***.***-1***18-aws
NEEDRESTART-KSTA: 3
NEEDRESTART-SVC: acpid.service
NEEDRESTART-SVC: chrony.service
NEEDRESTART-SVC: cron.service
NEEDRESTART-SVC: dbus.service
NEEDRESTART-SVC: getty@tty1.service
NEEDRESTART-SVC: networkd-dispatcher.service
NEEDRESTART-SVC: packagekit.service
NEEDRESTART-SVC: polkit.service
NEEDRESTART-SVC: rsyslog.service
NEEDRESTART-SVC: serial-getty@ttyS***.service
NEEDRESTART-SVC: snapd.service
NEEDRESTART-SVC: systemd-logind.service
NEEDRESTART-SVC: unattended-upgrades.service
NEEDRESTART-SVC: user@1***.service
debug1: channel ***: free: client-session, nchannels 1
Connection to *** closed by remote host.
Connection to *** closed.
Transferred: sent ***9***, received 1***584 bytes, in 83.*** seconds
Bytes per second: sent 34.9, received 1***9.3
debug1: Exit status -1

Anyone has an idea of how this can be done? Or if this type of approach is correct? Maybe splitting into 2 jobs? Initially, I wanted to put the “wait” part into a separate step, like this:

        - name: Server - Wait for it to be Back Online (back-end)
          shell: bash
            
          run: |
            echo "${{ secrets.CONFIG_SERVER_SSH_KEY }}" > id_rsa_server_private_key_temp.pem
            chmod 600 id_rsa_server_private_key_temp.pem
            echo 'Waiting for server to come back online...';
            until ssh -o StrictHostKeyChecking=no -i id_rsa_server_private_key_temp.pem ubuntu@${{ env.CONFIG_SERVER_BACKEND_IP }} 'echo "Server is back online"'; do
              echo "Server not yet available, retrying in 30 seconds...";
              sleep 30;
            done
            echo 'Status check: Server is back online, proceeding with the workflow.';
            rm id_rsa_server_private_key_temp.pem;

Server context:

  • AWS / EC2
  • Ubuntu
  • Fresh install

You may also like