freeCodeCamp/docs/flight-manuals/working-on-virtual-machines.md

Flight Manual for working on Virtual Machines

As a member of the staff or the dev-team, you may have been given access to our cloud service providers like Azure, Digital Ocean, etc.

Here are some handy commands that you can use to work on the Virtual Machines (VM), for instance performing maintenance updates or doing general houeskeeping.

Get a list of the VMs

[!NOTE] While you may already have SSH access to the VMs, that alone will not let you list VMs unless you been granted access to the cloud portals as well.

Azure

Install Azure CLI az: https://docs.microsoft.com/en-us/cli/azure/install-azure-cli

(One-time) Install on macOS with homebrew:

brew install azure-cli

(One-time) Login:

az login

Get the list of VM names and P addresses:

az vm list-ip-addresses --output table

Digital Ocean

Install Digital Ocean CLI doctl: https://github.com/digitalocean/doctl#installing-doctl

(One-time) Install on macOS with homebrew:

brew install doctl

(One-time) Login:

Authentication and context switching: https://github.com/digitalocean/doctl#authenticating-with-digitalocean

doctl auth init

Get the list of VM names and IP addresses:

doctl compute droplet list --format "ID,Name,PublicIPv4"

Spin a VM (or VM Scale Set)

Todo: Add instructions for spinning VM(s)

Keep VMs updated

You should keep the VMs up to date by performing updates and upgrades. This will ensure that the virtual machine is patched with latest security fixes.

[!WARNING] Before you run these commands:

• Make sure that the VM has been provisioned completely and there is no post-install steps running.
• If you are updating packages on a VM that is already serving an application, make sure the app has been stopped / saved. Package updates will cause network bandwidth, memory and/or CPU usage spikes leading to outages on running applications.

Update package information

sudo apt update

Upgrade installed packages

sudo apt upgrade -y

Cleanup unused packages

sudo apt autoremove -y

Work on Web Servers (Proxy)

We are running load balanced (Azure Load Balancer) instances for our web servers. These servers are running NGINX which reverse proxy all of the traffic to freeCodeCamp.org from various applications running on their own infrastructures.

The NGINX config is available on this repository.

First Install

Provisioning VMs with the Code

1. (Optional) Install NGINX and configure from repository.

The basic setup should be ready OOTB, via the cloud-init configuration. SSH and make changes as necessary for the particular instance(s).

If you did not use the cloud-init config previously use the below for manual setup of NGINX and error pages:

sudo su

cd /var/www/html
git clone https://github.com/freeCodeCamp/error-pages

cd /etc/
rm -rf nginx
git clone https://github.com/freeCodeCamp/nginx-config nginx

cd /etc/nginx

2. Install Cloudflare origin certificates and upstream application config.

Get the Cloudflare origin certificates from the secure storage and install at required locations.

OR

Move over existing certificates:

# Local
scp -r username@source-server-public-ip:/etc/nginx/ssl ./
scp -pr ./ssl username@target-server-public-ip:/tmp/

# Remote
rm -rf ./ssl
mv /tmp/ssl ./

Update Upstream Configurations:

vi configs/upstreams.conf

Add/update the source/origin application IP addresses.

3. Setup networking and firewalls.

Configure Azure firewalls and ufw as needed for ingress origin addresses.

4. Add the VM to the load balancer backend pool.

Configure and add rules to load balancer if needed. You may also need to add the VMs to load balancer backend pool if needed.

Logging and Monitoring

1. Check status for NGINX service using the below command:

sudo systemctl status nginx

2. Logging and monitoring for the servers are available at:

https://amplify.nginx.com

Updating Instances (Maintenance)

Config changes to our NGINX instances are maintained on GitHub, these should be deployed on each instance like so:

1. SSH into the instance and enter sudo

sudo su

2. Get the latest config code.

cd /etc/nginx
git fetch --all --prune
git reset --hard origin/master

3. Test and reload the config with Signals.

nginx -t
nginx -s reload

Work on API Instances

1. Install build tools for node binaries (node-gyp) etc.

sudo apt install build-essential

First Install

Provisioning VMs with the Code

1. Install Node LTS.

2. Update npm and install PM2 and setup logrotate and startup on boot

   npm i -g npm
   npm i -g pm2
   pm2 install pm2-logrotate
   pm2 startup
   

3. Clone freeCodeCamp, setup env and keys.

   git clone https://github.com/freeCodeCamp/freeCodeCamp.git
   cd freeCodeCamp
   git checkout production-current # or any other branch to be deployed
   

4. Create the .env from the secure credentials storage.

5. Create the google-credentials.json from the secure credentials storage.

6. Install dependencies

   npm ci
   

7. Build the server

   npm run ensure-env && npm run build:server
   

8. Start Instances

   cd api-server
   pm2 start production-start.js -i max --max-memory-restart 600M --name org
   

Logging and Monitoring

pm2 logs

pm2 monit

Updating Instances (Maintenance)

Code changes need to be deployed to the API instances from time to time. It can be a rolling update or a manual update. The later is essential when changing dependencies or adding enviroment variables.

[!DANGER] The automated pipelines are not handling dependencies updates at the minute. We need to do a manual update before any deployment pipeline runs.

1. Manual Updates - Used for updating dependencies, env variables.

1. Stop all instances

pm2 stop all

2. Install dependencies

npm ci

3. Build the server

npm run ensure-env && npm run build:server

4. Start Instances

pm2 start all --update-env && pm2 logs

2. Rolling updates - Used for logical changes to code.

pm2 reload all --update-env && pm2 logs

[!NOTE] We are handling rolling updates to code, logic, via pipelines. You should not need to run these commands. These are here for documentation.

Work on Client Instances

1. Install build tools for node binaries (node-gyp) etc.

sudo apt install build-essential

First Install

Provisioning VMs with the Code

1. Install Node LTS.

2. Update npm and install PM2 and setup logrotate and startup on boot

   npm i -g npm
   npm i -g pm2
   npm install -g serve
   pm2 install pm2-logrotate
   pm2 startup
   

3. Clone client config, setup env and keys.

   git clone https://github.com/freeCodeCamp/client-config.git client
   cd client
   

   git clone https://github.com/freeCodeCamp/client-config.git client
   cd client
   

   Start placeholder instances for the web client, these will be updated with artifacts from the Azure pipline.

   Todo: This setup needs to move to S3 or Azure Blob storage

   echo "serve -c ../../serve.json www -p 50505" >> client-start-primary.sh
   chmod +x client-start-primary.sh
   pm2 delete client-primary
   pm2 start  ./client-start-primary.sh --name client-primary
   echo "serve -c ../../serve.json www -p 52525" >> client-start-secondary.sh
   chmod +x client-start-secondary.sh
   pm2 delete client-secondary
   pm2 start  ./client-start-secondary.sh --name client-secondary
   

Logging and Monitoring

pm2 logs

pm2 monit

Updating Instances (Maintenance)

Code changes need to be deployed to the API instances from time to time. It can be a rolling update or a manual update. The later is essential when changing dependencies or adding enviroment variables.

[!DANGER] The automated pipelines are not handling dependencies updates at the minute. We need to do a manual update before any deployment pipeline runs.

1. Manual Updates - Used for updating dependencies, env variables.

1. Stop all instances

   pm2 stop all
   

2. Install or update dependencies

3. Start Instances

   pm2 start all --update-env && pm2 logs
   

2. Rolling updates - Used for logical changes to code.

pm2 reload all --update-env && pm2 logs

[!NOTE] We are handling rolling updates to code, logic, via pipelines. You should not need to run these commands. These are here for documentation.