CI/CD Basics

The basics of CI/CD pipelines, github actions and github pages

What is Continuous Integration/Continuous Deployment?

Essentially it is a method for defining automated virtual computing processes. In laymens terms it typically means writing a configuration file or script that will spin up a 'virtual computer' (though sometimes it will run on the host computer) and run through a series of defined tasks.

What is CI/CD Used for?

Any task that you might want a 'virtual computer' to run it on for example

  • Testing
  • Building a distribution
  • Automated updating
  • etc.

Pipeline/Workflow vs jobs/steps

A pipeline/workflow is the full definition of the task (can mean the file or the actions). Jobs/steps are the individual actions or group of actions that make up a pipeline/workflow

Environment

The environment the pipeline/workflow runs in including the operating system, any configuration variables, and any included binaries/languages

Strategy for making a pipeline

Imagine you have someone you are training to do the task. They have a brand new laptop and you need to help them get setup and do the task

An example

Last week we looked at using hugo with the ignite site, what are the steps needed to build the ignite site if you were given a brand new laptop?

Overview of steps for the example

First we need to clone the site so we have the files, next we need to install hugo, then we need to build them (hugo -d), and finally we need to take the files and put them somewhere other people can access them

Step 1: Clone the site so we have the files

First we need to clone the site down, this means we will need some sort of OS and git along with a URL to clone the files down. So once we have an os we need to run:

git clone https://github.com/Schulich-Ignite/website

Step 2: We need to install hugo

This means either installing go and installing hugo that way, or downloading a binary

Step 3: Build the files

Need to run the hugo command over the files we downloaded

hugo -d

Step 4: Deploy the files

Find a way to deploy those files so other people can access them (will cover this in detail tomorrow)

Some example systems

Every system is a little different. Don't worry about understanding these fully, we will cover one system in depth later these are just here to show other approaches

Jenkins

Example jenkinsfile (printing Hello world! at cli)

pipeline {
    agent any 
    stages {
        stage('Stage 1') {
            steps {
                echo 'Hello world!' 
            }
        }
    }
}
Site

Ansible

---
- name: This is a hello-world example
  hosts: ansibleclient01.local
  tasks:
    - name: Create a file called '/tmp/testfile.txt' with the content 'hello world'.
      copy:
        content: hello worldn
        dest: /tmp/testfile.txt
Site

Gitlab CI/CD

build-job:
  stage: build
  script:
    - echo "Hello World!"
Site

Travis CI

Site

Azure

Site

CI/CD is not always remote

Some CI/CD processes are run locally on people's devices like local testing pipelines (some people say this isn't ci/cd some say it is)

CI/CD vs PaaS (platforms as a service)

PaaS services like heroku are designed to be a one-stop shop for setting up a project, they tend to be more efficient but they are by design less flexible than standard CI/CD systems

Heroku

CI/CD vs containers

Containers are essentially a way to run code inside a 'mini computer' inside a computer. Some CI/CD systems use containers, but containers are not themselves CI/CD systems necessarily

Docker

Github Actions

Github actions is a remote CI/CD system that is integrated with github directly

Github Actions Files are YML/YAML

Github actions files are written in YAML/YML

YAML Syntax

For example this presentation has a pipeline

name: GH build for ezprez

on:
  push:
    branches:
      - master
      - main

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout repo
      uses: actions/checkout@v2
    - name: Setup Python
      uses: actions/setup-python@v2
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        python -m pip install --upgrade setuptools wheel
        python -m pip install ezprez
    - name: build html files
      run: |
        mkdir ~/Downloads
        python example.py
        mv Presentation/* .
        rm -rf Presentation
    - name: Deploy Docs
      uses: peaceiris/actions-gh-pages@v3
      with:
        github_token: ${{ secrets.GITHUB_TOKEN }}
        publish_dir: .

Github Actions File structure

They go in a folder in your repository called /.github/workflows (you can have multiple pipelines)

Triggers

This configuration determines when an action pipeline runs. Note that multiple pipelines can be run off a single trigger, and pipelines finishing can trigger other pipelines to start. After a name parameter you need to add an 'on'

name: GH build for ezprez

on:
  ... # This is where you add your trigger

On push (all branches)

name: GH build for ezprez

on:
  [push] # all branches

On push (specific branches)

name: GH build for ezprez

on:
  push:
    branches: # Only rub on a push to master or main
      - master
      - main

On Workflow dispatch (No Inputs)

name: GH build for ezprez

on:
  workflow_dispatch # Manually start the pipeline

On Workflow dispatch (Inputs)

name: GH build for ezprez

on:
  workflow_dispatch: # Manually start the pipeline
    inputs:
        doc_system: # Can access value ${{ github.event.inputs.doc_system }}
          description: Either pdoc, or mkdocs # Description shows under field
          default: pdoc # Provide a default
          required: true # Make field required or not

On Workflow dispatch (Inputs)

workflow-dispatch

On Schedule/Cron

Online editor for crontab format
name: GH build for ezprez

on:
  schedule:
    # Run at 5:05 pm on mondays and wednesdays
    - cron: '5 17 * * 1,3'

Jobs

Each job is essentially a set of steps with some associated state. Each job runs in it's own 'runner' with it's own 'virtual environment'(container)

name: name of workflow

on:
  ... # This is where you add your trigger

jobs:
  job_name:
    name: You can also specify a name here
    ... # environment definition

Environment definitions

The information used to configure all steps in a job

runs-on

This defines what OS the workflow runs on

name: name of workflow

on:
  ... # This is where you add your trigger

jobs:
  job_name:
    name: You can also specify a name here
    runs-on: ubuntu-latest # can also be a matrix you iterate over

Strategy

Used to create arrays that are itterated over to run a job multiple times

name: name of workflow

on:
  ... # This is where you add your trigger

jobs:
  job_name:
    name: You can also specify a name here
    runs-on: ${{ matrix.os }} # Would run this with ubuntu-latest first, then windows-latest, then macos-latest
    strategy:
      matrix:
        python_version: ['3.7', '3.8', '3.9', '3.10'] # Can use this later on
        os: [ubuntu-latest, windows-latest, macOS-latest]

Steps

This is where actual commands are run. Note that state is typically persistant across steps in the same job. Note you can use other people's actions by passing the repo name

name: name of workflow

on:
  ... # This is where you add your trigger

jobs:
  job_name:
    ... # your job config

    steps:
    - uses: actions/checkout@v2 # minimal example
    - name: Set up Python ${{ matrix.python_version }} # Using a more complex pre-built example
      uses: actions/setup-python@v1 # https://github.com/actions/setup-python
      with:
        python-version: ${{ matrix.python_version }}
    - name: Install dependencies # Example of running raw bash commands
      run: |
        pip install .
        pip install pytest

Using with

With can be used to define settings for each step in a job. There are some standard one's we will cover but keep in mind custom actions often have custom settings you can use alongside the with section. It is essentially where you can stick any configuration for a step you need

Secrets

In github you can set a secret, which is a key-value pair that is kept completely secret (like a password). Be aware once set you can only modify or delete, you cannot read the value again

secrets

With options: token

Tokens are used to authenticate an action and are what allows the action access to make changes or even just access content that is locked behind a password/github account. They use different formats, but here is an example

name: Publish Site
on:
  ...
jobs:
  build-deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Publish Site
        uses: chabad360/hugo-gh-pages@master
        with:
          githubToken: ${{ secrets.PERSONAL_TOKEN }} # In this case this is where you set a personal access token to your account

With options: artifacts

name: Create theme release # Taken from https://github.com/QU-UP/ezcv-themes/blob/main/.github/workflows/release.yml
on:
  workflow_dispatch:
    inputs:
      version_number:
        description: The version number to release
        default: "0.1.0"
        required: true
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout repo
      uses: actions/checkout@v2

    - name: Create zip files
      ... # creates a bunch of zip files
    - name: Create release draft
      uses: ncipollo/release-action@v1
      with:
        artifacts: "*.zip" # Include zip files in releases draft
        tag: ${{ github.event.inputs.version_number }}
        draft: true
        token: ${{ secrets.GITHUB_TOKEN }}

What does this look like?

actions-status

What does this look like?

actions-overview

What does this look like?

workflow-run-summary

Github Pages

Github pages is a host for static site files. Imagine a google drive/dropbox/onedrive but for html files so that people can access the files on the web. It also provides a full http server as well

Github Pages setup

You can either have your html files in your regular source or on a gh-pages branch. You need an index.html file for either

Github Pages setup

Once you have the content where you want it, you can configure the settings for github pages from the repository (by default .github.io is the domain used, and should be the repository name)

pages-setup

Github Pages deployment

Once you have finished the configuration a github action will run to actually deploy the files to github's servers

gh-pages-deploy

Thank you

This presentation was made with the help of

WebSlides

ezprez