Runbooks

This page explains the available options to configure Runbooks with your git server.

This guide provides step-by-step instructions for configuring Runbooks using the command line. Currently, the Webapp supports configuration exclusively with SSH private keys with limited options.

To start, make sure to install the hoop command line and login to your gateway instance:

hoop config create --api-url https://<API_URL>
hoop login

Public Repositories

To configure any public repositories

hoop admin create plugin runbooks --overwrite \
	-c GIT_URL=https://github.com/your-org/your-repo

Required Configuration:

GIT_URL (required) - the GIT URL of the repository (http or ssh)

Basic Credentials

It uses username and password to clone a repository via HTTP.

hoop admin create plugin runbooks --overwrite \
	-c GIT_URL=https://github.com/your-org/your-repo \
	-c GIT_PASSWORD=your-personal-access-token

Required Configuration:

GIT_URL (required) - the HTTP GIT URL of the repository
GIT_USER (optional) - the git username, defaults to oauth2 if it’s empty
GIT_PASSWORD (required) - the password or token that has read access to the repository

GitHub users could use personal tokens

SSH Private Keys

It uses a private key to clone the repository via SSH.

hoop admin create plugin runbooks --overwrite \
    -c GIT_URL=git@github.com:your-org/your-repo.git \
    -c GIT_SSH_KEY=path:$HOME/.ssh/your_key

GitHub users could follow the Setup Deploy Keys guide to generate a key.

Required Configuration:

GIT_URL (required) - the HTTP GIT URL of the repository
GIT_SSH_KEY (required) - the private key that has read access to the repository
GIT_SSH_USER (optional) - the git username, defaults to git if it’s empty
GIT_SSH_KEYPASS (optional) - the password of the key
GIT_SSH_KNOWN_HOSTS (optional) - the path to the known hosts file to use

We recommend using the option GIT_SSH_KNOWN_HOSTS to prevent MITM when cloning repositories. It will try to obtain the known hosts issuing the command ssh-keyscan in case this configuration is not set.

Testing

To test the integration, issue the command below, it will return the last commit from the directory. When you add a runbook file it will show in the items attribute.

hoop admin get runbooks -o json
{"items":[],"commit":"<git-sha>","commit_author":"author <author@email.tld>","commit_message":"<msg>\n"}

Runbook Hooks

This feature is available in version 1.36.11+.

This functionality enables easier integration with any internal workflow by leveraging Hoop as a hook system for your process. It requires specific files in the repository that will start the execution of a runbook when a specific event occurs.

File Path	Event Trigger	Environment Variables	Description
`hoop-hooks/session-open.runbook.py`	Session Open	`HOOP_RUNBOOK_HOOK_PAYLOAD`	It trigger when a session is being opened
`hoop-hooks/session-close.runbook.py`	Session Close	`HOOP_RUNBOOK_HOOK_PAYLOAD`	It trigger when a session is being closed

The environment variables are available when the runbook script is called, it contains the payload of the event that triggered the runbook.

JSON Payload

The attribute event_session_open will be available when the session is opened
The attribute event_session_close will be available when the session is closed

{
  "id": "7b769f42-a8a2-45cb-b0a1-e2d039385365",
  "sid": "8c59a347-62bd-4db2-a456-2b25963dfebf",
  "command": ["python3"],
  "event_session_open": {
    "verb": "connect",
    "connection_name": "pg-readonly",
    "connection_type": "database",
    "connection_subtype": "postgres",
    "connection_envs": {
      "PWD": "/",
      "HOME": "/root"
    },
    "input": "SELECT * FROM customers WHERE id = 1",
    "user_email": "user@domain.tld"
  },
  "event_session_close": {
    "exit_code": 0,
    "output": ""
  }
}

Example Python Script

The example below shows how to use the payload in a python script, it will print the connection name and the session id.

# hoop-hooks/session-open.runbook.py
import json, os

payload_json = os.getenv('HOOP_RUNBOOK_HOOK_PAYLOAD')
req = json.loads(payload_json)

print('id: ', req['id'])
print('session id: ', req['sid'])

ev = req['event_session_open'] or {}
print('connection_name: {}'.format(ev['connection_name']))

How the template engine works

Runbooks use the GO text/template as the template engine. A runbook is a template to be run against a connection, the placeholders are rendered by inputs provided by an HTTP client. To define an input, the runbook must be enclosed with {{ }} and the input name must start with a dot. - Example - {{ .myinput }} The input name must comply with the regular expression \.[a-zA-Z0-9_]+

Template Specification

A client could implement input validation based on how templates are created, the specification of inputs are derived from a runbook. The template below:

SELECT name FROM customers WHERE id = '{{ .customer_id }}'

Generates the following specification:

GET /api/plugins/runbooks/connections/:dbconn/templates

{
    "items": [
        {
            "name": "team/finops/sql/fetch-customer.runbook.sql",
            "metadata": {
                "customer_id": {
                    "description": "",
                    "required": false,
                    "type": "text"
                }
            }
        }
    ],
    "commit": "b96851b9cf7065f9af0977e506cd1970c60cc87c",
    "commit_author": "Author <author@domain.tld>",
    "commit_message": "<commit-message>"
}

Supported Fields

The specifications supports the following fields:

description - the description of the input
type - the type of the input
- text
- number
- tel
- time
- date
- url
- email
- select
required - if the this field is required
default - specifies a default value for an input if it’s empty

The fields indicates how a client could create inputs to a runbook, the fields are defined as function templates in an placeholder using the pipe character |. Example:

{{ .customer_id
  | description "the id of the customer"
  | required "customer_id is required"
  | type "number" }}

Template Functions

The template engine has auxiliary functions which helps to build better and secure templates:

required "<message>" - it will return the error if the input is empty
- <message> - the message to return when the condition doesn’t match
default "<value>" - add a default value to the input if it’s empty
- <value> - the default value to set
pattern "<regexp>" - a regexp pattern to validate the input
- <regexp> - the go regular expression to validate the input
description "<message>" - used as attribute specification to client input validation
- <message> - the description of the input
type "<type>" - used as attribute specification to client input validation
- <type> - the type of the input (see supported fields for a list of types)
placeholder "<message>" - used as attribute specification to client input validation
- <message> - the description of the placeholder
options "<option>" "..." - used as attribute specification to client input validation
- "<option>" "..." - a list of strings describing each option
squote - wraps the input with single quotes: '
dquote - wraps the input with double quotes: "
quotechar "<char>" - wraps the input with <char>
- <char> - the character to wrap the input
encodeb64 - encode the input as base64
decodeb64 - decode a base64 input
asenv "<environment>" - add the input as a environment variable
- <environment> - the name of the environment variable

The functions description and type always returns the value of the last command.

Usage

Template functions may be “chained” by separating a sequence of commands with pipeline characters ‘|’. In a chained pipeline, the result of each command is passed as the last argument of the following command. The output of the final command in the pipeline is the value of the pipeline. The output of a command will be either one value or two values, the second of which has type error. If that second value is present and evaluates to non-nil, the runbook will fail to execute describing what went wrong. Examples:

Encode the myinput as base64

myvar = {{ .myinput
            | encodeb64 }}

Wrap the myinput into single quote and encode the input as base64

myvar = {{ .myinput
            | squote
            | encodeb64 }}

Gives a description to myinput, encode the value as base64 and then wrap it using the character %

myvar = {{ .myinput
            | description "this is my input"
            | quotechar "%" }}

Specify the color as input type select with options red, white and black

myvar = {{ .color
          | type "select"
          | options "red" "white" "black"}}

asenv function

The asenv function allows defining inputs and mapping then as environment variables in the connection runtime. Instead of injecting the value as an input directly to the template, it will gather the value and inject as an environment variable when executing the session. The inputs could be just defined in a comment in the template, examples:

Python Connection Runtime

# {{ .customer_id | asenv "CUSTOMER_ID" }}
# {{ .country | asenv "COUNTRY_CODE" }}
import os
print os.environ['CUSTOMER_ID']
print os.environ['COUNTRY_CODE']

Bash Connection Runtime

# {{ .deployment | asenv "DEPLOYMENT_NAME" }}
# {{ .namespace | asenv "NAMESPACE" }}
kubectl rollout restart deploy/$DEPLOYMENT_NAME -n $NAMESPACE

Linter

The command line provides a linter to validate the runbook templates before executing them. It will check if the inputs are valid and if the template is well formed.

Validating a file

hoop runbooks lint /path/to/my/query.runbook.sql

Validating content from stdin

cat /path/to/my/query.runbook.sql | hoop runbooks lint
hoop runbooks lint <<< '{{ .status | description "The status of the resource" | default "online" }}'
hoop runbooks lint <<EOF
{{ .status | description "The status of the resource" | default "online" }}
EOF

Validating parameters

hoop runbooks lint -p customer_id=123 -p country=US <<EOF
{{ .customer_id | description "Customer ID" | required "customer_id is required" }}
{{ .country | description "Country" | required "country is required" }}
{{ .status | description "The status of the resource" | default "online" }}
EOF

Error Handling

If there are any errors, it will be reported to the standard error stream (stderr) and the command will return a non-zero exit code.

cat - > /tmp/invalid-runbook.runbook.py <<EOF
{{ .customer_id
    | description "Customer ID"
    | required "customer_id is required" }}

{{ .country
    | description "Country"
    | require "country is required" }}

{{ .status
    | description "The status of the resource"
    | default "online" }}

EOF

hoop runbooks lint /tmp/invalid-runbook.runbook.py
failed parsing runbook template, reason=(...) function "require" not defined

{{ .country
    | description "Country"
    | require "country is required" }}

{{ .status
    | description "The status of the resource"

Tips

Server Side Template Injection

Templates are subject to code injection depending on the runtime that you’re using. To mitigate this issue, follows these tips:

Use the pattern function to define the format of the input, specially in sql templates.

SELECT name FROM customers WHERE id = '{{ .customer_id | pattern "^[0-9]+$" }}'

This pattern guarantees that the input will be only a number, this prevents any user to inject any sql instruction

Use the asenv function to expose inputs as environment variables if your runtime supports it
This will map the customer_id input as an environment variable avoiding bash injections with shell control operators

# {{ .customer_id | asenv "CUSTOMER_ID" }}
/path/to/my/script.sh "$CUSTOMER_ID"

The same applies to language runtimes.

# {{ .customer_id | asenv "CUSTOMER_ID" }}
import os
if __name__ == '__main__':
    print(os.environ['CUSTOMER_ID'])

Introduction

Features

Clients

Quickstart

Setup & Administration

Concepts

Integrations

Public Repositories

Basic Credentials

SSH Private Keys

Testing

Runbook Hooks

JSON Payload

Example Python Script

How the template engine works

Template Specification

Supported Fields

Template Functions

Usage

asenv function

Linter

Tips

Server Side Template Injection

Introduction

Features

Clients

Quickstart

Setup & Administration

Concepts

Integrations

​Public Repositories

​Basic Credentials

​SSH Private Keys

​Testing

​Runbook Hooks

​JSON Payload

​Example Python Script

​How the template engine works

​Template Specification

​Supported Fields

​Template Functions

​Usage

​asenv function

​Linter

​Tips

​Server Side Template Injection

Public Repositories

Basic Credentials

SSH Private Keys

Testing

Runbook Hooks

JSON Payload

Example Python Script

How the template engine works

Template Specification

Supported Fields

Template Functions

Usage

asenv function

Linter

Tips

Server Side Template Injection