01 / apply it
yardr pack apply githubyardr carries this pack in its binary; the name is enough. yardr pack check github shows what it would change first. It goes into the workspace itself, and into no repo · depot.
yardr pack apply https://yardr.dev/packs/github/From this site, as the files below.
02 / the files
Every file, as it ships.
# GitHub as a pack of kind source: the integration, imported into the yard
# and into no depot. What GitHub is: how its webhooks are verified and
# read, how the yard replies, opens a thread and sets a status in it, and
# how it reads a pull request and its CI back.
# What a stage means there is the flow pack's to say (sources.<name>:
# hears); until the yard has this source, an import of that pack skips the
# lines, and the next apply sets them.
#
# yardr pack apply github
#
# The source is named github. While its Go adapter is still compiled in,
# the declared one wins every lookup when the yard has it; the apply
# folds what the yard recorded under github-declared into github.
name: github
kind: source
setup: SETUP.md
sources:
github:
declaration: github.yaml
#!/bin/bash
# ci: the CI state of one commit of a pull request: the pull request's head
# commit, base branch and state, the checks its base requires, and the
# checks on the commit. One JSON object on stdin, one on stdout, as the
# events key of the declaration says: in {repo, number, sha}, out {head,
# base, state, required, checks}. The daemon reads it as state for a train
# whose hook's table has an event=ci line for it, and the train's brief
# carries its snapshot. What the base requires comes from its protection
# and from the rulesets that apply to it, a repository using either or
# both; either 404s when it has nothing, which contributes no check. The
# checks are the check runs and the commit statuses on the commit, each in
# the tongue the core judges: completed or not, and a completed one
# concluded success, neutral, skipped, failure, timed_out or
# startup_failure. GITHUB_TOKEN (a personal token, or what the source's
# mint_command minted, sent as a Bearer token) and YARDR_SOURCE_BASE, the
# declaration's base, are in the environment. Needs curl and jq.
set -euo pipefail
in=$(cat)
arg() { jq -r --arg k "$1" '.[$k] // ""' <<<"$in"; }
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
# api <method> <url>: the answer's body, or the failure on stderr. The
# token goes to curl as a config line on its stdin, written by the shell
# itself: an argument can be read by every process on the machine while
# curl runs.
api() {
local m u code
m=$1
u=$2
code=$(printf 'header = "Authorization: Bearer %s"\n' "$GITHUB_TOKEN" \
| curl -sS -o "$tmp/out" -w '%{http_code}' -K - -X "$m" "$u" \
-H 'Accept: application/vnd.github+json' \
-H 'X-GitHub-Api-Version: 2022-11-28') || {
echo "could not reach $u (curl exit $?)" >&2
return 1
}
if [ "${code:0:1}" != 2 ]; then
echo "$u: HTTP $code: $(jq -r '.message // "no answer"' "$tmp/out" 2>/dev/null || head -c 200 "$tmp/out")" >&2
return 1
fi
cat "$tmp/out"
}
# api_maybe <method> <url>: api whose 404 is no answer, for the branch
# policy endpoints: GitHub 404s where a branch has no protection and where
# no ruleset applies.
api_maybe() {
local m u code
m=$1
u=$2
code=$(printf 'header = "Authorization: Bearer %s"\n' "$GITHUB_TOKEN" \
| curl -sS -o "$tmp/out" -w '%{http_code}' -K - -X "$m" "$u" \
-H 'Accept: application/vnd.github+json' \
-H 'X-GitHub-Api-Version: 2022-11-28') || {
echo "could not reach $u (curl exit $?)" >&2
return 1
}
[ "$code" = 404 ] && return 0
if [ "${code:0:1}" != 2 ]; then
echo "$u: HTTP $code: $(jq -r '.message // "no answer"' "$tmp/out" 2>/dev/null || head -c 200 "$tmp/out")" >&2
return 1
fi
cat "$tmp/out"
}
# pages <url with ?> <key>: every page's <key> as one array.
pages() {
local u k page all cs n
u=$1
k=$2
page=1
all='[]'
while :; do
cs=$(api GET "$u&per_page=100&page=$page") || return 1
n=$(jq --arg k "$k" '.[$k] | length' <<<"$cs") || return 1
all=$(jq -c --argjson a "$all" --arg k "$k" '$a + .[$k]' <<<"$cs") || return 1
[ "$n" -lt 100 ] && break
page=$((page + 1))
done
printf '%s' "$all"
}
# A repository as {repo} gives it: two names, nothing that would reach
# another endpoint.
valid_repo() { [[ $1 =~ ^[A-Za-z0-9][A-Za-z0-9-]*/[A-Za-z0-9._-]+$ ]]; }
repo=$(arg repo)
n=$(arg number)
sha=$(arg sha)
valid_repo "$repo" || {
echo "{repo} $repo: want owner/repo" >&2
exit 1
}
[[ $n =~ ^[0-9]+$ ]] || {
echo "{number} $n: want the pull request's number" >&2
exit 1
}
[[ $sha =~ ^[0-9a-fA-F]+$ ]] || {
echo "{sha} $sha: want the commit's SHA" >&2
exit 1
}
pr=$(api GET "$YARDR_SOURCE_BASE/repos/$repo/pulls/$n") || exit 1
head=$(jq -r '.head.sha // ""' <<<"$pr")
base=$(jq -r '.base.ref // ""' <<<"$pr")
state=$(jq -r '.state // ""' <<<"$pr")
[ -n "$head" ] && [ -n "$base" ] || {
echo "the answer holds no head or base" >&2
exit 1
}
# A base with a slash in it goes escaped: GitHub names the branch in the path.
esc=$(jq -n -r --arg b "$base" '$b|@uri')
prot=$(api_maybe GET "$YARDR_SOURCE_BASE/repos/$repo/branches/$esc") || exit 1
rules=$(api_maybe GET "$YARDR_SOURCE_BASE/repos/$repo/rules/branches/$esc") || exit 1
required=$(jq -n -c --argjson p "${prot:-null}" --argjson r "${rules:-null}" '
([($p | .protection.required_status_checks.contexts[]?)] +
[$r[]? | select(.type == "required_status_checks") | .parameters.required_status_checks[].context])
| reduce .[] as $x ([]; if index($x) then . else . + [$x] end)') || exit 1
# GitHub lists the latest check run of each name, and the combined status
# the latest status of each context.
runs=$(pages "$YARDR_SOURCE_BASE/repos/$repo/commits/$sha/check-runs?" check_runs) || exit 1
statuses=$(pages "$YARDR_SOURCE_BASE/repos/$repo/commits/$sha/status?" statuses) || exit 1
jq -n -c --arg head "$head" --arg base "$base" --arg state "$state" \
--argjson required "$required" --argjson runs "$runs" --argjson statuses "$statuses" '
{head: $head, base: $base, state: $state, required: $required,
checks: ([$runs[] | {name, status, conclusion, url: .html_url}] +
[$statuses[] | {name: .context, url: .target_url,
status: (if .state == "pending" then "pending" else "completed" end),
conclusion: (if .state == "success" then "success"
elif .state == "pending" then ""
else "failure" end)}])}'
#!/bin/bash
# git-credential: the credential helper a depot's git runs for github.com.
# The import copies it beside the declaration, under
# <home>/sources/github/github/git-credential, and a depot whose origin is
# on github.com points its own git at the copy:
# git -C <depot> config credential."https://github.com".helper <the copy>
# (in that depot's own .git/config alone, never --global: packs/github/SETUP.md).
# Git runs it with get, store or erase and its request (protocol, host, ...)
# on stdin. For protocol https and host github.com it answers username
# x-access-token and the GITHUB_TOKEN read from the github source's env
# file at the moment git asks, so a rotated token needs no re-turn-on;
# for any other host or protocol it answers nothing, and store and erase
# do nothing. The token is read as the core reads it (last GITHUB_TOKEN
# line wins, surrounding whitespace off, an empty value as good as unset)
# and is printed only to git on stdout: never in an argument, a file, a
# log or an error. Needs nothing but bash.
set -euo pipefail
# trim prints its argument without surrounding whitespace, as the core
# trims a credential's value before it is used.
trim() {
local v=$1
v=${v#"${v%%[![:space:]]*}"}
v=${v%"${v##*[![:space:]]}"}
printf '%s' "$v"
}
# yard_home is the yard's home: three directories above this helper as the
# import copied it (<home>/sources/github/github/git-credential), else
# YARDR_HOME, else the default home. The copy knows where it stands, so
# the helper needs no environment of its own.
yard_home() {
local self=$1 link dir
case $self in
*/*) ;;
*) self=$(command -v "$self" || printf '%s' "$self") ;;
esac
while [ -L "$self" ]; do
link=$(readlink "$self")
case $link in
/*) self=$link ;;
*) self=$(dirname "$self")/$link ;;
esac
done
case $self in
/*) ;;
*) self=$PWD/$self ;;
esac
case $self in
*/sources/github/github/git-credential)
dir=${self%/sources/github/github/git-credential}
[ -n "$dir" ] && printf '%s' "$dir" && return 0
;;
esac
printf '%s' "${YARDR_HOME:-$HOME/.yardr}"
}
op=${1:-}
case $op in
store | erase) exit 0 ;;
get) ;;
*)
echo "usage: git-credential {get|store|erase} (run by git, not by hand)" >&2
exit 2
;;
esac
# Git's request: key=value lines to a blank line. Only protocol and host
# matter here; anything else is git's own.
protocol=""
host=""
while IFS= read -r line || [ -n "$line" ]; do
line=${line%$'\r'}
[ -n "$line" ] || break
case $line in
protocol=*) protocol=${line#protocol=} ;;
host=*) host=${line#host=} ;;
esac
done
if [ "$protocol" != "https" ] || [ "$host" != "github.com" ]; then
exit 0
fi
envfile=$(yard_home "$0")/sources/github.env
raw=""
if [ -f "$envfile" ]; then
n=0
while IFS= read -r line || [ -n "$line" ]; do
n=$((n + 1))
line=${line%$'\r'}
line=${line#"${line%%[![:space:]]*}"}
case $line in
'' | '#'*) continue ;;
esac
case $line in
*=*) k=${line%%=*} v=${line#*=} ;;
*)
echo "git-credential: $envfile line $n: want KEY=VALUE" >&2
exit 0
;;
esac
k=$(trim "$k")
case $k in
'' | *[!A-Za-z0-9_]* | [0-9]*)
echo "git-credential: $envfile line $n: want KEY=VALUE" >&2
exit 0
;;
esac
if [ "$k" = "GITHUB_TOKEN" ]; then
raw=$v
fi
done <"$envfile"
fi
token=$(trim "$raw")
if [ -z "$token" ]; then
echo "git-credential: GITHUB_TOKEN is not set in $envfile" >&2
exit 0
fi
printf 'username=%s\npassword=%s\n' "x-access-token" "$token"
#!/bin/sh
# A verified GitHub webhook to the yard's envelope, as GitHub{} makes it:
# stdin {"headers":{…},"body":{…}}, stdout the envelope. Needs jq.
# internal/hooks feeds it the GitHub bodies its tests already have and
# compares the envelopes with GitHub{}.Normalize's. What differs on
# purpose: the envelope's train, which the core derives from head_branch
# after this returns, so this leaves it out.
exec jq -c '
def first_of(f): [f | select(. != null and . != "")] | .[0] // "";
# jq 1.6 and 1.7.1 encode some punctuation differently; pin it here so a
# change to the installed jq does not make a second thread URL for a branch.
def uri_segment: @uri
| gsub("!"; "%21")
| gsub("\\*"; "%2A")
| gsub("\u0027"; "%27")
| gsub("\\("; "%28")
| gsub("\\)"; "%29");
# trunc is hooks.truncate: whitespace collapsed, at most 500 runes. The jq
# POSIX space class includes the Unicode spaces split by Go strings.Fields.
def trunc: (gsub("^[[:space:]]+|[[:space:]]+$"; "") | gsub("[[:space:]]+"; " ")) | if length > 500 then .[0:499] + "…" else . end;
def ordered: reduce .[] as $x ([]; if index($x) then . else . + [$x] end);
# mentions is hooks.githubMentions: the lowercase logins a body mentions,
# once each, in order. Team mentions (@org/team) are left out: a team is
# not a crew member. An @login in code is text, not a mention.
def mentions($md):
((($md // "") | gsub("(?s)```.*?(```|$)|`[^`\n]*`"; " "))
| [scan("(?:^|[^A-Za-z0-9_@./-])@([A-Za-z0-9](?:-?[A-Za-z0-9])*)(/?)")
| select(.[1] != "/") | .[0] | ascii_downcase | select(length <= 39)]
| ordered);
def header($name): [.headers | to_entries[] | select(.key | ascii_downcase == $name) | .value[]] | .[0] // "";
def field($o; $k; $v): if ($v // "") == "" then $o else $o + {($k): [$v]} end;
(.body | if . == null then {} elif type == "object" then . else error("github payload is not JSON") end) as $p
| (header("x-github-event") | if . == "" then error("github delivery has no X-GitHub-Event") else . end) as $event
| (($p.repository // {}) | (.full_name // "")) as $repo
| ((if $p.sender == null then {} else {id: (($p.sender.login // "") | ascii_downcase), name: ($p.sender.login // "")} end)) as $actor
# What acted, as GitHub types it: user, bot or organization. A login
# ending in [bot] is a bot: GitHub Apps and Actions act as one.
| (if $p.sender == null then ""
else (((($p.sender.type // "") | ascii_downcase)) as $kind
| (if (($p.sender.login // "") | endswith("[bot]")) then "bot" elif $kind != "" then $kind else "user" end)) end) as $sender
| ({} | field(.; "event"; $event) | field(.; "action"; ($p.action // "")) | field(.; "repo"; $repo) | field(.; "sender"; $sender)) as $base
| {source: "github", type: $event, action: ($p.action // ""), actor: $actor, thread: {}, mentions: []}
| if $event == "pull_request" or $event == "issues" or $event == "issue_comment" or $event == "pull_request_review" then
((if $event == "pull_request" or $event == "pull_request_review" then $p.pull_request else $p.issue end) // null) as $item
| ((if $event == "issue_comment" then $p.comment elif $event == "pull_request_review" then $p.review else null end) // null) as $said
| (if $event == "issue_comment" then ($p.comment // {})
elif $event == "pull_request_review" then ($p.review // {}) else {} end) as $carries
| (($carries.body // "") | if $event == "pull_request" or $event == "issues" then ($item.body // "") else . end) as $text
| . + {url: (if $event == "pull_request" or $event == "issues"
then (if $item == null then "" else ($item.html_url // "") end)
else ($carries.html_url // "") end)}
# An edit re-sends the comment whole, so its mentions are counted
# against the last one seen, and a deletion forgets it. The
# description is re-sent whole with every action, so its mentions
# are counted against the last one seen.
| (if ($event == "issue_comment" and ($carries.id // 0) != 0) then
. + {message: (($carries.id | tostring)), entity: ("issuecomment-" + ($carries.id | tostring))}
elif ($event == "pull_request_review" and ($carries.id // 0) != 0) then
. + {entity: ("pullrequestreview-" + ($carries.id | tostring))}
elif (($event == "pull_request" or $event == "issues") and $item != null) then
. + {entity: ($repo + "#" + (($item.number // 0) | tostring))}
else . end)
| ($base | (if $event == "pull_request_review" then field(.; "state"; ((($carries.state // "") | ascii_downcase))) else . end)) as $f
| (if $item == null or ($item.number // 0) <= 0 then . + {fields: $f}
else
($f | field(.; "number"; (($item.number | tostring)))
| (if .state == null then field(.; "state"; ($item.state // "")) else . end)
| (if $item.merged == null then . else field(.; "merged"; ($item.merged | tostring)) end)
| field(.; "head_branch"; (($item.head // {}).ref // ""))
| field(.; "head_sha"; (($item.head // {}).sha // ""))
| field(.; "head_repo"; ((($item.head // {}).repo // {}).full_name // ""))
| field(.; "base"; (($item.base // {}).ref // ""))
# The size of the change, which the events of a pull request
# carry and those of an issue do not: the card a maintainer
# decides on shows it beside the head (a field is skipped
# when empty, and a count of 0 is kept). No apostrophe in a
# comment here: the program is one single-quoted word.
| field(.; "additions"; (if $item.additions == null then "" else ($item.additions | tostring) end))
| field(.; "deletions"; (if $item.deletions == null then "" else ($item.deletions | tostring) end))
| field(.; "changed_files"; (if $item.changed_files == null then "" else ($item.changed_files | tostring) end))
| ([$item.labels[]?.name | select(. != null and . != "")] as $labels
| if ($labels | length) == 0 then . else . + {label: $labels} end)
# What the writer is to the repository, which is whom a
# filter can trust (association=member): the writer of the
# comment or review the delivery carries, else the author
# of the issue or pull request. outsider is yes when that is
# given and is none of owner, member and collaborator: the
# yard prints such a delivery as information, not
# instructions, and leaves its text out of the notes it
# writes.
| (((if $said == null then ($item.author_association // "") else ($said.author_association // "") end) | ascii_downcase) as $writer
| field(.; "association"; $writer)
| field(.; "outsider"; (if $writer == "" or $writer == "owner" or $writer == "member" or $writer == "collaborator" then "" else "yes" end)))) as $f2
| . + {thread: {id: ($repo + "#" + (($item.number | tostring))),
title: ($item.title // ""), url: ($item.html_url // "")},
assignee: ((($item.assignee // {}).login // "") | ascii_downcase),
text: ($text | trunc), mentions: mentions($text), fields: $f2}
end)
elif $event == "workflow_run" or $event == "check_suite" then
((if $event == "workflow_run" then $p.workflow_run else $p.check_suite end) // null) as $s
| (if $s == null then . + {fields: $base}
else
(first_of($s.name, $s.display_title) | if . == "" then (($s.app // {}).name // "") else . end) as $name
| ([$name, first_of($s.conclusion, $s.status)] | map(select(. != ""))) as $words
| (($words + (if ($s.head_branch // "") == "" then [] else ["on", $s.head_branch] end)) | join(" ")) as $words
| . + {text: ($words | trunc),
url: (first_of($s.html_url,
(if $p.repository == null or ($s.head_sha // "") == "" then ""
else (($p.repository.html_url // "") + "/commit/" + $s.head_sha) end))),
fields: ($base | field(.; "conclusion"; ($s.conclusion // ""))
| field(.; "head_branch"; ($s.head_branch // ""))
| field(.; "head_sha"; ($s.head_sha // ""))
| field(.; "head_repo"; ((($s.head_repository // {}).full_name // "")))
| field(.; "workflow"; $name))}
# A run for a pull request'"'"'s branch belongs to its thread, so
# whoever takes part in the pull request hears how CI went.
| (if (($s.pull_requests // []) | length) > 0 and ((($s.pull_requests[0] // {}).number // 0) > 0) then
((($s.pull_requests[0].number | tostring))) as $n
| . + {thread: {id: ($repo + "#" + $n)}, fields: (.fields + {number: [$n]})}
# A run for no pull request belongs to its branch'"'"'s: the
# thread <repo>@<branch>, at the same URL for every run of it, so
# that a bead made for a failed run is found by the next. GitHub
# lists no pull request for a fork'"'"'s run, whose branch may be
# named as this repository'"'"'s: it is on the fork'"'"'s, so it has
# no thread here.
elif ($s.head_branch // "") != ""
and (($s.head_repository // {}).full_name // $repo) == $repo then
first_of(($p.repository // {}).html_url, "https://github.com/" + $repo) as $site
| . + {thread: {id: ($repo + "@" + $s.head_branch),
title: ("CI on " + $repo + "@" + $s.head_branch),
url: ($site + "/tree/" + ($s.head_branch | split("/") | map(uri_segment) | join("/")))}}
else . end)
end)
else . + {fields: $base}
end
'
#!/bin/bash
# pr: the pull request live: its head commit, base branch and state. One
# JSON object on stdin, one on stdout, as the events key of the declaration
# says: in {repo, number, url}, out {head, base, state}. The daemon reads
# it when no event has told the pull request's head. GITHUB_TOKEN (a
# personal token, or what the source's mint_command minted, sent as a
# Bearer token) and YARDR_SOURCE_BASE, the declaration's base, are in the
# environment. Needs curl and jq.
set -euo pipefail
in=$(cat)
arg() { jq -r --arg k "$1" '.[$k] // ""' <<<"$in"; }
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
# api <method> <url>: the answer's body, or the failure on stderr. The
# token goes to curl as a config line on its stdin, written by the shell
# itself: an argument can be read by every process on the machine while
# curl runs.
api() {
local m u code
m=$1
u=$2
code=$(printf 'header = "Authorization: Bearer %s"\n' "$GITHUB_TOKEN" \
| curl -sS -o "$tmp/out" -w '%{http_code}' -K - -X "$m" "$u" \
-H 'Accept: application/vnd.github+json' \
-H 'X-GitHub-Api-Version: 2022-11-28') || {
echo "could not reach $u (curl exit $?)" >&2
return 1
}
if [ "${code:0:1}" != 2 ]; then
echo "$u: HTTP $code: $(jq -r '.message // "no answer"' "$tmp/out" 2>/dev/null || head -c 200 "$tmp/out")" >&2
return 1
fi
cat "$tmp/out"
}
# A repository as {repo} gives it: two names, nothing that would reach
# another endpoint.
valid_repo() { [[ $1 =~ ^[A-Za-z0-9][A-Za-z0-9-]*/[A-Za-z0-9._-]+$ ]]; }
repo=$(arg repo)
n=$(arg number)
if [ -z "$repo" ] || [ -z "$n" ]; then
# By URL alone: https://github.com/<owner>/<repo>/pull/<n>.
url=$(arg url)
if [[ $url =~ ^https?://[^/]+/([A-Za-z0-9][A-Za-z0-9-]*/[A-Za-z0-9._-]+)/pull/([0-9]+)(/.*)?$ ]]; then
repo=${BASH_REMATCH[1]}
n=${BASH_REMATCH[2]}
else
echo "the pull request is named by neither {repo, number} nor {url}" >&2
exit 1
fi
fi
valid_repo "$repo" || {
echo "{repo} $repo: want owner/repo" >&2
exit 1
}
[[ $n =~ ^[0-9]+$ ]] || {
echo "{number} $n: want the pull request's number" >&2
exit 1
}
api GET "$YARDR_SOURCE_BASE/repos/$repo/pulls/$n" |
jq -c '{head: (.head.sha // ""), base: (.base.ref // ""), state: (.state // "")}
| if .head == "" or .base == "" then error("the answer holds no head or base") else . end'
#!/bin/bash
# thread: what the yard does in a GitHub thread: reply, posted, read, open,
# opened, state. One JSON object on stdin, one on stdout, as the thread key
# of the declaration says. GITHUB_TOKEN (a personal token, or what the
# source's mint_command minted, sent as a Bearer token) and YARDR_SOURCE_BASE,
# the declaration's base, are in the environment. Needs curl and jq.
set -euo pipefail
in=$(cat)
arg() { jq -r --arg k "$1" '.[$k] // ""' <<<"$in"; }
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
# api <method> <url> [data]: the answer's body, or the failure on stderr.
# The token goes to curl as a config line on its stdin, written by the
# shell itself: an argument can be read by every process on the machine
# while curl runs.
api() {
local m u code
m=$1
u=$2
if [ $# -ge 3 ]; then
code=$(printf 'header = "Authorization: Bearer %s"\n' "$GITHUB_TOKEN" \
| curl -sS -o "$tmp/out" -w '%{http_code}' -K - -X "$m" "$u" \
-H 'Accept: application/vnd.github+json' \
-H 'X-GitHub-Api-Version: 2022-11-28' -H 'Content-Type: application/json' -d "$3") || {
echo "could not reach $u (curl exit $?)" >&2
return 1
}
else
code=$(printf 'header = "Authorization: Bearer %s"\n' "$GITHUB_TOKEN" \
| curl -sS -o "$tmp/out" -w '%{http_code}' -K - -X "$m" "$u" \
-H 'Accept: application/vnd.github+json' \
-H 'X-GitHub-Api-Version: 2022-11-28') || {
echo "could not reach $u (curl exit $?)" >&2
return 1
}
fi
if [ "${code:0:1}" != 2 ]; then
echo "$u: HTTP $code: $(jq -r '.message // "no answer"' "$tmp/out" 2>/dev/null || head -c 200 "$tmp/out")" >&2
return 1
fi
cat "$tmp/out"
}
# pages <url with ?>: every page of a list endpoint as one array, oldest
# first as GitHub lists them.
pages() {
local u page all cs n
u=$1
page=1
all='[]'
while :; do
cs=$(api GET "$u&per_page=100&page=$page") || return 1
n=$(jq 'length' <<<"$cs") || return 1
all=$(jq -c --argjson a "$all" '$a + .' <<<"$cs") || return 1
[ "$n" -lt 100 ] && break
page=$((page + 1))
done
printf '%s' "$all"
}
# A repository as --in gives it: two names, nothing that would reach
# another endpoint.
valid_repo() { [[ $1 =~ ^[A-Za-z0-9][A-Za-z0-9-]*/[A-Za-z0-9._-]+$ ]]; }
# split <thread>: the repository and the number owner/repo#number names, or
# why it is no thread.
split() {
repo=${1%%#*}
n=${1##*#}
if [[ $1 != *#* ]] || ! valid_repo "$repo" || [[ ! $n =~ ^[0-9]+$ ]]; then
echo "thread $1: want owner/repo#number" >&2
return 1
fi
}
case "$1" in
reply) # GitHub takes no client-chosen id, so the key goes into the body
# as a marker that renders as nothing. Its comments are not threaded,
# so the parent is left out: every reply is top-level.
thread=$(arg thread)
body=$(arg body)
key=$(arg key)
split "$thread" || exit 1
[ -n "$key" ] && body="$body
<!-- yardr-reply $key -->"
api POST "$YARDR_SOURCE_BASE/repos/$repo/issues/$n/comments" \
"$(jq -n --arg b "$body" '{body: $b}')" |
jq -c '{id: (((.id // "") | tostring)), url: (.html_url // "")}' ;;
posted) # {} for none: a retry asks before it posts, so nothing is posted twice
thread=$(arg thread)
key=$(arg key)
split "$thread" || exit 1
pages "$YARDR_SOURCE_BASE/repos/$repo/issues/$n/comments?" |
jq -c --arg m "<!-- yardr-reply $key -->" \
'[.[] | select((.body // "") | contains($m))][0]
| if . == null then {} else {id: (((.id // "") | tostring)), url: (.html_url // "")} end' ;;
read) # every page, oldest first as GitHub lists them
thread=$(arg thread)
split "$thread" || exit 1
pages "$YARDR_SOURCE_BASE/repos/$repo/issues/$n/comments?" |
jq -c '{messages: [.[] | {id: (((.id // "") | tostring)), parent: "",
author: {id: (((.user.login // "") | ascii_downcase)), name: (.user.login // "")},
at: (.created_at // "0001-01-01T00:00:00Z"), body: (.body // ""), url: (.html_url // "")}]}' ;;
open) # GitHub takes no client-chosen id, so the key goes into the body
# as a marker. The thread is named by the repository's full name as
# GitHub spells it, which may differ in case from what was typed. An
# assignee is a login.
into=$(arg in)
title=$(arg title)
body=$(arg body)
key=$(arg key)
assignee=$(arg assignee)
valid_repo "$into" || {
echo "--in $into: want owner/repo" >&2
exit 1
}
if [ -n "$key" ]; then
marker="<!-- yardr-thread $key -->"
if [ -z "${body//[[:space:]]/}" ]; then
body=$marker
else
body=$(printf '%s' "$body" | sed -e 's/[[:space:]]*$//')
body="$body
$marker"
fi
fi
api POST "$YARDR_SOURCE_BASE/repos/$into/issues" \
"$(jq -n --arg t "$title" --arg b "$body" --arg a "$assignee" \
'{title: $t, body: $b} + (if $a != "" then {assignees: [$a]} else {} end)')" |
jq -c --arg in "$into" --arg a "$assignee" '
if (.number // 0) <= 0 then error("the answer names no issue") else . end
| if $a != "" and ([.assignees[]?.login? | ascii_downcase] | index($a | ascii_downcase)) == null
then error("issue was created, but GitHub did not assign \($a); the token may need push access to set assignees")
else . end
| ((.repository_url // "" | split("/repos/") | .[1:] | join("/repos/"))) as $tail
| (if ($tail | split("/") | length) == 2 then $tail else $in end) as $repo
| {id: "\($repo)#\(.number)", title: (.title // ""), url: (.html_url // "")}' ;;
opened) # posted's sibling for an open whose answer was lost: the key is
# carried in the body's marker. Newest first, bounded by since where
# the attempt was put on record (a minute earlier, for clocks that
# disagree); issues the bound leaves out cannot be it.
into=$(arg in)
key=$(arg key)
since=$(arg since)
valid_repo "$into" || {
echo "--in $into: want owner/repo" >&2
exit 1
}
q="state=all&sort=created&direction=desc"
if [ -n "$since" ]; then
since=$(TZ=UTC jq -n -r --arg s "$since" \
'try ((($s | strptime("%Y-%m-%dT%H:%M:%SZ") | mktime) - 60 | strftime("%Y-%m-%dT%H:%M:%SZ"))) catch $s')
q="$q&since=$since"
fi
pages "$YARDR_SOURCE_BASE/repos/$into/issues?$q" |
jq -c --arg m "<!-- yardr-thread $key -->" --arg in "$into" '
[.[] | select((.body // "") | contains($m))][0]
| if . == null then {}
else ((.repository_url // "" | split("/repos/") | .[1:] | join("/repos/"))) as $tail
| (if ($tail | split("/") | length) == 2 then $tail else $in end) as $repo
| {id: "\($repo)#\(.number)", title: (.title // ""), url: (.html_url // "")}
end' ;;
state) # started is open, done is closed as completed. An issue has no
# review status, and a pull request is refused one: a state would close
# it unmerged. A label:<name> adds the label, to a pull request too,
# and nothing is ever removed.
thread=$(arg thread)
state=$(arg state)
forward=$(arg forward)
split "$thread" || exit 1
issue=$(api GET "$YARDR_SOURCE_BASE/repos/$repo/issues/$n") || exit 1
# standing: open stands at started; closed as completed at done, closed
# as anything else at other, past every status.
standing=$(jq -r '
if .state != "closed" then "started"
elif .state_reason == null or .state_reason == "" or .state_reason == "completed" then "done"
else "other" end' <<<"$issue")
status=$(jq -r '
if .state == "closed" and .state_reason != null and .state_reason != "" and .state_reason != "completed"
then "closed (" + (.state_reason | gsub("_"; " ")) + ")"
else (.state // "") end' <<<"$issue")
state_now=$(jq -r '.state // ""' <<<"$issue")
if [ "$state_now" != open ] && [ "$state_now" != closed ]; then
echo "the answer names no state for $thread" >&2
exit 1
fi
case "$state" in
label:*)
name=${state#label:}
[ -n "$name" ] || {
echo "\"$state\" is no label: want label:<name>" >&2
exit 1
}
# A label has no order to stand in: it is added wherever the issue
# or pull request stands, unless it is there.
has=$(jq -r --arg l "$name" '[.labels[]?.name | select(. != null)] | map(select(ascii_downcase == ($l | ascii_downcase))) | .[0] // ""' <<<"$issue")
if [ -n "$has" ]; then
jq -n -c --arg s "labeled $has" '{status: $s, moved: false}'
exit 0
fi
api POST "$YARDR_SOURCE_BASE/repos/$repo/issues/$n/labels" \
"$(jq -n --arg l "$name" '{labels: [$l]}')" >/dev/null || exit 1
jq -n -c --arg s "labeled $name" '{status: $s, moved: true}'
exit 0 ;;
started) want=open ;;
done) want=closed ;;
review)
echo "an issue has no review status: the states it has are started and done" >&2
exit 1
;;
*)
echo "\"$state\" is no state here: want started, done" >&2
exit 1
;;
esac
# Exit 3 tells the yard the refusal is for good: a stage map's move is
# skipped at once, where any other exit is tried again until the yard
# gives up.
if [ "$(jq -r '.pull_request != null' <<<"$issue")" = true ]; then
echo "$thread is a pull request: the status of issues only is set" >&2
exit 3
fi
if [ "$forward" = true ]; then
# A stage map's move, which never moves backwards: an open issue
# stands at started, which is all that can be said of it, and a
# closed one is never reopened.
if [ "$state" = started ] || [ "$standing" = done ] || [ "$standing" = other ]; then
jq -n -c --arg s "$status" '{status: $s, moved: false}'
exit 0
fi
elif [ "$state_now" = "$want" ]; then
# thread state moves exactly: only a thread already in the status stays.
jq -n -c --arg s "$status" '{status: $s, moved: false}'
exit 0
fi
if [ "$want" = closed ]; then
data='{"state":"closed","state_reason":"completed"}'
else
data='{"state":"open"}'
fi
api PATCH "$YARDR_SOURCE_BASE/repos/$repo/issues/$n" "$data" >/dev/null || exit 1
jq -n -c --arg s "$want" --arg was "$status" '{status: $s, moved: true, was: $was}' ;;
*) echo "unknown: $1" >&2; exit 2 ;;
esac
# GitHub as a declared source, as GitHub{} has it: how the core verifies a
# GitHub delivery, and the command that turns a verified body into the
# envelope; what the yard does in a GitHub thread, by one command per
# thing; GitHub's issue and pull request URLs as a bead's link; and what
# git uses to reach its repositories. internal/hooks holds it to golden
# files: the envelopes for the bodies its tests have, and what the
# stand-in receives for reply, posted, read, open and state and for the
# pr and ci reads.
#
# The name is github. While its Go adapter is still compiled in, the
# declared one wins every lookup when the yard has it.
#
# A command is a list, run without a shell. A path that does not start
# with / is a file beside this one, which `yardr source add` copies into
# the yard.
name: github
about: GitHub issues, pull requests and CI. Hears its webhooks and speaks in its threads.
# The name of the variable, never the value. A command gets it in its
# environment under this name, and gets no other variable of the yard.
# hook.normalize is handed no credential at all. Where the env sets the
# mint's two names, the core runs the source's mint_command and hands what
# it mints under GITHUB_TOKEN instead; one of them alone is refused.
credential:
env: GITHUB_TOKEN
header: "Authorization: Bearer {credential}"
mint: [GITHUB_APP_ID, GITHUB_APP_PRIVATE_KEY_FILE]
# ---- what it sends us ----------------------------------------------------
hook:
# Adapter.Verify stays data, checked by the core: the signing secret is
# handed to no command, and an unverified body reaches none. GitHub signs
# no timestamp, so a replay cannot be told by age; the signed body drops
# it instead (bodies).
verify:
scheme: hmac-sha256
signed: body
signature: { header: X-Hub-Signature-256, encoding: hex, prefix: "sha256=" }
delivery: { header: X-GitHub-Delivery }
# Per hook, not per source: GitHub sends an event's body to every webhook
# on the repository, and each hook records its own.
bodies: unique-per-hook
# GitHub's deliveries follow commits, not a rhythm: an idle hour after
# the last push is no break, so the quiet check does not time them.
untimed: true
# Adapter.Normalize. stdin {"headers":{…},"body":{…}}, stdout the
# envelope. The core sets source and delivery itself, and derives the
# envelope's train from its head_branch, which the command leaves out.
normalize: [github/normalize]
# The keys its events' fields may carry: a subscription's --where may
# name them beside type, action, team, project, repo and label.
# outsider is yes on a delivery whose text was written by someone who is
# none of the repository's own people (association owner, member or
# collaborator): the yard marks its line as information, not instructions.
keys: [event, number, state, merged, conclusion, workflow, head_branch, head_sha, head_repo, base, train, sender, association, outsider, additions, deletions, changed_files]
# ---- what it sends about pull requests and CI ------------------------------
# Which envelopes are CI runs, which tell the pull request's head commit,
# and the commands that read the pull request and its CI back. The daemon
# judges a run against the head the last head_from event carried, or the
# head the pr command reads live, and reads CI as state for a train whose
# hook's table has an event=ci line for it; a train's brief carries the ci
# command's snapshot. Both run as a thread command does, with the
# credential under GITHUB_TOKEN. The shapes are in docs/integrations.md.
events:
runs: [workflow_run, check_suite] # the types that are CI runs, whatever became of them
head_from: [pull_request, pull_request_review] # the types that tell the pull request's head
pr: [github/pr] # in {repo, number, url} out {head, base, state}
ci: [github/ci] # in {repo, number, sha} out {head, base, state, required, checks}
# ---- what the yard does in its threads ------------------------------------
# One command per thing, each handed its object on stdin and the
# credential under GITHUB_TOKEN, with YARDR_SOURCE_BASE from base below.
# github/thread is the script: reply comments on the issue or pull request
# with the key in a marker comment, posted finds what a key left ({} for
# none), read lists the thread's comments oldest first, open starts an
# issue in the repository in, opened finds what an open left ({} for none:
# posted's sibling for open), and state moves to what the word means:
# exactly, or never backwards with forward (a stage map's move). The shapes
# are in docs/integrations.md.
thread:
reply: [github/thread, reply] # in {thread, parent, body, key} out {id, url}
posted: [github/thread, posted] # in {thread, key} out {id, url} or {}
read: [github/thread, read] # in {thread} out {messages: [...]}
place: repository # what open's in names
open: [github/thread, open] # in {in, title, body, key, assignee?} out {id, title, url}
assignee: key # the assignee open takes: a login
opened: [github/thread, opened] # in {in, key, since?} out {id, title, url} or {}
state: [github/thread, state] # in {thread, state, forward?} out {status, moved, was?}
states: [started, done] # the words it has a status for: an issue is open or closed
labels: true # and a label:<name> adds the label, to any thread
# ---- who a crew member is in it -------------------------------------------
# GitHub logins are the same in any case, and its deliveries carry them
# lowercase: a key is kept as they carry it, however it was typed.
identity:
fold: lower
# ---- its issues as a bead's link ------------------------------------------
# The host its issue URLs are on, what reads an issue from a URL's path,
# and over the groups: the issue's one fixed form, and the id a thread
# command takes for it. Groups fold before the templates expand, so one
# spelling names the issue however it was typed. A pull request is the
# second shape, over the same groups in the same order, so one names the
# other; repos is what a repository's path matches, as a depot's origin
# remote carries it past the host.
link:
host: github.com
issue: '^/([A-Za-z0-9](?:-?[A-Za-z0-9])*)/([A-Za-z0-9._-]+)/issues/([1-9][0-9]*)(?:/.*)?$'
fixed: "https://github.com/{1}/{2}/issues/{3}"
thread: "{1}/{2}#{3}"
fold: lower
pull:
issue: '^/([A-Za-z0-9](?:-?[A-Za-z0-9])*)/([A-Za-z0-9._-]+)/pull/([1-9][0-9]*)(?:/.*)?$'
fixed: "https://github.com/{1}/{2}/pull/{3}"
thread: "{1}/{2}#{3}"
repos: '[A-Za-z0-9][A-Za-z0-9-]*/[A-Za-z0-9._-]+'
# ---- git, in this source's repositories -----------------------------------
# What git uses to reach them: the credential helper git runs for
# github.com. A depot whose origin is there points its own git at the copy
# (SETUP.md), which answers git's protocol from the source's env file at
# the moment git asks; the yard never runs it. For any other host or
# protocol it answers nothing, and store and erase do nothing.
git:
credential: [github/git-credential]
# ---- (today) what an agent may ask of it, through `yardr call` ------------
# Who the credential acts as, and nothing else: the file's calls are what
# keep its base and credential (a file with none has neither), and one read
# call is what that takes. It names no check call on purpose: no one
# endpoint answers for a static token and for the JWT a scope-free mint
# returns, so `yardr source check` looks at the credential and the thread
# commands above, and sends nothing.
base: https://api.github.com
calls:
user:
kind: read
about: Who the credential acts as.
request: { method: GET, path: /user }
answer: { select: [login, type] }
# Setting up GitHub
The pack wires GitHub into the yard as the declared source
`github`: its webhooks are verified and read, and the yard
replies, opens issues and sets their status through the commands the pack
carries. What a stage means in GitHub is a flow pack's to say, not this
file's.
1. **The webhook, on the repository.** Settings → Webhooks → Add webhook:
the URL `https://<public host>/hooks/<hook name>` — the path ends in
the hook's name, the one `yardr hook add` is given below — content type
`application/json`, a secret, and the events Issues, Issue comments,
Pull requests, Pull request reviews, Workflow runs and Check suites.
The yard reads those deliveries; anything else it cannot use.
2. **The hook and its secret.** The webhook's secret goes in a file, mode
0600, which the hook names. A new file takes its mode from the
shell's umask, so narrow it first. The secret is never a command's
word: that line would stay in the shell's history. Each command below
is one thing to type, in a block of its own; run the second, type or
paste the secret blind and press Enter:
```sh
umask 077
```
```sh
IFS= read -rs v && printf '%s' "$v" > ~/.yardr/github.secret; unset v
```
```sh
chmod 600 ~/.yardr/github.secret
```
```sh
yardr hook add gh --source github --secret-file ~/.yardr/github.secret
# hook gh saved: POST /hooks/gh (github)
```
One hook serves every crew member.
A crew member sets the hook up by two asks on one bead, the secret
first (`--secret-hook gh --source github`, then `--run "hook add gh
--source github"`): the operator pastes the secret typed unseen, the
server keeps it in `<home>/hooks/gh.secret` (mode 0600), and
approving the hook makes `POST /hooks/gh`. The manual steps above
remain available; `--map`, `--says` and `--as` stay in a terminal.
3. **The crew member's credential.** A token in an env file of the crew
member's own, mode 0600, never in the store. Have the crew member ask
for it, and answer on its card, in footplate or in `yardr chat`
(`ctrl+o`), where the field shows dots:
```sh
yardr bead ask <bead> "The token it answers with" --secret github --for-crew <name> --env GITHUB_TOKEN --context "Make a fine-grained token for this one repository: on GitHub, open Settings, then Developer settings, then Fine-grained tokens, and make a token for this repository with Contents set to read and write, and Issues and Pull requests set to read and write."
```
The yard's server writes it into the crew member's env file, or with
none set into `~/.yardr/crew/<name>.env`, which is then set as `yardr
crew set` sets it. The running session does not see the new value:
restart the crew member after giving it. By hand it is yours instead.
A new file takes its mode from the shell's umask, so narrow it first.
The token is never a command's word: that line would stay in the
shell's history. Each command below is one thing to type, in a block
of its own; run the second, type or paste the token blind and press
Enter:
```sh
umask 077
```
```sh
IFS= read -rs v && printf 'GITHUB_TOKEN=%s' "$v" > ~/.yardr/crew/<name>.env; unset v
```
```sh
chmod 600 ~/.yardr/crew/<name>.env
```
```sh
yardr crew set <name> env_file=~/.yardr/crew/<name>.env
```
The token reads and writes the repository's issues and pull requests;
a token that may only read answers nothing. Restart the crew member
after changing the file. Or the member speaks as a GitHub App: the
App's id and key file beside the token's place, and the yard mints an
installation token narrowed to each repository from them, with the
umask narrowed first as above. The id and the key file's path are
typed as usual: they are no secret, and the key itself is never
typed:
```sh
umask 077
```
```sh
printf '%s\n' 'GITHUB_APP_ID=<app id>' 'GITHUB_APP_PRIVATE_KEY_FILE=<key file>' > ~/.yardr/crew/<name>.env
chmod 600 ~/.yardr/crew/<name>.env
yardr crew set <name> env_file=~/.yardr/crew/<name>.env
install contrib/github-mint/github-mint ~/bin/ # from main
yardr source set github mint_command=$HOME/bin/github-mint
# source github: mint command /Users/me/bin/github-mint
```
`yardr source check github` looks the setup over — the
source's own env file, the mint, and that every thread command runs —
and sends nothing:
```sh
yardr source set github env_file=~/.yardr/crew/<name>.env
# source github: GITHUB_TOKEN is set in /Users/me/.yardr/crew/<name>.env
yardr source check github
# github: credential: GITHUB_TOKEN is set in /Users/me/.yardr/crew/<name>.env
# github: its file names no check call (check: <a read call without parameters>), so nothing was sent
# github: command thread.reply (github/thread reply): can be run
# ...
```
The source's file may name the crew member's where one member is all
the yard has; a yard with several keeps its own, with the App's names
where it mints.
4. **Its identity.** The login of the user the token acts as, looked up on
GitHub by hand — or the App's bot, where it speaks as one:
```sh
yardr crew identity <name> --source github --key <login>
yardr crew identity <name> --source github --key '<slug>[bot]'
```
A crew member setting itself up proposes the login instead, from its
session, and the operator approves it on the card:
```sh
yardr bead ask <bead> "Who am I on GitHub?" --run "crew identity <name> --source github --key <login>"
```
A bot login (`<slug>[bot]`) has characters no run ask carries, so it
stays typed. A login is kept lowercase, however it was typed. What the
bot does is then its own, and @\<slug\> in a comment mentions it.
5. **The grants.** Opening issues and setting their status are the
operator's to grant:
```sh
yardr crew set <name> open_threads=github set_state=github
```
6. **The daemon's reads.** Judging a run against the pull request's head
and reading CI as state run with the hook's crew member's credential,
read from the member's env file (step 3) as for a move; nothing ambient
is tried. The hook acts as that member, and adding the member re-saves
the hook, so the source and secret go again:
```sh
yardr hook add gh --source github --secret-file ~/.yardr/github.secret --as <name> \
--says "event=ci conclusion=success stage=in-pr -> done" \
--says "event=ci conclusion=failure stage=in-pr -> changes"
# hook gh saved: POST /hooks/gh (github)
```
The lines are the flow's (`yardr docs line` writes them for its train flow):
a hook with an `event=ci` line and no `--as` is refused, as is one
whose source declares no `ci` command. A hook whose table has no
`event=ci` line may skip the `--as`: events still move beads, and only
the live reads need the member. The train's brief carries the same read
from the source's own env file (step 3), since the brief has no hook.
7. **Test it.** As the crew member: open an issue, mention it from GitHub,
move it:
```sh
yardr thread open github --in <owner>/<repo> - -b - # stdin: the title (Hello from the yard), then the body (testing the wiring)
# comment @<login> on the issue in GitHub: it lands in the crew member's inbox as mentioned
yardr thread state <owner>/<repo>#<n> started
```
`yardr thread list <name>` shows the thread's id.
To open an issue assigned to a person, give the yard's user their
login once, then name them. GitHub requires the credential to have push
access to set assignees; the command reports an error if GitHub creates
the issue but drops the assignee:
```sh
yardr user identity <user> --source github --key <their login>
yardr thread open github --in <owner>/<repo> --assign <user> - # stdin: the title (For you)
```
8. **Pushing the depot's landings.** A depot whose origin is on github.com
pushes its landings because git itself is handed the source's token.
The pack brings the credential helper; the depot's git is pointed at
the copy the import made, in that depot's own config alone:
```sh
git -C <depot> config credential."https://github.com".helper ~/.yardr/sources/github/github/git-credential
```
Never `--global`: the person's own login stays as it is for everything
else (on a Mac, the keychain), and only this depot's git asks the
helper. Only the helper's name stands in the config: the token is in no
remote URL, group `cmd`, log or config, and is read from the source's
env file at the moment git asks, so a rotated token needs no re-turn-on.
The token needs the `contents: write` permission to push; where the
source mints from a GitHub App instead of a static token, add a
`GITHUB_TOKEN` with that permission to the source's env file for pushes
(the yard keeps minting for its own calls). Whether the depot pushes at
all is the assembly group's `YARDR_PUSH`, not the pack's.