The task is to write a tool which will be started from a Jenkin’s job by a cron and will check an organization’s public repositories list in the Github.
A Docker-image build and a Jenkins job are described in the Jenkins: a job to check a Github organization’s public repositories list post.
Then it has to compare received repositories list with a predefined list with allowed repositories and if they will be not equal – send an alert to a Slack channel.
The idea is to have such a check in case if developers accidentally will create a public repository instead of private or will change a private repository to the public and get a notification about such issue.
It could be written with bash and curl, or Python with urllib and use Github API directly, but I want to have some Go experience so will use it.
In fact – this is my the very first self-written Golang code which I didn’t use a lot and even don’t know its syntax, but will try to use some existing knowledge with C/Python, add some logic and of course with the Google’s help and write something workable.
To work with API Github the go-github package will be used.
Let’s begin.
Add imports, the function was created automatically with the vim-go plugin:
package main
import (
"fmt"
"github.com/google/go-github/github"
)
func main() {
fmt.Println("vim-go")
}
Add $GOPATH:
[simterm]
$ sudo mkdir /usr/local/go $ sudo chown setevoy:setevoy /usr/local/go/ $ export GOPATH=/usr/local/go && export GOBIN=/usr/local/go/bin
[/simterm]
Install package:
[simterm]
$ go get # _/home/setevoy/Scripts/Go/GTHB ./gitrepos.go:5:2: imported and not used: "github.com/google/go-github/github"
[/simterm].
Contents
Getting repositories list from Github
Copy and paste the first example from the package’s README, an organization name will be taken from the $GITHUB_ORG_NAMEand let’s try get something.
Set the variable:
[simterm]
$ export GITHUB_ORG_NAME="rtfmorg"
[/simterm]
And code:
package main
import (
"context"
"fmt"
"github.com/google/go-github/github"
)
func main() {
client := github.NewClient(nil)
opt := &github.RepositoryListByOrgOptions{Type: "public"}
repos, _, _ := client.Repositories.ListByOrg(context.Background(), os.Getenv("GITHUB_ORG_NAME"), opt)
fmt.Printf(repos)
}
In the original example, the organization name was hardcoded but here we will take it from a variable which will be set in a Jenkin’s job using os.Getenv.
Add context and os imports.
Run the code:
[simterm]
$ go run go-github-public-repos-checker.go # command-line-arguments ./go-github-public-repos-checker.go:17:15: cannot use repos (type []*github.Repository) as type string in argument to fmt.Printf
[/simterm]
Errr…
Okay.
I thought client.Repositories.ListByOrg will return actually list – just because of the List in its name 🙂
Check what data type we have in the repos object. Use reflect:
package main
import (
"os"
"context"
"fmt"
"reflect"
"github.com/google/go-github/github"
)
func main() {
client := github.NewClient(nil)
opt := &github.RepositoryListByOrgOptions{Type: "public"}
repos, _, _ := client.Repositories.ListByOrg(context.Background(), os.Getenv("GITHUB_ORG_NAME"), opt)
fmt.Println(reflect.TypeOf(repos).String())
}
Run:
[simterm]
$ go run go-github-public-repos-checker.go []*github.Repository
[/simterm]
Good… It’s really list ([]), apparently pointed to a structure – github.Repository.
Let’s check with the go doc:
[simterm]
$ go doc github.Repository
type Repository struct {
ID *int64 `json:"id,omitempty"`
NodeID *string `json:"node_id,omitempty"`
Owner *User `json:"owner,omitempty"`
Name *string `json:"name,omitempty"`
FullName *string `json:"full_name,omitempty"`
Description *string `json:"description,omitempty"`
...
[/simterm]
Yup – it’s structure and also we see all its fields.
To shut up Go with the “imported and not used: “reflect” message – set it as _reflect:
package main
import (
"os"
"context"
"fmt"
_"reflect"
"github.com/google/go-github/github"
)
func main() {
client := github.NewClient(nil)
opt := &github.RepositoryListByOrgOptions{Type: "public"}
repos, _, _ := client.Repositories.ListByOrg(context.Background(), os.Getenv("GITHUB_ORG_NAME"), opt)
for _, repo := range repos {
fmt.Printf("Repo: %s\n", *repo.Name)
}
}
Here from the repos list we are getting an element’s ID and its value.
Can add ID’s as well:
...
for id, repo := range repos {
fmt.Printf("%d Repo: %s\n", id, *repo.Name)
}
...
[simterm]
$ go run go-github-public-repos-checker.go 0 Repo: org-repo-1-pub 1 Repo: org-repo-2-pub
[/simterm]
Comparing lists in Go
Now need to add another one list which will keep repositories list from Github.
Allowed to be public repositories will be passed from Jenkins in a list view separated by spaces:
[simterm]
$ export ALLOWED_REPOS="1 2"
[/simterm]
Attempt number one
Create allowedRepos of the string type and save allowed public repositories list in it:
...
allowedRepos := []string{os.Getenv("ALLOWED_REPOS")}
for id, repo := range repos {
fmt.Printf("%d Repo: %s\n", id, *repo.Name)
}
fmt.Printf("Allowed repos: %s\n", allowedRepos)
...
But here is an issue which I faced with a bit later when started comparing lists. Will see it shortly.
Run the code:
[simterm]
$ go run go-github-public-repos-checker.go 0 Repo: org-repo-1-pub 1 Repo: org-repo-2-pub Allowed repos: [1 2]
[/simterm]
Okay – it works.
Now need to make checks between two repositories lists – repos with repositories from Github and allowedRepos.
The first solution was next:
...
allowedRepos := []string{os.Getenv("ALLOWED_REPOS")}
fmt.Printf("Allowed repos: %s\n", allowedRepos)
for id, repo := range repos {
for _, i := range allowedRepos {
if i == *repo.Name {
fmt.Printf("Index: %d, repo %s found in Allowed as %s\n", id, *repo.Name, i)
} else {
fmt.Printf("ALARM: repo %s was NOT found in Allowed!\n", *repo.Name)
}
}
}
...
Run:
[simterm]
$ go run go-github-public-repos-checker.go Allowed repos: [1 2] ALARM: repo org-repo-1-pub was NOT found in Allowed! ALARM: repo org-repo-2-pub was NOT found in Allowed!
[/simterm]
Looks like it works?
Repositories with the 1 and 2 names != org-repo-1-pub and org-repo-2-pub.
But the problem which might be already obvious for some readers appeared when I did a “back-check”, i.e. when I set $ALLOWED_REPOS with real names to get the OK result:
[simterm]
$ export ALLOWED_REPOS="org-repo-1-pub org-repo-2-pub"
[/simterm]
Check:
[simterm]
$ go run go-github-public-repos-checker.go Allowed repos: [org-repo-1-pub org-repo-2-pub] ALARM: repo org-repo-1-pub was NOT found in Allowed! ALARM: repo org-repo-2-pub was NOT found in Allowed!
[/simterm]
And why?
Because of in the allowedRepos we have not a list aka slice and Go – but just a string.
Let’s add the i variable’s output and indexes numbers:
...
for r_id, repo := range repos {
for a_id, i := range allowedRepos {
fmt.Printf("ID: %d Type: %T Value: %s\n", a_id, allowedRepos, i)
if i == *repo.Name {
fmt.Printf("Index: %d, repo %s found in Allowed as %s\n", r_id, *repo.Name, i)
} else {
fmt.Printf("ALARM: repo %s was NOT found in Allowed!\n", *repo.Name)
}
}
}
...
Check:
[simterm]
$ go run go-github-public-repos-checker.go Allowed repos: [org-repo-1-pub org-repo-2-pub] ID: 0 Type: []string Value: org-repo-1-pub org-repo-2-pub ALARM: repo org-repo-1-pub was NOT found in Allowed! ID: 0 Type: []string Value: org-repo-1-pub org-repo-2-pub ALARM: repo org-repo-2-pub was NOT found in Allowed!
[/simterm]
allowedRepos is really a list but with the only 0 element which keeps the “org-repo-1-pub org-repo-2-pub” value.
Attempt number two
To make this working – need to convert the allowedRepos to a real list.
Let’s use the strings package:
...
// allowedRepos := []string{os.Getenv("ALLOWED_REPOS")}
allowedRepos := strings.Fields(os.Getenv("ALLOWED_REPOS"))
fmt.Printf("Allowed repos: %s\n", allowedRepos)
for r_id, repo := range repos {
for a_id, i := range allowedRepos {
fmt.Printf("ID: %d Type: %T Value: %s\n", a_id, allowedRepos, i)
if i == *repo.Name {
fmt.Printf("Index: %d, repo %s found in Allowed as %s\n", r_id, *repo.Name, i)
} else {
fmt.Printf("ALARM: repo %s was NOT found in Allowed!\n", *repo.Name)
}
}
}
...
allowedRepos now will be filled with the strings.Fields().
Check result:
[simterm]
$ go run go-github-public-repos-checker.go Allowed repos: [org-repo-1-pub org-repo-2-pub] ID: 0 Type: []string Value: org-repo-1-pub Index: 0, repo org-repo-1-pub found in Allowed as org-repo-1-pub ID: 1 Type: []string Value: org-repo-2-pub ALARM: repo org-repo-1-pub was NOT found in Allowed! ID: 0 Type: []string Value: org-repo-1-pub ALARM: repo org-repo-2-pub was NOT found in Allowed! ID: 1 Type: []string Value: org-repo-2-pub Index: 1, repo org-repo-2-pub found in Allowed as org-repo-2-pub
[/simterm]
Okay – much better now but this also will not work as there some “false-positive” results because of each repos‘s element is compared with each allowedRepos‘s element.
Attempt number three
Let’s rewrite it and now let’s try to use a repo‘s indexes to chose an element from the allowedRepos:
...
for r_id, repo := range repos {
fmt.Printf("%d %s\n", r_id, *repo.Name)
if *repo.Name != allowedRepos[r_id] {
fmt.Printf("ALARM: repo %s was NOT found in Allowed!\n", *repo.Name)
} else {
fmt.Printf("Repo %s found in Allowed\n", *repo.Name)
}
}
...
I.e. from the repos we getting an element’s ID and then checking the allowedRepos‘s element with the same ID.
Run:
[simterm]
$ go run go-github-public-repos-checker.go Allowed repos: [org-repo-1-pub org-repo-2-pub] 0 org-repo-1-pub Repo org-repo-1-pub found in Allowed 1 org-repo-2-pub Repo org-repo-2-pub found in Allowed
[/simterm]
Nice!
But another issue can happen now…
What if the order in both lists will differ?
Set instead of the:
[simterm]
$ export ALLOWED_REPOS="org-repo-1-pub org-repo-2-pub"
[/simterm]
Repositories names in the reversed order:
[simterm]
$ export ALLOWED_REPOS="org-repo-2-pub org-repo-1-pub"
[/simterm]
Check:
[simterm]
$ go run go-github-public-repos-checker.go Allowed repos: [org-repo-2-pub org-repo-1-pub] 0 org-repo-1-pub ALARM: repo org-repo-1-pub was NOT found in Allowed! 1 org-repo-2-pub ALARM: repo org-repo-2-pub was NOT found in Allowed!
[/simterm]
D’oh!
Attempt number four
The next idea was to use the reflect module and its DeepEqual() function.
Add one more list called actualRepos, add repositories taken from Github with the append and then compare them:
...
allowedRepos := strings.Fields(os.Getenv("ALLOWED_REPOS"))
var actualRepos []string
for _, repo := range repos {
actualRepos = append(actualRepos, *repo.Name)
}
fmt.Printf("Allowed: %s\n", allowedRepos)
fmt.Printf("Actual: %s\n", actualRepos)
fmt.Println("Slice equal: ", reflect.DeepEqual(allowedRepos, actualRepos))
...
Run:
[simterm]
$ go run go-github-public-repos-checker.go Allowed: [org-repo-2-pub org-repo-1-pub] Actual: [org-repo-1-pub org-repo-2-pub] Slice equal: false
[/simterm]
And again no… Although if revert order back – it will work:
[simterm]
$ export ALLOWED_REPOS="org-repo-1-pub org-repo-2-pub" $ go run go-github-public-repos-checker.go Allowed: [org-repo-1-pub org-repo-2-pub] Actual: [org-repo-1-pub org-repo-2-pub] Slice equal: true
[/simterm]
So, need to go back to the loop’s variant but to check somehowrepo‘s value in the whole allowedRepos list and only if it’s absent – create an alert.
Attempt number five
The solution was next: create a dedicated function which will check all allowedRepos‘s elements in a loop and will return true if the value will be found and false otherwise. Then – we can use this in themain()‘s loop.
Let’s try.
Create a isAllowedRepo()function which will accept two arguments – a repository name to be checked and the allowed repositories list and will return a boolean value:
...
func isAllowedRepo(repoName string, allowedRepos []string) bool {
for _, i := range allowedRepos {
if i == repoName {
return true
}
}
return false
}
...
Then in the main() – run a loop over all repos‘s elements, pass them one by one to the isAllowedRepo() and then print a result:
...
for _, repo := range repos {
fmt.Printf("Checking %s\n", *repo.Name)
fmt.Println(isAllowedRepo(*repo.Name, allowedRepos))
}
...
Let’s test.
First, restore allowed repositories list in the initial order:
[simterm]
$ export ALLOWED_REPOS="org-repo-1-pub org-repo-2-pub" $ go run go-github-public-repos-checker.go Checking org-repo-1-pub true Checking org-repo-2-pub true
[/simterm]
Good!
And reversed order:
[simterm]
$ export ALLOWED_REPOS="org-repo-2-pub org-repo-1-pub" $ go run go-github-public-repos-checker.go Checking org-repo-1-pub true Checking org-repo-2-pub true
[/simterm]
Still works…
Now remove one of the allowed repositories:
[simterm]
$ export ALLOWED_REPOS="org-repo-2-pub" $ go run go-github-public-repos-checker.go Checking org-repo-1-pub false Checking org-repo-2-pub true
[/simterm]
Great!
Now when we will get false – we can raise an alert.
The whole code now. Leaving “as is” just for history:
package main
import (
"os"
"context"
"fmt"
_"reflect"
"strings"
"github.com/google/go-github/github"
)
func isAllowedRepo(repoName string, allowedRepos []string) bool {
for _, i := range allowedRepos {
if i == repoName {
return true
}
}
return false
}
func main() {
client := github.NewClient(nil)
opt := &github.RepositoryListByOrgOptions{Type: "public"}
repos, _, _ := client.Repositories.ListByOrg(context.Background(), os.Getenv("GITHUB_ORG_NAME"), opt)
// allowedRepos := []string{os.Getenv("ALLOWED_REPOS")}
allowedRepos := strings.Fields(os.Getenv("ALLOWED_REPOS"))
// var actualRepos []string
/*
for _, repo := range repos {
actualRepos = append(actualRepos, *repo.Name)
}
fmt.Printf("Allowed: %s\n", allowedRepos)
fmt.Printf("Actual: %s\n", actualRepos)
fmt.Println("Slice equal: ", reflect.DeepEqual(allowedRepos, actualRepos))
*/
for _, repo := range repos {
fmt.Printf("Checking %s\n", *repo.Name)
fmt.Println(isAllowedRepo(*repo.Name, allowedRepos))
}
/*
for r_id, repo := range repos {
for _, i := range allowedRepos {
fmt.Printf("Checking %s and %s\n", *repo.Name, i)
if i == *repo.Name {
fmt.Printf("Index: %d, repo %s found in Allowed as %s\n", r_id, *repo.Name, i)
break
} else {
fmt.Printf("ALARM: repo %s was NOT found in Allowed!\n", *repo.Name)
}
}
}
*/
/*
for r_id, repo := range repos {
fmt.Printf("%d %s\n", r_id, *repo.Name)
if *repo.Name != allowedRepos[r_id] {
fmt.Printf("ALARM: repo %s was NOT found in Allowed!\n", *repo.Name)
} else {
fmt.Printf("Repo %s found in Allowed\n", *repo.Name)
}
}
*/
/*
for r_id, repo := range repos {
for a_id, i := range allowedRepos {
fmt.Printf("ID: %d Type: %T Value: %s\n", a_id, allowedRepos, i)
if i == *repo.Name {
fmt.Printf("Index: %d, repo %s found in Allowed as %s\n", r_id, *repo.Name, i)
} else {
fmt.Printf("ALARM: repo %s was NOT found in Allowed!\n", *repo.Name)
}
}
}
*/
}
Golang Slack
And the final thing is to add a Slack notification.
Use ashwanthkumar/slack-go-webhook here.
Configure Slack’s WebHook, get its URL.
Add a new sendSlackAlarm() function which will accept two arguments – repository name and its URL:
...
func sendSlackAlarm(repoName string, repoUrl string) {
webhookUrl := os.Getenv("SLACK_URL")
text := fmt.Sprintf(":scream: ALARM: repository *%s* was NOT found in Allowed!", repoName)
attachment := slack.Attachment{}
attachment.AddAction(slack.Action{Type: "button", Text: "RepoURL", Url: repoUrl, Style: "danger"})
payload := slack.Payload{
Username: "Github checker",
Text: text,
Channel: os.Getenv("SLACK_CHANNEL"),
IconEmoji: ":scream:",
Attachments: []slack.Attachment{attachment},
}
err := slack.Send(webhookUrl, "", payload)
if len(err) > 0 {
fmt.Printf("error: %s\n", err)
}
}
...
Add the sendSlackAlarm() execution to the main() if the isAllowedRepo() returned false:
...
for _, repo := range repos {
fmt.Printf("\nChecking %s\n", *repo.Name)
if isAllowedRepo(*repo.Name, allowedRepos) {
fmt.Printf("OK: repo %s found in Allowed\n", *repo.Name)
} else {
fmt.Printf("ALARM: repo %s was NOT found in Allowed!\n", *repo.Name)
sendSlackAlarm(*repo.Name, *repo.HTMLURL)
}
}
...
repo.HTMLURL we found from the go doc github.Repository.
Add $SLACK_URL and $SLACK_CHANNEL environment variables:
[simterm]
$ export SLACK_URL="https://hooks.slack.com/services/T1641GRB9/BA***WRE" $ export SLACK_CHANNEL="#general"
[/simterm]
Restore the full repositories list on reversed order:
[simterm]
$ export ALLOWED_REPOS="org-repo-2-pub org-repo-1-pub"
[/simterm]
Check:
[simterm]
$ go run go-github-public-repos-checker.go Checking org-repo-1-pub OK: repo org-repo-1-pub found in Allowed Checking org-repo-2-pub OK: repo org-repo-2-pub found in Allowed
[/simterm]
Okay…
Remove one allowed:
[simterm]
$ export ALLOWED_REPOS="org-repo-2-pub"
[/simterm]
Check again:
[simterm]
$ go run go-github-public-repos-checker.go Checking org-repo-1-pub ALARM: repo org-repo-1-pub was NOT found in Allowed! Checking org-repo-2-pub OK: repo org-repo-2-pub found in Allowed
[/simterm]
And Slack notification:
Done.
The script is available in the setevoy-tools Github repository.
