Summon in GNU Guix

ryan · September 19, 2020, 11:03pm

Hi folks! I’ve created Guix packages for summon and summon-conjur and I’m ready to submit them upstream. But before I do, I wanted to give the community an opportunity to review the packages and offer any criticism or suggestions for improvement first.

Rationale

Once these packages are available upstream, a Guix user on any Linux distro will be able to install both tools by typing guix install summon summon-conjur. Or, they can create a nice environment for hacking on them, with all their build dependencies downloaded & ready to go, by typing guix environment summon summon-conjur.

Package definitions

The package definitions are here: https://github.com/ryanprior/guix-packages/blob/master/testing/summon.scm

You’ll find definitions for:

summon & summon-conjur
conjur-api-go
other dependencies that weren’t included in Guix yet, so I added them as part of the same effort

Testing

Guix runs the tests provided with the packages as part of the build process and fails the build when the tests fail, so users can feel fairly certain they are getting a result that works properly. I’ve disabled the tests for conjur-api-go because they appear to require a running Conjur server, which makes sense for API tests. But, to facilitate deterministic and reproducible builds, Guix sandboxes the build and test environments so they don’t have any network connectivity. All the rest of the tests are run and pass.

To demonstrate this for yourself, follow these steps:

install Guix as per these instructions
guix pull to pull the latest version of the dependencies
git clone https://github.com/ryanprior/guix-packages.git
guix build -L./guix-packages summon summon-conjur
guix environment -L./guix-packages --ad-hoc summon summon-conjur (this starts a new subshell)
summon -h; summon-conjur -h

Package descriptions

Part of the packaging process is writing useful descriptions to help people find them. Here are the ones I wrote, taking from readmes and homepages where I could:

	summon	summon-conjur
synopsis	Fetches secrets and makes them available to a process	Fetches secrets from a Conjur service
description	Summon fetches secrets using a provider program and a configuration file, then launches a subprocess with access to those secrets via its environment or a memory-mapped temporary file. When the subprocess exits, it removes the secrets.	The summon-conjur utility fetches a secret from Conjur, printing it to stdout.

Updating packages

Guix doesn’t have designated package maintainers, so any community member can submit updates to packages. To make it easier to find updateable packages, Guix provides the refresh command which scans for updates.

To find out whether summon and summon-conjur packages are up to date, I can run:

ryan@swallowtail$ guix refresh summon summon-conjur go-github.com-cyberark-conjur-api
gnu/packages/golang.scm:5757:13: 0.6.0 is already the latest version of go-github.com-cyberark-conjur-api
gnu/packages/cybersecurity.scm:54:13: 0.5.3 is already the latest version of summon-conjur
gnu/packages/cybersecurity.scm:90:13: 0.8.2 is already the latest version of summon

If you want to update the package version, you follow these steps:

clone the Guix repository
run guix refresh as above, which prints the file names and line numbers of the package definitions in Guix.
edit the package definition’s “version” string to reflect the new updated version number
clone the package repo and check-out the tag corresponding to the new version, eg. git clone -b v0.8.2 https://github.com/cyberark/summon.git /tmp/summon
hash the repo, eg guix hash -rx /tmp/summon
replace the hash string in the package definition with the new hash
commit your changes
run git format-patch master to create a patch file for your update
send the patch file to guix-patches@gnu.org

Of course, I’ll be around to help out with this process, so you can always ping me if you’re just putting a release out and I can handle the package update or walk somebody through the process.

Post your questions and feedback!

Please share your questions, ideas, or other feedback with me so I can make sure we have great high-quality packages in Guix. Thank you!

joe.garcia · September 21, 2020, 2:32pm

This is great work and an excellent write-up, @ryan! I’m giving it a go in my Vagrant playground right now.

joe.garcia · September 21, 2020, 2:59pm

@ryan:

I ran GuixSD in Vagrant using the following Vagrantfile:

gist.github.com

https://gist.github.com/infamousjoeg/67adf2bd86778422c4f25f47b580805d

Vagrantfile

# Run the following command in the root directory where
# this Vagrantfile is located:
# $ git clone https://github.com/ryanprior/guix-packages.git

Vagrant.require_version ">= 2.1.3"
Vagrant.configure("2") do |config|
    config.vm.box = "palfrey/guixsd"
    config.vm.synced_folder ".", "/vagrant", disabled: true
    config.vm.provision "file", source: "./guix-packages", destination: "$HOME/guix-packages"
end

After running guix build -L./guix-packages summon summon-conjur as the vagrant user in the VM, I receive the following build errors:

command "go" "install" "-v" "-x" "-ldflags=-s -w" "github.com/cyberark/conjur-api-go/conjurapi" failed with status 1
builder for `/gnu/store/sp85gpwlg08viiwz6qnsnn2qliz5690h-go-github.com-cyberark-conjur-api-0.6.0.drv' failed with exit code 1
build of /gnu/store/sp85gpwlg08viiwz6qnsnn2qliz5690h-go-github.com-cyberark-conjur-api-0.6.0.drv failed
View build log at '/var/log/guix/drvs/sp/85gpwlg08viiwz6qnsnn2qliz5690h-go-github.com-cyberark-conjur-api-0.6.0.drv.bz2'.
building /gnu/store/6b8gdkc6wkqv1747r54zdkq10f6y2c9n-go-gopkg-in-yaml-v3-3-checkout.drv...
cannot build derivation `/gnu/store/gi1pv7zjmvjrxb2y88zwnyvqvz0x6z34-summon-conjur-0.5.3.drv': 1 dependencies couldn't be built
guix build: error: build of `/gnu/store/gi1pv7zjmvjrxb2y88zwnyvqvz0x6z34-summon-conjur-0.5.3.drv' failed

Have you experienced this? Is this because of a root requirement somewhere? Any suggestions?

EDIT: It’s not root – same failure after sudo su.

EDIT 2: Full build log:

starting phase `build'
src/github.com/cyberark/conjur-api-go/conjurapi/config.go:12:2: cannot find package "gopkg.in/yaml.v2" in any of:
	/gnu/store/xacmln5jbjx28yb73cmp0v3i2g2wca8g-go-1.13.9/src/gopkg.in/yaml.v2 (from $GOROOT)
	/tmp/guix-build-go-github.com-cyberark-conjur-api-0.6.0.drv-0/src/gopkg.in/yaml.v2 (from $GOPATH)
Building 'github.com/cyberark/conjur-api-go/conjurapi' failed.
Here are the results of `go env`:
GO111MODULE="off"
GOARCH="amd64"
GOBIN="/gnu/store/dhl8ylyqijzxmp8cnbzwxxnrzcbgfij9-go-github.com-cyberark-conjur-api-0.6.0/bin"
GOCACHE="/tmp/go-cache"
GOENV="/homeless-shelter/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/tmp/guix-build-go-github.com-cyberark-conjur-api-0.6.0.drv-0"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/gnu/store/xacmln5jbjx28yb73cmp0v3i2g2wca8g-go-1.13.9"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/gnu/store/xacmln5jbjx28yb73cmp0v3i2g2wca8g-go-1.13.9/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="/gnu/store/x3jx25cd3q363mr7nbgzrhrv1vza6cf7-gcc-7.4.0/bin/gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/guix-build-go-github.com-cyberark-conjur-api-0.6.0.drv-0/go-build904119001=/tmp/go-build -gno-record-gcc-switches"
command "go" "install" "-v" "-x" "-ldflags=-s -w" "github.com/cyberark/conjur-api-go/conjurapi" failed with status 1

ryan · September 21, 2020, 3:07pm

That image says it was built 5 months ago, so it’s maybe missing definitions for some dependencies. Try running guix pull to pull the latest package definitions, then running the build again. I’ll add that in to the instructions above as well.

joe.garcia · September 21, 2020, 3:25pm

@ryan:

Successful guix pull followed by same failure on build of conjur-api-go.

I installed Guix on an Ubuntu VM from latest and received the same error there, as well.

izgerij · September 21, 2020, 3:28pm

@ryan thanks for sharing this! I’d like to understand better the rationale for using Guix vs other Linux package managers for distributing our Linux-friendly binaries. We already support Homebrew for OSX or Linux - what is the added benefit of also distributing our Linux binaries using Guix?

ryan · September 22, 2020, 3:07am

@joe.garcia I did manage to get the packages to build on Guix System with Vagrant. I did two things differently from you:

I ran guix pull twice. I don’t know why that would do anything; in theory it should do all the necessary work in one go and maybe it’s a bug that it didn’t. After my first guix pull I just had a gut feeling though and I was like “let’s try it again and see that ‘nothing to be done’ message.”

But darn if it didn’t do another whole round of work. (Not near as long as the first go-round thankfully.)
I cloned the repo from GitHub instead of provisioning it locally with Vagrant. The steps I took to do this were:
a) guix install git nss-certs
b) log-out and vagrant ssh again
c) git clone https://github.com/ryanprior/guix-packages

Then after all that, it built the two packages no problem. Possibly you were really close both times :X

@izgerij I am a big fan of Homebrew and I love that they’ve built a system that works across three major dev platforms.

Guix has some unique strengths:

All its package operations are atomic: when you install, remove, upgrade, or downgrade packages, they all succeed or no changes are made, like a database transaction.
It has unlimited “undo” - any changes you make to your packages are recorded, and you can return to any previous state.
It has time-travel: you can install just one package from a newer version of Guix without upgrading your whole system, you can install a bunch of packages exactly as they were on some particular date in the past, etc.
It’s security-focused: its toolchain is bootstrapped from a tiny, audited core and you can perform a full bootstrapped build of any package (at the cost of extra CPU time) to get closer to zero-trust builds. Guix packages are reproducible byte-for-byte, so it’s easy to “challenge” the binary package servers by building your own package and comparing.
It’s super automation friendly: you can create environments with different sets of package, services, and configurations, and they won’t cause conflicts. Guix can also turn such an environment into a tarball, a VM image or a Docker image to deploy it elsewhere.

Lots more info here: https://guix.gnu.org/manual/en/html_node/Features.html

sgnn7 · September 22, 2020, 3:52pm

Hy @ryan,
This seems like a great addition to the tooling around Summon and thank you for exploring this avenue of improving UX! With that said I think that GUIX adoption rate is really not high enough for it to be taken on as an “official” C&I project at this time as it would effectively put it on the lower end of work queues from the start and on a likely path to abandonware. I think if you can add some automated container installation E2E tests (maybe via GitHub actions) we definitely could link to this project and its artifacts from Summon docs though. As for your original question about should you publish these packages, I think that’s fine but some E2E tests as mentioned above would be really nice to see to ensure that they are installable/runnable.

joe.garcia · September 22, 2020, 4:29pm

That was the ticket, @ryan!

@izgeri: I upvote this contribution to upstream!

vagrant@guixsd ~ [env]$ summon -h; summon-conjur -h
NAME:
   summon - Parse secrets.yml and export environment variables

USAGE:
   summon [global options] command [command options] [arguments...]

VERSION:
   0.8.2

COMMANDS:
   help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   -p value, --provider value     Path to provider for fetching secrets
   -e value, --environment value  Specify section/environment to parse from secrets.yaml
   -f value                       Path to secrets.yml (default: "secrets.yml")
   --up                           Go up in the directory hierarchy until the secrets file is found
   -D value                       var=value causes substitution of value to $var
   --yaml value                   secrets.yml as a literal string
   --ignore value, -i value       Ignore the specified key if is isn't accessible or doesn't exist
   --ignore-all, -I               Ignore inaccessible or missing keys
   --all-provider-versions, -V    List of all of the providers in the default path and their versions(if they have the --version tag)
   --help, -h                     show help
   --version, -v                  print the version
Usage of summon-conjur:
  -h, --help
    show help
  -V, --version
    show version
  -v, --verbose
    be verbose

Topic		Replies	Views
Summon/ Summon-conjur for SOLARIS Summon opensource , summon , summon-conjur	4	707	September 28, 2020
Windows & Summon-Conjur Issues Secrets Management - Conjur, Secrets Hub & CP summon	3	633	August 19, 2021
Environment Variables Python CLI Secrets Management - Conjur, Secrets Hub & CP opensource , conjur-cli	2	825	September 26, 2019
Credentials for Summon (using /etc/conjur.identity) Secrets Management - Conjur, Secrets Hub & CP	5	308	September 27, 2023
Question about using environment variables in Summon Summon summon , summon-conjur , pcf , conjur-buildpack , dap	2	866	December 3, 2019