Go Project Development Environment

At initial release, the Go programming language development flow was built around a special directory called GOPATH, which would contain the source code for all of the packages you are working with, plus any executables built with go install, and cached package archives ready for linking.

I've generally always been in the camp of wanting each project I work on to have a separated development environment, so I can switch between projects and work freely without accidentally impacting the development environment for some other project.

For Go then, I embraced the less-popular per-project GOPATH style. In practice that meant that my main development directory contained lots of directories that were themselves GOPATHs for different projects, like this:

dev/alamatic/src/github.com/alamatic/alamatic/...
dev/envy/src/envy.pw/cli/...
dev/hcl2/src/github.com/hashicorp/hcl2/...
dev/terraform/src/github.com/hashicorp/terraform/...
...

In my early days working on Go projects I wrote some simple bash functions to help with switching between these, so e.g. I could begin working on Terraform with just a couple of commands:

$ cd dev/terraform/src/github.com/hashicorp/terraform
$ go-here activate
Activating GOPATH=/home/mart/dev/terraform

go-here activate would set GOPATH and also set PATH to include the bin directory under that GOPATH, so I can then just start using go install and running terraform as normal.

This has been my daily driver workflow for a number of years now, to the point where I was no longer really even thinking about GOPATHs and was was just activating my per-project dev environments and getting on with work.

Working in "Modules Mode"

Go 1.11 introduced Go Modules, and with it a new work style that no longer requires source code to live in GOPATH. You do still have a GOPATH, but its duties are reduced to being the home of a cache of installed upstream modules and (by default) the bin directory as before.

Ever since upgrading to Go 1.11 my daily dance has had a few extra steps:

$ cd dev/terraform/src/github.com/hashicorp/terraform
$ go-here activate
$ export GO111MODULE=on
$ export GOFLAGS=-mod=vendor

The final step here is because Terraform is still using "vendoring" in modules mode. That probably won't be true for ever, but it's important for now to ensure that I'm always testing against the vendored dependencies that would be used in a real release.

As well as adding some extra friction, the above workflow also still assumes a GOPATH per project. But is that still important?

My primary goal, implementation details aside, is to be confident that working on one project won't impact the dev enviroment of any other. In modules mode, there are some different things to isolate:

The bin directory where go install writes to, as before. By default this is $GOPATH/bin, but we can override it with GOBIN.
The git work tree of the project itself, which no longer needs to be in GOPATH at all.
The cache of all of the other modules the current project depends on, which lives in $GOPATH/pkg/mod.

It turns out that the module cache is, unlike $GOPATH/src in GOPATH mode, designed to allow multiple versions of the same module to be installed at once. The modules are arranged into directories containing their version numbers:

pkg/mod/github.com/hashicorp/go-uuid@v1.0.1
pkg/mod/github.com/hashicorp/hcl2@v0.0.0-20190515223218-4b22149b7cef

As long as we use good discipline in not modifying any of the upstream dependencies while working on a project, there's no harm in sharing this module cache between projects. Indeed, the go tool marks all of the files in here as read-only by default to reinforce the fact that it's a shared cache and not to be edited.

That then just leaves the bin directory. The default Go workflow still assumes that all projects share a single directory for tools, but I don't find that satisfactory because lots of projects depend on tools for code generation and using the wrong version of these tools can produce inconsistent results.

Therefore I've established my own convention for creating per-project bin directories: I place them in a subdirectory of $GOPATH/bin generated from the main package path, such as $GOPATH/bin/github.com/hashicorp/terraform for Terraform.

With that minor quirk of overriding GOBIN, I am now comfortable with sharing the same GOPATH across all projects. However, I still need to remember to set GOBIN properly when switching between projects.

To automate that, I added a new subcommand to my go-here script, creating the following project-selection workflow:

$ cd dev/terraform # (this is now directly the git repository, not a GOPATH)
$ go-here mod
vendor directory is present, so setting GOFLAGS=-mod=vendor
Activating module github.com/hashicorp/terraform with GOBIN=/home/mart/go/bin/github.com/hashicorp/terraform

As the output above indicates, there's a special case in the mod subcommand to detect a vendor directory and assume this project therefore requires vendor mode, which it then forces on by setting GOFLAGS.

The updated go-here, which now supports both my old and new workflows, is included in full below:

function _go-here {
    DIR="$1"
    export GOPATH="$DIR"
    export PATH="$GOPATH/bin":$PATH
    echo Activating GOPATH="$GOPATH"
}

function _go-here_mod {
    DIR="$1"
    cd "$DIR"
    if [ -d "vendor" ]; then
        echo "vendor directory is present, so setting GOFLAGS=-mod=vendor"
        export GOFLAGS=-mod=vendor
    fi
    export GO111MODULE=on
    export MODULE_PATH=$(go list .)
    export GOBIN="$HOME/go/bin/$MODULE_PATH"
    export PATH="$GOBIN:$PATH"
    mkdir -p "$GOBIN"
    echo Activating module "$MODULE_PATH" with GOBIN="${GOBIN}"
}

function go-here {
    COMMAND="$1"
    THIS_DIR="$(realpath .)"

    case "$COMMAND" in

    init)

        if [ "$(ls -A "$THIS_DIR")" ]; then
            echo >&2 "Can't init a GOPATH in $THIS_DIR: not empty"
            return 1
        fi

        echo Initializing "$THIS_DIR" as GOPATH

        mkdir -- "$THIS_DIR/bin"
        mkdir -- "$THIS_DIR/src"
        mkdir -- "$THIS_DIR/pkg"

        _go-here "$THIS_DIR"

        INITIAL_PACKAGE="$2"

        if [ "$INITIAL_PACKAGE" != "" ]; then
            echo Getting "$INITIAL_PACKAGE"
            go get -u -t -- "$INITIAL_PACKAGE"
        fi

        return 0

    ;;

    activate)

        while [ "$THIS_DIR" != "/" ]; do
            if [ -d "$THIS_DIR"/src ]; then
                _go-here "$THIS_DIR"
                return 0
            fi
            THIS_DIR="$(dirname -- "$THIS_DIR")"
        done

        >&2 echo Failed to find suitable GOPATH from here
        return 1

    ;;

    mod)

        while [ "$THIS_DIR" != "/" ]; do
            if [ -f "$THIS_DIR"/go.mod ]; then
                _go-here_mod "$THIS_DIR"
                return 0
            fi
            THIS_DIR="$(dirname -- "$THIS_DIR")"
        done

        >&2 echo Failed to find suitable module root from here
        return 1

    ;;

    *)
        >&2 cat <<EOT
Usage: go-here <command>

Commands:
    go-here mod
        Looks for a suitable module root by walking up from the current
        directory until a "go.mod" file is found and then creates a
        project-specific GOBIN directory using the module path.

    go-here init
        Creates an empty GOPATH skeleton in the current directory (which must
        be empty) and immediately activates it.

    go-here init <initial-package>
        Like the plain "go-here init" except also runs "go get" with the
        given package to install an initial package (and its dependencies)
        into the new GOPATH.

    go-here activate
        Looks for a suitable GOPATH by walking up from the current directory
        until a "src" subdirectory is found and then activates the discovered
        directory.

EOT
        return 2
    ;;

    esac

}

Selecting Go Versions

One problem that neither my old or new workflows directly addressed was selecting which version of Go a project is built with. This can be important, because the Go standard library behavior often changes in subtle ways and so when debugging it's helpful to ensure you are using the same version of Go that a release was built from.

My go-here script doesn't do anything about this because I solve this problem using goenv.

My old helper go-here activate actually partially defeated goenv by overriding the GOPATH, but that was okay as long as a particular project stayed on a particular version of Go. Switching to a new version required rebuilding all of the tools in the per-project bin to ensure they were compatible with the new Go version.

The new go-here mod command is designed with goenv in mind, though: it intentionally doesn't touch GOPATH at all so that GOPATH can be a separate directory per Go version: ~/go/1.12.5 rather than just ~/go. Switching a project to a new Go version therefore switches to a different GOPATH too, allowing for a separate set of tools built against that version and separate cache directories per version.

goenv recognizes a file called .go-version in the project root, allowing the project's exact current Go version to be recorded. Most importantly, it's therefore included in release tags so that I can easily discover which Go version I need to best approximate the environment a release was built from when debugging.

The .go-version file is also a good team communication tool: switching to a new Go version can be done via pull-request workflow just like any other change, ensuring that there's an opportunity for team discussion and a mark in the version control history for where the change happened in case it helps with diagnosing any unexpected regressions in subsequent releases.

Is it worth it?

A lot of Go developers don't worry about sharing directories between their projects, and in many cases they've always had a shared GOPATH and bin directory.

As someone who works on lots of different open source projects and will often pivot quickly between them on different days, I find this extra project-switching step valuable to avoid the annoyingly-common situation of switching back to a previous project and finding that it no longer builds or runs exactly as it did before, and then having to track down what has changed since my environment was last working correctly.

There is a danger here, of course: my local development environment for a project contains some artifacts that are not represented in the project's version control system, and thus if I were to lose that directory and have to rebuild it I may still find that the result doesn't match what I previously had.

To avoid this, I do try to still ensure that everything needed to rebuild the environment is captured exactly in the project repository. The problem I'm trying to guard against is returning to an already-configured project environment and assuming it's still matching what was recorded in the repository when actually something has been changed.

Go has no automatic way to specify exact tool dependency versions and verify that they match what is installed, so I would not be informed explicitly if e.g. the wrong version of stringer for the project is now in GOBIN, because it has changed since I originally set up the work tree.

If you only work on one project or on a set of closely-related projects with similar dependencies then separating this is likely overkill. The default Go Modules workflow is still a great improvement over the GOPATH workflow for project separation, with only the bin directory shared across projects.