`GOEXPERIMENT=systemcrypto` doesn't override `MS_GO_NOSYSTEMCRYPTO=1`

[bdhill-arista]

Microsoft build of Go version

go1.26.0-1

What is your operating system and platform?

Fedora 43 on ARM64

Output of go env in your module/workspace:

AR='ar'
CC='gcc'
CGO_CFLAGS='-O2
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2
CGO_ENABLED='1'
CGO_FFLAGS='-O2
CGO_LDFLAGS='-O2
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOARCH='arm64'
GOARM64='v8.0'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/root/.cache/ms-go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT='nosystemcrypto'
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC
GOHOSTARCH='arm64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/go/src/gofipstest/go.mod'
GOMODCACHE=''
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH=''
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/root/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/root/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/root/go/pkg/tool/linux_arm64'
GOVCS=''
GOVERSION='go1.26.0'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

Tested the latest Go 1.26.0-1 release with two different configuration methods:

1. GOEXPERIMENT=systemcrypto CGO_ENABLED=1
2. -tags=goexperiment.systemcrypto,cgo

Reproducible with the Dockerfile below.

Dockerfile

FROM fedora:43@sha256:781b7642e8bf256e9cf75d2aa58d86f5cc695fd2df113517614e181a5eee9138

# Install cgo-related dependencies
RUN set -eux; \
    dnf -y update; \
    dnf -y install \
    ca-certificates \
    crypto-policies \
    crypto-policies-scripts \
    gcc-c++ \
    gcc \
    git \
    glibc-devel \
    openssl \
    openssl-libs \
    pkgconf \
    sudo \
    ; \
    dnf clean all

# Install toolchain
ENV PATH="/root/go/bin:${PATH}"
ENV GOROOT="/root/go"
WORKDIR /root
RUN curl -fsSL https://aka.ms/golang/release/latest/go1.26.0-1.linux-arm64.tar.gz | tar -xzf - && \
    go version

# Prepare FIPS config
RUN cat > /etc/pki/tls/openssl_fips.cnf <<EOF
.include /etc/ssl/openssl.cnf

[openssl_init]
providers = provider_sect
alg_section = algorithm_sect

[provider_sect]
fips = fips_sect

[fips_sect]
activate = 1

[algorithm_sect]
default_properties = fips=yes
EOF
# Check it works
RUN OPENSSL_CONF=/etc/pki/tls/openssl_fips.cnf openssl list -providers && openssl md5 /etc/hosts || echo "MD5 disabled as expected in FIPS mode"

# Create test files
WORKDIR /go/src/gofipstest
RUN go mod init gofipstest
COPY <<EOF ./main.go
package main

import (
   "crypto/fips140"
   "crypto/sha1"
   "fmt"
)

func main() {
   sha1.New()
   fmt.Println("   Includes fips.go =", included())
   fmt.Println("   FIPS Mode =", fips140.Enabled())
}
EOF

COPY <<EOF ./fips.go
//go:build goexperiment.opensslcrypto && cgo && linux
package main

func included() bool {
   return true
}
EOF

COPY <<EOF ./nofips.go
//go:build !goexperiment.opensslcrypto || !cgo || !linux
package main

func included() bool {
   return false
}
EOF

# Run tests
ENV MS_GO_NOSYSTEMCRYPTO=1
SHELL [ "/usr/bin/bash", "-c" ]
RUN <<EOF
set -euo pipefail
echo "go env"
go env
echo "--"

echo "TestWithTagsOnly (-tags=goexperiment.systemcrypto,cgo)"
go build -tags=goexperiment.systemcrypto,cgo -o /tmp/gofipstest ./...

echo "-- FIPS Disabled"
/tmp/gofipstest

echo "-- FIPS Enabled"
OPENSSL_CONF=/etc/pki/tls/openssl_fips.cnf GODEBUG=fips140=on /tmp/gofipstest && echo "PASSED" || echo "FAILED"

echo "--"

echo "TestWithEnvsOnly (CGO_ENABLED=1 GOEXPERIMENT=systemcrypto)"
CGO_ENABLED=1 GOEXPERIMENT=systemcrypto go build -o /tmp/gofipstest ./...

echo "-- FIPS Disabled"
/tmp/gofipstest

echo "-- FIPS Enabled"
OPENSSL_CONF=/etc/pki/tls/openssl_fips.cnf GODEBUG=fips140=on /tmp/gofipstest && echo "PASSED" || echo "FAILED"
EOF

What did you see happen?

I noticed that using GOEXPERIMENT environment variable instead of goexperiment build tag results in different FIPS runtime behavior.

1. GOEXPERIMENT=systemcrypto doesn't set the goexperiment.opensslcrypto build tag (could be all goexperiment.* tags, idk).

2. Binaries built with -tags=goexperiment.systemcrypto are able to run in FIPS mode, but binaries built with GOEXPERIMENT=systemcrypto CGO_ENABLED=1 panic instead.

See:

#18 0.110 TestWithTagsOnly (-tags=goexperiment.systemcrypto,cgo)
#18 4.007 -- FIPS Disabled
#18 4.010    Includes fips.go = true
#18 4.010    FIPS Mode = false
#18 4.010 -- FIPS Enabled
#18 4.016    Includes fips.go = true
#18 4.016    FIPS Mode = true
#18 4.016 PASSED
#18 4.016 --
#18 4.016 TestWithEnvsOnly (CGO_ENABLED=1 GOEXPERIMENT=systemcrypto)
#18 5.992 -- FIPS Disabled
#18 5.995    Includes fips.go = false
#18 5.995    FIPS Mode = false
#18 5.996 -- FIPS Enabled
#18 6.000 panic: FIPS mode requested (environment variable GODEBUG=fips140=on) but no supported crypto backend is enabled
#18 6.000
#18 6.000 goroutine 1 [running]:
#18 6.000 crypto/internal/backend.init.0()
#18 6.000 	/root/go/src/crypto/internal/backend/nobackend.go:19 +0xc0
#18 6.002 FAILED

What did you expect to see?

I expected them to behave the same way in both cases.

[bdhill-arista]

Looks like its because of MS_GO_NOSYSTEMCRYPTO=1

If I build with: MS_GO_NOSYSTEMCRYPTO=0 CGO_ENABLED=1 GOEXPERIMENT=systemcrypto go build -o /tmp/gofipstest ./... then it works as expected.

[bdhill-arista]

I assumed MS_GO_NOSYSTEMCRYPTO=1 would be overridden by GOEXPERIMENT=systemcrypto, but doesn't seem to be the case.

From https://github.com/microsoft/go/blob/microsoft/release-branch.go1.26/docs/go1.26.md#configuration

You can disable systemcrypto at build time by setting the environment variable MS_GO_NOSYSTEMCRYPTO to 1. This is now the preferred method for disabling systemcrypto when necessary.

I was using MS_GO_NOSYSTEMCRYPTO=1 as a drop-in replacement for GOEXPERIMENT=nosystemcrypto, so the change in behavior is unexpected.

[dagood]

I was using MS_GO_NOSYSTEMCRYPTO=1 as a drop-in replacement for GOEXPERIMENT=nosystemcrypto, so the change in behavior is unexpected.

Just to make sure it's clear: you can continue to use GOEXPERIMENT=nosystemcrypto. We specifically added MS_GO_NOSYSTEMCRYPTO for use cases where setting GOEXPERIMENT isn't reasonable.

It's true that we do recommend MS_GO_NOSYSTEMCRYPTO over changing GOEXPERIMENT, but only because it's simpler (for fresh use cases) and more compatible with some use cases.

Looks like its because of MS_GO_NOSYSTEMCRYPTO=1

If I build with: MS_GO_NOSYSTEMCRYPTO=0 CGO_ENABLED=1 GOEXPERIMENT=systemcrypto go build -o /tmp/gofipstest ./... then it works as expected.

Yes, and as of 1.25 changing systemcrypto to be enabled by default, you should actually be able to do this:

export MS_GO_NOSYSTEMCRYPTO=1
MS_GO_NOSYSTEMCRYPTO=0 go build ./...

---

I actually hit a similar problem you did, and I tried to update the docs to clarify:

• Docs: clarify interaction of backend env vars; add more links #2007

https://github.com/microsoft/go/blob/microsoft/main/eng/doc/fips/README.md#build-option-to-use-go-crypto-if-the-backend-compatibility-check-fails

Note

MS_GO_NOSYSTEMCRYPTO=1 has precedence over GOEXPERIMENT values. For example, setting MS_GO_NOSYSTEMCRYPTO=1 and GOEXPERIMENT=systemcrypto builds a program that uses Go standard library cryptography.

Unfortunately, the release notes are (intentionally) terse and don't get into the interaction. I think we can at least add a link, though.

[bdhill-arista]

Thanks! So basically PEBKAC on my end :')

Just to make sure it's clear: you can continue to use GOEXPERIMENT=nosystemcrypto. We specifically added MS_GO_NOSYSTEMCRYPTO for use cases where setting GOEXPERIMENT isn't reasonable.

Ah okay, to recap:

1. GOEXPERIMENT=systemcrypto is enabled by default
2. GOEXPERIMENT=nosystemcrypto can disable it, but causes upstream Go builds to fail with go: unknown GOEXPERIMENT systemcrypto
3. MS_GO_NOSYSTEMCRYPTO=1 will unconditionally disable GOEXPERIMENT without impacting upstream Go builds

IIUC, the build tags are distinct from GOEXPERIMENT values, and cannot be disabled with Go-FIPS environment settings?

I actually hit a similar problem you did, and I tried to update the docs to clarify:

Thanks for the link! :)

From reading the changelog, it sounds like platform-specific build tags will remain supported, but GOEXPERIMENT will converge to systemcrypto?

The per-platform GOEXPERIMENTs (opensslcrypto, cngcrypto, darwincrypto) have been removed.
The systemcrypto GOEXPERIMENT is now enabled by default on all platforms where it's supported (Windows, Linux and macOS). To disable it, set GOEXPERIMENT=nosystemcrypto.
Using any of the removed experiments will result in a build error.

[dagood]

So basically PEBKAC on my end :')

Well, when we talked about this today as a team, we're actually considering changing it. 😄 In general, we're aiming to avoid any situation where it seems like systemcrypto should be enabled, but it ends up not being enabled--clear footgun. Of course, changing it means breaking someone else's hypothetically valid scenario, so we have some weighing to do.

Ah okay, to recap: [...]

That looks right to me, maybe with the small note that technically the GOEXPERIMENT environment variable isn't set to some value by default, rather it's handled internally.

Funnily enough, though, MS_GO_NOSYSTEMCRYPTO=1 is currently handled by detecting the env and then simply appending nosystemcrypto to the GOEXPERIMENT env var before it's processed. (So I believe your scenario ends up as if you're setting GOEXPERIMENT=systemcrypto,nosystemcrypto.)

IIUC, the build tags are distinct from GOEXPERIMENT values, and cannot be disabled with Go-FIPS environment settings?

Yes, they're distinct, and the way I'd put it is that the build process looks at the final GOEXPERIMENT setting and sets (or doesn't set) some build tags. After that, there simply isn't a way to remove specific build tags by appending something to a build command, systemcrypto-related or not. The "no" prefix is only a GOEXPERIMENT settings feature.

From reading the changelog, it sounds like platform-specific build tags will remain supported, but GOEXPERIMENT will converge to systemcrypto?

I'm actually not sure if that's the intent. The experiment code doc in main (1.27) says:

+	// SystemCrypto enables the system crypto backend for the target GOOS.
+	// The specific backend used depends on the platform:
+	//   - Linux: OpenSSL
+	//   - Windows: CNG
+	//   - Darwin: CommonCrypto/Security.framework
+	//
+	// Platform-specific code should use build constraints like:
+	//   //go:build goexperiment.systemcrypto && linux
+	//   //go:build goexperiment.systemcrypto && windows
+	//   //go:build goexperiment.systemcrypto && darwin
+	SystemCrypto bool

@gdams, @qmuntal, can you clarify if we plan to keep backend-specific tags?

[qmuntal]

@gdams, @qmuntal, can you clarify if we plan to keep backend-specific tags?

We won't keep backend-specific tags. In fact, they will be removed in Go 1.27.

[dagood]

Thanks, submitted a few PRs to try to help with this going forward:

• docs: rework "Build option to use Go crypto" #2173
• [release-branch.go1.26] Refer to FIPS 1.26 changelog, remove unclear statement #2174