@nebulon would all binaries not run (eg. /bin/sh from within the base) or just Go binaries that you compiled within the buildx pipeline? If it's the former, it may not be using the right base image. If it's the latter and the former works, perhaps setting the GOARCH variable via a build-arg would solve it.
Note: I personally have not used buildx yet, but from what I can see it's a simpler, automatic version of what I'm trying to do with qemu that handles the manifest for you. So I think you should just be able to build without the muckiness of all the build-args I pass, but if not you can play with mixing those in until it works.
I think buildx is supported on my laptop, so I can give it a try, but it's not supported on my CI box yet, so I haven't switched.