unix: generate all Linux go files from source

Right now the process for adding in new constants, errors, or syscalls
for Linux is a pain and unreliable. The scripts are designed to be run
on the target architecture and use the header files installed on the
user's system. This makes it hard to generate files for all the
architectures or to have consistency between users. See golang/go#15282.

This CL fixes this issue by making all of the files for the 11 supported
architectures directly from source checkouts of Linux, glibc, and bluez.
This is done using Docker, the gcc cross-compilers, and qemu emulation.
Previously discussed here:
    https://go-review.googlesource.com/c/37589/

A README.md file is also added to explain how all the parts of the build
system work.

In order to get the build working for all the architectures, I made
some changes to the other scripts called from mkall_linux.go:
  - Files only used for generating linux code, moved to linux/
  - linux/mksysnum.pl supports a specified CC compiler.
  - The generated C code in mkerrors.sh changed to avoid a warning
  - mkerrors.sh headers changed to fix powerpc64 bug in sys/ioctl.h
  - linux/types.go no longer needs to export Ptrace structs in lowercase

Build instructions:
  - Host system needs to be x86-64 Linux
  - Install Docker (https://docs.docker.com/engine/installation/)
  - ./mkall.sh (That's it!!!)

Change-Id: I87067c14442ba12f8d51991349a43a9d73f38ae0
Reviewed-on: https://go-review.googlesource.com/37943Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>

unix/README.md

+# Building `sys/unix`
+
+The sys/unix package provides access to the raw system call interface of the
+underlying operating system. See: https://godoc.org/golang.org/x/sys/unix
+
+Porting Go to a new architecture/OS combination or adding syscalls, types, or
+constants to an existing architecture/OS pair requires some manual effort;
+however, there are tools that automate much of the process.
+
+## Build Systems
+
+There are currently two ways we generate the necessary files. We are currently
+migrating the build system to use containers so the builds are reproducible.
+This is being done on an OS-by-OS basis. Please update this documentation as
+components of the build system change.
+
+### Old Build System (currently for `GOOS != "Linux" || GOARCH == "sparc64"`)
+
+The old build system generates the Go files based on the C header files
+present on your system. This means that files
+for a given GOOS/GOARCH pair must be generated on a system with that OS and
+architecture. This also means that the generated code can differ from system
+to system, based on differences in the header files.
+
+To avoid this, if you are using the old build system, only generate the Go
+files on an installation with unmodified header files. It is also important to
+keep track of which version of the OS the files were generated from (ex.
+Darwin 14 vs Darwin 15). This makes it easier to track the progress of changes
+and have each OS upgrade correspond to a single change.
+
+To build the files for your current OS and architecture, make sure GOOS and
+GOARCH are set correctly and run `mkall.sh`. This will generate the files for
+your specific system. Running `mkall.sh -n` shows the commands that will be run.
+
+Requirements: bash, perl, go
+
+### New Build System (currently for `GOOS == "Linux" && GOARCH != "sparc64"`)
+
+The new build system uses a Docker container to generate the go files directly
+from source checkouts of the kernel and various system libraries. This means
+that on any platform that supports Docker, all the files using the new build
+system can be generated at once, and generated files will not change based on
+what the person running the scripts has installed on their computer.
+
+The OS specific files for the new build system are located in the `${GOOS}`
+directory, and the build is coordinated by the `${GOOS}/mkall.go` program. When
+the kernel or system library updates, modify the Dockerfile at
+`${GOOS}/Dockerfile` to checkout the new release of the source.
+
+To build all the files under the new build system, you must be on an amd64/Linux
+system and have your GOOS and GOARCH set accordingly. Running `mkall.sh` will
+then generate all of the files for all of the GOOS/GOARCH pairs in the new build
+system. Running `mkall.sh -n` shows the commands that will be run.
+
+Requirements: bash, perl, go, docker
+
+## Component files
+
+This section describes the various files used in the code generation process.
+It also contains instructions on how to modify these files to add a new
+architecture/OS or to add additional syscalls, types, or constants. Note that
+if you are using the new build system, the scripts cannot be called normally.
+They must be called from within the docker container.
+
+### asm files
+
+The hand-written assembly file at `asm_${GOOS}_${GOARCH}.s` implements system
+call dispatch. There are three entry points:
+```
+  func Syscall(trap, a1, a2, a3 uintptr) (r1, r2, err uintptr)
+  func Syscall6(trap, a1, a2, a3, a4, a5, a6 uintptr) (r1, r2, err uintptr)
+  func RawSyscall(trap, a1, a2, a3 uintptr) (r1, r2, err uintptr)
+```
+The first and second are the standard ones; they differ only in how many
+arguments can be passed to the kernel. The third is for low-level use by the
+ForkExec wrapper. Unlike the first two, it does not call into the scheduler to
+let it know that a system call is running.
+
+When porting Go to an new architecture/OS, this file must be implemented for
+each GOOS/GOARCH pair.
+
+### mksysnum
+
+Mksysnum is a script located at `${GOOS}/mksysnum.pl` (or `mksysnum_${GOOS}.pl`
+for the old system). This script takes in a list of header files containing the
+syscall number declarations and parses them to produce the corresponding list of
+Go numeric constants. See `zsysnum_${GOOS}_${GOARCH}.go` for the generated
+constants.
+
+Adding new syscall numbers is mostly done by running the build on a sufficiently
+new installation of the target OS (or updating the source checkouts for the
+new build system). However, depending on the OS, you make need to update the
+parsing in mksysnum.
+
+### mksyscall.pl
+
+The `syscall.go`, `syscall_${GOOS}.go`, `syscall_${GOOS}_${GOARCH}.go` are
+hand-written Go files which implement system calls (for unix, the specific OS,
+or the specific OS/Architecture pair respectively) that need special handling
+and list `//sys` comments giving prototypes for ones that can be generated.
+
+The mksyscall.pl script takes the `//sys` and `//sysnb` comments and converts
+them into syscalls. This requires the name of the prototype in the comment to
+match a syscall number in the `zsysnum_${GOOS}_${GOARCH}.go` file. The function
+prototype can be exported (capitalized) or not.
+
+Adding a new syscall often just requires adding a new `//sys` function prototype
+with the desired arguments and a capitalized name so it is exported. However, if
+you want the interface to the syscall to be different, often one will make an
+unexported `//sys` prototype, an then write a custom wrapper in
+`syscall_${GOOS}.go`.
+
+### types files
+
+For each OS, there is a hand-written Go file at `${GOOS}/types.go` (or
+`types_${GOOS}.go` on the old system). This file includes standard C headers and
+creates Go type aliases to the corresponding C types. The file is then fed
+through godef to get the Go compatible definitions. Finally, the generated code
+is fed though mkpost.go to format the code correctly and remove any hidden or
+private identifiers. This cleaned-up code is written to
+`ztypes_${GOOS}_${GOARCH}.go`.
+
+The hardest part about preparing this file is figuring out which headers to
+include and which symbols need to be `#define`d to get the actual data
+structures that pass through to the kernel system calls. Some C libraries
+preset alternate versions for binary compatibility and translate them on the
+way in and out of system calls, but there is almost always a `#define` that can
+get the real ones.
+See `types_darwin.go` and `linux/types.go` for examples.
+
+To add a new type, add in the necessary include statement at the top of the
+file (if it is not already there) and add in a type alias line. Note that if
+your type is significantly different on different architectures, you may need
+some `#if/#elif` macros in your include statements.
+
+### mkerrors.sh
+
+This script is used to generate the system's various constants. This doesn't
+just include the error numbers and error strings, but also the signal numbers
+an a wide variety of miscellaneous constants. The constants come from the list
+of include files in the `includes_${uname}` variable. A regex then picks out
+the desired `#define` statements, and generates the corresponding Go constants.
+The error numbers and strings are generated from `#include <errno.h>`, and the
+signal numbers and strings are generated from `#include <signal.h>`. All of
+these constants are written to `zerrors_${GOOS}_${GOARCH}.go` via a C program,
+`_errors.c`, which prints out all the constants.
+
+To add a constant, add the header that includes it to the appropriate variable.
+Then, edit the regex (if necessary) to match the desired constant. Avoid making
+the regex too broad to avoid matching unintended constants.
+
+
+## Generated files
+
+### `zerror_${GOOS}_${GOARCH}.go`
+
+A file containing all of the system's generated error numbers, error strings,
+signal numbers, and constants. Generated by `mkerrors.sh` (see above).
+
+### `zsyscall_${GOOS}_${GOARCH}.go`
+
+A file containing all the generated syscalls for a specific GOOS and GOARCH.
+Generated by `mksyscall.pl` (see above).
+
+### `zsysnum_${GOOS}_${GOARCH}.go`
+
+A list of numeric constants for all the syscall number of the specific GOOS
+and GOARCH. Generated by mksysnum (see above).
+
+### `ztypes_${GOOS}_${GOARCH}.go`
+
+A file containing Go types for passing into (or returning from) syscalls.
+Generated by godefs and the types file (see above).

unix/linux/Dockerfile

+FROM ubuntu:16.04
+
+# Dependencies to get the git sources and go binaries
+RUN apt-get update && apt-get install -y \
+        curl \
+        git \
+    && rm -rf /var/lib/apt/lists/*
+
+# Get the git sources. If not cached, this takes O(5 minutes).
+WORKDIR /git
+RUN git config --global advice.detachedHead false
+# Linux Kernel: Released 19 Feb 2017
+RUN git clone --branch v4.10 --depth 1 https://kernel.googlesource.com/pub/scm/linux/kernel/git/torvalds/linux
+# GNU C library: Released 05 Feb 2017 (we should try to get a secure way to clone this)
+RUN git clone --branch glibc-2.25 --depth 1 git://sourceware.org/git/glibc.git
+
+# Get Go 1.8 (https://github.com/docker-library/golang/blob/master/1.8/Dockerfile)
+ENV GOLANG_VERSION 1.8
+ENV GOLANG_DOWNLOAD_URL https://golang.org/dl/go$GOLANG_VERSION.linux-amd64.tar.gz
+ENV GOLANG_DOWNLOAD_SHA256 53ab94104ee3923e228a2cb2116e5e462ad3ebaeea06ff04463479d7f12d27ca
+
+RUN curl -fsSL "$GOLANG_DOWNLOAD_URL" -o golang.tar.gz \
+    && echo "$GOLANG_DOWNLOAD_SHA256  golang.tar.gz" | sha256sum -c - \
+    && tar -C /usr/local -xzf golang.tar.gz \
+    && rm golang.tar.gz
+
+ENV PATH /usr/local/go/bin:$PATH
+
+# Linux and Glibc build dependencies
+RUN apt-get update && apt-get install -y \
+        gawk make python \
+        gcc gcc-multilib \
+        gettext  texinfo \
+    && rm -rf /var/lib/apt/lists/*
+# Emulator and cross compilers
+RUN apt-get update && apt-get install -y \
+        qemu \
+        gcc-aarch64-linux-gnu       gcc-arm-linux-gnueabi     \
+        gcc-mips-linux-gnu          gcc-mips64-linux-gnuabi64 \
+        gcc-mips64el-linux-gnuabi64 gcc-mipsel-linux-gnu      \
+        gcc-powerpc64-linux-gnu     gcc-powerpc64le-linux-gnu \
+        gcc-s390x-linux-gnu         gcc-sparc64-linux-gnu     \
+    && rm -rf /var/lib/apt/lists/*
+
+# Let the scripts know they are in the docker environment
+ENV GOLANG_SYS_BUILD docker
+WORKDIR /build
+ENTRYPOINT ["go", "run", "linux/mkall.go", "/git/linux", "/git/glibc"]

unix/mksysnum_linux.pl → unix/linux/mksysnum.pl

@@ -10,11 +10,18 @@ if($ENV{'GOARCH'} eq "" || $ENV{'GOOS'} eq "") {
 	exit 1;
 }
 
-my $command = "mksysnum_linux.pl ". join(' ', @ARGV);
+# Check that we are using the new build system if we should
+if($ENV{'GOLANG_SYS_BUILD'} ne "docker") {
+	print STDERR "In the new build system, mksysnum should not be called directly.\n";
+	print STDERR "See README.md\n";
+	exit 1;
+}
+
+my $command = "$0 ". join(' ', @ARGV);
 
 print <<EOF;
 // $command
-// MACHINE GENERATED BY THE ABOVE COMMAND; DO NOT EDIT
+// Code generated by the command above; see README.md. DO NOT EDIT.
 
 // +build $ENV{'GOARCH'},$ENV{'GOOS'}
 
@@ -38,8 +45,8 @@ sub fmt {
 }
 
 my $prev;
-open(GCC, "gcc -E -dD @ARGV |") || die "can't run gcc";
-while(<GCC>){
+open(CC, "$ENV{'CC'} -E -dD @ARGV |") || die "can't run $ENV{'CC'}";
+while(<CC>){
 	if(/^#define __NR_Linux\s+([0-9]+)/){
 		# mips/mips64: extract offset
 		$offset = $1;

unix/types_linux.go → unix/linux/types.go

@@ -5,7 +5,7 @@
 // +build ignore
 
 /*
-Input to cgo -godefs.  See also mkerrors.sh and mkall.sh
+Input to cgo -godefs.  See README.md
 */
 
 // +godefs map struct_in_addr [4]byte /* in_addr */
@@ -20,7 +20,6 @@ package unix
 #define _GNU_SOURCE
 
 #include <dirent.h>
-#include <fcntl.h>
 #include <netinet/in.h>
 #include <netinet/tcp.h>
 #include <netpacket/packet.h>
@@ -36,13 +35,11 @@ package unix
 #include <sys/resource.h>
 #include <sys/select.h>
 #include <sys/signal.h>
-#include <sys/stat.h>
 #include <sys/statfs.h>
 #include <sys/sysinfo.h>
 #include <sys/time.h>
 #include <sys/times.h>
 #include <sys/timex.h>
-#include <sys/types.h>
 #include <sys/un.h>
 #include <sys/user.h>
 #include <sys/utsname.h>
@@ -52,16 +49,64 @@ package unix
 #include <linux/rtnetlink.h>
 #include <linux/icmpv6.h>
 #include <asm/termbits.h>
+#include <asm/ptrace.h>
 #include <time.h>
 #include <unistd.h>
 #include <ustat.h>
 #include <utime.h>
-#include <bluetooth/bluetooth.h>
-#include <bluetooth/hci.h>
 #include <linux/can.h>
 #include <linux/if_alg.h>
 #include <linux/vm_sockets.h>
 
+// On mips64, the glibc stat and kernel stat do not agree
+#if (defined(__mips__) && _MIPS_SIM == _MIPS_SIM_ABI64)
+
+// Use the stat defined by the kernel with a few modifications. These are:
+//	* The time fields (like st_atime and st_atimensec) use the timespec
+//	  struct (like st_atim) for consitancy with the glibc fields.
+//	* The padding fields get different names to not break compatibility.
+//	* st_blocks is signed, again for compatibility.
+struct stat {
+	unsigned int		st_dev;
+	unsigned int		st_pad1[3]; // Reserved for st_dev expansion
+
+	unsigned long		st_ino;
+
+	mode_t			st_mode;
+	__u32			st_nlink;
+
+	uid_t			st_uid;
+	gid_t			st_gid;
+
+	unsigned int		st_rdev;
+	unsigned int		st_pad2[3]; // Reserved for st_rdev expansion
+
+	off_t			st_size;
+
+	// These are declared as speperate fields in the kernel. Here we use
+	// the timespec struct for consistancy with the other stat structs.
+	struct timespec		st_atim;
+	struct timespec		st_mtim;
+	struct timespec		st_ctim;
+
+	unsigned int		st_blksize;
+	unsigned int		st_pad4;
+
+	long			st_blocks;
+};
+
+// These are needed because we do not include fcntl.h or sys/types.h
+#include <linux/fcntl.h>
+#include <linux/fadvise.h>
+
+#else
+
+// Use the stat defined by glibc
+#include <fcntl.h>
+#include <sys/types.h>
+
+#endif
+
 #ifdef TCSETS2
 // On systems that have "struct termios2" use this as type Termios.
 typedef struct termios2 termios_t;
@@ -87,6 +132,13 @@ struct sockaddr_any {
 	char pad[sizeof(union sockaddr_all) - sizeof(struct sockaddr)];
 };
 
+// copied from /usr/include/bluetooth/hci.h
+struct sockaddr_hci {
+        sa_family_t     hci_family;
+        unsigned short  hci_dev;
+        unsigned short  hci_channel;
+};;
+
 // copied from /usr/include/linux/un.h
 struct my_sockaddr_un {
 	sa_family_t sun_family;
@@ -421,11 +473,11 @@ const SizeofInotifyEvent = C.sizeof_struct_inotify_event
 type PtraceRegs C.PtraceRegs
 
 // Structures contained in PtraceRegs on s390x (exported by mkpost.go)
-type ptracePsw C.ptracePsw
+type PtracePsw C.ptracePsw
 
-type ptraceFpregs C.ptraceFpregs
+type PtraceFpregs C.ptraceFpregs
 
-type ptracePer C.ptracePer
+type PtracePer C.ptracePer
 
 // Misc
 

unix/mkall.sh

@@ -3,75 +3,9 @@
 # Use of this source code is governed by a BSD-style
 # license that can be found in the LICENSE file.
 
-# The unix package provides access to the raw system call
-# interface of the underlying operating system.  Porting Go to
-# a new architecture/operating system combination requires
-# some manual effort, though there are tools that automate
-# much of the process.  The auto-generated files have names
-# beginning with z.
-#
-# This script runs or (given -n) prints suggested commands to generate z files
-# for the current system.  Running those commands is not automatic.
-# This script is documentation more than anything else.
-#
-# * asm_${GOOS}_${GOARCH}.s
-#
-# This hand-written assembly file implements system call dispatch.
-# There are three entry points:
-#
-# 	func Syscall(trap, a1, a2, a3 uintptr) (r1, r2, err uintptr);
-# 	func Syscall6(trap, a1, a2, a3, a4, a5, a6 uintptr) (r1, r2, err uintptr);
-# 	func RawSyscall(trap, a1, a2, a3 uintptr) (r1, r2, err uintptr);
-#
-# The first and second are the standard ones; they differ only in
-# how many arguments can be passed to the kernel.
-# The third is for low-level use by the ForkExec wrapper;
-# unlike the first two, it does not call into the scheduler to
-# let it know that a system call is running.
-#
-# * syscall_${GOOS}.go
-#
-# This hand-written Go file implements system calls that need
-# special handling and lists "//sys" comments giving prototypes
-# for ones that can be auto-generated.  Mksyscall reads those
-# comments to generate the stubs.
-#
-# * syscall_${GOOS}_${GOARCH}.go
-#
-# Same as syscall_${GOOS}.go except that it contains code specific
-# to ${GOOS} on one particular architecture.
-#
-# * types_${GOOS}.c
-#
-# This hand-written C file includes standard C headers and then
-# creates typedef or enum names beginning with a dollar sign
-# (use of $ in variable names is a gcc extension).  The hardest
-# part about preparing this file is figuring out which headers to
-# include and which symbols need to be #defined to get the
-# actual data structures that pass through to the kernel system calls.
-# Some C libraries present alternate versions for binary compatibility
-# and translate them on the way in and out of system calls, but
-# there is almost always a #define that can get the real ones.
-# See types_darwin.c and types_linux.c for examples.
-#
-# * zerror_${GOOS}_${GOARCH}.go
-#
-# This machine-generated file defines the system's error numbers,
-# error strings, and signal numbers.  The generator is "mkerrors.sh".
-# Usually no arguments are needed, but mkerrors.sh will pass its
-# arguments on to godefs.
-#
-# * zsyscall_${GOOS}_${GOARCH}.go
-#
-# Generated by mksyscall.pl; see syscall_${GOOS}.go above.
-#
-# * zsysnum_${GOOS}_${GOARCH}.go
-#
-# Generated by mksysnum_${GOOS}.
-#
-# * ztypes_${GOOS}_${GOARCH}.go
-#
-# Generated by godefs; see types_${GOOS}.c above.
+# This script runs or (given -n) prints suggested commands to generate files for
+# the Architecture/OS specified by the GOARCH and GOOS environment variables.
+# See README.md for more information about how the build system works.
 
 GOOSARCH="${GOOS}_${GOARCH}"
 
@@ -84,6 +18,7 @@ zsysctl="zsysctl_$GOOSARCH.go"
 mksysnum=
 mktypes=
 run="sh"
+cmd=""
 
 case "$1" in
 -syscalls)
@@ -98,6 +33,7 @@ case "$1" in
 	;;
 -n)
 	run="cat"
+	cmd="echo"
 	shift
 esac
 
@@ -109,6 +45,14 @@ case "$#" in
 	exit 2
 esac
 
+if [[ "$GOOS" -eq "linux" ]] && [[ "$GOARCH" != "sparc64" ]]; then
+	# Use then new build system
+	# Files generated through docker (use $cmd so you can Ctl-C the build or run)
+	$cmd docker build --tag generate:$GOOS $GOOS
+	$cmd docker run --interactive --tty --volume $(dirname "$(readlink -f "$0")"):/build generate:$GOOS
+	exit
+fi
+
 GOOSARCH_in=syscall_$GOOSARCH.go
 case "$GOOSARCH" in
 _* | *_ | _)
@@ -167,64 +111,6 @@ freebsd_arm)
 	# API consistent across over platforms.
 	mktypes="GOARCH=$GOARCH go tool cgo -godefs -- -fsigned-char"
 	;;
-linux_386)
-	mkerrors="$mkerrors -m32"
-	mksyscall="./mksyscall.pl -l32"
-	mksysnum="./mksysnum_linux.pl /usr/include/asm/unistd_32.h"
-	mktypes="GOARCH=$GOARCH go tool cgo -godefs"
-	;;
-linux_amd64)
-	unistd_h=$(ls -1 /usr/include/asm/unistd_64.h /usr/include/x86_64-linux-gnu/asm/unistd_64.h 2>/dev/null | head -1)
-	if [ "$unistd_h" = "" ]; then
-		echo >&2 cannot find unistd_64.h
-		exit 1
-	fi
-	mkerrors="$mkerrors -m64"
-	mksysnum="./mksysnum_linux.pl $unistd_h"
-	mktypes="GOARCH=$GOARCH go tool cgo -godefs"
-	;;
-linux_arm)
-	mkerrors="$mkerrors"
-	mksyscall="./mksyscall.pl -l32 -arm"
-	mksysnum="curl -s 'http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/plain/arch/arm/include/uapi/asm/unistd.h' | ./mksysnum_linux.pl -"
-	mktypes="GOARCH=$GOARCH go tool cgo -godefs"
-	;;
-linux_arm64)
-	unistd_h=$(ls -1 /usr/include/asm/unistd.h /usr/include/asm-generic/unistd.h 2>/dev/null | head -1)
-	if [ "$unistd_h" = "" ]; then
-		echo >&2 cannot find unistd_64.h
-		exit 1
-	fi
-	mksysnum="./mksysnum_linux.pl $unistd_h"
-	# Let the type of C char be signed for making the bare syscall
-	# API consistent across over platforms.
-	mktypes="GOARCH=$GOARCH go tool cgo -godefs -- -fsigned-char"
-	;;
-linux_ppc64)
-	GOOSARCH_in=syscall_linux_ppc64x.go
-	unistd_h=/usr/include/asm/unistd.h
-	mkerrors="$mkerrors -m64"
-	mksysnum="./mksysnum_linux.pl $unistd_h"
-	mktypes="GOARCH=$GOARCH go tool cgo -godefs"
-	;;
-linux_ppc64le)
-	GOOSARCH_in=syscall_linux_ppc64x.go
-	unistd_h=/usr/include/powerpc64le-linux-gnu/asm/unistd.h
-	mkerrors="$mkerrors -m64"
-	mksysnum="./mksysnum_linux.pl $unistd_h"
-	mktypes="GOARCH=$GOARCH go tool cgo -godefs"
-	;;
-linux_s390x)
-	GOOSARCH_in=syscall_linux_s390x.go
-	unistd_h=/usr/include/asm/unistd.h
-	mkerrors="$mkerrors -m64"
-	mksysnum="./mksysnum_linux.pl $unistd_h"
-	# Let the type of C char be signed to make the bare sys
-	# API more consistent between platforms.
-	# This is a deliberate departure from the way the syscall
-	# package generates its version of the types file.
-	mktypes="GOARCH=$GOARCH go tool cgo -godefs -- -fsigned-char"
-	;;
 linux_sparc64)
 	GOOSARCH_in=syscall_linux_sparc64.go
 	unistd_h=/usr/include/sparc64-linux-gnu/asm/unistd.h
@@ -288,7 +174,6 @@ esac
 	if [ -n "$mksysctl" ]; then echo "$mksysctl |gofmt >$zsysctl"; fi
 	if [ -n "$mksysnum" ]; then echo "$mksysnum |gofmt >zsysnum_$GOOSARCH.go"; fi
 	if [ -n "$mktypes" ]; then
-		echo "echo // +build $GOARCH,$GOOS > ztypes_$GOOSARCH.go";
-		echo "$mktypes types_$GOOS.go | go run mkpost.go >>ztypes_$GOOSARCH.go";
+		echo "$mktypes types_$GOOS.go | go run mkpost.go > ztypes_$GOOSARCH.go";
 	fi
 ) | $run

unix/mkerrors.sh

@@ -16,6 +16,15 @@ if test -z "$GOARCH" -o -z "$GOOS"; then
 	exit 1
 fi
 
+# Check that we are using the new build system if we should
+if [[ "$GOOS" -eq "linux" ]] && [[ "$GOARCH" != "sparc64" ]]; then
+	if [[ "$GOLANG_SYS_BUILD" -ne "docker" ]]; then
+		echo 1>&2 "In the new build system, mkerrors should not be called directly."
+		echo 1>&2 "See README.md"
+		exit 1
+	fi
+fi
+
 CC=${CC:-cc}
 
 if [[ "$GOOS" -eq "solaris" ]]; then
@@ -102,6 +111,36 @@ includes_Linux='
 #endif
 #define _GNU_SOURCE
 
+// <sys/ioctl.h> is broken on powerpc64, as it fails to include definitions of
+// these structures. We just include them copied from <bits/termios.h>.
+#if defined(__powerpc__)
+struct sgttyb {
+        char    sg_ispeed;
+        char    sg_ospeed;
+        char    sg_erase;
+        char    sg_kill;
+        short   sg_flags;
+};
+
+struct tchars {
+        char    t_intrc;
+        char    t_quitc;
+        char    t_startc;
+        char    t_stopc;
+        char    t_eofc;
+        char    t_brkc;
+};
+
+struct ltchars {
+        char    t_suspc;
+        char    t_dsuspc;
+        char    t_rprntc;
+        char    t_flushc;
+        char    t_werasc;
+        char    t_lnextc;
+};
+#endif
+
 #include <bits/sockaddr.h>
 #include <sys/epoll.h>
 #include <sys/inotify.h>
@@ -391,7 +430,7 @@ echo '#include <signal.h>' | $CC -x c - -E -dM $ccflags |
 	sort >_signal.grep
 
 echo '// mkerrors.sh' "$@"
-echo '// MACHINE GENERATED BY THE COMMAND ABOVE; DO NOT EDIT'
+echo '// Code generated by the command above; see README.md. DO NOT EDIT.'
 echo
 echo "// +build ${GOARCH},${GOOS}"
 echo
@@ -453,7 +492,7 @@ intcmp(const void *a, const void *b)
 int
 main(void)
 {
-	int i, j, e;
+	int i, e;
 	char buf[1024], *p;
 
 	printf("\n\n// Error table\n");

unix/mkpost.go

@@ -8,10 +8,11 @@
 // modify the generated types. It is used to clean up
 // the sys API in an architecture specific manner.
 //
-// mkpost is run after cgo -godefs by mkall.sh.
+// mkpost is run after cgo -godefs; see README.md.
 package main
 
 import (
+	"bytes"
 	"fmt"
 	"go/format"
 	"io/ioutil"
@@ -21,42 +22,67 @@ import (
 )
 
 func main() {
+	// Get the OS and architecture (using GOARCH_TARGET if it exists)
+	goos := os.Getenv("GOOS")
+	goarch := os.Getenv("GOARCH_TARGET")
+	if goarch == "" {
+		goarch = os.Getenv("GOARCH")
+	}
+	// Check that we are using the new build system if we should be.
+	if goos == "linux" && goarch != "sparc64" {
+		if os.Getenv("GOLANG_SYS_BUILD") != "docker" {
+			os.Stderr.WriteString("In the new build system, mkpost should not be called directly.\n")
+			os.Stderr.WriteString("See README.md\n")
+			os.Exit(1)
+		}
+	}
+
 	b, err := ioutil.ReadAll(os.Stdin)
 	if err != nil {
 		log.Fatal(err)
 	}
-	s := string(b)
 
-	goarch := os.Getenv("GOARCH")
-	goos := os.Getenv("GOOS")
-	if goarch == "s390x" && goos == "linux" {
-		// Export the types of PtraceRegs fields.
-		re := regexp.MustCompile("ptrace(Psw|Fpregs|Per)")
-		s = re.ReplaceAllString(s, "Ptrace$1")
+	// If we have empty Ptrace structs, we should delete them. Only s390x emits
+	// nonempty Ptrace structs.
+	ptraceRexexp := regexp.MustCompile(`type Ptrace((Psw|Fpregs|Per) struct {\s*})`)
+	b = ptraceRexexp.ReplaceAll(b, nil)
 
-		// Replace padding fields inserted by cgo with blank identifiers.
-		re = regexp.MustCompile("Pad_cgo[A-Za-z0-9_]*")
-		s = re.ReplaceAllString(s, "_")
+	// Replace the control_regs union with a blank identifier for now.
+	controlRegsRegex := regexp.MustCompile(`(Control_regs)\s+\[0\]uint64`)
+	b = controlRegsRegex.ReplaceAll(b, []byte("_ [0]uint64"))
 
-		// Replace other unwanted fields with blank identifiers.
-		re = regexp.MustCompile("X_[A-Za-z0-9_]*")
-		s = re.ReplaceAllString(s, "_")
+	// Remove fields that are added by glibc
+	// Note that this is unstable as the identifers are private.
+	removeFieldsRegex := regexp.MustCompile(`X__glibc\S*`)
+	b = removeFieldsRegex.ReplaceAll(b, []byte("_"))
 
-		// Replace the control_regs union with a blank identifier for now.
-		re = regexp.MustCompile("(Control_regs)\\s+\\[0\\]uint64")
-		s = re.ReplaceAllString(s, "_ [0]uint64")
+	// We refuse to export private fields on s390x
+	if goarch == "s390x" && goos == "linux" {
+		// Remove cgo padding fields
+		removeFieldsRegex := regexp.MustCompile(`Pad_cgo_\d+`)
+		b = removeFieldsRegex.ReplaceAll(b, []byte("_"))
+
+		// Remove padding, hidden, or unused fields
+		removeFieldsRegex = regexp.MustCompile(`X_\S+`)
+		b = removeFieldsRegex.ReplaceAll(b, []byte("_"))
 	}
 
+	// Remove the first line of warning from cgo
+	b = b[bytes.IndexByte(b, '\n')+1:]
+	// Modify the command in the header to include:
+	//  mkpost, our own warning, and a build tag.
+	replacement := fmt.Sprintf(`$1 | go run mkpost.go
+// Code generated by the command above; see README.md. DO NOT EDIT.
+
+// +build %s,%s`, goarch, goos)
+	cgoCommandRegex := regexp.MustCompile(`(cgo -godefs .*)`)
+	b = cgoCommandRegex.ReplaceAll(b, []byte(replacement))
+
 	// gofmt
-	b, err = format.Source([]byte(s))
+	b, err = format.Source(b)
 	if err != nil {
 		log.Fatal(err)
 	}
 
-	// Append this command to the header to show where the new file
-	// came from.
-	re := regexp.MustCompile("(cgo -godefs [a-zA-Z0-9_]+\\.go.*)")
-	b = re.ReplaceAll(b, []byte("$1 | go run mkpost.go"))
-
-	fmt.Printf("%s", b)
+	os.Stdout.Write(b)
 }

unix/mksyscall.pl

@@ -69,6 +69,16 @@ if($ARGV[0] =~ /^-/) {
 	exit 1;
 }
 
+# Check that we are using the new build system if we should
+if($ENV{'GOOS'} eq "linux" || $ENV{'GOARCH'} ne "sparc64") {
+	if($ENV{'GOLANG_SYS_BUILD'} ne "docker") {
+		print STDERR "In the new build system, mksyscall should not be called directly.\n";
+		print STDERR "See README.md\n";
+		exit 1;
+	}
+}
+
+
 sub parseparamlist($) {
 	my ($list) = @_;
 	$list =~ s/^\s*//;
@@ -300,7 +310,7 @@ if($errors) {
 
 print <<EOF;
 // $cmdline
-// MACHINE GENERATED BY THE COMMAND ABOVE; DO NOT EDIT
+// Code generated by the command above; see README.md. DO NOT EDIT.
 
 // +build $tags
 

unix/mksyscall_solaris.pl

@@ -258,7 +258,7 @@ if($errors) {
 
 print <<EOF;
 // $cmdline
-// MACHINE GENERATED BY THE COMMAND ABOVE; DO NOT EDIT
+// Code generated by the command above; see README.md. DO NOT EDIT.
 
 // +build $tags
 

unix/mksysnum_darwin.pl

@@ -16,7 +16,7 @@ my $command = "mksysnum_darwin.pl " . join(' ', @ARGV);
 
 print <<EOF;
 // $command
-// MACHINE GENERATED BY THE ABOVE COMMAND; DO NOT EDIT
+// Code generated by the command above; see README.md. DO NOT EDIT.
 
 // +build $ENV{'GOARCH'},$ENV{'GOOS'}
 

unix/mksysnum_dragonfly.pl

@@ -17,7 +17,7 @@ my $command = "mksysnum_dragonfly.pl " . join(' ', @ARGV);
 
 print <<EOF;
 // $command
-// MACHINE GENERATED BY THE ABOVE COMMAND; DO NOT EDIT
+// Code generated by the command above; see README.md. DO NOT EDIT.
 
 // +build $ENV{'GOARCH'},$ENV{'GOOS'}
 

unix/mksysnum_freebsd.pl

@@ -17,7 +17,7 @@ my $command = "mksysnum_freebsd.pl " . join(' ', @ARGV);
 
 print <<EOF;
 // $command
-// MACHINE GENERATED BY THE ABOVE COMMAND; DO NOT EDIT
+// Code generated by the command above; see README.md. DO NOT EDIT.
 
 // +build $ENV{'GOARCH'},$ENV{'GOOS'}
 

unix/mksysnum_netbsd.pl

@@ -17,7 +17,7 @@ my $command = "mksysnum_netbsd.pl " . join(' ', @ARGV);
 
 print <<EOF;
 // $command
-// MACHINE GENERATED BY THE ABOVE COMMAND; DO NOT EDIT
+// Code generated by the command above; see README.md. DO NOT EDIT.
 
 // +build $ENV{'GOARCH'},$ENV{'GOOS'}
 

unix/mksysnum_openbsd.pl

@@ -17,7 +17,7 @@ my $command = "mksysnum_openbsd.pl " . join(' ', @ARGV);
 
 print <<EOF;
 // $command
-// MACHINE GENERATED BY THE ABOVE COMMAND; DO NOT EDIT
+// Code generated by the command above; see README.md. DO NOT EDIT.
 
 // +build $ENV{'GOARCH'},$ENV{'GOOS'}
 

unix/types_darwin.go

@@ -5,7 +5,7 @@
 // +build ignore
 
 /*
-Input to cgo -godefs.  See also mkerrors.sh and mkall.sh
+Input to cgo -godefs.  See README.md
 */
 
 // +godefs map struct_in_addr [4]byte /* in_addr */

unix/types_dragonfly.go

@@ -5,7 +5,7 @@
 // +build ignore
 
 /*
-Input to cgo -godefs.  See also mkerrors.sh and mkall.sh
+Input to cgo -godefs.  See README.md
 */
 
 // +godefs map struct_in_addr [4]byte /* in_addr */

unix/types_freebsd.go

@@ -5,7 +5,7 @@
 // +build ignore
 
 /*
-Input to cgo -godefs.  See also mkerrors.sh and mkall.sh
+Input to cgo -godefs.  See README.md
 */
 
 // +godefs map struct_in_addr [4]byte /* in_addr */

unix/types_netbsd.go

@@ -5,7 +5,7 @@
 // +build ignore
 
 /*
-Input to cgo -godefs.  See also mkerrors.sh and mkall.sh
+Input to cgo -godefs.  See README.md
 */
 
 // +godefs map struct_in_addr [4]byte /* in_addr */

unix/types_openbsd.go

@@ -5,7 +5,7 @@
 // +build ignore
 
 /*
-Input to cgo -godefs.  See also mkerrors.sh and mkall.sh
+Input to cgo -godefs.  See README.md
 */
 
 // +godefs map struct_in_addr [4]byte /* in_addr */

unix/types_solaris.go

@@ -5,7 +5,7 @@
 // +build ignore
 
 /*
-Input to cgo -godefs.  See also mkerrors.sh and mkall.sh
+Input to cgo -godefs.  See README.md
 */
 
 // +godefs map struct_in_addr [4]byte /* in_addr */

unix/zerrors_linux_mipsle.go

-// mkerrors.sh
-// MACHINE GENERATED BY THE COMMAND ABOVE; DO NOT EDIT
+// mkerrors.sh -Wall -Werror -static -I/tmp/include
+// Code generated by the command above; see README.md. DO NOT EDIT.
 
 // +build mipsle,linux
 
 // Created by cgo -godefs - DO NOT EDIT
-// cgo -godefs -- _const.go
+// cgo -godefs -- -Wall -Werror -static -I/tmp/include _const.go
 
 package unix
 
@@ -36,7 +36,7 @@ const (
 	AF_KEY                           = 0xf
 	AF_LLC                           = 0x1a
 	AF_LOCAL                         = 0x1
-	AF_MAX                           = 0x2a
+	AF_MAX                           = 0x2b
 	AF_MPLS                          = 0x1c
 	AF_NETBEUI                       = 0xd
 	AF_NETLINK                       = 0x10
@@ -45,6 +45,7 @@ const (
 	AF_PACKET                        = 0x11
 	AF_PHONET                        = 0x23
 	AF_PPPOX                         = 0x18
+	AF_QIPCRTR                       = 0x2a
 	AF_RDS                           = 0x15
 	AF_ROSE                          = 0xb
 	AF_ROUTE                         = 0x10
@@ -160,21 +161,21 @@ const (
 	B75                              = 0x2
 	B921600                          = 0x1007
 	B9600                            = 0xd
-	BLKBSZGET                        = 0x80081270
-	BLKBSZSET                        = 0x40081271
-	BLKFLSBUF                        = 0x1261
-	BLKFRAGET                        = 0x1265
-	BLKFRASET                        = 0x1264
-	BLKGETSIZE                       = 0x1260
-	BLKGETSIZE64                     = 0x80081272
-	BLKRAGET                         = 0x1263
-	BLKRASET                         = 0x1262
-	BLKROGET                         = 0x125e
-	BLKROSET                         = 0x125d
-	BLKRRPART                        = 0x125f
-	BLKSECTGET                       = 0x1267
-	BLKSECTSET                       = 0x1266
-	BLKSSZGET                        = 0x1268
+	BLKBSZGET                        = 0x40041270
+	BLKBSZSET                        = 0x80041271
+	BLKFLSBUF                        = 0x20001261
+	BLKFRAGET                        = 0x20001265
+	BLKFRASET                        = 0x20001264
+	BLKGETSIZE                       = 0x20001260
+	BLKGETSIZE64                     = 0x40041272
+	BLKRAGET                         = 0x20001263
+	BLKRASET                         = 0x20001262
+	BLKROGET                         = 0x2000125e
+	BLKROSET                         = 0x2000125d
+	BLKRRPART                        = 0x2000125f
+	BLKSECTGET                       = 0x20001267
+	BLKSECTSET                       = 0x20001266
+	BLKSSZGET                        = 0x20001268
 	BOTHER                           = 0x1000
 	BPF_A                            = 0x10
 	BPF_ABS                          = 0x20
@@ -238,6 +239,7 @@ const (
 	CAN_MTU                          = 0x10
 	CAN_NPROTO                       = 0x7
 	CAN_RAW                          = 0x1
+	CAN_RAW_FILTER_MAX               = 0x200
 	CAN_RTR_FLAG                     = 0x40000000
 	CAN_SFF_ID_BITS                  = 0xb
 	CAN_SFF_MASK                     = 0x7ff
@@ -403,6 +405,7 @@ const (
 	ETH_P_MPLS_MC                    = 0x8848
 	ETH_P_MPLS_UC                    = 0x8847
 	ETH_P_MVRP                       = 0x88f5
+	ETH_P_NCSI                       = 0x88f8
 	ETH_P_PAE                        = 0x888e
 	ETH_P_PAUSE                      = 0x8808
 	ETH_P_PHONET                     = 0xf5
@@ -438,6 +441,7 @@ const (
 	FALLOC_FL_KEEP_SIZE              = 0x1
 	FALLOC_FL_NO_HIDE_STALE          = 0x4
 	FALLOC_FL_PUNCH_HOLE             = 0x2
+	FALLOC_FL_UNSHARE_RANGE          = 0x40
 	FALLOC_FL_ZERO_RANGE             = 0x10
 	FD_CLOEXEC                       = 0x1
 	FD_SETSIZE                       = 0x400
@@ -1180,7 +1184,7 @@ const (
 	RTAX_UNSPEC                      = 0x0
 	RTAX_WINDOW                      = 0x3
 	RTA_ALIGNTO                      = 0x4
-	RTA_MAX                          = 0x18
+	RTA_MAX                          = 0x19
 	RTCF_DIRECTSRC                   = 0x4000000
 	RTCF_DOREDIRECT                  = 0x1000000
 	RTCF_LOG                         = 0x2000000
@@ -1277,7 +1281,7 @@ const (
 	RTM_SETLINK                      = 0x13
 	RTM_SETNEIGHTBL                  = 0x43
 	RTNH_ALIGNTO                     = 0x4
-	RTNH_COMPARE_MASK                = 0x11
+	RTNH_COMPARE_MASK                = 0x19
 	RTNH_F_DEAD                      = 0x1
 	RTNH_F_LINKDOWN                  = 0x10
 	RTNH_F_OFFLOAD                   = 0x8
@@ -1312,6 +1316,7 @@ const (
 	SCM_RIGHTS                       = 0x1
 	SCM_TIMESTAMP                    = 0x1d
 	SCM_TIMESTAMPING                 = 0x25
+	SCM_TIMESTAMPING_OPT_STATS       = 0x36
 	SCM_TIMESTAMPNS                  = 0x23
 	SCM_WIFI_STATUS                  = 0x29
 	SHUT_RD                          = 0x0
@@ -1553,6 +1558,7 @@ const (
 	TCP_REPAIR                       = 0x13
 	TCP_REPAIR_OPTIONS               = 0x16
 	TCP_REPAIR_QUEUE                 = 0x14
+	TCP_REPAIR_WINDOW                = 0x1d
 	TCP_SAVED_SYN                    = 0x1c
 	TCP_SAVE_SYN                     = 0x1b
 	TCP_SYNCNT                       = 0x7
@@ -1688,6 +1694,7 @@ const (
 	VMADDR_CID_RESERVED              = 0x1
 	VMADDR_PORT_ANY                  = 0xffffffff
 	VMIN                             = 0x4
+	VM_SOCKETS_INVALID_VERSION       = 0xffffffff
 	VQUIT                            = 0x1
 	VREPRINT                         = 0xc
 	VSTART                           = 0x8

unix/zsyscall_linux_386.go

 // mksyscall.pl -l32 -tags linux,386 syscall_linux.go syscall_linux_386.go
-// MACHINE GENERATED BY THE COMMAND ABOVE; DO NOT EDIT
+// Code generated by the command above; see README.md. DO NOT EDIT.
 
 // +build linux,386
 

unix/zsyscall_linux_amd64.go

 // mksyscall.pl -tags linux,amd64 syscall_linux.go syscall_linux_amd64.go
-// MACHINE GENERATED BY THE COMMAND ABOVE; DO NOT EDIT
+// Code generated by the command above; see README.md. DO NOT EDIT.
 
 // +build linux,amd64
 

unix/zsyscall_linux_arm.go

 // mksyscall.pl -l32 -arm -tags linux,arm syscall_linux.go syscall_linux_arm.go
-// MACHINE GENERATED BY THE COMMAND ABOVE; DO NOT EDIT
+// Code generated by the command above; see README.md. DO NOT EDIT.
 
 // +build linux,arm
 

unix/zsyscall_linux_arm64.go

 // mksyscall.pl -tags linux,arm64 syscall_linux.go syscall_linux_arm64.go
-// MACHINE GENERATED BY THE COMMAND ABOVE; DO NOT EDIT
+// Code generated by the command above; see README.md. DO NOT EDIT.
 
 // +build linux,arm64
 

unix/zsyscall_linux_mips.go

 // mksyscall.pl -b32 -arm -tags linux,mips syscall_linux.go syscall_linux_mipsx.go
-// MACHINE GENERATED BY THE COMMAND ABOVE; DO NOT EDIT
+// Code generated by the command above; see README.md. DO NOT EDIT.
 
 // +build linux,mips