• Austin Clements's avatar
    runtime/internal/sys: implement Ctz and Bswap in assembly for 386 · 9c8809f8
    Austin Clements authored
    Ctz is a hot-spot in the Go 1.7 memory manager. In SSA it's
    implemented as an intrinsic that compiles to a few instructions, but
    on the old backend (all architectures other than amd64), it's
    implemented as a fairly complex Go function. As a result, switching to
    bitmap-based allocation was a significant hit to allocation-heavy
    workloads like BinaryTree17 on non-SSA platforms.
    
    For unknown reasons, this hit 386 particularly hard. We can regain a
    lot of the lost performance by implementing Ctz in assembly on the
    386. This isn't as good as an intrinsic, since it still generates a
    function call and prevents useful inlining, but it's much better than
    the pure Go implementation:
    
    name                      old time/op    new time/op    delta
    BinaryTree17-12              3.59s ± 1%     3.06s ± 1%  -14.74%  (p=0.000 n=19+20)
    Fannkuch11-12                3.72s ± 1%     3.64s ± 1%   -2.09%  (p=0.000 n=17+19)
    FmtFprintfEmpty-12          52.3ns ± 3%    52.3ns ± 3%     ~     (p=0.829 n=20+19)
    FmtFprintfString-12          156ns ± 1%     148ns ± 3%   -5.20%  (p=0.000 n=18+19)
    FmtFprintfInt-12             137ns ± 1%     136ns ± 1%   -0.56%  (p=0.000 n=19+13)
    FmtFprintfIntInt-12          227ns ± 2%     225ns ± 2%   -0.93%  (p=0.000 n=19+17)
    FmtFprintfPrefixedInt-12     210ns ± 1%     208ns ± 1%   -0.91%  (p=0.000 n=19+17)
    FmtFprintfFloat-12           375ns ± 1%     371ns ± 1%   -1.06%  (p=0.000 n=19+18)
    FmtManyArgs-12               995ns ± 2%     978ns ± 1%   -1.63%  (p=0.000 n=17+17)
    GobDecode-12                9.33ms ± 1%    9.19ms ± 0%   -1.59%  (p=0.000 n=20+17)
    GobEncode-12                7.73ms ± 1%    7.73ms ± 1%     ~     (p=0.771 n=19+20)
    Gzip-12                      375ms ± 1%     374ms ± 1%     ~     (p=0.141 n=20+18)
    Gunzip-12                   61.8ms ± 1%    61.8ms ± 1%     ~     (p=0.602 n=20+20)
    HTTPClientServer-12         87.7µs ± 2%    86.9µs ± 3%   -0.87%  (p=0.024 n=19+20)
    JSONEncode-12               20.2ms ± 1%    20.4ms ± 0%   +0.53%  (p=0.000 n=18+19)
    JSONDecode-12               65.3ms ± 0%    65.4ms ± 1%     ~     (p=0.385 n=16+19)
    Mandelbrot200-12            4.11ms ± 1%    4.12ms ± 0%   +0.29%  (p=0.020 n=19+19)
    GoParse-12                  3.75ms ± 1%    3.61ms ± 2%   -3.90%  (p=0.000 n=20+20)
    RegexpMatchEasy0_32-12       104ns ± 0%     103ns ± 0%   -0.96%  (p=0.000 n=13+16)
    RegexpMatchEasy0_1K-12       805ns ± 1%     803ns ± 1%     ~     (p=0.189 n=18+18)
    RegexpMatchEasy1_32-12       111ns ± 0%     111ns ± 3%     ~     (p=1.000 n=14+19)
    RegexpMatchEasy1_1K-12      1.00µs ± 1%    1.00µs ± 1%   +0.50%  (p=0.003 n=19+19)
    RegexpMatchMedium_32-12      133ns ± 2%     133ns ± 2%     ~     (p=0.218 n=20+20)
    RegexpMatchMedium_1K-12     41.2µs ± 1%    42.2µs ± 1%   +2.52%  (p=0.000 n=18+16)
    RegexpMatchHard_32-12       2.35µs ± 1%    2.38µs ± 1%   +1.53%  (p=0.000 n=18+18)
    RegexpMatchHard_1K-12       70.9µs ± 2%    72.0µs ± 1%   +1.42%  (p=0.000 n=19+17)
    Revcomp-12                   1.06s ± 0%     1.05s ± 0%   -1.36%  (p=0.000 n=20+18)
    Template-12                 86.2ms ± 1%    84.6ms ± 0%   -1.89%  (p=0.000 n=20+18)
    TimeParse-12                 425ns ± 2%     428ns ± 1%   +0.77%  (p=0.000 n=18+19)
    TimeFormat-12                517ns ± 1%     519ns ± 1%   +0.43%  (p=0.001 n=20+19)
    [Geo mean]                  74.3µs         73.5µs        -1.05%
    
    Prior to this commit, BinaryTree17-12 on 386 was 33% slower than at
    the go1.6 tag. With this commit, it's 13% slower.
    
    On arm and arm64, BinaryTree17-12 is only ~5% slower than it was at
    go1.6. It may be worth implementing Ctz for them as well.
    
    I consider this change low risk, since the functions it replaces are
    simple, very well specified, and well tested.
    
    For #16117.
    
    Change-Id: Ic39d851d5aca91330134596effd2dab9689ba066
    Reviewed-on: https://go-review.googlesource.com/24640Reviewed-by: 's avatarRick Hudson <rlh@golang.org>
    Reviewed-by: 's avatarKeith Randall <khr@golang.org>
    Run-TryBot: Austin Clements <austin@google.com>
    TryBot-Result: Gobot Gobot <gobot@golang.org>
    9c8809f8
Name
Last commit
Last update
..
cgo Loading commit data...
debug Loading commit data...
internal Loading commit data...
msan Loading commit data...
pprof Loading commit data...
race Loading commit data...
testdata Loading commit data...
trace Loading commit data...
Makefile Loading commit data...
alg.go Loading commit data...
append_test.go Loading commit data...
asm.s Loading commit data...
asm_386.s Loading commit data...
asm_amd64.s Loading commit data...
asm_amd64p32.s Loading commit data...
asm_arm.s Loading commit data...
asm_arm64.s Loading commit data...
asm_mips64x.s Loading commit data...
asm_ppc64x.h Loading commit data...
asm_ppc64x.s Loading commit data...
asm_s390x.s Loading commit data...
atomic_arm64.s Loading commit data...
atomic_mips64x.s Loading commit data...
atomic_pointer.go Loading commit data...
atomic_ppc64x.s Loading commit data...
callers_test.go Loading commit data...
cgo.go Loading commit data...
cgo_mips64x.go Loading commit data...
cgo_mmap.go Loading commit data...
cgo_ppc64x.go Loading commit data...
cgocall.go Loading commit data...
cgocallback.go Loading commit data...
cgocheck.go Loading commit data...
chan.go Loading commit data...
chan_test.go Loading commit data...
chanbarrier_test.go Loading commit data...
closure_test.go Loading commit data...
compiler.go Loading commit data...
complex.go Loading commit data...
complex_test.go Loading commit data...
cpuprof.go Loading commit data...
cputicks.go Loading commit data...
crash_cgo_test.go Loading commit data...
crash_nonunix_test.go Loading commit data...
crash_test.go Loading commit data...
crash_unix_test.go Loading commit data...
debug.go Loading commit data...
defs1_linux.go Loading commit data...
defs1_netbsd_386.go Loading commit data...
defs1_netbsd_amd64.go Loading commit data...
defs1_netbsd_arm.go Loading commit data...
defs1_solaris_amd64.go Loading commit data...
defs2_linux.go Loading commit data...
defs3_linux.go Loading commit data...
defs_arm_linux.go Loading commit data...
defs_darwin.go Loading commit data...
defs_darwin_386.go Loading commit data...
defs_darwin_amd64.go Loading commit data...
defs_darwin_arm.go Loading commit data...
defs_darwin_arm64.go Loading commit data...
defs_dragonfly.go Loading commit data...
defs_dragonfly_amd64.go Loading commit data...
defs_freebsd.go Loading commit data...
defs_freebsd_386.go Loading commit data...
defs_freebsd_amd64.go Loading commit data...
defs_freebsd_arm.go Loading commit data...
defs_linux.go Loading commit data...
defs_linux_386.go Loading commit data...
defs_linux_amd64.go Loading commit data...
defs_linux_arm.go Loading commit data...
defs_linux_arm64.go Loading commit data...
defs_linux_mips64x.go Loading commit data...
defs_linux_ppc64.go Loading commit data...
defs_linux_ppc64le.go Loading commit data...
defs_linux_s390x.go Loading commit data...
defs_nacl_386.go Loading commit data...
defs_nacl_amd64p32.go Loading commit data...
defs_nacl_arm.go Loading commit data...
defs_netbsd.go Loading commit data...
defs_netbsd_386.go Loading commit data...
defs_netbsd_amd64.go Loading commit data...
defs_netbsd_arm.go Loading commit data...
defs_openbsd.go Loading commit data...
defs_openbsd_386.go Loading commit data...
defs_openbsd_amd64.go Loading commit data...
defs_openbsd_arm.go Loading commit data...
defs_plan9_386.go Loading commit data...
defs_plan9_amd64.go Loading commit data...
defs_plan9_arm.go Loading commit data...
defs_solaris.go Loading commit data...
defs_solaris_amd64.go Loading commit data...
defs_windows.go Loading commit data...
defs_windows_386.go Loading commit data...
defs_windows_amd64.go Loading commit data...
duff_386.s Loading commit data...
duff_amd64.s Loading commit data...
duff_arm.s Loading commit data...
duff_arm64.s Loading commit data...
duff_mips64x.s Loading commit data...
duff_ppc64x.s Loading commit data...
env_plan9.go Loading commit data...
env_posix.go Loading commit data...
env_test.go Loading commit data...
error.go Loading commit data...
export_arm_test.go Loading commit data...
export_futex_test.go Loading commit data...
export_linux_test.go Loading commit data...
export_mmap_test.go Loading commit data...
export_test.go Loading commit data...
export_windows_test.go Loading commit data...
extern.go Loading commit data...
fastlog2.go Loading commit data...
fastlog2_test.go Loading commit data...
fastlog2table.go Loading commit data...
funcdata.h Loading commit data...
futex_test.go Loading commit data...
gc_test.go Loading commit data...
gcinfo_test.go Loading commit data...
go_tls.h Loading commit data...
hash32.go Loading commit data...
hash64.go Loading commit data...
hash_test.go Loading commit data...
hashmap.go Loading commit data...
hashmap_fast.go Loading commit data...
heapdump.go Loading commit data...
iface.go Loading commit data...
iface_test.go Loading commit data...
lfstack.go Loading commit data...
lfstack_32bit.go Loading commit data...
lfstack_64bit.go Loading commit data...
lfstack_test.go Loading commit data...
lock_futex.go Loading commit data...
lock_sema.go Loading commit data...
malloc.go Loading commit data...
malloc_test.go Loading commit data...
map_test.go Loading commit data...
mapspeed_test.go Loading commit data...
mbarrier.go Loading commit data...
mbitmap.go Loading commit data...
mcache.go Loading commit data...
mcentral.go Loading commit data...
mem_bsd.go Loading commit data...
mem_darwin.go Loading commit data...
mem_linux.go Loading commit data...
mem_plan9.go Loading commit data...
mem_windows.go Loading commit data...
memclr_386.s Loading commit data...
memclr_amd64.s Loading commit data...
memclr_arm.s Loading commit data...
memclr_arm64.s Loading commit data...
memclr_mips64x.s Loading commit data...
memclr_plan9_386.s Loading commit data...
memclr_plan9_amd64.s Loading commit data...
memclr_ppc64x.s Loading commit data...
memclr_s390x.s Loading commit data...
memmove_386.s Loading commit data...
memmove_amd64.s Loading commit data...
memmove_arm.s Loading commit data...
memmove_arm64.s Loading commit data...
memmove_linux_amd64_test.go Loading commit data...
memmove_mips64x.s Loading commit data...
memmove_nacl_amd64p32.s Loading commit data...
memmove_plan9_386.s Loading commit data...
memmove_plan9_amd64.s Loading commit data...
memmove_ppc64x.s Loading commit data...
memmove_s390x.s Loading commit data...
memmove_test.go Loading commit data...
mfinal.go Loading commit data...
mfinal_test.go Loading commit data...
mfixalloc.go Loading commit data...
mgc.go Loading commit data...
mgcmark.go Loading commit data...
mgcsweep.go Loading commit data...
mgcwork.go Loading commit data...
mheap.go Loading commit data...
mkduff.go Loading commit data...
mkfastlog2table.go Loading commit data...
mknacl.sh Loading commit data...
mmap.go Loading commit data...
mprof.go Loading commit data...
msan.go Loading commit data...
msan0.go Loading commit data...
msan_amd64.s Loading commit data...
msize.go Loading commit data...
mstats.go Loading commit data...
mstkbar.go Loading commit data...
netpoll.go Loading commit data...
netpoll_epoll.go Loading commit data...
netpoll_kqueue.go Loading commit data...
netpoll_nacl.go Loading commit data...
netpoll_solaris.go Loading commit data...
netpoll_stub.go Loading commit data...
netpoll_windows.go Loading commit data...
noasm.go Loading commit data...
norace_linux_test.go Loading commit data...
norace_test.go Loading commit data...
os2_freebsd.go Loading commit data...
os2_nacl.go Loading commit data...
os2_openbsd.go Loading commit data...
os2_plan9.go Loading commit data...
os2_solaris.go Loading commit data...
os3_plan9.go Loading commit data...
os3_solaris.go Loading commit data...
os_android.go Loading commit data...
os_darwin.go Loading commit data...
os_darwin_arm.go Loading commit data...
os_darwin_arm64.go Loading commit data...
os_dragonfly.go Loading commit data...
os_freebsd.go Loading commit data...
os_freebsd_arm.go Loading commit data...
os_linux.go Loading commit data...
os_linux_arm.go Loading commit data...
os_linux_arm64.go Loading commit data...
os_linux_generic.go Loading commit data...
os_linux_mips64x.go Loading commit data...
os_linux_noauxv.go Loading commit data...
os_linux_s390x.go Loading commit data...
os_nacl.go Loading commit data...
os_nacl_arm.go Loading commit data...
os_netbsd.go Loading commit data...
os_netbsd_386.go Loading commit data...
os_netbsd_amd64.go Loading commit data...
os_netbsd_arm.go Loading commit data...
os_openbsd.go Loading commit data...
os_openbsd_arm.go Loading commit data...
os_plan9.go Loading commit data...
os_plan9_arm.go Loading commit data...
os_solaris.go Loading commit data...
os_windows.go Loading commit data...
panic.go Loading commit data...
print.go Loading commit data...
proc.go Loading commit data...
proc_runtime_test.go Loading commit data...
proc_test.go Loading commit data...
race.go Loading commit data...
race0.go Loading commit data...
race_amd64.s Loading commit data...
rdebug.go Loading commit data...
rt0_android_386.s Loading commit data...
rt0_android_amd64.s Loading commit data...
rt0_android_arm.s Loading commit data...
rt0_android_arm64.s Loading commit data...
rt0_darwin_386.s Loading commit data...
rt0_darwin_amd64.s Loading commit data...
rt0_darwin_arm.s Loading commit data...
rt0_darwin_arm64.s Loading commit data...
rt0_dragonfly_amd64.s Loading commit data...
rt0_freebsd_386.s Loading commit data...
rt0_freebsd_amd64.s Loading commit data...
rt0_freebsd_arm.s Loading commit data...
rt0_linux_386.s Loading commit data...
rt0_linux_amd64.s Loading commit data...
rt0_linux_arm.s Loading commit data...
rt0_linux_arm64.s Loading commit data...
rt0_linux_mips64x.s Loading commit data...
rt0_linux_ppc64.s Loading commit data...
rt0_linux_ppc64le.s Loading commit data...
rt0_linux_s390x.s Loading commit data...
rt0_nacl_386.s Loading commit data...
rt0_nacl_amd64p32.s Loading commit data...
rt0_nacl_arm.s Loading commit data...
rt0_netbsd_386.s Loading commit data...
rt0_netbsd_amd64.s Loading commit data...
rt0_netbsd_arm.s Loading commit data...
rt0_openbsd_386.s Loading commit data...
rt0_openbsd_amd64.s Loading commit data...
rt0_openbsd_arm.s Loading commit data...
rt0_plan9_386.s Loading commit data...
rt0_plan9_amd64.s Loading commit data...
rt0_plan9_arm.s Loading commit data...
rt0_solaris_amd64.s Loading commit data...
rt0_windows_386.s Loading commit data...
rt0_windows_amd64.s Loading commit data...
rune.go Loading commit data...
runtime-gdb.py Loading commit data...
runtime-gdb_test.go Loading commit data...
runtime-lldb_test.go Loading commit data...
runtime.go Loading commit data...
runtime1.go Loading commit data...
runtime2.go Loading commit data...
runtime_linux_test.go Loading commit data...
runtime_mmap_test.go Loading commit data...
runtime_test.go Loading commit data...
runtime_unix_test.go Loading commit data...
select.go Loading commit data...
sema.go Loading commit data...
signal1_unix.go Loading commit data...
signal2_unix.go Loading commit data...
signal_386.go Loading commit data...
signal_amd64x.go Loading commit data...
signal_arm.go Loading commit data...
signal_arm64.go Loading commit data...
signal_darwin.go Loading commit data...
signal_darwin_386.go Loading commit data...
signal_darwin_amd64.go Loading commit data...
signal_darwin_arm.go Loading commit data...
signal_darwin_arm64.go Loading commit data...
signal_dragonfly.go Loading commit data...
signal_dragonfly_amd64.go Loading commit data...
signal_freebsd.go Loading commit data...
signal_freebsd_386.go Loading commit data...
signal_freebsd_amd64.go Loading commit data...
signal_freebsd_arm.go Loading commit data...
signal_linux_386.go Loading commit data...
signal_linux_amd64.go Loading commit data...
signal_linux_arm.go Loading commit data...
signal_linux_arm64.go Loading commit data...
signal_linux_mips64x.go Loading commit data...
signal_linux_ppc64x.go Loading commit data...
signal_linux_s390x.go Loading commit data...
signal_mips64x.go Loading commit data...
signal_nacl.go Loading commit data...
signal_nacl_386.go Loading commit data...
signal_nacl_amd64p32.go Loading commit data...
signal_nacl_arm.go Loading commit data...
signal_netbsd.go Loading commit data...
signal_netbsd_386.go Loading commit data...
signal_netbsd_amd64.go Loading commit data...
signal_netbsd_arm.go Loading commit data...
signal_openbsd.go Loading commit data...
signal_openbsd_386.go Loading commit data...
signal_openbsd_amd64.go Loading commit data...
signal_openbsd_arm.go Loading commit data...
signal_plan9.go Loading commit data...
signal_ppc64x.go Loading commit data...
signal_sigtramp.go Loading commit data...
signal_solaris.go Loading commit data...
signal_solaris_amd64.go Loading commit data...
signal_unix.go Loading commit data...
signal_windows.go Loading commit data...
sigpanic_unix.go Loading commit data...
sigqueue.go Loading commit data...
sigqueue_plan9.go Loading commit data...
sigtab_linux_generic.go Loading commit data...
sigtab_linux_mips64x.go Loading commit data...
slice.go Loading commit data...
softfloat64.go Loading commit data...
softfloat64_test.go Loading commit data...
softfloat_arm.go Loading commit data...
sqrt.go Loading commit data...
sqrt_test.go Loading commit data...
stack.go Loading commit data...
stack_test.go Loading commit data...
string.go Loading commit data...
string_test.go Loading commit data...
stubs.go Loading commit data...
stubs2.go Loading commit data...
stubs32.go Loading commit data...
stubs_android.go Loading commit data...
symtab.go Loading commit data...
symtab_test.go Loading commit data...
sys_arm.go Loading commit data...
sys_arm64.go Loading commit data...
sys_darwin_386.s Loading commit data...
sys_darwin_amd64.s Loading commit data...
sys_darwin_arm.s Loading commit data...
sys_darwin_arm64.s Loading commit data...
sys_dragonfly_amd64.s Loading commit data...
sys_freebsd_386.s Loading commit data...
sys_freebsd_amd64.s Loading commit data...
sys_freebsd_arm.s Loading commit data...
sys_linux_386.s Loading commit data...
sys_linux_amd64.s Loading commit data...
sys_linux_arm.s Loading commit data...
sys_linux_arm64.s Loading commit data...
sys_linux_mips64x.s Loading commit data...
sys_linux_ppc64x.s Loading commit data...
sys_linux_s390x.s Loading commit data...
sys_mips64x.go Loading commit data...
sys_nacl_386.s Loading commit data...
sys_nacl_amd64p32.s Loading commit data...
sys_nacl_arm.s Loading commit data...
sys_netbsd_386.s Loading commit data...
sys_netbsd_amd64.s Loading commit data...
sys_netbsd_arm.s Loading commit data...
sys_nonppc64x.go Loading commit data...
sys_openbsd_386.s Loading commit data...
sys_openbsd_amd64.s Loading commit data...
sys_openbsd_arm.s Loading commit data...
sys_plan9_386.s Loading commit data...
sys_plan9_amd64.s Loading commit data...
sys_plan9_arm.s Loading commit data...
sys_ppc64x.go Loading commit data...
sys_s390x.go Loading commit data...
sys_solaris_amd64.s Loading commit data...
sys_windows_386.s Loading commit data...
sys_windows_amd64.s Loading commit data...
sys_x86.go Loading commit data...
syscall2_solaris.go Loading commit data...
syscall_nacl.h Loading commit data...
syscall_solaris.go Loading commit data...
syscall_windows.go Loading commit data...
syscall_windows_test.go Loading commit data...
textflag.h Loading commit data...
time.go Loading commit data...
tls_arm.s Loading commit data...
tls_arm64.h Loading commit data...
tls_arm64.s Loading commit data...
tls_mips64x.s Loading commit data...
tls_ppc64x.s Loading commit data...
tls_s390x.s Loading commit data...
trace.go Loading commit data...
traceback.go Loading commit data...
type.go Loading commit data...
typekind.go Loading commit data...
unaligned1.go Loading commit data...
unaligned2.go Loading commit data...
vdso_linux_amd64.go Loading commit data...
vdso_none.go Loading commit data...
vlop_386.s Loading commit data...
vlop_arm.s Loading commit data...
vlop_arm_test.go Loading commit data...
vlrt.go Loading commit data...
wincallback.go Loading commit data...
write_err.go Loading commit data...
write_err_android.go Loading commit data...
zcallback_windows.go Loading commit data...
zcallback_windows.s Loading commit data...