• Austin Clements's avatar
    runtime: wake idle Ps when enqueuing GC work · 0bae74e8
    Austin Clements authored
    If the scheduler has no user work and there's no GC work visible, it
    puts the P to sleep (or blocks on the network). However, if we later
    enqueue more GC work, there's currently nothing that specifically
    wakes up the scheduler to let it start an idle GC worker. As a result,
    we can underutilize the CPU during GC if Ps have been put to sleep.
    
    Fix this by making GC wake idle Ps when work buffers are put on the
    full list. We already have a hook to do this, since we use this to
    preempt a random P if we need more dedicated workers. We expand this
    hook to instead wake an idle P if there is one. The logic we use for
    this is identical to the logic used to wake an idle P when we ready a
    goroutine.
    
    To make this really sound, we also fix the scheduler to re-check the
    idle GC worker condition after releasing its P. This closes a race
    where 1) the scheduler checks for idle work and finds none, 2) new
    work is enqueued but there are no idle Ps so none are woken, and 3)
    the scheduler releases its P.
    
    There is one subtlety here. Currently we call enlistWorker directly
    from putfull, but the gcWork is in an inconsistent state in the places
    that call putfull. This isn't a problem right now because nothing that
    enlistWorker does touches the gcWork, but with the added call to
    wakep, it's possible to get a recursive call into the gcWork
    (specifically, while write barriers are disallowed, this can do an
    allocation, which can dispose a gcWork, which can put a workbuf). To
    handle this, we lift the enlistWorker calls up a layer and delay them
    until the gcWork is in a consistent state.
    
    Fixes #14179.
    
    Change-Id: Ia2467a52e54c9688c3c1752e1fc00f5b37bbfeeb
    Reviewed-on: https://go-review.googlesource.com/32434
    Run-TryBot: Austin Clements <austin@google.com>
    TryBot-Result: Gobot Gobot <gobot@golang.org>
    Reviewed-by: 's avatarDmitry Vyukov <dvyukov@google.com>
    0bae74e8
Name
Last commit
Last update
..
cgo Loading commit data...
debug Loading commit data...
internal Loading commit data...
msan Loading commit data...
pprof Loading commit data...
race Loading commit data...
testdata Loading commit data...
trace Loading commit data...
HACKING.md Loading commit data...
Makefile Loading commit data...
alg.go Loading commit data...
append_test.go Loading commit data...
asm.s Loading commit data...
asm_386.s Loading commit data...
asm_amd64.s Loading commit data...
asm_amd64p32.s Loading commit data...
asm_arm.s Loading commit data...
asm_arm64.s Loading commit data...
asm_mips64x.s Loading commit data...
asm_mipsx.s Loading commit data...
asm_ppc64x.h Loading commit data...
asm_ppc64x.s Loading commit data...
asm_s390x.s Loading commit data...
atomic_arm64.s Loading commit data...
atomic_mips64x.s Loading commit data...
atomic_mipsx.s Loading commit data...
atomic_pointer.go Loading commit data...
atomic_ppc64x.s Loading commit data...
callers_test.go Loading commit data...
cgo.go Loading commit data...
cgo_mmap.go Loading commit data...
cgo_ppc64x.go Loading commit data...
cgo_sigaction.go Loading commit data...
cgocall.go Loading commit data...
cgocallback.go Loading commit data...
cgocheck.go Loading commit data...
chan.go Loading commit data...
chan_test.go Loading commit data...
chanbarrier_test.go Loading commit data...
closure_test.go Loading commit data...
compiler.go Loading commit data...
complex.go Loading commit data...
complex_test.go Loading commit data...
cpuflags_amd64.go Loading commit data...
cpuidlow_amd64.s Loading commit data...
cpuprof.go Loading commit data...
cputicks.go Loading commit data...
crash_cgo_test.go Loading commit data...
crash_nonunix_test.go Loading commit data...
crash_test.go Loading commit data...
crash_unix_test.go Loading commit data...
debug.go Loading commit data...
defs1_linux.go Loading commit data...
defs1_netbsd_386.go Loading commit data...
defs1_netbsd_amd64.go Loading commit data...
defs1_netbsd_arm.go Loading commit data...
defs1_solaris_amd64.go Loading commit data...
defs2_linux.go Loading commit data...
defs3_linux.go Loading commit data...
defs_arm_linux.go Loading commit data...
defs_darwin.go Loading commit data...
defs_darwin_386.go Loading commit data...
defs_darwin_amd64.go Loading commit data...
defs_darwin_arm.go Loading commit data...
defs_darwin_arm64.go Loading commit data...
defs_dragonfly.go Loading commit data...
defs_dragonfly_amd64.go Loading commit data...
defs_freebsd.go Loading commit data...
defs_freebsd_386.go Loading commit data...
defs_freebsd_amd64.go Loading commit data...
defs_freebsd_arm.go Loading commit data...
defs_linux.go Loading commit data...
defs_linux_386.go Loading commit data...
defs_linux_amd64.go Loading commit data...
defs_linux_arm.go Loading commit data...
defs_linux_arm64.go Loading commit data...
defs_linux_mips64x.go Loading commit data...
defs_linux_mipsx.go Loading commit data...
defs_linux_ppc64.go Loading commit data...
defs_linux_ppc64le.go Loading commit data...
defs_linux_s390x.go Loading commit data...
defs_nacl_386.go Loading commit data...
defs_nacl_amd64p32.go Loading commit data...
defs_nacl_arm.go Loading commit data...
defs_netbsd.go Loading commit data...
defs_netbsd_386.go Loading commit data...
defs_netbsd_amd64.go Loading commit data...
defs_netbsd_arm.go Loading commit data...
defs_openbsd.go Loading commit data...
defs_openbsd_386.go Loading commit data...
defs_openbsd_amd64.go Loading commit data...
defs_openbsd_arm.go Loading commit data...
defs_plan9_386.go Loading commit data...
defs_plan9_amd64.go Loading commit data...
defs_plan9_arm.go Loading commit data...
defs_solaris.go Loading commit data...
defs_solaris_amd64.go Loading commit data...
defs_windows.go Loading commit data...
defs_windows_386.go Loading commit data...
defs_windows_amd64.go Loading commit data...
duff_386.s Loading commit data...
duff_amd64.s Loading commit data...
duff_arm.s Loading commit data...
duff_arm64.s Loading commit data...
duff_mips64x.s Loading commit data...
duff_ppc64x.s Loading commit data...
env_plan9.go Loading commit data...
env_posix.go Loading commit data...
env_test.go Loading commit data...
error.go Loading commit data...
export_arm_test.go Loading commit data...
export_futex_test.go Loading commit data...
export_linux_test.go Loading commit data...
export_mmap_test.go Loading commit data...
export_test.go Loading commit data...
export_windows_test.go Loading commit data...
extern.go Loading commit data...
fastlog2.go Loading commit data...
fastlog2_test.go Loading commit data...
fastlog2table.go Loading commit data...
funcdata.h Loading commit data...
futex_test.go Loading commit data...
gc_test.go Loading commit data...
gcinfo_test.go Loading commit data...
go_tls.h Loading commit data...
hash32.go Loading commit data...
hash64.go Loading commit data...
hash_test.go Loading commit data...
hashmap.go Loading commit data...
hashmap_fast.go Loading commit data...
heapdump.go Loading commit data...
iface.go Loading commit data...
iface_test.go Loading commit data...
lfstack.go Loading commit data...
lfstack_32bit.go Loading commit data...
lfstack_64bit.go Loading commit data...
lfstack_test.go Loading commit data...
lock_futex.go Loading commit data...
lock_sema.go Loading commit data...
malloc.go Loading commit data...
malloc_test.go Loading commit data...
map_test.go Loading commit data...
mapspeed_test.go Loading commit data...
mbarrier.go Loading commit data...
mbitmap.go Loading commit data...
mcache.go Loading commit data...
mcentral.go Loading commit data...
mem_bsd.go Loading commit data...
mem_darwin.go Loading commit data...
mem_linux.go Loading commit data...
mem_plan9.go Loading commit data...
mem_windows.go Loading commit data...
memclr_386.s Loading commit data...
memclr_amd64.s Loading commit data...
memclr_arm.s Loading commit data...
memclr_arm64.s Loading commit data...
memclr_mips64x.s Loading commit data...
memclr_mipsx.s Loading commit data...
memclr_plan9_386.s Loading commit data...
memclr_plan9_amd64.s Loading commit data...
memclr_ppc64x.s Loading commit data...
memclr_s390x.s Loading commit data...
memmove_386.s Loading commit data...
memmove_amd64.s Loading commit data...
memmove_arm.s Loading commit data...
memmove_arm64.s Loading commit data...
memmove_linux_amd64_test.go Loading commit data...
memmove_mips64x.s Loading commit data...
memmove_mipsx.s Loading commit data...
memmove_nacl_amd64p32.s Loading commit data...
memmove_plan9_386.s Loading commit data...
memmove_plan9_amd64.s Loading commit data...
memmove_ppc64x.s Loading commit data...
memmove_s390x.s Loading commit data...
memmove_test.go Loading commit data...
mfinal.go Loading commit data...
mfinal_test.go Loading commit data...
mfixalloc.go Loading commit data...
mgc.go Loading commit data...
mgcmark.go Loading commit data...
mgcsweep.go Loading commit data...
mgcsweepbuf.go Loading commit data...
mgcwork.go Loading commit data...
mheap.go Loading commit data...
mkduff.go Loading commit data...
mkfastlog2table.go Loading commit data...
mknacl.sh Loading commit data...
mksizeclasses.go Loading commit data...
mmap.go Loading commit data...
mprof.go Loading commit data...
msan.go Loading commit data...
msan0.go Loading commit data...
msan_amd64.s Loading commit data...
msize.go Loading commit data...
mstats.go Loading commit data...
mstkbar.go Loading commit data...
net_plan9.go Loading commit data...
netpoll.go Loading commit data...
netpoll_epoll.go Loading commit data...
netpoll_kqueue.go Loading commit data...
netpoll_nacl.go Loading commit data...
netpoll_solaris.go Loading commit data...
netpoll_stub.go Loading commit data...
netpoll_windows.go Loading commit data...
noasm.go Loading commit data...
norace_linux_test.go Loading commit data...
norace_test.go Loading commit data...
os2_freebsd.go Loading commit data...
os2_nacl.go Loading commit data...
os2_openbsd.go Loading commit data...
os2_plan9.go Loading commit data...
os2_solaris.go Loading commit data...
os3_plan9.go Loading commit data...
os3_solaris.go Loading commit data...
os_android.go Loading commit data...
os_darwin.go Loading commit data...
os_darwin_arm.go Loading commit data...
os_darwin_arm64.go Loading commit data...
os_dragonfly.go Loading commit data...
os_freebsd.go Loading commit data...
os_freebsd_arm.go Loading commit data...
os_linux.go Loading commit data...
os_linux_arm.go Loading commit data...
os_linux_arm64.go Loading commit data...
os_linux_be64.go Loading commit data...
os_linux_generic.go Loading commit data...
os_linux_mips64x.go Loading commit data...
os_linux_mipsx.go Loading commit data...
os_linux_noauxv.go Loading commit data...
os_linux_ppc64x.go Loading commit data...
os_linux_s390x.go Loading commit data...
os_nacl.go Loading commit data...
os_nacl_arm.go Loading commit data...
os_netbsd.go Loading commit data...
os_netbsd_386.go Loading commit data...
os_netbsd_amd64.go Loading commit data...
os_netbsd_arm.go Loading commit data...
os_openbsd.go Loading commit data...
os_openbsd_arm.go Loading commit data...
os_plan9.go Loading commit data...
os_plan9_arm.go Loading commit data...
os_solaris.go Loading commit data...
os_windows.go Loading commit data...
panic.go Loading commit data...
plugin.go Loading commit data...
print.go Loading commit data...
proc.go Loading commit data...
proc_runtime_test.go Loading commit data...
proc_test.go Loading commit data...
race.go Loading commit data...
race0.go Loading commit data...
race_amd64.s Loading commit data...
rdebug.go Loading commit data...
rt0_android_386.s Loading commit data...
rt0_android_amd64.s Loading commit data...
rt0_android_arm.s Loading commit data...
rt0_android_arm64.s Loading commit data...
rt0_darwin_386.s Loading commit data...
rt0_darwin_amd64.s Loading commit data...
rt0_darwin_arm.s Loading commit data...
rt0_darwin_arm64.s Loading commit data...
rt0_dragonfly_amd64.s Loading commit data...
rt0_freebsd_386.s Loading commit data...
rt0_freebsd_amd64.s Loading commit data...
rt0_freebsd_arm.s Loading commit data...
rt0_linux_386.s Loading commit data...
rt0_linux_amd64.s Loading commit data...
rt0_linux_arm.s Loading commit data...
rt0_linux_arm64.s Loading commit data...
rt0_linux_mips64x.s Loading commit data...
rt0_linux_mipsx.s Loading commit data...
rt0_linux_ppc64.s Loading commit data...
rt0_linux_ppc64le.s Loading commit data...
rt0_linux_s390x.s Loading commit data...
rt0_nacl_386.s Loading commit data...
rt0_nacl_amd64p32.s Loading commit data...
rt0_nacl_arm.s Loading commit data...
rt0_netbsd_386.s Loading commit data...
rt0_netbsd_amd64.s Loading commit data...
rt0_netbsd_arm.s Loading commit data...
rt0_openbsd_386.s Loading commit data...
rt0_openbsd_amd64.s Loading commit data...
rt0_openbsd_arm.s Loading commit data...
rt0_plan9_386.s Loading commit data...
rt0_plan9_amd64.s Loading commit data...
rt0_plan9_arm.s Loading commit data...
rt0_solaris_amd64.s Loading commit data...
rt0_windows_386.s Loading commit data...
rt0_windows_amd64.s Loading commit data...
runtime-gdb.py Loading commit data...
runtime-gdb_test.go Loading commit data...
runtime-lldb_test.go Loading commit data...
runtime.go Loading commit data...
runtime1.go Loading commit data...
runtime2.go Loading commit data...
runtime_linux_test.go Loading commit data...
runtime_mmap_test.go Loading commit data...
runtime_test.go Loading commit data...
runtime_unix_test.go Loading commit data...
select.go Loading commit data...
sema.go Loading commit data...
sigaction_linux.go Loading commit data...
signal_386.go Loading commit data...
signal_amd64x.go Loading commit data...
signal_arm.go Loading commit data...
signal_arm64.go Loading commit data...
signal_darwin.go Loading commit data...
signal_darwin_386.go Loading commit data...
signal_darwin_amd64.go Loading commit data...
signal_darwin_arm.go Loading commit data...
signal_darwin_arm64.go Loading commit data...
signal_dragonfly.go Loading commit data...
signal_dragonfly_amd64.go Loading commit data...
signal_freebsd.go Loading commit data...
signal_freebsd_386.go Loading commit data...
signal_freebsd_amd64.go Loading commit data...
signal_freebsd_arm.go Loading commit data...
signal_linux_386.go Loading commit data...
signal_linux_amd64.go Loading commit data...
signal_linux_arm.go Loading commit data...
signal_linux_arm64.go Loading commit data...
signal_linux_mips64x.go Loading commit data...
signal_linux_mipsx.go Loading commit data...
signal_linux_ppc64x.go Loading commit data...
signal_linux_s390x.go Loading commit data...
signal_mips64x.go Loading commit data...
signal_mipsx.go Loading commit data...
signal_nacl.go Loading commit data...
signal_nacl_386.go Loading commit data...
signal_nacl_amd64p32.go Loading commit data...
signal_nacl_arm.go Loading commit data...
signal_netbsd.go Loading commit data...
signal_netbsd_386.go Loading commit data...
signal_netbsd_amd64.go Loading commit data...
signal_netbsd_arm.go Loading commit data...
signal_openbsd.go Loading commit data...
signal_openbsd_386.go Loading commit data...
signal_openbsd_amd64.go Loading commit data...
signal_openbsd_arm.go Loading commit data...
signal_plan9.go Loading commit data...
signal_ppc64x.go Loading commit data...
signal_sighandler.go Loading commit data...
signal_solaris.go Loading commit data...
signal_solaris_amd64.go Loading commit data...
signal_unix.go Loading commit data...
signal_windows.go Loading commit data...
sigqueue.go Loading commit data...
sigqueue_plan9.go Loading commit data...
sigtab_linux_generic.go Loading commit data...
sigtab_linux_mipsx.go Loading commit data...
sizeclasses.go Loading commit data...
slice.go Loading commit data...
softfloat64.go Loading commit data...
softfloat64_test.go Loading commit data...
softfloat_arm.go Loading commit data...
sqrt.go Loading commit data...
sqrt_test.go Loading commit data...
stack.go Loading commit data...
stack_test.go Loading commit data...
string.go Loading commit data...
string_test.go Loading commit data...
stubs.go Loading commit data...
stubs2.go Loading commit data...
stubs32.go Loading commit data...
stubs_android.go Loading commit data...
stubs_asm.go Loading commit data...
symtab.go Loading commit data...
symtab_test.go Loading commit data...
sys_arm.go Loading commit data...
sys_arm64.go Loading commit data...
sys_darwin_386.s Loading commit data...
sys_darwin_amd64.s Loading commit data...
sys_darwin_arm.s Loading commit data...
sys_darwin_arm64.s Loading commit data...
sys_dragonfly_amd64.s Loading commit data...
sys_freebsd_386.s Loading commit data...
sys_freebsd_amd64.s Loading commit data...
sys_freebsd_arm.s Loading commit data...
sys_linux_386.s Loading commit data...
sys_linux_amd64.s Loading commit data...
sys_linux_arm.s Loading commit data...
sys_linux_arm64.s Loading commit data...
sys_linux_mips64x.s Loading commit data...
sys_linux_mipsx.s Loading commit data...
sys_linux_ppc64x.s Loading commit data...
sys_linux_s390x.s Loading commit data...
sys_mips64x.go Loading commit data...
sys_mipsx.go Loading commit data...
sys_nacl_386.s Loading commit data...
sys_nacl_amd64p32.s Loading commit data...
sys_nacl_arm.s Loading commit data...
sys_netbsd_386.s Loading commit data...
sys_netbsd_amd64.s Loading commit data...
sys_netbsd_arm.s Loading commit data...
sys_nonppc64x.go Loading commit data...
sys_openbsd_386.s Loading commit data...
sys_openbsd_amd64.s Loading commit data...
sys_openbsd_arm.s Loading commit data...
sys_plan9_386.s Loading commit data...
sys_plan9_amd64.s Loading commit data...
sys_plan9_arm.s Loading commit data...
sys_ppc64x.go Loading commit data...
sys_s390x.go Loading commit data...
sys_solaris_amd64.s Loading commit data...
sys_windows_386.s Loading commit data...
sys_windows_amd64.s Loading commit data...
sys_x86.go Loading commit data...
syscall2_solaris.go Loading commit data...
syscall_nacl.h Loading commit data...
syscall_solaris.go Loading commit data...
syscall_windows.go Loading commit data...
syscall_windows_test.go Loading commit data...
textflag.h Loading commit data...
time.go Loading commit data...
tls_arm.s Loading commit data...
tls_arm64.h Loading commit data...
tls_arm64.s Loading commit data...
tls_mips64x.s Loading commit data...
tls_mipsx.s Loading commit data...
tls_ppc64x.s Loading commit data...
tls_s390x.s Loading commit data...
trace.go Loading commit data...
traceback.go Loading commit data...
type.go Loading commit data...
typekind.go Loading commit data...
unaligned1.go Loading commit data...
unaligned2.go Loading commit data...
utf8.go Loading commit data...
vdso_linux_amd64.go Loading commit data...
vdso_none.go Loading commit data...
vlop_386.s Loading commit data...
vlop_arm.s Loading commit data...
vlop_arm_test.go Loading commit data...
vlrt.go Loading commit data...
wincallback.go Loading commit data...
write_err.go Loading commit data...
write_err_android.go Loading commit data...
zcallback_windows.go Loading commit data...
zcallback_windows.s Loading commit data...