• Russ Cox's avatar
    [release-branch.go1.8] runtime: do not call wakep from enlistWorker, to avoid possible deadlock · bcda91c1
    Russ Cox authored
    We have seen one instance of a production job suddenly spinning to
    100% CPU and becoming unresponsive. In that one instance, a SIGQUIT
    was sent after 328 minutes of spinning, and the stacks showed a single
    goroutine in "IO wait (scan)" state.
    
    Looking for things that might get stuck if a goroutine got stuck in
    scanning a stack, we found that injectglist does:
    
    	lock(&sched.lock)
    	var n int
    	for n = 0; glist != nil; n++ {
    		gp := glist
    		glist = gp.schedlink.ptr()
    		casgstatus(gp, _Gwaiting, _Grunnable)
    		globrunqput(gp)
    	}
    	unlock(&sched.lock)
    
    and that casgstatus spins on gp.atomicstatus until the _Gscan bit goes
    away. Essentially, this code locks sched.lock and then while holding
    sched.lock, waits to lock gp.atomicstatus.
    
    The code that is doing the scan is:
    
    	if castogscanstatus(gp, s, s|_Gscan) {
    		if !gp.gcscandone {
    			scanstack(gp, gcw)
    			gp.gcscandone = true
    		}
    		restartg(gp)
    		break loop
    	}
    
    More analysis showed that scanstack can, in a rare case, end up
    calling back into code that acquires sched.lock. For example:
    
    	runtime.scanstack at proc.go:866
    	calls runtime.gentraceback at mgcmark.go:842
    	calls runtime.scanstack$1 at traceback.go:378
    	calls runtime.scanframeworker at mgcmark.go:819
    	calls runtime.scanblock at mgcmark.go:904
    	calls runtime.greyobject at mgcmark.go:1221
    	calls (*runtime.gcWork).put at mgcmark.go:1412
    	calls (*runtime.gcControllerState).enlistWorker at mgcwork.go:127
    	calls runtime.wakep at mgc.go:632
    	calls runtime.startm at proc.go:1779
    	acquires runtime.sched.lock at proc.go:1675
    
    This path was found with an automated deadlock-detecting tool.
    There are many such paths but they all go through enlistWorker -> wakep.
    
    The evidence strongly suggests that one of these paths is what caused
    the deadlock we observed. We're running those jobs with
    GOTRACEBACK=crash now to try to get more information if it happens
    again.
    
    Further refinement and analysis shows that if we drop the wakep call
    from enlistWorker, the remaining few deadlock cycles found by the tool
    are all false positives caused by not understanding the effect of calls
    to func variables.
    
    The enlistWorker -> wakep call was intended only as a performance
    optimization, it rarely executes, and if it does execute at just the
    wrong time it can (and plausibly did) cause the deadlock we saw.
    
    Comment it out, to avoid the potential deadlock.
    
    Fixes #19112.
    Unfixes #14179.
    
    Change-Id: I6f7e10b890b991c11e79fab7aeefaf70b5d5a07b
    Reviewed-on: https://go-review.googlesource.com/37093
    Run-TryBot: Russ Cox <rsc@golang.org>
    Reviewed-by: 's avatarAustin Clements <austin@google.com>
    Reviewed-on: https://go-review.googlesource.com/37022
    TryBot-Result: Gobot Gobot <gobot@golang.org>
    bcda91c1
Name
Last commit
Last update
..
cgo Loading commit data...
debug Loading commit data...
internal Loading commit data...
msan Loading commit data...
pprof Loading commit data...
race Loading commit data...
testdata Loading commit data...
trace Loading commit data...
HACKING.md Loading commit data...
Makefile Loading commit data...
alg.go Loading commit data...
append_test.go Loading commit data...
asm.s Loading commit data...
asm_386.s Loading commit data...
asm_amd64.s Loading commit data...
asm_amd64p32.s Loading commit data...
asm_arm.s Loading commit data...
asm_arm64.s Loading commit data...
asm_mips64x.s Loading commit data...
asm_mipsx.s Loading commit data...
asm_ppc64x.h Loading commit data...
asm_ppc64x.s Loading commit data...
asm_s390x.s Loading commit data...
atomic_arm64.s Loading commit data...
atomic_mips64x.s Loading commit data...
atomic_mipsx.s Loading commit data...
atomic_pointer.go Loading commit data...
atomic_ppc64x.s Loading commit data...
callers_test.go Loading commit data...
cgo.go Loading commit data...
cgo_mmap.go Loading commit data...
cgo_ppc64x.go Loading commit data...
cgo_sigaction.go Loading commit data...
cgocall.go Loading commit data...
cgocallback.go Loading commit data...
cgocheck.go Loading commit data...
chan.go Loading commit data...
chan_test.go Loading commit data...
chanbarrier_test.go Loading commit data...
closure_test.go Loading commit data...
compiler.go Loading commit data...
complex.go Loading commit data...
complex_test.go Loading commit data...
cpuflags_amd64.go Loading commit data...
cpuidlow_amd64.s Loading commit data...
cpuprof.go Loading commit data...
cputicks.go Loading commit data...
crash_cgo_test.go Loading commit data...
crash_nonunix_test.go Loading commit data...
crash_test.go Loading commit data...
crash_unix_test.go Loading commit data...
debug.go Loading commit data...
defs1_linux.go Loading commit data...
defs1_netbsd_386.go Loading commit data...
defs1_netbsd_amd64.go Loading commit data...
defs1_netbsd_arm.go Loading commit data...
defs1_solaris_amd64.go Loading commit data...
defs2_linux.go Loading commit data...
defs3_linux.go Loading commit data...
defs_arm_linux.go Loading commit data...
defs_darwin.go Loading commit data...
defs_darwin_386.go Loading commit data...
defs_darwin_amd64.go Loading commit data...
defs_darwin_arm.go Loading commit data...
defs_darwin_arm64.go Loading commit data...
defs_dragonfly.go Loading commit data...
defs_dragonfly_amd64.go Loading commit data...
defs_freebsd.go Loading commit data...
defs_freebsd_386.go Loading commit data...
defs_freebsd_amd64.go Loading commit data...
defs_freebsd_arm.go Loading commit data...
defs_linux.go Loading commit data...
defs_linux_386.go Loading commit data...
defs_linux_amd64.go Loading commit data...
defs_linux_arm.go Loading commit data...
defs_linux_arm64.go Loading commit data...
defs_linux_mips64x.go Loading commit data...
defs_linux_mipsx.go Loading commit data...
defs_linux_ppc64.go Loading commit data...
defs_linux_ppc64le.go Loading commit data...
defs_linux_s390x.go Loading commit data...
defs_nacl_386.go Loading commit data...
defs_nacl_amd64p32.go Loading commit data...
defs_nacl_arm.go Loading commit data...
defs_netbsd.go Loading commit data...
defs_netbsd_386.go Loading commit data...
defs_netbsd_amd64.go Loading commit data...
defs_netbsd_arm.go Loading commit data...
defs_openbsd.go Loading commit data...
defs_openbsd_386.go Loading commit data...
defs_openbsd_amd64.go Loading commit data...
defs_openbsd_arm.go Loading commit data...
defs_plan9_386.go Loading commit data...
defs_plan9_amd64.go Loading commit data...
defs_plan9_arm.go Loading commit data...
defs_solaris.go Loading commit data...
defs_solaris_amd64.go Loading commit data...
defs_windows.go Loading commit data...
defs_windows_386.go Loading commit data...
defs_windows_amd64.go Loading commit data...
duff_386.s Loading commit data...
duff_amd64.s Loading commit data...
duff_arm.s Loading commit data...
duff_arm64.s Loading commit data...
duff_mips64x.s Loading commit data...
duff_ppc64x.s Loading commit data...
env_plan9.go Loading commit data...
env_posix.go Loading commit data...
env_test.go Loading commit data...
error.go Loading commit data...
export_arm_test.go Loading commit data...
export_futex_test.go Loading commit data...
export_linux_test.go Loading commit data...
export_mmap_test.go Loading commit data...
export_test.go Loading commit data...
export_windows_test.go Loading commit data...
extern.go Loading commit data...
fastlog2.go Loading commit data...
fastlog2_test.go Loading commit data...
fastlog2table.go Loading commit data...
funcdata.h Loading commit data...
futex_test.go Loading commit data...
gc_test.go Loading commit data...
gcinfo_test.go Loading commit data...
go_tls.h Loading commit data...
hash32.go Loading commit data...
hash64.go Loading commit data...
hash_test.go Loading commit data...
hashmap.go Loading commit data...
hashmap_fast.go Loading commit data...
heapdump.go Loading commit data...
iface.go Loading commit data...
iface_test.go Loading commit data...
lfstack.go Loading commit data...
lfstack_32bit.go Loading commit data...
lfstack_64bit.go Loading commit data...
lfstack_test.go Loading commit data...
lock_futex.go Loading commit data...
lock_sema.go Loading commit data...
malloc.go Loading commit data...
malloc_test.go Loading commit data...
map_test.go Loading commit data...
mapspeed_test.go Loading commit data...
mbarrier.go Loading commit data...
mbitmap.go Loading commit data...
mcache.go Loading commit data...
mcentral.go Loading commit data...
mem_bsd.go Loading commit data...
mem_darwin.go Loading commit data...
mem_linux.go Loading commit data...
mem_plan9.go Loading commit data...
mem_windows.go Loading commit data...
memclr_386.s Loading commit data...
memclr_amd64.s Loading commit data...
memclr_arm.s Loading commit data...
memclr_arm64.s Loading commit data...
memclr_mips64x.s Loading commit data...
memclr_mipsx.s Loading commit data...
memclr_plan9_386.s Loading commit data...
memclr_plan9_amd64.s Loading commit data...
memclr_ppc64x.s Loading commit data...
memclr_s390x.s Loading commit data...
memmove_386.s Loading commit data...
memmove_amd64.s Loading commit data...
memmove_arm.s Loading commit data...
memmove_arm64.s Loading commit data...
memmove_linux_amd64_test.go Loading commit data...
memmove_mips64x.s Loading commit data...
memmove_mipsx.s Loading commit data...
memmove_nacl_amd64p32.s Loading commit data...
memmove_plan9_386.s Loading commit data...
memmove_plan9_amd64.s Loading commit data...
memmove_ppc64x.s Loading commit data...
memmove_s390x.s Loading commit data...
memmove_test.go Loading commit data...
mfinal.go Loading commit data...
mfinal_test.go Loading commit data...
mfixalloc.go Loading commit data...
mgc.go Loading commit data...
mgcmark.go Loading commit data...
mgcsweep.go Loading commit data...
mgcsweepbuf.go Loading commit data...
mgcwork.go Loading commit data...
mheap.go Loading commit data...
mkduff.go Loading commit data...
mkfastlog2table.go Loading commit data...
mknacl.sh Loading commit data...
mksizeclasses.go Loading commit data...
mmap.go Loading commit data...
mprof.go Loading commit data...
msan.go Loading commit data...
msan0.go Loading commit data...
msan_amd64.s Loading commit data...
msize.go Loading commit data...
mstats.go Loading commit data...
mstkbar.go Loading commit data...
net_plan9.go Loading commit data...
netpoll.go Loading commit data...
netpoll_epoll.go Loading commit data...
netpoll_kqueue.go Loading commit data...
netpoll_nacl.go Loading commit data...
netpoll_solaris.go Loading commit data...
netpoll_stub.go Loading commit data...
netpoll_windows.go Loading commit data...
noasm.go Loading commit data...
norace_linux_test.go Loading commit data...
norace_test.go Loading commit data...
os2_freebsd.go Loading commit data...
os2_nacl.go Loading commit data...
os2_openbsd.go Loading commit data...
os2_plan9.go Loading commit data...
os2_solaris.go Loading commit data...
os3_plan9.go Loading commit data...
os3_solaris.go Loading commit data...
os_android.go Loading commit data...
os_darwin.go Loading commit data...
os_darwin_arm.go Loading commit data...
os_darwin_arm64.go Loading commit data...
os_dragonfly.go Loading commit data...
os_freebsd.go Loading commit data...
os_freebsd_arm.go Loading commit data...
os_linux.go Loading commit data...
os_linux_arm.go Loading commit data...
os_linux_arm64.go Loading commit data...
os_linux_be64.go Loading commit data...
os_linux_generic.go Loading commit data...
os_linux_mips64x.go Loading commit data...
os_linux_mipsx.go Loading commit data...
os_linux_noauxv.go Loading commit data...
os_linux_ppc64x.go Loading commit data...
os_linux_s390x.go Loading commit data...
os_nacl.go Loading commit data...
os_nacl_arm.go Loading commit data...
os_netbsd.go Loading commit data...
os_netbsd_386.go Loading commit data...
os_netbsd_amd64.go Loading commit data...
os_netbsd_arm.go Loading commit data...
os_openbsd.go Loading commit data...
os_openbsd_arm.go Loading commit data...
os_plan9.go Loading commit data...
os_plan9_arm.go Loading commit data...
os_solaris.go Loading commit data...
os_windows.go Loading commit data...
panic.go Loading commit data...
plugin.go Loading commit data...
print.go Loading commit data...
proc.go Loading commit data...
proc_runtime_test.go Loading commit data...
proc_test.go Loading commit data...
race.go Loading commit data...
race0.go Loading commit data...
race_amd64.s Loading commit data...
rdebug.go Loading commit data...
rt0_android_386.s Loading commit data...
rt0_android_amd64.s Loading commit data...
rt0_android_arm.s Loading commit data...
rt0_android_arm64.s Loading commit data...
rt0_darwin_386.s Loading commit data...
rt0_darwin_amd64.s Loading commit data...
rt0_darwin_arm.s Loading commit data...
rt0_darwin_arm64.s Loading commit data...
rt0_dragonfly_amd64.s Loading commit data...
rt0_freebsd_386.s Loading commit data...
rt0_freebsd_amd64.s Loading commit data...
rt0_freebsd_arm.s Loading commit data...
rt0_linux_386.s Loading commit data...
rt0_linux_amd64.s Loading commit data...
rt0_linux_arm.s Loading commit data...
rt0_linux_arm64.s Loading commit data...
rt0_linux_mips64x.s Loading commit data...
rt0_linux_mipsx.s Loading commit data...
rt0_linux_ppc64.s Loading commit data...
rt0_linux_ppc64le.s Loading commit data...
rt0_linux_s390x.s Loading commit data...
rt0_nacl_386.s Loading commit data...
rt0_nacl_amd64p32.s Loading commit data...
rt0_nacl_arm.s Loading commit data...
rt0_netbsd_386.s Loading commit data...
rt0_netbsd_amd64.s Loading commit data...
rt0_netbsd_arm.s Loading commit data...
rt0_openbsd_386.s Loading commit data...
rt0_openbsd_amd64.s Loading commit data...
rt0_openbsd_arm.s Loading commit data...
rt0_plan9_386.s Loading commit data...
rt0_plan9_amd64.s Loading commit data...
rt0_plan9_arm.s Loading commit data...
rt0_solaris_amd64.s Loading commit data...
rt0_windows_386.s Loading commit data...
rt0_windows_amd64.s Loading commit data...
runtime-gdb.py Loading commit data...
runtime-gdb_test.go Loading commit data...
runtime-lldb_test.go Loading commit data...
runtime.go Loading commit data...
runtime1.go Loading commit data...
runtime2.go Loading commit data...
runtime_linux_test.go Loading commit data...
runtime_mmap_test.go Loading commit data...
runtime_test.go Loading commit data...
runtime_unix_test.go Loading commit data...
select.go Loading commit data...
sema.go Loading commit data...
sigaction_linux.go Loading commit data...
signal_386.go Loading commit data...
signal_amd64x.go Loading commit data...
signal_arm.go Loading commit data...
signal_arm64.go Loading commit data...
signal_darwin.go Loading commit data...
signal_darwin_386.go Loading commit data...
signal_darwin_amd64.go Loading commit data...
signal_darwin_arm.go Loading commit data...
signal_darwin_arm64.go Loading commit data...
signal_dragonfly.go Loading commit data...
signal_dragonfly_amd64.go Loading commit data...
signal_freebsd.go Loading commit data...
signal_freebsd_386.go Loading commit data...
signal_freebsd_amd64.go Loading commit data...
signal_freebsd_arm.go Loading commit data...
signal_linux_386.go Loading commit data...
signal_linux_amd64.go Loading commit data...
signal_linux_arm.go Loading commit data...
signal_linux_arm64.go Loading commit data...
signal_linux_mips64x.go Loading commit data...
signal_linux_mipsx.go Loading commit data...
signal_linux_ppc64x.go Loading commit data...
signal_linux_s390x.go Loading commit data...
signal_mips64x.go Loading commit data...
signal_mipsx.go Loading commit data...
signal_nacl.go Loading commit data...
signal_nacl_386.go Loading commit data...
signal_nacl_amd64p32.go Loading commit data...
signal_nacl_arm.go Loading commit data...
signal_netbsd.go Loading commit data...
signal_netbsd_386.go Loading commit data...
signal_netbsd_amd64.go Loading commit data...
signal_netbsd_arm.go Loading commit data...
signal_openbsd.go Loading commit data...
signal_openbsd_386.go Loading commit data...
signal_openbsd_amd64.go Loading commit data...
signal_openbsd_arm.go Loading commit data...
signal_plan9.go Loading commit data...
signal_ppc64x.go Loading commit data...
signal_sighandler.go Loading commit data...
signal_solaris.go Loading commit data...
signal_solaris_amd64.go Loading commit data...
signal_unix.go Loading commit data...
signal_windows.go Loading commit data...
sigqueue.go Loading commit data...
sigqueue_plan9.go Loading commit data...
sigtab_linux_generic.go Loading commit data...
sigtab_linux_mipsx.go Loading commit data...
sizeclasses.go Loading commit data...
slice.go Loading commit data...
softfloat64.go Loading commit data...
softfloat64_test.go Loading commit data...
softfloat_arm.go Loading commit data...
sqrt.go Loading commit data...
sqrt_test.go Loading commit data...
stack.go Loading commit data...
stack_test.go Loading commit data...
string.go Loading commit data...
string_test.go Loading commit data...
stubs.go Loading commit data...
stubs2.go Loading commit data...
stubs32.go Loading commit data...
stubs_android.go Loading commit data...
stubs_asm.go Loading commit data...
symtab.go Loading commit data...
symtab_test.go Loading commit data...
sys_arm.go Loading commit data...
sys_arm64.go Loading commit data...
sys_darwin_386.s Loading commit data...
sys_darwin_amd64.s Loading commit data...
sys_darwin_arm.s Loading commit data...
sys_darwin_arm64.s Loading commit data...
sys_dragonfly_amd64.s Loading commit data...
sys_freebsd_386.s Loading commit data...
sys_freebsd_amd64.s Loading commit data...
sys_freebsd_arm.s Loading commit data...
sys_linux_386.s Loading commit data...
sys_linux_amd64.s Loading commit data...
sys_linux_arm.s Loading commit data...
sys_linux_arm64.s Loading commit data...
sys_linux_mips64x.s Loading commit data...
sys_linux_mipsx.s Loading commit data...
sys_linux_ppc64x.s Loading commit data...
sys_linux_s390x.s Loading commit data...
sys_mips64x.go Loading commit data...
sys_mipsx.go Loading commit data...
sys_nacl_386.s Loading commit data...
sys_nacl_amd64p32.s Loading commit data...
sys_nacl_arm.s Loading commit data...
sys_netbsd_386.s Loading commit data...
sys_netbsd_amd64.s Loading commit data...
sys_netbsd_arm.s Loading commit data...
sys_nonppc64x.go Loading commit data...
sys_openbsd_386.s Loading commit data...
sys_openbsd_amd64.s Loading commit data...
sys_openbsd_arm.s Loading commit data...
sys_plan9_386.s Loading commit data...
sys_plan9_amd64.s Loading commit data...
sys_plan9_arm.s Loading commit data...
sys_ppc64x.go Loading commit data...
sys_s390x.go Loading commit data...
sys_solaris_amd64.s Loading commit data...
sys_windows_386.s Loading commit data...
sys_windows_amd64.s Loading commit data...
sys_x86.go Loading commit data...
syscall2_solaris.go Loading commit data...
syscall_nacl.h Loading commit data...
syscall_solaris.go Loading commit data...
syscall_windows.go Loading commit data...
syscall_windows_test.go Loading commit data...
textflag.h Loading commit data...
time.go Loading commit data...
tls_arm.s Loading commit data...
tls_arm64.h Loading commit data...
tls_arm64.s Loading commit data...
tls_mips64x.s Loading commit data...
tls_mipsx.s Loading commit data...
tls_ppc64x.s Loading commit data...
tls_s390x.s Loading commit data...
trace.go Loading commit data...
traceback.go Loading commit data...
type.go Loading commit data...
typekind.go Loading commit data...
unaligned1.go Loading commit data...
unaligned2.go Loading commit data...
utf8.go Loading commit data...
vdso_linux_amd64.go Loading commit data...
vdso_none.go Loading commit data...
vlop_386.s Loading commit data...
vlop_arm.s Loading commit data...
vlop_arm_test.go Loading commit data...
vlrt.go Loading commit data...
wincallback.go Loading commit data...
write_err.go Loading commit data...
write_err_android.go Loading commit data...
zcallback_windows.go Loading commit data...
zcallback_windows.s Loading commit data...