Only forget about it if the context timed out, as the comment says.

Fixes #20703.

Change-Id: Ie6234f1a32f85e6bfd052dc24a33aa63b8883c37
Reviewed-on: https://go-review.googlesource.com/45999
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>