Otherwise we can fail to remove a unreadable empty directory.

Fixes #29178

Change-Id: I43d5c89fce57a86626abe2a1c2bbf145716e087b
Reviewed-on: https://go-review.googlesource.com/c/153720
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>