Use memoization to avoid repeated recomputation of nested
node sizes. Speeds up testdata/slow.input by several orders
of magnitude.

- added respective test case
- added timeout to test code
- deleted some unrelated unused code

Fixes #1628.

R=rsc, r
CC=golang-dev
https://golang.org/cl/4274075