• Russ Cox's avatar
    go/constant: make string addition compute actual string lazily · 010d8948
    Russ Cox authored
    It is natural for tools to take a large string concatenation like
    
    	"1" + "1" + "1" + ... + "1"
    
    and translate that into a sequence of go/constant calls:
    
    	x := constant.MakeString("1")
    	x = constant.BinaryOp(x, token.ADD, constant.MakeString("1"))
    	x = constant.BinaryOp(x, token.ADD, constant.MakeString("1"))
    	x = constant.BinaryOp(x, token.ADD, constant.MakeString("1"))
    	x = constant.BinaryOp(x, token.ADD, constant.MakeString("1"))
    	...
    
    If the underlying representation of a string constant is a Go string,
    then this leads to O(N²) memory for the concatenation of N strings,
    allocating memory for "1", "11", "111", "1111", and so on.
    This makes go/types and in particular cmd/vet run out of memory
    (or at least use far too much) on machine-generated string concatenations,
    such as those generated by go-bindata.
    
    This CL allows code like the above to operate efficiently, by delaying
    the evaluation of the actual string constant value until it is needed.
    Now the representation of a string constant is either a string or an
    explicit addition expression. The addition expression is turned into
    a string the first time it is requested and then cached for future use.
    This slows down the use of single strings, but analyses are likely not
    dominated by that use anyway. It speeds up string concatenations,
    especially large ones, significantly.
    
    On my Mac running 32-bit code:
    
    name               old time/op    new time/op    delta
    StringAdd/1-8         160ns ± 2%     183ns ± 1%  +13.98%  (p=0.000 n=10+10)
    StringAdd/4-8         650ns ± 1%     927ns ± 4%  +42.73%  (p=0.000 n=10+10)
    StringAdd/16-8       3.93µs ± 1%    2.78µs ± 2%  -29.12%  (p=0.000 n=8+9)
    StringAdd/64-8       37.3µs ± 9%    10.1µs ± 5%  -73.06%  (p=0.000 n=10+10)
    StringAdd/256-8       513µs ± 5%      38µs ± 1%  -92.63%  (p=0.000 n=10+10)
    StringAdd/1024-8     5.67ms ± 4%    0.14ms ± 2%  -97.45%  (p=0.000 n=8+10)
    StringAdd/4096-8     77.1ms ± 9%     0.7ms ± 2%  -99.10%  (p=0.000 n=10+9)
    StringAdd/16384-8     1.33s ± 7%     0.00s ±10%  -99.64%  (p=0.000 n=10+10)
    StringAdd/65536-8     21.5s ± 4%      0.0s ± 8%  -99.89%  (p=0.000 n=10+10)
    
    name               old alloc/op   new alloc/op   delta
    StringAdd/1-8          232B ± 0%      256B ± 0%  +10.34%  (p=0.000 n=10+10)
    StringAdd/4-8        1.20kB ± 0%    1.24kB ± 0%   +3.33%  (p=0.000 n=10+10)
    StringAdd/16-8       14.7kB ± 0%     4.6kB ± 0%  -68.87%  (p=0.000 n=10+10)
    StringAdd/64-8        223kB ± 0%      16kB ± 0%  -92.66%  (p=0.000 n=10+10)
    StringAdd/256-8      3.48MB ± 0%    0.07MB ± 0%  -98.07%  (p=0.000 n=10+10)
    StringAdd/1024-8     55.7MB ± 0%     0.3MB ± 0%  -99.53%  (p=0.000 n=10+10)
    StringAdd/4096-8      855MB ± 0%       1MB ± 0%  -99.88%  (p=0.000 n=10+10)
    StringAdd/16384-8    13.5GB ± 0%     0.0GB ± 0%  -99.97%  (p=0.000 n=9+10)
    StringAdd/65536-8     215GB ± 0%       0GB ± 0%  -99.99%  (p=0.000 n=10+10)
    
    name               old allocs/op  new allocs/op  delta
    StringAdd/1-8          3.00 ± 0%      3.00 ± 0%     ~     (all equal)
    StringAdd/4-8          9.00 ± 0%     11.00 ± 0%  +22.22%  (p=0.000 n=10+10)
    StringAdd/16-8         33.0 ± 0%      25.0 ± 0%  -24.24%  (p=0.000 n=10+10)
    StringAdd/64-8          129 ± 0%        75 ± 0%  -41.86%  (p=0.000 n=10+10)
    StringAdd/256-8         513 ± 0%       269 ± 0%  -47.56%  (p=0.000 n=10+10)
    StringAdd/1024-8      2.05k ± 0%     1.04k ± 0%  -49.29%  (p=0.000 n=10+10)
    StringAdd/4096-8      8.19k ± 0%     4.12k ± 0%  -49.77%  (p=0.000 n=10+10)
    StringAdd/16384-8     32.8k ± 0%     16.4k ± 0%  -49.97%  (p=0.000 n=9+10)
    StringAdd/65536-8      131k ± 0%       66k ± 0%  -50.11%  (p=0.000 n=10+10)
    
    https://perf.golang.org/search?q=upload:20180105.2
    
    Fixes #23348 (originally reported as cmd/vet failures in comments on #23222).
    
    This makes constant.Values of Kind String no longer meaningful for ==, which
    required fixes in go/types. While there, also fix go/types handling of constant.Values
    of Kind Int (for uint64), Float, and Complex.
    
    Change-Id: I80867bc9c4232c5c9b213443ff16645434a68b36
    Reviewed-on: https://go-review.googlesource.com/86395
    Run-TryBot: Russ Cox <rsc@golang.org>
    TryBot-Result: Gobot Gobot <gobot@golang.org>
    Reviewed-by: 's avatarRobert Griesemer <gri@golang.org>
    010d8948
Name
Last commit
Last update
..
value.go Loading commit data...
value_test.go Loading commit data...