Skip to content

G
golang
Switch branch/tag
  1. 15 Nov, 2018 8 commits
    • Alan Donovan's avatar
      cmd/vet: switch to x/tools/go/analysis implementation · cb413099
      Alan Donovan authored 
      This change deletes the legacy implementation of vet, replacing it
with a short main.go that merely selects the desired analyzers and
calls into the "unitchecker" implementation vendored from
golang.org/x/tools/go/analysis.

Unlike the full vet checker (x/tools/go/analysis/cmd/vet), the 'lite'
unitchecker cannot also be run standalone (as 'go tool vet' or
cmd/vet); it must be invoked by 'go vet'.
This design was chosen to avoid vendoring many
additional dependencies into GOROOT, in particular go/packages. If
go/packages should someday become part of the standard library, there
will be considerable opportunity for simplification.

This change also patches the vendored analysisflag package
(by adding patch.go) so that it fully supports the build
system's -V flag protocol.

Also:
- remove stale internal/unitchecker/ tree
  (belonged in https://go-review.googlesource.com/c/149778).
- move vet legacy flags (-all, -v, -source, -tags) into analysisflags
  as all drivers will need them, not just unitchecker.
  I will upstream this change.

A sampling of tests from the cmd/vet testsuite have been preserved as
a smoke test, to ensure that each analyzer is being run, and for
convenience when evaluating changes. Comprehensive tests for each
analyzer live upstream in x/tools. The tests have been heavily reduced
and reorganized so that they conform to the structure required by 'go
vet'.

Change-Id: I84b38caeef733e65deb95234b3b87b5f61046def
Reviewed-on: https://go-review.googlesource.com/c/149609Reviewed-by: 's avatarRuss Cox <rsc@golang.org>
      cb413099
    • Austin Clements's avatar
      runtime: eliminate mheap.busy* lists · b00a6d8b
      Austin Clements authored 
      The old whole-page reclaimer was the only thing that used the busy
span lists. Remove them so nothing uses them any more.

Change-Id: I4007dd2be08b9ef41bfdb0c387215c73c392cc4c
Reviewed-on: https://go-review.googlesource.com/c/138960
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: 's avatarRick Hudson <rlh@golang.org>
Reviewed-by: 's avatarMichael Knyszek <mknyszek@google.com>
      b00a6d8b
    • Austin Clements's avatar
      runtime: implement efficient page reclaimer · 5333550b
      Austin Clements authored 
      When we attempt to allocate an N page span (either for a large
allocation or when an mcentral runs dry), we first try to sweep spans
to release N pages. Currently, this can be extremely expensive:
sweeping a span to emptiness is the hardest thing to ask for and the
sweeper generally doesn't know where to even look for potentially
fruitful results. Since this is on the critical path of many
allocations, this is unfortunate.

This CL changes how we reclaim empty spans. Instead of trying lots of
spans and hoping for the best, it uses the newly introduced span marks
to efficiently find empty spans. The span marks (and in-use bits) are
in a dense bitmap, so these spans can be found with an efficient
sequential memory scan. This approach can scan for unmarked spans at
about 300 GB/ms and can free unmarked spans at about 32 MB/ms. We
could probably significantly improve the rate at which is can free
unmarked spans, but that's a separate issue.

Like the current reclaimer, this is still linear in the number of
spans that are swept, but the constant factor is now so vanishingly
small that it doesn't matter.

The benchmark in #18155 demonstrates both significant page reclaiming
delays, and object reclaiming delays. With "-retain-count=20000000
-preallocate=true -loop-count=3", the benchmark demonstrates several
page reclaiming delays on the order of 40ms. After this change, the
page reclaims are insignificant. The longest sweeps are still ~150ms,
but are object reclaiming delays. We'll address those in the next
several CLs.

Updates #18155.

Fixes #21378 by completely replacing the logic that had that bug.

Change-Id: Iad80eec11d7fc262d02c8f0761ac6998425c4064
Reviewed-on: https://go-review.googlesource.com/c/138959
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: 's avatarRick Hudson <rlh@golang.org>
      5333550b
    • Austin Clements's avatar
      runtime: mark span when marking any object on the span · ba1698e9
      Austin Clements authored 
      This adds a mark bit for each span that is set if any objects on the
span are marked. This will be used for sweeping.

For #18155.

The impact of this is negligible for most benchmarks, and < 1% for
GC-heavy benchmarks.

name                       old time/op  new time/op  delta
Garbage/benchmem-MB=64-12  2.18ms ± 0%  2.20ms ± 1%  +0.88%  (p=0.000 n=16+18)

(https://perf.golang.org/search?q=upload:20180928.1)

name                      old time/op    new time/op    delta
BinaryTree17-12              2.68s ± 1%     2.68s ± 1%    ~     (p=0.707 n=17+19)
Fannkuch11-12                2.28s ± 0%     2.39s ± 0%  +4.95%  (p=0.000 n=19+18)
FmtFprintfEmpty-12          40.3ns ± 4%    39.4ns ± 2%  -2.27%  (p=0.000 n=17+18)
FmtFprintfString-12         67.9ns ± 1%    68.3ns ± 1%  +0.55%  (p=0.000 n=18+19)
FmtFprintfInt-12            75.7ns ± 1%    76.1ns ± 1%  +0.44%  (p=0.005 n=18+19)
FmtFprintfIntInt-12          123ns ± 1%     121ns ± 1%  -1.00%  (p=0.000 n=18+18)
FmtFprintfPrefixedInt-12     150ns ± 0%     148ns ± 0%  -1.33%  (p=0.000 n=16+13)
FmtFprintfFloat-12           208ns ± 0%     204ns ± 0%  -1.92%  (p=0.000 n=13+17)
FmtManyArgs-12               501ns ± 1%     498ns ± 0%  -0.55%  (p=0.000 n=19+17)
GobDecode-12                6.24ms ± 0%    6.25ms ± 1%    ~     (p=0.113 n=20+19)
GobEncode-12                5.33ms ± 0%    5.29ms ± 1%  -0.72%  (p=0.000 n=20+18)
Gzip-12                      220ms ± 1%     218ms ± 1%  -1.02%  (p=0.000 n=19+19)
Gunzip-12                   35.5ms ± 0%    35.7ms ± 0%  +0.45%  (p=0.000 n=16+18)
HTTPClientServer-12         77.9µs ± 1%    77.7µs ± 1%  -0.30%  (p=0.047 n=20+19)
JSONEncode-12               8.82ms ± 0%    8.93ms ± 0%  +1.20%  (p=0.000 n=18+17)
JSONDecode-12               47.3ms ± 0%    47.0ms ± 0%  -0.49%  (p=0.000 n=17+18)
Mandelbrot200-12            3.69ms ± 0%    3.68ms ± 0%  -0.25%  (p=0.000 n=19+18)
GoParse-12                  3.13ms ± 1%    3.13ms ± 1%    ~     (p=0.640 n=20+20)
RegexpMatchEasy0_32-12      76.2ns ± 1%    76.2ns ± 1%    ~     (p=0.818 n=20+19)
RegexpMatchEasy0_1K-12       226ns ± 0%     226ns ± 0%  -0.22%  (p=0.001 n=17+18)
RegexpMatchEasy1_32-12      71.9ns ± 1%    72.0ns ± 1%    ~     (p=0.653 n=18+18)
RegexpMatchEasy1_1K-12       355ns ± 1%     356ns ± 1%    ~     (p=0.160 n=18+19)
RegexpMatchMedium_32-12      106ns ± 1%     106ns ± 1%    ~     (p=0.325 n=17+20)
RegexpMatchMedium_1K-12     31.1µs ± 2%    31.2µs ± 0%  +0.59%  (p=0.007 n=19+15)
RegexpMatchHard_32-12       1.54µs ± 2%    1.53µs ± 2%  -0.78%  (p=0.021 n=17+18)
RegexpMatchHard_1K-12       46.0µs ± 1%    45.9µs ± 1%  -0.31%  (p=0.025 n=17+19)
Revcomp-12                   391ms ± 1%     394ms ± 2%  +0.80%  (p=0.000 n=17+19)
Template-12                 59.9ms ± 1%    59.9ms ± 1%    ~     (p=0.428 n=20+19)
TimeParse-12                 304ns ± 1%     312ns ± 0%  +2.88%  (p=0.000 n=20+17)
TimeFormat-12                318ns ± 0%     326ns ± 0%  +2.64%  (p=0.000 n=20+17)

(https://perf.golang.org/search?q=upload:20180928.2)

Change-Id: I336b9bf054113580a24103192904c8c76593e90e
Reviewed-on: https://go-review.googlesource.com/c/138958
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: 's avatarRick Hudson <rlh@golang.org>
Reviewed-by: 's avatarMichael Knyszek <mknyszek@google.com>
      ba1698e9
    • Austin Clements's avatar
      runtime: record in-use spans in a page-indexed bitmap · 69e666e4
      Austin Clements authored 
      This adds a bitmap indexed by page number that marks the starts of
in-use spans. This will be used to quickly find in-use spans with no
marked objects for sweeping.

For #18155.

Change-Id: Icee56f029cde502447193e136fa54a74c74326dd
Reviewed-on: https://go-review.googlesource.com/c/138957
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: 's avatarRick Hudson <rlh@golang.org>
Reviewed-by: 's avatarMichael Knyszek <mknyszek@google.com>
      69e666e4
    • Austin Clements's avatar
      runtime: track all heap arenas in a slice · e500ffd8
      Austin Clements authored 
      Currently, there's no efficient way to iterate over the Go heap. We're
going to need this for fast free page sweeping, so this CL adds a
slice of all allocated heap arenas. This will also be useful for
generational GC.

For #18155.

Change-Id: I58d126cfb9c3f61b3125d80b74ccb1b2169efbcc
Reviewed-on: https://go-review.googlesource.com/c/138076
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: 's avatarRick Hudson <rlh@golang.org>
      e500ffd8
    • Alan Donovan's avatar
      cmd/vet/all: use x/tools/go/analysis/cmd/vet not cmd/vet · b2df0bd5
      Alan Donovan authored 
      cmd/vet/all applies vet to all packages in the standard tree.
It is run for every configuration using this command:
  GO_BUILDER_NAME=misc-vetall go tool dist test
by the misc-vetall builder (see chart at build.golang.org).

Ideally we would switch to 'go vet', but it effectively does a partial
build. This means that its analysis has accurate type information, so
it reports slightly fewer spurious diagnostics. However, it is more
than twice as slow.

Instead, cmd/vet/all builds and runs
golang.org/x/tools/go/analysis/cmd/vet, which uses x/tools/go/packages
to load the entire std lib from source. It takes about 4min to run all
OS/ARCH pairs. An important consequence is that golang.org/x/tools
must be on your $GOPATH to run cmd/vet/all. The test has been
temporarily modified to warn and skip if this is not the case.

This is a preparatory step for switching to the new
cmd/vet based on vet-lite.

Whitelist changes:
- The two "deadcode" diagnostics removed from the whitelist were due
  to if-conditions that could now be proven false.
- The asmdecl warnings are now printed with the log.Printf prefix,
  so they are discarded by the parser and needn't be whitelisted.

Change-Id: I6486508b0de2cd947c897523af086a408cbaf4a8
Reviewed-on: https://go-review.googlesource.com/c/149097Reviewed-by: 's avatarBrad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
      b2df0bd5
    • Alan Donovan's avatar
      cmd/vendor: update to golang.org/x/tools@f62bfb54 · a18638c0
      Alan Donovan authored 
      Change-Id: I3b3035784ce89ba2ac5ab8f6448c45a3d38fa97d
Reviewed-on: https://go-review.googlesource.com/c/149778
Run-TryBot: Alan Donovan <adonovan@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: 's avatarIan Lance Taylor <iant@golang.org>
      a18638c0
  3. 14 Nov, 2018 14 commits
  5. 13 Nov, 2018 18 commits