Skip to content

G
golang
Switch branch/tag
  1. 15 Nov, 2018 6 commits
    • Austin Clements's avatar
      runtime: implement efficient page reclaimer · 5333550b
      Austin Clements authored 
      When we attempt to allocate an N page span (either for a large
allocation or when an mcentral runs dry), we first try to sweep spans
to release N pages. Currently, this can be extremely expensive:
sweeping a span to emptiness is the hardest thing to ask for and the
sweeper generally doesn't know where to even look for potentially
fruitful results. Since this is on the critical path of many
allocations, this is unfortunate.

This CL changes how we reclaim empty spans. Instead of trying lots of
spans and hoping for the best, it uses the newly introduced span marks
to efficiently find empty spans. The span marks (and in-use bits) are
in a dense bitmap, so these spans can be found with an efficient
sequential memory scan. This approach can scan for unmarked spans at
about 300 GB/ms and can free unmarked spans at about 32 MB/ms. We
could probably significantly improve the rate at which is can free
unmarked spans, but that's a separate issue.

Like the current reclaimer, this is still linear in the number of
spans that are swept, but the constant factor is now so vanishingly
small that it doesn't matter.

The benchmark in #18155 demonstrates both significant page reclaiming
delays, and object reclaiming delays. With "-retain-count=20000000
-preallocate=true -loop-count=3", the benchmark demonstrates several
page reclaiming delays on the order of 40ms. After this change, the
page reclaims are insignificant. The longest sweeps are still ~150ms,
but are object reclaiming delays. We'll address those in the next
several CLs.

Updates #18155.

Fixes #21378 by completely replacing the logic that had that bug.

Change-Id: Iad80eec11d7fc262d02c8f0761ac6998425c4064
Reviewed-on: https://go-review.googlesource.com/c/138959
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: 's avatarRick Hudson <rlh@golang.org>
      5333550b
    • Austin Clements's avatar
      runtime: mark span when marking any object on the span · ba1698e9
      Austin Clements authored 
      This adds a mark bit for each span that is set if any objects on the
span are marked. This will be used for sweeping.

For #18155.

The impact of this is negligible for most benchmarks, and < 1% for
GC-heavy benchmarks.

name                       old time/op  new time/op  delta
Garbage/benchmem-MB=64-12  2.18ms ± 0%  2.20ms ± 1%  +0.88%  (p=0.000 n=16+18)

(https://perf.golang.org/search?q=upload:20180928.1)

name                      old time/op    new time/op    delta
BinaryTree17-12              2.68s ± 1%     2.68s ± 1%    ~     (p=0.707 n=17+19)
Fannkuch11-12                2.28s ± 0%     2.39s ± 0%  +4.95%  (p=0.000 n=19+18)
FmtFprintfEmpty-12          40.3ns ± 4%    39.4ns ± 2%  -2.27%  (p=0.000 n=17+18)
FmtFprintfString-12         67.9ns ± 1%    68.3ns ± 1%  +0.55%  (p=0.000 n=18+19)
FmtFprintfInt-12            75.7ns ± 1%    76.1ns ± 1%  +0.44%  (p=0.005 n=18+19)
FmtFprintfIntInt-12          123ns ± 1%     121ns ± 1%  -1.00%  (p=0.000 n=18+18)
FmtFprintfPrefixedInt-12     150ns ± 0%     148ns ± 0%  -1.33%  (p=0.000 n=16+13)
FmtFprintfFloat-12           208ns ± 0%     204ns ± 0%  -1.92%  (p=0.000 n=13+17)
FmtManyArgs-12               501ns ± 1%     498ns ± 0%  -0.55%  (p=0.000 n=19+17)
GobDecode-12                6.24ms ± 0%    6.25ms ± 1%    ~     (p=0.113 n=20+19)
GobEncode-12                5.33ms ± 0%    5.29ms ± 1%  -0.72%  (p=0.000 n=20+18)
Gzip-12                      220ms ± 1%     218ms ± 1%  -1.02%  (p=0.000 n=19+19)
Gunzip-12                   35.5ms ± 0%    35.7ms ± 0%  +0.45%  (p=0.000 n=16+18)
HTTPClientServer-12         77.9µs ± 1%    77.7µs ± 1%  -0.30%  (p=0.047 n=20+19)
JSONEncode-12               8.82ms ± 0%    8.93ms ± 0%  +1.20%  (p=0.000 n=18+17)
JSONDecode-12               47.3ms ± 0%    47.0ms ± 0%  -0.49%  (p=0.000 n=17+18)
Mandelbrot200-12            3.69ms ± 0%    3.68ms ± 0%  -0.25%  (p=0.000 n=19+18)
GoParse-12                  3.13ms ± 1%    3.13ms ± 1%    ~     (p=0.640 n=20+20)
RegexpMatchEasy0_32-12      76.2ns ± 1%    76.2ns ± 1%    ~     (p=0.818 n=20+19)
RegexpMatchEasy0_1K-12       226ns ± 0%     226ns ± 0%  -0.22%  (p=0.001 n=17+18)
RegexpMatchEasy1_32-12      71.9ns ± 1%    72.0ns ± 1%    ~     (p=0.653 n=18+18)
RegexpMatchEasy1_1K-12       355ns ± 1%     356ns ± 1%    ~     (p=0.160 n=18+19)
RegexpMatchMedium_32-12      106ns ± 1%     106ns ± 1%    ~     (p=0.325 n=17+20)
RegexpMatchMedium_1K-12     31.1µs ± 2%    31.2µs ± 0%  +0.59%  (p=0.007 n=19+15)
RegexpMatchHard_32-12       1.54µs ± 2%    1.53µs ± 2%  -0.78%  (p=0.021 n=17+18)
RegexpMatchHard_1K-12       46.0µs ± 1%    45.9µs ± 1%  -0.31%  (p=0.025 n=17+19)
Revcomp-12                   391ms ± 1%     394ms ± 2%  +0.80%  (p=0.000 n=17+19)
Template-12                 59.9ms ± 1%    59.9ms ± 1%    ~     (p=0.428 n=20+19)
TimeParse-12                 304ns ± 1%     312ns ± 0%  +2.88%  (p=0.000 n=20+17)
TimeFormat-12                318ns ± 0%     326ns ± 0%  +2.64%  (p=0.000 n=20+17)

(https://perf.golang.org/search?q=upload:20180928.2)

Change-Id: I336b9bf054113580a24103192904c8c76593e90e
Reviewed-on: https://go-review.googlesource.com/c/138958
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: 's avatarRick Hudson <rlh@golang.org>
Reviewed-by: 's avatarMichael Knyszek <mknyszek@google.com>
      ba1698e9
    • Austin Clements's avatar
      runtime: record in-use spans in a page-indexed bitmap · 69e666e4
      Austin Clements authored 
      This adds a bitmap indexed by page number that marks the starts of
in-use spans. This will be used to quickly find in-use spans with no
marked objects for sweeping.

For #18155.

Change-Id: Icee56f029cde502447193e136fa54a74c74326dd
Reviewed-on: https://go-review.googlesource.com/c/138957
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: 's avatarRick Hudson <rlh@golang.org>
Reviewed-by: 's avatarMichael Knyszek <mknyszek@google.com>
      69e666e4
    • Austin Clements's avatar
      runtime: track all heap arenas in a slice · e500ffd8
      Austin Clements authored 
      Currently, there's no efficient way to iterate over the Go heap. We're
going to need this for fast free page sweeping, so this CL adds a
slice of all allocated heap arenas. This will also be useful for
generational GC.

For #18155.

Change-Id: I58d126cfb9c3f61b3125d80b74ccb1b2169efbcc
Reviewed-on: https://go-review.googlesource.com/c/138076
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: 's avatarRick Hudson <rlh@golang.org>
      e500ffd8
    • Alan Donovan's avatar
      cmd/vet/all: use x/tools/go/analysis/cmd/vet not cmd/vet · b2df0bd5
      Alan Donovan authored 
      cmd/vet/all applies vet to all packages in the standard tree.
It is run for every configuration using this command:
  GO_BUILDER_NAME=misc-vetall go tool dist test
by the misc-vetall builder (see chart at build.golang.org).

Ideally we would switch to 'go vet', but it effectively does a partial
build. This means that its analysis has accurate type information, so
it reports slightly fewer spurious diagnostics. However, it is more
than twice as slow.

Instead, cmd/vet/all builds and runs
golang.org/x/tools/go/analysis/cmd/vet, which uses x/tools/go/packages
to load the entire std lib from source. It takes about 4min to run all
OS/ARCH pairs. An important consequence is that golang.org/x/tools
must be on your $GOPATH to run cmd/vet/all. The test has been
temporarily modified to warn and skip if this is not the case.

This is a preparatory step for switching to the new
cmd/vet based on vet-lite.

Whitelist changes:
- The two "deadcode" diagnostics removed from the whitelist were due
  to if-conditions that could now be proven false.
- The asmdecl warnings are now printed with the log.Printf prefix,
  so they are discarded by the parser and needn't be whitelisted.

Change-Id: I6486508b0de2cd947c897523af086a408cbaf4a8
Reviewed-on: https://go-review.googlesource.com/c/149097Reviewed-by: 's avatarBrad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
      b2df0bd5
    • Alan Donovan's avatar
      cmd/vendor: update to golang.org/x/tools@f62bfb54 · a18638c0
      Alan Donovan authored 
      Change-Id: I3b3035784ce89ba2ac5ab8f6448c45a3d38fa97d
Reviewed-on: https://go-review.googlesource.com/c/149778
Run-TryBot: Alan Donovan <adonovan@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: 's avatarIan Lance Taylor <iant@golang.org>
      a18638c0
  3. 14 Nov, 2018 14 commits
  5. 13 Nov, 2018 18 commits
  7. 12 Nov, 2018 2 commits
    • Emmanuel T Odeke's avatar
      net: preserve unexpired context values for LookupIPAddr · 5d392600
      Emmanuel T Odeke authored 
      To avoid any cancelation of the parent context from affecting
lookupGroup operations, Resolver.LookupIPAddr previously used
an entirely new context created from context.Background().
However, this meant that all the values in the parent context
with which LookupIPAddr was invoked were dropped.

This change provides a custom context implementation
that only preserves values of the parent context by composing
context.Background() and the parent context. It only falls back
to the parent context to perform value lookups if the parent
context has not yet expired.
This context is never canceled, and has no deadlines.

Fixes #28600

Change-Id: If2f570caa26c65bad638b7102c35c79d5e429fea
Reviewed-on: https://go-review.googlesource.com/c/148698
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
Reviewed-by: 's avatarBrad Fitzpatrick <bradfitz@golang.org>
      5d392600
    • Filippo Valsorda's avatar
      crypto/tls: don't modify Config.Certificates in BuildNameToCertificate · 70e3b1df
      Filippo Valsorda authored 
      The Config does not own the memory pointed to by the Certificate slice.
Instead, opportunistically use Certificate.Leaf and let the application
set it if it desires the performance gain.

This is a partial rollback of CL 107627. See the linked issue for the
full explanation.

Fixes #28744

Change-Id: I33ce9e6712e3f87939d9d0932a06d24e48ba4567
Reviewed-on: https://go-review.googlesource.com/c/149098Reviewed-by: 's avatarEmmanuel Odeke <emm.odeke@gmail.com>
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
      70e3b1df