Fixes #9637
Fixes #10120

Change-Id: I3728239089efb94d04cd4115c9f840afd7badeaf
Reviewed-on: https://go-review.googlesource.com/9715Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>

Not a language change.

Fixes #11350.

Change-Id: I9b905f17d1ef2722cab4bae38a037270165c7d95
Reviewed-on: https://go-review.googlesource.com/11369Reviewed-by: Rob Pike <r@golang.org>

Port of https://go-review.googlesource.com/11365

Fixes #11357.

Change-Id: Icd20fa038696a8853d1d14477e1c1132938b3e2e
Reviewed-on: https://go-review.googlesource.com/11368Reviewed-by: Alan Donovan <adonovan@google.com>

Port of https://go-review.googlesource.com/11363.

Fixes #11367.

Change-Id: Ie0a82bcfab782c514d1947e7a6b190e286afd159
Reviewed-on: https://go-review.googlesource.com/11367Reviewed-by: Alan Donovan <adonovan@google.com>

When heapBitsSetType repeats a source bitmap with a scalar tail
(typ.ptrdata < typ.size), it lays out the tail upon reaching the end
of the source bitmap by simply increasing the number of bits claimed
to be in the incoming bit buffer. This causes later iterations to read
the appropriate number of zeros out of the bit buffer before starting
on the next repeat of the source bitmap.

Currently, however, later iterations of the loop continue to read bits
from the source bitmap *regardless of the number of bits currently in
the bit buffer*. The bit buffer can only hold 32 or 64 bits, so if the
scalar tail is large and the padding bits exceed the size of the bit
buffer, the read from the source bitmap on the next iteration will
shift the incoming bits into oblivion when it attempts to put them in
the bit buffer. When the buffer does eventually shift down to where
these bits were supposed to be, it will contain zeros. As a result,
words that should be marked as pointers on later repetitions are
marked as scalars, so the garbage collector does not trace them. If
this is the only reference to an object, it will be incorrectly freed.

Fix this by adding logic to drain the bit buffer down if it is large
instead of reading more bits from the source bitmap.

Fixes #11286.

Change-Id: I964432c4b9f1cec334fc8c3da0ff16460203feb6
Reviewed-on: https://go-review.googlesource.com/11360Reviewed-by: Russ Cox <rsc@golang.org>

h_spans can be accessed concurrently without synchronization from
other threads, which means it needs the appropriate memory barriers on
weakly ordered machines. It happens to already have the necessary
memory barriers because all accesses to h_spans are currently
protected by the heap lock and the unlocks happen in exactly the
places where release barriers are needed, but it's easy to imagine
that this could change in the future. Document the fact that we're
depending on the barrier implied by the unlock.

Related to issue #9984.

Change-Id: I1bc3c95cd73361b041c8c95cd4bb92daf8c1f94a
Reviewed-on: https://go-review.googlesource.com/11361Reviewed-by: Rick Hudson <rlh@golang.org>
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>

Port of https://go-review.googlesource.com/11344 to std repo.

Fixes #11325.

Change-Id: I634beaf77cbaeb09de50aa1410e8c53fc37b19df
Reviewed-on: https://go-review.googlesource.com/11317Reviewed-by: Alan Donovan <adonovan@google.com>

There are two conditions to worry about:

1) The shift count cannot be negative. Since the evaluator uses unsigned
arithmetic throughout, this means checking that the high bit of
the shift count is always off, which is done by converting to int64
and seeing if the result is negative.

2) For right shifts, the value cannot be negative. We don't want a
high bit in the value because right shifting a value depends on the
sign, and for clarity we always want unsigned shifts.

Next step is to build some testing infrastructure for the parser.

Change-Id: I4c46c79989d02c107fc64954403fc18613763f1d
Reviewed-on: https://go-review.googlesource.com/11326Reviewed-by: Russ Cox <rsc@golang.org>

It was otherwise not being preserved across
specific Decode->Encode->Decode calls.

Fixes #11287

Change-Id: I40602da7fa39ec67403bed52ff403f361c6171bb
Reviewed-on: https://go-review.googlesource.com/11256Reviewed-by: Nigel Tao <nigeltao@golang.org>

Documentation change only.

Fixes #11247.

Change-Id: Ib412de2d643292dbe42b56dee955bdb877aee81b
Reviewed-on: https://go-review.googlesource.com/11329Reviewed-by: David Symonds <dsymonds@golang.org>

This appears to be some legacy which is no longer used.

Change-Id: I469beb59a90853e8de910158f179b32f1aa14c7d
Reviewed-on: https://go-review.googlesource.com/11304Reviewed-by: Andrew Gerrand <adg@golang.org>
Run-TryBot: Andrew Gerrand <adg@golang.org>

Some of those consts were supposed to be vars.

Caught by Ingo Oeser.

Change-Id: Ifc12e4a8ee61ebf5174e4ad923956c546dc096e2
Reviewed-on: https://go-review.googlesource.com/11296Reviewed-by: Andrew Gerrand <adg@golang.org>

The change that "fixed" LSH was incorrect, and the fix for RSH was poor.
Make both use a correct, simple test: if the 64-bit value as a signed
integer is negative, it's an error.

Really fixes #11278.

Change-Id: I72cca03d7ad0d64fd649fa33a9ead2f31bd2977b
Reviewed-on: https://go-review.googlesource.com/11325Reviewed-by: Andrew Gerrand <adg@golang.org>

And vice versa.

The flags are tightly coupled so make the connection clear.

Change-Id: I505f76be631ffa6e489a441c2f3c717aa09ec802
Reviewed-on: https://go-review.googlesource.com/11324Reviewed-by: Andrew Gerrand <adg@golang.org>

This CL removes the single and racy use of mheap.arena_end outside
of the bookkeeping done in mHeap_init and mHeap_Alloc.
There should be no way for heapBitsForSpan to see a pointer to
an invalid span. This CL makes the check for this more precise by
checking that the pointer is between mheap_.arena_start and
mheap_.arena_used instead of mheap_.arena_end.

Change-Id: I1200b54353ee1eda002d92645fd8d26048600ceb
Reviewed-on: https://go-review.googlesource.com/11342Reviewed-by: Austin Clements <austin@google.com>

Change-Id: I35d20ed958c32d464b2c9d849403b6e3f99b6482
Reviewed-on: https://go-review.googlesource.com/11343Reviewed-by: Russ Cox <rsc@golang.org>

In order to avoid a race with a concurrent write barrier or garbage
collector thread, any update to arena_used must be preceded by mapping
the corresponding heap bitmap and spans array memory. Otherwise, the
concurrent access may observe that a pointer falls within the heap
arena, but then attempt to access unmapped memory to look up its span
or heap bits.

Commit d57c889a fixed all of the places where we updated arena_used
immediately before mapping the heap bitmap and spans, but it missed
the one place where we update arena_used and depend on later code to
update it again and map the bitmap and spans. This creates a window
where the original race can still happen. This commit fixes this by
mapping the heap bitmap and spans before this arena_used update as
well. This code path is only taken when expanding the heap reservation
on 32-bit over a hole in the address space, so these extra mmap calls
should have negligible impact.

Fixes #10212, #11324.

Change-Id: Id67795e6c7563eb551873bc401e5cc997aaa2bd8
Reviewed-on: https://go-review.googlesource.com/11340
Run-TryBot: Austin Clements <austin@google.com>
Reviewed-by: Rick Hudson <rlh@golang.org>
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>

The unsynchronized accesses to mheap_.arena_used in the concurrent
part of the garbage collector look like a problem waiting to happen.
In fact, they are safe, but the reason is somewhat subtle and
undocumented. This commit documents this reasoning.

Related to issue #9984.

Change-Id: Icdbf2329c1aa11dbe2396a71eb5fc2a85bd4afd5
Reviewed-on: https://go-review.googlesource.com/11254Reviewed-by: Dmitry Vyukov <dvyukov@google.com>

Historically we have declined to try to provide real support for URLs
that contain %2F in the path, but they seem to be popping up more
often, especially in (arguably ill-considered) REST APIs that shoehorn
entire paths into individual path elements.

The obvious thing to do is to introduce a URL.RawPath field that
records the original encoding of Path and then consult it during
URL.String and URL.RequestURI. The problem with the obvious thing
is that it breaks backward compatibility: if someone parses a URL
into u, modifies u.Path, and calls u.String, they expect the result
to use the modified u.Path and not the original raw encoding.

Split the difference by treating u.RawPath as a hint: the observation
is that there are many valid encodings of u.Path. If u.RawPath is one
of them, use it. Otherwise compute the encoding of u.Path as before.

If a client does not use RawPath, the only change will be that String
selects a different valid encoding sometimes (the original passed
to Parse).

This ensures that, for example, HTTP requests use the exact
encoding passed to http.Get (or http.NewRequest, etc).

Also add new URL.EscapedPath method for access to the actual
escaped path. Clients should use EscapedPath instead of
reading RawPath directly.

All the old workarounds remain valid.

Fixes #5777.
Might help #9859.
Fixes #7356.
Fixes #8767.
Fixes #8292.
Fixes #8450.
Fixes #4860.
Fixes #10887.
Fixes #3659.
Fixes #8248.
Fixes #6658.
Reduces need for #2782.

Change-Id: I77b88f14631883a7d74b72d1cf19b0073d4f5473
Reviewed-on: https://go-review.googlesource.com/11302Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

The test was translated from shell incorrectly,
and it depended on having hg installed, which
may not be the case.

Moved repo to GitHub, updated code, and fixed
go list ... command to be expected to succeed.

Fixes test for #8181.

Change-Id: I7f3e8fb20cd16cac5ed24de6fd952003bc5e08d4
Reviewed-on: https://go-review.googlesource.com/11301Reviewed-by: Ian Lance Taylor <iant@golang.org>

Change-Id: I3463826aa760aa5984dec4fc043b95fd2a5120ac
Reviewed-on: https://go-review.googlesource.com/11240Reviewed-by: Andrew Gerrand <adg@golang.org>

In the parser, the shift value is always a uint64.

Change-Id: I9b50295a9f7d174ed1f6f9baf78ec0ed43db417f
Reviewed-on: https://go-review.googlesource.com/11322Reviewed-by: Andrew Gerrand <adg@golang.org>

A header of ": value" results in an empty key. Do not add
it to the headers, because RFC7230 (section 3.2) says that
field-names are tokens, which are one or more characters.

Fixes #11205.

Change-Id: I883be89da1489dc84f98523786b019d1d0169d46
Reviewed-on: https://go-review.googlesource.com/11242Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

Fixes #11278.

Change-Id: Ic46fda0f42cefedc3f6085c0e77e67616ce4955e
Reviewed-on: https://go-review.googlesource.com/11297Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

Most important: skip test on darwin/arm64 for unclear reasons.

First cut at the test missed this feature of go doc: when asking for
the docs for a type, include any function that looks like it constructs
a that type as a return value.

Change-Id: I124e7695e5d365e2b12524b541a9a4e6e0300fbc
Reviewed-on: https://go-review.googlesource.com/11295Reviewed-by: Rob Pike <r@golang.org>

Some Linux kernels apparently have a sysctl that prohibits
nonprivileged processes from creating user namespaces.  If we see a
failure for that reason, skip the test.

Fixes #11261.

Change-Id: I82dfcaf475eea4eaa387941373ce7165df4848ad
Reviewed-on: https://go-review.googlesource.com/11269Reviewed-by: Mikio Hara <mikioh.mikioh@gmail.com>

nacl is really giving a hard time. avoid all external dependencies in the test.
Worked with trybots, failed in the build. No explanation, but this should fix it.

TBR=rsc

Change-Id: Icb644286dbce88f17ee3d96ad90efba34a80a92d
Reviewed-on: https://go-review.googlesource.com/11291Reviewed-by: Rob Pike <r@golang.org>

Refactor main a bit to make it possible to run tests without an exec every time.
(Makes a huge difference in run time.)

Add a silver test. Not quite golden, since it looks for pieces rather than the
full output, and also includes tests for what should not appear.

Fixes #10920.

Change-Id: I6a4951cc14e61763379754a10b0cc3484d30c267
Reviewed-on: https://go-review.googlesource.com/11272Reviewed-by: Russ Cox <rsc@golang.org>
Run-TryBot: Rob Pike <r@golang.org>

This sometime worries new contributors.
Hopefully mentioning it here will help.

Fixes #11300.

Change-Id: Ica7f10d749731704ac6a2c39c7dcba389996011e
Reviewed-on: https://go-review.googlesource.com/11236Reviewed-by: Ian Lance Taylor <iant@golang.org>

All of the heavy-lifting was done by minux@, with his external-linking support
for darwin/arm64: golang.org/cl/8781

Change-Id: I7c9fbc19246f418c065c92fb2c13c00026ff0f82
Reviewed-on: https://go-review.googlesource.com/11127
Run-TryBot: Srdjan Petrovic <spetrovic@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>

Change-Id: Ia13d1fa450e88e278b81048b99686395ca474c99
Reviewed-on: https://go-review.googlesource.com/11259Reviewed-by: Russ Cox <rsc@golang.org>

Fixes #11284.

Change-Id: I4ecc4e4cd3c1b3467b43e4ba9666ea6db5fb61a5
Reviewed-on: https://go-review.googlesource.com/11268Reviewed-by: Alan Donovan <adonovan@google.com>

Change-Id: I42cfdb389282478ce0e29436464f2048ed087429
Reviewed-on: https://go-review.googlesource.com/11290Reviewed-by: Rob Pike <r@golang.org>

When GO15VENDOREXPERIMENT=1 is in the environment,
this CL changes the resolution of import paths according to
the Go 1.5 vendor proposal:

	If there is a source directory d/vendor, then,
	when compiling a source file within the subtree rooted at d,
	import "p" is interpreted as import "d/vendor/p" if that exists.

	When there are multiple possible resolutions,
	the most specific (longest) path wins.

	The short form must always be used: no import path can
	contain “/vendor/” explicitly.

	Import comments are ignored in vendored packages.

The goal of these changes is to allow authors to vendor (copy) external
packages into their source trees without any modifications to the code.
This functionality has been achieved in tools like godep, nut, and gb by
requiring GOPATH manipulation. This alternate directory-based approach
eliminates the need for GOPATH manipulation and in keeping with the
go command's use of directory layout-based configuration.

The flag allows experimentation with these vendoring semantics once
Go 1.5 is released, without forcing them on by default. If the experiment
is deemed a success, the flag will default to true in Go 1.6 and then be
removed in Go 1.7.

For more details, see the original proposal by Keith Rarick at
https://groups.google.com/d/msg/golang-dev/74zjMON9glU/dGhnoi2IMzsJ.

Change-Id: I2c6527e777d14ac6dc43c53e4b3ff24f3279216e
Reviewed-on: https://go-review.googlesource.com/10923Reviewed-by: Andrew Gerrand <adg@golang.org>

The -importmap option takes an argument of the form old=new
and specifies that import "old" should be interpreted as if it said
import "new". The option may be repeated to specify multiple mappings.

This option is here to support the go command's new -vendor flag.

Change-Id: I31b4ed4249b549982a720bf61bb230462b33c59b
Reviewed-on: https://go-review.googlesource.com/10922Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>

Currently its possible for the garbage collector to observe
uninitialized memory or stale heap bitmap bits on weakly ordered
architectures such as ARM and PPC. On such architectures, the stores
that zero newly allocated memory and initialize its heap bitmap may
move after a store in user code that makes the allocated object
observable by the garbage collector.

To fix this, add a "publication barrier" (also known as an "export
barrier") before returning from mallocgc. This is a store/store
barrier that ensures any write done by user code that makes the
returned object observable to the garbage collector will be ordered
after the initialization performed by mallocgc. No barrier is
necessary on the reading side because of the data dependency between
loading the pointer and loading the contents of the object.

Fixes one of the issues raised in #9984.

Change-Id: Ia3d96ad9c5fc7f4d342f5e05ec0ceae700cd17c8
Reviewed-on: https://go-review.googlesource.com/11083Reviewed-by: Rick Hudson <rlh@golang.org>
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
Reviewed-by: Minux Ma <minux@golang.org>
Reviewed-by: Martin Capitanio <capnm9@gmail.com>
Reviewed-by: Russ Cox <rsc@golang.org>

These tests were broken by https://go-review.googlesource.com/#/c/11227/
which fixed the LZW encoder to reject invalid input.

For TestNoPalette, the LZW encoder with a litWidth of 2 now rejects an
input byte of 128, so we change 128 to 3, as 3 <= (1<<2 - 1).

For TestPixelOutsidePaletteRange, the LZW encoder similarly rejects an
input byte of 255. Prior to golang.org/cl/11227, the encoder (again with
a litWidth of 2) accepted the 255 input byte, but masked it with (1<<2 -
1), so that the 255 test case was effectively the same as the 3 test
case. After that LZW CL, the 255 input byte is simply invalid, so we
remove it as a test case. The test still tests pixels outside of the
palette range, since 3 >= the length of the global palette, which is 2.

Change-Id: I50be9623ace016740e34801549c15f83671103eb
Reviewed-on: https://go-review.googlesource.com/11273Reviewed-by: David Symonds <dsymonds@golang.org>

Change-Id: I129d70304ae4e4694d9217826b18b341e3834d3c
Reviewed-on: https://go-review.googlesource.com/11201Reviewed-by: Andrew Gerrand <adg@golang.org>

Change-Id: I5917bea8bb35b0e725dcc56a68f3a70137cfc180
Reviewed-on: https://go-review.googlesource.com/9387Reviewed-by: Ian Lance Taylor <iant@golang.org>

This is a documentation fix that reflects the current reality.

Fixes #9673.

Change-Id: Ie436b277dfd1b68b13c67813d29c238d2c23b820
Reviewed-on: https://go-review.googlesource.com/11221Reviewed-by: Russ Cox <rsc@golang.org>