• Charles L. Dorian's avatar
    math: improve Atan, Asin and Acos accuracy · 35c3afdb
    Charles L. Dorian authored
    pkg/math/all_test.go tests Atan (and therefore Asin and Acos) to a
    relative accuracy of 4e-16, but the test vector misses values where
    the old algorithm was in error by more than that. For example:
    
    x            newError   oldError
    0.414215746  1.41e-16  -4.24e-16
    0.414216076  1.41e-16  -4.24e-16
    0.414217632  1.41e-16  -4.24e-16
    0.414218770  1.41e-16  -4.24e-16
    0.414225466  0         -5.65e-16
    0.414226244  1.41e-16  -4.24e-16
    0.414228756  0         -5.65e-16
    0.414235089  0         -5.65e-16
    0.414237070  0         -5.65e-16
    
    R=rsc, golang-dev
    CC=golang-dev
    https://golang.org/cl/6302093
    35c3afdb
atan.go 2.96 KB