Estimators

AbstractEstimator

Estimators are used for estimation of the gradient of quantizers for which the true gradient does not exist.

Estimators are used for dispatch on backward pass, i.e. for each quantizer (for example Sign) and it is necessary to define specific method for pullback function.

STE(threshold::Real = 2)

It is the simplest estimator used in all quantizers like Sign, Heaviside or Ternary.

It requires a real positive number for threshold parameter, in case a negative number is supplied an ArgumentError exception is thrown.

Threshold is used, in the pullback function, to determine the range of values for which the gradient is calculated by an estimation function. Sign

PolynomialSTE()

Currently is only supported for binary Sign quantizer.

Does not require any additional parameters, simply indicates that a polynomial approximation of sign function is used and its respective derivative.

SwishSTE(β:Real = 5)

Currently is only supported for binary Sign quantizer.

It requires a real positive number for β parameter, in case a negative number is supplied an ArgumentError exception is thrown.

β is used as the parameter in the calculation of the swish function and its derivative in the pullback function. Sign

StochasticSTE()

To be defined.