Layers

The QuantDense module defines a custom dense layer for neural networks that combines quantization and sparsity techniques for efficient inference. This module includes various constructors and methods to create and utilize quantized dense layers.

Constructor

• weight (learnable weights used for the dense layer)
• bias (learnable biases used for the dense layer)
• σ (activation function applied to the layer's output)
• weight_quantizer (weight quantization function)
• weight_sparsifier (weight sparsification function)
• output_quantizer (output quantization function)
• batchnorm (optional batch normalization layer)

Functor

QuantDense serves as a functor and it applies the layer to the input data x. It performs quantization of weights, sparsification, batch normalization (if enabled), and output quantization. If necessary the function resahpes the input.

using Random; Random.seed!(3);
x = Float32.([1 2]);
kwargs = (;
    init = (dims...) -> ClippedArray(dims...; lo = -1, hi = 1),
    output_quantizer = Ternary(1),
    weight_quantizer = Sign(),
    weight_sparsifier = identity,
    batchnorm = true,
)

qd = QuantDense(1 => 2, identity; kwargs...)

qd(x)

This function converts a QuantDense layer to a logic-compatible layer by applying the weight quantization and sparsification techniques. It returns a Dense layer suitable for efficient inference.

The FeatureQuantizer module defines a struct and associated functions for feature quantization within a neural network. Feature quantization involves discretizing input features using learnable weights and biases, and applying an output quantization function.

function (q::FeatureQuantizer)(x)

FeatureQuantizer serves as a functor and it applies the feature quantization operation to the input data x using the weights and biases stored in the FeatureQuantizer object and returns the quantized output.

This is an internal function and performs the forward pass of the feature quantization layer for input data with multiple dimensions. It computes the weighted sum of input data, applies biases, and quantizes the result. There is also a version for supplying one-dimensional input vector.

This function defines the reverse-mode automatic differentiation (AD) rule for the _forward_pass function. It specifies how gradients are propagated backward through the quantization layer during backpropagation.

• project_w: A function to project the gradient with respect to weights.
• project_b: A function to project the gradient with respect to biases.
• project_x: A function to project the gradient with respect to input data.

This function returns a tuple containing gradients with respect to weights, biases, and input data, along with the projected gradients for each.

This internal function is used for reverse-mode AD during backpropagation. It computes gradients for the feature quantization layer and returns the gradients for weights, biases, and input data. Δy is the gradient with respect to the output.