hxtorch.spiking.functional.surrogates.saturate
A collection of surrogate functions for the torch.clamp() function.
There are Classes deriving from torch.autograd.Function, that use torch.clamp() in the forward pass, and only use the surrogate gradient in the backward pass, but also python functions (having a _func suffix in their names) that implement the surrogate in forward direction. To enable the use of functools.partial on the apply methods of the torch.autograd.Function classes, these are wrapped.
Classes
|
Forward-pass uses torch.clamp(), but backward pass uses the identity as a surrogate gradient. |
|
Forward-pass uses torch.clamp(), but backward pass uses the gradient of a linear function with exponential roll off towards the saturation bounds. |
Functions
-
hxtorch.spiking.functional.surrogates.saturate.clamp(input: torch.Tensor, lower: torch.Tensor, upper: torch.Tensor) → torch.Tensor Wrapper for Clamp.apply()
-
hxtorch.spiking.functional.surrogates.saturate.exponential_rolloff(input: torch.Tensor, lower: torch.Tensor, upper: torch.Tensor, rolloff_margin: float = 0.03, rolloff_margin_abs: float = 0.05) → torch.Tensor Wrapper for ExponentialRolloff.apply()
-
hxtorch.spiking.functional.surrogates.saturate.exponential_rolloff_func(input: torch.Tensor, lower: torch.Tensor, upper: torch.Tensor, rolloff_margin: float = 0.03, rolloff_margin_abs: float = 0.05) → torch.Tensor Linear function capped at lower- and upper bounds with a roll off between the linear and the constant sections. :param input: Tensor, to which the function is applied to. :param lower: Lower threshold. If infinite, the roll off on the lower end
is not applied at all.
- Parameters
upper – Upper threshold.
rolloff_margin – Size of the margin from the bounds inwards, in which the roll off is active. Value relative to the distance between the bounds.
rolloff_margin_abs – Absolute size of the margin from the bounds inwards, in which the roll off is active. This value is needed as a fallback, in case one of the thresholds is infinite.