Clipping Utils¶
-
class
opacus.utils.clipping.ConstantFlatClipper(flat_value)[source]¶ A clipper that clips all gradients in such a way that their norm is at most equal to a specified value. This value is shared for all layers in a model. Note that the process of clipping really involves multiplying all gradients by a scaling factor. If this scaling factor is > 1.0, it is instead capped at 1.0. The net effect is that the final norm of the scaled gradients will be less than the specified value in such a case. Thus it is better to think of the specified value as an upper bound on the norm of final clipped gradients.
- Parameters
flat_value (
float) – Constant value that is used to normalize gradients such that their norm equals this value before clipping. This threshold value is used for all layers.
-
calc_clipping_factors(norms)[source]¶ Calculates the clipping factor based on the given norm of gradients for all layers, so that the new norm of clipped gradients is at most equal to
self.flat_value.
-
property
is_per_layer¶ Returns indicator as to whether different clipping is applied to each layer in the model. For this clipper, it is False.
- Return type
- Returns
Flag with value False
-
class
opacus.utils.clipping.ConstantPerLayerClipper(flat_values)[source]¶ A clipper that clips all gradients in such a way that their norm is at most equal to a specified value. This value is specified for each layer in a model. Note that the process of clipping really involves multiplying all gradients by a scaling factor. If this scaling factor is > 1.0, it is instead capped at 1.0. The net effect is that the final norm of the scaled gradients will be less than the specified value in such a case. Thus it is better to think of the specified value as an upper bound on the norm of final clipped gradients.
- Parameters
flat_values (
List[float]) – List of values that is used to normalize gradients for each layer such that the norm equals the corresponding value before clipping.
-
calc_clipping_factors(norms)[source]¶ Calculates separate clipping factors for each layer based on its corresponding norm of gradients, such that its new norm is at most equal to the flat value specified for that layer when instantiating the object of
ConstantPerLayerClipper.
-
property
is_per_layer¶ Returns indicator as to whether different clipping is applied to each layer in the model. For this clipper, it is True.
- Return type
- Returns
Flag with value True
-
class
opacus.utils.clipping.NormClipper[source]¶ An abstract class to calculate the clipping factor
-
calc_clipping_factors(norms)[source]¶ Calculates the clipping factor(s) based on the given parameters. A concrete subclass must implement this.
-
property
is_per_layer¶ Depending on type of clipper, returns indicator as to whether different clipping is applied to each layer in the model.
- Return type
- Returns
Flag indicator as to whether different clipping is applied to each layer in the model.
-