- class opacus.optimizers.ddp_perlayeroptimizer.DistributedPerLayerOptimizer(optimizer, *, noise_multiplier, max_grad_norm, expected_batch_size, loss_reduction='mean', generator=None, secure_mode=False)¶
DPOptimizerthat implements per layer clipping strategy and is compatible with distibured data parallel
Optimizer) – wrapped optimizer.
float) – noise multiplier
int]) – batch_size used for averaging gradients. When using Poisson sampling averaging denominator can’t be inferred from the actual batch size. Required is
loss_reduction="mean", ignored if
str) – Indicates if the loss reduction (for aggregating the gradients) is a sum or a mean operation. Can take values “sum” or “mean”
generator – torch.Generator() object used as a source of randomness for the noise
bool) – if
Trueuses noise generation approach robust to floating point arithmetic attacks. See
- property accumulated_iterations: int¶
Returns number of batches currently accumulated and not yet processed.
In other words
accumulated_iterationstracks the number of forward/backward passed done in between two optimizer steps. The value would typically be 1, but there are possible exceptions.
Used by privacy accountants to calculate real sampling rate.
- Return type:
Adds noise to clipped gradients. Stores clipped and noised result in
Performs gradient clipping. Stores clipped and aggregated gradients into p.summed_grad``