DPRNN

class opacus.layers.dp_rnn.DPGRU(input_size, hidden_size, num_layers=1, bias=True, batch_first=False, dropout=0, bidirectional=False, proj_size=0)[source]

Applies a multi-layer gated recurrent unit (GRU) RNN to an input sequence.

DP-friendly drop-in replacement of the torch.nn.GRU module. Refer to torch.nn.GRU documentation for the model description, parameters and inputs/outputs.

After training this module can be exported and loaded by the original torch.nn implementation for inference.

class opacus.layers.dp_rnn.DPGRUCell(input_size, hidden_size, bias)[source]

A gated recurrent unit (GRU) cell

DP-friendly drop-in replacement of the torch.nn.GRUCell module to use in DPGRU. Refer to torch.nn.GRUCell documentation for the model description, parameters and inputs/outputs.

Initialize internal Module state, shared by both nn.Module and ScriptModule.

forward(input, hx=None, batch_size_t=None)[source]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Return type:

Tensor

class opacus.layers.dp_rnn.DPLSTM(input_size, hidden_size, num_layers=1, bias=True, batch_first=False, dropout=0, bidirectional=False, proj_size=0)[source]

Applies a multi-layer long short-term memory (LSTM) RNN to an input sequence.

DP-friendly drop-in replacement of the torch.nn.LSTM module. Refer to torch.nn.LSTM documentation for the model description, parameters and inputs/outputs.

After training this module can be exported and loaded by the original torch.nn implementation for inference.

class opacus.layers.dp_rnn.DPLSTMCell(input_size, hidden_size, bias)[source]

A long short-term memory (LSTM) cell.

DP-friendly drop-in replacement of the torch.nn.LSTMCell module to use in DPLSTM. Refer to torch.nn.LSTMCell documentation for the model description, parameters and inputs/outputs.

Initialize internal Module state, shared by both nn.Module and ScriptModule.

forward(input, hx=None, batch_size_t=None)[source]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Return type:

Tuple[Tensor, Tensor]

class opacus.layers.dp_rnn.DPRNN(input_size, hidden_size, num_layers=1, bias=True, batch_first=False, dropout=0, bidirectional=False, proj_size=0, nonlinearity='tanh')[source]

Applies a multi-layer Elman RNN with :math:` anh` or :math:` ext{ReLU}` non-linearity to an input sequence.

DP-friendly drop-in replacement of the torch.nn.RNN module. Refer to torch.nn.RNN documentation for the model description, parameters and inputs/outputs.

After training this module can be exported and loaded by the original torch.nn implementation for inference.

class opacus.layers.dp_rnn.DPRNNBase(mode, input_size, hidden_size, num_layers=1, bias=True, batch_first=False, dropout=0.0, bidirectional=False, proj_size=0, cell_params=None)[source]

Base class for all RNN-like sequence models.

DP-friendly drop-in replacement of the torch.nn.RNNBase module. After training this module can be exported and loaded by the original torch.nn implementation for inference.

This module implements multi-layer (Type-2, see [this issue](https://github.com/pytorch/pytorch/issues/4930#issuecomment-361851298)) bi-directional sequential model based on abstract cell. Cell should be a subclass of DPRNNCellBase.

Limitations: - proj_size > 0 is not implemented - this implementation doesn’t use cuDNN

forward(input, state_init=None)[source]

Forward pass of a full RNN, containing one or many single- or bi-directional layers. Implemented for an abstract cell type.

Note: proj_size > 0 is not supported here. Cell state size is always equal to hidden state size.

Return type:

Tuple[Union[Tensor, PackedSequence], Union[Tensor, Tuple[Tensor, Tensor]]]

Inputs: input, h_0/(h_0, c_0)
input: Input sequence. Tensor of shape [T, B, D] ([B, T, D] if batch_first=True)

or PackedSequence.

h_0: Initial hidden state for each element in the batch. Tensor of shape [L*P, B, H]. Default to zeros. c_0: Initial cell state for each element in the batch. Only for cell types with an additional state.

Tensor of shape [L*P, B, H]. Default to zeros.

Outputs: output, h_n/(h_n, c_n)
output: Output features (h_t) from the last layer of the model for each t. Tensor of

shape [T, B, P*H] ([B, T, P*H] if batch_first=True), or PackedSequence.

h_n: Final hidden state for each element in the batch. Tensor of shape [L*P, B, H]. c_n: Final cell state for each element in the batch. Tensor of shape [L*P, B, H].

where

T = sequence length B = batch size D = input_size H = hidden_size L = num_layers P = num_directions (2 if bidirectional=True else 1)

forward_layer(x, h_0, c_0, batch_sizes, cell, max_batch_size, seq_length, is_packed, reverse_layer)[source]

Forward pass of a single RNN layer (one direction). Implemented for an abstract cell type.

Inputs: x, h_0, c_0

x: Input sequence. Tensor of shape [T, B, D] or PackedSequence if is_packed = True. h_0: Initial hidden state. Tensor of shape [B, H]. c_0: Initial cell state. Tensor of shape [B, H]. Only for cells with additional

state c_t, e.g. DPLSTMCell.

Outputs: h_t, h_last, c_last
h_t: Final hidden state, output features (h_t) for each timestep t. Tensor of

shape [T, B, H] or list of length T with tensors [B, H] if PackedSequence is used.

h_last: The last hidden state. Tensor of shape [B, H]. c_last: The last cell state. Tensor of shape [B, H]. None if cell has no additional state.

where

T = sequence length B = batch size D = input_size (for this specific layer) H = hidden_size (output size, for this specific layer)

Parameters:

  • batch_sizes (Tensor) – Contains the batch sizes as stored in PackedSequence

  • cell (DPRNNCellBase) – Module implementing a single cell of the network, must be an instance of DPRNNCell

  • max_batch_size (int) – batch size

  • seq_length (int) – sequence length

  • is_packed (bool) – whether PackedSequence is used as input

  • reverse_layer (bool) – if True, it will run forward pass for a reversed layer

Return type:

Tuple[Union[Tensor, List[Tensor]], Tensor, Tensor]

iterate_layers(*args)[source]

Iterate through all the layers and through all directions within each layer.

Arguments should be list-like of length num_layers * num_directions where each element corresponds to (layer, direction) pair. The corresponding elements of each of these lists will be iterated over.

Example

num_layers = 3 bidirectional = True

for layer, directions in self.iterate_layers(self.cell, h):
for dir, (cell, hi) in directions:

print(layer, dir, hi)

# 0 0 h[0] # 0 1 h[1] # 1 0 h[2] # 1 1 h[3] # 2 0 h[4] # 2 1 h[5]

class opacus.layers.dp_rnn.DPRNNCell(input_size, hidden_size, bias, nonlinearity='tanh')[source]

An Elman RNN cell with tanh or ReLU non-linearity.

DP-friendly drop-in replacement of the torch.nn.RNNCell module to use in DPRNN. Refer to torch.nn.RNNCell documentation for the model description, parameters and inputs/outputs.

Initialize internal Module state, shared by both nn.Module and ScriptModule.

forward(input, hx=None, batch_size_t=None)[source]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Return type:

Tensor

class opacus.layers.dp_rnn.DPRNNCellBase(input_size, hidden_size, bias, num_chunks)[source]

Initialize internal Module state, shared by both nn.Module and ScriptModule.

class opacus.layers.dp_rnn.RNNLinear(in_features, out_features, bias=True)[source]

Applies a linear transformation to the incoming data: \(y = xA^T + b\)

This module is the same as a torch.nn.Linear` layer, except that in the backward pass the grad_samples get accumulated (instead of being concatenated as in the standard nn.Linear).

When used with PackedSequence`s, additional attribute `max_batch_len is defined to determine the size of per-sample grad tensor.

Initialize internal Module state, shared by both nn.Module and ScriptModule.

opacus.layers.dp_rnn.apply_permutation(tensor, dim, permutation)[source]

Permute elements of a tensor along a dimension dim. If permutation is None do nothing.