explicit and implicit layers

explicit

• Inputs $\to$ outputs
• Forward pass in a neural network classically means applying the function $f$ at each layer

$z=f(x;\theta )$

implicit

• Specify the conditions that we want the layer’s output to satisfy. For example, a layer with

$g(x,z;\theta )=0$

• The output of the layer is the solution to this equation for $z$.
  • relatively few or even just one implicit layer are often enough
  • Calculate gradients using implicit function theorem rather than backpropagating through each step of the solve (memory intensive)

examples

• Fixed point layer
  • $g(x,z)=\tanh (wz+x)$
  • Solving for $z$ is the equivalent of repeatedly applying a standard dense layer with activation to the same input.
• (neural) ODE / SDE / equilibrium models