Residual loss
The residual loss plugs in the neural network approximation to the PDE and measures the residual. For example, when solving the (1D) heat equation \[ u_t = u_{xx}, \hspace{2em} (t,x)\in[0,T]\times\Omega \] with given initial condition \(u(t=0,x)=u_0(x)\). Residual loss at a collocation point \((t,x)\) is \[ L(\theta; (t,x)) = [u_t(t,x;\theta) - u_{xx}(t,x;\theta)]^2. \] The total residual loss on the domain of interest is usually designed to be \[ L_\mathrm{res}(\theta) \propto \int_{0}^{T} \int_\Omega | u_t(t,x;\theta) - u_{xx}(t,x;\theta)|^2 \mathrm{d}x \mathrm{d}t \]
The calculation of total residual loss is equivalent to perform the numerical integral.
If the PDE is changed to (inviscid) Burger's equation \[ u_t + u u_x = 0, \hspace{2em} (t,x)\in (0,\infty)\times\mathbb{R} \] with Riemann IVP: \(u(t=0, x)=1-\mathrm{Heaviside}(x)\), then residual loss at a collocation point \((t,x)\) is \[ L(\theta; (t,x)) = [u_t(t,x;\theta) + u(t,x;\theta)\cdot u_x(t,x;\theta)]^2. \]
We almost never encounter the problem of lack of training data or overfitting issues in the residual loss: we can always choose sufficiently many collocation points \((t,x)_{n}\) for \(n=1,\cdots, N\) from the domain of interest to achieve \(N\gg\dim\theta\).