In fluid dynamics, Rayleigh's equation or Rayleigh stability equation is a linear ordinary differential equation to study the hydrodynamic stability of a parallel, incompressible and inviscid shear flow. The equation is:^[1]

${\displaystyle (U-c)(\varphi ''-k^{2}\varphi )-U''\varphi =0,}$

with ${\displaystyle U(z)}$ the flow velocity of the steady base flow whose stability is to be studied and ${\displaystyle z}$ is the cross-stream direction (i.e. perpendicular to the flow direction). Further ${\displaystyle \varphi (z)}$ is the complex valued amplitude of the infinitesimal streamfunction perturbations applied to the base flow, ${\displaystyle k}$ is the wavenumber of the perturbations and ${\displaystyle c}$ is the phase speed with which the perturbations propagate in the flow direction. The prime denotes differentiation with respect to ${\displaystyle z.}$

The equation is named after Lord Rayleigh, who introduced it in 1880.^[2] The Orr–Sommerfeld equation – introduced later, for the study of stability of parallel viscous flow – reduces to Rayleigh's equation when the viscosity is zero.^[3]

Rayleigh's equation, together with appropriate boundary conditions, most often poses an eigenvalue problem. For given (real-valued) wavenumber ${\displaystyle k}$ and mean flow velocity ${\displaystyle U(z),}$ the eigenvalues are the phase speeds ${\displaystyle c,}$ and the eigenfunctions are the associated streamfunction amplitudes ${\displaystyle \varphi (z).}$ In general, the eigenvalues form a continuous spectrum. In certain cases there may further be a discrete spectrum of complex conjugate pairs of ${\displaystyle c.}$ Since the wavenumber ${\displaystyle k}$ occurs only as a square ${\displaystyle k^{2}}$ in Rayleigh's equation, a solution (i.e. ${\displaystyle \varphi (z)}$ and ${\displaystyle c}$) for wavenumber ${\displaystyle +k}$ is also a solution for the wavenumber ${\displaystyle -k.}$^[3]

Rayleigh's equation only concerns two-dimensional perturbations to the flow. From Squire's theorem it follows that the two-dimensional perturbations are less stable than three-dimensional perturbations.

If a real-valued phase speed ${\displaystyle c}$ is in between the minimum and maximum of ${\displaystyle U(z),}$ the problem has so-called critical layers near ${\displaystyle z=z_{\mathrm {crit} }}$ where ${\displaystyle U(z_{\mathrm {crit} })=c.}$ At the critical layers Rayleigh's equation becomes singular. These were first being studied by Lord Kelvin, also in 1880.^[4] His solution gives rise to a so-called cat's eye pattern of streamlines near the critical layer, when observed in a frame of reference moving with the phase speed ${\displaystyle c.}$^[3]

Consider a parallel shear flow ${\displaystyle U(z)}$ in the ${\displaystyle x}$ direction, which varies only in the cross-flow direction ${\displaystyle z.}$^[1] The stability of the flow is studied by adding small perturbations to the flow velocity ${\displaystyle u(x,z,t)}$ and ${\displaystyle w(x,z,t)}$ in the ${\displaystyle x}$ and ${\displaystyle z}$ directions, respectively. The flow is described using the incompressible Euler equations, which become after linearization – using velocity components ${\displaystyle U(z)+u(x,z,t)}$ and ${\displaystyle w(x,z,t):}$

${\displaystyle {\begin{aligned}&\partial _{t}u+U\,\partial _{x}u+w\,U'=-{\frac {1}{\rho }}\partial _{x}p,\\&\partial _{t}w+U\,\partial _{x}w=-{\frac {1}{\rho }}\partial _{z}p\qquad {\text{and}}\\&\partial _{x}u+\partial _{z}w=0,\end{aligned}}}$

with ${\displaystyle \partial _{t}}$ the partial derivative operator with respect to time, and similarly ${\displaystyle \partial _{x}}$ and ${\displaystyle \partial _{z}}$ with respect to ${\displaystyle x}$ and ${\displaystyle z.}$ The pressure fluctuations ${\displaystyle p(x,z,t)}$ ensure that the continuity equation ${\displaystyle \partial _{x}u+\partial _{z}w=0}$ is fulfilled. The fluid density is denoted as ${\displaystyle \rho }$ and is a constant in the present analysis. The prime ${\displaystyle U'}$ denotes differentiation of ${\displaystyle U(z)}$ with respect to its argument ${\displaystyle z.}$

The flow oscillations ${\displaystyle u}$ and ${\displaystyle w}$ are described using a streamfunction ${\displaystyle \psi (x,z,t),}$ ensuring that the continuity equation is satisfied:

${\displaystyle u=+\partial _{z}\psi \quad {\text{ and }}\quad w=-\partial _{x}\psi .}$

Taking the ${\displaystyle z}$- and ${\displaystyle x}$-derivatives of the ${\displaystyle x}$- and ${\displaystyle z}$-momentum equation, and thereafter subtracting the two equations, the pressure ${\displaystyle p}$ can be eliminated:

${\displaystyle \partial _{t}\left(\partial _{x}^{2}\psi +\partial _{z}^{2}\psi \right)+U\,\partial _{x}\left(\partial _{x}^{2}\psi +\partial _{z}^{2}\psi \right)-U''\,\partial _{x}\psi =0,}$

which is essentially the vorticity transport equation, ${\displaystyle \partial _{x}^{2}\psi +\partial _{z}^{2}\psi }$ being (minus) the vorticity.

Next, sinusoidal fluctuations are considered:

${\displaystyle \psi (x,z,t)=\Re \left\{\varphi (z)\,\exp(ik(x-ct))\right\},}$

with ${\displaystyle \varphi (z)}$ the complex-valued amplitude of the streamfunction oscillations, while ${\displaystyle i}$ is the imaginary unit (${\displaystyle i^{2}=-1}$) and ${\displaystyle \Re \left\{\cdot \right\}}$ denotes the real part of the expression between the brackets. Using this in the vorticity transport equation, Rayleigh's equation is obtained.

The boundary conditions for flat impermeable walls follow from the fact that the streamfunction is a constant at them. So at impermeable walls the streamfunction oscillations are zero, i.e. ${\displaystyle \varphi =0.}$ For unbounded flows the common boundary conditions are that ${\displaystyle \lim _{z\to \pm \infty }\varphi (z)=0.}$