In the mathematics of moduli theory, given an algebraic, reductive, Lie group ${\displaystyle G}$ and a finitely generated group ${\displaystyle \pi }$, the ${\displaystyle G}$-character variety of ${\displaystyle \pi }$ is a space of equivalence classes of group homomorphisms from ${\displaystyle \pi }$ to ${\displaystyle G}$:

${\displaystyle {\mathfrak {R}}(\pi ,G)=\operatorname {Hom} (\pi ,G)/\!\sim \,.}$

This article needs additional citations for verification. (January 2017)

More precisely, ${\displaystyle G}$ acts on ${\displaystyle \operatorname {Hom} (\pi ,G)}$ by conjugation, and two homomorphisms are defined to be equivalent (denoted ${\displaystyle \sim }$) if and only if their orbit closures intersect. This is the weakest equivalence relation on the set of conjugation orbits, ${\displaystyle \operatorname {Hom} (\pi ,G)/G}$, that yields a Hausdorff space.

Formally, and when the reductive group is defined over the complex numbers ${\displaystyle \mathbb {C} }$, the ${\displaystyle G}$-character variety is the spectrum of prime ideals of the ring of invariants (i.e., the affine GIT quotient).

${\displaystyle \mathbb {C} [\operatorname {Hom} (\pi ,G)]^{G}.}$

Here more generally one can consider algebraically closed fields of prime characteristic. In this generality, character varieties are only algebraic sets and are not actual varieties. To avoid technical issues, one often considers the associated reduced space by dividing by the radical of 0 (eliminating nilpotents). However, this does not necessarily yield an irreducible space either. Moreover, if we replace the complex group by a real group we may not even get an algebraic set. In particular, a maximal compact subgroup generally gives a semi-algebraic set. On the other hand, whenever ${\displaystyle \pi }$ is free we always get an honest variety; it is singular however.

An interesting class of examples arise from Riemann surfaces: if ${\displaystyle X}$ is a Riemann surface then the ${\displaystyle G}$-character variety of ${\displaystyle X}$, or Betti moduli space, is the character variety of the surface group ${\displaystyle \pi =\pi _{1}(X)}$

${\displaystyle {\mathcal {M}}_{B}(X,G)={\mathfrak {R}}(\pi _{1}(X),G)}$.

For example, if ${\displaystyle G=\mathrm {SL} (2,\mathbb {C} )}$ and ${\displaystyle X}$ is the Riemann sphere punctured three times, so ${\displaystyle \pi =\pi _{1}(X)}$ is free of rank two, then Henri G. Vogt, Robert Fricke, and Felix Klein proved^[1][2] that the character variety is ${\displaystyle \mathbb {C} ^{3}}$; its coordinate ring is isomorphic to the complex polynomial ring in 3 variables, ${\displaystyle \mathbb {C} [x,y,z]}$. Restricting to ${\displaystyle G=\mathrm {SU} (2)}$ gives a closed real three-dimensional ball (semi-algebraic, but not algebraic).

Another example, also studied by Vogt and Fricke–Klein is the case with ${\displaystyle G=\mathrm {SL} (2,\mathbb {C} )}$ and ${\displaystyle X}$ is the Riemann sphere punctured four times, so ${\displaystyle \pi =\pi _{1}(X)}$ is free of rank three. Then the character variety is isomorphic to the hypersurface in ${\displaystyle \mathbb {C} ^{7}}$ given by the equation

${\displaystyle a^{2}+b^{2}+c^{2}+d^{2}+x^{2}+y^{2}+z^{2}-(ab+cd)x-(ad+bc)y-(ac+bd)z+abcd+xyz-4=0.}$

This character variety appears in the theory of the sixth Painleve equation,^[3] and has a natural Poisson structure such that ${\displaystyle a,b,c,d}$ are Casimir functions, so the symplectic leaves are affine cubic surfaces of the form ${\displaystyle xyz+x^{2}+y^{2}+z^{2}+c_{1}x+c_{2}y+c_{3}z=c_{4}}$

This construction of the character variety is not necessarily the same as that of Marc Culler and Peter Shalen (generated by evaluations of traces), although when ${\displaystyle G=\mathrm {SL} (n,\mathbb {C} )}$ they do agree, since Claudio Procesi has shown that in this case the ring of invariants is in fact generated by only traces. Since trace functions are invariant by all inner automorphisms, the Culler–Shalen construction essentially assumes that we are acting by ${\displaystyle G=\mathrm {SL} (n,\mathbb {C} )}$ on ${\displaystyle {\mathfrak {R}}=\operatorname {Hom} (\pi ,H)}$ even if ${\displaystyle G\neq H}$.^{[clarification needed]}

For instance, when ${\displaystyle \pi }$ is a free group of rank 2 and ${\displaystyle G=\mathrm {SO} (2)}$, the conjugation action is trivial and the ${\displaystyle G}$-character variety is the torus

${\displaystyle S^{1}\times S^{1}.}$

But the trace algebra is a strictly small subalgebra (there are fewer invariants). This provides an involutive action on the torus that needs to be accounted for to yield the Culler–Shalen character variety. The involution on this torus yields a 2-sphere. The point is that up to ${\displaystyle \mathrm {SO} (2)}$-conjugation all points are distinct, but the trace identifies elements with differing anti-diagonal elements (the involution).

There is an interplay between these moduli spaces and the moduli spaces of principal bundles, vector bundles, Higgs bundles, and geometric structures on topological spaces, given generally by the observation that, at least locally, equivalent objects in these categories are parameterized by conjugacy classes of holonomy homomorphisms of flat connections. In other words, with respect to a base space ${\displaystyle M}$ for the bundles or a fixed topological space for the geometric structures, the holonomy homomorphism is a group homomorphism from ${\displaystyle \pi _{1}(M)}$ to the structure group ${\displaystyle G}$ of the bundle.^{[citation needed]}

The coordinate ring of the character variety has been related to skein modules in knot theory.^[4][5] The skein module is roughly a deformation (or quantization) of the character variety. It is closely related to topological quantum field theory in dimension 2+1.

• Geometric invariant theory