Let $\mathbb{K}$ be a field and $V$ a vector space over $\mathbb{K}$ Let $G$ be a group with a specified representation on $V$ denoted by $g . v$ for $v \in V$ and $g \in G$

An invariant scalar product (with respect to the action of $G$ on $V$ is a scalar product $\left( \cdot \lvert \cdot \right)$ on $V$ (i.e. a non-degenerate, symmetric $\mathbb{K}$ bilinear form) such that for any $g \in G, u,v \in V$ we have

$$ \left( g . u \lvert g . v \right) = \left( u \lvert v \right) $$

Now let $\mathfrak{g}$ be a Lie algebra over $\mathbb{K}$ with a representation on $V$ denoted by $X . v$ for $X \in \mathfrak{g}, v \in V$ Then an invariant scalar product (with respect to the action of $\mathfrak{g}$ is a scalar product on $V$ such that for any $X \in \mathfrak{g}, u,v \in V$ we have

$$ \left( X . u \lvert v \right) = - \left( u \lvert X . v \right) $$

An invariant scalar product on a Lie algebra $\mathfrak{g}$ is by definition an invariant scalar product as above where the representation is the adjoint representation of $\mathfrak{g}$ on itself. In this case invariance is usualy written $\left( [X, Y] \mid Z \right) = \left( X \mid [Y, Z] \right)$

Examples

For example if $G = O(n)$ the orthogonal subgroup of $n \times n$ real matricies and $\mathbb{R}^n$ is the natural representation for $O(n)$ then the standard Euclidean scalar product on $\mathbb{R}^n$ is an invariant scalar product. Invariance in this example follows from the definition of $O(n)$

As another example if $\mathfrak{g}$ is a complex semi-simple Lie algebra then the Killing form $\kappa(X,Y) := Tr(ad_X \cdot ad_Y)$ is an invariant scalar product on $\mathfrak{g}$ itself via the adjoint representation. Invariance in this example follows from the fact that the trace operator is associative, i.e. $Tr([Y,X] \cdot Z) = - Tr([X,Y] \cdot Z) = - Tr(X \cdot [Y,Z])$ Thus an invariant scalar product (with respect to a Lie algebra representation) is sometimes called an associative scalar product.