Lemma 1   Let $G$ act transitively on a set $S$ . If $H$ is a normal subgroup of $G$ , then the transitivity classes of the action, restricted to $H$ , form a set of blocks for the action of $G$ .

Proof. If $T$ , $U$ are any transitivity classes for the restricted action, let $t \in T$ , $u \in U$ , and $g \in G$ such that $gt = u$ . Then $x \mapsto gx$ is a bijective map from $T$ onto $U$ (here we use normality). Hence any element of $G$ maps transitivity classes to transitivity classes.

Corollary 2   Let $G$ act primitively on a set $S$ . If $H$ is a normal subgroup of $G$ , then either $H$ acts transitively on $S$ , or $H$ lies in the kernel of the action. If the action is faithful, then either $H = \{1\}$ or $H$ is transitive.

Theorem 3   Let $G$ be a group acting primitively and faithfully on a set $S$ . Let $K$ be the stabilizer of some point $s_0 \in S$ , and assume that $K$ is simple. Then if $H$ is a nontrivial proper normal subgroup of $G$ , then $G$ is isomorphic to the semidirect product of $H$ by $K$ . $H$ can be identified with $S$ in such a way that $1 \in H$ is identified with $s_0$ , the action of $H$ becomes left multiplication, and the action of $K$ becomes conjugation.

Proof. Since $H \cap K$ is a normal subgroup of $K$ , it is either $\{1\}$ or $K$ .

If $H \cap K = K$ , then $K \subset H$ , and since $K$ is maximal and $H$ is proper, we have $K=H$ . Since $H$ is normal and $H$ stabilizes $s_0$ , then $H$ stabilizes every point (since the action is transitive). Since the action is faithful, $K = H = \{1\}$ , a contradiction. (This contradiction can also be reached by applying the corollary.)

Therefore, $H \cap K = \{1\}$ . So no element of $H$ , other than 1, fixes $s_0$ . Thus $H$ acts freely and transitively on $S$ . For any $g \in G$ , if $g s_0 = s$ and $h s = s_0$ , then $h g s_0 = s_0$ , hence $h g$ is in $K$ . Thus $G$ is generated by $H$ and $K$ . Since $H$ is normal and $H \cap K = \{1\}$ , $G$ is the (internal) semidirect product of $H$ by $K$ .

Theorem 4   Let $G$ be a group acting faithfully on a set $S$ . Let $s_0 \in S$ and let $K$ be the stabilizer of $s_0$ . Assume $K$ is simple.

1. Assume the action of $G$ is doubly transitive, and let $H$ be a nontrivial proper normal subgroup of $G$ . Then $H$ is an elementary abelian $p$ -group for some prime $p$ . Furthermore, $K$ is isomorphic to a subgroup of $GL(n,\F_p)$ , and $G$ is isomorphic to a subgroup of $AGL(n,\F_p)$ , the group of affine transformations of $H$ .

2. If the action of $G$ is triply transitive and $|S|>3$ , then any nontrivial proper normal subgroup of $G$ is an elementary abelian 2-group.

3. If the action of $G$ is quadruply transitive and $|S|>4$ , then $G$ is simple.

Proof. For part 1, use the identification of $H$ with $S$ given by the previous theorem. Since the action is doubly transitive, the action by conjugation of $K$ is transitive on $H - \{1\}$ . Therefore, all non-identity elements of $H$ have the same order, which must therefore be some prime $p$ . Hence $H$ is a $p$ -group. The center $Z(H)$ is nontrivial, and is preserved by all automorphisms. By double transitivity again, there is an automorphism taking any nontrivial element to any other; hence $H$ is abelian. Therefore $H$ is an elementary abelian $p$ -group.

For part 2, we know from part 1 that $H$ is isomorphic to an elementary abelian $p$ -group and $K$ acts as linear transformations of $H$ . Since the action of $G$ is triply transitive, the action of $K$ on the nonzero elements is doubly transitive. However, if $p>2$ , then the linearity of the action disallows double transitivity (if $x \mapsto y$ , then $2x \mapsto 2y$ so we do not have complete freedom for any two elements since $H$ contains some element besides 0, $y$ and $2y$ .)

(We note that when $|S|=3$ , we have the example $G=S_3$ , $H=A_3$ , $K=S_2$ .)

Here is an example illustrating part 2. The group $AGL(n,\F_2)$ acts triply transitively on $\F_2^n$ , and the stabilizer of a point is $GL(n,\F_2)$ , which is simple if $n>2$ . $AGL(n, \F_2)$ contains the normal subgroup of translations, an elementary abelian 2-group.

For part 3, note that the action of $GL(n,\F_2)$ on $\F_2^n$ , $n>2$ , is not triply transitive on nonzero elements, so the only conclusion left is that $G$ is simple.

Corollary 5   The Mathieu groups $M_{21}$ , $M_{22}$ , $M_{23}$ , and $M_{24}$ are simple.

Proof. We take it as known that $M_{21} \cong PSL(3, \F_4)$ is simple. Since $M_n$ has $M_{n-1}$ as point stabilizer, and has a triply transitive action on a set of $n$ elements, we may work our way inductively up to $M_{24}$ , using the previous theorem. The index of $M_{n-1}$ in $M_n$ is $n$ , which is not a power of 2. Hence in all cases, $M_n$ is simple.

Corollary 6   The alternating groups $A_n$ are simple for $n \ge 5$ .

Proof. Since the natural action of $A_n$ on $n$ letters is quadruply transitive for $n \ge 6$ , and the point stabilizer of $A_n$ is $A_{n-1}$ , we may apply the theorem to deduce the simplicity of the alternating groups $A_n$ , $n \ge 5$ , from the simplicity of $A_5 \cong PSL(2,\F_4) \cong PSL(2,\F_5)$ .