THE FUNDAMENTAL LEMMA FOR

THE SHALIKA SUBGROUP OF GL(4)

Solomon Friedberg and Hervé Jacquet

Abstract. The authors establish the fundamental lemma for a relative trace formula. This trace formula compares generic automorphic representations of GSp(4) with automorphic representations of GL(4) which are distinguished with respect to a character of the Shalika subgroup, the subgroup of matrices of 2 × 2 block form

( \begin{matrix}g & 0 \\ 0 & g \end{matrix} ) ( \begin{matrix}I & 0 \\ X & O \end{matrix} ) .

The fundamental lemma, giving the equality of two orbital integrals, amounts to a comparison of certain exponential sums arising from these two different groups. On the GSp(4) side one has the Kloosterman sums for this group, and on the GL(4) side certain, new, relative Kloosterman sums. To show that these are equal for each relevant Weyl group element, the authors compute the Mellin transforms of the sums and match them in all cases.

The file available from the MSRI preprint server gives the first chapter of the manuscript. The entire manuscript, 147 pages long, is available as an MSRI preprint.

Typeset by AMS-TEX