THE FUNDAMENTAL LEMMA FOR

THE SHALIKA SUBGROUP OF GL(4)

by Solomon Friedberg and Hervé Jacquet

University of California, Santa Cruz Columbia University

Department of Mathematics, University of California Santa Cruz, Santa Cruz, CA 95064

E-mail address: friedbe@cats.ucsc.edu

Department of Mathematics, Columbia University, New York, NY 10027 E-mail address: hj@math.columbia.edu

Typeset by AMS-TEX

ABSTRACT

The authors establish the fundamental lemma for a relative trace formula. This trace formula compares generic automorphic representations of GSp(4) with automorphic representations of GL(4) which are distinguished with respect to a character of the Shalika subgroup, the subgroup of matrices of 2 × 2 block form ( \begin{matrix}g & 0 \\ X & g \end{matrix} ). The fundamental lemma, giving the equality of two orbital integrals, amounts to a comparison of certain exponential sums arising from these two different groups. On the GSp(4) side one has the Kloosterman sums for this group, and on the GL(4) side certain, new, relative Kloosterman sums. To show that these are equal for each relevant Weyl group element, the authors compute the Mellin transforms of the sums and match them in all cases.

1991 Mathematics Subject Classification. Primary 11F70; Secondary 11F46, 11F72, 11L05, 22E50.

Key words and phrases. relative trace formula, fundamental lemma, Kloosterman integral, relevant double coset, Shalika subgroup.

Research supported by National Science Foundation grants DMS-8821762 and DMS-9123845 (Friedberg), and DMS-8801759 and DMS-9101637 (Jacquet). Friedberg’s research also supported by the Sloan Foundation. Research at MSRI is supported in part by National Science Foundation grant DMS-9022140.

CONTENTS

Chapter I. Introduction and Statement of Results 1 1. Introduction 1 2. The Shalika Subgroup H of GL(4): Relevant Double Cosets

and Kloosterman Integrals 3 3. The Symplectic Group GSp(4): Relevant Double Cosets and

Kloosterman Integrals 7 4. The Main Theorem 9 5. Kloosterman Integrals and Gauss Sums 10

Chapter II. Evaluation of the Integral for the Main H-

Relevant Double Cosets 11 1. Plan of the Computation 11 2. The Unipotent Integral 12 3. The Toroidal Integral 25 4. Two Supplementary Integrals 36 5. Evaluation of the Mellin Transform I: The Ramified Case 38 6. Evaluation of the Mellin Transform II: The Case χ_{1} Ramified,

χ_{2} Unramified 40 7. Evaluation of the Mellin Transform III: The Case χ_{1} Unrami-

fied, χ_{2} Ramified 44 8. Evaluation of the Mellin Transform IV: The Unramified Case 55

Chapter III. Evaluation of the Integrals for the Other

H-Relevant Double Cosets 62 1. Evaluation of the Integral M_{F} (χ; 1), I: Unipotent Integration 62 2. Evaluation of the Integral M_{F} (χ; 1), II: Toroidal Integration

and Simplification 69 3. Evaluation of the Integral M_{F} (χ; (132)), I: Unipotent Integra-

tion 73 4. Evaluation of the Integral M_{F} (χ; (132)), II: Toroidal Integra-

tion and Simplification 79 Chapter IV. Evaluation of the GSp(4) Integral for the

Main Double Cosets 81 1. Computation of the Integral J(t) 81 2. Simplification of the Integrals Giving J(t) 87 3. Computation of the Mellin Transform 93 4. Evaluation of the Mellin Transform I: The Ramified Case 102 5. Evaluation of the Mellin Transform II: The Case χ_{1} Ramified,

χ_{2} Unramified 104 6. Evaluation of the Mellin Transform III: The Case χ_{1} Unrami-

fied, χ_{2} Ramified 112 7. Evaluation of the Mellin Transform IV: The Unramified Case 119

Chapter V. Evaluation of the GSp(4) Integrals for the

Other Relevant Double Cosets 131 1. Computation of the Integral J(w_{1}, t) 131 2. Evaluation of the Mellin Transform M_{J}(χ; w_{1}) 134 3. Computation of the Integral J(w_{2}, t) 137 4. Evaluation of the Mellin Transform M_{J}(χ; w_{2}) 140 References 143 CHAPTER I INTRODUCTION AND STATEMENT OF RESULTS

1. Introduction

(1.1) Let K be a number field and let G be the group GL(4) regarded as an algebraic group over K. Suppose that π is a cuspidal automorphic representation of G. Thus, conjecturally, π is associated to a representation τ of the hypothetical Langlands group L_{K} into ^{L}G^{0} = GL(4, C). Suppose furthermore that π is selfcontragredient. Then either the symmetric square L-function or the exterior square L-function of π has a pole at s = 1. Assume that the exterior square L-function has a pole at s = 1. Conjecturally, this means that ∧^{2}τ has a fixed vector, and hence that τ factors through the symplectic group in 4 variables Sp(4, C), which is the Lgroup of SO(5, K). Thus it is predicted that π is the functorial image of a cuspidal automorphic representation π′ (or rather a packet of representations) of the group SO(5). On the other hand, the condition that the exterior square L-function has a pole at s = 1 can be read directly on the space of the representation π, due to the integral representation for the exterior square L-function given by Jacquet and Shalika [JS]. More precisely, let H be the Shalika subgroup of GL(4)

H = \{( \begin{matrix}g \\ & g \end{matrix} ) \binom{I_{2}}{X I_{2}} | g ∈ GL(2), X ∈ M_{2×2}\} ,

again regarded as an algebraic group over K. If ψ is an additive character of the adeles 𝔸 of K trivial on K define a character still denoted ψ of H(𝔸) by

(1.1)  ( \begin{matrix}g \\ & g \end{matrix} ) \binom{I_{2}}{X I_{2}} ↦ ψ(trX).

Then the exterior square L-function has a pole at s = 1 if and only if π is distinguished with respect to ψ, in the sense that there is a vector ϕ in the representation π such that

∫ ϕ(h) ψ(h) dh ≠ 0.

More generally, we can consider automorphic representations which are essentially self-contragredient, that is, self-contragredient up to twist by a character. This amounts to replacing the exterior square L-function in the above discussion by a suitable twist. A pole at s = 1 conjecturally implies that the representation τ factors through the group of symplectic similitudes GSp(4, C), which is the L-group of GSp(4, K). This means that the representation π should be distinguished with respect to a character of the form h ↦ χ(det g) ψ(trX).

(1.2) The above considerations suggest the existence of a trace formula of the following type. For simplicity, we restrict ourselves to the study of self-contragredient representations with trivial central character, and we consider only the character ψ of the Shalika subgroup. Let Φ be a smooth function of compact support on G(𝔸). Define as usual a kernel function

K_{Φ}(g_{1}, g_{2}) = ∑_{ξ∈G(K)} Φ(g_{1}^{−1}ξg_{2}).

Similarly, let G′ be the group GSp(4) regarded as an algebraic group over K, let Φ′ be a smooth function of compact support on G′(𝔸), and define the kernel function

K_{Φ′}(g′_{1}, g′_{2}) = ∑_{ξ′∈G′(K)} Φ′(g′_{1}^{−1}ξ′g′_{2}).

The trace formula in question takes the form:

(1.2) ∫ K_{Φ}(h^{−1}, n) ψ(h) θ(n) dh dn = ∫ K_{Φ′}(^{t}n_{1}, n_{2}z) θ′(n_{1}^{−1}n_{2}) dn_{1} dn_{2} dz.

Here h is integrated over the Shalika subgroup, n, n_{1}, n_{2} are integrated over the unipotent radicals of the respective Borel subgroups, θ, θ′ are nondegenerate characters, and z is integrated over the center of GSp(4). Both sides can be computed in terms of orbital integrals, that is, integrals taken over double cosets of the appropriate groups. Not all double cosets contribute. There is a bijection between the relevant double cosets of the two groups, and the above identity holds if the corresponding orbitals integrals are equal. This is described in more detail below. Assuming that there are enough pairs Φ, Φ′ satisfying this condition of matching orbital integrals, spectral expansions of the kernels should allow one to derive from this identity another identity asserting that

∫ K^{cusp}(h^{−1}, n) ψ(h) θ(n) dh dn = ∫ K^{cusp}(^{t}n_{1}, n_{2}z) θ′(n_{1}^{−1}n_{2}) dn_{1} dn_{2} dz.

Here the exponent cusp is used to indicate that only the cuspidal part of the kernel is considered. (In fact, one expects that some discrete terms will have to be added to the cuspidal kernel.)

What are the applications of such an identity? Via the twisted trace formula, one expects to be able to establish that (essentially) self-contragredient automorphic representations of GL(4) whose (possibly twisted) exterior square L-function has a pole are functorial images from the group GSp(4). The present trace formula would then establish that in any global packet of representations of GSp(4) there is at least one element with a global Whittaker model.

We caution that the details of this program are far from completed. In particular, we have not discussed the relation between representations of SO(5) and GL(4). In the case at hand, the Weil representation might be a quicker way to arrive at the same result. However, for higher ranks, the Weil representation is not available and one has to resort to a trace formula of the above type. In that case the group GSp(4) is to be replaced by a spin group whose L-group is the GSp(2n, C).

2. THE SHALIKA SUBGROUP H 3

(1.3) If Φ and Φ′ are each products of local functions Φ_{v}, Φ′_{v}, the matching of orbital integrals for (1.2) reduces to a local question. At almost all places the functions Φ_{v}, Φ′_{v} are the characteristic functions of the respective maximal compact subgroups. In this paper, we match the orbital integrals of these characteristic functions, that is, we establish the “fundamental lemma” for the trace formula at hand. The orbital integral on GSp(4) is of Kloosterman type, while the orbital integral on GL(4) is a new sort of relative Kloosterman integral. This matching is the first crucial step in establishing the formula (1.2). However, our method of proof is rather computational, and thus leaves something to be desired. Essentially, we have to show that functions of two (or one) variables are equal. We do so by showing that their formal Mellin transforms are equal, and this is a series of lengthy calculations. It is remarkable that the resulting formulae are relatively simple. Thus, it is possible that the Mellin transforms of our functions have some independent interest.

This use of formal Mellin transforms was introduced, in the context of quadratic base change on GL(2), by Jacquet and Ye [JY1]. A number of additional papers related to the relative trace formula are listed in the References.

The main theorem of this work is given below in Section 4 of this Chapter. This theorem was announced in [FJ].

(1.4) In the rest of this paper we pass to the local situation. Throughout we fix the following notation. Let K now denote a nonarchimedean local field of residual characteristic not equal to 2, O be its ring of integers, and ϖ be a local uniformizer of K. Let q denote the cardinality of the residue field O/ϖO, and | | denote the normalized absolute value on K, so that |ϖ| = q^{−1}. Fix an additive character ψ of K with the property that the largest fractional ideal on which ψ is trivial is precisely O.

(1.5) Equations are numbered consecutively throughout each section. The fifth numbered equation of Chapter 1, Section 2 is referred to as (2.5) throughout Chapter 1, and as (1.2.5) in the other chapters. The numbering of Lemmas, Propositions, and Theorems (together) is similar. Sections are also divided into numbered paragraphs, for the convenience of the reader.

2. The Shalika Subgroup H of GL(4): Relevant Double Cosets and Kloosterman Integrals

(2.1) For convenience, let us write H instead of H(K) for the Shalika subgroup of GL(4, K):

H = \{( \begin{matrix}g \\ & g \end{matrix} ) \binom{I_{2}}{X I_{2}} | g ∈ GL(2, K), X ∈ M_{2×2}(K)\} .

If B is a Borel subgroup of GL(2, K), let H(B) be the subgroup of the Shalika subgroup defined by

H(B) = \{( \begin{matrix}b \\ & b \end{matrix} ) \binom{I_{2}}{X I_{2}} | b ∈ B, X ∈ M_{2×2}(K)\} .

When working with the Shalika subgroup (this Section and Chapters 2 and 3), we let N denote the subgroup of GL(4, K) consisting of upper triangular unipotent matrices.

The additive character ψ gives rise to a character of the group H, which we shall again denote by ψ, by the formula (1.1). In addition, let θ denote the character of N given by

θ(n) = ψ(n_{1,2} + n_{2,3} + n_{3,4}).

(2.2) A double coset HrN of GL(4, K) is said to be H-relevant if the map

hrn ↦ ψ(h)θ(n)

is well defined. This is the case if and only if

(2.1)  θ(n) = ψ(rnr^{−1})  for all n ∈ r^{−1}Hr ∩ N.

It is precisely these double cosets which enter into the relative trace formula.

To make these explicit, let W denote the Weyl group of GL(4) with respect to the standard Borel subgroup B_{4}. Then W is isomorphic to the permutation group S_{4}. To make this explicit, associate with each permutation σ ∈ S_{4} the permutation matrix with entries 1 in the (i, σ(i)) positions and 0 elsewhere. Note that with this convention, permutations are multiplied from the right, so (12)(13)=(123) rather than (132), and if w is the permutation matrix corresponding to σ, then w^{−1}m_{i,j}w = m_{σ(i),σ(j)}. Let diag( ) denote the 4 × 4 diagonal matrix. Then we have

Proposition 2.1. The H-relevant double cosets of GL(4, K) are

(1) H (1243) N

(2) H diag(−a_{1}, a_{1}, 1, 1) N

(3) H (132) diag(a_{1}, 1, 1, 1) N

(4) H (12) diag(a_{1}a_{2}, a_{1}, 1, 1) N with a_{1}, a_{2} ∈ K^{×}.

Proof. We first determine a complete set of coset representatives for the double cosets H\GL(4, K)/B_{4}. Let P_{2,2} be the standard parabolic subgroup of type (2,2), and let W_{2,2} be the subgroup of W corresponding to P_{2,2}. Identify W with the group of 4 × 4 permutation matrices as above. By the Bruhat decomposition, we have the disjoint union

GL(4, K) = ⋃_{w∈W_{2,2}\W} P_{2,2} w B_{4}.

Now a set of coset representatives for H\P_{2,2} is given by matrices of the form

( \begin{matrix}g \\ & I_{2} \end{matrix} ) with g ∈ GL(2, K). Such a g may be written g = b_{1}w_{1}b_{2} with w_{1} a

2 × 2 permutation matrix and b_{1} and b_{2} either upper or lower triangular (with independent choices). Changing on the left by an element of the Shalika subgroup,

H ( \begin{matrix}g \\ & I_{2} \end{matrix} ) w B_{4} = H ( \begin{matrix}w_{1}b_{2} \\ & b_{1}^{−1} \end{matrix} ) w B_{4}.

But for correct choices of upper/lower triangularity of the b_{i}, they may each be pulled past the w and absorbed in B_{4}. It follows that for any w ∈ W,

P_{2,2}wB_{4} = HwB_{4} ∪ H(12)wB_{4}.

This gives the decomposition

(2.2)  GL(4, K) = ⋃_{<(34)>\W} HwB_{4}.  (disjoint union)

Next let us determine which of the twelve double cosets HwB_{4} in (2.2) may contain a relevant double coset HwaN, for some diagonal a. Suppose that w corresponds to a permutation σ. Consideration of the characters ψ and θ shows that (2.1) holds only when the following conditions are satisfied:

(1) If σ(3) < σ(1), then σ(1) = σ(3) + 1.

(2) If σ(4) < σ(2), then σ(2) = σ(4) + 1.

(3) If σ(3) < σ(2), then σ(2) ≠ σ(3) + 1.

(4) If σ(4) < σ(1), then σ(1) ≠ σ(4) + 1.

(5) If σ(1) < σ(2) and σ(3) < σ(4), then either σ(1) + 1 < σ(2) and σ(3) + 1 <

σ(4), or σ(1) + 1 = σ(2) and σ(3) + 1 = σ(4). Using these conditions, one finds that if a double coset HwaN is relevant, then (modulo left multiplication by (34)), σ must be one of the permutations 1, (12), (132), and (1243).

To complete the proof of the Proposition, one checks that for each of these four permutations, the double cosets of the form HwaN satisfying (2.1) are precisely those of the form listed. We omit this calculation.

(2.3) For each relevant double coset, we now introduce an integral of the unit element Ξ of the GL(4) Hecke algebra. The integral is to be compared with an appropriate symplectic integral. To facilitate this comparison, it is useful to consider a Mellin transform of the integral against a quasicharacter of the appropriate torus.

The integral is described as follows. Let Ξ be the characteristic function of the maximal compact subgroup K = GL(4, O) of GL(4, K). Suppose that the double coset HwaN is relevant. Then we consider

(2.3)  F (w, a) = ∫_{N} ∫_{H/H∩wNw^{−1}} Ξ(hwan) ψ(h) θ(n) dh dn,

where dh is the quotient measure obtained from the normalized right Haar measure on H.

A first simplification of (2.3) is given as follows. Let B denote a Borel subgroup of GL(2, K), and let K_{1} denote the compact subgroup GL(2, O). Then with suitable normalizations of the Haar measures

∫_{GL(2,K)} dg = ∫_{K_{1}} dk_{1} ∫_{B} db.

Moving the matrix ( \begin{matrix}k_{1} & 0 \\ 0 & k_{1} \end{matrix} ) ∈ K to the left and using the K-invariance of Ξ, we

may do the K_{1} integral. We find that the integral (2.3) is unchanged if we replace H by the subgroup H(B).

We shall compute the integral F ((1243), I_{4}) directly momentarily. To study the remaining integrals, let χ_{1}, χ_{2} be quasicharacters of K. Let us introduce the following formal Mellin transforms of F (w, a):

(2.4) M_{F} (χ_{1}; 1) = ∫_{K^{×}} F (1, diag(−a_{1}, a_{1}, 1, 1)) χ_{1}(a_{1}) d^{×}a_{1}

M_{F} (χ_{1}; (132)) = ∫_{K^{×}} F ((132), diag(a_{1}, 1, 1, 1)) χ_{1}(a_{1}) d^{×}a_{1} (2.5)

M_{F} (χ_{1}, χ_{2}; (12)) = ∫_{(K^{×})^{2}} F ((12), diag(a_{1}a_{2}, a_{1}, 1, 1)) χ_{1}(a_{1}) χ_{2}(a_{2}) d^{×}a_{1} d^{×}a_{2}. (2.6)

The meaning of these formal integrals is the following. Let ϕ_{A} be the characteristic function of the compact set q^{−A} ≤ |x| ≤ q^{A}, and introduce the integrals

(2.7)  M_{F} (χ_{1}; 1; A) = ∫_{K^{×}} F (1, diag(−a_{1}, a_{1}, 1, 1)) χ_{1}(a_{1}) ϕ_{A}(a_{1}) d^{×}a_{1},

(2.8) M_{F} (χ_{1}; (132); A) = ∫_{K^{×}} F ((132), diag(a_{1}, 1, 1, 1)) χ_{1}(a_{1}) ϕ_{A}(a_{1}) d^{×}a_{1},
M_{F} (χ_{1}, χ_{2}; (12); A) = ∫_{(K^{×})^{2}} F ((12), diag(a_{1}a_{2}, a_{1}, 1, 1)) χ_{1}(a_{1}) χ_{2}(a_{2})

× ϕ_{A}(a_{1}) ϕ_{A}(a_{2}) d^{×}a_{1} d^{×}a_{2}. (2.9)

The integrands then have compact support. If the quasicharacters χ_{1}, χ_{2} are ramified, then these integrals have limits as A tends to infinity. The formal Mellin transform is by definition this limit. In (2.7) and (2.8), when the quasicharacter χ_{1} is unramified, these integrals represent Laurent polynomials in χ_{1}(ϖ). As A tends to infinity, these Laurent polynomials approach a Laurent series; this is by definition the formal Mellin transform. Here the topology on the space of Laurent series is that of pointwise convergence of the coefficients. The treatment of (2.9) is similar; the formal Mellin transform is a Laurent series in χ_{1}(ϖ) and χ_{2}(ϖ) when both quasicharacters are unramified, and is a Laurent series in one variable in the mixed cases.

For a fixed relevant w, we occasionally suppress the dependence of M_{F} on w.

(2.4) The quantity M_{F} (χ_{1}, χ_{2}; (12)) is evaluated in Chapter 2 of these notes. In Chapter 3 we evaluate the remaining two Mellin transforms. First, though, let us dispose of F ((1243), I_{4}).

3. THE SYMPLECTIC GROUP GSp(4) 7

Proposition 2.2. F ((1243), I_{4}) = 1.

Proof. A direct computation shows that with w = (1243)

H ∩ wNw^{−1} = \{( \begin{matrix}1 & x \\ & 1 \\ & & 1 & x \\ & & & 1 \end{matrix} ) ( \begin{matrix}1 \\ & 1 \\ r & s & 1 \\ & u & & 1 \end{matrix} )\} .

Replacing H by H(B) where B is the standard Borel subgroup of GL(2) consisting of upper triangular matrices, we thus arrive at the expression

F ((1243), I_{4}) = ∫ Ξ (( \begin{matrix}1 \\ & 1 \\ & & 1 \\ γ & & & 1 \end{matrix} ) ( \begin{matrix}t_{3} \\ & t_{4} \\ & & t_{3} \\ & & & t_{4} \end{matrix} ) (1243)n)

× θ(n) |\frac{t_{3}}{t_{4}} |^{2} dγ d^{×}t_{3} d^{×}t_{4} dn

= ∫ Ξ (( \begin{matrix}t_{3} \\ & t_{3} \\ & γt_{3} & t_{4} \\ & & & t_{4} \end{matrix} ) n) θ(n) |\frac{t_{3}}{t_{4}} |^{2} dγ d^{×}t_{3} d^{×}t_{4} dn,

where for the last step we have moved (1243) to the left and used the K-invariance of Ξ. But a matrix calculation (or the use of Plücker coordinates, as in Chapter 2 below) shows that the matrix

( \begin{matrix}t_{3} \\ & t_{3} \\ & γt_{3} & t_{4} \\ & & & t_{4} \end{matrix} )

is in KN if and only if it is in K. Thus the integrand is nonzero only when n ∈ K, and it is easy to see that the integral reduces to 1.

3. The Symplectic Group GSp(4): Relevant Double Cosets and Kloosterman Integrals

(3.1) We consider the similitude symplectic group G′(K) = GSp(4, K) for the skew symmetric matrix

(3.1)  w_{0} = ( \begin{matrix}0 & w_{s} \\ −w_{s} & 0 \end{matrix} )

where we have set:

w_{s} = ( \begin{matrix}0 & 1 \\ 1 & 0 \end{matrix} ) .

We denote by T the group of diagonal matrices in G′(K). It consists of all matrices of the form

(3.2)  t = diag(a_{1}, a_{2}, a_{2}^{−1}λ, a_{1}^{−1}λ) .

The scalar λ is the similitude ratio. If λ = 1 the matrix t is in the symplectic group and the positive roots take the following values on t:

(3.3)  α_{1} = a_{1}a_{2}^{−1} , α_{2} = a_{2}^{2} , α_{1}α_{2} = a_{1}a_{2} , α_{1}^{2}α_{2} = a_{1}^{2} .

We let Z be the center of G′(K). When working with the group GSp(4, K) (this Section and Chapters 4 and 5), we let N denote the group of upper triangular symplectic matrices with unit diagonal. Thus N is the semi-direct product of the group V of matrices of the form:

(3.4)  v = ( \begin{matrix}1 & x & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & −x \\ 0 & 0 & 0 & 1 \end{matrix} ) ,

and the group U of matrices of the form:

(3.5)  u = ( \begin{matrix}1 & 0 & p & r \\ 0 & 1 & s & p \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{matrix} ) .

We define a generic character θ of N by

(3.6)  θ(u) = ψ(s) , θ(v) = ψ(x) .

Also, let \overline{N} be the opposite (transposed) subgroup to N.

(3.2) A double coset \overline{N}rN is said to be relevant if the map

tn_{1}rn_{2} ↦ θ(n_{1}n_{2})

is well defined. This is the case if and only if

(3.7)  θ(n) = θ(^{t}(rnr^{−1}))  for all n ∈ r^{−1}\overline{N}r ∩ N.

It is precisely these cosets which enter into the relative trace formula. Relevant double cosets are easily described: as in any split group, they are associated with the standard parabolic subgroups P = MU of G′. Let w_{M} be the longest element in W ∩ M and t an element of the center of M. Then the double coset of w_{M}t is relevant and all relevant double cosets have a unique representative of this form for a suitable M.

More precisely, let w_{1} and w_{2} be the following matrices:

(3.8)  w_{1} = ( \begin{matrix}1 & 0 & 0 & 0 \\ 0 & 0 & −1 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 1 \end{matrix} ) , w_{2} = ( \begin{matrix}0 & 1 & 0 & 0 \\ 1 & 0 & 0 & 0 \\ 0 & 0 & 0 & 1 \\ 0 & 0 & 1 & 0 \end{matrix} ) .

Then the a set of representatives for the relevant double cosets consists of the matrices of the form

(1) t

(2) w_{1} diag(a_{1}, a_{2}, −a_{2}, −a_{1}^{−1}a_{2}^{2})

(3) w_{2} diag(a_{1}, a_{1}, a_{1}^{−1}λ, a_{1}^{−1}λ)

(4) w_{0} diag(a_{1}, a_{1}, a_{1}, a_{1}).

(3.3) Let Ξ′ be the characteristic function of GL(4, O) ∩ GSp(4, F). For each relevant double coset \overline{N}wtN we consider the integrals

(3.9)  J(w, t) = ∫_{N∩w^{−1}\overline{N}w\N} ∫_{N} Ξ′(^{t}n_{2}wtn_{1})θ(n_{1}n_{2}) dn_{2} dn_{1},

(3.10)  J′(w, t) = ∫_{Z} ∫_{N∩w^{−1}\overline{N}w\N} ∫_{N} Ξ′(^{t}n_{2}wztn_{1})θ(n_{1}n_{2}) dn_{2} dn_{1}dz.

Our main result compares these integrals to the integrals F (w, a) introduced in paragraph (2.3).

(3.4) First, for completeness, we record the value of J(w_{0}, I_{4}). The proof is similar to that given in paragraph (2.4).

Proposition 3.1.

J(w_{0}, I_{4}) = 1.

To study the remaining integrals, let χ_{1}, χ_{2} be quasicharacters of K^{×}. Let us introduce the following formal Mellin transforms of the functions J′(w, t):

M_{J}(χ_{1}; w_{1}) = ∫_{K^{×}} J′ (w_{1}, diag(a, 1, −1, −a^{−1})) χ_{1}(a) d^{×}a, (3.11)

M_{J}(χ_{1}; w_{2}) = ∫_{K^{×}} J′ (w_{2}, diag(a, a, 1, 1)) χ_{1}(a) d^{×}a, (3.12)

M_{J}(χ_{1}, χ_{2}; 1) = ∫_{(K^{×})^{2}} J′ (1, diag(a_{1}a_{2}, a_{2}, 1, a_{1}^{−1})) χ_{1}(a_{1}) χ_{2}(a_{2}) d^{×}a_{1} d^{×}a_{2}. (3.13)

The quantity M_{J}(χ_{1}, χ_{2}; 1) is evaluated in Chapter 4 and the other quantities in Chapter 5.

4. The Main Theorem

(4.1) To state the main theorem, we let σa ↔ wt be the bijection between relevant double coset representatives on the two groups described as follows. The double cosets wt for GSp(4) are taken modulo the center. Then the bijection is given by

(12) diag(a_{1}a_{2}, a_{1}, 1, 1) ↔ diag(−a_{1}a_{2}, −a_{2}, 1, a_{1}^{−1})
diag(−a_{1}, a_{1}, 1, 1) ↔ w_{1} diag(a_{1}, 1, −1, −a_{1}^{−1})
(132) diag(a_{2}, 1, 1, 1) ↔ w_{2} diag(a_{2}, a_{2}, 1, 1)
(1243) ↔ w_{0}.

The main theorem states that the orbital integrals introduced above for the two different groups are in fact equal.

Main Theorem. For the bijection of relevant cosets σa ↔ wt, the relative orbital integral on GL(4) with respect to the Shalika subgroup is equal to the Kloosterman orbital integral on GSp(4):

F (σ, a) = J′(w, t).

To establish the Main Theorem, we shall compare these two functions of the form f(a_{1}, a_{2}) (resp. f(a_{1}), f(a_{2})), by comparing their formal Mellin transforms in the sense described above. As we shall show in this work, these formal Mellin transforms are in fact equal. For the main relevant orbits, these Mellin transforms are computed in Chapters II (M_{F}) and IV (M_{J}), and the comparison is also included in Chapter IV. For the other two nontrivial relevant orbits, the Mellin transforms are computed in Chapters III (M_{F}) and V (M_{J}), and the comparisons are included in Chapter V.

5. Kloosterman Integrals and Gauss Sums

(5.1) We end this Chapter with a brief review of Kloosterman integrals and of Gauss sums. For r, s ∈ K^{×}, the (GL(2)) Kloosterman integral Kl(r, s) is defined by the formula

Kl(r, s) = ∫_{|rx|=1} ψ (x − \frac{s}{rx}) dx.

Here dx denotes the Haar measure on K, normalized so that O has measure 1. The following proposition allows one to evaluate this integral explicitly when |rs| ≠ 1.

Proposition 5.1. The Kloosterman integral satisfies the following properties:

(1) Kl(r, s) = |rs|^{−1}Kl(s^{−1}, r^{−1}).

(2) Suppose |rs| < 1.

(i) If |r| ≥ 1, then Kl(r, s) = (1 − q^{−1})|r|^{−1}. (ii) If |r| < 1, |s| > 1, then Kl(r, s) = 0. (iii) If |r| < |ϖ|, |s| ≤ 1, then Kl(r, s) = 0. (iv) If |r| = |ϖ|, |s| ≤ 1, then Kl(r, s) = −1.

(5.2) If χ is a ramified quasicharacter of K and f is a positive integer, we shall say that χ has conductor f if χ is trivial on the subgroup 1 + ϖ^{f}O of O^{×}, but not on the subgroup (1 + ϖ^{f−1}O) ∩ O^{×}. If χ is a ramified quasicharacter of conductor f, let G(χ) denote the Gauss sum

G(χ) = ∫_{|s|=q^{f}} χ(s)ψ(s) ds.

Then one has

(5.1)  ∫_{|s|=q^{h}} χ(s)ψ(s) ds = δ_{h,f}G(χ),

where δ_{h,f} denotes the Kronecker delta, for if h ≠ f then the oscillations give zero.

REFERENCES

[F] S. Friedberg, Poincaré series for GL(n): Fourier expansion, Kloosterman sums, and algebreo-geometric estimates, Math. Zeitschrift 196 (1987), 165-188.

[FJ] S. Friedberg and H. Jacquet, Le lemme fondamental pour le sous-groupe de Shalika de GL(4), Comptes Rendus de l’Académie des Sciences, Paris 316 (1993), 407-410.

[G] D. Goldfeld, Kloosterman zeta functions for GL(n, Z), Proc. Intern. Cong. Math. Berkeley 1 (1986), 417-424.

[I] H. Iwaniec, On Waldspurger’s Theorem, Acta Arith. 49 (1987), 205-212.

[J1] H. Jacquet, On the nonvanishing of some L-functions, Proc. Indian Acad. Sci. (Math. Sci.) 97 (1987), 117-155.

[J2] , Relative Kloosterman integrals for GL(3): II, Canadian J. Math. 44 (1992), 1220-1240.

[J3] , The continuous spectrum of the relative trace formula for GL(3) over a quadratic extension, Israel J. Math. (to appear).

[JL] H. Jacquet and K. F. Lai, A relative trace formula, Compositio Math. 54 (1985), 243-310.

[JLR] H. Jacquet, K. F. Lai, and S. Rallis, A trace formula for symmetric spaces, Duke Math. J. 70 (1993), 305-372.

[JS] H. Jacquet and J. Shalika, Exterior square L-functions, in Automorphic Forms, Shimura Varieties, and L-functions, Academic Press, 1990, pp. 143–226.

[JY1] H. Jacquet and Y. Ye, Une remarque sur le changement de base quadratique, Comptes Rendus de l’Académie des Sciences, Paris 311 (1990), 671–676.

[JY2] , Relative Kloosterman integrals for GL(3), Bull. Soc. Math. France 120 (1992), 263-295.

[JY3] , Distinguished representations and quadratic base change for GL(3) (preprint).

[M] Z. Mao, Relative Kloosterman integrals for GL(3): III, Canadian J. Math. 45 (1993), 1211-1230.

[S] G. Stevens, Poincaré series on GL(r) and Kloosterman sums, Math. Ann. 277 (1987), 21-51.

[Y1] Y. Ye, Kloosterman integrals and base change for GL(2), J. reine angew. Math. 400 (1989), 57-121.

[Y2] , The Fundamental lemma of a relative trace formula for GL(3), Compositio Math. 89 (1993), 121-162.

[Y3] , An integral transform and its applications, Math. Ann. (to appear).

[Z] D. Zagier, Modular forms associated to real quadratic fields, Invent. Math. 30 (1975), 1-46.