R-GROUPS AND ELLIPTIC REPRESENTATIONS FOR SIMILITUDE GROUPS

David Goldberg*

Abstract. The tempered spectrum of the similitude groups of non-degenerate symplectic, hermitian, or split orthogonal forms defined over p-adic groups of characteristic zero is studied. The components of representations induced from discrete series of proper parabolic subgroups are classified in terms of R-groups. Multiplicity one is proved. The tempered elliptic spectrum is identified, and the relation between elliptic characters appearing in a given induced representation is determined. Those irreducible tempered representations which are not elliptic and not fully induced from elliptic tempered representations are described.

Introduction. We continue our program of studying the explicit description of the tempered spectrum of reductive groups defined over a p-adic field of characteristic zero. The problem can be broken down into two steps. The first step is to classify the discrete series representations of G and of all its Levi subgroups. The second step is to determine the components of those representations which are parabolically induced from a discrete series representation of a proper parabolic subgroup P of G. It is this second step that we shall concern ourselves with. This problem also has two distinct parts. The first is to determine the criteria for an induced representation to be reducible when P is a maximal proper parabolic subgroup. This involves the computation of Plancherel measures. The second part is to use knowledge of the rank one Plancherel measures to construct the Knapp-Stein R-group. This gives one a combinatorial algorithm for determining the structure of the induced representations. Those irreducible tempered representations whose characters fail to vanish on the regular elliptic set are called elliptic tempered representations, and are of particular interest.

Typeset by AMS-TEX 2 DAVID GOLDBERG

*Partially supported by NSF Postdoctoral Fellowship DMS9206246

Research at MSRI supported in part by NSF grant DMS9022140

1991 AMS Subject Classification: 22E50, 22E35

In [6,7,10] we determined the possible R-groups for Sp_{2n}, SO_{n}, U_{n}, and SL_{n}. Here we built upon the work of Keys [18,19]. Arthur gives a criteria for determining the elliptic tempered representations in terms of R-groups in [2]. In [15] Herb described the elliptic tempered representations for Sp_{2n} and SO_{n}. For SO_{2n} she exhibited irreducible tempered representations which are neither elliptic nor irreducibly induced from an elliptic tempered representation of a proper parabolic subgroup. Such representations do not exist for F = ℝ, [21], and this is the first known example of them in the p-adic case. Using some results of Herb and Arthur, we described the elliptic representations for SL_{n} and the quasi-split unitary groups U_{n} [6,10]. The problem of determining the zeros of Plancherel measures has been explored by Shahidi in [24,25,26,27,28,29]. In [29] Shahidi determined the reducibility criteria for the Siegel parabolic subgroups of Sp_{2n} and SO_{n}. In [24] he determined the criteria for parabolics of SO_{2n} whose Levi component is of the form GL_{k} × SO_{2m} where k is even. In [9] we determined the criteria for the Siegel parabolics of U_{n}. Our joint work with Shahidi [12] has determined this criteria for symplectic and quasi-split orthogonal groups in the case where M = GL_{k} × G(m), with k even. Our continuing joint work with Shahidi will will examine other maximal parabolic subgroups of classical groups.

Here we compute the possible R-groups for the similitude group of a nondegenerate symplectic, symmetric, or quasi-split hermitian form, defined over F. We show that all the R-groups are elementary two groups (cf. Theorem 2.6). Moreover, the R-groups are all contained in the subgroup of the Weyl group consisting of sign changes. For GU_{2n} this extends the work of Keys [19]. We then undertake the computation of the elliptic tempered representations. Suppose M is a Levi subgroup of G. Then, for some collection of positive integers m_{1}, …, m_{r}, and some m ≥ 0, we have M = M(F) ≃ GL_{m_{1}} × ⋯ × GL_{m_{r}} × G(m), where G(m) = GSp_{2m}, GU_{2m}, GO_{2m+1}, GU_{2m+1}, or GO_{2m} as appropriate (cf. Section 2). If σ is a discrete series representation of M, then the induced representation i_{G,M}(σ) has elliptic constituents if and only if the longest possible sign change w_{0} = C_{1} … C_{r} is in R(σ) (cf. Proposition 3.3). We make this more explicit on a case by case basis (cf. Theorems 3.4 and 3.6). If G = GO_{2n+1} or GU_{2n+1},

R-GROUPS AND ELLIPTIC REPRESENTATIONS FOR SIMILITUDE GROUPS 3

then every irreducible tempered representation of G = G(F) is either elliptic, or is irreducibly induced from an elliptic tempered representation of a proper Levi subgroup (cf. Theorem 3.4). For G = GU_{2n}, GSp_{2n}, or GO_{2n}, this statement fails to be true. We classify those irreducible tempered representations which are non-elliptic, and are not irreducibly induced from elliptic tempered representations (cf. Theorems 3.4 and 3.6).

We also show that when an induced representation has elliptic components, then the elliptic characters of those components satisfy a particularly nice relation. More specifically, suppose κ is an irreducible representation of the R-group R, and π_{κ} is the irreducible subrepresentation of i_{G,M}(σ) attached to κ [2,19]. Let Θ_{κ}^{e} be the restriction of the character of π_{κ} to the regular elliptic set. Then Θ_{κ}^{e} = κ(C_{1} … C_{r})Θ_{1}^{e}, where 1 is the trivial character (cf. Theorem 3.9). This result is similar to the ones obtained by Herb in [15], and is in fact motivated by that work.

I would like to thank Freydoon Shahidi and Rebecca Herb for their comments on this work. I would also like to thank the Mathematical Sciences Research Institute, in Berkeley, for providing the pleasant atmosphere in which this work was concluded.

Section 1 Preliminaries.

Suppose F is a nonarchimedean local field of characteristic zero and residual characteristic q. Let G be a connected reductive quasi–split algebraic group defined over F. We denote by G the F -rational points of G. We let G^{e} be the collection of regular elliptic elements of G, i.e., the set of regular elements whose centralizers in G are compact modulo the center of G.

We denote by E_{c}(G) the collection of equivalence classes of irreducible admissible representations of G. We make no distinction between an irreducible admissible representation π of G, and its class [π] in E_{c}(G). If π ∈ E_{c}(G), then the distribution character Θ_{π} of π is given by a locally integrable function [13]. We let Θ_{π}^{e} be the restriction of Θ_{π} to G^{e}. We say that π is elliptic if Θ_{π}^{e} ≠ 0. We

4 DAVID GOLDBERG

let E_{2}(G), E_{t}(G), and E_{e}(G) be the collection of discrete series, tempered, and tempered elliptic classes in E_{c}(G), respectively. Then E_{2}(G) ⊂ E_{e}(G) ⊂ E_{t}(G).

We fix a maximal torus T of G, and let T_{d} be the maximal split subtorus of T. Denote by Φ = Φ(G, T_{d}) the set of roots of T_{d} in G. Let Δ be a choice of simple roots in Φ, and let Φ^{+} = Φ^{+}(G, T_{d}) be the positive roots with respect to this choice of Δ. The choice of Δ also determines a Borel subgroup B = TU of G. If θ ⊆ Δ, then we let A_{θ} be the split subtorus of T_{d} determined by θ. Then A_{θ} is the split component of a unique parabolic subgroup P_{θ} = M_{θ}N_{θ} of G containing B.

Suppose A = A_{θ} for some θ. Then M = M_{θ} = Z_{G}(A) is a Levi subgroup of G with split component A. We denote by W (G, A) the Weyl group N_{G}(A)/M. If w ∈ W (G, A), we let w̃ ∈ N_{G}(A) be a representative for w.

Suppose (σ, V) ∈ E_{2}(M). Let

V (σ) = {f ∈ C^{∞}(G, V)|f(mng) = σ(m)δ_{P}^{1/2}(m)f(g), ∀m ∈ M, n ∈ N, g ∈ G}.

Here δ_{P} denotes the modular function of P. The unitarily induced representation, Ind_{P}^{G}(σ), of G on V (σ) is given by right translation. Since the class of Ind_{P}^{G}(σ) depends only on M and not on the choice of N, we may denote its class by i_{G,M}(σ). We now recall the theory of R-groups which allows us to determine the structure of Ind_{P}^{G}(σ). For more details see [7,11,18,25,30,31]. For σ ∈ E_{2}(M), we let W (σ) = {w ∈ W (G, A)|w̃σ = σ}. Here w̃σ(m) = σ(w̃^{−1}mw̃). Since the class of w̃σ is independent of the choice of w̃, we may denote it by wσ. Let Φ(P, A) be the reduced roots of A in P. For β ∈ Φ(P, A), we let A_{β} be the subtorus of A defined by β. Let M_{β} = Z_{G}(A_{β}). Then M is a maximal Levi subgroup of M_{β}. Let N_{β} = M_{β} ∩ N, and ^{∗}P_{β} = MN_{β}. Then ^{∗}P_{β} is a maximal parabolic subgroup of M_{β}. Let μ_{β}(σ) be the Plancherel measure of σ with respect to β [14,31]. The value of μ_{β} is given by ratios of certain values of Langlands L-functions [28]. For our purposes, it is enough to know that μ_{β}(σ) = 0 if and only if W (M_{β}, A) ∩ W (σ) ≠ {1} and i_{M_{β},M}(σ) is irreducible. Let Δ′ = {β ∈ Φ(P, A)|μ_{β}(σ) = 0}. Then ±Δ′ is a subroot system of Φ [5, VI,

R-GROUPS AND ELLIPTIC REPRESENTATIONS FOR SIMILITUDE GROUPS 5

§2, Proposition 9]. Thus, the group W′ generated by the reflections determined by the elements of Δ′ is a subgroup of W (σ). Let R = R(σ) = {w ∈ W (σ)| wβ > 0, ∀β ∈ Δ′}.

Theorem 1.1 (Knapp–Stein, Silberger [20,30,32]). For any σ ∈ E_{2}(M) we have W (σ) = R ⋉ W′. Moreover, the commuting algebra C(σ) of Ind_{P}^{G}(σ) has dimension |R|. □

We call R the Knapp–Stein R-group attached to σ. Let w ∈ W (σ), and choose T_{w} : V ⟶ V which defines an isomorphism between σ and w̃σ. Let w_{1}, w_{2} ∈ R, and define η(w_{1}, w_{2}) by T_{w_{1}w_{2}} = η(w_{1}, w_{2})T_{w_{1}}T_{w_{2}}. Then η: R × R ⟶ ℂ^{×} is a 2–cocycle.

Theorem 1.2 (Keys [19]).

(a) The commuting algebra C(σ) of Ind_{P}^{G}(σ) is isomorphic to ℂ[R]_{η}, the group algebra of R twisted by the cocycle η.

(b) Suppose η is a coboundary. Let R̂ be the set of equivalence classes of irreducible representations of R. Then there is a natural one-to–one correspondence, κ ⟼ π_{κ}, between R̂ and the equivalence classes of subrepresentations of i_{G,M}(σ) such that dim κ = dim Hom_{G}(π_{κ}, i_{G,M}(σ)). In particular, if R is abelian and η is a coboundary, then C(σ) ≃ ℂ[R], and i_{G,M}(σ) decomposes as |R| inequivalent irreducible subrepresentations. □

Let 𝔞 = Hom(X(M)_{F} , ℤ) be the real Lie algebra of A. (Here X(M)_{F} is the collection of F -rational characters of M .) For w ∈ W (G, A), we let 𝔞_{w} = {H ∈ 𝔞|w · H = H}. Let Z be the split component of G, and let 𝔷 be its real Lie algebra. Suppose σ ∈ E_{2}(M), and let R be the R-group of σ. We let 𝔞_{R} = ⋂_{w∈R} 𝔞_{w}. In [2] Arthur gives an explicit criteria determining when i_{G,M}(σ)

has an elliptic subrepresentation. We give a weak version of his results which is sufficient for our needs.

Theorem 1.3 (Arthur [2]). Suppose R is abelian and η is a coboundary.

6 DAVID GOLDBERG

Then the following are equivalent:

(a) i_{G,M}(σ) has an elliptic constituent;

(b) all the constituents of i_{G,M}(σ) are elliptic;

(c) there is a w ∈ R with 𝔞_{w} = 𝔷. □

We are also interested in which tempered representations are not elliptic, and whether they appear as irreducibly induced from tempered representations. If a tempered representation is irreducibly induced, we wish to know when the inducing representation is elliptic. The next two results allow us to determine these things. We again state a weaker version of Arthur’s result (Theorem 1.4) than the one given in [2].

Suppose M′ ⊃ M is a Levi subgroup of G satisfying Arthur’s compatibility condition with respect to Δ′ [2, §2]. Then R′ = R ∩ W (M′, A) is the R-group attached to i_{M′,M}(σ). If κ′ ∈ R̂′, then we let τ_{κ′} to be the irreducible component of i_{M′,M}(σ) corresponding to κ′. Let

R̂(κ′) = {κ ∈ R̂ | κ(w) = κ′(w), ∀w ∈ R′}.

We let π_{κ} be the irreducible constituent of i_{G,M}(σ) corresponding to κ.

Theorem 1.4 (Arthur [2, §2]). Suppose that R is abelian and C(σ) ≃ ℂ[R]. Then, for any κ′ ∈ R̂′, we have

i_{G,M′}(τ_{κ′}) = ⨁_{κ∈R̂(κ′)}π_{κ}. □

Proposition 1.5 (Herb [15]). Suppose R is abelian and C(σ) ≃ ℂ[R]. Let π be an irreducible constituent of i_{G,M}(σ). Then π = i_{G,M′}(τ) for a proper Levi subgroup M′ and some τ ∈ E_{t}(M′), if and only if 𝔞_{R} ≠ 𝔷. Moreover, M′ and τ can be chosen so that τ is elliptic if and only if there is a w_{0} ∈ R with 𝔞_{R} = 𝔞_{w_{0}}. □

Section 2. The Similitude Groups and their R-groups.

R-GROUPS AND ELLIPTIC REPRESENTATIONS FOR SIMILITUDE GROUPS 7

We now describe the possible R-groups that can arise when G is one of the following groups: GSp_{2n}, GO_{n}, or GU_{n}. Thus, G is the similitude group of a non–degenerate symplectic, symmetric, or hermitian form. In the last case we assume that the hermitian form defines the quasi–split unitary group of rank [\frac{n}{2}], and we let E/F be the quadratic extension of F over which G splits. Denote by x ↦ \overline{x} the Galois automorphism of E over F. We let N(x) = x\overline{x} be the norm map.

Let J be the form with respect to which G is defined. Then, if G = GSp_{2n} or GO_{n},

G = {g ∈ GL_{n}| ^{t}gJg = λ(g)J for some λ(g) ∈ G_{m}}.

For the unitary similitude groups

G = \{g ∈ Res_{E/F} GL_{n}| ^{t}\overline{g}Jg = λ(g)J, for some λ(g) ∈ Res_{E/F} G_{m}\}.

In each case we call λ the multiplier character of G. If ω is a character of F^{×}, then we also denote by ω the character of G(F) given by ω ∘ λ. We fix the following forms for the various G. We let

w_{ℓ} = ( \begin{matrix}& & & & & & 1 \\ & & & & & · \\ & & & & · \\ & & & · \\ & & 1 \\ & 1 \\ 1 \end{matrix} ) ∈ M_{ℓ}(F).

Then we take J = ( \begin{matrix}0 & w_{n} \\ −w_{n} & 0 \end{matrix} ) if G = GSp_{2n}. For G = GO_{n}, we take

J = w_{n}. For the unitary groups, we fix an element y ∈ E with \overline{y} = −y. Then we let

J = y ( \begin{matrix}0 & w_{n} \\ −w_{n} & 0 \end{matrix} ) if G = GU_{2n} and
J = ( \begin{matrix}0 & 0 & yw_{n} \\ 0 & 1 & 0 \\ y(−w_{n}) & 0 & 0 \end{matrix} ) if G = GU_{2n+1}.

8 DAVID GOLDBERG

We will often denote GSp_{2n}, GO_{2n}, GO_{2n+1}, GU_{2n}, or GO_{2n+1} by G(n). We formally let G(0) = GL_{1} if G ≠ GU_{2n+1} and G(0) = Res_{E/F} GL_{1} if G = GU_{2n+1}.

Let T be the maximal torus of diagonal elements in G. We given an explicit description of T = T(F) in each case. We adopt the convention of denoting a diagonal matrix by diag{(x_{1}, …, x_{k})}. If G = GSp_{2n} or GO_{2n}, then

(2.1)  T = {diag{x_{1}, …, x_{n}, λx_{n}^{−1}, …, λx_{1}^{−1}}|x_{i}, λ ∈ F^{×}}.

If G = GO_{2n+1}, then

(2.2.)  T = {diag{x_{1}, …, x_{n}, λ, λ^{2}x_{n}^{−1}, …, λ^{2}x_{1}^{−1}}|x_{i}, λ ∈ F^{×}}

If G = GU_{2n}, then

(2.3)  T = \{diag(x_{1}, …, x_{n}, λ\overline{x}_{n}^{−1}, …, λ\overline{x}_{1}^{−1})|λ ∈ F^{×}, x_{i} ∈ E^{×}\},

While if G = GU_{2n+1}, then

(2.4)  T = \{diag\{x_{1}, …, x_{n}, λ, N(λ)\overline{x}_{n}^{−1}, …, N(λ)\overline{x}_{1}^{−1}\}|x_{i}, λ ∈ E^{×}\}.

In each case we let T_{d} be the maximal F split subtorus of T. For G = GSp_{2n} or GO_{n}, we have T = T_{d}. For G = GU_{2n}, T_{d} is given by (2.1), while T_{d} for GU_{2n+1} is given by (2.2). We sometimes denote an element of T by t(x_{1}, …, x_{n}, λ).

Note that in each case W (G, T_{d}) is given by S_{n} ⋉ ℤ_{2}^{n}. We use standard cycle notation for the elements of S_{n}. Thus,

(ij): t(x_{1}, …, x_{n}, λ) ⟼ t(x_{1}, …, x_{j}, …, x_{i}, …, x_{n}, λ).

We let c_{i} be the i-th sign change,

c_{i}: t(x_{1}, …, x_{n}, λ) ⟼ t(x_{1}, …, x_{i−1}, λx_{i}^{−1}, …, x_{n}, λ).

We call any product of the c_{i} ’s a sign change. We remark that for G = SO_{2n}, only those sign changes which are products of an even number of c_{i} ’s are in

R-GROUPS AND ELLIPTIC REPRESENTATIONS FOR SIMILITUDE GROUPS 9

W (G, T_{d}). However, if G = GO_{2n}, then the sign changes c_{i} can be represented in G. Finally we remark that W (G, T_{d}) = ⟨(ij)⟩ ⋉ ⟨c_{i}⟩.

Let Δ be the choice of simple roots of T_{d} in G for which the associated Borel subgroup consists of upper triangle matrices. Suppose θ ⊂ Δ. Let P = P_{θ} be the associated parabolic subgroup, and write P = MN for the Levi decomposition of P.

For some positive integers m_{1}, …, m_{r}, and some m ≥ 0 we have,

(2.5)  M = M(F) ≃ GL_{m_{1}} × … × GL_{m_{r}} × G(m),

where

GL_{m_{i}} = \{ \begin{matrix}GL_{m_{i}}(F), & if G ≠ GU_{n}; \\ GL_{m_{i}}(E), & if G = GU_{n}. \end{matrix}

In particular, we may assume that M consists of block diagonal matrices. Let ε: GL_{m_{i}} ⟶ GL_{m_{i}} be given by ε(g) = ^{t}\overline{g}^{−1} (where for GL_{m_{i}}(F), the Galois automorphism is of course trivial), and we let ε′(g) = w_{m_{i}}ε(g)w_{m_{i}}^{−1}. Then we may assume that

M = {diag{g_{1}, …, g_{r}, g, λ(g)ε′(g_{r}), …, λ(g)ε′(g_{1})}| g_{i} ∈ GL_{m_{i}}, g ∈ G(m)} .

We may denote an element of M by (g_{1}, …, g_{r}, g). Note that

A = A(F) = \{diag\{x_{1}I_{m_{1}}, …, x_{r}I_{m_{r}}, I_{k}, \overline{x}_{r}^{−1}I_{m_{r}} … \overline{x}_{1}^{−1}I_{m_{1}}\}\},

where k is chosen appropriately to be 2m or 2m + 1. For 1 ≤ i ≤ r, we let

b_{i} = ∑_{j=1}^{i} m_{j}, and set b_{0} = 0. For 1 ≤ i ≤ j ≤ r − 1, we let α_{ij} = e_{b_{i}} − e_{b_{j}+1},

and β_{ij} = e_{b_{i}} + e_{b_{j}+1}. For 1 ≤ i ≤ r, we set

γ_{i} = \{ \begin{matrix}2e_{b_{i}}, & if G = GSp_{2n} or GU_{2n}; \\ e_{b_{i}}, & if G = GO_{2n+1} or GU_{2n+1}; \\ e_{b_{i}−1} + e_{b_{i}}, & if G = GO_{2n}. \end{matrix}

Then Φ(P, A) = {α_{ij}, β_{ij}}_{1≤i≤j≤r−1} ∪ {γ_{i}}_{1≤i≤r}.

Note that the Weyl group W (G, A) is a subgroup of S_{r} ⋉ ℤ_{2}^{r}. Namely, if m_{i} = m_{j}, then the permutation:

w_{ij} = ∏_{k=1}^{m_{i}}(b_{i−1} + k b_{j−1} + k)

is in W (G, A), and the map w_{ij} ↦ (ij) gives an isomorphism of this permutation group with a subgroup of S_{r}. For each i, we have the sign change

C_{i} = ∏_{j=1}^{m_{i}} c_{b_{i−1}+j}

is in W (G, A). We call C_{i} a block sign change. The C_{i} ’s generate the subgroup ℤ_{2}^{r} in the semidirect product. Note that, for (g_{1}, …, g_{r}, g) ∈ M, we have

w_{ij}(g_{1}, … g_{r}, g) = (g_{1}, … g_{j}, … g_{i} … g_{r}, g), and
C_{i}(g_{1} … g_{r}, g) = (g_{1}, … λ(g)ε(g_{i}), …, g_{r}, g).

Suppose σ ∈ E_{2}(M). Then σ ≃ σ_{1} ⊗ σ_{2} ⊗ … ⊗ σ_{r} ⊗ ρ, for some σ_{i} ∈ E_{2}(GL_{m_{i}}) and ρ ∈ E_{2}(G(m)). Let ω_{i} be the central character of σ_{i}. Note that if w_{ij} ∈ W (G, A), then

w_{ij}σ ≃ σ_{1} ⊗ … ⊗ σ_{j} ⊗ … ⊗ σ_{i} ⊗ … ⊗ σ_{r} ⊗ ρ.

If 1 ≤ i ≤ r, then

C_{i}σ ≃ σ_{1} ⊗ … ⊗ σ_{i}^{ε} ⊗ … ⊗ σ_{r} ⊗ (ρ ⊗ ω_{i}).

Here σ_{i}^{ε}(g_{i}) = σ_{i}(ε(g_{i})). Thus, W (σ) ≠ {1} if and only if at least one of the following holds:

(2.6)  σ_{i} ≃ σ_{j} for some i ≠ j,
(2.7)  σ_{i} ≃ σ_{j}^{ε} and ρ ⊗ ω_{i}ω_{j} ≃ ρ, or ,

for some subset {i_{j}} ⊆ {1, …, r},

(2.8)  σ_{i_{j}} ≃ σ_{i_{j}}^{ε} for all j and ρ ⊗ (∏_{j} ω_{i_{j}} ) ≃ ρ.

R-GROUPS AND ELLIPTIC REPRESENTATIONS FOR SIMILITUDE GROUPS 11

Note that if (2.6) holds then w_{ij} ∈ W (σ). If (2.7) holds then w_{ij}C_{i}C_{j} ∈ W (σ). Finally, (2.8) implies ∏_{j} C_{i_{j}} ∈ W (σ). Note that in (2.7), since σ_{i} ≃ σ_{j}^{ε}, we have

ω_{i} = \{ \begin{matrix}ω_{j}^{−1} & if G ≠ GU_{n}, \\ ω̄_{j}^{−1} & if G = GU_{n}. \end{matrix}

Since

λ(G(m)) = \{ \begin{matrix}F^{×}, & if G = GSp_{2n}, GO_{2n}, or GU_{2n}; \\ F^{x^{2}}, & if G = GO_{2n+1}; \\ N_{E/F} E^{×}, & if G = GU_{2n+1}, \end{matrix}

(see [16,22]), we see that ω_{i}ω_{j} ∘ λ = 1 if σ_{i} ≃ σ_{j}^{ε}. Thus, (2.7) simply becomes σ_{i} ≃ σ_{j}^{ε}.

We begin the computation of the possible R-groups with the following standard result. Recall that Δ′ = {β ∈ Φ(P, A)|μ_{β}(σ) = 0}.

Lemma 2.1. The reduced root α_{ij} ∈ Δ′ if and only if σ_{i} ≃ σ_{j+1}. Similarly, β_{ij} ∈ Δ′ if and only if σ_{i} ≃ σ_{j+1}^{ε}.

Proof. Note that

M_{α_{ij}} ≃ ( ∏_{k≠i,j+1} GL_{m_{k}} ) × GL_{m_{i}+m_{j+1}} × G(m) ≃ M_{β_{ij}}.

Thus, by the results of Ol’šanskiǐ [23], Bernstein–Zelevinsky [4,33], and Jacquet [17], i_{M_{α_{ij}} ,M}(σ) and i_{M_{β_{ij}} ,M}(σ) are both irreducible for all σ ∈ E_{2}(M). Thus, μ_{α_{ij}}(σ) = 0 if and only if there is some w ≠ 1 in W (M_{α_{ij}}, A) with wσ ≃ σ. If m_{i} ≠ m_{j+1}, then W (M_{α_{ij}}, A) = {1}. However, if m_{i} = m_{j+1}, then W (M_{α_{ij}}, A) = {w_{ij}, 1}, and so μ_{α_{ij}}(σ) = 0 if and only if σ_{i} ≃ σ_{j+1}. Since β_{ij} = C_{j+1}α_{ij}, we see that μ_{β_{ij}}(σ) = 0 if and only if μ_{α_{ij}}(C_{ij}σ) = 0, which is equivalent to σ_{i} ≃ σ_{j+1}^{ε}. □

We now show that the R-group of any σ ∈ E_{2}(M) is an elementary 2–group. The proof of the following lemma is based on a technique of Keys [18].

Lemma 2.2. Suppose w = sc ∈ R, with s ∈ S_{r} and c ∈ ℤ_{2}^{r}. Then s = 1.

Proof. By conjugating by a sign change, we may assume that c changes the sign of at most one e_{b_{i}} in each orbit of s. Suppose s has a non–trivial cycle, which we assume is (1 … j +1). If c changes no signs among {e_{b_{1}}, …, e_{b_{j+1}}}, then we see that wσ ≃ σ implies σ_{1} ≃ σ_{2} ≃ … ≃ σ_{j+1}. By Lemma 2.1, we see that α_{1j} ∈ Δ′, while wα_{1j} < 0. This contradicts the assumption that w ∈ R. Suppose that c changes the sign of e_{b_{j+1}}. Then wσ ≃ σ implies σ_{1} ≃ … ≃ σ_{j+1} ≃ σ_{1}^{ε}. So by Lemma 2.1, β_{1j} ∈ Δ′. But wβ_{1j} = sα_{1j} < 0, and again we have a contradiction. Thus, s = 1. □

Suppose that, for some subset B ⊂ {1, 2, …, r}, we have C_{B} = ∏_{j∈B} C_{j} ∈ R.

Set ω_{B} = ∏_{j∈B} ω_{j}. Then, for each j ∈ B, σ_{j} ≃ σ_{j}^{ε}, and ρ ⊗ ω_{B} ≃ ρ. Let

N_{ε}F^{×} = \{ \begin{matrix}N_{E/F} E^{×}, & if G = GU_{n}; \\ F^{x^{2}}, & if G = GSp_{2n} or GO_{n}. \end{matrix}

If σ_{i} ≃ σ_{i}^{ε} then ω_{i}|_{N_{ε}F^{×}} = 1. If G = GSp_{2n}, GO_{2n}, or GU_{2n}, then λ(G(m)) = F^{×}, while if G = GO_{2n+1}, or GU_{2n+1}, then λ(G(m)) = N_{ε}F^{×}. Thus, for these last two groups, the condition ρ ⊗ ω_{B} ≃ ρ is trivial. Therefore, for G = GO_{2n+1} or GU_{2n+1}, we see that C_{B} ∈ R only if σ_{i} ≃ σ_{i}^{ε} for each i ∈ B.

Definition 2.3. Let σ ∈ E_{2}(GL_{k}) and ρ ∈ E_{2}(G(m)). We say that condition X_{m,k,G}(σ⊗ρ) holds if i_{G(m+k),GL_{k}×G(m)}(σ⊗ρ) is reducible. Note that a necessary condition for X_{m,k,G}(σ ⊗ ρ) to hold is that ρ ⊗ ω_{σ} ≃ ρ, where ω_{σ} is the central character of σ.

Definition 2.4. If w ∈ W (G, A), then we let R(w) = {α ∈ Φ(P, A)|wα < 0}. Note that w ∈ R if and only if w ∈ W (σ) and R(w) ∩ Δ′ = ∅.

Lemma 2.5. Suppose G = GO_{2n+1} or GU_{2n+1}. Suppose P = MN is a parabolic subgroup of G with M = M(F) ≃ GL_{m_{1}} × … × GL_{m_{r}} × G(m). Let

σ ∈ E_{2}(M), with σ ≃ σ_{1} ⊗ … ⊗ σ_{r} ⊗ ρ. If c ∈ R, and c = C_{B} = ∏_{j∈B} C_{j}, then

C_{j} ∈ R for each j ∈ B.

R-GROUPS AND ELLIPTIC REPRESENTATIONS FOR SIMILITUDE GROUPS 13

Proof. Note that, for each j ∈ B, we have R(C_{j}) ⊂ R(c). Thus, if C_{j}β < 0 then C_{B}β < 0, and hence β ∉ Δ′. Now it is enough to note that, since cσ ≃ σ, we have σ_{j} ≃ σ_{j}^{ε} for each j ∈ B, and thus C_{j} ∈ W (σ). Therefore, C_{j} ∈ R. □

In order to give an explicit description of the R-groups, we need to define some terms. Let ρ ∈ E_{2}(G(m)), and set

X(ρ) = {χ ∈ (F^{×}/N_{ε}F^{×})^{∧} |ρ ⊗ χ ≃ ρ}.

Let

J_{1}(σ) = {i|X_{m,m_{i},G}(σ_{i} ⊗ ρ) holds}.

For each i ∈ J_{1}(σ), we have ω_{i} ∈ X(ρ). For each χ ∈ (F^{×}/N_{ε}F^{×})^{∧} \ X(ρ), let

J_{χ}(σ) = {i|σ_{i} ≃ σ_{i}^{ε} and ω_{i}|_{F^{×}} = χ}.

For χ = 1 or χ ∉ X(ρ), we set

I_{χ}(σ) = {i ∈ J_{χ}(σ)|σ_{i} ≄ σ_{j}, ∀i > j},

and let d_{χ} = d_{χ}(σ) = |I_{χ}(σ)| be the number of inequivalent σ_{i} for i ∈ J_{χ}(σ).

For a nonempty subset S ⊂ (F^{×}/N_{ε}F^{×})^{∧}, we say that S is minimally ρtrivial if ∏_{χ∈S} χ ∈ X(ρ), and ∏_{χ∈S′} χ ∉ X(ρ) for any ∅ ⊊ S′ ⊊ S. We let

Λ(σ) =

{S ⊂ (F^{×}/N_{ε}F^{×})^{∧} \ X(ρ)|S is minimally ρ − trivial and d_{χ}(σ) ≥ 1, ∀χ ∈ S}.

Let {S_{1}, …, S_{k}} ⊆ Λ(σ). We let

X({S_{1}, …, S_{k}}) = {χ | χ ∈ S_{j} for an odd number of j}.

We say that Λ(σ)′ is a basis for Λ(σ) if, for every S ∈ Λ(σ), there is some {S_{1}, …, S_{k}} ⊂ Λ(σ)′ with S = X({S_{1}, …, S_{k}}), and Λ(σ)′ is minimal with respect to this property.

Theorem 2.6. Let M be a Levi subgroup of G, with

M ≃ GL_{m_{1}} × … × GL_{m_{r}} × G(m).

Suppose σ ∈ E_{2}(M), with σ ≃ σ_{1} ⊗ … ⊗ σ_{r} ⊗ ρ. Let R = R(σ).

(a) If G = GO_{2n+1} or GU_{2n+1}, then R ≃ ℤ_{2}^{d}, with d = d_{1}, i.e., d is the number of inequivalent σ_{i} such that X_{m,m_{i},G}(σ ⊗ ρ) holds.

(b) If G = GSp_{2n}, GO_{2n}, or GU_{2n}, then R ≃ ℤ_{2}^{d} with

d = d_{1} + ∑_{{χ∉X(ρ) \atop dχ≥1}}(d_{χ} − 1) + |Λ(σ)′|,

for any basis Λ(σ)′ of Λ(σ).

Proof. (a) By Lemma 2.5, is enough to show that C_{i} ∈ R if and only if i ∈ I_{1}(σ). Suppose C_{i} ∈ W (σ). Note that R(C_{i}) = {α_{ij}, β_{ij}}_{i≤j≤r−1} ∪ {γ_{i}}. Note that if σ_{j+1} ≃ σ_{i}, for some j ≥ i, then α_{ij} and β_{ij} ∈ Δ′, and thus C_{i} ∉ R. Direct computation shows that M_{γ_{i}} ≃ ( ∏_{k≠j} GL_{m_{k}}) × G(m + m_{i}). Since C_{i}σ ≃ σ, we

see that γ_{i} ∈ Δ′ if and only if X_{m,m_{i},G}(σ_{i} ⊗ ρ) does not hold. Thus, C_{i} ∈ R if and only if X_{m,m_{i},G}(σ_{i} ⊗ ρ) holds and σ_{i} ≄ σ_{j} for all j > i, i.e., if and only if i ∈ I_{1}(σ).

(b) Suppose B ⊆ {1, 2, …, r} with C_{B} = ∏_{j∈B} C_{j} ∈ R. By the argument of

part (a), we see that, for each j ∈ B, σ_{k} ≄ σ_{j} for all k > j. Note again that M_{γ_{j}} ≃ ( ∏_{k≠j} GL_{m_{k}}) × G(m + m_{j}). Therefore, γ_{j} ∈ Δ′ if and only if C_{j}σ ≃ σ and

X_{m,m_{j},G}(σ ⊗ ρ) fails. Since C_{j}σ ≃ σ if and only if σ_{j} ≃ σ_{j}^{ε}, and ω_{j} ∈ X(ρ), we see that those j ∈ B for which ω_{j} ∈ X(ρ) are all elements of I_{1}(σ). Let

R_{1} = R_{1}(σ) = ⟨C_{j} | j ∈ I_{1}(σ)⟩.

Then we have just shown that R_{1}(σ) ⊆ R(σ). If j ∈ B and ω_{j} ∉ X(ρ), then C_{j} ∉ W (σ), and thus γ_{j} ∉ Δ′. Let χ ∈ (F^{×}/N_{ε}F^{×})^{∧}\X(ρ), and define

R_{χ}(σ) = ⟨C_{i}C_{j}|i, j ∈ I_{χ}(σ)⟩.

R-GROUPS AND ELLIPTIC REPRESENTATIONS FOR SIMILITUDE GROUPS 15

If C_{i}C_{j} ∈ R_{χ}(σ), then ω_{i}ω_{j}|_{F^{×}} = 1, so C_{i}C_{j} ∈ W (σ). Note that R(C_{i}C_{j}) = R(C_{i})∪R(C_{j}). Moreover, by the definition of I_{χ}(σ), and Lemma 2.1, we see that, for all k ≥ i and ℓ ≥ j, the reduced roots α_{ik}, α_{jℓ}, β_{ik}, and β_{jℓ} can not be in Δ′. Since γ_{i}, γ_{j} ∉ Δ′, we see that C_{i}C_{j} ∈ R, and thus R_{χ}(σ) ⊆ R.

Since each R_{χ}(σ) ⊂ R(σ), we can multiply C_{B} by an element C′ of

R_{1} × ∏_{χ∉X(ρ)} R_{χ}(σ)

so that C_{B′} = C′C_{B} has the property that B′ ∩ I_{1}(σ) = ∅, and ω_{i} ≠ ω_{j} for all i ≠ j in B′. Since ω_{B} ∈ X(ρ), we see that {ω_{j}|_{F^{×}} |j ∈ B′} can be partitioned into minimally ρ-trivial subsets. For each χ ∈ (N_{ε}F^{×}\F^{×})^{∧} \X(ρ), with d_{χ} > 0, we fix an i_{χ} ∈ I_{χ}(σ). Let S ∈ Λ(σ), and define C_{S} = ∏_{χ∈S} C_{i_{χ}}. Since ∏_{χ∈S} χ ∈

X(ρ), we see that C_{S} ∈ W (σ), and by the definition of I_{χ}(σ), we see that C_{S} ∈ R. Suppose C′ = C_{D} ∈ R has the property that {ω_{j}|_{F^{×}}|j ∈ D} = S and |D| = |S|. Then

C_{D} = C_{S} · ∏_{j∈D} C_{j} C_{i_{ω_{j}}} ,

and each C_{j}C_{i_{ω_{j}}} ∈ R_{ω_{j}}(σ). Thus, all the factors in the product are in R. Fix a basis Λ(σ)′ for Λ(σ). Suppose S ∈ Λ(σ), but S ∉ Λ(σ)′. Then S = X({S_{1}, …, S_{k}}) for some {S_{1}, …, S_{k}} ⊆ Λ(σ)′. By the definition of C_{S}, and of X({S_{1}, …, S_{k}}), we have C_{S} = C_{S_{1}}C_{S_{2}} … C_{S_{k}}. Thus, for some {S_{1}, …, S_{ℓ}} ⊆

Λ(σ)′, and some C′′ ∈ ∏_{χ∉X(ρ)} R_{χ}(σ), we have C_{B′} = C_{S_{1}} … C_{S_{ℓ}} · C′′. Therefore,

if we define

R′(σ) = ⟨C_{S}|S ∈ Λ(σ)′}⟩,

then we have seen that

R = (∏_{χ} R_{χ}(σ)) · R′(σ).

Moreover, by considering central characters, this is a direct product. Therefore,

R ≃ ℤ_{2}^{d_{1}} × ( ∏_{χ∉X(ρ)} ℤ_{2}^{d_{χ}−1}) × ℤ_{2}^{|Λ(σ)′|},

as claimed. □

Remark. If G = GU_{2n}, then N_{ε}F^{×} = N_{E/F} E^{×} is of index two in F^{×}, and so the local class field theory character ω_{E/F} is the only non-trivial element of (F^{×}/N_{ε}F^{×})^{∧}. Therefore, the group R′ is trivial, and R ≃ ℤ_{2}^{d}, with d = d_{1}, or d_{1} + d_{ω_{E/F}} − 1 depending on whether or not d_{ω_{E/F}} = 0 or not. We note the similarity between the R-group structure for GU_{2n} and that for SO_{2n}. We will see that the theory of elliptic representations for GU_{2n} also parallels that that of SO_{2n}.

Section 3 Elliptic Representations.

We now describe the tempered elliptic representations of G. We first show that the 2–cocycle η of R described in Section 1 is trivial. We use an idea from [26].

Lemma 3.1. Let G = GSp_{2n}, GO_{n}, or GU_{n}. Suppose M is a Levi subgroup of G and σ ∈ E_{2}(M). Denote by R the R-group associated to i_{G,M}(σ). Then the commuting algebra C(σ) is isomorphic to ℂ[R].

Proof. Suppose that G ≠ GO_{2n}. Let G_{1}(n) = {g ∈ G(n)|λ(g) = 1}^{∘}. That is, G_{1}(n) is the symplectic, special orthogonal, or quasi-split unitary group associated to the form J. Let M_{1} be the Levi component of G_{1}(n) given by M_{1} = M ∩ G_{1}(n). If

M = GL_{m_{1}} × ⋯ × GL_{m_{r}} × G(m),

then

M_{1} = GL_{m_{1}} × ⋯ × GL_{m_{r}} × G_{1}(m),

Let P_{1} = M_{1}N. Suppose that τ is an irreducible subrepresentation of σ|_{M_{1}}. Then τ ≃ σ_{1} ⊗ ⋯ ⊗ σ_{r} ⊗ ρ_{1}, for some irreducible component ρ_{1} of ρ|_{G_{1}(m)}. Let A(ν, τ, w) and A(ν, σ, w) be the standard intertwining operators attached to either σ and τ, respectively. [19,25,28]. Since N ⊂ G_{1}(n), (A(ν, σ, w)f)|_{G_{1}} = A(ν, τ, w)f|_{G_{1}} for any f ∈ Ind_{P}^{G}(τ). Thus, for each reduced root β, we have μ_{β}(σ) = μ_{β}(τ). We let A′(τ, w) and A′(σ, w) be the normalized self intertwining

R-GROUPS AND ELLIPTIC REPRESENTATIONS FOR SIMILITUDE GROUPS 17

operators (see [10,15]). Then these operators satisfy

A′(σ, w_{1}w_{2}) = η(w_{1}, w_{2})A′(σ, w_{1})A′(σ, w_{2})

for each w_{1}, w_{2} ∈ R(σ), and

A′(τ, w_{1}w_{2}) = η(w_{1}, w_{2})A′(τ, w_{1})A′(τ, w_{2})

for w_{1}, w_{2} ∈ R(τ).

Note that each element of R can be represented by an element in G_{1}. Suppose that C_{B} ∈ R. Then for each i ∈ B, σ_{i} ≃ σ_{i}^{ε}. Thus, by the results of [7] and [6], we have C_{B} ∈ W (τ). If C_{B}β < 0, then μ_{β}(σ) ≠ 0, and thus μ_{β}(τ) ≠ 0. Consequently, C_{B} ∈ R(τ), i.e., R(σ) ⊂ R(τ). Now let w_{1}, w_{2} ∈ R(σ), and f ∈ Ind_{P}^{G}(σ), and f_{1} = f|_{G_{1}}. Then

η(w_{1}, w_{2})A′(σ, w_{1})A′(σ, w_{2})f|_{G_{1}} = A′(w_{1}w_{2}, σ)f|G_{1}.

Thus,

η(w_{1}, w_{2})A′(w_{1}, τ)A′(w_{2}, τ)f_{1} = A′(w_{1}w_{2}, τ)f_{1}.

On the other hand, the results of [6,15] show that

A′(w_{1}, τ)A′(w_{2}, τ) = A′(w_{1}w_{2}, τ),

and thus η(w_{1}, w_{2}) = 1.

We are left to prove the lemma for G = GO_{2n}. The problem with the above argument is that there may be R-group elements which are not elements of the Weyl group attached to the corresponding parabolic subgroup of SO_{2n}. We can remedy this by looking at restriction to M_{1} as above, and inducing to O_{2n}. Let G_{1} = SO_{2n}, and take M_{1} = M∩G_{1}, and P_{1} = P∩G_{1} as above. Let G_{2} = O_{2n}. If m > 0, then for each i with m_{i} odd, we choose the representative for C_{i} as in [7]. Then C_{i} also represents an element of W (G_{1}, A_{1}), where this is the obvious Weyl group. Suppose σ ∈ E_{2}(M), and τ is an irreducible subrepresentation of σ|_{M_{1}}. In [8] we examine the decomposition of the representation

Ind_{P_{1}}^{G_{2}}(τ) = Ind_{G_{1}}^{G_{2}} (Ind_{P_{1}}^{G_{1}} (τ)) .

18 DAVID GOLDBERG

In [1], Arthur extends the definition of the unnormalized intertwining operators A(ν, τ, w) to the case of disconnected groups whose component group is cyclic. Consider these operators for G_{2}. Since N ⊂ G_{2}, this integral operator is again the same one which gives the unnormalized operators A(ν, σ, w) for GO_{2n}. In the general case, Arthur did not prove the analytic continuation of these operators. However, since A(ν, σ, w) can be analytically continued, and A(ν, σ, w)f|_{G_{2}} = A(ν, τ, w)(f|_{G_{2}}), we see that indeed the operators for G_{2} can be analytically continued.

Let C_{B} ∈ R. Then C_{B} can be represented in G_{2}, and is a representative for an element of W (G_{2}, A_{1}). Moreover, from the results of [7,8], C_{B}τ ≃ τ. We claim that C_{B} ∈ R_{G_{2}}(τ), where this last object is the R-group given in [8] which determines the structure of Ind_{P_{1}}^{G_{2}}(τ). It is enough to show that if i ∈ B, then μ_{γ_{i}}(τ) ≠ 0. If C_{i} represents an element of W (G_{1}, A_{1}), then the argument used for GO_{2n+1}, GSp_{2n}, and GU_{n} above shows that μ_{γ_{i}}(τ) = μ_{γ_{i}}(σ). Thus, we are reduced to the case where m_{i} is odd, and m = 0. Then, since W (M_{γ_{i}}, A_{1}) = {1}, we have μ_{γ_{i}}(τ) ≠ 0, as pointed out in [7]. Thus, C_{B} ∈ R_{G_{2}}(σ).

Now, the operators A′(w, σ) restrict to elements A′(τ, w) of the commuting algebra of Ind_{P_{1}}^{G_{2}}(τ). Thus, the cocycle η also determines the composition of these operators. However, these operators can also be defined using operators which intertwine τ and wτ, and η is also the cocycle that arises in this way. The argument given in Proposition 2.3 of [15] shows that, we can choose T′_{w} for each w in R_{G_{2}}(τ), satisfying T′_{w}τ = wτT′_{w}, with T′_{w_{1}w_{2}} = T′_{w_{1}}T′_{w_{2}}. But this shows that

A′(w_{1}w_{2}, τ) = A′(w_{1}, τ)A′(w_{2}, τ),

and therefore η ≡ 1. This completes the lemma. □

Corollary 3.2. For any σ ∈ E_{2}(M), the induced representation i_{G,M}(σ) decomposes as a direct sum of |R| inequivalent representations. □

Proposition 3.3. Let G = GSp_{2n}, GO_{n}, or GU_{n}. Suppose M is a Levi subgroup of G, with M ≃ GL_{m_{1}} × … × GL_{m_{r}} × G(m). Let σ ∈ E_{2}(M) with

R-GROUPS AND ELLIPTIC REPRESENTATIONS FOR SIMILITUDE GROUPS 19

σ ≃ σ_{1} ⊗ … ⊗ σ_{r} ⊗ ρ. Suppose R is the R-group attached to i_{G,M}(σ). Then the following are equivalent:

(a) i_{G,M}(σ) has an elliptic constituent;

(b) all the constituent of i_{G,M}(σ) are elliptic;

(c) C_{1} … C_{r} ∈ R.

Proof. By Lemma 3.1, and Theorems 1.3 and 2.6, conditions (a) and (b) are equivalent. Furthermore, both are equivalent to 𝔞_{w} = 𝔷 = {0} for some w ∈ R. Note that

𝔞 = {diag{x_{1}I_{m_{1}}, …, x_{r}I_{m_{r}}, 0_{k}, −x_{r}I_{m_{r}}, …, −x_{1}I_{m_{1}}}|x_{i} ∈ ℝ},

where k = 2m or 2m+1 as appropriate. We may denote an element of 𝔞 by x = (x_{1}, …, x_{r}). We further note that C_{i}: (x_{1}, …, x_{i}, …, x_{r}) ↦ (x_{1}, …, −x_{i}, …, x_{r}). Thus, if C = C_{B}, we have 𝔞_{C} = {(x_{1}, …, x_{r})|x_{i} = 0, ∀i ∈ B}. Consequently, 𝔞_{C} = {0} if and only if B = {1, …, r}. □

We now give more explicit criteria for i_{G,M}(σ) to have elliptic components. In order to do this, we need to consider three cases. The first two are handled in the next result, and are similar to the results of [15]. We use the same type of arguments found there.

Theorem 3.4.

(a) If G = GO_{2n+1} or GU_{2n+1}, then the following hold:

(i) Conditions (a)–(c) of Proposition 3.3 hold if and only if R ≃ ℤ_{2}^{r}.

(ii) If π ∈ E_{t}(G), then either π ∈ E_{e}(G), or there is a proper Levi subgroup M′ of G and some τ′ ∈ E_{e}(M′) with π = i_{G,M′}(τ).

(b) If G = GU_{2n} then the following hold:

(i) Conditions (a)–(c) of Proposition 3.3 hold if and only if d = r − 1 and d_{ω_{E/F}} > 0 is even, or d = r and d_{ω_{E/F}} = 0.

(ii) If d < r − 1, or d = r − 1 and d_{ω_{E/F}} = 1, then each irreducible subrepresentation π of i_{G,M}(σ) is of the form π = i_{G,M′}(τ), for

20 DAVID GOLDBERG

some proper Levi subgroup M′ of G, and some τ ∈ E_{t}(M′). We can choose M′ and τ with τ ∈ E_{e}(M′) if and only if d_{ω_{E/F}} is even or d_{ω_{E/F}} = 1.

(iii) Suppose d = r − 1, d_{ω_{E/F}} ≥ 3 is odd, and π is an irreducible subrepresentation of i_{G,M}(σ). Then π cannot be irreducibly induced from a tempered representation of a proper Levi subgroup.

Proof. (a) From Lemma 2.5, we see that if G = GO_{2n+1} or GU_{2n+1}, then C_{1} … C_{r} ∈ R if and only if R = ⟨C_{i}|1 ≤ i ≤ r⟩ ≃ ℤ_{2}^{r}. Thus, (i) holds. Further, Lemma 2.5 implies that for any σ ∈ E_{2}(M), there is a subset B_{σ} of {1, 2, …, r} with R = ⟨C_{i}|i ∈ B_{σ}⟩. Let C = C_{B_{σ}}. Then

𝔞_{R} = {(x_{1}, …, x_{r})|x_{i} = 0, ∀i ∈ B_{σ}} = 𝔞_{C}.

Thus, Theorem 1.5 implies (ii).

(b) Let d_{2} = d_{ω_{E/F}} . By the remark following Theorem 2.6, we know that R ≃ ℤ_{2}^{d_{1}+d_{2}−1} if d_{2} > 0, and R ≃ ℤ_{2}^{d_{1}} if d_{2} = 0. So, if d_{2} = 0, we have C_{1} … C_{r} ∈ R if and only if C_{i} ∈ R for each 1 ≤ i ≤ r, which is equivalent to R ≃ ℤ_{2}^{r}. On the other hand, if d_{2} > 0, then the longest element of R consists of d_{1} + 2[\frac{d_{2}}{2} ] block sign changes. Hence, if d_{2} is odd, it is impossible for C_{1} … C_{r} ∈ R. On the other hand, if d_{2} is even, then we see that C_{1} … C_{r} ∈ R if and only if d_{1} + d_{2} = r, which is equivalent to R ≃ ℤ_{2}^{r−1}. This proves (i).

We note that if d < r −1, then there is some i ∈ {1, …, r} for which C_{i}C_{B} ∉ R for all B ⊆ {1, …, i − 1, i + 1, …, r}. Thus, for each w ∈ R, the element E_{i} = (0, …, 1, …, 0) is in 𝔞_{w}. (Here the 1 is in the i-th coordinate.) Thus, E_{i} ∈ 𝔞_{R}, which implies 𝔞_{R} ≠ {0}. Therefore, by Theorem 1.5, every irreducible subrepresentation π of i_{G,M}(σ) is of the form π = i_{G,M′}(τ) for some M′ ⊊ G and τ ∈ E_{t}(M′).

Suppose now that d_{2} = d_{ω_{E/F}} is even. Let I_{2}(σ) = I_{ω_{E/F}} (σ). Then

C_{0} = ∏_{i∈I_{2}(σ)} C_{i}

R-GROUPS AND ELLIPTIC REPRESENTATIONS FOR SIMILITUDE GROUPS 21

is in R. So if B = I_{1}(σ) ∪ I_{2}(σ), we have C_{B} ∈ R, and 𝔞_{C_{B}} = 𝔞_{R}. Therefore, if d < r − 1 and d_{2} is even, then we can choose (M′, τ), with τ ∈ E_{e}(M′).

Suppose that d_{2} = 1. Then ω_{E/F} ∉ X(ρ). If ω_{1} = ω_{E/F} , then C_{1}C_{B} ∉ R, for all B ⊂ {2, …, r}. Let C = C_{I_{1}(σ)}. Then 𝔞_{R} = 𝔞_{C}, and therefore we can choose (M′, τ) and τ ∈ E_{e}(M′), with π = i_{G,M′}(τ).

Now suppose d_{2} ≥ 3 is odd. Without loss of generality, we assume that I_{1}(σ) = {k+1, k+2, …, k+d_{1}}, and I_{2}(σ) = {k+d_{1}+1, …, r−1, r}, with k = r−(d_{1}+d_{2}). Then 𝔞_{R} = {(x_{1} … x_{r})|x_{k+1} = x_{k+2} = … = x_{r} = 0}. Since C_{k+1} … C_{r} ∉ R, we see that 𝔞_{R} ≠ 𝔞_{w} for any w ∈ R. Therefore, π can not be of the form i_{G,M′}(τ) for some M′ ⊊ G and τ ∈ E_{e}(M′). □

Definition 3.5. Let G = GSp_{2n} or GO_{2n}. Suppose σ ≃ σ_{1} ⊗ ⋯ ⊗ σ_{r} ⊗ ρ ∈ E_{2}(M). We set

O(G, σ) = {χ ∈ (F^{×}/F^{×^{2}})^{∧} \ X(ρ) | d_{χ} is odd } .

We also set

O_{1}(G, σ) = {χ ∈ O(G, σ) | d_{χ} = 1, and χ ∉ S, ∀S ∈ Λ(σ)} .

Finally, we let I_{0}(σ) = I_{1}(σ) ∪ ⋃_{χ∉X(ρ)} I_{χ}(σ).

Theorem 3.6. Suppose G = GSp_{2n} or GO_{2n}. Let M ≃ GL_{m_{1}} × … × GL_{m_{r}} × G(m). Assume that σ ≃ σ_{1} ⊗ ⋯ ⊗ σ_{r} ⊗ ρ ∈ E_{2}(M) and R is the R-group of σ.

(a) i_{G,M}(σ) has elliptic constituents if and only if

(i) O(G, σ) can be partitioned into minimally ρ-trivial subsets, and

(ii) I_{0}(σ) = {1, …, r}.

(b) Suppose π ∈ E_{t}(G) is not elliptic, and π ⊆ i_{G,M}(σ). Then we can find a proper Levi subgroup M′ of G and a τ ∈ E_{t}(M′) with π = i_{G,M′}(τ) if and only if I_{0}(σ) ⊊ {1, …, r}, or O_{1}(G, σ) is nonempty. We can choose (M′, τ) with τ ∈ E_{e}(M′) if and only if O(G, σ)\O_{1}(G, σ) can be partitioned into minimally ρ-trivial subsets.

22 DAVID GOLDBERG

Proof. (a) If C_{1} … C_{r} ∈ R, then I_{0}(σ) = {1, …, r}. So condition (ii) is trivial. Suppose this is the case. For each χ ∈ O(G, σ), we choose an i_{χ} ∈ {1, …, r}, with ω_{i_{χ}} = χ. We then let

C_{χ} = ∏_{i∈I_{χ}(σ)\{i_{χ}}} C_{i}.

Since d_{χ} is odd, C_{χ} ∈ R. For each χ ∉ X(ρ) with d_{χ} even, we let

C_{χ} = ∏_{i∈I_{χ}(σ)} C_{i}.

Finally let

C_{0} = ( ∏_{i∈I_{1}(σ)} C_{i}) ∏_{χ∉X(ρ)} C_{χ}.

Then C_{0} ∈ R, and

C_{0} = ∏_{i∉{i_{χ}|χ∈O(G,σ)}} C_{i}.

Thus, C_{1} … C_{r} ∈ R if and only if C′ = ∏_{O(G,σ)} C_{i_{χ}} ∈ R. From the proof of

Theorem 2.6, we know that C′ ∈ R if and only if ∏_{O(G,σ)} ω_{i_{χ}} = ∏_{O(G,σ)} χ ∈ X(ρ).

This shows that (i) and (ii) are equivalent to C_{1} … C_{r} ∈ R.

(b) Note that

𝔞_{R} = ⋂_{w∈R} 𝔞_{w} ⊆ {(x_{1} … x_{r}) | x_{i} = 0, ∀i ∈ I_{0}(σ)} .

So in particular, if I_{0}(σ) ⊊ {1, …, r}, then 𝔞_{R} ⊋ {0}. Thus, in this case, we can find M′ and τ ∈ E_{t}(M′) as desired. Now suppose that I_{0}(σ) = {1, …, r}. If χ ∉ X(ρ) and ω_{i} = ω_{j} = χ for some i ≠ j, then C = C_{i}C_{j} ∈ R and 𝔞_{C} = {(x_{1} … x_{r})|x_{i} = x_{j} = 0} ⊆ 𝔞_{R}. Therefore, if d_{χ} ≠ 1 for all χ ∉ X(ρ) then we see that 𝔞_{R} = {0}, and so i_{G,M′}(τ) = π is impossible. So now suppose ω_{1} = χ and ω_{i} ≠ χ for all i > 1. Then there is a subset B ⊂ {2, 3, …, r} with C_{1}C_{B} ∈ R if and only if ω_{B} = χ, which is equivalent to {χ}∪{ω_{j}|j ∈ B′} ∈ Λ(σ)

R-GROUPS AND ELLIPTIC REPRESENTATIONS FOR SIMILITUDE GROUPS 23

for some B′ ⊆ {2, …, r}. Thus, if no such B′ exists then {(1, 0, …, 0)} ∈ 𝔞_{R}, and we can choose M′ and τ ∈ E_{t}(M′) with π = i_{G,M′}(τ). On the other hand, suppose that, for each χ with d_{χ} = 1, there is an S ∈ Λ(σ) with χ ∈ S. Then, for each 1 ≤ i ≤ r, there is a subset B of {1, …, i − 1, i + 1, …, r} with C_{i}C_{B} ∈ R. Thus, for each i, {(x_{1} … x_{r})|x_{i} = 0} ⊆ 𝔞_{R}, which implies 𝔞_{R} = {0}. Therefore, finding M′ and τ is impossible.

Finally, we assume that 𝔞_{R} = {(x_{1}, …, x_{k}, 0, …, 0)|x_{i} ∈ ℝ}, with k ≥ 1. We need to determine when C_{k+1} … C_{r} ∈ R. Let χ ∈ O(G, σ). If χ ∈ O_{1}(G, σ), and ω_{i_{χ}} = χ, then i_{χ} ≤ k. For all other χ ∈ O(G, σ), if i ∈ I_{χ}(σ), then we have i > k. Suppose χ ∈ O(G, σ)\O_{1}(G, σ). As in the proof of part (a), we fix i_{χ} ∈ I_{χ}(σ) and let

C_{χ} = ∏_{i∈I_{χ}(σ)\{i_{χ}}} C_{i}.

If χ ∉ O(G, σ) let

C_{χ} = ∏_{i∈I_{χ}(σ)} C_{i}.

Set

C_{0} = ∏_{i∈I_{1}(σ)} C_{i} ∏_{χ∉X(ρ)} C_{χ}.

Then C_{0} ∈ R. Since

C_{k+1} … C_{r} = C_{0} · ∏_{χ∈O(G,σ)\O_{1}(G,σ)} C_{i_{χ}},

we see that C_{k+1} … C_{r} ∈ R if and only if

∏_{O(G,σ)\O_{1}(G,σ)} χ ∈ X(ρ),

which says that O(G, σ) \ O_{1}(G, σ) can be partitioned into minimally ρ-trivial subsets. □

24 DAVID GOLDBERG

Example. We note that in all previous cases where the elliptic tempered representations were determined, the following phenomenon always held: Given a parabolic subgroup P = MN of G, there is a fixed finite group R_{M}, so that, for any σ ∈ E_{2}(M), the representation i_{G,M}(σ) has elliptic components if and only if R(σ) ≃ R_{M}. In fact, if M is a Levi subgroup that admits elliptic tempered representations, then i_{G,M}(σ) has elliptic constituents if and only if |R(σ)| = |R_{M}|. With the similitude groups, we see that such R_{M} do not exist. We look at the following example. Suppose that G = GSp_{6n} or GO_{6n} for some n, and M ≃ GL_{n} × GL_{n} × GL_{n} × GL_{1}. Let σ ≃ σ_{1} ⊗ σ_{2} ⊗ σ_{3} ⊗ ρ. In this case ρ is a one dimensional character, so ρ ⊗ ω = ρ if and only if ω = 1. We suppose that the σ_{i} are pairwise inequivalent discrete series representations of GL_{n}(F), and that σ_{i} ≃ σ̃_{i} ≃ σ_{i}^{ε} for each i. Here σ̃_{i} is the smooth contragredient of σ_{i} [3,§7]. If X_{n,0,G}(σ_{i} ⊗ ρ) holds for each i, then R ≃ ℤ_{2} × ℤ_{2} × ℤ_{2}, and i_{G,M}(σ) has eight elliptic constituents. On the other hand, if X_{n,0,G}(σ_{1} ⊗ ρ) holds, while ω_{2} = ω_{3} ≠ 1, then R =< C_{1}, C_{2}C_{3} > . Since C_{1}C_{2}C_{3} ∈ R, we see that i_{G,M}(σ) has four elliptic constituents. Finally, if the ω_{i} are all non-trivial, and ω_{3} = ω_{1}ω_{2}, then R =< C_{1}C_{2}C_{3} >, and i_{G,M}(σ) has two elliptic constituents. □

Suppose now that G is any of the similitude groups that we have been studying. Further suppose M is a Levi subgroup of G, and σ ∈ E_{2}(M) is such that all the constituents of i_{G,M}(σ) are elliptic. Let R̂ be the collection of irreducible representations of R. Since R is abelian, dim κ = 1 for all κ ∈ R̂. We let π_{κ} be the irreducible subrepresentation of i_{G,M}(σ) canonically parameterized by κ [19], and we denote by Θ_{κ}^{e} its character on the regular elliptic set of G. Then we see that

∑_{κ∈R̂} Θ_{κ}^{e} = 0.

We now make this linear dependence precise by generalizing an argument of Herb [15]. We need some preliminary results.

Lemma 3.7. Suppose i_{G,M}(σ) has elliptic constituents, and R ≃ ℤ_{2}^{d}. For 1 ≤ i ≤ r, let M_{i} be the standard Levi component of G, with split component A_{i},

R-GROUPS AND ELLIPTIC REPRESENTATIONS FOR SIMILITUDE GROUPS 25

such that M_{i} ≃ GL_{m_{i}} × G(n − m_{i}), and W (G, A_{i}) = ⟨C_{i}⟩. Then the R-group R_{i} attached to i_{M_{i},M}(σ) is of order ℤ_{2}^{d−1}.

Proof. Since i_{G,M}(σ) has elliptic constituents, C_{1} … C_{r} ∈ R, and thus Δ′ = ∅. Therefore, M_{i} satisfies Arthur’s compatibility condition, which implies R_{i} = R ∩ W (M_{i}, A). Suppose R = ⟨s_{1}, …, s_{d}⟩. We assume that s_{1} = C_{i}C_{B} for some B ⊆ {1, …, i − 1, i + 1, …, r}. We claim that it is possible to choose {s_{1}, …, s_{d}} so that s_{1} is the only generator of this form. For j > 1, we suppose that s_{j} = C_{B_{j}}. Set s′_{j} = s_{j} if i ∉ B_{j}, and s′_{j} = s_{1}s_{j} otherwise. Then {s_{1}, s′_{2}, …, s′_{d}} is a set of generators of the desired form. Therefore, R_{i} = R ∩ W (M_{i}, A) = ⟨s′_{2}, …, s′_{d}⟩ ≃ ℤ_{2}^{d−1}. □

Lemma 3.8. Suppose that M ≃ GL_{m_{1}} × ⋯ × GL_{m_{r}} × G(m). Let σ ∈ E_{2}(M), and suppose that i_{G,M}(σ) has elliptic constituents. We further suppose that R = R(σ) ≃ ℤ_{2}^{d}. Then there is a set of generators Ω = {s_{1}, …, s_{d}} for R so that, for each i, Ω \ {s_{i}} generates R_{j_{i}}, for some j_{i}.

Proof. By Lemma 3.7 we can choose a set of generators {s_{11}, …, s_{1d}} for R with R_{j_{1}} =< s_{12}, …, s_{1d} > for some j_{1}. Now suppose 1 ≤ k < d, and we have chosen a generating set Ω_{k} = {s_{k1}, …, s_{kd}} for R with the property that, for each 1 ≤ i ≤ k the set Ω_{k} \ {s_{ki}} generates R_{j_{i}} for some j_{i}. Suppose s_{k(k+1)} = C_{B}, for some B ⊂ {1, …, r}. Since

s_{k(k+1)} ∈ ⋂_{i=1}^{k} R_{j_{i}},

we know that j_{i} ∉ B for each 1 ≤ i ≤ k. Fix j_{k+1} ∈ B. Suppose that i ≠ k + 1 and s_{ki} = C_{B_{i}}. For i ≠ k + 1, we let

s_{(k+1)i} = \{ \begin{matrix}s_{ki} & if j_{k+1} ∉ B_{i} \\ s_{k(k+1)}s_{ki} & if j_{k+1} ∈ B_{i}, \end{matrix}

and take s_{(k+1)(k+1)} = s_{k(k+1)}. Set Ω_{k+1} = {s_{(k+1)1}, …, s_{(k+1)d}}. By our choice of Ω_{k}, Ω_{k+1}, and by Lemma 3.7, we see that for 1 ≤ i ≤ k, the group R_{j_{i}} is

26 DAVID GOLDBERG

generated by Ω_{k+1} \ {s_{(k+1)i}}. Moreover, we have chosen the s_{(k+1)i} = C_{D_{i}} so that j_{k+1} ∉ D_{i}, for i ≠ k +1. Thus, R_{j_{k+1}} is generated by Ω_{k+1} \{s_{(k+1)(k+1)}}. Consequently, by induction, we can choose Ω = Ω_{d} with the desired property. □

Theorem 3.9. Let G = GSp_{2n}, GO_{n}, or GU_{n}. Suppose M is a Levi subgroup of G, with M ≃ GL_{m_{1}} × … × GL_{m_{r}} × G(m). Further suppose that σ ∈ E_{2}(M) satisfies conditions (a)–(c) of Proposition 3.3. Let R = R(σ). Suppose κ ∈ R̂ and that π_{κ} is the irreducible subrepresentation of i_{G,M}(σ) parameterized by κ. Denote its character on G^{e} by Θ_{κ}^{e}. We set w_{0} = C_{1} … C_{r}, and let ε(κ) = κ(w_{0}). Then Θ_{κ}^{e} = ε(κ)Θ_{1}^{e}, where 1 is the trivial character.

Proof. By Lemma 3.8, we can choose a set of generators Ω = {s_{1}, …, s_{d}} for R with the property that Ω \ {s_{i}} generates R_{j_{i}} for some j_{i}. We fix a choice of such an Ω. Let κ ∈ R̂. Denote by s(κ) the number of s_{i} ∈ Ω for which κ(s_{i}) = −1. If s(κ) = 0, then κ = 1, and the claim is trivially true. Assume that the statement of the theorem is true whenever s(κ) ≤ s. Suppose that s(κ) = s+1. Without loss of generality, we suppose that κ(s_{1}) = −1. We are assuming that < s_{2}, …, s_{d} >= R_{j} for some j. Let κ_{j} = κ|_{R_{j}}. If ξ ∈ R̂_{j}, then

R̂(ξ) = {χ ∈ R̂ | χ|_{R_{j}} = ξ} = {ξ^{+}, ξ^{−}},

where ξ^{±} is determined by ξ^{±}(s_{1}) = ±1. Note that κ = κ_{j}^{−}, and consequently s(κ_{j}^{+}) = s(κ) − 1 = s. Therefore, by our induction hypothesis Θ_{κ_{j}^{+}}^{e} = ε(κ_{j}^{+})Θ_{1}^{e}. Let τ_{κ_{j}} be the irreducible subrepresentation of i_{M_{j},M}(σ) parameterized by κ_{j}. Then, by Theorem 1.4, we have i_{G,M_{j}}(τ_{κ_{j}}) = π_{κ_{j}^{+}} ⊕π_{κ_{j}^{−}}, and thus, Θ_{κ}^{e} = −Θ_{κ_{j}^{+}}^{e} =

−ε(κ_{j}^{+})Θ_{1}^{e}. It is enough to show that ε(κ) = −ε(κ_{j}^{+}). To see this, write w_{0} = s_{1}w, with w ∈ R_{j}. Now we have

ε(κ) = κ(w_{0}) = κ(s_{1})κ(w) = −κ_{j}^{+}(s_{1})κ_{j}^{+}(w) = −κ_{j}^{+}(w_{0}) = −ε(κ_{j}^{+}).

Therefore, Θ_{κ}^{e} = ε(κ)Θ_{1}^{e}, and the theorem follows by induction. □

R-GROUPS AND ELLIPTIC REPRESENTATIONS FOR SIMILITUDE GROUPS 27

References

1. J. Arthur, Intertwining operators and residues I. Weighted characters, J. Funct. Anal. 84 (1989), 19–84.

2. , On elliptic tempered characters, Acta Math. 171 (1993), 73–138.

3. I. N. Bernstein and A. V. Zelevinsky, Representations of the group GL(n, F) where F is a local non-archimedean local field, Russian Math. Surveys 33 (1976), 1–68.

4. , Induced representations of reductive p-adic groups. I, Ann. Sci. École Norm. Sup. (4) 10 (1977), 441–472.

5. N. Bourbaki, Groupes et Algèbres de Lie, Chapitres 4,5, et 6, Hermann, Paris, 1968.

6. D. Goldberg, R-groups and elliptic representations for unitary groups, Proc. Amer. Math. Soc., to appear.

7. , Reducibility of induced representations for Sp(2n) and SO(n) , Amer. J. Math., to appear.

8. , Reducibility for non-connected p-adic groups with G^{∘} of prime index, Canad. J. Math., to appear.

9. , Some results on reducibility for unitary groups and local Asai L-functions, J. Reine Angew. Math. 448 (1994), 65–95.

10. , R-groups and elliptic representations for SL_{n} , Pacific J. Math. 165 (1994), 77–92.

11. D. Goldberg and F. Shahidi, Automorphic L-functions, intertwining operators, and the irreducible tempered representations of p-adic groups, to appear.

12. , On the tempered spectrum of quasi-split classical groups, preprint.

13. Harish-Chandra, Harmonic Analysis on Reductive p-adic Groups, Springer-Verlag, Notes by G. van Dijk, New York–Heidelberg–Berlin, 1970.

14. , Harmonic analysis on reductive p-adic groups, Proc. Sympos. Pure Math. 26 (1973), 167–192.

15. R. A. Herb, Elliptic representations for Sp(2n) and SO(n) , Pacific J. Math. 161 (1993), 347–358.

16. N. Jacobson, A note on hermitian forms, Bull. Amer. Math. Soc. 46 (1940), 264–268.

17. H. Jacquet, Generic representations, in Non Commutatuve Harmonic Analysis, Lecture Notes in Mathematics, no. 587, Springer-Verlag, New York–Heidelberg–Berlin, 1977, 91–101.

18. C. D. Keys, On the decomposition of reducible principal series representations of p-adic Chevalley groups, Pacific J. Math. 101 (1982), 351–388.

19. , L-indistinguishability and R-groups for quasi split groups: unitary groups in even dimension, Ann. Sci. École Norm. Sup. (4) 20 (1987), 31–64.

20. A. W. Knapp and E. M. Stein, Irreducibility theorems for the principal series, in Conference on Harmonic Analysis, Lecture Notes in Mathematics, no. 266, Springer-Verlag, New York–Heidelberg–Berlin, 1972, 197–214.

21. A. W. Knapp and G. Zuckerman, Classification of irreducible tempered representations of semisimple Lie groups, Proc. Nat. Acad. Sci. U.S.A. 73 (1976), 2178–2180.

22. W. Landherr, Äquivalenze Hermitscher formen über einem beliebigen algebraischen zahlkörper, Abh. Math. Sem. Univ. Hamburg 11 (1936), 245–248.

23. G. I. Ol’šanskiǐ, Intertwining operators and complementary series in the class of representations induced from parabolic subgroups of the general linear group over a locally compact division algebra, Math. USSR-Sb. 22 (1974), 217–254.

24. F. Shahidi, The notion of norm and the representation theory of orthogonal groups, Invent. Math., To appear.

25. , On certain L-functions, Amer. J. Math. 103 (1981), 297–355.

26. , Some results on L-indistinguishability for SL(r) , Canad. J. Math. 35 (1983), 1075–1109.

27. , Fourier transforms of intertwining operators and Placherel measures for GL(n) , Amer. J. Math. 106 (1984), 67–111.

28. , A proof of Langlands conjecture for Plancherel measures; complementary series for p-adic groups, Ann. of Math. (2) 132 (1990), 273–330.

29. , Twisted endoscopy and reducibility of induced representations for p-adic groups, Duke Math. J. 66 (1992), 1–41.

30. A. J. Silberger, The Knapp-Stein dimension theorem for p-adic groups, Proc. Amer. Math. Soc. 68 (1978), 243–246.

31. , Introduction to Harmonic Analysis on Reductive p-adic Groups, Mathematical Notes, no. 23, Princeton University Press, Princeton, NJ, 1979.

32. , The Knapp-Stein dimension theorem for p-adic groups. Correction, Proc. Amer. Math. Soc. 76 (1979), 169–170.

33. A. V. Zelevinsky, Induced representations of reductive p-adic groups II, on irreducible representations of GL(n) , Ann. Sci. École Norm. Sup. (4) 13 (1980), 165–210.

Department of Mathematics, Purdue University, West Lafayette, IN 47907

Current address: Mathematical Sciences Research Institute, Berkeley, CA 94720-5070