STEIN EXTENSIONS OF RIEMANN SYMMETRIC SPACES AND DUALITIES OF ORBITS ON FLAG MANIFOLDS

SIMON GINDIKIN* AND TOSHIHIKO MATSUKI

Abstract. It is known that K_{ℂ}-orbits S and G_{ℝ}-orbits S′ on a complex flag manifold are in one-to-one correspondence by the condition that S ∩ S′ is nonempty and compact. We may replace K_{ℂ} by some conjugate xK_{ℂ}x^{−1} so that the correspondence is preserved. We will show that this replacement is related to the domain introduced by Akhiezer and Gindikin.

1. Introduction

Let G_{ℂ} be a connected complex semisimple Lie group and G_{ℝ} a connected real form of G_{ℂ}. Let K_{ℂ} be the complexification in G_{ℂ} of a maximal compact subgroup K of G_{ℝ}. Let X = G_{ℂ}/P be a flag manifold of G_{ℂ} where P is an arbitrary parabolic subgroup of G_{ℂ}. Then there exists a natural one-to-one correspondence between the set K_{ℂ}\X of K_{ℂ}-orbits S and the set G_{ℝ}\X of G_{ℝ}-orbits S′ on X given by the condition:

(1.1) S ↔ S′ ⟺ S ∩ S′ is non-empty and compact

([M4], c.f. [MUV] for an extension to sheaf theory). It is possible to interpret this correspondence as the duality between K_{ℂ}-orbits and G_{ℝ}-orbits on flag manifolds X = G_{ℂ}/P: dual orbits have non-empty compact intersections.

It turns out that for a fixed group G_{ℝ}, this duality between its orbits with K_{ℂ}- orbits is inherited by some conjugate subgroups xK_{ℂ}x^{−1} in G_{ℂ}. In other words, let M_{ℝ} = G_{ℝ}/K be the Riemann symmetric space, canonically embedded in its complexification M_{ℂ} = G_{ℂ}/K_{ℂ}. Then in (1.1), instead of K_{ℂ}, we can, of course, take K_{ℂ}(z) the isotropy subgroup of any point z ∈ M_{ℝ}. Our aim is to find the set M̃ ⊃ M_{ℝ} of such z ∈ M_{ℂ} that (1.1) holds for K_{ℂ}(z). This problem is one of several problems about Stein extensions M̃ of Riemann symmetric spaces M_{ℝ}, on which can be extended different analytic or geometrical constructions on M_{ℝ}. There is an informal conjecture that is an universal Stein extension for many such problems which in [G] was suggested to call by complex crowns of Riemann symmetric spaces.

For every K_{ℂ}-orbit S on X = G_{ℂ}/P, define a subset

C(S) = {x ∈ G_{ℂ} | xS ∩ S′ is non-empty and compact}

of G_{ℂ} where S′ is the G_{ℝ}-orbit on X given by (1.1).

Remark 1.1. (1) Let h ∈ G_{ℝ} and k ∈ K_{ℂ}. Then we have

hxkS ∩ S′ = hxS ∩ hS′ = h(xS ∩ S′).

* The first author is partly supported by NSF grant DMS 00-70816.

Hence the sets C(S) are left G_{ℝ}-invariant and right K_{ℂ}-invariant. So we can also define a G_{ℝ}-invariant subset M̃(S) = C(S)/K_{ℂ} of M_{ℂ}

(2) Let Z denote the center of G_{ℂ}. Then the sets C(S) are Z-invariant because Z ⊂ P. Hence the sets C(S) depend only on Lie algebras. Note that the K_{ℂ}-orbit structure and G_{ℝ}-orbit structure on X depend only on Lie algebras since we assume that K_{ℂ} and G_{ℝ} are connected.

Now we can propose the following problem.

Problem 1.2. What is the set

C = ⋂ C(S) ?

Here we take the intersection for all K_{ℂ}-orbits S on X and all flag manifolds X = G_{ℂ}/P of G_{ℂ}. Clearly, M̃ = ⋂ M̃(S) = C/K_{ℂ}.

Our conjecture is that the sets C and M̃ correspond to the domain D_{0} from [AG]. This domain includes M_{ℝ} and is the maximal domain which unifies G_{ℝ}-orbits with compact isotropy subgroups. These orbits admit the next description.

Let 𝔤_{ℝ} = 𝔨 ⊕ 𝔭 denote the Cartan decomposition of 𝔤_{ℝ} = Lie(G_{ℝ}) with respect to K. Let 𝔱 be a maximal abelian subspace in i𝔭. Put

𝔱^{+} = \{Y ∈ 𝔱 | |α(Y)| < \frac{π}{2} for all α ∈ Σ\}

where Σ is the restricted root system of 𝔤_{ℂ} with respect to 𝔱. Then we can define a left G_{ℝ}-invariant right K_{ℂ}-invariant open subset in G_{ℂ} by

D̃_{0} = G_{ℝ}(exp 𝔱^{+})K_{ℂ}

and a G_{ℝ}-invariant open subset D_{0} = D̃_{0}/K_{ℂ} in M_{ℂ}.

Conjecture 1.3. C = D̃_{0}Z. (Equivalently, M̃ = ZD_{0}.)

The aim of this paper is to prove:

Theorem 1.4. C = D̃_{0}Z for all classical cases and exceptional Hermitian cases.

We will prove this theorem by case-by-case checkings in Section 2-7 and by a general trick in Section 8. We will deal with the case where G_{ℝ} is Hermitian type in Section 2, SL(n, ℝ)-case in Section 3, SO(p, q)-case in Section 4, cases of complex classical Lie groups in Section 5, GL(n, ℍ)-case in Section 6 and Sp(p, q)-case in Section 7. (The essential part in SL(n, ℝ)-case was already proved in [AG], Example 1.)

Let us explain a structure of these results. Let B denote a Borel subgroup of G_{ℂ} and let C_{B} denote the intersection of all C(S) such that S are closed K_{ℂ}-orbits on G_{ℂ}/B.

In Section 2, we assume that G_{ℝ} is Hermitian type. This case in many relations is a special one. Firstly, we consider the natural embedding

ι : M_{ℂ} → G_{ℂ}/P × G_{ℂ}/\overline{P}

([WZ]) where P and \overline{P} are the usual maximal parabolic subgroups of G_{ℂ} containing K_{ℂ}. Then we will prove in Proposition 2.2 that

ι(D_{0}) = G_{ℝ}P/P × G_{ℝ}\overline{P} /\overline{P} (≅ M_{ℝ} × \overline{M_{ℝ}}). (1.2)

Let B be a Borel subgroup of G_{ℂ} contained in P and let consider two special closed K_{ℂ}-orbits S_{1} = K_{ℂ}B/B = P/B and S_{2} = K_{ℂ}w_{0}B/B = \overline{P} w_{0}/B on X = G_{ℂ}/B where w_{0} is the longest element in the Weyl group. (The corresponding open G_{ℝ}- orbits S′_{1} and S′_{2} are used to realize holomorphic and anti-holomorphic discrete series representations of G_{ℝ}.) Then the above realization (1.2) of D_{0} clearly implies that

D̃_{0} = C(S_{1}) ∩ C(S_{2}).

The second part of the section is dedicated to the proof of

C_{B} = C(S_{1}) ∩ C(S_{2})

(Proposition 2.4) which was already proved in [WZ] for classical cases by case-by-case checkings. But we will give, in this paper, a simple general proof.

Remark 1.5. Let Q be a parabolic subgroup of G_{ℂ} containing B and π : G_{ℂ}/B → G_{ℂ}/Q the canonical projection. Then a K_{ℂ}-orbit S (resp. G_{ℝ}-orbit S′) is called to be of holomorphic type if

S = π(S_{1}) or π(S_{2}) (resp. S′ = π(S′_{1}) or π(S′_{2})).

Clearly, this definition is equal to the one in [WZ] when S′ is open in G_{ℂ}/Q. By this definition, S is automatically closed and S′ is open in G_{ℂ}/Q. By the result in [WZ], we have

M̃(S) = \begin{cases}G_{ℝ}P/K_{ℂ} & if S = π(S_{1}), \\ G_{ℝ}\overline{P} /K_{ℂ} & if S = π(S_{2}). \end{cases}

All five sections from 3 to 7 have the same structure. We first take a minimal flag manifold X_{0} such that the K_{ℂ}-orbit structure on X_{0} is nontrivial and take a closed K_{ℂ}-orbit S_{0} (S_{+} in Section 7). Then we will show that C(S_{0}) is essentially equal to D̃_{0}Z. (In Section 5, 6 and 7, they are equal. In Section 3 and 4, they are not equal in general but

C(S_{0})_{0} = D̃_{0}. (1.3)

where C(S)_{0} denote the connected component of C(S) containing the identity.) The most essential idea in these sections is that the problem is reduced to some nonstandard problems of linear algebra in a classical style as in [AG], Example 1 which we will solve by elementary calculus.

Here are samples of these problems. For SL(n, ℝ), we need to describe such (complex) nondegenerate symmetric forms m that corresponding quadrics (^{t}xmx = 0) in the projective space P^{n−1}(ℂ) have no real points. Of course this set contains the symmetric space M_{ℝ} of real positive matrixes.

For SL(n, ℂ), we need to describe such m ∈ SL(n, ℂ) that (^{t}\overline{x}mx = 0) has no real points in the complex projective space. Again hermitian positive m (Riemann symmetric space M_{ℝ}) satisfy these condition but we want to describe all of them.

One more example: for SO(p, q) we take the quadric corresponding to the canonical (p, q)-form in the complex projective space and describe (p−1)-sections on which the corresponding hermitian form will be positive. Evidently this set includes the real Grassmannian Gr_{p}(ℝ). In all cases these sets are G_{ℝ}-invariant and we describe them by a parameterization of orbits, which has a similar structure in all these and some other cases.

Next we consider full flag manifolds X = G_{ℂ}/B and show that

C(S) = D̃_{0}Z (1.4)

for all closed K_{ℂ}-orbits S on X.

In Section 8, we will show in general (Proposition 8.1) that

C = C_{B}.

So we have proved Theorem 1.4 for all classical groups and exceptional groups of Hermitian type.

Moreover we will show in Proposition 8.3 that

C(S)_{0} = C_{0} (1.5)

for the open K_{ℂ}-orbit S on X = G_{ℂ}/B for any G_{ℝ}. This result has an interesting application (see below).

In the appendix, we will discuss some geometric consideration for SO(p, q)-case: We may embed D_{0} into the Grassmann manifold X_{1} = Gr_{p,p+q} of SL(p + q, C) in this case. It turns out that the image is bigger than the expected Hermitian symmetric domain which is an open SU(p, q)-orbit on X_{1}. Though this consideration is independent of our conjecture, it seems to be important for geometric studies on D_{0}.

We have seen in (1.3), (1.4) and (1.5) that for typical orbits S we have

C(S)_{0} = D̃_{0}. (1.6)

We can also prove it for all closed K_{ℂ}-orbits S on X = G_{ℂ}/P (P is arbitrary) of nonholomorphic type (c.f. [WZ]) for all classical cases. It will be published in a subsequent paper. Of course, we assume that X is not K_{ℂ}-homogeneous. For non-Hermitian cases, every closed K_{ℂ}-orbit (open G_{ℝ}-orbit) is nonholomorphic type.

By (1.6), we expect that

Conjecture 1.6. C(S)_{0} = D̃_{0} for all K_{ℂ}-orbits S on X = G_{ℂ}/P of nonholomorphic type. (Every non-closed K_{ℂ}-orbit on X and each orbit for groups of non Hermitian type is defined to be of nonholomorphic type by Remark 1.5.)

Apparently, the new element which we need for proving of this conjecture is a proof of the inclusions C(S)_{0} ⊂ D̃_{0}. (The opposite inclusion is already proved in Theorem 1.4 for all classical cases and exceptional Hermitian cases.)

This problem (Conjecture 1.6) for different types of orbits corresponds to different geometrical problems, some of which were studied extensively. Open G_{ℝ}-orbits S′ are called by flag domains and they are connected with realizations of discrete series and some other unitary representations. Then xS are compact complex cycles and we have the problem on the parameterization of cycles in flag domains. Wolf proved that for these cases C(S) are Stein manifolds ([W]). Wolf and Zierau proved that M̃(S)_{0} ⊃ M_{ℝ} × \overline{M_{ℝ}} for Hermitian classical groups and orbits S of nonholomorphic type ([WZ]). By our computation, it seems that their proof of the opposite inclusion contains a gap.

Another interesting special case is when S is open. There is only one such orbit and it is Zariski open. Correspondingly, there is a unique closed G_{ℝ}-orbit S′ which is contained in S. The supplement S̃ to S can be defined by an polynomial equation D(g) = 0, where the function D satisfies the condition:

D(kgp) = D(g)χ(p)

for a character χ of P. (This function D is a K_{ℂ}-invariant element of a class-one finite dimensional representation of G_{ℂ} which is realized, by the Borel-Weil theorem, on the space of holomorphic cross sections of the holomorphic line bundle over G_{ℂ}/P with some strictly dominant class-one integral character of P.) In the case P = B, this condition is the obstruction to the existence of the Iwasawa decomposition on G_{ℂ}. Then we can define C(S) as

C(S) = {x | D(x^{−1}g) ≠ 0, g ∈ S′}

which is clearly a Stein manifold because C(S) is obtained by deleting from G_{ℂ} an infinite family of analytic hypersurfaces. If P = B, then we proved in Proposition 8.3 that C(S)_{0} = C_{0} and we proved C_{0} = D̃_{0} for all classical groups. Hence D_{0} = D̃_{0}/K_{ℂ} is a Stein manifold for all these groups. In such a way, we prove the conjecture from [AG] for all these groups. The functions D (let us call them determinant functions) must play essential role in the analysis on flag manifolds and symmetric spaces and deserve a special investigation. In simplest cases they reduce to the norm-functions on symmetric cones (Jordan algebras); for SU(p, q) they constructed in [BGW].

Finally, we want to explain, by the following simplest example, the relation to the structure of the double coset decomposition G_{ℝ}\G_{ℂ}/K_{ℂ} which was studied in [M5].

Example 1.7. Let G_{ℂ} = SL(2, ℂ), G_{ℝ} = SL(2, ℝ) and K_{ℂ} = SO(2, ℂ). Then the flag manifold X = G_{ℂ}/P ≅ P^{1}(ℂ) = ℂ ∪ {∞} is decomposed to K_{ℂ}-orbits as

X = S_{1} ⊔ S_{2} ⊔ S_{3}

where S_{1} = {i} and S_{2} = {−i}. X is also decomposed to G_{ℝ}-orbits as

X = S′_{1} ⊔ S′_{2} ⊔ S′_{3}

where S′_{1} = {z ∈ X | Im z > 0} and S′_{2} = {z ∈ X | Im z < 0}. By the results of [M5], we can take representatives x of the double coset decomposition G_{ℝ}\G_{ℂ}/K_{ℂ} as follows. (Then xS_{1} and xS_{2} are as in the remaining columns.)

x  xS_{1}  xS_{2}  xS_{1} ⊂ xS_{2} ⊂
a_{θ} (0 ≤ θ < \frac{π}{4} )  \{i tan (θ + \frac{π}{4} )\} \{−i cot (θ + \frac{π}{4} )\}  S′_{1}  S′_{2}
a_{π/4}  {∞}  {0}  S′_{3}  S′_{3}
a_{θ} (\frac{π}{4} < θ ≤ \frac{π}{2} ) \{i tan (θ + \frac{π}{4} )\} \{−i cot (θ + \frac{π}{4} )\}  S′_{2}  S′_{1}
b_{t}a_{π/4} (t > 0)  {i coth t}  {i tanh t}  S′_{1}  S′_{1}
b_{t}a_{π/4} (t < 0)  {i coth t}  {i tanh t}  S′_{2}  S′_{2}
c_{+}a_{π/4}  {∞}  {i}  S′_{3}  S′_{1}
c_{−}a_{π/4}  {∞}  {−i}  S′_{3}  S′_{2}
d_{+}a_{π/4}  {−i}  {0}  S′_{2}  S′_{3}
d_{−}a_{π/4}  {i}  {0}  S′_{1}  S′_{3}

Here

a_{θ} = (\begin{matrix}cos θ & i sin θ \\ i sin θ & cos θ \end{matrix}) , b_{t} = (\begin{matrix}cosh t & i sinh t \\ −i sinh t & cosh t \end{matrix}) , c_{±} = (\begin{matrix}1 & ±i \\ 0 & 1 \end{matrix}) and d_{±} = (\begin{matrix}1 & 0 \\ ±i & 1 \end{matrix}) .

By the above list, we can see

C = C(S_{1}) ∩ C(S_{2}) = ⨆_{0≤θ<π/4} G_{ℝ}a_{θ}K_{ℂ} = D̃_{0}.

On the other hand, C(S_{3}) consists of four connected components

⨆_{0≤θ<π/4} G_{ℝ}a_{θ}K_{ℂ},  ⨆_{π/4<θ≤π/2} G_{ℝ}a_{θ}K_{ℂ},  ⨆_{t>0} G_{ℝ}b_{t}a_{π/4}K_{ℂ} and ⨆_{t<0} G_{ℝ}b_{t}a_{π/4}K_{ℂ}.

Note that there are four non-closed G_{ℝ}-K_{ℂ} double cosets represented by c_{±}a_{π/4} and d_{±}a_{π/4} which have common boundary represented by a_{π/4}.

Acknowledgement: The authors are grateful to H. Ochiai who pointed out an elementary mistake in our first draft. While we were correcting it, the proofs were very much improved.

2. Hermitian cases

Suppose that G_{ℝ} is simple and of Hermitian type. Then 𝔨 = Lie (K) has a onedimensional center and a nontrivial central element in i𝔨 defines a maximal parabolic subgroup

P = K_{ℂ} exp 𝔫

of G_{ℂ} where

𝔫 = ⨁_{α∈Δ_{n}^{+}} 𝔤_{ℂ}(𝔧, α)

is the nilpotent radical of Lie (P). Here 𝔧 is a maximal torus of 𝔨 and 𝔤_{ℂ}(𝔧, α) denote the root space for a root α in the root system Δ = Δ(𝔤_{ℂ}, 𝔧). Let x ↦ \overline{x} denote the conjugation of G_{ℂ} with respect to G_{ℝ}. Then we have

\overline{P} = K_{ℂ} exp \overline{𝔫} and \overline{𝔫} = ⨁_{α∈Δ_{n}^{+}} 𝔤_{ℂ}(𝔧, −α).

Note that

𝔭_{ℂ} = 𝔫 ⊕ \overline{𝔫}.

Let {β_{1}, …, β_{ℓ}} be a maximal strongly othogonal system in Δ_{n}^{+} and X_{1}, …, X_{ℓ} be nonzero elements of 𝔤_{ℂ}(𝔧, β_{1}), …, 𝔤_{ℂ}(𝔧, β_{ℓ}), respectively. Then we can define maximal abelian subspaces

𝔞 = ℝ(X_{1} + \overline{X_{1}}) ⊕ ⋯ ⊕ ℝ(X_{ℓ} + \overline{X_{ℓ}})

of 𝔭 and

𝔱 = ℝ(X_{1} − \overline{X_{1}}) ⊕ ⋯ ⊕ ℝ(X_{ℓ} − \overline{X_{ℓ}})

of i𝔭. Put A = exp 𝔞, T = exp 𝔱 and T^{+} = exp 𝔱^{+}. Though the following fact seems to be known, we will give a proof for the sake of completeness.

Lemma 2.1. (1) AP = T^{+}P.

(2) G_{ℝ}P = KT^{+}P.

Proof. (1) Let 𝔩_{ℂ} be the complex Lie subalgebra of 𝔤_{ℂ} generated by \{X_{1}, \overline{X_{1}}, X_{2}, \overline{X_{2}}, …, X_{ℓ}, \overline{X_{ℓ}}\}. Then 𝔩_{ℂ} is isomorphic to the direct sum of ℓ-copies of 𝔰l(2, ℂ). Note that the set 𝔱^{+} is characterized by the subset Δ(𝔩_{ℂ}, 𝔱) of Σ = Δ(𝔤_{ℂ}, 𝔱) because Σ is of type C_{ℓ} or BC_{ℓ} and Δ(𝔩_{ℂ}, 𝔱) is the set of longest roots in Σ. Let L_{ℂ} denote the analytic subgroup of G_{ℂ} for 𝔩_{ℂ}. Then by a computation in SL(2, ℂ), we have

A(L_{ℂ} ∩ P) = T^{+}(L_{ℂ} ∩ P).

Hence we have

AP = T^{+}P.

(2) is clear by the Cartan decomposition G_{ℝ} = KAK. q.e.d.

Consider the natural embedding

ι : G_{ℂ}/K_{ℂ} ↪ G_{ℂ}/P × G_{ℂ}/\overline{P}

given by

xK_{ℂ} ↦ (xP, x\overline{P})

([WZ]).

Proposition 2.2. ι(D_{0}) = G_{ℝ}P/P × G_{ℝ}\overline{P} /\overline{P}. (Here D_{0} = D̃_{0}/K_{ℂ}.)

Proof. By Lemma 2.1, we have

ι(T^{+}K_{ℂ}/K_{ℂ}) ⊂ G_{ℝ}P/P × G_{ℝ}\overline{P} /\overline{P}

and hence

ι(D_{0}) ⊂ G_{ℝ}P/P × G_{ℝ}\overline{P} /\overline{P} .

Conversely, let g and h be elements of G_{ℝ}. Then we have only to find an x ∈ D_{0} such that ι(x) = (gP, h\overline{P}). By G_{ℝ}-action, we have only to consider (h^{−1}gP, \overline{P}). Write h^{−1}g = k_{1}ak_{2} with k_{1}, k_{2} ∈ K and a ∈ A. Then we have only to consider (aP, \overline{P}) since

(k_{1}^{−1}h^{−1}gP, k_{1}^{−1}\overline{P}) = (aP, \overline{P}).

Furthermore we have only to consider (a^{1/2}P, a^{−1/2}\overline{P}). Let τ denote the conjugation in G_{ℂ} with respect to the compact real form U = K exp i𝔭. Then we have τ(P) = \overline{P} since P is invariant by the holomorphic involution defining K_{ℂ} and we also have τ(a^{1/2}) = a^{−1/2}. Write

a^{1/2} = tp

with some t ∈ T^{+} and p ∈ P by Lemma 2.1 (1). Then applying τ on the both sides, we get

a^{−1/2} = tτ(p).

Hence we have

ι(tK_{ℂ}) = (a^{1/2}P, a^{−1/2}\overline{P}).

Thus we have proved ι(D_{0}) = G_{ℝ}P/P × G_{ℝ}\overline{P} /\overline{P}. q.e.d.

Let Δ^{+} be a positive system of Δ(𝔤_{ℂ}, 𝔧) containing Δ_{n}^{+}. Then Δ^{+} defines a Borel subgroup B of G_{ℂ} such that B ⊂ P. Let w_{0} be the longest element (with respect to Δ^{+}) of the Weyl group W = W (𝔧). Then the two closed sets

S_{1} = P = K_{ℂ}P = K_{ℂ}B and S_{2} = \overline{P} w_{0} = K_{ℂ}\overline{P} w_{0} = K_{ℂ}(w_{0}Bw_{0}^{−1})w_{0} = K_{ℂ}w_{0}B

are contained in the open sets

S′_{1} = G_{ℝ}P = G_{ℝ}B and S′_{2} = G_{ℝ}\overline{P} w_{0} = G_{ℝ}w_{0}B,

respectively (c.f. [M2]). Let x be an element of G_{ℂ}. Then Proposition 2.2 implies that

x ∈ D̃_{0} ⟺ xS_{1} ⊂ S′_{1} and xS_{2} ⊂ S′_{2}

Let W_{K} be the subgroup of W defined by W_{K} = N_{K_{ℂ}}(𝔧)/Z_{K_{ℂ}}(𝔧). Then the set of closed K_{ℂ}-B double cosets in G_{ℂ} is parametrized by W_{K}\W ([M1], [R]).

Lemma 2.3. For w ∈ W, we have

K_{ℂ}(BwB)^{cl} ∩ K_{ℂ}w_{0}(Bw_{0}^{−1}wB)^{cl} = K_{ℂ}wB.

Proof. We can choose a w′ = w_{K}w ∈ W_{K}w such that

K_{ℂ}(BwB)^{cl} = (Bw′B)^{cl}

since P = K_{ℂ}B and (BwB)^{cl} are both expressed as products of parabolic subgoups P_{α} = B ∪ Bw_{α}B with α ∈ Ψ where Ψ denote the set of simple roots in Δ^{+}. It is known that

(Bw′B)^{cl} ∩ (B^{−}w′B)^{cl} = w′B

where B^{−} = w_{0}Bw_{0}^{−1}. So we have

K_{ℂ}(BwB)^{cl} ∩ K_{ℂ}w_{0}(Bw_{0}^{−1}wB)^{cl} = (Bw′B)^{cl} ∩ K_{ℂ}(B^{−}wB)^{cl}
= (Bw′B)^{cl} ∩ K_{ℂ}(B^{−}w′B)^{cl}
= K_{ℂ}w′B
= K_{ℂ}wB

since K_{ℂ}(B^{−}wB)^{cl} = (\overline{P} wB)^{cl} = (\overline{P} w′B)^{cl} = K_{ℂ}(B^{−}w′B)^{cl} and since w_{K} is contained in K_{ℂ}. q.e.d.

Proposition 2.4. Let x be an element of G_{ℂ}. If xS_{1} ⊂ S′_{1} and xS_{2} ⊂ S′_{2}, then xS ⊂ S′ for all closed K_{ℂ}-B double cosets S in G_{ℂ}.

Proof. Let w ∈ W. Then by Lemma 2.3, we have

S_{1}(BwB)^{cl} ∩ S_{2}(Bw_{0}^{−1}wB)^{cl} = K_{ℂ}wB.

We also have

S′_{1}(BwB)^{cl} ∩ S′_{2}(Bw_{0}^{−1}wB)^{cl} = G_{ℝ}wB

since the correspondence between K_{ℂ}-orbits and G_{ℝ}-orbits is compatible with the right P_{α}-action for α ∈ Ψ ([M2]). Since xS_{1} ⊂ S′_{1} and xS_{2} ⊂ S′_{2}, we have

xS_{1}(BwB)^{cl} ⊂ S′_{1}(BwB)^{cl} and xS_{2}(Bw_{0}^{−1}wB)^{cl} ⊂ S′_{2}(Bw_{0}^{−1}wB)^{cl}.

Hence we have

xK_{ℂ}wB ⊂ G_{ℝ}wB. q.e.d.
3. SL(n, ℝ)-case

Let G_{ℂ} = SL(n, ℂ), K_{ℂ} = SO(n, ℂ) and G_{ℝ} = SL(n, ℝ) with n ≥ 3. First consider the flag manifold X_{0} = P^{n−1}(ℂ) = (ℂ^{n} − {0})/ℂ^{×}. Then there are two K_{ℂ}-orbits and two G_{ℝ}-orbits on X_{0}. The closed K_{ℂ}-orbit is defined by

S_{0} : z_{1}^{2} + ⋯ + z_{n}^{2} = 0

on X_{0} and the open G_{ℝ}-orbit is

S′_{0} = X_{0} − P^{n−1}(ℝ).

We can take

𝔱^{+} = \{(\begin{matrix}iθ_{1} & & & & 0 \\ & . & . & . \\ 0 & & & & iθ_{n} \end{matrix}) ∈ 𝔰𝔩(n, ℂ) | |θ_{j} − θ_{k}| < \frac{π}{2} for all j and k\} .

Put

y = (\begin{matrix}e^{iπ/n}I_{n−1} & 0 \\ 0 & −e^{iπ/n} \end{matrix}) = e^{iπ/n} (\begin{matrix}I_{n−1} & 0 \\ 0 & −1 \end{matrix})

and let Y denote the subgroup of G_{ℂ} generated by y. Note that y normalizes the sets G_{ℝ}, K_{ℂ} and T^{+} and that y^{2} generates the center Z of G_{ℂ}. (If n is odd, then Y = Z.) Though the following fact was already proved in Example 1 of [AG], we will give a detailed proof here because an elementary calculus used in this proof is the key idea of this paper.

Proposition 3.1. xS_{0} ⊂ S′_{0} ⟺ x ∈ G_{ℝ}T^{+}Y K_{ℂ}.

Proof. Note that the projective cone xS_{0} ⊂ X_{0} is defined by

tzmz = 0

where m = ^{t}x^{−1}x^{−1}. So the condition xS_{0} ⊂ S′_{0} implies that

tzmz ≠ 0 for all z ∈ ℝ^{n} − {0}. (3.1)

Assume (3.1). Since F = ℝ^{n} − {0} is simply connected, we can define a smooth function ξ : F → ℝ by

ξ(z) = arg(^{t}zmz)

taking some branch of arg. Since P^{n−1}(ℝ) = F/ℝ^{×} is compact, the image R of ξ is a closed interval in ℝ. So we can take a v_{1} ∈ F such that ξ(v_{1}) is on the boundary of R and that |^{t}v_{1}mv_{1}| = 1. Let u be an element of ℝ^{n} and consider the function

η(t) = ξ(v_{1} + tu) = arg(^{t}v_{1}mv_{1} + 2t(^{t}v_{1}mu) + t^{2}(^{t}umu))

of t ∈ ℝ. Then we have

tv_{1}mu ∈ ℝ^{t}v_{1}mv_{1} for all u ∈ ℝ^{n}

since η(0) = ξ(v_{1}) is on the boundary of R. Put

U_{1} = {u ∈ ℝ^{n} | ^{t}v_{1}mu = 0}.

Then U_{1} is an n − 1-dimensional subspace of ℝ^{n} which does not contain v_{1}.

Next consider the restriction of ξ on U_{1} − {0} and take a v_{2} ∈ U_{1} − {0} such that ξ(v_{2}) is on the boundary of ξ(U_{1} − {0}) and that |^{t}v_{2}mv_{2}| = 1. Repeating this procedure, we can take basis v_{1}, …, v_{n} of ℝ^{n} such that

tv_{j}mv_{k} = 0 if j ≠ k

and that |^{t}v_{j}mv_{j}| = 1 for j = 1, …, n. Hence if we put g = (v_{1} ⋯ v_{n}) ∈ G_{ℝ}, then we have

tgmg = (\begin{matrix}λ_{1} & & & & 0 \\ & . & . & . \\ 0 & & & & λ_{n} \end{matrix})

where λ_{j} = ^{t}v_{j}mv_{j} ∈ U(1). Write z = gc with c ∈ ℝ^{n}. Then we have

tzmz = ^{t}c^{t}gmgc = λ_{1}c_{1}^{2} + ⋯ + λ_{n}c_{n}^{2}.

Considering the convex real cone in ℂ generated by λ_{1}, …, λ_{n}, we see that

(3.2) (3.1) ⟺ {λ_{1}, …, λ_{n}} ⊂ {z ∈ ℂ | Re μz > 0} for some μ ∈ ℂ.

Hence we can write

tg^{t}x^{−1}x^{−1}g = ^{t}gmg = t^{2} = ^{t}tt

with some t ∈ T^{+}Y. Thus we have

x ∈ gt^{−1}K_{ℂ} ⊂ G_{ℝ}T^{+}Y K_{ℂ}.

Converse assertion is clear by the above arguments. q.e.d.

Now consider the full flag manifold X of G_{ℂ} consisting of full flags

V_{1} ⊂ ⋯ ⊂ V_{n−1} (3.3)

where dim_{ℂ} V_{j} = j. Then such a flag is contained in a closed K_{ℂ}-orbit on X if and only if

Q(V_{j}, V_{n−j}) = {0} for all j.

Here Q( , ) denote the canonical quadratic form on ℂ^{n} defined by Q(z, w) = z_{1}w_{1} + ⋯ + z_{n}w_{n}. On the other hand, a flag (3.3) is contained in an open G_{ℝ}-orbit on X if and only if

V_{j} ∩ \overline{V_{n−j}} = \{0\} for all j.

Let S be a closed K_{ℂ}-orbit on X. Then every flag (3.3) contained in xS satisfies

Q_{x}(V_{j}, V_{n−j}) = {0} for all j (3.4)

where Q_{x}(z, w) = Q(x^{−1}z, x^{−1}w) = ^{t}zmw.

Assume (3.1). Then we can show that xS is contained in an open G_{ℝ}-orbit as follows. Let (3.3) be an element of xS and z an element of V_{j} ∩ \overline{V_{n−j}}. Then by (3.4), we have

Q_{x}(z, \overline{z}) = 0.

Write z = c_{1}v_{1} + ⋯ + c_{n}v_{n} with the basis v_{1}, …, v_{n} of ℝ^{n} given in the proof of Proposition 3.1. Then we have

0 = Q_{x}(z, \overline{z}) = ^{t}zm\overline{z} = |c_{1}|^{2}λ_{1} + ⋯ + |c_{n}|^{2}λ_{n}.

By (3.2), this implies c_{1} = ⋯ = c_{n} = 0 and z = 0. Hence xS is contained in an open G_{ℝ}-orbit on X.

Let 𝔧 be a maximal torus of 𝔨 = 𝔬(n). Then the set of closed K_{ℂ}-orbits on X are parametrized by the left coset

W_{K_{ℂ}}(𝔧)\W_{G_{ℂ}}(𝔧)

where W_{∗}(𝔧) = N_{∗}(𝔧)/Z_{∗}(𝔧) ([M1], [R]). First suppose n is odd. Then we have W_{K_{ℂ}}(𝔧) = W_{G_{ℂ}}(𝔧) (B_{[n/2]}-type) and hence X has only one closed K_{ℂ}-orbit. Since Y = Z in this case, we have G_{ℝ}T^{+}Y K_{ℂ} = D̃_{0}Z.

Next suppose that n is even. Then we have |W_{K_{ℂ}}(𝔧)\W_{G_{ℂ}}(𝔧)| = 2 since W_{K_{ℂ}}(𝔧) is D_{n/2}-type and W_{G_{ℂ}}(𝔧) is B_{n/2}-type. But since W_{O(n)}(𝔧) = W_{G_{ℂ}}(𝔧) and since y can be replaced by an element

(\begin{matrix}I_{n−1} & 0 \\ 0 & −1 \end{matrix})

of O(n), the two closed K_{ℂ}-orbits on X are written as

S and yS.

Since y^{2} ∈ Z, we have Y = Z ⊔ Zy and hence

G_{ℝ}T^{+}Y K_{ℂ} = D̃_{0}Z ⊔ D̃_{0}Zy.

Thus we have

xS ⊂ S′ and xyS ⊂ yS′ if x ∈ D̃_{0}Z

and

xS ⊂ yS′ and xyS ⊂ S′ if x ∈ D̃_{0}Zy.

Conversely if xS ⊂ S′, then we have xS_{0} ⊂ S′_{0} by a general argument as in Section 8.

By the above arguments, we get:

Proposition 3.2. x ∈ D̃_{0}Z ⟺ xS ⊂ S′ for a closed K_{ℂ}-orbit S on X.

4. SO(p, q)-case

Let Q denote the quadratic form on ℂ^{n} (n = p + q) defined by

Q(z, w) = z_{1}w_{1} + ⋯ + z_{p}w_{p} − z_{p+1}w_{p+1} − ⋯ − z_{n}w_{n}

and put

G_{ℂ} = {g ∈ SL(n, ℂ) | Q(gz, gw) = Q(z, w) for all z, w ∈ ℂ^{n}} ≅ SO(n, ℂ),
K̃_{ℂ} = {g ∈ G_{ℂ} | gV_{+} = V_{+}} = {g ∈ G_{ℂ} | gV_{−} = V_{−}}
and G̃_{ℝ} = G_{ℂ} ∩ SL(n, ℝ) = SO(p, q)

where V_{+} = ℂe_{1} ⊕ ⋯ ⊕ ℂe_{p} and V_{−} = ℂe_{p+1} ⊕ ⋯ ⊕ ℂe_{n} with the canonical basis e_{1}, …, e_{n} of ℂ^{n}. Let K_{ℂ} and G_{ℝ} denote the connected components of K̃_{ℂ} and G̃_{ℝ}, respectively, containing the identity. Let F be the cone in ℂ^{n} defined by

F = {z ∈ ℂ^{n} | Q(z, z) = 0, z ≠ 0}.

Then X_{0} = F/ℂ^{×} is a flag manifold of G_{ℂ}.

Suppose that p ≥ 3 and q ≥ 1. Then

S_{0} = (V_{+} ∩ F)/ℂ^{×}

is a closed K̃_{ℂ}-orbit on X_{0}. Since S_{0} is connected, it is also a closed K_{ℂ}-orbit on X_{0}. Put

S′_{0} = \{z ∈ F | Q(z, \overline{z}) > 0\}/ℂ^{×}.

Then S′_{0} is an open G̃_{ℝ}-orbit (G_{ℝ}-orbit) on X_{0} containing S_{0}.

Let ℓ = min(p, q) and write

Y (θ_{1}, …, θ_{ℓ}) = iθ_{1}(E_{1,n} + E_{n,1}) + ⋯ + iθ_{ℓ}(E_{ℓ,n−ℓ+1} + E_{n−ℓ+1,ℓ})

where E_{j,k} are the matrix units. Then

𝔱 = {Y (θ_{1}, …, θ_{ℓ}) | θ_{1}, …, θ_{ℓ} ∈ ℝ}

is a maximal abelian subspace of i𝔭 and the set 𝔱^{+} is written as

𝔱^{+} = \{Y (θ_{1}, …, θ_{ℓ}) | |θ_{j}| + |θ_{k}| < \frac{π}{2} for all j ≠ k\}.

Put

y = (\begin{matrix}−1 & & 0 \\ & I_{n−2} \\ 0 & & −1 \end{matrix}) .

Then we have

K̃_{ℂ} = K_{ℂ} ⊔ K_{ℂ}y and G̃_{ℝ} = G_{ℝ} ⊔ G_{ℝ}y.

Since yT^{+}y^{−1} = T^{+}, we have

G̃_{ℝ}T^{+}K̃_{ℂ} = G_{ℝ}T^{+}K_{ℂ} ⊔ G_{ℝ}T^{+}K_{ℂ}y = D̃_{0} ⊔ D̃_{0}y.

Proposition 4.1. Let x be an element of G_{ℂ}. Then

x ∈ G̃_{ℝ}T^{+}K̃_{ℂ} ⟺ xS_{0} ⊂ S′_{0}.

Proof. Suppose that xS_{0} ⊂ S′_{0}. This implies that

Q(z, \overline{z}) > 0 for any z ∈ xV_{+} ∩ F.

We will find good basis v_{1}, …, v_{p} of V = xV_{+} as follows.

Let f(z) be the restriction of the function Q(z, \overline{z}) on the quadratic hypersurface H = {z ∈ V | Q(z, z) = 1} in V. Then by the above assumption, f(z) goes to +∞ when z tends to the boundary of H. So f(z) has its minimum at some point v_{1} in H. Hence by elementary calculus, we have

v ∈ V, Q(v, v_{1}) = 0 ⟹ Re Q(v, \overline{v_{1}}) = 0.

Since Q(v, v_{1}) = 0 implies Q(iv, v_{1}) = 0, we also have

v ∈ V, Q(v, v_{1}) = 0 ⟹ Re Q(iv, \overline{v_{1}}) = 0.

Hence we get

v ∈ V, Q(v, v_{1}) = 0 ⟹ Q(v, \overline{v_{1}}) = 0.

Next consider the orthogonal compliment U_{1} of v_{1} in V and take a v_{2} ∈ U_{1} ∩ H by the same procedure. Repeating this procedure, we get basis v_{1}, …, v_{p} of V such that

(4.1) Q(v_{j}, v_{k}) = δ_{jk} and that Q(v_{j}, \overline{v_{k}}) = 0 if j ≠ k.

Write v_{j} = u_{j} + iu′_{j} with u_{j}, u′_{j} ∈ ℝ^{n}. Then the 2p-vectors u_{1}, …, u_{p}, u′_{1}, …, u′_{p} in ℝ^{n} are mutually orthogonal by (4.1) and

Q(u_{j}, u_{j}) − Q(u′_{j}, u′_{j}) = Q(v_{j}, v_{j}) = 1,
Q(u_{j}, u_{j}) + Q(u′_{j}, u′_{j}) = Q(v_{j}, \overline{v_{j}}).

If Q(v_{2}, \overline{v_{2}}) > 1, then since Q(v_{2}, \overline{v_{2}}) ≤ ⋯ ≤ Q(v_{p}, \overline{v_{p}}), we have

Q(u_{j}, u_{j}) > 0 and Q(u′_{j}, u′_{j}) > 0 for j = 2, …, p,

a contradiction to the signature of Q. Hence we have

Q(v_{2}, \overline{v_{2}}) ≤ 1.

Since v_{1} + iv_{2} ∈ F, we have

0 < Q(v_{1} + iv_{2}, \overline{v_{1}} − i\overline{v_{2}}) = Q(v_{1}, \overline{v_{1}}) + Q(v_{2}, \overline{v_{2}}) (4.2) by the assumption. So we have

Q(v_{1}, \overline{v_{1}}) > −1

and hence

Q(u_{1}, u_{1}) > 0.

Since

Q(v_{1}, \overline{v_{1}}) ≤ ⋯ ≤ Q(v_{p}, \overline{v_{p}}), (4.3)

we have

0 < Q(u_{1}, u_{1}) ≤ ⋯ ≤ Q(u_{p}, u_{p}).

Hence by the signature of Q, we have Q(u′_{p}, u′_{p}) ≤ 0 and therefore Q(u_{p}, u_{p}) ≤ 1. Hence we can write

Q(u_{j}, u_{j}) = cos^{2} θ_{j} and Q(u′_{j}, u′_{j}) = − sin^{2} θ_{j}

with

\frac{π}{2} > θ_{1} ≥ ⋯ ≥ θ_{p} ≥ 0 and θ_{1} + θ_{2} < \frac{π}{2} (4.4)

by (4.2). If q < p, then we have

θ_{q+1} = ⋯ = θ_{p} = 0

by the signature of Q. Clearly we can choose an element g of G̃_{ℝ} = SO(p, q) such that

gv_{1} = (cos θ_{1})e_{1} + (iε sin θ_{1})e_{n} and

gv_{j} = (cos θ_{j})e_{j} + (i sin θ_{j})e_{n−j+1} for j = 2, …, p (4.5)

with ε = ±1. This implies

gxV_{+} = gV = exp Y (εθ_{1}, θ_{2}, …, θ_{ℓ})V_{+}

and therefore

x ∈ g^{−1} exp Y (εθ_{1}, θ_{2}, …, θ_{ℓ})K̃_{ℂ} ⊂ G̃_{ℝ}T^{+}K̃_{ℂ}.

Conversely, suppose that x ∈ G̃_{ℝ}T^{+}K̃_{ℂ}. Then by the action of the Weyl group WK̃(T) (B_{ℓ}-type or D_{ℓ}-type, K̃ = K̃_{ℂ} ∩ G̃_{ℝ}), we have

x ∈ g^{−1} exp Y (εθ_{1}, θ_{2}, …, θ_{ℓ})K̃_{ℂ}

with some g ∈ G̃_{ℝ} and θ_{1}, …, θ_{p} satisfying the conditions in (4.4). So we can take basis v_{1}, …, v_{p} of V = xV_{+} given by (4.5) which satisfy the conditions (4.2) and (4.3). Let z be an element of V ∩ F. Take an element v in the plane spanned by z and v_{1} such that Q(v, v_{1}) = 0 and that Q(v, v) = 1. Then we can write

v = a_{2}v_{2} + ⋯ + a_{p}v_{p}

with a_{j} ∈ ℂ such that

a_{2}^{2} + ⋯ + a_{p}^{2} = 1.

We have

Q(v, \overline{v}) = |a_{2}|^{2}Q(v_{2}, \overline{v_{2}}) + ⋯ + |a_{p}|^{2}Q(v_{p}, \overline{v_{p}})
≥ (|a_{2}|^{2} + ⋯ + |a_{p}|^{2})Q(v_{2}, \overline{v_{2}})
≥ Q(v_{2}, \overline{v_{2}})

since 0 < Q(v_{2}, \overline{v_{2}}) ≤ ⋯ ≤ Q(v_{p}, \overline{v_{p}}) by (4.2) and (4.3). Since z = a(v_{1} ± iv) for some a ∈ ℂ^{×}, we have

Q(z, \overline{z}) = |a|^{2}Q(v_{1} ± iv, \overline{v_{1} ± iv})
= |a|^{2}(Q(v_{1}, \overline{v_{1}}) + Q(v, \overline{v}))
≥ |a|^{2}(Q(v_{1}, \overline{v_{1}}) + Q(v_{2}, \overline{v_{2}}))
> 0

by (4.2). q.e.d.

Now let us consider the full flag manifold. The full flag manifold X = G_{ℂ}/B consists of full flags

V_{1} ⊂ ⋯ ⊂ V_{[n/2]}

where V_{j} are j-dimensional complex linear subspaces V_{j} contained in F ∪ {0}. Such a flag is contained in a closed K̃_{ℂ}-orbit if and only if

V_{j} = (V_{j} ∩ V_{+}) ⊕ (V_{j} ∩ V_{−}) for all j = 1, …, n′

where n′ = [p/2] + [q/2]. It is clear that

p_{j} = dim(V_{j} ∩ V_{+}) and q_{j} = dim(V_{j} ∩ V_{−}) (j = 1, …, n′)

are the invariants of closed K̃_{ℂ}-orbits. Hence there are \binom{n′}{[p/2]} closed K̃_{ℂ}-orbits on X.

On the other hand, a flag is contained in an open G̃_{ℝ}-orbit if and only if the hermitian form Q(z, \overline{z}) is nondegenerate on each V_{j} (j = 1, …, n′). So the signatures (p_{j}, q_{j}) of Q(z, \overline{z}) on V_{j} (j = 1, …, n′) are the invariants of open G̃_{ℝ}-orbits on X. We can see that closed K̃_{ℂ}-orbits and open G̃_{ℝ}-orbits on X are in one-to-one correspondence by these numbers (p_{j}, q_{j}) (j = 1, …, n′).

Suppose that x ∈ G̃_{ℝ}T^{+}K̃_{ℂ}. Then we want to show that xS ⊂ S′ for all closed K̃_{ℂ}-orbits S on X. By Proposition 4.1, we have

Q(z, \overline{z}) > 0 for any z ∈ xV_{+} ∩ F.

We also have

Q(z, \overline{z}) < 0 for any z ∈ xV_{−} ∩ F.

by the same arguments as in the proof of Proposition 4.1. Let

V_{1} ⊂ ⋯ ⊂ V_{[n/2]}

be a flag contained in S. Then since x(V_{j} ∩ V_{+}) = xV_{j} ∩ xV_{+} ⊂ xV_{+} ∩ F, we have

Q(z, \overline{z}) > 0 for all z ∈ x(V_{j} ∩ V_{+}).

Similarly, we also have

Q(z, \overline{z}) < 0 for all z ∈ x(V_{j} ∩ V_{−}).

Since xV_{j} = x(V_{j} ∩ V_{+}) ⊕ x(V_{j} ∩ V_{−}) for j = 1, …, n′, the signature of Q(z, \overline{z}) on xV_{j} is (dim(V_{j} ∩ V_{+}), dim(V_{j} ∩ V_{−})) for j = 1, …, n′. Thus we have proved

xS ⊂ S′.

Now let us consider K_{ℂ}-orbits on X. Let 𝔧 be a maximal torus of 𝔨. Then the set of closed K_{ℂ}-orbits and K̃_{ℂ}-orbits on X are parametrized by the left cosets

W_{K_{ℂ}}(𝔧)\W_{G_{ℂ}}(𝔧) and WK̃_{ℂ}(𝔧)\W_{G_{ℂ}}(𝔧),

respectively, where W_{∗}(𝔧) = N_{∗}(𝔧)/Z_{∗}(𝔧). First suppose that either p or q is even. Then we can see that WK̃_{ℂ}(𝔧) is bigger than W_{K_{ℂ}}(𝔧). Hence in this case, every closed K̃_{ℂ}-orbit S on X is decomosed into two closed K_{ℂ}-orbits as

S = S_{1} ⊔ yS_{1}.

Since xS ⊂ S′, we see that

xS_{1} ⊂ S′_{1} and xyS_{1} ⊂ yS′_{1} if x ∈ D̃_{0},

and that

xS_{1} ⊂ yS′_{1} and xyS_{1} ⊂ S′_{1} if x ∈ D̃_{0}y.

On the other hand, if p and q are both odd, then we can replace y by the central element −I_{n}. So the K_{ℂ}-orbits and K̃_{ℂ}-orbits on X coincide.

Conversely, suppose that xS_{1} ⊂ S′_{1} for some closed K_{ℂ}-orbit S_{1} on X. Let

V_{1} ⊂ ⋯ ⊂ V_{[n/2]}

be a flag contained in S_{1}. Then every flag contained in S_{1} is of the form

kV_{1} ⊂ ⋯ ⊂ kV_{[n/2]}

with some k ∈ K_{ℂ} and

V_{1} ⊂ V_{+} or V_{1} ⊂ V_{−}.

If V_{1} ⊂ V_{+}, then by the property of S′_{1}, we have

Q(z, \overline{z}) is positive definite on xkV_{1} for every k ∈ K_{ℂ}

and therefore

Q(z, \overline{z}) > 0 for all z ∈ xV_{+} ∩ F

since V_{+} ∩ F is K_{ℂ}-homogeneous (p ≥ 3). Hence by Proposition 4.1, we have

x ∈ G̃_{ℝ}T^{+}K̃_{ℂ}

and

x ∈ D̃_{0}Z

by the preceding arguments. When V_{1} ⊂ V_{−} and q ≥ 3, we also get x ∈ D̃_{0}Z by the same arguments.

Summarizing the above arguments, we have:

Proposition 4.2. (1) Suppose p ≠ 2 and q ≠ 2. Then the following three conditions on x ∈ G_{ℂ} are equivalent:

(a) x ∈ D̃_{0}Z.

(b) xS ⊂ S′ for a closed K_{ℂ}-orbit S on X.

(c) xS ⊂ S′ for all closed K_{ℂ}-orbits S on X.

(2) Suppose p ≥ 3 and q = 2. Then the following three conditions on x ∈ G_{ℂ} are equivalent:

(a) x ∈ D̃_{0}Z.

(b) xS ⊂ S′ for a closed K_{ℂ}-orbit S on X consisting of flags V_{1} ⊂ ⋯ ⊂ V_{[n/2]} such that V_{1} ⊂ V_{+}.

(c) xS ⊂ S′ for all closed K_{ℂ}-orbits S on X.

Remark 4.3. When p ≥ 3 and q = 2, G_{ℝ} is a Lie group of Hermitian type. Open G_{ℝ}-orbits S′ on X in the above condition (b) are called “nonholomorphic type” in [WZ]. On the other hand, the two open G_{ℝ}-orbits S′_{1} and S′_{2} on X which we considered in Section 2 are called “holomorphic type”.

5. Cases of complex classical Lie groups

Let G be a connected complex simple Lie group and G_{ℂ} = G × G. Take a compact real form K of G and let τ be the conjugation of G with respect to K. Then

G_{ℝ} = {(g, τ(g)) | g ∈ G} ≅ G

is a real form of G_{ℂ}. We can take

K_{ℂ} = {(g, g) | g ∈ G} ≅ G.

Let X = G/B be the full flag manifold of G. Then by the correspondence (g_{1}, g_{2}) ↦ g_{1}^{−1}g_{2}, we have a natural bijection

K_{ℂ}\G_{ℂ}/B × B ≅ B\G/B.

Since the right hand side has only one closed double coset B = BeB by the Bruhat decomposition, the full flag manifold X × X of G_{ℂ} has only one closed K_{ℂ}-orbit

S = {(m, m) | m ∈ X}.

When G is classical, we can take a minimal flag manifold X_{0} = F/ℂ^{×} of G where F is given as follows.

\begin{matrix}G & F \\ SL(n, ℂ) & ℂ^{n} − \{0\} \\ SO(n, ℂ) & \{z ∈ ℂ^{n} − \{0\} | z_{1}^{2} + ⋯ + z_{n}^{2} = 0\} \\ Sp(n, ℂ) & ℂ^{2n} − \{0\} \end{matrix}

If we write X_{0} = G/P, then we have a natural bijection

K_{ℂ}\G_{ℂ}/P × P ≅ P \G/P ≅ W_{P} \W/W_{P} .

When W is ∗_{n}-type (∗ = A, B, C or D), we can see that W_{P} is ∗_{n−1}-type. Hence we have

|W_{P} \W/W_{P} | = \begin{cases}2 & if G = SL(n, ℂ), \\ 3 & if G = SO(n, ℂ) (n ≥ 5) or G = Sp(n, ℂ) (n ≥ 2), \\ 4 & if G = SO(4, ℂ). \end{cases}

The flag manifold X_{0} × X_{0} of G_{ℂ} has only one closed K_{ℂ}-orbit

S_{0} = {(m, m) | m ∈ X_{0}} = p{(z, z) | z ∈ F }

where p : F × F → X_{0} × X_{0} is the canonical projection. When G = SO(n, ℂ) with n ≥ 5, the K_{ℂ}-orbit decomposition of X_{0} × X_{0} is

X_{0} × X_{0} = S_{0} ⊔ {p(z, w) | ℂz ≠ ℂw, Q(z, w) = 0} ⊔ {p(z, w) | Q(z, w) ≠ 0}

where Q(z, w) = z_{1}w_{1} + ⋯ + z_{n}w_{n}. When G = Sp(n, ℂ) with n ≥ 2, the K_{ℂ}-orbit decomposition of X_{0} × X_{0} is

X_{0} × X_{0} = S_{0} ⊔ {p(z, w) | ℂz ≠ ℂw, ^{t}zJw = 0} ⊔ {p(z, w) | ^{t}zJw ≠ 0}

where J is the canonical alternating matrix defining G.

We will prove the following proposition in this section.

Proposition 5.1. Let G = SL(n, ℂ), SO(n, ℂ) or Sp(n, ℂ). Then the following three conditions on x ∈ G_{ℂ} are equivalent:

(i) x ∈ D̃_{0}Z, (ii) xS_{0} ⊂ S′_{0}, (iii) xS ⊂ S′. (Remark. (iii) implies (ii) by a general argument as in Section 8. Since SO(3, ℂ) and SO(4, ℂ) are locally isomorphic to SL(2, ℂ) and SL(2, ℂ) × SL(2, ℂ), respectively, we may assume n ≥ 5 for SO(n, ℂ)-case.)

Proof. Since G = SL(n, ℂ), SO(n, ℂ) or Sp(n, ℂ), we can take

τ(g) = ^{t}\overline{g}^{−1} for g ∈ G

as usual. Then the open G_{ℝ}-orbit on X_{0} × X_{0} is

S′_{0} = p\{(w, z) ∈ F × F | ^{t}\overline{z}w ≠ 0\}.

The space G_{ℂ}/K_{ℂ} is identified with G by the map

φ : (g, h)K_{ℂ} ↦ gh^{−1}.

By this identification, the left G_{ℝ}-action on G_{ℂ}/K_{ℂ} corresponds to the G_{ℝ}-action on G given by

((h,^{t}\overline{h}^{−1}), g) ↦ hg^{t}\overline{h}. (5.1)

Write x = (g_{1}, g_{2}) with g_{1}, g_{2} ∈ G. Then we have

xS_{0} = (g_{1}, g_{2})S_{0} = (g, e)S_{0} = p{(gz, z) | z ∈ F }

where g = φ(x) = g_{1}g_{2}^{−1}. So the condition xS_{0} ⊂ S′_{0} implies that

t\overline{z}gz ≠ 0 for all z ∈ F. (5.2)

A. SL(n, ℂ)-case We can take

𝔱^{+} = \{((\begin{matrix}iθ_{1} & & & & 0 \\ & . & . & . \\ 0 & & & & iθ_{n} \end{matrix}) , − (\begin{matrix}iθ_{1} & & & & 0 \\ & . & . & . \\ 0 & & & & iθ_{n} \end{matrix})) | |θ_{j} − θ_{k}| < \frac{π}{2} for all j and k\} .

(ii) ⟹ (i). Assume (5.2). Then we can diagonalize g by the G_{ℝ}-action (5.1) by an argument similar to Proposition 3.1 as follows. Since F = ℂ^{n} − {0} is simply connected, we can define a smooth function ξ : F → ℝ by

ξ(z) = arg(^{t}\overline{z}gz)

for z ∈ F. Then the image R of ξ is a bounded closed interval in ℝ since X_{0} = P^{n−1}(ℂ) = F/ℂ^{×} is compact. Take v_{1} so that ξ(v_{1}) is on the boundary of R and that |^{t}\overline{v_{1}}gv_{1}| = 1. Let u be an element of ℂ^{n} and consider the function

η(t) = ξ(v_{1} + tu) = arg(^{t}\overline{v_{1}}gv_{1} + t(^{t}\overline{v_{1}}gu +^{t}\overline{u}gv_{1}) + t^{2}(^{t}\overline{u}gu))

of t ∈ ℝ. Then we have

t\overline{v_{1}}gu +^{t}\overline{u}gv_{1} ∈ ℝ^{t}\overline{v_{1}}gv_{1} for all u ∈ ℂ^{n} (5.3)

since η(0) = ξ(v_{1}) is on the boundary of R. Put

U_{1} = \{u ∈ ℂ^{n} | ^{t}\overline{u}gv_{1} = 0\}.

Then U_{1} is an n − 1-dimensional complex subspace of ℂ^{n} which does not contain v_{1}. By (5.3), we have

t\overline{v_{1}}gU_{1} ⊂ ℝ^{t}\overline{v_{1}}gv_{1}.

Hence we have

t\overline{v_{1}}gU_{1} = \{0\}

because the left hand side is a complex subspace of ℂ.

Next consider the restriction of ξ on U_{1} − {0} and take a v_{2} ∈ U_{1} − {0} such that ξ(v_{2}) is on the boundary of ξ(U_{1} − \{0\}) and that |^{t}\overline{v_{2}}gv_{2}| = 1. Repeating this procedure, we can take basis v_{1}, …, v_{n} of ℂ^{n} such that

t\overline{v_{j}}gv_{k} = 0 if j ≠ k

and that |^{t}\overline{v_{j}}gv_{j}| = 1 for j = 1, …, n. Hence if we put h = (v_{1} ⋯ v_{n}) ∈ G, then we have

t\overline{h}gh = (\begin{matrix}λ_{1} & & & & 0 \\ & . & . & . \\ 0 & & & & λ_{n} \end{matrix})

where λ_{j} = ^{t}\overline{v_{j}}gv_{j} ∈ U(1). By (5.2), λ_{1}, …, λ_{n} satisfy the condition in (3.2). Hence we have

t\overline{h}gh ∈ φ(t)Z′ = φ(tZ)

with some t ∈ T^{+}. Here Z′ = φ(Z) is the center of G. This implies that

x = (g_{1}, g_{2}) ∈ (^{t}\overline{h}^{−1}, h)tZK_{ℂ} ⊂ G_{ℝ}T^{+}K_{ℂ}Z = D̃_{0}Z.

(i) ⟹ (ii) is clear by the above arguments.

(ii) ⟹ (iii). Elements of the full flag manifold X ×X of G_{ℂ} are pairs of sequences

(V_{1} ⊂ ⋯ ⊂ V_{n−1}, V′_{1} ⊂ ⋯ ⊂ V′_{n−1}) (5.4)

where V_{j} and V′_{j} are j-dimensional complex subspaces of ℂ^{n}. The open G_{ℝ}-orbit S′ on X ×X consists of flags (5.4) such that the canonical hermitian form is nondegenerate on the pair (V_{j}, V′_{j}) for every j = 1, …, n − 1. (i.e. For every v ∈ V_{j}, there exists a u ∈ V′_{j} such that ^{t}\overline{u}v ≠ 0.) For x = (g_{1}, g_{2}) ∈ G_{ℂ}, we can see that the set xS consists of flags of the form

(gV_{1} ⊂ ⋯ ⊂ gV_{n−1}, V_{1} ⊂ ⋯ ⊂ V_{n−1})

(g = φ(x) = g_{1}g_{2}^{−1}).

Suppose that xS_{0} ⊂ S′_{0}. Then by (5.2), ^{t}\overline{z}gz ≠ 0 for every z ∈ ℂ^{n} − \{0\}. So the canonical hermitian form is clearly nondegenerate on the pair (gV_{j}, V_{j}) for j = 1, …, n − 1. Hence we have xS ⊂ S′.

B. SO(n, ℂ)-case (n ≥ 5)

We can take

𝔱^{+} = \{(Y (θ_{1}, …, θ_{m}), −Y (θ_{1}, …, θ_{m})) | |θ_{j}| + |θ_{k}| < \frac{π}{2} if j ≠ k\}.

Here

Y (θ_{1}, …, θ_{m}) = \begin{cases}(\begin{matrix}θ_{1}J_{1} & & & & 0 \\ & . & . & . \\ 0 & & & & θ_{m}J_{1} \end{matrix}) & if n = 2m, \\ (\begin{matrix}θ_{1}J_{1} & & & & & 0 \\ & . & . & . \\ & & & & θ_{m}J_{1} \\ 0 & & & & & 1 \end{matrix}) & if n = 2m + 1 \end{cases}

with

J_{1} = (\begin{matrix}0 & 1 \\ −1 & 0 \end{matrix}) .

Let Q denote the canonical quadratic form on ℂ^{n} defined by

Q(z, w) = z_{1}w_{1} + ⋯ + z_{n}w_{n}.

Then (5.2) is written as

(5.2′)  Q(gz, \overline{z}) ≠ 0 for all z ∈ F.

(ii) ⟹ (i). Assume (5.2’). Since F = {z ∈ ℂ^{n} − {0} | Q(z, z) = 0} is simply connected, we can define a smooth function ξ_{g} : F → ℝ by

ξ_{g}(z) = arg Q(gz, \overline{z}) for z ∈ F.

Since X_{0} = F/ℂ^{×} is compact, the image R of ξ_{g} is a bounded closed interval in ℝ. If g′ = \overline{h}^{−1}gh with an h ∈ G, then we can put

ξ_{g′}(z) = ξ_{g}(hz) (5.5)

because Q(g′z, \overline{z}) = Q(\overline{h}^{−1}ghz, \overline{z}) = Q(ghz, \overline{hz}).

Let v be an element of F such that ξ_{g}(v) = θ_{1} is on the boundary of R. Replacing g by a twisted conjugate \overline{h_{0}}^{−1}gh_{0} with an element h_{0} ∈ G such that v = h_{0}(e_{1} + ie_{2}), we may assume

v = e_{1} + ie_{2},

by (5.5), where {e_{1}, …, e_{n}} is the canonical basis of ℝ^{n}.

Since Q(v, \overline{v}) = 2 ≠ 0 and since Q(gv, \overline{v}) ≠ 0, two points (ℂv, ℂ\overline{v}) and (ℂgv, ℂ\overline{v}) are in the open K_{ℂ}-orbit on X_{0} × X_{0}. So we can take an h ∈ G such that

\overline{h}ℂ\overline{v} = ℂ\overline{v} and that \overline{h}ℂv = ℂgv.

Hence

\overline{h}^{−1}ghv ∈ \overline{h}^{−1}gℂv = ℂv

and so we can write

\overline{h}^{−1}ghv = μv

with some μ ∈ ℂ^{×}. Take an element h′ of G such that

h′v = |μ|^{−1/2}v and that h′\overline{v} = |μ|^{1/2}\overline{v}.

Then we have

\overline{h′}^{−1}\overline{h}^{−1}ghh′v = \frac{μ}{|μ|}v.

Rewrite \overline{h′}^{−1}\overline{h}^{−1}ghh′ as g. Then we have

ξ_{g}(v) = θ_{1}

by (5.5) since hh′v ∈ ℂv. Hence we have

gv = \frac{μ}{|μ|}v = e^{iθ_{1}}v. (5.6)

Let u be an element of U_{1} = ℂe_{3} ⊕ ⋯ ⊕ ℂe_{n} and consider the function

η(t) = ξ_{g}(v + tu) = arg(Q(gv, \overline{v}) + tQ(gv, \overline{u}) + tQ(gu, \overline{v}) + t^{2}Q(gu, \overline{u}))

of t ∈ ℝ. Then we have

\frac{dη}{dt} (0) = 0

since η(0) = ξ_{g}(v) is on the boundary of R and since the line v + tu is tangent to F at t = 0. Since gv = e^{iθ_{1}}v, we have

Q(gv, \overline{u}) = 0.

Hence

Q(gu, \overline{v}) ∈ ℝQ(gv, \overline{v})

for all u ∈ U_{1}. Since U_{1} is a complex subspace of ℂ^{n}, this implies

Q(gu, \overline{v}) = 0

22 S. GINDIKIN AND T. MATSUKI

and therefore

g^{−1}\overline{v} ∈ ℂv ⊕ ℂ\overline{v}.

Since Q(v, g^{−1}\overline{v}) = Q(gv, \overline{v}) ≠ 0 and since g^{−1}\overline{v} ∈ F, we have

g^{−1}\overline{v} ∈ ℂ\overline{v}.

Hence

g\overline{v} = β\overline{v}

with some β ∈ ℂ^{×}. Since 2e^{iθ_{1}}β = Q(gv, g\overline{v}) = Q(v, \overline{v}) = 2, we have β = e^{−iθ_{1}}. Thus we get g\overline{v} = e^{−iθ_{1}} \overline{v}. (5.7)

By (5.6) and (5.7), g is of the form

(\begin{matrix}B_{θ_{1}} & 0 \\ 0 & g′ \end{matrix})

with g′ ∈ SO(n − 2, ℂ) where

B_{θ} = (\begin{matrix}cos θ & sin θ \\ − sin θ & cos θ \end{matrix}) .

So by induction, we can show that a twisted conjugate g_{0} of g is of the form

g_{0} = a(θ_{1}, …, θ_{m}) = \begin{cases}(\begin{matrix}B_{θ_{1}} & & & & 0 \\ & . & . & . \\ 0 & & & & B_{θ_{m}} \end{matrix} ) & if n = 2m, \\ (\begin{matrix}B_{θ_{1}} & & & & & 0 \\ & . & . & . \\ & & & & B_{θ_{m}} \\ 0 & & & & & 1 \end{matrix}) & if n = 2m + 1. \end{cases}

By the action of K = SO(n), we may assume

π ≥ |θ_{1}| ≥ ⋯ ≥ |θ_{m}| ≥ 0.

First consider the case where n is odd. Suppose |θ_{1}| + |θ_{2}| ≥ π. Then we have

Q(g_{0}(e_{1} + iαe_{3} + iβe_{n}), e_{1} − iαe_{3} − iβe_{n}) = cos θ_{1} + α^{2} cos θ_{2} + β^{2} = 0 for some (α, β) ∈ [0, 1] × [0, 1] satisfying α^{2} + β^{2} = 1 because

cos θ_{1} + cos θ_{2} ≤ 0 and cos θ_{1} + 1 ≥ 0.

Since e_{1} + iαe_{3} + iβe_{n} ∈ F, this contradicts to (5.2’). Hence

|θ_{1}| + |θ_{2}| < π

and so we have

g_{0} ∈ φ(T^{+}),

which implies (i).

Next consider the case where n is even (n = 2m ≥ 6). Suppose |θ_{1}| + |θ_{m}| ≤ π. Then we can show |θ_{1}| + |θ_{2}| < π as follows. If |θ_{1}| + |θ_{2}| ≥ π, then we have

Q(g_{0}(e_{1} + iαe_{3} + iβe_{2m−1}), e_{1} − iαe_{3} − iβe_{2m−1}) = cos θ_{1} + α^{2} cos θ_{2} + β^{2} cos θ_{m} = 0

for some (α, β) ∈ [0, 1] × [0, 1] satisfying α^{2} + β^{2} = 1 because

cos θ_{1} + cos θ_{2} ≤ 0 and cos θ_{1} + cos θ_{m} ≥ 0.

Since e_{1} + iαe_{3} + iβe_{2m−1} ∈ F, this contradicts to (5.2’). Hence

|θ_{1}| + |θ_{2}| < π

and so we have g_{0} ∈ φ(T^{+}).

On the other hand, suppose |θ_{1}| + |θ_{m}| ≥ π. Then we can show |θ_{m−1}| + |θ_{m}| > π by the same argument as above. Hence we have

g_{0} = −a(θ_{1} + π, …, θ_{m} + π) ∈ −φ(T^{+}) ⊂ φ(T^{+}Z).

(i) ⟹ (ii). Let g = εa(θ_{1}, …, θ_{m}) with

ε = ±1, |θ_{j}| + |θ_{k}| < π if j ≠ k.

Then we have only to show (5.2’). Clearly we may assume ε = 1. By the action of K = SO(n), we may assume

|θ_{1}| ≥ ⋯ ≥ |θ_{m}| ≥ 0.

Let z = z_{1}e_{1} + ⋯ + z_{n}e_{n} be an element of F. Then we have

Q(gz, \overline{z}) = ∑_{j=1}^{m}((z_{2j−1} cos θ_{j} + z_{2j} sin θ_{j})\overline{z_{2j−1}} + (−z_{2j−1} sin θ_{j} + z_{2j} cos θ_{j})\overline{z_{2j}}) + δz_{n}\overline{z_{n}}

= ∑_{j=1}^{m}(|z_{2j−1}|^{2} + |z_{2j}|^{2}) cos θ_{j} + δ|z_{n}|^{2} + 2i ∑_{j=1}^{m} Im(\overline{z_{2j−1}}z_{2j}) sin θ_{j}

where δ = n − 2m.

If |θ_{1}| < π/2, then Re Q(gz, \overline{z}) > 0. So we may assume that |θ_{1}| ≥ π/2.

Suppose Im Q(gz, \overline{z}) = 0. Since | sin θ_{1}| > | sin θ_{2}| ≥ ⋯ ≥ | sin θ_{m}| ≥ 0, we have

|Im(\overline{z_{1}}z_{2})| ≤ |Im(\overline{z_{3}}z_{4})| + ⋯ + |Im(\overline{z_{2m−1}}z_{2m})|.

On the other hand, since z ∈ F, we have

|z_{1}^{2} + z_{2}^{2}| = |z_{3}^{2} + ⋯ + z_{n}^{2}| ≤ |z_{3}^{2} + z_{4}^{2}| + ⋯ + |z_{2m−1}^{2} + z_{2m}^{2}| + δ|z_{n}^{2}|.

Since |z_{2j−1}^{2} + z_{2j}^{2}|^{2} + 4|Im(\overline{z_{2j−1}}z_{2j})|^{2} = (|z_{2j−1}|^{2} + |z_{2j}|^{2})^{2}, we get

|z_{1}|^{2} + |z_{2}|^{2} ≤ |z_{3}|^{2} + ⋯ + |z_{n}|^{2}.

Hence

Re Q(gz, \overline{z}) ≥ (|z_{1}|^{2} + |z_{2}|^{2}) cos θ_{1} + (|z_{3}|^{2} + ⋯ + |z_{n}|^{2}) cos θ_{2}
= (|z_{1}|^{2} + |z_{2}|^{2})(cos θ_{1} + cos θ_{2}) + (|z_{3}|^{2} + ⋯ + |z_{n}|^{2} − |z_{1}|^{2} − |z_{2}|^{2}) cos θ_{2}
> 0

because cos θ_{2} ≥ cos θ_{1} + cos θ_{2} > 0 and z ≠ 0. Thus the assertion is proved.

(ii) ⟹ (iii). Elements of the full flag manifold X ×X of G_{ℂ} are pairs of sequences

(V_{1} ⊂ ⋯ ⊂ V_{[n/2]}, V′_{1} ⊂ ⋯ ⊂ V′_{[n/2]}) (5.8)

where V_{j} and V′_{j} are j-dimensional complex subspaces contained in F ∪ {0}. The open G_{ℝ}-orbit S′ on X × X consists of flags (5.8) such that the canonical hermitian form is nondegenerate on the pair (V_{j}, V′_{j}) for every j = 1, …, [n/2]. So we can prove (ii) ⟹ (iii) by the same argument as in the case of SL(n, ℂ).

C. Sp(n, ℂ)-case

Define G = Sp(n, ℂ) by

G = {g ∈ GL(2n, ℂ) | ^{t}gJg = J}

where

J = (\begin{matrix}J_{1} & & & & 0 \\ & . & . & . \\ 0 & & & & J_{1} \end{matrix}) and J_{1} = (\begin{matrix}0 & 1 \\ −1 & 0 \end{matrix}) .

Then we can take

𝔱^{+} = \{(Y (θ_{1}, …, θ_{n}), −Y (θ_{1}, …, θ_{n})) | |θ_{j}| < \frac{π}{4} for j = 1, …, n\}

where

Y (θ_{1}, …, θ_{n}) = (\begin{matrix}θ_{1} & & & & & & 0 \\ & −θ_{1} \\ & & . & . & . \\ & & & & & θ_{n} \\ 0 & & & & & & −θ_{n} \end{matrix}) .

(ii) ⟹ (i). Assume (5.2). Since F = ℂ^{2n} −{0} is simply connected, we can define a smooth function ξ_{g} : F → ℝ by

ξ_{g}(z) = arg^{t}\overline{z}gz for z ∈ F.

Since X_{0} = F/ℂ^{×} is compact, the image R of ξ_{g} is a bounded closed interval in ℝ. If g′ = ^{t}\overline{h}gh with an h ∈ G, then we can put

ξ_{g′}(z) = ξ_{g}(hz). (5.9)

Let v be an element of F such that ξ_{g}(v) = θ_{1} is on the boundary of R. Replacing g by a conjugate ^{t}\overline{h_{0}}gh_{0} with an element h_{0} ∈ G such that v = h_{0}e_{1}, we may assume

v = e_{1}

by (5.9).

Since ^{t}e_{2}Je_{1} = −1 ≠ 0 and since

te_{2}Jge_{1} = −^{t}e_{1}ge_{1} = −^{t}\overline{e_{1}}ge_{1} ≠ 0,

two points (ℂe_{1}, ℂe_{2}) and (ℂge_{1}, ℂe_{2}) are in the open K_{ℂ}-orbit on X_{0} × X_{0}. So we can take an h ∈ G such that

t\overline{h}ℂe_{2} = ℂe_{2} and that ^{t}\overline{h}ℂge_{1} = ℂe_{1}.

STEIN EXTENSIONS OF RIEMANN SYMMETRIC SPACES 25

Hence we have

he_{1} = J^{−1}^{t}h^{−1}Je_{1} = −J^{−1}^{t}h^{−1}e_{2} ∈ J^{−1}ℂe_{2} = ℂe_{1}

and therefore

t\overline{h}ghe_{1} ∈ ^{t}\overline{h}gℂe_{1} = ℂe_{1}.

Write ^{t}\overline{h}ghe_{1} = μe_{1} with μ ∈ ℂ^{×} and take an element

h′ = (\begin{matrix}|μ|^{−1/2} & & 0 \\ & |μ|^{1/2} \\ 0 & & I_{2n−2} \end{matrix})

of G. Then we have

t\overline{h′} ^{t}\overline{h}ghh′e_{1} = \frac{μ}{|μ|}e_{1}.

Rewrite ^{t}\overline{h′} ^{t}\overline{h}ghh′ as g. Then we have

ξ_{g}(e_{1}) = θ_{1}

by (5.9) since hh′e_{1} ∈ ℂe_{1}. Hence we have

ge_{1} = \frac{μ}{|μ|}e_{1} = e^{iθ_{1}}e_{1}. (5.10)

Let u be an element of U_{1} = ℂe_{2} ⊕ ⋯ ⊕ ℂe_{2n} and consider the function

η(t) = ξ_{g}(e_{1} + tu) = arg(^{t}\overline{e_{1}}ge_{1} + t^{t}\overline{u}ge_{1} + t^{t}\overline{e_{1}}gu + t^{2} ^{t}\overline{u}gu)

of t ∈ ℝ. Then we have

\frac{dη}{dt} (0) = 0

since η(0) = ξ_{g}(e_{1}) is on the boundary of R. Since ge_{1} = e^{iθ_{1}}e_{1}, we have

t\overline{u}ge_{1} = 0.

Hence

t\overline{e_{1}}gu ∈ ℝ^{t}\overline{e_{1}}ge_{1}

for all u ∈ U_{1}. Since U_{1} is a complex subspace of ℂ^{n}, this implies

t\overline{e_{1}}gu = 0

and therefore

tge_{1} ∈ ℂe_{1}.

Hence we have

g^{−1}e_{2} = J^{−1}^{t}gJe_{2} = J^{−1}^{t}ge_{1} ∈ J^{−1}ℂe_{1} = ℂe_{2}.

So we can write ge_{2} = βe_{2} and we can compute β as

β = ^{t}e_{1}Jge_{2} = ^{t}e_{1} ^{t}g^{−1}Je_{2} = ^{t}(g^{−1}e_{1})Je_{2} = e^{−iθ_{1}}.

Thus we have ge_{2} = e^{−iθ_{1}}e_{2}. (5.11)

By (5.10) and (5.11), g is of the form

(\begin{matrix}e^{iθ_{1}} & & 0 \\ & e^{−iθ_{1}} \\ 0 & & g′ \end{matrix})

with g′ ∈ Sp(n−1, ℂ). So by induction, we can show that g is conjugate to a diagonal matrix

g_{0} = a(θ_{1}, …, θ_{n}) = exp Y (θ_{1}, …, θ_{n}).

Since

t\overline{z}g_{0}z = e^{iθ_{1}}|z_{1}|^{2} + e^{−iθ_{1}}|z_{2}|^{2} + ⋯ + e^{iθ_{n}}|z_{2n−1}|^{2} + e^{−iθ_{n}}|z_{2n}|^{2},

it is clear that ^{t}\overline{z}g_{0}z ≠ 0 for all z ∈ F if and only if

Re e^{iθ_{1}}, …, Re e^{iθ_{n}} are all positive or all negative.

Hence it is equivalent to

g_{0} ∈ φ(T^{+}Z).

(i) ⟹ (ii) is included in the above argument.

(ii) ⟹ (iii). Elements of the full flag manifold X ×X of G_{ℂ} are pairs of sequences

(V_{1} ⊂ ⋯ ⊂ V_{n}, V′_{1} ⊂ ⋯ ⊂ V′_{n}) (5.12)

where V_{j} and V′_{j} are j-dimensional complex subspaces on which the canonical symplectic form for G = Sp(n, ℂ) vanish. The open G_{ℝ}-orbit S′ on X × X consists of flags (5.12) such that the canonical hermtian form is nondegenerate on the pair (V_{j}, V′_{j}) for every j = 1, …, n. So we can prove (ii) ⟹ (iii) by the same argument as in the case of SL(n, ℂ). q.e.d.

6. GL(n, ℍ)-case

Let G_{ℂ} = SL(2n, ℂ), K_{ℂ} = {g ∈ G_{ℂ} | ^{t}gJg = J} ≅ Sp(n, ℂ) and G_{ℝ} = {g ∈ G_{ℂ} | J\overline{g}J^{−1} = g\} (≅ GL(n, ℍ) modulo center). Here

J = (\begin{matrix}J_{1} & & & & 0 \\ & . & . & . \\ 0 & & & & J_{1} \end{matrix}) and J_{1} = (\begin{matrix}0 & 1 \\ −1 & 0 \end{matrix}) .

As a flag manifold X_{0}, we consider the Grassmann manifold

X_{0} = {V ⊂ ℂ^{2n} | dim_{ℂ} V = 2}.

(Note that P^{2n−1}(ℂ) is K_{ℂ}-homogeneous.) There are two K_{ℂ}-orbits and two G_{ℝ}- orbits on X_{0}. The closed K_{ℂ}-orbit S_{0} on X_{0} is

S_{0} = {V ∈ X_{0} | ^{t}uJv = 0 for all u, v ∈ V }

and the open G_{ℝ}-orbit S′_{0} on X_{0} is

S′_{0} = \{V ∈ X_{0} | J\overline{V} ∩ V = \{0\}\}.

We can take

𝔱^{+} = \{(\begin{matrix}iθ_{1} & & & & & & 0 \\ & iθ_{1} \\ & & . & . & . \\ & & & & & iθ_{n} \\ 0 & & & & & & iθ_{n} \end{matrix}) ∈ 𝔰𝔩(2n, ℂ) | |θ_{j} − θ_{k}| < \frac{π}{2} for all j and k\} .

Proposition 6.1. xS_{0} ⊂ S′_{0} ⟺ x ∈ D̃_{0}Z.

Proof. Suppose that V ∈ S_{0} and xV ∈ X_{0} − S′_{0} (= the closed G_{ℝ}-orbit). Note that

xV ∈ X_{0} − S′_{0} ⟺ J\overline{xV} = xV ⟺ xV = ℂv ⊕ ℂJ\overline{v} for v ∈ xV − \{0\}.

Since x^{−1}v, x^{−1}J\overline{v} ∈ V ∈ S_{0}, we have

t(x^{−1}J\overline{v})Jx^{−1}v = ^{t}\overline{v}^{t}J^{t}x^{−1}Jx^{−1}v = 0.

Hence the condition xS_{0} ⊂ S′_{0} is equivalent to

t\overline{v}gv ≠ 0 for all v ∈ ℂ^{2n} − \{0\} (6.1)

where g = ^{t}J^{t}x^{−1}Jx^{−1}. Note that gJ and ^{t}Jg are skew symmetric.

Assume (6.1). Since ℂ^{2n}−{0} is simply connected, we can define a smooth function

ξ(v) = arg^{t}\overline{v}gv

of v ∈ ℂ^{2n}−{0}. The range R of ξ is a bounded closed interval in ℝ since P^{2n−1}(ℂ) = (ℂ^{2n} − {0})/ℂ^{×} is compact. Take a v_{1} ∈ ℂ^{2n} − {0} such that ξ(v_{1}) is on the boudary of R and that ^{t}\overline{v_{1}}gv_{1} = λ_{1} ∈ U(1). Put

U_{1} = \{u ∈ ℂ^{2n} | ^{t}\overline{v_{1}}gu = ^{t}\overline{v_{1}}gJ\overline{u} = 0\}.

Then U_{1} is a complex subspace of ℂ^{2n} such that

J\overline{U_{1}} = U_{1}. (6.2)

Since ^{t}\overline{v_{1}}gv_{1} ≠ 0 by (6.1), U_{1} is a 2n − 2-dimensional complex subspace of ℂ^{2n} satisfying

U_{1} ∩ (ℂv_{1} ⊕ ℂJ\overline{v_{1}}) = \{0\}.

Let u be an element of U_{1} and define a function

η(t) = ξ(v_{1} + tu) = arg(^{t}\overline{v_{1}}gv_{1} + t^{t}\overline{v_{1}}gu + t^{t}\overline{u}gv_{1} + t^{2} ^{t}\overline{u}gu)

of t ∈ ℝ. Since η(0) = ξ(v_{1}) is on the boundary of R and since ^{t}\overline{v_{1}}gu = 0, we have

t\overline{u}gv_{1} ∈ ℝ^{t}\overline{v_{1}}gv_{1}

for all u ∈ U_{1}. Since U_{1} is a complex subspace of ℂ^{2n}, this implies

t\overline{u}gv_{1} = 0.

By (6.2), we also have

t\overline{J\overline{u}}gv_{1} = 0.

Since ^{t}Jg is skew symmetric, we have

t\overline{J\overline{v_{1}}}gu = ^{t}v_{1} ^{t}Jgu = −^{t}u^{t}Jgv_{1} = −^{t}\overline{J\overline{u}}gv_{1} = 0.

On the other hand, since gJ is skew symmetric, we also have

t\overline{u}gJ\overline{v_{1}} = −^{t}\overline{v_{1}}gJ\overline{u} = 0

by the definition of U_{1}. Thus we have proved

t\overline{v_{1}}gu = ^{t}\overline{J\overline{v_{1}}}gu = ^{t}\overline{u}gv_{1} = ^{t}\overline{u}gJ\overline{v_{1}} = 0

for all u ∈ U_{1}.

Since gJ and ^{t}Jg are skew symmetric, we have

t\overline{v_{1}}gJ\overline{v_{1}} = 0, ^{t}\overline{J\overline{v_{1}}}gv_{1} = ^{t}v_{1} ^{t}Jgv_{1} = 0

and

t\overline{J\overline{v_{1}}}gJ\overline{v_{1}} = ^{t}v_{1} ^{t}JgJ\overline{v_{1}} = −^{t}v_{1} ^{t}gJ^{2}\overline{v_{1}} = ^{t}v_{1} ^{t}g\overline{v_{1}} = ^{t}\overline{v_{1}}gv_{1} = λ_{1}.

Hence we have

(\begin{matrix}^{t}\overline{v_{1}}gv_{1} & ^{t}\overline{v_{1}}gJ\overline{v_{1}} \\ ^{t}\overline{J\overline{v_{1}}}gv_{1} & ^{t}\overline{J\overline{v_{1}}}gJ\overline{v_{1}} \end{matrix}) = (\begin{matrix}λ_{1} & 0 \\ 0 & λ_{1} \end{matrix})

Next we take an element v_{2} ∈ U_{1} − {0} such that ξ(v_{2}) is on the boundary of ξ(U_{1} − \{0\}) and that ^{t}\overline{v_{2}}gv_{2} = λ_{2} ∈ U(1). Repeating this procedure, we can take basis v_{1}, J\overline{v_{1}}, …, v_{n}, J\overline{v_{n}} of ℂ^{2n} such that

t\overline{v_{j}}gv_{k} = ^{t}\overline{v_{j}}gJ\overline{v_{k}} = ^{t}(\overline{J\overline{v_{j}}})gv_{k} = ^{t}(\overline{J\overline{v_{j}}})gJ\overline{v_{k}} = 0 if j ≠ k

and that

(\begin{matrix}^{t}\overline{v_{j}}gv_{j} & ^{t}\overline{v_{j}}gJ\overline{v_{j}} \\ ^{t}\overline{J\overline{v_{j}}}gv_{j} & ^{t}\overline{J\overline{v_{j}}}gJ\overline{v_{j}} \end{matrix}) = (\begin{matrix}λ_{j} & 0 \\ 0 & λ_{j} \end{matrix})

for j = 1, …, n.

Let h be an element of GL(2n, ℂ) such that

he_{1} = v_{1}, he_{2} = −J\overline{v_{1}}, …, he_{2n−1} = v_{n}, he_{2n} = −J\overline{v_{n}}.

Then h is an element of G̃_{ℝ} = \{h ∈ GL(2n, ℂ) | J\overline{h}J^{−1} = h\} ≅ GL(n, ℍ) and

t\overline{h}gh = (\begin{matrix}λ_{1} & & & & & & 0 \\ & λ_{1} \\ & & . & . & . \\ & & & & & λ_{n} \\ 0 & & & & & & λ_{n} \end{matrix}) .

We can see that det h ∈ ℝ and moreover that det h > 0 since G̃_{ℝ} is connected. Since g = ^{t}J^{t}x^{−1}Jx^{−1} ∈ SL(n, ℂ), we have

(det h)^{2} = λ_{1}^{2} ⋯ λ_{n}^{2} ∈ U(1).

So we have det h = 1 which implies h ∈ G_{ℝ} and we also have

λ_{1}^{2} ⋯ λ_{n}^{2} = 1.

We have

th^{t}x^{−1}Jx^{−1}h = J^{t}\overline{h}^{t}J^{t}x^{−1}Jx^{−1}h = J^{t}\overline{h}gh = (\begin{matrix}λ_{1}J_{1} & & & & 0 \\ & . & . & . \\ 0 & & & & λ_{n}J_{1} \end{matrix}) .

We can take the pfaffian as a polynomial function of degree n on the space of skew symmetric matrices such that

Pf J = 1.

Then we have

Pf (\begin{matrix}λ_{1}J_{1} & & & & 0 \\ & . & . & . \\ 0 & & & & λ_{n}J_{1} \end{matrix}) = λ_{1} ⋯ λ_{n} = 1 (6.3)

because Pf(^{t}mJm) = 1 for any m ∈ SL(2n, ℂ).

Let c be an element of ℂ^{2n}. Then we have

t\overline{c}^{t}\overline{h}ghc = λ_{1}(|c_{1}|^{2} + |c_{2}|^{2}) + ⋯ + λ_{n}(|c_{2n−1}|^{2} + |c_{2n}|^{2}).

Hence the condition (6.1) is equivalent to the condition (3.2). So we can write

(\begin{matrix}λ_{1}J_{1} & & & & 0 \\ & . & . & . \\ 0 & & & & λ_{n}J_{1} \end{matrix}) = t^{2}z^{2}J = ^{t}(tz)Jtz

with some t ∈ T^{+} and z ∈ Z by using the condition (6.3). Hence x^{−1}hz^{−1}t^{−1} ∈ K_{ℂ} and therefore

x ∈ hz^{−1}t^{−1}K_{ℂ} ⊂ D̃_{0}Z.

Converse assertion is clear by the above arguments.

Now let us consider the full flag manifold X consisting of full flags

V_{1} ⊂ ⋯ ⊂ V_{2n−1}

where V_{j} is a j-dimensional complex subspace in ℂ^{2n}. There is only one closed K_{ℂ}- orbit S on X consisting of flags satisfying

u ∈ V_{j}, v ∈ V_{2n−j} ⟹ ^{t}uJv = 0

for all j. On the other hand, the corresponding open G_{ℝ}-orbit S′ consists of flags satisfying

J\overline{V_{j}} ∩ V_{2n−j} = \{0\}

for all j.

Suppose xS_{0} ⊂ S′_{0}. Then we can show xS ⊂ S′ as follows. Any flag contained in xS is written as

xV_{1} ⊂ ⋯ ⊂ xV_{2n−1}

with some

V_{1} ⊂ ⋯ ⊂ V_{2n−1}

in S. Let v be an element of J\overline{xV_{j}} ∩ xV_{2n−j}. Then since x^{−1}\overline{J^{−1}v} ∈ V_{j} and x^{−1}v ∈ V_{2n−j}, we have

0 = ^{t}(x^{−1}\overline{J^{−1}v})Jx^{−1}v = ^{t}\overline{v}^{t}J^{t}x^{−1}Jx^{−1}v = ^{t}\overline{v}gv.

Hence v = 0 by (6.1). Thus we have proved S_{0} ⊂ S′_{0} ⟹ xS ⊂ S′.

The assertion xS ⊂ S′ ⟹ xS_{0} ⊂ S′_{0} is proved by a general argument as in Section 8. Thus we have shown:

Proposition 6.2. xS_{0} ⊂ S′_{0} ⟺ xS ⊂ S′.

7. Sp(p, q)-case

Let G_{ℂ} = {g ∈ GL(2n, ℂ) | ^{t}gJg = J} ≅ Sp(n, ℂ) where J is defined as in Section 5 and 6. Put V_{+} = ℂe_{1} ⊕ ⋯ ⊕ ℂe_{2p} and V_{−} = ℂe_{2p+1} ⊕ ⋯ ⊕ ℂe_{n} with the canonical basis e_{1}, …, e_{2n} of ℂ^{2n} and let Q be the hermitian form on ℂ^{2n} defined by

Q(z, w) = ^{t}zI_{2p,2q}\overline{w} = z_{1}\overline{w_{1}} + ⋯ + z_{2p}\overline{w_{2p}} − z_{2p+1}\overline{w_{2p+1}} − ⋯ − z_{2n}\overline{w_{2n}}.

Then we define subgroups

K_{ℂ} = {g ∈ G_{ℂ} | gV_{+} = V_{+}} = {g ∈ G_{ℂ} | gV_{−} = V_{−}} ≅ Sp(p, ℂ) × Sp(q, ℂ)

(q = n − p) and

G_{ℝ} = {g ∈ G_{ℂ} | Q(gz, gw) = Q(z, w) for all z, w ∈ ℂ^{2n}} ≅ Sp(p, q)

of G_{ℂ}. We can take a flag manifold

X_{0} = P^{2n−1}(ℂ) = (ℂ^{2n} − {0})/ℂ^{×}

of G_{ℂ}. There are three K_{ℂ}-orbits on X_{0} consisting of two closed orbits S_{±} = (V_{±} − {0})/ℂ^{×} and the compliment S_{0}. There are also three G_{ℝ}-orbits S′_{±} and S′_{0} on X_{0} defined by the signature of Q. Let ℓ = min(p, q) and write

Y (θ_{1}, …, θ_{ℓ}) = θ_{1}(E_{2p+1,1} + E_{2p+2,2} − E_{1,2p+1} − E_{2,2p+2}) + ⋯
+ θ_{ℓ}(E_{2p+2ℓ−1,2ℓ−1} + E_{2p+2ℓ,2ℓ} − E_{2ℓ−1,2p+2ℓ−1} − E_{2ℓ,2p+2ℓ}).

Then we can take

𝔱^{+} = {Y (θ_{1}, …, θ_{ℓ}) | |θ_{j}| < π/4 for all j = 1, …, ℓ}.

Proposition 7.1. x ∈ G_{ℝ}T^{+}K_{ℂ} ⟺ xS_{+} ⊂ S′_{+} ⟺ xS_{−} ⊂ S′_{−}.

Proof. The condition xS_{+} ⊂ S′_{+} is equivalent to

(7.1) Q is positive definite on V = xV_{+}

Assume (7.1). Let H denote the hypersurface in V × V defined by

H = {(u, v) ∈ V × V | ^{t}uJv = 1}

and consider the function

f(u, v) = Q(u, u)Q(v, v)

of (u, v) ∈ V × V. Then the range of f|_{H} is of the form

[M, ∞)

with some M > 0 because we can write

\{|^{t}uJv| | u, v ∈ V, Q(u, u) = Q(v, v) = 1\} = [0, \frac{1}{M} ]

by (7.1).

Let (u_{1}, v_{1}) be an element of H such that

f(u_{1}, v_{1}) = M.

Clearly, we may choose u_{1} and v_{1} so that

Q(u_{1}, u_{1}) = Q(v_{1}, v_{1}) = \sqrt{M}.

Put λ_{1} = \sqrt{M}. By elementary calculus, we have

tu_{1}Jw = 0 ⟹ Re Q(v_{1}, w) = 0

for w ∈ ℂ^{2n}. Since w can be replaced by iw, we have

tu_{1}Jw = 0 ⟹ Im Q(v_{1}, w) = 0

and therefore

tu_{1}Jw = 0 ⟹ Q(v_{1}, w) = 0.

In the same way, we also have

tv_{1}Jw = 0 ⟹ Q(u_{1}, w) = 0.

Put U_{1} = {w ∈ V | ^{t}u_{1}Jw = ^{t}v_{1}Jw = 0}. Then the alternating form ^{t}zJw is nondegenerate on U_{1} since it is nondegenerate on V. By the same argument as above, we can take (u_{2}, v_{2}) ∈ H_{1} = (U_{1} × U_{1}) ∩ H such that f|_{H_{1}} attains its minimum at (u_{2}, v_{2}) and that

Q(u_{2}, u_{2}) = Q(v_{2}, v_{2}) = λ_{2} ≥ λ_{1}.

Repeating this procedure, we can get basis u_{1}, v_{1}, …, u_{p}, v_{p} of V satisfying

tu_{j}Jv_{j} = 1,  Q(u_{j}, u_{j}) = Q(v_{j}, v_{j}) = λ_{j},  Q(u_{j}, v_{j}) = 0

and

tu_{j}Ju_{k} = ^{t}u_{j}Jv_{k} = ^{t}v_{j}Jv_{k} = Q(u_{j}, u_{k}) = Q(u_{j}, v_{k}) = Q(v_{j}, v_{k}) = 0 if j ≠ k (7.2)

where 0 < λ_{1} ≤ ⋯ ≤ λ_{p}.

Put J′ = I_{2p,2q}J. Then we have

Q(u, J′\overline{v}) = ^{t}uJv, Q(J′\overline{u}, J′\overline{v}) = Q(v, u) and ^{t}(J′\overline{u})JJ′\overline{u} = \overline{^{t}uJv} (7.3)

for any u, v ∈ ℂ^{2n}. Put

w_{j}^{(1)} = u_{j} + J′\overline{v_{j}}, w_{j}^{(2)} = −J′\overline{w_{j}^{(1)}}, w_{j}^{(3)} = u_{j} − J′\overline{v_{j}}, w_{j}^{(4)} = −J′\overline{w_{j}^{(3)}}

for j = 1, …, p. Then by (7.2) and (7.3), we have

Q(w_{j}^{(r)}, w_{k}^{(s)}) = ^{t}w_{j}^{(r)}Jw_{k}^{(s)} = 0 if j ≠ k (7.4)

32 S. GINDIKIN AND T. MATSUKI

for every r, s = 1, 2, 3, 4. On the other hand, we have

Q(w_{j}^{(1)}, w_{j}^{(1)}) = Q(u_{j}, u_{j}) + Q(u_{j}, J′\overline{v_{j}}) + Q(J′\overline{v_{j}}, u_{j}) + Q(J′\overline{v_{j}}, J′\overline{v_{j}})
= Q(u_{j}, u_{j}) +^{t}u_{j}Jv_{j} + \overline{^{t}u_{j}Jv_{j}} + Q(v_{j}, v_{j})
= 2λ_{j} + 2,

Q(w_{j}^{(3)}, w_{j}^{(3)}) = 2λ_{j} − 2, Q(w_{j}^{(1)}, w_{j}^{(3)}) = 0 and

tw_{j}^{(1)}Jw_{j}^{(3)} = ^{t}(u_{j} + J′\overline{v_{j}})J(u_{j} − J′\overline{v_{j}}) = −2^{t}u_{j}JJ′\overline{v_{j}} = 2Q(u_{j}, v_{j}) = 0. Hence we have

(7.5) Q(w_{j}^{(r)}, w_{j}^{(s)}) = \begin{cases}2λ_{j} + 2 & if r = s = 1 or r = s = 2 \\ 2λ_{j} − 2 & if r = s = 3 or r = s = 4 \\ 0 & otherwise \end{cases}

and

(7.6) ^{t}w_{j}^{(r)}Jw_{j}^{(s)} = \begin{cases}2λ_{j} + 2 & if (r, s) = (1, 2) \\ 2λ_{j} − 2 & if (r, s) = (3, 4) \\ 0 & if r ≤ s and (r, s) ≠ (1, 2), (3, 4) \end{cases}

by (7.3).

By (7.4) and (7.5), the 4p vectors w_{j}^{(r)} are mutually orthogonal with respect to Q. Since 2λ_{j} + 2 > 2 > 0 and since the signature of Q is (2p, 2q), we have

2λ_{j} − 2 ≤ 0

and hence λ_{j} ≤ 1. Moreover if λ_{j} = 1, then

w_{j}^{(3)} = w_{j}^{(4)} = 0

since Q is negative definite on the orthogonal compliment of the 2p-dimensional space spanned by w_{1}^{(1)}, …, w_{p}^{(1)}, w_{1}^{(1)}, …, w_{p}^{(2)}. (If p > q, then we have automatically λ_{j} = 1 and w_{j}^{(3)} = w_{j}^{(4)} = 0 for j = q + 1, …, p since λ_{1} ≤ ⋯ ≤ λ_{p}.) Since 0 < λ_{1} ≤ ⋯ ≤ λ_{p} ≤ 1, we can write λ_{j} = cos^{2} θ_{j} − sin^{2} θ_{j} with

\frac{π}{4} ≥ θ_{1} ≥ ⋯ ≥ θ_{p} ≥ 0.

Then 2λ_{j} + 2 = 4 cos^{2} θ_{j} and 2λ_{j} − 2 = −4 sin^{2} θ_{j}. By (7.4), (7.5) and (7.6), we can take an element g ∈ G_{ℝ} such that

gw_{j}^{(1)} = 2 cos θ_{j}e_{2j−1}, gw_{j}^{(2)} = 2 cos θ_{j}e_{2j}

for j = 1, …, p and that gw_{j}^{(3)} = 2 sin θ_{j}e_{2p+2j−1}, gw_{j}^{(4)} = 2 sin θ_{j}e_{2p+2j} for j = 1, …, ℓ′ where ℓ′ = max{j | θ_{j} ≠ 0}. So we have

gu_{j} = \frac{1}{2}g(w_{j}^{(1)} + w_{j}^{(3)}) = cos θ_{j}e_{2j−1} + sin θ_{j}e_{2p+2j−1}
gv_{j} = \frac{1}{2}g(w_{j}^{(2)} + w_{j}^{(4)}) = cos θ_{j}e_{2j} + sin θ_{j}e_{2p+2j}

for j = 1, …, p. This implies

gxV_{+} = gV = exp Y (θ_{1}, …, θ_{ℓ})V_{+}

and therefore

x ∈ g^{−1} exp Y (θ_{1}, …, θ_{ℓ})K_{ℂ} ⊂ G_{ℝ}T^{+}K_{ℂ}.

Conversely, let v = ∑_{j=1}^{2p} c_{j}e_{j} ∈ V_{+} and x = exp Y (θ_{1}, …, θ_{ℓ}) with |θ_{j}| < π/4. Then we have

Q(xv, xv) = (|c_{1}|^{2}+|c_{2}|^{2})(cos^{2} θ_{1}−sin^{2} θ_{1})+⋯+(|c_{2p−1}|^{2}+|c_{2p}|^{2})(cos^{2} θ_{p}−sin^{2} θ_{p}) > 0

if v ≠ 0. (Here θ_{ℓ+1} = ⋯ = θ_{p} = 0 if p > q.) Hence xS_{+} ⊂ S′_{+}.

We also have x ∈ G_{ℝ}T^{+}K_{ℂ} ⟺ xS_{−} ⊂ S′_{−} by the same argument. q.e.d.

Now let us consider the full flag manifold. The full flag manifold X = G_{ℂ}/B consists of full flags

V_{1} ⊂ ⋯ ⊂ V_{n}

where V_{j} are j-dimensional complex linear subspaces V_{j} such that ^{t}zJw = 0 for all z, w ∈ V_{j}. Such a flag is contained in a closed K_{ℂ}-orbit if and only if

V_{j} = (V_{j} ∩ V_{+}) ⊕ (V_{j} ∩ V_{−}) for all j = 1, …, n.

It is clear that

p_{j} = dim(V_{j} ∩ V_{+}) and q_{j} = dim(V_{j} ∩ V_{−}) (j = 1, …, n)

are the invariants of closed K_{ℂ}-orbits. Hence there are \binom{n}{p} closed K_{ℂ}-orbits on X.

On the other hand, a flag is contained in an open G_{ℝ}-orbit if and only if the hermitian form Q is nondegenerate on each V_{j} (j = 1, …, n). So the signatures (p_{j}, q_{j}) of Q on V_{j} (j = 1, …, n) are the invariants of open G_{ℝ}-orbits on X. We can see that closed K_{ℂ}-orbits and open G_{ℝ}-orbits on X are in one-to-one correspondence by these numbers (p_{j}, q_{j}) (j = 1, …, n).

Suppose that x ∈ G_{ℝ}T^{+}K_{ℂ}. Then we want to show that xS ⊂ S′ for all closed K_{ℂ}-orbits S on X. By Proposition 7.1, Q is positive defnite on xV_{+} and negative definite on xV_{−}. Let

V_{1} ⊂ ⋯ ⊂ V_{n}

be a flag contained in S. Then Q is positive definite on xV_{j} ∩ xV_{+} = x(V_{j} ∩ V_{+}) and negative definite on xV_{j} ∩ xV_{−} = x(V_{j} ∩ V_{−}) for every j = 1, …, n. So the signature of Q on xV_{j} is (dim(V_{j} ∩ V_{+}), dim(V_{j} ∩ V_{−})) for j = 1, …, n. Thus we have proved xS ⊂ S′.

Conversely, if xS ⊂ S′ for some closed K_{ℂ}-orbit on X, then we have xS_{+} ⊂ S′_{+} or xS_{−} ⊂ S′_{−} by a general argument as in Section 8 and hence x ∈ G_{ℝ}T^{+}K_{ℂ} by Proposition 7.1.

Summarizing the above arguments, we have:

Proposition 7.2. x ∈ G_{ℝ}T^{+}K_{ℂ} ⟺ xS ⊂ S′ for a closed K_{ℂ}-orbit on X.

8. Arguments on full flag manifolds

Let B be a Borel subgroup of G_{ℂ}. Let Ψ denote the set of simple roots corresponding to B. For α ∈ Ψ, we can take a parabolic subgroup P_{α} = B ∪ Bw_{α}B of G_{ℂ}.

Proposition 8.1. Let x be an element of G_{ℂ}. If xS ⊂ S′ for all closed K_{ℂ}-B double cosets in G_{ℂ}, then

xS ∩ S′ is non-empty and closed in G_{ℂ} for all S ∈ K_{ℂ}\G_{ℂ}/P

for an arbitrary parabolic subgroup P of G_{ℂ}.

Proof. First assume that P = B. We will prove this by induction on dim S. If S ∈ K_{ℂ}\G_{ℂ}/B is not closed, then there exists an α ∈ Ψ such that

(1) SP_{α} = S ∪ S_{1} ∪ S_{2}, dim S_{1} = dim S_{2} = dim S − 2, or

(2) SP_{α} = S ∪ S_{1}, dim S_{1} = dim S − 2.

Consider the case (1). (The case (2) is easier than (1).) It is known ([M3], [Sp]) that the closure S_{1}^{cl} of S_{1} is written as

S_{1}^{cl} = S_{0}P_{α_{1}} ⋯ P_{α_{ℓ}}

with some closed K_{ℂ}-B double coset S_{0} and α_{1}, …, α_{ℓ} ∈ Ψ. So by the assumption, we have

xS_{1} ⊂ xS_{1}^{cl} = xS_{0}P_{α_{1}} ⋯ P_{α_{ℓ}} ⊂ S′_{0}P_{α_{1}} ⋯ P_{α_{ℓ}} = ⋃{T′ ∈ G_{ℝ}\G_{ℂ}/B | T ⊂ S_{1}^{cl}}. (8.1)

(Note that the right hand side is open in G_{ℂ} since S′_{0} is open.) Hence we have

xS_{1} ∩ S′ = ϕ and xS_{1} ∩ S′_{2} = ϕ. (8.2)

because S and S_{2} are not contained in S_{1}^{cl}. In the same way, we also have

xS_{2} ∩ S′ = ϕ and xS_{2} ∩ S′_{1} = ϕ. (8.3)

By (8.2), we have

(xS_{1} ∩ S′_{1})P_{α} = (xS_{1} ∩ (S′_{1} ∪ S′_{2} ∪ S′))P_{α} = (xS_{1} ∩ S′_{1}P_{α})P_{α} = xS_{1}P_{α} ∩ S′_{1}P_{α}.

By the assumption of induction, this set is closed in G_{ℂ} (compact in G_{ℂ}/B). On the other hand, we have

xS_{1}P_{α}∩S′_{1}P_{α} = (xS_{1}∪xS_{2}∪xS)∩(S′_{1}∪S′_{2}∪S′) = ((xS_{1}∪xS_{2}∪xS)∩(S′_{1}∪S′_{2}))∪(xS∩S′)

since

(xS_{1} ∪ xS_{2} ∪ xS) ∩ S′ = xS ∩ S′ (8.4)

by (8.2) and (8.3). Hence the set xS ∩S′ is closed in G_{ℂ} because it is the compliment of the open set

⋃{T′ ∈ G_{ℝ}\G_{ℂ}/B | T ⊂ (S_{1})^{cl}} ∪ ⋃{T′ ∈ G_{R}\G_{ℂ}/B | T ⊂ (S_{2})^{cl}}

in the closed set xS_{1}P_{α} ∩ S′_{1}P_{α}.

By (8.4), we have

(xS ∩ S′)P_{α} = (xSP_{α} ∩ S′)P_{α} = xSP_{α} ∩ S′P_{α} = xS_{1}P_{α} ∩ S′_{1}P_{α}.

This set is non-empty since xS_{1} ∩ S′_{1} is non-empty by the assumption of induction. Hence xS ∩ S′ is non-empty.

Now consider an arbitrary parabolic subgroup P. We may assume that B ⊂ P. Take a K_{ℂ}-B double coset S_{1} contained in S such that S_{1} is relatively closed in S. Then by (8.1), we have

xS_{1} ∩ T′ = ϕ

for K_{ℂ}-B double cosets T contained in S − S_{1} because T ⊄ (S_{1})^{cl}. Hence we have

xS_{1} ∩ S′ = xS_{1} ∩ S′_{1}

since S′ = ⋃_{T ⊂S} T′ as in [M2]. Thus the set

xS ∩ S′ = xS_{1}P ∩ S′ = (xS_{1} ∩ S′)P = (xS_{1} ∩ S′_{1})P

is closed in G_{ℂ} since (xS_{1} ∩ S′_{1})/B is compact. q.e.d.

Remark 8.2. (More precise arguments when P = B) First consider the case (1). Let y be an element of xS ∩ S′. Then we have

yP_{α} = (yP_{α} ∩ xS_{1}) ∪ (yP_{α} ∩ xS_{2}) ∪ (yP_{α} ∩ xS)
= (yP_{α} ∩ S′_{1}) ∪ (yP_{α} ∩ S′_{2}) ∪ (yP_{α} ∩ S′)

Here yP_{α}/B ≅ P^{1}(ℂ), (yP_{α} ∩ xS_{1})/B ≅ {a point}, (yP_{α} ∩ xS_{2})/B ≅ {a point} and (yP_{α} ∩ S′)/B ≅ S^{1}. By (8.2) and (8.3), we have

yP_{α} ∩ xS_{1} ⊂ yP_{α} ∩ S′_{1} and yP_{α} ∩ xS_{2} ⊂ yP_{α} ∩ S′_{2}.

We also have the same formulas when y ∈ xS_{1} ∩ S′_{1}. Thus we can construct a smooth surjection of (xS ∩ S′)/B onto (xS_{1} ∩ S′_{1})/B with fibers diffeomorphic to S^{1}.

Next consider the case (2). Let y be an element of xS ∩ S′. Then we have

yP_{α} = (yP_{α} ∩ xS_{1}) ∪ (yP_{α} ∩ xS)
= (yP_{α} ∩ S′_{1}) ∪ (yP_{α} ∩ S′)

There are two cases:

(2a) (yP_{α} ∩ xS_{1})/B ≅ {a point} and (yP_{α} ∩ S′)/B ≅ {a point},

(2b) (yP_{α} ∩ xS_{1})/B ≅ {two points} and (yP_{α} ∩ S′)/B ≅ S^{1}.

In (2a)-case, we get a diffeomorphism of (xS ∩ S′)/B onto (xS_{1} ∩ S′_{1})/B. On the other hand, in (2b)-case, we have a S^{1}-to-{2 points} correspondence.

Proposition 8.3. Let S be the open K_{ℂ}-orbit on X. Then

C(S)_{0} = C_{0}.

Here ∗_{0} denote the connected component of ∗ containing the identity e.

Proof. Let [0, 1] ∋ t ↦ x_{t} be a continuous curve in C(S) such that x_{0} = e. Then by Proposition 8.1, we have only to show that x_{t} ∈ C(S_{1}) for all closed K_{ℂ}-orbits S_{1} on X. Suppose x_{t} ∉ C(S_{1}) for some t and S_{1}. Then we will get a contradiction as follows.

We may assume that x_{t}S_{1}∩S′_{2} ≠ ϕ for some G_{ℝ}-orbit S′_{2} contained in the boundary of S′_{1}. Put ℓ = dim_{ℂ} S − dim_{ℂ} S_{2} and let α_{1}, …, α_{ℓ} be a sequence of simple roots such that

dim_{ℂ} S_{2}P_{α_{1}} ⋯ P_{α_{k}} = dim_{ℂ} S_{2} + k

for all k = 1, …, ℓ. Since S_{2}P_{α_{1}} ⋯ P_{α_{k}} ⊃ S, we have

S′_{2}P_{α_{1}} ⋯ P_{α_{k}} ⊃ S′

(cf. [M2]) and therefore

S′_{2} ⊂ S′P_{α_{k}} ⋯ P_{α_{1}}.

Since x_{t} ∈ C(S), we have S′ ⊂ x_{t}S. So we have

S′_{2} ⊂ x_{t}SP_{α_{k}} ⋯ P_{α_{1}}

and hence

ϕ ≠ x_{t}S_{1} ∩ x_{t}SP_{α_{k}} ⋯ P_{α_{1}} = x_{t}(S_{1} ∩ SP_{α_{k}} ⋯ P_{α_{1}}).

But since the set SP_{α_{k}} ⋯ P_{α_{1}} does not contain K_{ℂ}-orbits whose dimensions are smaller than dim_{ℂ} S_{2}, the right hand side of the above formula must be ϕ. Thus we have got a contradiction. q.e.d.

9. Appendix for SO(p, q)-case

Retain the notations in Section 4. Assume p ≤ q. The orthogonal compliment 𝔪 of 𝔰𝔬(p, q) + 𝔰(𝔲(p) ⊕ 𝔲(q)) in 𝔰𝔲(p, q) is written as

𝔪 = \{(\begin{matrix}0 & −i^{t}B \\ iB & 0 \end{matrix}) | B is a q × p real matrix\} .

So if we write

Z(t_{1}, …, t_{p}) = it_{1}(E_{n,1} − E_{1,n}) + ⋯ + it_{p}(E_{n−p+1,p} − E_{p,n−p+1}),

then we can take

𝔞 = {Z(t_{1}, …, t_{p}) | t_{1}, …, t_{p} ∈ ℝ}

as a maximal abelian subspace of 𝔪. By a fundamental decomposition theorem (c.f.

[R] Theorem 10), we have

SU(p, q) = SO(p, q)_{0}AS(U(p) × U(q)) (9.1)

where A = exp 𝔞.

Let X be the Grassmann manifold consisting of p-dimensional complex subspaces of ℂ^{n}. Let D′ denote the SU(p, q)-orbit on X containing V_{+} = ℂe_{1} ⊕⋯⊕ℂe_{p}. Then by (9.1), we have

D′ = SO(p, q)_{0}AV_{+}.

Let 𝔱 be as in Section 4 and define

𝔱^{+}′ = {Y (θ_{1}, …, θ_{p}) | |θ_{j}| < π/4 for all j = 1, …, p}

which is a subset of 𝔱^{+}. Put T^{+}′ = exp 𝔱^{+}′. Then we have

AV_{+} = T^{+}′V_{+}

because

(exp Z(t_{1}, …, t_{p}))e_{j} = (cosh t_{j})e_{j} + i(sinh t_{j})e_{n−j+1}

and

(exp Y (θ_{1}, …, θ_{p}))e_{j} = (cos θ_{j})e_{j} + i(sin θ_{j})e_{n−j+1}.

(This is a typical example of Lemma 2.1 (1).) Hence D_{0} = SO(p, q)_{0}T^{+}V_{+} is bigger than D′ = SO(p, q)_{0}T^{+}′V_{+}.

References

[AG] D. N. Akhiezer and S. G. Gindikin. On Stein extensions of real symmetric spaces. Math. Ann., 286:1–12, 1990.

[BGW] L. Barchini, S. G. Gindikin and H. W. Wong. Determinant functions and geometry of the flag manifold for SU(p, q). J. of Lie Theory, 6:191-206, 1996.

[G] S. Gindikin. Tube domains in Stein symmetric spaces. Positivity in Lie theory: Open problems, W. de Gruyter, 81–98, 1998.

[M1] T. Matsuki. The orbits of affine symmetric spaces under the action of minimal parabolic subgroups. J. Math. Soc. Japan, 31:331–357, 1979.

[M2] T. Matsuki. Orbits on affine symmetric spaces under the action of parabolic subgroups. Hiroshima Math. J., 12:307–320, 1982.

[M3] T. Matsuki. Closure relations for orbits on affine symmetric spaces under the action of minimal parabolic subgroups. Adv. Studies in Pure Math., 14:541–559, 1988.

[M4] T. Matsuki. Closure relations for orbits on affine symmetric spaces under the action of parabolic subgroups. Intersections of associated orbits. Hiroshima Math. J., 18:59–67, 1988.

[M5] T. Matsuki. Double coset decompositions of reductive Lie groups arising from two involutions. J. of Algebra, 197:49–91, 1997.

[MUV] I. Mirković, T. Uzawa and K. Vilonen. Matsuki correspondence for sheaves. Invent. Math., 109:231–245, 1992.

[R] W. Rossmann. The structure of semisimple symmetric spaces. Canad. J. Math., 31:157–180, 1979.

[Sp] T. A. Springer. Some results on algebraic groups with involutions. Adv. Studies in Pure Math., 6:525–534, 1984.

[W] J. A. Wolf. The Stein condition for cycle spaces of open orbits on complex flag manifolds. Ann. of Math., 136:541–555, 1992.

[WZ] J. A. Wolf and R. Zierau. Linear cycle spaces in flag domains. Math. Ann., 316:529–545, 2000.

Simon Gindikin, Department of Mathematics, Hill Center, Rutgers University, 110 Frelinghysen Road, Piscataway, NJ 08854-8019, U.S.A.

  E-mail address: gindikin@math.rutgers.edu

Toshihiko Matsuki, Faculty of Integrated Human Studies, Kyoto University, Kyoto 606-8501, Japan

  E-mail address: matsuki@math.h.kyoto-u.ac.jp