ON THE BOUNDARY ORBIT ACCUMULATION SET

FOR A DOMAIN WITH NON-COMPACT

AUTOMORPHISM GROUP

Alexander V. Isaev Steven G. Krantz

Abstract. For a smoothly bounded pseudoconvex domain D ⊂ ℂ^{n} of finite type with non-compact holomorphic automorphism group Aut(D), we show that the set S(D) of all boundary accumulation points for Aut(D) is a compact subset of ∂D and, if S(D) contains at least three points, it is a perfect set and thus has the power of the continuum. Moreover, we show that in this case, S(D) is either connected or the number of its connected components is uncountable. We also discuss how S(D) relates to other invariant subsets of ∂D.

Suppose that D ⊂ ℂ^{n} is a smoothly bounded domain, i.e. D is bounded and ∂D is C^{∞}-smooth. We assume that the group Aut(D) of holomorphic automorphisms of D is non-compact; this means, thanks to a classical result of H. Cartan, that there is a point q ∈ ∂D such that for some p ∈ D and a sequence {f_{j}} ⊂ Aut(D), one has that f_{j}(p) → q as j → ∞. Such a point q is called a boundary accumulation point for Aut(D) (see [GK1] for a discussion of this matter).

Let S(D) denote the set of all boundary accumulation points for Aut(D). Existing examples of domains with non-compact automorphism groups (see [FIK] for a discussion of the case of Reinhardt domains), for which the set S(D) can be found explicitly, indicate that this set should enjoy some explicit regularity properties.

For instance, let us for the moment restrict attention to smoothly bounded Reinhardt domains. It follows from [FIK] that, for such a domain D, S(D) is always a compact, connected smooth submanifold of ∂D. For such domains one can also observe other interesting properties of S(D) such as the constancy and minimality of the rank of the Levi form of ∂D along S(D) (see [H] for the genesis of these ideas); there is also a certain relation between this rank and the dimensions of the orbits of the action of Aut(D) on D. Similarly, the type in the sense of D’Angelo [D’A1] is constant and maximal along S(D). Many of these properties, when considered for general domains, appear to be related to the conjecture of Greene/Krantz [GK2] which states that every boundary accumulation point for a smoothly bounded domain must be of finite type.

In the present paper we begin a systematic study of the set S(D) for a fairly general class of domains, and obtain foundational results on its topology and the

Typeset by AMS-TEX

1991 Mathematics Subject Classification. 32M05.

Key words and phrases. Non-Compact Automorphism Groups, Boundary Accumulation Points.

This work was completed while Isaev was an Alexander von Humboldt Fellow at the University of Wuppertal. Research at MSRI by Krantz was supported in part by NSF Grant DMS-9022140.

relation of S(D) to other invariant subsets of ∂D. We thank H. Boas, R. Remmert, J. Wolf and S. Fu for stimulating remarks and suggestions concerning this work. We are also grateful to K. Diederich for a very valuable discussion of the results of this paper.

We say that ∂D is variety-free at q ∈ ∂D if there are no non-trivial germs of complex varieties lying in ∂D and passing through q.

Proposition 1. Let D be a bounded domain in ℂ^{n}. Suppose that ∂D is varietyfree at each point of S(D). Then S(D) is compact.

Proof. We need only to prove that S(D) is closed. Let {q_{k}} be a sequence of points from S(D) such that q_{k} → q ∈ ∂D as k → ∞. Since ∂D is variety-free at each point q_{k}, then for every q_{k} there is a sequence {f_{k}^{j}} from Aut(D) such that f_{k}^{j} converges to the constant map q_{k} in all of D as j → ∞ (see [GK1]).

Fix now a sequence {ϵ_{k}}, ϵ_{k} > 0, ϵ_{k} → 0 as k → ∞. Next, fix a point p ∈ D and for every k find an index j(k) such that |f_{k}^{j(k)}(p) − q_{k}| < ϵ_{k}. It is now obvious that f_{k}^{j(k)}(p) → q as k → ∞. Hence q ∈ S(D) and S(D) is closed.

The proposition is proved.

Remark. For smoothly bounded domains the variety-free assumption in Proposition 1 would follow from the conjecture of Greene/Krantz.

Theorem 2. Suppose that D ⊂ ℂ^{n} is a smoothly bounded pseudoconvex domain of finite type. Then, if S(D) contains at least three points, it is a perfect set and thus has the power of the continuum. Moreover, in this case, S(D) is either connected, or the number of its connected components is uncountable.

Proof. We note that any automorphism of D extends to a C^{∞}-automorphism of \overline{D} (see e.g. [D’A2]).

Assume that S(D) contains at least three points. We will first show that S(D) cannot have isolated points. Indeed, let q ∈ S(D) be an isolated point. Let {f_{j}} be a sequence in Aut(D) such that f_{j} → q in all of D as j → ∞. By passing to a subsequence we can also assume that f_{j}^{−1} → r in all of D where r ∈ S(D).

Suppose first that r = q. Since S(D) contains at least three points, one can find two distinct points s, t ∈ S(D), s, t ≠ q. Then by Theorem 1 of [B], f_{j}(s) → q, f_{j}(t) → q as j → ∞. Since q is an isolated point of S(D) and each f_{j} preserves S(D), we conclude that, for all sufficiently large j, one has f_{j}(s) = f_{j}(t) = q. This is impossible since every f_{j} is a one-to-one mapping on \overline{D}.

Suppose now that r ≠ q. Then there is s ∈ S(D) such that s ≠ q, r. Then by [B], f_{j}(q) → q, f_{j}(s) → q as j → ∞ which implies as before that for all sufficiently large j, f_{j}(q) = f_{j}(s) = q, but this is again impossible since the f_{j} are one-to-one on \overline{D}. Thus if S(D) has at least three elements then S(D) does not have isolated points, and thus is a perfect set.

Assume now that S(D) is disconnected and the number of its connected components is not uncountable. Let S(D) = ∪_{k}S_{k}(D) be the decomposition of S(D) into the disjoint union of its connected components. We will show that, for every k_{0}, every q ∈ S_{k_{0}}(D) and every neighborhood U of q there exists k_{1} ≠ k_{0} such that U ∩ S_{k_{1}} ≠ ∅. This implies that the number of connected components of S(D) is infinite and that each of the sets X_{m} = S(D) \ ∪_{k=1}^{m}S_{k}(D) is open and dense in S(D). Then since S(D) is compact by Proposition 1 and since ∩_{m=1}^{∞}X_{m} = ∅, the Baire Category Theorem gives a contradiction.

Let S_{k_{0}}(D) be a component of S(D) and q ∈ S_{k_{0}} be such that there exists a neighborhood U of q that does not contain points from S_{k}(D) with k ≠ k_{0}. Since S(D) does not have isolated points, S_{k_{0}}(D) is not a one-point set. Therefore, by decreasing U if necessary we can assume that S_{k_{0}}(D) \ U ≠ ∅. Let {f_{j}} ⊂ Aut(D) be such that f_{j} → q, f_{j}^{−1} → r in all of D, with r ∈ S(D) as j → ∞.

Suppose first that r ∈ S_{k_{0}}(D). Then by [B], for any other connected component S_{k_{1}}(D) of S(D) with k_{1} ≠ k_{0}, one has f_{j}(S_{k_{1}}(D)) ⊂ U for all sufficiently large j; this is impossible since U does not contain an entire component of S(D). If r ∉ S_{k_{0}}(D), then for all sufficiently large j, f_{j}(S_{k_{0}}(D)) ⊂ U which is again impossible.

Thus, S(D) either is connected or has uncountably many components.

The theorem is proved.

As we noted in the proof of Theorem 2 above, for a smoothly bounded pseudoconvex domain of finite type, the set S(D) is invariant under the extension of an automorphism of D to the boundary. In the following proposition we show that S(D) is generically the smallest invariant subset of ∂D.

Proposition 3. Let D ⊂ ℂ^{n} be a smoothly bounded pseudoconvex domain of finite type with non-compact automorphism group. Suppose that A ⊂ ∂D is nonempty, compact and invariant under Aut(D). Assume further that A is not a one-point subset of S(D). Then S(D) ⊂ A.

In particular, if Aut(D) does not have fixed points in ∂D, then S(D) is the smallest compact subset of ∂D invariant under Aut(D).

Proof. Since A is closed, it is sufficient to show that every point of S(D) belongs to \overline{A}. Let q ∈ S(D) and \{f_{j}\} ⊂ Aut(D) be such that f_{j} → q, f_{j}^{−1} → r in all of D as j → ∞, for some r ∈ S(D). Since A is not a one-point subset of S(D), there is a point a ∈ A, a ≠ r. Then, by [B], f_{j}(a) → q as j → ∞. Since A is invariant under any f_{j}, we see that f_{j}(a) ∈ A for all j and thus q is either an accumulation point for A, or, if f_{j}(a) = q for some index j, then q ∈ A.

The proposition is proved.

We now derive several corollaries from the above proposition regarding particular sets A.

Fix 0 ≤ k ≤ n − 1 and denote by L_{k}(D) the set of all points from ∂D where the rank of the Levi form of ∂D does not exceed k. Clearly, each set L_{k}(D) is a compact subset of ∂D and is invariant under any automorphism of D. Let l_{1} denote the minimal rank of the Levi form on ∂D and l_{2} the minimal rank of the Levi form on ∂D \ L_{l_{1}}(D). For these sets, Proposition 3 gives the following corollary (that was first proved in [H]). Note that the proof in [H] was also based on the results of [B].

Corollary 4. Let D be as in Proposition 3. Then either

(i) S(D) ⊂ L_{l_{1}}(D), or (ii) L_{l_{1}}(D) is a one-point subset of S(D) and S(D) ⊂ L_{l_{2}}(D).

Proof. If L_{l_{1}}(D) is not a one-point subset of S(D) then, by Proposition 3, S(D) ⊂ L_{l_{1}}(D). Suppose now that L_{l_{1}}(D) is a one-point subset of S(D). Then, since L_{l_{1}}(D) is strictly contained in L_{l_{2}}(D), one has S(D) ⊂ L_{l_{2}}(D).

The corollary is proved.

By a similar argument, one can endeavor to prove an analogous property of the type τ(q), q ∈ ∂D, in the sense of D’Angelo. Indeed, denote by T_{k}(D) the set of all points q ∈ ∂D where τ(q) is at least k. We choose t_{1} and t_{2} such that T_{t_{1}}(D) ≠ ∅, t_{2} < t_{1}, and there exists a point of type t_{2} in ∂D \ T_{t_{1}}(D). Since τ is invariant under automorphisms of D, so is every set T_{k}(D). However, the sets T_{k}(D) do not have to be closed, as the type function τ may not be upper-semicontinuous on ∂D (see e.g. an example in [D’A2], p. 136). Therefore, for the type we only have a somewhat weaker result.

Corollary 5. Let D be as in Proposition 3. Then either

(i) S(D) ⊂ \overline{T_{t_{1}}(D)}, or (ii) T_{t_{1}}(D) is a one-point subset of S(D) and S(D) ⊂ \overline{T_{t_{2}}(D)}.

In place of the type function τ, one can consider the multiplicity function μ on ∂D (see [D’A2], p. 145 for the definition), which is also invariant under the extensions of automorphisms to ∂D. It should be noted that, for q ∈ ∂D, the number τ(q) is finite if and only if μ(q) is finite. In contrast with τ, however, the function μ is upper-semicintinuous on ∂D. Analogously to what we have done above for the function τ, denote by M_{k}(D) the set of all points q ∈ ∂D, where μ(q) is at least k and choose m_{1} and m_{2} such that m_{1} = max_{q∈∂D} μ(q), m_{2} < m_{1}, and there exists a point of multiplicity m_{2} in ∂D \ M_{m_{1}}(D). Due to the upper-semicintinuity and invariance of μ, each set M_{k}(D) is a compact subset of ∂D that is invariant under Aut(D). This observation gives the following analogue of Corollary 6 for M_{m_{1}}, M_{m_{2}}.

Corollary 6. Let D be as in Proposition 3. Then either

(i) S(D) ⊂ M_{m_{1}}(D), or (ii) M_{m_{1}}(D) is a one-point subset of S(D) and S(D) ⊂ M_{m_{2}}(D).

The proof of Corollaries 5, 6 is completely analogous to that of Corollary 4.

Remarks.

1. It is plausible that Theorem 2 and Corollaries 4–6 hold without the assumptions of pseudoconvexity and finite type.

2. We note that, in complex dimension 2, the type τ is upper-semicontinuous. As a result, Corollary 5 can be stated in this case without passing to the closures of the T_{t_{j}}. Also, in complex dimension 2, Corollary 5 is a consequence of the explicit classification of smoothly bounded pseudoconvex domains of finite type with noncompact automorphism groups [BP1].

3. For a smoothly bounded circular domain, the set S(D) clearly cannot be a one- or two-point set. Thus, Theorem 2 gives that, for smoothly bounded pseudoconvex circular domains of finite type, S(D) is always a perfect set. Next, since for such domains the automorphism group cannnot have fixed points on the boundary, Corollary 4 implies that in this case the Levi form of ∂D has constant rank along S(D) and minimizes its rank over ∂D on S(D) (see also [H]). It also should be noted here that, by the results of [BP2], every smoothly bounded convex domain of finite type with non-compact automorphism group is biholomorphically equivalent to a certain polynomially defined domain that admits an action of the two-dimensional torus 𝕋^{2}. Therefore, for any such a domain, S(D) also is a perfect set, and the rank of the Levi form is constant and minimal on S(D).

4. The results of [FIK] imply that, for a smoothly bounded Reinhardt domain D, the type is constant along S(D) and maximizes on S(D) the type over ∂D. It is an interesting question whether there exists an analogue of this fact for more general domains (cf. Corollaries 5, 6). Note that one can make a statement analogous to Corollary 6 for the multitype introduced in [C], since the multitype function is upper-semicontinuous with respect to lexicographic ordering.

5. It also follows from [FIK] that, for a smoothly bounded Reinhardt domain D, the real dimension of any orbit of the action of Aut(D) on D is at least 2(k + 1), where k is the rank of the Levi form of ∂D along S(D). Moreover, there is precisely one orbit of minimal dimension 2(k + 1) (see [K] for a discussion of this phenomenon). Also, the orbit of minimal dimension approaches every point of S(D) non-tangentially, whereas any other orbit approaches every point of S(D) only along tangential directions. It would be interesting to know if similar statements hold for more general, e.g. circular, domains. The fact that there exists an orbit that approaches S(D) non-tangentially would be very important for a proof of the Greene/Krantz conjecture. It also could be used to show that S(D) is a smooth submanifold of ∂D.

We wish to conclude this paper with a list of immediate open problems that arise from the above discussion and which complement some of the preceding remarks.

Open Problems.

1. For a smoothly (C^{∞}) bounded domain D ⊂ ℂ^{n}, can the set S(D) be a oneor two-point set? Note that the reference [GK3] gives an example of a domain with C^{1−ϵ} boundary, for which S(D) has only two points. It appears that this example can be modified, using a parabolic group of automorphisms, so that S(D) has just one point. We shall explore this matter further, and additionally investigate increasing the boundary smoothness, in a future paper. Indications are that the case of finite boundary smoothness will be different from the case of infinite boundary smoothness.

2. For a smoothly bounded domain D, can the set S(D) have uncountably many components, for example, can it be a Cantor-type set?

3. For a smoothly bounded domain D, is the set S(D) always a smooth submanifold of ∂D? Note that the results of [FIK] imply that, for a smoothly bounded Reinhardt domain, S(D) is always a smooth submanifold of ∂D that is diffeomorphic to a sphere of odd dimension.

4. Is it always true that the rank of the Levi form is in fact constant and minimal along S(D) and that the type is constant and maximal along S(D)?

References

[B] Bell, S., Compactness of families of holomorphic mappings up to the boundary, Lecture Notes in Mathematics, Springer-Verlag 1268 (1986), 29–42.

[BP1] Bedford, E., Pinchuk, S., Domains in ℂ^{n+1} with non-compact automorphism groups, J. Geom. Anal. 1 (1991), 165–191.

[BP2] Bedford, E., Pinchuk, S., Convex domains with non-compact automorphism group (translated from Russian), Russian Acad. Sci. Sb. Math. 82 (1995), 1–20.

[C] Catlin, D., Boundary invariants of pseudoconvex domains, Ann. Math. 120 (1984), 529– 586.

[D’A1] D’Angelo, J., Real hypersurfaces, orders of contact, and applications, Ann. Math. 115

(1982), 615–637. [D’A2] D’Angelo, J., Several complex variables and the geometry of real hypersurfaces, Boca

Raton, CRS Press, 1993, Studies in Advanced Mathematics - XIII.

[FIK] Fu, S., Isaev, A. V., Krantz, S. G., Reinhardt domains with non-compact automorphism groups, Math. Res. Letters 3 (1996), 109–122.

[GK1] Greene, R. E., Krantz, S. G., Biholomorphic self-maps of domains, in C. Berenstein (Ed.) Complex Analysis II, Lecture Notes in Mathematics, Springer-Verlag 1276 (1987), 136–207.

[GK2] Greene, R. E., Krantz, S. G., Techniques for studying automorphisms of weakly pseudoconvex domains, in J. E. Fornæss (Ed.) Several Complex Variables: Proceedings of the Mittag-Leffler Institute, 1987-1988, Math. Notes, Vol. 38, Princeton University Press, 1993, 389–410.

[GK3] Greene, R. E., Krantz, S. G., Stability of the Carathéodory and Kobayashi metrics and applications to biholomorphic mappings, in Y.-T. Siu (Ed.) Complex Analysis of Several Complex Variables, Proceedings of the Symposium on Several Complex Variables held at the University of Wisconsin, Madison, 1982; Proc. Symp. Pure Math. 41 (1984), 77-93.

[H] Huang, X., Some applications of Bell’s theorem to weakly pseudoconvex domains, Pacific J. Math. 158 (1993), 305–315.

[K] Krantz, S. G., Survey of some recent ideas concerning automorphism groups of domains, in Proceedings of a Conference in Honor of Pierre Dolbeault, Hermann, 1995.

Alexander V. Isaev Centre for Mathematics and Its Applications, The Australian National University,

Canberra, ACT 0200, Australia Bergische Universität, Gesamthochschule Wuppertal, Mathematik (FB 07), Gauss-

strasse 20, 42097 Wuppertal, Germany E-mail address: Alexander.Isaev@anu.edu.au, Alexander.Isaev@math.uni-wuppertal.de

Steven G. Krantz Department of Mathematics, Washington University, St.Louis, MO 63130, USA MSRI, 1000 Centennial Drive, Berkeley, CA 94720, USA E-mail address: sk@math.wustl.edu, krantz@msri.org