EXAMPLES OF DOMAINS

WITH NON-COMPACT AUTOMORPHISM GROUPS

Siqi Fu, A. V. Isaev, and S. G. Krantz

Abstract. We give an example of a bounded, pseudoconvex, circular domain in ℂ^{n} for any n ≥ 3 with smooth real-analytic boundary and non-compact automorphism group, which is not biholomorphically equivalent to any Reinhardt domain. We also give an analogous example in ℂ^{2}, where the domain is bounded, non-pseudoconvex, is not equivalent to any Reinhardt domain, and the boundary is smooth real-analytic at all points except one.

Let D be a bounded or, more generally, a hyperbolic domain in ℂ^{n}. Denote by Aut(D) the group of biholomorphic self-mappings of D. The group Aut(D), with the topology given by uniform convergence on compact subsets of D, is in fact a Lie group [Kob].

A domain D is called Reinhardt if the standard action of the n-dimensional torus 𝕋^{n} on ℂ^{n},

z_{j} ↦ e^{iϕ^{j}}z_{j}, ϕ_{j} ∈ ℝ, j = 1, …, n,

leaves D invariant. For certain classes of domains with non-compact automorphism groups, Reinhardt domains serve as standard models up to biholomorphic equivalence (see e.g. [R], [W], [BP], [GK1], [Kod]).

It is an intriguing question whether any domain in ℂ^{n} with non-compact automorphism group and satisfying some natural geometric conditions is biholomorphically equivalent to a Reinhardt domain. The history of the study of domains with non-compact automorphism groups shows that there were expectations that the answer to this question would be positive (see [Kra]). In this note we give examples that show that the answer is in fact negative.

While the domain that we shall consider in Theorem 1 below has already been noted in the literature [BP], it has never been proved that this domain is not biholomorphically equivalent to a Reinhardt domain. Note that this domain is circular, i.e. it is invariant under the special rotations

z_{j} ↦ e^{iϕ}z_{j}, ϕ ∈ ℝ, j = 1, …, n.

Our first result is the following

Typeset by AMS-TEX

Mathematics Subject Classification: 32A07, 32H05, 32M05

Keywords and Phrases: Automorphism groups, Reinhardt domains, circular domains.

Theorem 1. There exists a bounded, pseudoconvex, circular domain Ω ⊂ ℂ^{3} with smooth real-analytic boundary and non-compact automorphism group, which is not biholomorphically equivalent to any Reinhardt domain.

Proof. Consider the domain

Ω = \{|z_{1}|^{2} + |z_{2}|^{4} + |z_{3}|^{4} + (\overline{z_{2}}z_{3} + \overline{z_{3}}z_{2})^{2} < 1\}.

The domain Ω is invariant under the action of the two-dimensional torus 𝕋^{2}

\begin{matrix}z_{1} ↦ e^{iϕ^{1}}z_{1},  ϕ_{1} ∈ ℝ, \\ z_{j} ↦ e^{iϕ^{2}}z_{j},  ϕ_{2} ∈ ℝ, j = 2, 3, \end{matrix}

and therefore is circular. It is also a pseudoconvex, bounded domain with smooth real-analytic boundary. The automorphism group Aut(Ω) is non-compact since it contains the following subgroup

\begin{matrix}z_{1} ↦ \frac{z_{1} − a}{1 − \overline{a}z_{1}} , \\ z_{2} ↦ \frac{(1 − |a|^{2})^{\frac{1}{4}} z_{2}}{(1 − \overline{a}z_{1})^{\frac{1}{2}}} , \\ z_{3} ↦ \frac{(1 − |a|^{2})^{\frac{1}{4}} z_{3}}{(1 − \overline{a}z_{1})^{\frac{1}{2}}} , \end{matrix}  (1)

for a complex parameter a with |a| < 1.

We are now going to explicitly determine Aut(Ω). Let F = (f_{1}, f_{2}, f_{3}) be an automorphism of Ω. Then, since Ω is bounded, pseudoconvex and has real-analytic boundary, F extends smoothly to \overline{Ω} [BL]. Therefore, F must preserve the rank of the Levi form L_{∂Ω}(q) of ∂Ω at every q ∈ ∂Ω. The only points where L_{∂Ω} ≡ 0 are those of the form (e^{iα}, 0, 0), α ∈ ℝ. These points must be preserved by F. This observation implies that f_{j}(e^{iα}, 0, 0) = 0 for all α ∈ ℝ, j = 2, 3. Restricting f_{2}, f_{3} to the unit disc Ω ∩ {z_{2} = z_{3} = 0}, we see that f_{j}(z_{1}, 0, 0) = 0 for all |z_{1}| ≤ 1, j = 2, 3. Therefore, F (0) = (b, 0, 0) for some |b| < 1. Taking the composition of F and the automorphism G of the form (1) with a = b, we find that the mapping G ∘ F preserves the origin. Since Ω is circular, it follows from a theorem of H. Cartan [C] that G ∘ F must be linear. Therefore, any automorphism of Ω is the composition of a linear automorphism and an automorphism of the form (1).

The above argument also shows that any linear automorphism of Ω can be written as

z_{1} ↦ e^{iϕ_{1}}z_{1},
z_{2} ↦ az_{2} + bz_{3},
z_{3} ↦ cz_{3} + dz_{3},

where ϕ_{1} ∈ ℝ, a, b, c, d ∈ ℂ, and the transformation in the variables (z_{2}, z_{3}) is an automorphism of the section Ω ∩ {z_{1} = 0}. Further, since the only points of ∂Ω where rank L_{∂Ω} = 1 are those of the form (z_{1}, w, ±w) with w ≠ 0 and since automorphisms of Ω preserve such points, it follows that any linear automorphism of Ω is in fact given by

z_{1} ↦ e^{iϕ_{1}}z_{1},
z_{2} ↦ e^{iϕ_{2}}z_{σ(2)},
z_{3} ↦ ±e^{iϕ_{2}}z_{σ(3)},

where ϕ_{1}, ϕ_{2} ∈ ℝ, and σ is a permutation of the set {2, 3}.

The preceding description of Aut(Ω) implies that dim Aut(Ω) = 4. That is to say, each of the four connected components of Aut(Ω) is parametrized by the point a from the unit disc and by the rotation parameters ϕ_{1}, ϕ_{2}.

Suppose now that Ω is biholomorphically equivalent to a Reinhardt domain D ⊂ ℂ^{3}. Since Ω is bounded, it follows that D is hyperbolic. It follows from [Kru] that any hyperbolic Reinhardt domain G ⊂ ℂ^{n} can be biholomorphically mapped onto its normilized form G̃ for which the identity component Aut_{0}(G̃) of Aut(G̃) is described as follows. There exist integers 0 ≤ s ≤ t ≤ p ≤ n and n_{i} ≥ 1, i = 1, …, p, with ∑_{i=1}^{p} n_{i} = n, and real numbers α_{i}^{k}, i = 1, …, s, k = t + 1, …, p, and β_{j}^{k}, j = s + 1, …, t, k = t + 1, …, p such that if we set z^{i} = (z_{n_{1}+···+n_{i−1}+1}, …, z_{n_{1}+···+n_{i}}), i = 1, …, p, then Aut_{0}(G̃) is given by the mappings

\begin{matrix}z^{i} ↦ \frac{A^{i}z^{i} + b^{i}}{c^{i}z^{i} + d^{i}} , i = 1, …, s, \\ z^{j} ↦ B^{j}z^{j} + e^{j}, j = s + 1, …, t, \\ z^{k} ↦ C^{k} \frac{∏_{j=s+1}^{t} exp (−β_{j}^{k} (2\overline{e^{j}}^{T} B^{j}z^{j} + |e^{j}|^{2})) z^{k}}{∏_{i=1}^{s}(c^{i}z^{i} + d^{i})^{2α_{i}^{k}}} , k = t + 1, …, p, \end{matrix} (2)

where

( \begin{matrix}A^{i} & b^{i} \\ c^{i} & d^{i} \end{matrix} ) ∈ SU(n_{i}, 1), i = 1, …, s,

B^{j} ∈ U(n_{j}), e^{j} ∈ ℂ^{n_{j}}, j = s + 1, …, t,

C^{k} ∈ U(n_{k}), k = t + 1, …, p.

The normalized form G̃ is written as

\begin{matrix}G = \{|z^{1}| & < 1, …, |z^{s}| < 1, \\ (\frac{z^{t+1}}{∏_{i=1}^{s} (1 − |z^{i}|^{2})^{α_{i}^{t+1}} ∏_{j=s+1}^{t} exp (−β_{j}^{t+1} |z^{j}|^{2})} , …, \\ \frac{z^{p}}{∏_{i=1}^{s} (1 − |z^{i}|^{2})^{α_{i}^{p}} ∏_{j=s+1}^{t} exp (−β_{j}^{p} |z^{j}|^{2})}) ∈ G̃_{1}\}, \end{matrix}  (3)

where G̃_{1} := G̃ ⋂ {z^{i} = 0, i = 1, …, t} is a hyperbolic Reinhardt domain in ℂ^{n_{t+1}} × ⋯ × ℂ^{n_{p}}.

It is now easy to see that, for any hyperbolic Reinhardt domain D ⊂ ℂ^{3} written in a normilized form D̃, Aut_{0}(D̃) given by formulas (2) cannot have dimension equal to 4.

This completes the proof.

Remark. The theorem can be easily extended to ℂ^{n} for any n ≥ 3 (just replace |z_{1}|^{2} in the defining function of Ω by ∑_{j=1}^{n−2} |z_{j}|^{2}, z_{2} by z_{n−1}, z_{3} by z_{n}).

There is considerable evidence that, in complex dimension two, an example such as that constructed in Theorem 1 does not exist. Certainly the example provided above depends on the decoupling, in the domain Ω, of the variables z_{2}, z_{3} from the variable z_{1}. Such decoupling is not possible when the dimension is only two.

The work of Bedford and Pinchuk (see [BP] and references therein) suggests that the only smoothly bounded domains in ℂ^{2} with non-compact automorphism groups are (up to biholomorphic equivalence) the complex ellipsoids

Ω_{α} = {(z_{1}, z_{2}) ∈ ℂ^{2} : |z_{1}|^{2} + |z_{2}|^{2α} < 1},

where α is a positive integer. It is also a plausible conjecture that any bounded domain in ℂ^{2} with non-compact automorphism group and a boundary of finite smoothness C^{k} for k ≥ 1, is biholomorphically equivalent to some Ω_{α}, where α ≥ 1 and is not necessarily an integer. Of course all the domains Ω_{α} are pseudoconvex and Reinhard.

However, as the following theorem shows, if we allow the boundary to be nonsmooth at just one point, then the domain may be non-pseudoconvex and be nonequivalent to any Reinhardt domain.

Theorem 2. There exists a bounded, non-pseudoconvex domain Ω ⊂ ℂ^{2} with noncompact automorphism group such that ∂Ω is smooth real-analytic everywhere except one point (this exceptional point is an orbit accumulation point for the automorphism group action), and such that Ω is not biholomorphically equivalent to any Reinhardt domain.

For the proof of Theorem 2, we first need the following lemma.

Lemma A. If Ω ⊂ ℂ^{2} is a bounded, non-pseudoconvex, simply-connected domain such that the identity component Aut_{0}(Ω) of the automorphism group Aut(Ω) is non-compact, then Ω is not biholomorphically equivalent to any Reinhardt domain.

Proof of Lemma A. Suppose that Ω is biholomorphically equivalent to a Reinhardt domain D. Since Ω is bounded, it follows that D is hyperbolic. Also, since Aut_{0}(Ω) is non-compact, then so is Aut_{0}(D). We are now going to show that any such domain D is either pseudoconvex, or not simply-connected, or cannot be biholomorphically equivalent to a bounded domain. This result clearly implies the lemma.

We can now assume that the domain D is written in its normalized form D̃ as in (3), and Aut_{0}(D̃) is given by formulas (2). Then, since Aut_{0}(D̃) is non-compact, it must be that t > 0. Next, if p = t, then D̃ is either non-hyperbolic (for s < t), or (for s = t) is the unit ball or the unit polydisc and therefore is pseudoconvex. Thus we can assume that t = 1, p = 2, n_{1} = n_{2} = 1.

Let D̃_{1} ⊂ ℂ be the hyperbolic Reinhardt domain analogous to G̃_{1} that was defined above (see (3)). Clearly, there are the following possibilities for D̃_{1}:

(i) D̃_{1} = {0 < |z_{2}| < R}, 0 < R < ∞; (ii) D̃_{1} = {r < |z_{2}| < R}, 0 < r < R ≤ ∞; (iii) D̃_{1} = {|z_{2}| < R}, 0 < R < ∞.

For the cases (i), (ii), D̃ is always not simply-connected, and therefore we will concentrate on the case (iii). If s = 0, then D̃ is not hyperbolic since it contains the complex line {z_{2} = 0}. Thus we can assume that s = 1. Next observe that, for α_{1}^{2} ≥ 0, the domain D̃ is always pseudoconvex. Thus we may take α_{1}^{2} < 0. Then the domain D̃ has the form

D̃ = \{|z_{1}| < 1, |z_{2}| < \frac{R}{(1 − |z_{1}|^{2})^{γ}} \} , γ > 0.

We will now show that the above domain D̃ cannot be biholomorphically equivalent to a bounded domain. More precisely, we will show that any bounded holomorphic function on D̃ is independent of z_{2}.

Let f(z_{1}, z_{2}) be holomorphic on D̃ and |f| < M for some M > 0. For every ρ such that |ρ| ≤ \frac{R}{2}, the disc Δ_{ρ} = \{|z_{1}| < 1, z_{2} = ρ\} is contained in D̃. We will show that ∂f/∂z_{2} ≡ 0 on every such Δ_{ρ}, which implies that ∂f/∂z_{2} ≡ 0 everywhere in D̃.

Fix a point (μ, ρ) ∈ Δ_{ρ} and restrict f to the disc Δ′_{μ} = {z_{1} = μ, |z_{2}| < R_{μ}}, where R_{μ} = R/2(1 − |μ|^{2})^{γ} . Clearly, (μ, ρ) ∈ Δ′_{μ} and \overline{Δ′_{μ}} ⊂ D̃. By the Cauchy Integral Formula

f(μ, z_{2}) = \frac{1}{2πi} ∫_{∂Δ′_{μ}} \frac{f(μ, ζ)}{ζ − z_{2}} dζ,

for |z_{2}| < R_{μ}, and therefore

\frac{∂f}{∂z_{2}} (μ, ρ) = \frac{1}{2πi} ∫_{∂Δ′_{μ}} \frac{f(μ, ζ)}{(ζ − ρ)^{2}} dζ.

Hence

|\frac{∂f}{∂z_{2}} (μ, ρ)| ≤ \frac{MR_{μ}}{(R_{μ} − |ρ|)^{2}} .

Letting |μ| → 1 and taking into account that R_{μ} → ∞, we see that |∂f/∂z_{2}(μ, ρ)| → 0 as |μ| → 1. Therefore, ∂f/∂z_{2} ≡ 0 on Δ_{ρ}.

The lemma is proved.

Proof of Theorem 2. We will now present a domain that satisfies the conditions of the lemma. Set

Ω = \{|z_{1}|^{2} + |z_{2}|^{4} + 8|z_{1} − 1|^{2} (\frac{z_{2}^{2}}{z_{1} − 1} − \frac{3}{2} \frac{|z_{2}|^{2}}{|z_{1} − 1|} + \frac{\overline{z_{2}}^{2}}{\overline{z_{1}} − 1})^{2} < 1\} .

The domain Ω is plainly bounded since the third term on the left is non-negative. Next, the identity component Aut_{0}(Ω) of its automorphism group is non-compact since it contains the subgroup

\begin{matrix}z_{1} ↦ \frac{z_{1} − a}{1 − az_{1}} , \\ z_{2} ↦ \frac{(1 − a^{2})^{\frac{1}{4}} z_{2}}{(1 − az_{1})^{\frac{1}{2}}} , \end{matrix}

where a ∈ (−1, 1).

Further, Ω is simply-connected, since the family of mappings F_{τ} (z_{1}, z_{2}) = (z_{1}, τz_{2}), 0 ≤ τ ≤ 1, retracts Ω inside itself, as τ → 0, to the unit disc {|z_{1}| < 1, z_{2} = 0} (which is simply-connected).

To show that Ω is not pseudoconvex, consider its unbounded realization. Namely, under the mapping

\begin{matrix}z_{1} ↦ \frac{z_{1} + 1}{z_{1} − 1}, \\ z_{2} ↦ \frac{\sqrt{2}z_{2}}{\sqrt{z_{1} − 1}}, \end{matrix}  (4)

the domain Ω is transformed into the domain

Ω′ = \{Re z_{1} + \frac{1}{4}|z_{2}|^{4} + 2 (z_{2}^{2} − \frac{3}{2}|z_{2}|^{2} + \overline{z_{2}}^{2})^{2} < 0\} .

It is easy to see that at the boundary point (−\frac{3}{4}, 1) ∈ ∂Ω′ the Levi form of ∂Ω′ is equal to −|z_{2}|^{2}, and thus is negative-definite. Therefore, Ω is non-pseudoconvex.

Hence, by Lemma A, Ω is not biholomorphically equivalent to any Reinhardt domain.

Next, if ϕ denotes the defining function of Ω, the following holds at every boundary point of Ω except (1, 0):

\frac{∂ϕ}{∂z_{1}} = \frac{1}{z_{1} − 1} (−\frac{z_{2}}{2} \frac{∂ϕ}{∂z_{2}} + 1 − \overline{z_{1}}) ,

and therefore grad ϕ does not vanish at every such point. Hence, ∂Ω is smooth real-analytic everywhere except at (1, 0).

The theorem is proved.

Remarks.

1. The hypothesis of simple connectivity in Lemma A is automatically satisfied if, for example, the boundary of the domain is locally variety-free and smooth near some orbit accumulation point for the automorphism group of the domain (see e.g. [GK2]). For a smoothly bounded domain it would follow from a conjecture of Greene/Krantz [GK3].

2. Tedious calculations show that the boundary of the domain Ω in Theorem 2 is quite pathological near the exceptional point (1, 0). It is not Lipschitz-smooth of any positive degree. It would be interesting to know whether there is an example with Lipschitz-1 boundary at the bad point.

In fact, many more examples similar to that in Theorem 2 can be constructed in the following way. Let

Ω′ = {(z_{1}, z_{2}) ∈ ℂ^{2} : Re z_{1} + P (z_{2}) < 0},  (5)

where P = |z_{2}|^{2m}+Q(z_{2}) is a homogeneous non-plurisubharmonic polynomial, m is a positive integer, and Q(z_{2}) is positive away from the origin. Then, by a mapping analogous to (4), Ω′ can be transformed into a bounded domain Ω. The domain Ω is simply-connected, non-pseudoconvex, Aut_{0}(Ω) is non-compact, and ∂Ω is smooth real-analytic everywhere except at the point (1, 0). For all such examples, ∂Ω is not Lipschitz-smooth of any positive degree at (1, 0).

It is also worth noting that, in the example contained in Theorem 2, the point (−1, 0) is also an orbit accumulation point, but ∂Ω is smooth real-analytic at this point.

3. It is conceivable that the domain Ω as in Theorem 2 has an alternative, smoothly bounded realization, but it looks plausible that if in formula (5) we allow P (z_{2}) to be an arbitrary homogeneous polynomial positive away from the origin with no harmonic term, then domain (5) does not have a bounded realization with C^{1}- smooth boundary, unless P (z_{2}) = c|z_{2}|^{2m}, where c > 0 and m is a positive integer.

Acknowledgements. This work was completed while the second author was an Alexander von Humboldt Fellow at the Research at MSRI by the third author supported in part by NSF Grant DMS-9022140. University of Wuppertal.

References

[BL] Bell, S., Ligocka, E., A simplification and extension of Fefferman’s theorem on biholomorphic mappings, Invent. Math. 57 (1980), 283–289.

[BP] Bedford, E., Pinchuk, S., Domains in ℂ^{n+1} with non-compact automorphism groups, J. Geom. Anal. 1 (1991), 165–191.

[C] Cartan, H., Les transformations analytiques des domaines cerclés les uns dans les autres, Compt. Rendus de l’Académie des Sciences de Paris 190 (1930), 718–720.

[GK1] Greene, R. E., Krantz, S. G., Characterization of certain weakly pseudoconvex domains with non-compact automorphism groups, Lecture Notes in Mathematics 1268, Springer-Verlag, 1987, 121–157.

[GK2] Greene, R. E, Krantz, S. G., Invariants of Bergman geometry and the automorphism groups of domains in ℂ^{n}, Proceedings of a Conference on Complex Analysis and Geometry held in Cetraro, 1989, Mediterranean Press, 1992, 107–136.

[GK3] Greene, R. E., Krantz, S. G., Techniques for studying automorphisms of weakly pseudoconvex domains, in J. E. Fornaess (Ed.) Several Complex Variables: Proceedings of the Mittag-Leffler Institute, 1987-1988, Math. Notes, Vol. 38, Princeton University Press, 1993, 389–410.

[Kob] Kobayashi, S., Intrinsic distances, measures and geometric function theory, Bull. Amer. Math. Soc. 82 (1976), 357–416.

[Kod] Kodama, A., A characterization of certain domains with good boundary points in the sense of Greene-Krantz, Kodai. Math. J. 12 (1989), 257–269.

[Kra] Krantz, S. G., Convexity in complex analysis, Several Complex Variables and Complex Geometry, Part 1 (Santa Cruz, CA, 1989), Proc. Symp. Pure Math. 52, Part 1 (1991), 119–137.

[Kru] Kruzhilin, N. G., Holomorphic automorphisms of hyperbolic Reinhardt domains (translated from Russian), Math. USSR-Izv. 32 (1989), 15–38.

[R] Rosay, J. P., Sur une caractérisation de la boule parmi les domaines de ℂ^{n} par son groupe d’automorphismes, Ann. Inst. Fourier (Grenoble) 29 (1979), 91–97.

[W] Wong, B., Characterization of the unit ball in ℂ^{n} by its automorphism group, Invent. Math. 41 (1977), 253–257.

Siqi Fu Department of Mathematics University of California, Irvine, CA 92717 USA E-mail address: sfu@math.uci.edu

A. V. Isaev Centre for Mathematics and Its Applications The Australian National University Canberra, ACT 0200 AUSTRALIA E-mail address: Alexander.Isaev@anu.edu.au and Bergische Universität Gesamthochschule Wuppertal Mathematik (FB 07) Gaussstrasse 20 42097 Wuppertal GERMANY E-mail address: Alexander.Isaev@math.uni-wuppertal.de

S. G. Krantz Department of Mathematics Washington University, St.Louis, MO 63130 USA E-mail address: sk@math.wustl.edu and MSRI 1000 Centennial Drive Berkeley, California 94720 USA E-mail address: krantz@msri.org