GEODESIC CONJUGACY IN TWO-STEP NILMANIFOLDS

Carolyn Gordon and Yiping Mao

Abstract. Two Riemannian manifolds are said to have C^{k}-conjugate geodesic flows if there exist an C^{k} diffeomorphism between their unit tangent bundles which intertwines the geodesic flows. We obtain a number of rigidity results for the geodesic flows on compact 2-step Riemannian nilmanifolds: For generic 2-step nilmanifolds the geodesic flow is C^{2} rigid. For special classes of 2-step nilmanifolds, we show that the geodesic flow is C^{0} or C^{2} rigid. In particular, there exist continuous families of 2-step nilmanifolds whose Laplacians are isospectral but whose geodesic flows are not C^{0} conjugate.

Introduction

Two Riemannian manifolds (M, g) and (N, h) are said to have C^{k}-conjugate geodesic flows if there is a C^{k} diffeomorphism F : S(M, g) → S(N, h) which intertwines the geodesic flows on S(M, g) and S(N, h). Here S(M, g) and S(N, h) are the unit tangent bundles of (M, g) and (N, h) respectively. We call F a C^{k}-geodesic conjugacy from M to N. A compact Riemannian manifold is said to be C^{k}-geodesically rigid within a given class of manifolds if any Riemannian manifold N in that class whose geodesic flow is C^{k}-conjugate to that of M is isometric to M.

A. Weinstein ([W]) exhibited a zoll surface of non-constant curvature whose geodesic flow is conjugate to that of the round sphere. On the other hand, two flat tori with C^{1}-conjugate geodesic flows must be isometric. Therefore, a natural question arises:

Question. Which compact Riemannian manifolds are C^{k}-geodesically rigid in a given class of manifolds?

This question is central to the study of negatively curved manifolds. Many important open problems in the field will follow if all negatively curved manifolds can be shown to be geodesically rigid (see [BFL], [EHS], [Ka], [Kk]). For negatively curved surfaces , C. Croke ([C]) and J. Otal ([O]) have independently answered this question affirmatively. Recently, C. Croke and B. Kleiner ([CK]) proved that compact Riemannian manifolds with a parallel vector field are geodesically rigid. For negatively curved manifolds of higher dimension, the question is still open. Recently, G. Besson, G. Courtois and G. Gallot proved that if a manifold has

Typeset by AMS-TEX

1991 Mathematics Subject Classification. 58G25, 53C22.

Key words and phrases. Geodesic conjugacy, almost inner derivation, marked length spectrum.

The first author was supported in part by a grant from the National Science Foundation.

Research at MSRI is supported by NSF grant #DMS 9022140.

a geodesic flow which is C^{1}-conjugate to the geodesic flow of a rank one locally symmetric space M , then it is isometric to M.

In this article, we will consider the question above for the class of 2-step Riemannian nilmanifolds. This question was also asked by P. Eberlein in [E].

A k-step Riemannian nilmanifold is a compact quotient of a k-step nilpotent Lie group G by a discrete subgroup together with a Riemannian metric whose lift to G is left-invariant. Note that a one-step nilmanifold is just a flat torus. Thus the 2-step nilmanifolds may be viewed as the simplest generalization of flat tori, yet they have a much richer geometry.

We are interested in this problem in part because of its relationship to spectral geometry. Two Riemannian manifolds are said to be isospectral if the associated Laplace-Beltrami operators have the same eigenvalue spectrum. A continuous family M_{t} (−ϵ < t < ϵ) of Riemannian manifolds is said to be an isospectral deformation of M_{0} if the manifolds are pairwise isospectral. Since the Laplacian can be viewed as the quantum analog of the geodesic flow, one might expect that continuous families of isospectral manifolds would have conjugate geodesic flows. E. N. Wilson and the first author ([GW]) gave a method for constructing isospectral deformations of Riemannian nilmanifolds, in particular, 2-step nilmanifolds. Moreover, P. Eberlein ([E]) showed that if a pair of 2-step nilmanifolds have C^{1} conjugate geodesic flows, then they must both lie in one of these continuous families. Nonetheless, we shall see that, at least for some of the isospectral deformations, the geodesic flows are not even C^{0} conjugate.

The main results in this paper are the following:

Theorem 1.

Almost all compact 2-step nilmanifolds are C^{2}-geodesically rigid within the class of all compact nilmanifolds.

(See section three for a precise statement.)

Theorem 2. There is a large class of compact 2-step Riemannian nilmanifolds such that any Riemannian nilmanifold whose geodesic flow is C^{0} conjugate to a nilmanifold M in this class is isometric to M. Included in this class are many continuous families of isospectral manifolds.

We remark that R. Kuwabara [Ku] has shown that for some of the isospectral deformations of 2-step nilmanifolds mentioned above, the geodesic flows restricted to dense open subsets of the tangent bundles are symplectically conjugate.

A. Kaplan [Kap] introduced the notion of 2-step nilmanifolds of Heisenberg type. (See section one for the definition.)

Theorem 3. Compact 2-step nilmanifolds of Heisenberg type are C^{0}-geodesically rigid within the class of compact Riemannian nilmanifolds.

Finally, Eberlein [E] defined a notion of Riemannian 2-step nilmanifolds in resonance. This algebraic condition is closely associated with the geometric condition that the set of vectors lying in closed orbits of the geodesic flow are dense in the unit tangent bundle. (See [Ma], [LP].) We define a notion of nilmanifold strongly in resonance and prove:

Theorem 4. Compact 2-step nilmanifolds which are strongly in resonance are C^{2}-geodesically rigid within the class of compact Riemannian nilmanifolds.

The paper is organized as follows: In section one, we review the geometry of 2-step nilmanifolds, define various special classes of 2-step nilmanifolds, including those cited in the theorems above, and review the construction of isospectral deformations of nilmanifolds. Section two establishes a number of results concerning conjugacies of geodesic flows between arbitrary 2-step nilmanifolds and culminates in Theorem 2. Theorems 1, 3 and 4 are established in section three.

The authors would like to thank Patrick Eberlein, Jens Heber, and David Webb for helpful discussions. Some of the results of this paper were announced in [GM].

1. Preliminaries

In this section, we give a brief introduction to the geometry of 2-step Lie groups that will be used in the proof of the theorems. We recommend [E] as a reference.

A Riemannian nilmanifold is a quotient M = Γ\N of a nilpotent Lie group N by a discrete subgroup Γ, together with a Riemannian metric g whose lift to N, also denoted g, is left-invariant. We say the nilmanifold has step size k if N is k-step nilpotent.

1.1 Notation. Let N be a 2-step nilpotent Lie group with a left invariant metric g. The metric g defines an inner product <, > on the Lie algebra N of N. Let Z = [N , N] and let V denote the orthogonal complement of Z in N relative to <, >.

Since N is 2-step nilpotent, the Campbell-Baker-Hausdorff theorem gives the product rule

exp(x) exp(y) = exp(x + y + \frac{1}{2}[x, y])

for all x, y ∈ N , where exp : N → N is the Lie group exponential map.

For z in Z, define a skew symmetric linear transformation J(z) : V → V by J(z)x = (ad(x))^{∗}z for x ∈ V, where (ad(x))^{∗} denotes the adjoint of ad(x). Or, equivalently

(∗) < J(z)x, y >=< [x, y], z > for x, y ∈ V, z ∈ Z. The operators J(z) contain all geometric information concerning the manifold (N, g).

Conversely, given inner product spaces V and Z and a linear map J : Z → 𝔰o(V), we can construct a 2-step nilpotent Lie algebra N together with an inner product, by setting N = V ⨁ Z and defining the Lie bracket so that Z is central and [ , ] : V × V → Z is given by (∗). The inner product defines a left-invariant Riemannian metric on the associated simply-connected nilpotent Lie group N.

1.2 Example. The Heisenberg group of dimension 2n + 1 is the simply-connected Lie group with Lie algebra N = span{x_{1}, ⋯, x_{n}, y_{1}, ⋯, y_{n}, z}, where [x_{i}, y_{i}] = z, 1 ≤ i ≤ n, and all other brackets of basis elements are zero. Giving N the inner product for which the basis above is orthonormal, we have

Z = span{z},
V = span{x_{1}, ⋯, x_{n}, y_{1}, ⋯, y_{n}}

and J(z)x_{i} = y_{i}, J(z)y_{i} = −x_{i}. Thus J(z) can be viewed as a complex structure on the vector space V.

1.3 Definition.

(1) A 2-step nilpotent Lie group N with a left-invariant Riemannian metric g is of Heisenberg type if J^{2}(z) = −|z|^{2} Id on V, for all z ∈ Z.

(2) (N, g) is said to be nonsingular if J(z) is a nonsingular linear transformation for all z ∈ Z, z ≠ 0.

(3) (N, g) is said to be in resonance if for all z ≠ 0 ∈ Z, every ratio of non-zero eigenvalues of J(z) is rational, equivalently, for each z ∈ Z, there exists a constant t such that e^{tJ(z)} = Id, where t may depend on z.

(4) (N, g) is said to be strongly in resonance if for each z ∈ Z, there exists a constant t such that e^{tJ(z)} = − Id, where t may depend on z.

(5) (N, g) is said to be irrational if for z in a dense set of Z, 1, θ_{1}(z), ⋯, θ_{p}(z) are linearly independent over the field of rational numbers, where \{±\sqrt{−1}θ_{i}(z), i = 1, 2, ⋯, p} are the distinct eigenvalues of J(z). ( We choose θ_{i}(z) ≥ 0).

1.4 Remark.

(1) The notion of Lie group of Heisenberg type is due to A. Kaplan ([Kn]). These groups have a surprisingly rich and varied geometry and have been studied by several authors (e.g., [E], [Rie]). It is easy to see that the Lie groups of Heisenberg type are nonsingular and strongly in resonance.

(2) P. Eberlein ([E]) defined the term “in resonance”. The nilmanifolds of Heisenberg type are in resonance, in fact, strongly in resonance. M. Mast ([Mt]) gave the first examples of nilmanifolds in resonance which are not of Heisenberg type. We give an additional example below.

(3) The condition “strongly in resonance” implies “in resonance”. However, our examples will show that “strongly in resonance” is indeed stronger than “in resonance”.

(4) Generically, all 2-step nilpotent Lie groups are irrational. In this case, for z in a dense set of Z, there is a sequence of t_{i} such that lim_{i→∞} e^{t_{i}J(z)} = − Id .

1.5 Example. Let V and Z be inner product spaces with orthonormal bases {X_{1}, ⋯, X_{8}} and {Z_{1}, Z_{2}}, respectively, and define J : Z → 𝔰o(V) so that for Z = z_{1}Z_{1} + z_{2}Z_{2}, J(Z) has the following matrix representation with respect to the basis above of V:

( \begin{matrix}0 & 0 & −λ_{1}z_{1} & −λ_{1}z_{2} & 0 & 0 & 0 & 0 \\ 0 & 0 & λ_{1}z_{2} & −λ_{1}z_{1} & 0 & 0 & 0 & 0 \\ λ_{1}z_{1} & −λ_{1}z_{2} & 0 & 0 & 0 & 0 & 0 & 0 \\ λ_{1}z_{2} & λ_{1}z_{1} & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & −λ_{2}z_{1} & −λ_{2}z_{2} \\ 0 & 0 & 0 & 0 & 0 & 0 & λ_{2}z_{2} & −λ_{2}z_{1} \\ 0 & 0 & 0 & 0 & λ_{2}z_{1} & −λ_{2}z_{2} & 0 & 0 \\ 0 & 0 & 0 & 0 & λ_{2}z_{2} & λ_{2}z_{1} & 0 & 0 \end{matrix} )

The distinct eigenvalues of J(Z) are \{±\sqrt{−1}λ_{1}|Z|, ±\sqrt{−1}λ_{2}|Z|\}.

As in 1.1, the data (V, Z, J) defines a simply-connected 2-step nilpotent Lie group N with a left-invariant Riemannian metric g. If, say, λ_{2} = 2λ_{1}, then (N, g) is in resonance but not strongly in resonance. If, say, λ_{2} = 3λ_{1}, then (N, g) is strongly in resonance but is not of Heisenberg type.

We now consider geodesics on a 2-step nilmanifold. Let x, y ∈ N regarded as left invariant vector fields on N. Recall that the formula for the covariant derivative ▽_{x}y normally contains 6 terms but in this case three of them vanish since g(x, y) is constant in N. One obtains

▽_{x}y = \frac{1}{2}\{[x, y] − (ad(x))^{∗}y − (ad(y))^{∗}x\}

Therefore, one obtains the following lemma.

1.6 Lemma. ([E])

(1) ▽_{x}y = \frac{1}{2}[x, y], for x, y ∈ V.

(2) ▽_{x}z = ▽_{z}x = −\frac{1}{2}J(z)x, for x ∈ V, z ∈ Z.

(3) ▽_{z}z^{∗} = 0, for z, z^{∗} ∈ Z.

Using lemma 1.6, one can calculate geodesics in N.

Let σ(t) be a geodesic with σ(0) = e and σ′(0) = x + z ∈ N, where e is the identity element of N, x ∈ V and z ∈ Z. In exponential coordinates, we write σ(t, x + z) = exp(x(t) + z(t)) where x(t) ∈ V, z(t) ∈ Z, x′(0) = x, z′(0) = z. We denote J(z) by J. One obtains:

1.7 Lemma. ([E])

(1) σ′(t) = dL_{σ(t)}(e^{tJ}x + z), for t ∈ ℝ, where L_{m} denotes the left translation on N by m ∈ N.

(2) x(t) = tx_{1} + (e^{tJ} − id)J^{−1}x_{2}, for t ∈ ℝ, where x_{1} ∈ Ker(J(z)), x_{2} ∈ Ker(J(z))^{⊥} and x_{1} + x_{2} = x.

(3) z(t) = tz_{1}(t) + z_{2}(t),

z_{1}(t) = z + \frac{1}{2}[x_{1}, (e^{tJ} + Id)J^{−1}x_{2}] + \frac{1}{2} ∑_{i=1}^{N}[J^{−1}ξ_{i}, ξ_{i}],

z_{2}(t) = [x_{1}, (Id −e^{tJ})J^{−2}x_{2}] − \frac{1}{2}[e^{tJ}J^{−1}x_{2}, J^{−1}x_{2}]

− \frac{1}{2} ∑_{i≠j=1}^{N} \frac{1}{θ_{j}^{2} − θ_{i}^{2}} \{[e^{tJ}Jξ_{i}, e^{tJ}J^{−1}ξ_{j}] − [e^{tJ}Jξ_{i}, e^{tJ}ξ_{j}]\}

+ \frac{1}{2} ∑_{i≠j=1}^{N} \frac{1}{θ_{j}^{2} − θ_{i}^{2}} \{[Jξ_{i}, J^{−1}ξ_{j}] − [ξ_{i}, ξ_{j}]\},

where e^{tJ} = ∑_{n} \frac{t^{n}}{n!} J^{n}, \{±θ_{i}\sqrt{−1},; i = 1, ⋯, p\} are the distinct eigenvalues of J(z) and {ξ_{j}} ⊆ Ker(J(z))^{⊥} satisfies Σ_{j}ξ_{j} = x_{2} and J^{2}(z)ξ_{j} = −θ_{j}^{2}ξ_{j}.

1.8 Remark.

(1) If N is of Heisenberg type, then x_{1} = 0, and ±\sqrt{−1}|z| are the only eigenvalues of J(z), so

x(t) = (cos(t | z |) − 1)J^{−1}x + \frac{sin(t | z |)}{| z |}x,
z_{1}(t) = (1 + \frac{| x |^{2}}{2 | z |^{2}} )z,
z_{2}(t) = −\frac{sin(t | z |) | x |^{2}}{2 | z |^{3}} z

(2) If x = 0 or z = 0, then the geodesics become respectively σ(t) = exp(tz) or σ(t) = exp(tx). More generally, if J(z)x = 0, then the geodesic is given by σ(t) = exp(t(x + z)).

A geodesic σ in N descends to a closed geodesic in Γ\N if and only if there exists γ ∈ Γ and l ∈ ℝ such that σ(t + l) = γσ(t) for all t ∈ ℝ.

1.9 Lemma. ([E]) Let φ = exp(v^{∗} + z^{∗}) be an arbitrary element of N. Let z^{∗∗} be the component of z^{∗} orthogonal to [v^{∗}, V]. Let σ(t) be a unit speed geodesic such that φσ(t) = σ(t + l) for all t ∈ R and some l > 0. Let a = σ(0) and ω^{∗} = (|v^{∗}|^{2} + |z^{∗∗}|^{2})^{\frac{1}{2}} . Then

(1) |v^{∗}| ≤ l ≤ ω^{∗}.

(2) l = ω^{∗} if and only if the following conditions hold:

σ(t) = a exp(\frac{t}{ω^{∗}} (v^{∗} + z^{∗∗})
z^{∗∗} = z^{∗} + [v^{∗}, ξ], where a = exp(ξ).

(3) l = |v^{∗}| if and only if z^{∗∗} = 0 and σ(t) = a · exp(t\frac{v^{∗}}{|v^{∗}|}).

We next recall some basic facts about uniform discrete subgroups of N. See [Rn] for details.

1.10. If Γ is a uniform discrete subgroup of N, then log Γ ∩ Z is a lattice of full rank in the derived algebra Z, and π_{v}(log Γ) is a lattice of full rank in V, where π_{v} : N → V is the projection with kernel Z. For x, y ∈ log Γ, we have exp(x) exp(y) exp(−x) exp(−y) = exp([x, y]) ∈ Γ, so [x, y] ∈ log Γ∩Z. In particular, if we choose a basis of N consisting of elements of log Γ, then the constants of structure are rational. Thus, letting N_{Q} denote the rational span of log Γ, then N_{Q} has the structure of a rational Lie algebra. We will say a subspace of N is rational if it has a basis consisting of elements of log Γ. For example, image(ad(x)) is rational for all x ∈ log Γ.

We conclude this section with a brief discussion of isospectral nilmanifolds. The notion of almost inner automorphisms (1.11 below) will play a key role in the sequel.

1.11 Definition.

(a) Let Γ be a uniform discrete subgroup of a simply-connected nilpotent Lie group N. An automorphism Φ of N is said to be Γ-almost inner if Φ(γ) is conjugate to γ for all γ ∈ Γ. The automorphism is said to be almost inner if Φ(x) is conjugate to x for all x ∈ N .

(b) A derivation ϕ of the Lie algebra N is said to be Γ-almost inner, respectively almost inner, if ϕ(x) ∈ image(ad(x)) for all x ∈ log Γ, respectively, for all x ∈ N.

1.12 Remark. (See [GW])

(a) The Γ-almost inner automorphisms and the almost inner automorphisms form connected Lie subgroups of Aut(N). In many cases, these groups properly contain the group Inn(N) of inner automorphisms. The spaces of Γ-almost inner (respectively, almost inner) derivations of N are the Lie algebras of these groups of automorphisms. In particular, if ϕ is a (Γ-)almost inner derivation, then there exists a one-parameter family Φ_{t} of (Γ-)almost inner automorphisms of N such that Φ_{t}_{∗} = e^{tϕ}. Conversely, if Φ is a (Γ- )almost inner automorphism of N, then Φ_{∗} = e^{ϕ} for some (Γ-)almost inner derivation of N.

(b) Note that a Γ-almost inner derivation ϕ satisfies ϕ(V) ⊆ [N , N] and ϕ(Z) = 0 if Z is central. In particular, if N is 2-step nilpotent, then (letting Z = [N , N] as before), we have ϕ(N) ⊆ Z and ϕ(Z) = 0, so ϕ^{2} = 0. Thus e^{tϕ} = Id +tϕ.

1.13 Proposition. Let (Γ\N, g) be a compact nilmanifold and let Φ be a Γ- almost inner automorphism of N. Then (Φ(Γ)\N, g) is isospectral to (Γ\N, g). Conversely, if N is 2- step nilpotent and if (Γ_{t})_{t≥0} is a continuous family of discrete subgroups of N such that the family of manifolds (Γ_{t}\N, g) are all isospectral, then there exists a family {Φ_{t}} of Γ_{0}-almost inner automorphisms of N such that Γ_{t} = Φ_{t}(Γ_{0}) for all t.

The first statement is proven in [GW] for almost inner automorphisms and in [G] for Γ-almost inner automorphisms. The converse is given in [OP] and [P].

1.14 Remark. If Φ is an inner automorphism of N, say Φ is conjugation by a ∈ N, then (Φ(Γ)\N, g) is isometric to (Γ\N, g). The isometry is induced from the isometry L_{a} of (N, g) given by left translation. However, if ϕ is a Γ-almost inner derivation which is not inner and {Φ_{t}} is the corresponding family of automorphisms, then the deformation Φ_{t}(Γ)\N, g) is non-trivial.

2. Geodesic Conjugacy

Throughout this section, (N, g) will denote an arbitrary simply-connected 2-step nilpotent Lie group with left-invariant Riemannian metric and Γ a uniform discrete subgroup of N. We continue to use the notation N = Z + V introduced in 1.1 for the Lie algebra of N.

2.1 Notation and Remarks.

(a) The left-invariant vector fields on N induce global vector fields on Γ\N. Thus the tangent bundles of both N and Γ\N are completely parallelizable, and the unit tangent bundles may be identified with

S(N, g) = N × S(N)
S(Γ\N, g) = Γ\N × S(N)

where S(N) is the unit sphere in N relative to the Riemannian inner product. We will write S(N) for S(N, g) if g is understood.

(b) For x ∈ N, the left action L_{x} : N → N induces a diffeomorphism (L_{x})_{∗} of S(N). Under the identification above,

(L_{x})_{∗}(n, u) = (xn, u)

for n ∈ N, u ∈ S(N).

(c) Now suppose that (Γ^{∗}\N^{∗}, g^{∗}) is another 2-step nilmanifold and that

F : S(Γ\N, g) → S(Γ^{∗}\N^{∗}, g^{∗})

is a homeomorphism intertwining the geodesic flows. By (a), the fundamental group of S(Γ\N) is isomorphic to Γ, so F induces an isomorphism

F_{∗} : Γ → Γ^{∗}.

2.2 Proposition. ([E]) Suppose (Γ\N, g) and (Γ^{∗}\N^{∗}, g^{∗}) are compact 2-step nilmanifolds and F : S(Γ\N) → S(Γ^{∗}\N^{∗}) is a homeomorphism intertwining their geodesic flows. Then there exists a Γ-almost inner automorphism Φ of N (see definition 1.11) such that (Γ^{∗}\N^{∗}, g^{∗}) is isometric to (Γ\N, g). Moreover the isomorphism F_{∗} : Γ → Γ^{∗} is given by F_{∗} = Ψ_{∗} ∘ Φ|_{Γ} where

Ψ : (Ψ(Γ)\N, g) → (Γ^{∗}\N^{∗}, g^{∗})

is an isometry and Ψ_{∗} : Φ(Γ) → Γ^{∗} is the induced map on fundamental groups.

2.3 Notation and Remarks.

(a) By Proposition 2.2, we may replace (Γ^{∗}\N^{∗}, g^{∗}) by (Φ(Γ)\N, g), so F : S(Γ\N, g) → S(Φ(Γ)\N, g), and we may assume that F_{∗} = Φ|_{Γ} for some Γ-almost inner automorphism Φ of N. Let

F̃ : S(N) → S(N)

be the lift of F. In the notation of 2.1 (b), we have

F̃ ∘ L_{γ}_{∗} = L_{Φ(γ)}_{∗} ∘ F̃,

for all γ ∈ Γ. Also, denoting by G^{t} the geodesic flow of (N, g), we have

F̃ ∘ G^{t} = G^{t} ∘ F̃.

(b) For (n, u) ∈ S(N), write

F̃(n, u) = (exp(A(n, u) + B(n, u))n, I(n, u) + H(n, u))

(c) (See Remark 1.12) There exists a Γ-almost inner derivation of N, which we denote by ϕ, such that the differential Φ_{∗} : N → N is given by Φ_{∗} = Id +ϕ. We have ϕ(Z) = 0 and ϕ(V) ⊆ Z.

2.4 Proposition. We use the notation in 2.3. For (n, u) ∈ S(N) and γ ∈ Γ, (a) B(γn, u) = B(n, u),

(b) H(γn, u) = H(n, u),

(c) I(γn, u) = I(n, u),

(d) A(γn, u) = A(n, u) + ϕ(log γ) − [B(n, u), log γ].

Proof. By 2.1 (b) and 2.3, we have

(exp(A(γn, u) + B(γn, u))γn, I(γn, u) + H(γn, u))
= F̃(γn, u) = L_{Φ(γ)}_{∗}F̃(n, u)
= (Φ(γ) exp(A(n, u) + B(n, u))n, I(n, u) + H(n, u)).

This yields (b) and (c). Using 2.3 (c) and the fact that

exp(x) exp(y) = exp(x + y + \frac{1}{2}[x, y])

for x, y ∈ N , a straight-forward computation yields (a) and (d).

2.5 Proposition. For a unit vector u ∈ N, let σ(t, u) be the geodesic in N with σ(0, u) = e and σ̇(0, u) = u. Write σ(t, u) = exp(X(t, u)+Z(t, u)) with X(t, u) ∈ V and Z(t, u) ∈ Z. Defining J : Z → 𝔰o(V) as in 1.1 and letting G^{t} be the geodesic flow of (N, g), we have:

(a) I(G^{t}(n, u)) = e^{tJ(H(n,u))}I(n, u),

(b) H(G^{t}(n, u)) = H(n, u),

(c) B(G^{t}(n, u)) = B(n, u) + X(t, I(n, u) + H(n, u)) − X(t, u)

Proof. The proposition follows from the expression for the geodesics in Lemma 1.7.

As seen in Lemma 1.7, if u ∈ V or u ∈ Z, then the geodesic σ(t, u) in N with σ(0, u) = e, σ̇(0, u) = u is given by σ(t, u) = exp(tu). For n ∈ N, the geodesic with initial conditions (n, u) is just the translation n exp(tu). The corresponding orbit of the geodesic flow in N is the curve (n exp(tu), u).

2.6 Proposition. In the notation of 1.1 and 2.3, we have:

(a) F : Γ\N × {z} → Φ(Γ)\N × {z} for all z ∈ Z. That is, H(n, z) = z and I(n, z) = 0.

(b) F : Γ\N × {v} → Φ(Γ)\N × {v} for all v ∈ Z. That is, H(n, v) = 0 and I(n, v) = v.

2.7 Lemma. ([E]) Let v ∈ π_{v}(log Γ) and ϵ > 0 be given. Then there exists an element ξ ∈ log Γ such that ξ = kv + z_{0} for some positive integer k and some element z_{0} ∈ Z with |z_{0}| < ϵ.

Proof of Proposition 2.6. Recall that free homotopy classes of closed curves in Γ\N correspond to conjugacy classes in Γ. For γ ∈ Γ (respectively, Φ(Γ)), we will denote by [γ] (respectively, [γ]^{∗}) the corresponding free homotopy class.

(a) Let

Z_{u} = {z ∈ Z, ∥z∥ = 1}
Z_{Γ} = {z ∈ Z_{u} |rz ∈ log(Γ), for some r > 0}

Since Γ is a uniform lattice, Z_{Γ} is dense in Z_{u}. Thus we need only show that H(n, z) = z for z ∈ Z_{Γ}.

The almost inner automorphism Φ restricts to the identity on the center of N. In particular, log Γ ∩ Z = log Φ(Γ) ∩ Z and Z_{Γ} = Z_{Φ(Γ)}. Let z_{0} ∈ Z_{Γ} and let r_{0} satisfy r_{0}z_{0} ∈ log Γ. By Lemma 1.9 (with v^{∗} = 0), the longest geodesics in the free homotopy class [r_{0}z_{0}] are precisely the projections to Γ\N of exp(tz_{0}) and all its left translations. Thus the submanifold Γ\N ×{z_{0}} of S(Γ\N) is foliated by all the longest periodic orbits of the geodesic flow in the free homotopy class [r_{0}z_{0}] (viewed now as a free homotopy class of curves in S(Γ\N)). Similarly Φ(Γ)\N × {z_{0}} is foliated by the longest periodic orbits of the flow in the class [r_{0}z_{0}]^{∗} of S(Φ(Γ)\N). Hence the geodesic conjugacy F must map Γ\N × {z_{0}} onto Γ^{∗}\N × {z_{0}}. This proves (a).

(b) The proof of (b) is similar but considerably more complicated. Let

V_{u} = {v ∈ V| ∥v∥ = 1}
V_{Γ} = {v ∈ V_{u}|rv ∈ π_{v}(log Γ), for some r > 0}

where π_{v} : N ⟶ V is the orthogonal projection. Since Γ is a uniform lattice, V_{Γ} is dense in V_{u} and we only need to show that I(n, v) = v for all v ∈ V_{Γ}.

For a fixed v ∈ V_{Γ}, there is an r > 0, such that rv ∈ π_{v}(log Γ). According to Lemma 2.7, for ϵ > 0, there is a positive integer k_{ϵ} and an element z_{ϵ} ∈ z with ∥z_{ϵ}∥ < ϵ such that

(2.6.1)  k_{ϵ}rv + z_{ϵ} ∈ log(Γ).

Decompose

(2.6.2)  z_{ϵ} = z_{ϵ,0} + z_{ϵ,1}

with z_{ϵ,0} ∈ Im(ad(v))^{⊥} and z_{ϵ,1} ∈ Im(ad(v)). Let

(2.6.3)  v_{0} = \frac{k_{ϵ}rv}{∥k_{ϵ}rv + z_{ϵ,0}∥}

(2.6.4)  z_{0} = \frac{z_{ϵ,0}}{∥k_{ϵ}rv + z_{ϵ,0}∥}

For n ∈ N, the curve n exp t(v_{0} + z_{0}) is a geodesic (see Lemma 1.7). Set

N_{0} = {n ∈ N | γ_{n}(t) = n exp t(v_{0} + z_{0}) descends to a closed geodesic in Γ\N}.

We are going to prove that N_{0} is dense in N.

For n = exp(x + w) ∈ N, x ∈ V, w ∈ Z, we get,

γ_{n}(t) = n exp(v_{0} + z_{0}) = exp(tv_{0} + x + w + \frac{t}{2}[x, v_{0}] + tz_{0}),
γ_{n}(t + l) = exp((t + l)v_{0} + x + w + \frac{t + l}{2} [x, v_{0}] + (t + l)z_{0})

= exp(lv_{0} + l[x, v_{0}] + lz_{0}) exp(x + tv_{0} + w + \frac{t}{2}[x, v_{0}] + tz_{0})

= exp(lv_{0} + l[x, v_{0}] + lz_{0})γ_{n}(t).

Therefore, γ_{n}(t) descends to a closed geodesic in Γ\N if and only if

(2.6.5)  exp(lv_{0} + l[x, v_{0}] + lz_{0}) ∈ Γ.

So,

N_{0} = {n = exp(x + w) ∈ N | exp(lv_{0} + l[x, v_{0}] + lz_{0}) ∈ Γ for some l ∈ ℝ}.

By (2.6.1) - (2.6.4), we have

∥k_{ϵ}rv + z_{ϵ,0}∥v_{0} + ∥k_{ϵ}rv + z_{ϵ,0}∥z_{0} + z_{ϵ,1} ∈ log Γ.

Since z_{ϵ,1} ∈ Im(ad(v)), there is a u ∈ V such that [u, v_{0}] = \frac{z_{ϵ,1}}{∥k_{ϵ}rv+z_{ϵ,0}∥}. Letting r_{1} = ∥k_{ϵ}rv + z_{ϵ,0}∥, we get

(2.6.6)  r_{1}(v_{0} + [u, v_{0}] + z_{0}) ∈ log(Γ)

Let Z_{Q} be the rational span of Z ∩ log(Γ); i.e., Z_{Q} = Z ∩ N_{Q} in the notation of 1.10. By the remark in 1.10, Z_{Q} ∩ Im(ad(v_{0})) is dense in Im(ad(v_{0})) and thus so is \frac{1}{r_{1}} (Z_{Q} ∩ Im(ad(v_{0}))).

Note that

V = ker(ad(v_{0})) ⨁ ker(ad(v_{0}))^{⊥}

and

ad(v_{0}) : ker(ad(v_{0}))^{⊥} → Im(ad(v_{0}))

is an isomorphism. So, ad(v_{0})^{−1}(r_{1}^{−1}(Z_{Q} ∩ Im(ad(v_{0})))) is dense in ker(ad(v_{0})^{⊥}.

Denote

A(v_{0}) = ad(v_{0})^{−1}(r_{1}^{−1}(Z_{Q} ∩ Im(ad(v_{0}))).

Fix u such that r_{1}(v_{0} + [u, v_{0}] + z_{0}) ∈ log Γ. Then, u + A(v_{0}) + ker(ad(v_{0})) is dense in V.

For ū ∈ {u + A(v_{0}) + ker(ad(v_{0}))}, we get,

[ū, v_{0}] = [u, v_{0}] mod (r_{1}^{−1}Z_{Q}),

that is,

r_{1}[ū, v_{0}] = r_{1}[u, v_{0}] mod (Z_{Q}).

Then there is an integer q such that

qr_{1}[ū, v_{0}] = qr_{1}[u, v_{0}] mod (log Γ).

Therefore, by (2.6.6),

qr_{1}(v_{0} + [ū, v_{0}] + z_{0}) ∈ log Γ.

Noting that n exp(Z) ⊆ N_{0} whenever n ∈ N_{0}, it follows from (2.6.5) that N_{0} is dense in N.

Now, for n = exp(x + w) ∈ N_{0}, γ_{n}(t) = n · exp(v_{0} + z_{0}) descends to a closed geodesic in Γ\N, say of length l. This is the longest closed geodesic in the free homotopy class [exp(lv_{0} + l[x, v_{0}] + lz_{0})]. Thus F sends γ_{n}(t) to a closed geodesic in Φ(Γ)\N in the free homotopy class [exp(lv_{0} + l[x + ξ, v_{0}] + lz_{0})] which must be the longest one, too.

According to Lemma 1.9, this geodesic must have the form: n_{∗} exp(t(v_{0} + z_{0})) for some n_{∗} ∈ N, that is, F̃(n, v_{0} + z_{0}) = (n_{∗}, v_{0} + z_{0}). Therefore, we have proved that

H(n, v_{0} + z_{0}) = z_{0} and I(n, v_{0} + z_{0}) = v_{0}, for n ∈ N_{0}.

Since N_{0} is dense in N, the above identities hold for all n ∈ N.

Let ϵ → 0 in (2.6.3). Recalling that ∥z_{ϵ,0}∥ < ϵ and k_{ϵ} > 0 is an integer, we have

∥z_{0}∥ ≤ \frac{∥z_{ϵ,0}∥}{rk_{ϵ}} ≤ \frac{z_{ϵ,0}}{r} → 0.
v_{0} = \frac{rk_{ϵ}v}{∥rk_{ϵ}v + z_{ϵ,0}∥} = \frac{v}{∥v + \frac{z_{ϵ,0}}{rk_{ϵ}} ∥} → v.

So,

I(n, v) = lim_{ϵ→0} I(n, v_{0}) = v and H(n, v) = 0

for all v ∈ V_{Γ}. As noted above, the proposition follows.

2.8 Proposition. For n ∈ N, v ∈ V, z ∈ Z,

\begin{matrix}A(G^{t}(n, v)) = A(n, v)  A(G^{t}(n, z)) = A(n, z) \\ B(G^{t}(n, v)) = B(n, v)  B(G^{t}(n, z)) = B(n, z) \end{matrix}

Proof. Immediate from Propositions 2.5 and 2.6.

2.9 Notation and Remarks. The derived group [N, N] of N is a simply-connected central subgroup with Lie algebra [N , N] = Z, and exp : Z → [N, N] is a vector space isomorphism. By the remarks in 1.10, Γ∩[N, N]\[N, N] is a torus; we denote it by T. Note that T acts isometrically on Γ\N by left translations.

By Proposition 2.4, B(γn, u) = B(n, u), I(γn, u) = I(n, u) for all γ ∈ Γ, and A(γn, u) = A(n, u) for γ ∈ [N, N] ∩ Γ. Thus we can define the averages over T:

B̄(n, u) = ∫_{T} B(x · n, u) dx,
Ā(n, u) = ∫_{T} A(x · n, u) dx.
Ī(n, u) = ∫_{T} I(x · n, u) dx.

for (n, u) ∈ S(N). Here dx is the Haar measure of total volume one.

2.10 Proposition. In the notation of 1.1, 2.3 and 2.9,

ϕ(v) = [B̄(n, \frac{v}{|v|}), v],

B̄(n, v) = B̄(e, v)

for all (n, v) ∈ S(N) with v ∈ V.

Proof. By Proposition 2.4, for all n ∈ N, v ∈ V, with ∥v∥ = 1, and γ ∈ Γ, we have

ϕ(log γ) = A(γn, v) − A(n, v) + [B(n, v), log γ],

and therefore,

ϕ(log γ) = Ā(γn, v) − Ā(n, v) + [B̄(n, v), log γ].

Letting π_{v} : N → V be the orthogonal projection and recalling that ϕ(z) = 0, we thus have, for η ∈ π_{v}(log(Γ)),

ϕ(η) = Ā(exp(η)n, v) − Ā(n, v) + [B̄(n, v), η].

Next note that N/[N, N] is a simply-connected abelian Lie group. Letting π : N → N/[N, N] be the projection, then π(Γ)\π(N) is a torus T̄. The Lie algebra of N/[N, N] is N \Z. Under the identification of N \Z with V, the exponential map V → N/[N, N] carries π_{v}(log Γ) isometrically to π(Γ). Thus the torus T̄ is isomorphic to π_{v}(log Γ)\V.

Since B̄(yn, v) = B̄(n, v) for y ∈ [N, N] and B̄(γn, v) = B̄(n, v) by Proposition 2.4, the map B̄(·, v) may be viewed as a map on the torus T̄. Since G^{t}(n, v) = (n exp(tv), v), Proposition 2.8 implies that B̄(n exp(tv), v) = B̄(n, v) for all t ∈ ℝ. In particular, if v ∈ V is a unit vector such that the projection of v to the torus π_{v}(log Γ)\V ≅ T̄ is a generator of the torus - i.e., the projection of {tv|t ∈ ℝ} is dense in the torus - then

B̄(n, v) = B̄(n exp(tv), v) ≡ B̄(e, v).

By Kronecker’s theorem, the generators form a dense set in the torus. Hence

B̄(n, v) ≡ B̄(e, v)

for all unit vectors v ∈ V. (Note: we can’t expect to have Ā(n, v) = Ā(e, v) since Ā(γn, v) ≠ Ā(n, v) for general γ ∈ Γ .)

Hence, for η ∈ π_{v}(log Γ), n ∈ N, and v ∈ V with ∥v∥ = 1,

ϕ(η) = Ā(exp(η)n, v) − Ā(n, v) + [B̄(e, v), η].

From this identity, we get that for all n ∈ N,

Ā(exp(η)n, v) − Ā(n, v) = Ā(exp(η), v) − Ā(e, v).

In particular, for ξ ∈ π_{v} log(Γ),

Ā(exp(η) exp(ξ), v) − Ā(exp(ξ)), v) = Ā(exp(η), v) − Ā(e, v).

Therefore,

Ā(exp(η) exp(ξ), v) − Ā(e, v) = (Ā(exp(η), v) − Ā(e, v)) + (Ā(exp(ξ)), v) − Ā(e, v)).

Let

E(η, v) = (Ā(exp(η), v) − Ā(e, v)), for η ∈ π_{v} log Γ.

Then,

E(η + ξ, v) = E(η, v) + E(ξ, v), for η , ξ ∈ π_{v} log(Γ).

Let

V_{Q} = span_{Q}{π_{v} log Γ} ⊆ V

and let v_{1}, ⋯, v_{p} ∈ π_{v} log Γ be a basis of V_{Q}.

For w = Σ_{i} q_{i}v_{i} = \frac{1}{k} Σ_{i} k_{i}v_{i} ∈ V_{Q}, where k, k_{i} are integers, and for all v ∈ V with ∥v∥ = 1,

ϕ(w) = \frac{1}{k} ϕ(Σ_{i}k_{i}v_{i}) = \frac{1}{k} E(Σ_{i}k_{i}v_{i}, v) + [B̄(e, v), w].

In particular, for v = \frac{w}{∥w∥} = \frac{Σ_{i}k_{i}v_{i}}{∥Σ_{i}k_{i}v_{i}∥}, since

Ā(exp(tv), v) = Ā(e, v),

we get,

ϕ(w) = \frac{1}{k} (Ā(exp(Σ_{i}k_{i}v_{i}), \frac{Σ_{i}k_{i}v_{i}}{∥Σ_{i}k_{i}v_{i}∥}) − Ā(e, \frac{Σ_{i}k_{i}v_{i}}{∥Σ_{i}k_{i}v_{i}∥})) + [B̄(e, \frac{w}{∥w∥}), w]

= [B̄(e, \frac{w}{∥w∥}), w].

Noting that V_{Q} is dense in V and B̄ is continuous, we have proven that for all v ∈ V with v ≠ 0,

ϕ(v) = [B̄(e, \frac{v}{∥v∥}), v].

Proposition 2.10 gives us a strengthening of Eberlein’s result (Proposition 2.2) as follows:

2.11 Definition. Let ϕ be an almost inner derivation of a 2-step nilpotent Lie algebra N. By the definition, there exists a map ξ : N → N such that ϕ(x) = [ξ(x), x] for all x ∈ N. Note that ξ is not uniquely defined. For x ∈ Z, ξ(x) is completely arbitrary, thus only the values of ξ on V are of interest. We will say ϕ is of continuous type if ξ can be chosen to be continuous on V \ {0}. We will also say the corresponding almost inner automorphism Φ of N (with differential Id +ϕ as in 1.12 (a)) is of continuous type.

2.12 Theorem. Suppose (Γ\N, g) and (Γ^{∗}\N^{∗}, g^{∗}) are compact 2-step nilmanifolds whose geodesic flows are C^{0}-conjugate. Then there exists an almost-inner automorphism Φ of N of continuous type such that (Γ^{∗}\N^{∗}, g^{∗}) is isometric to (Φ(Γ)\N, g).

The theorem follows immediately from Propositions 2.2 and 2.10.

2.13 Corollary. There exist 2-step nilmanifolds (Γ\N, g) satisfying:

(a) Any 2-step nilmanifold whose geodesic flow is C^{0}-conjugate to (Γ\N, g) must be isometric to (Γ\N, g).

(b) (Γ\N, g) is isospectrally deformable, i.e., there exists a continuous family M_{t} of 2-step nilmanifolds with M_{0} = (Γ\N, g) such that M_{t} is isospectral but not isometric to M_{0} for all t.

Proof. We only need to find a nilmanifold (Γ\N, g) such that (i) every almost inner derivation of N of continuous type is inner and (ii) N admits a Γ-almost inner derivation ϕ which is non-inner. By Remark 1.14 and Theorem 2.12, condition (i) implies (a). By 1.12, 1.13 and 1.14, condition (ii) implies (b). We will exhibit an example below.

2.14 Example. (see [GM] and [GW]). Let N be the six-dimensional simplyconnected nilpotent Lie group with Lie algebra

N = span{X_{1}, X_{2}, Y_{1}, Y_{2}, Z_{1}, Z_{2}}

with

[X_{1}, Y_{1}] = [X_{2}, Y_{2}] = Z_{1}, [X_{1}, Y_{2}] = Z_{2}

and with all other brackets of basis vectors trivial.

The almost inner derivations of N are the linear maps which send X_{2} and Y_{1} to multiples of Z_{1}, send X_{1} and Y_{2} into span{Z_{1}, Z_{2}} and send Z_{1} and Z_{2} to 0. These form a 6-dimensional subspace containing the inner derivations as a fourdimensional subspace. The derivations ϕ_{1}, respectively ϕ_{2}, which send X_{1}, respectively Y_{2}, to Z_{2} and send all other basis vectors to zero span a two-dimensional family of non-inner almost inner derivations. No almost inner derivation of N is of continuous type. If M = (Γ\N, g) is any nilmanifold associated with N, then there exists a continuous two-parameter family of manifolds strongly Laplace isospectral to M but not C^{0}-geodesically conjugate to M.

See [GM] for further examples.

3. Special Classes of Nilmanifolds

In this section, we focus on the special classes of 2-step nilmanifolds introduced in Definition 1.3. We shall prove the following:

3.1 Theorem. If the geodesic flow of a nilmanifold (Γ^{∗}\N^{∗}, g^{∗}) is C^{0}-conjugate to that of a nilmanifold (Γ\N, g) of Heisenberg type, then (Γ^{∗}\N^{∗}, g^{∗}) is isometric to (Γ\N, g).

3.2 Theorem. If the geodesic flow of a 2-step nilmanifold (Γ^{∗}\N^{∗}, g^{∗}) is C^{2}- conjugate to that of a 2-step nilmanifold (Γ\N, g) which is strongly in resonance, then (Γ^{∗}\N^{∗}, g^{∗}) is isometric to (Γ\N, g).

3.3 Theorem. If the geodesic flow of a 2-step nilmanifold (Γ^{∗}\N^{∗}, g^{∗}) is C^{2}- conjugate to that of an irrational 2-step nilmanifold (Γ\N, g), then (Γ^{∗}\N^{∗}, g^{∗}) is isometric to (Γ\N, g).

Note that irrationality is a generic condition for 2-step nilmanifolds, so Theorem 3.3 says that generic 2-step nilmanifolds are geodesically rigid.

We continue to use the conventions of 2.3. In particular, we replace Γ^{∗}\N^{∗} by (Φ(Γ)\N, g) where Φ is a Γ-almost inner automorphism of N. To prove Theorems 3.1 - 3.3, we must in each case show that the associated almost inner derivation ϕ is inner. The key to the proofs is the following result, valid for arbitrary 2-step nilmanifolds.

3.4 Proposition. Let (Γ\N, g) be a 2-step nilmanifold and let ϕ be a Γ-almost inner derivation of the associated Lie algebra N. Thus in the notation of 1.1, there exists a map ξ : V → V such that ϕ(x) = [ξ(x), x] for all x ∈ V. If ξ can be chosen so that ξ(e^{J(z)}x) = ξ(x) for all x ∈ V, z ∈ Z, then ϕ is an inner derivation.

Proof. Pick v_{1} ∈ V and denote by V_{1} the subspace of V spanned by all vectors

{e^{J(z_{1})}e^{J(z_{2})} ⋯ e^{J(z_{k})}v_{1}| z_{1}, z_{2}, ⋯, z_{k} ∈ Z, k = 0, 1, 2, ⋯ }.

Then, pick v_{2} ∈ V such that v_{2} ⊥ V_{1}. Denote by V_{2} the subspace of V spanned by

{e^{J(z_{1})}e^{J(z_{2})} ⋯ e^{J(z_{k})}v_{2}| z_{1}, z_{2}, ⋯, z_{k} ∈ Z, k = 0, 1, 2, ⋯ }.

We claim that

(3.4.1)  V_{1} ⊥ V_{2} and [V_{1}, V_{2}] = 0.

In fact, let x = e^{J(z_{1})} ⋯ e^{J(z_{k})}v_{1}, y = e^{J(z̃_{1})} ⋯ e^{J(z̃_{l})}v_{2}. Then

< x, y > =< e^{J(z_{1})} ⋯ e^{J(z_{k})}v_{1}, e^{J(z̃_{1})} ⋯ e^{J(z̃_{l})}v_{2} >
=< e^{J(−z̃_{1})} ⋯ e^{J(−z̃_{l})}e^{J(z_{1})} ⋯ e^{J(z_{k})}v_{1}, v_{2} >
= 0,

since v_{2} ⊥ V_{1}.

Similarly, we can prove that for any z ∈ Z, < e^{tJ(z)}x, y >= 0. Taking the derivative with respect to t at t = 0 in this identity, we get < J(z)x, y >= 0, that is < z, [x, y] >= 0. Since z ∈ Z is arbitrary, we conclude that [x, y] = 0. Therefore, we have confirmed our claim.

Repeating this process, we can find a finite sequence of vectors v_{1}, v_{2}, ⋯, v_{p} ∈ V such that

V = V_{1} ⨁ V_{2} ⨁ ⋯ ⨁ V_{p},

where V_{i} is the subspace spanned by {e^{J(z_{1})}e^{J(z_{2})} ⋯ e^{J(z_{k})}v_{i}| z_{1}, z_{2}, ⋯, z_{k} ∈ Z, k = 0, 1, 2, ⋯ }. Moreover, V_{i} ⊥ V_{j} and [V_{i}, V_{j}] = 0 for i ≠ j.

Since ϕ is linear and ξ(e^{J(z)}v) = ξ(v), it is easy to see that

ϕ(x_{i}) = [ξ(v_{i}), x_{i}], for x_{i} ∈ V_{i}.

Let ξ_{i} be the projection of ξ(v_{i}) to V_{i}. We get,

(3.4.2)  ϕ(x_{i}) = [ξ_{i}, x_{i}], for x_{i} ∈ V_{i}.

Now, for x ∈ V, we write x = x_{1} + x_{2} + ⋯ + x_{p}, where x_{i} ∈ V_{i}. By 3.4.1 and 3.4.2,

ϕ(x) = Σ_{i}ϕ(x_{i})
= Σ_{i}[ξ_{i}, x_{i}]
= Σ_{i}[ξ, x_{i}]
= [ξ, x]

This proves that ϕ is an inner derivation of N.

We next prove a technical result valid for arbitrary non-singular 2-step nilmanifolds. This Lemma will be used in the proofs of Theorems 3.2 and 3.3.

3.5 Lemma. We use the notation of 2.1 and 2.3 and assume that (Γ\N, g) is a non-singular 2-step nilmanifold and F : S(Γ\N) → S(Φ(Γ)\N) is a C^{2}-geodesic conjugacy. Then for (n, v + z) ∈ S(N), with v ∈ V, z ∈ Z, we have:

(a) lim_{s→0} \frac{H(n,cos(s)v+sin(s)z)}{s}

= \frac{d}{ds}|_{s=0}H(n, cos(s)v + sin(s)z) = z.

(b) lim_{s→0} \frac{d}{ds}(\frac{H(n,cos(s)v+sin(s)z)}{s}) = 0.

(c) e^{tJ(z)} \frac{d}{ds}|_{s=0}I(n, cos(s)v + sin(s)z)

= lim_{s→0} \frac{d}{dλ}|_{λ=0}I(nσ(\frac{t}{s}, cos(s)v + sin(s)z), e^{tJ(z)} cos(λ)v + sin(λ)z). where σ(t, v + z) is the geodesic in N with σ(0) = e, σ′(0) = v + z.

Proof. (a) The first equality in (i) follows from Proposition 2.6. In particular, the limit exists. Write v(s) = cos(s)v and z(s) = sin(s)z. According to Proposition 2.5,

I(nσ(t, v(s) + z(s)), e^{tJ(z(s))}v(s) + z(s))
=e^{tJ(H(n,v(s)+z(s)))}I(n, v(s) + z(s)).

Replacing t by \frac{t}{s} in the above identity, we get

(3.5.1) I(nσ(\frac{t}{s}, v(s)+z(s)), e^{tJ( \frac{z(s)}{s})}v(s)+z(s)) = e^{tJ( \frac{H(n,v(s)+z(s))}{s})}I(n, v(s)+z(s)).

18 CAROLYN GORDON AND YIPING MAO

Now, for an arbitrary sequence s_{n} → 0, there exists a subsequence s_{n_{i}} such that nσ(\frac{t}{s_{n_{i}}} , v(s_{n_{i}}) + z(s_{n_{i}})) → g mod(Γ) for some element g in N( g may depend on the subsequence). Therefore, from I(γn, v + z) = I(n, v + z) for γ ∈ Γ, we get,

lim_{i→∞}I(nσ(\frac{t}{s_{n_{i}}} , v(s_{n_{i}}) + z(s_{n_{i}})), e^{tJ( \frac{z(sn_{i})}{sn_{i}})}v(s_{n_{i}}) + z(s_{n_{i}}))
=I(g, e^{tJ(z)}v) = e^{tJ(z)}v.

Therefore,

lim_{s→0} I(nσ(\frac{t}{s}, v(s) + z(s)), e^{tJ( \frac{z(s)}{s})}v(s) + z(s)) = e^{tJ(z)}v.

Hence by (3.5.1) and Proposition 2.6,

lim_{s→0} e^{tJ( \frac{H(n,v(s)+z(s))}{s})}v = lim_{s→0} e^{tJ( \frac{H(n,v(s)+z(s))}{s})}I(n, v(s) + z(s)) = e^{tJ(z)}v,

so,

e^{tJ(lim_{s→0} \frac{H(n,v(s)+z(s))}{s})}v = e^{tJ(z)}v.

Therefore,

J(lim_{s→0} \frac{H(n, v(s) + z(s))}{s})v = J(z)v,

that is,

J(lim_{s→0} \frac{H(n, v(s) + z(s))}{s} − z)v = 0.

Since N is nonsingular, lim_{s→0} \frac{H(n,v(s)+z(s))}{s} = z.

(b) First, we show that lim_{s→0} \frac{d}{ds}(\frac{H(n,cos(s)v+sin(s)z)}{s}) exists.

In fact, by (a), lim_{s→0} \frac{H(n,cos(s)v+sin(s)z)}{s} = z, so

lim_{s→0} \frac{d}{ds}(\frac{H(n, cos(s)v + sin(s)z)}{s})

= lim_{s→0} \frac{1}{s}(\frac{H(n, cos(s)v + sin(s)z)}{s} − z)

= \frac{1}{2} \frac{d^{2}}{ds^{2}} |_{s=0}H(n, cos(s)v + sin(s)z).

The last equality holds since by Taylor’s formula, we have

\frac{1}{s}(\frac{H(n, cos(s)v + sin(s)z)}{s} − z)

= \frac{1}{2} \frac{d^{2}}{ds^{2}} H(n, cos(s̄)v + sin(s̄)z) for some 0 ≤ s̄ ≤ s.

Now, taking the derivative with respect to s in the identity (3.5.1) and then letting s → 0, we get,

\frac{d}{ds}|_{s=0}I(nσ(\frac{t}{s}, v(s) + z(s)), e^{tJ( \frac{z(s)}{s})}v(s) + z(s))
=te^{tJ(z)}J(\frac{d}{ds}|_{s=0} \frac{H(n, v(s) + z(s))}{s})v
+e^{tJ(z)} \frac{d}{ds}|_{s=0}I(n, v(s) + z(s)).

Equivalently,

J(\frac{d}{ds}|_{s=0} \frac{H(n, v(s) + z(s))}{s})v
= − \frac{1}{t} \frac{d}{ds}|_{s=0}I(n, v(s) + z(s))
(3.5.2)  +\frac{1}{t}e^{−tJ(z)} \frac{d}{ds}|_{s=0}I(nσ(\frac{t}{s}, v(s) + z(s)), e^{tJ( \frac{z(s)}{s})}v(s) + z(s)).

Now, by Proposition 2.6,

\frac{d}{ds}|_{s=0}I(nσ(\frac{t}{s}, v(s) + z(s)), e^{tJ( \frac{z(s)}{s})}v(s) + z(s))
= lim_{s→0} \frac{I(nσ(\frac{t}{s}, v(s) + z(s)), e^{tJ( \frac{z(s)}{s})}v(s) + z(s)) − e^{tJ(z)}v}{s}
= lim_{s→0} \frac{I(nσ(\frac{t}{s}, v(s) + z(s)), e^{tJ( \frac{z(s)}{s})}v(s) + z(s)) − e^{tJ( \frac{z(s)}{s})}v(s)}{s}
= lim_{s→0} \frac{I(nσ(\frac{t}{s}, v(s) + z(s)), e^{tJ( \frac{z(s)}{s})}v(s) + z(s)) − I(nσ(\frac{t}{s}, v(s) + z(s)), e^{tJ( \frac{z(s)}{s})}v(s))}{s}
= lim_{s→0} \frac{d}{dλ}|_{λ=s̄}I(nσ(\frac{t}{s}, v(s) + z(s)), e^{tJ( \frac{z(s)}{s})}v(s) + z(λ)) where 0 ≤ s̄ ≤ s

Using the above equality and the facts that I(γn, v + z) = I(n, v + z) and F is C^{2}, we can prove that

\frac{d}{ds}|_{s=0}I(nσ(\frac{t}{s}, v(s) + z(s)), e^{tJ( \frac{z(s)}{s})}v(s) + z(s))

is bounded for all t ∈ ℝ. Furthermore, since the e^{tJ(z)} are unitary operators, the right hand side of (3.5.2) goes to 0 when t → ∞. Therefore, the identity (3.5.2) becomes

J(\frac{d}{ds}|_{s=0} \frac{H(n, v(s) + z(s))}{s})v = 0

Since N is non-singular, we get \frac{d}{ds}|_{s=0} \frac{H(n,v(s)+z(s))}{s} = 0.

(c) Putting J(\frac{d}{ds}|_{s=0} \frac{H(n,v(s)+z(s))}{s})v = 0 back into the identity (3.5.2), we get (c).

We now turn to Theorem 3.1.

3.6 Lemma. In the notation of 2.3, if N is of Heisenberg type, then H(n, v+z) ≡ z for all (n, v + z) ∈ SN, where v ∈ V, z ∈ Z.

Proof. First, we take (n, v + z) ∈ S(N) such that

\frac{2πk}{|z|} (1 + \frac{|v|^{2}}{2|z|^{2}} )z ∈ log(Γ ∩ [N, N])

for some integer k (take k to be minimum). The set of such points (n, v + z) are dense in S(N), so, we only need to prove that H(n, v + z) = z for these points. Let

γ = exp(\frac{2πk}{|z|} (1 + \frac{|v|^{2}}{2|z|^{2}} )z) ∈ Γ ∩ [N, N].

Since, by 1.8,

G^{t}(n, v + z)
=(n exp[(e^{tJ(z)} − Id)J^{−1}(z)v + t(1 + \frac{|v|^{2}}{2|z|^{2}} )z − \frac{sin(t|z|)|v|^{2}}{2|z|^{3}} z], e^{tJ(z)}v + z),

and since

e^{2πkJ( \frac{z}{|z|})} = Id,

we see that

G^{\frac{2πk}{|z|}} (n, v + z) = dL_{γ}(n, v + z).

So,

F̃G^{\frac{2πk}{|z|}} = F̃ dL_{γ}(n, v + z),

and since Φ(γ) = γ, as γ is central, 2.3 implies that

G^{\frac{2πk}{|z|}} F̃(n, v + z) = dL_{γ} F̃(n, v + z).

Now, denote

F̃(n, v + z) = (n_{1}, v_{1} + z_{1})

and

t_{0} = \frac{2πk}{|z|} .

We have,

G^{t_{0}} F̃(n, v + z) =
(n_{1} exp[(e^{t_{0}J(z_{1})} − Id)J^{−1}(z_{1})v + t_{0}(1 + \frac{|v_{1}|^{2}}{2|z_{1}|^{2}} )z_{1} − \frac{sin(t_{0}|z_{1}|)|v_{1}|^{2}}{2|z_{1}|^{3}} z_{1}], e^{t_{0}J(z_{1})}v_{1} + z_{1}),

and

dL_{γ} F̃(n, v + z) = (γn_{1}, v_{1} + z_{1}).

Comparing these identities, we see that

e^{t_{0}J(z_{1})} = Id

and

t_{0}(1 + \frac{|v_{1}|^{2}}{2|z_{1}|^{2}} )z_{1} = t_{0}(1 + \frac{|v|^{2}}{2|z|^{2}} )z.

That is,

e^{t_{0}J(z)} = e^{t_{0}J(z_{1})} = Id,

and

z_{1} = λz

for some positive real number λ. Note that λ must be a positive integer since e^{t_{0}J(z)} = e^{λt_{0}J(z)}.

Now, we have proved that for each (n, v + z) ∈ S(N), there exists a positive integer λ such that H(n, v+z) = λz. Since S(N) is connected, this λ is independent of (n, v + z). Furthermore, λ = 1 since H(n, z) = z by Proposition 2.5.

Proof of Theorem 3.1. As in 2.3, we may assume that (Γ^{∗}\N^{∗}, g^{∗}) = (Φ(Γ)\N, g) for some Γ-almost inner automorphism Φ of N. We only need to show that the corresponding Γ-almost inner derivation ϕ is inner.

According to Proposition 2.10,

ϕ(v) = [B̄(e, \frac{v}{|v|}), v], for 0 ≠ v ∈ V.

On the other hand, since ϕ is linear, we get

ϕ(v) = −ϕ(−v)

= −[B̄(e, −\frac{v}{|v|}), −v]

= [B̄(e, −\frac{v}{|v|}), v]

Therefore, Writing

B̃(v) = \frac{1}{2}(B̄(e, \frac{v}{|v|}) + B̄(e, −\frac{v}{|v|}),

we get

ϕ(v) = [B̃(v), v], for 0 ≠ v ∈ V.

According to Proposition 3.4, we only need to show that

B̃(e^{J(z)}v) = B̃(v)

for all z ∈ Z, v ≠ 0 ∈ V.

By Proposition 2.5, Lemma 1.7 and Lemma 3.6, we get, for (n, v + z) ∈ S(N),

(3.1.1) B(G^{t}(n, v + z)) = B(n, v + z) + (e^{tJ(z)} − Id)J^{−1}(z)(I(n, v + z) − v),

and

(3.1.2) B(G^{t}(n, −v + z)) = B(n, −v + z) + (e^{tJ(z)} − Id)J^{−1}(z)(I(n, −v + z) + v).

By Proposition 2.5 and Lemma 3.6,

I(G^{t}(n, v + z)) = e^{tJ(z)}I(n, v + z).

Taking t = \frac{π}{|z|}, we get

(3.1.3)  I(nσ(\frac{π}{|z|}, v + z), −v + z) = −I(n, v + z),

where σ(t, v + z) is the geodesic in N with σ(0) = e, σ′(0) = v + z.

By (3.1.1) - (3.1.3), we have

B(G^{t}(n, v + z)) + B(G^{t}(nσ(\frac{π}{|z|}, v + z), −v + z))

= B(n, v + z) + B(nσ(\frac{π}{|z|}, v + z), −v + z).

Hence,

B̄(G^{t}(n, v + z)) + B̄(G^{t}(nσ(\frac{π}{|z|}, v + z), −v + z))
=B̄(n, v + z) + B̄(nσ(\frac{π}{|z|}, v + z), −v + z).

Replacing v by cos(s)v, z by sin(s)z, t by \frac{t}{sin(s)} in the above identity and recalling that B̄(n, v) is independent of n (Proposition 2.10), we get, by letting s → 0,

B̄(e, e^{tJ(z)}v) + B̄(e, −e^{tJ(z)}v) = B̄(e, v) + B̄(e, −v).

That is B̃(e^{J(z)}v) = B̃(v).

Proof of Theorems 3.2 and 3.3.

As in the proof of Theorem 3.1, we set

B̃(v) = \frac{1}{2}(B̄(e, \frac{v}{|v|}) + B̄(e, −\frac{v}{|v|}).

We need only show that B̃(e^{J(z)}v) = B̃(v) for all z ∈ Z, v ≠ 0 ∈ V.

Note that under the hypotheses of either theorem, N is non-singular. According to Proposition 2.5 and Lemma 1.7, for (n, v + z) ∈ S(N),

B̄(G^{\frac{t}{s}} (n, cos(s)v + sin(s)z) − B̄(n, cos(s)v + sin(s)z)
=(e^{tJ( \frac{H(n,cos(s)v+sin(s)z)}{s})} − Id)J^{−1}(H(n, cos(s)v + sin(s)z))Ī(n, cos(s)v + sin(s)z)
−(e^{tJ( \frac{sin(s)}{s} z)} − Id)J^{−1}(sin(s)z) cos(s)v
=(e^{tJ( \frac{H(n,cos(s)v+sin(s)z)}{s})} − Id)J^{−1}(\frac{H(n, cos(s)v + sin(s)z}{s})\frac{Ī(n, cos(s)v + sin(s)z)}{s}
−(e^{tJ( \frac{sin(s)}{s} z)} − Id)J^{−1}(\frac{sin(s)}{s}z)\frac{cos(s)}{s}v

where Ī(n, v + z) is defined in 2.9. The last equality follows from the fact that J^{−1}(\frac{z}{s}) = J(\frac{z}{s})^{−1}.

Letting s → 0 in the above identity and using Lemma 3.5 (a) and (b) and Lemma 1.7, we get

(3.2.1) B̄(n, e^{tJ(z)}v) − B̄(n, v) = (e^{tJ(z)} − Id)J^{−1}(z)\frac{d}{ds}|_{s=0}Ī(n, cos(s)v + sin(s)z).

By Proposition 2.10, B̄(n, v) is independent of n, thus so is

(e^{tJ(z)} − Id)J^{−1}(z)\frac{d}{ds}|_{s=0}Ī(n, cos(s)v + sin(s)z).

Therefore, by Lemma 3.5 (c),

e^{tJ(z)} \frac{d}{ds}|_{s=0}Ī(n, cos(s)v + sin(s)z)
= \frac{d}{ds}|_{s=0}Ī(n, e^{tJ(z)} cos(s)v + sin(s)z).

If N is strongly in resonance, we can pick a t such that e^{tJ(z)} = − Id . If N is irrational, there is a sequence {t_{i}} such that lim_{i→∞} e^{t_{i}J(z)} = − Id. In both cases, we have,

\frac{d}{ds}|_{s=0}Ī(n, − cos(s)v + sin(s)z)
= − \frac{d}{ds}|_{s=0}Ī(n, cos(s)v + sin(s)z).

Substituting this identity into (3.2.1), we get B̃(e^{J(z)}v) = B̃(v). This completes the proofs of Theorems 3.2 and 3.3.

References

[BFL] Y. Benoist, P. Foulon & F. Labourie, Flots dÁnosov á distributions stable et instable diffŕentiables, (to appearJ. Amer. Math. Soc.).

[Bd] P. Bérard, Transplantation et isospectralité, Part I, Math. Ann. 292 (1992), 547–559.

[BGS] W. Ballmann, M. Gromov, and V. Schroeder, Manifolds of nonpositive curvature, Birkhäuser, Boston, 1985.

[BPV] R.Brooks, P. Perry, and P. Yang, Isospectral sets of conformally equivalent metrics, Duke Math. J. 58 (1989), 131–150.

[BT] R. Brooks and R. Tse, Isospectral surfaces of small genus, Nagoya Math. J. 107 (1987), 13–24.

[Br] P. Buser, Isospectral Riemannian surfaces, Ann. Inst. Fourier 36 (1986), 167–192.

[CV] Y. Colin de Verdière, Spectre du Laplacian et longeur des géodesiques periodiques II, Comp. Math. 27 (1993), 159–184.

[CS] J. H. Conway and N. J. Sloane, Four–dimensional lattices with the same theta series, Internat. Math. Research Notices (Duke Math. J.) 1992, no. 4, 93–96.

[C] C. Croke, Rigidity for surfaces of non-negative curvature, Comm. Math. Helvetici 65 (1990), 150–169.

[CK] C. Croke & B. Kleiner, Conjugacy and rigidity for manifolds with a parallel vector field,, J. Diff. Geom. 39 (1994), 659-680.

[DG] D. DeTurck and C. Gordon, Isospectral deformations II: trace formula, metrics, and potentials, Comm. Pure Appl. Math. 42 (1989), 1067–1095.

[DG] J. Duistermaat and V. Guillemin, The spectrum of positive elliptic operators and periodic bicharacteristics, Invent. Math. 29 (1975), 30–79.

[E] P. Eberlein, Geometry of 2-step nilpotent groups with a left–invariant metric, Preprint.

[EHS] P. Eberlein, U. Hamenstadt & V. Schroeder, Manifolds of nonpositive curvature,, Proc. Symp. Pure Math. 54 (1993), 179-227.

[G] C. Gordon, The Laplace spectra versus the length spectra of Riemannian Manifolds, Contemp. Math. 51 (1986), 63–80.

[GM] C. Gordon and Y. Mao, Comparisons of Laplace spectra, lenght spaectra and geodesic flows of some riemannian manifolds, Math. Research Letter 1 (1994), 677–688.

[GW] C. Gordon and E. N. Wilson, Isospectral deformations of compact solvmanifolds, J. Diff. Geom. 19 (1984), 241–256.

[Gt] R. Gornet, Spectral geometry on higher- step Riemannian nilmanifolds, Ph.D. thesis, Washington University–St. Louis, 1993.

[—] , The length spectrum and representation theory on two and three-step nilpotent

Lie groups, to appear in Contemp. Math.

[GK] V. Guillemin & D. Kazdan, Some inverse spectral results for negatively curved n-manifolds., Proc. Symp. Pure Math. 36 (1980), 153-180.

[Ka] M. Kaina, Geodesic flows on negatively curved manifolds with smooth stable and unstable foliations,, Erg. Theo. Dy. Sys. 8 (1988), 215-239.

[Kn] A. Kaplan, Riemannian nilmanifolds attached to Clifford modules, Geom. Dedicata 11 (1981), 127–136.

[Kk] A. Katok, Entropy and closed geodesics,, Erg. Theo. Dy. Sys. 2 (1982), 339-367.

[Ku] R. Kuwabara, A note on the deformations of Hamiltonian systems on nilmanifolds, J. Math. Tokushima 26 (1992), 19–29.

[LP] K. B. Lee and Keun Park, Closed geodesics in 2-step nilmanifolds,, Preprint.

[Mt] M. Mast, Closed geodesics in 2-step nilmanifolds,, Ph.D. thesis, University of North Carolina.

[Mr] J. Milnor, Eigenvalues of the Laplace operator on certain manifolds, Proc. Nat. Acad. Sci. 51 (1964), 542.

[O] J. Otal, Sur les longueurs des géodesiques d’une metrique a courbure negative dans le disque, Comm. Math. Helvetici 65 (1990).

[OP] H. Ouyang and H. Pesce, Déformations isospectrales sur les nilvariétés de rang deux, C. R. Acad. Sci. Paris, Série I 314 (1992), 621–623.

[P] H. Pesce, Calcul du spectre d’une nilvariété de rang deux et applications, to appear in Trans. Amer. Math. Soc.

[—] , Variétés hyperboliques et elliptiques fortement isospectrales, Preprint. [—] , Representations relativement équivalentes et variétés riemanniennes isospec-

trales, Preprint.

[Rn] M. Raghunathan, Discrete subgroups of Lie groups, Springer-Verlag,, 1972.

[Rm] C. Riehm, Explicit spin representations and Lie algebras of Heisenberg type, J. London Math. Soc. 29 (1984), no. 2, 49 -62.

[S] T. Sunada, Riemannian coverings and isospectral manifolds, Ann. Math. 121 (1985), 169–186.

[W] A. Weinstein, Fourier integral operators, quantization and the spectra of Riemannian manifolds., Colloques Internationaux C.N.R.S. 273 (1975), 289-298.

Department of Mathematics, Dartmouth College, Hanover, NH 03755 E-mail address: carolyn.s.gordon@dartmouth.edu

Department of Mathematics, Texas Tech University, Lubbock, TX 79409 E-mail address: ymao@ttmath.ttu.edu