INVOLUTIONS ON MODULI SPACES AND REFINEMENTS OF THE VERLINDE FORMULA

JØRGEN ELLEGAARD ANDERSEN AND GREGOR MASBAUM

Abstract. The moduli space M of semi-stable rank 2 bundles with trivial determinant over a complex curve Σ carries involutions naturally associated to 2-torsion points on the Jacobian of the curve. For every lift of a 2-torsion point to a 4-torsion point, we define a lift of the involution to the determinant line bundle L. We obtain an explicit presentation of the group generated by these lifts in terms of the order 4 Weil pairing. This is related to the triple intersections of the components of the fixed point sets in M, which we also determine completely using the order 4 Weil pairing. The lifted involutions act on the spaces of holomorphic sections of powers of L, whose dimensions are given by the Verlinde formula. We compute the characters of these vector spaces as representations of the group generated by our lifts, and we obtain an explicit isomorphism (as group representations) with the combinatorial-topological TQFT-vector spaces of [BHMV]. As an application, we describe a ‘brick decomposition’, with explicit dimension formulas, of the Verlinde vector spaces. We also obtain similar results in the twisted (i.e., degree one) case.

Contents

Introduction and Motivation. 1 1. Basic definitions and notation. 3 2. Statement of the main results. 5 3. Relationship with the [BHMV]-theory. 7 4. Brick decomposition and dimension formulas. 10 5. The fixed point varieties. 13 6. Intersections and triple intersections. 16 7. The action of J^{(2)} on the Hecke correspondence. 20 8. Investigation of the involutions ρ_{a} and ρ′_{α}. 23 9. The trace computation. 28 References 31

Date: October 22, 1997. Joint Aarhus University, Institut de Mathematiques de Jussieu and MSRI preprint. Research at MSRI is supported in part by NSF grant DMS-9022140. J.E.A. is supported in part by NSF grant DMS-93-09653 and by the Danish Research Council.

Introduction and Motivation.

The celebrated Verlinde formula [Ve] gives the dimension of certain vector spaces of so-called ‘conformal blocks’ appearing in conformal field theory. In this paper, we will take the point of view of algebraic geometry and think of the conformal blocks as holomorphic sections of powers of the determinant line bundle over moduli spaces of semi-stable bundles with fixed determinant over a complete non-singular complex curve Σ. According to Atiyah [At2] and Witten [Wi], the spaces of holomorphic sections should also fit into 2 + 1-dimensional ‘Topological Quantum Field Theories’ (TQFT). This geometric construction has been studied quite a lot (see e.g. [ADW, MS, RSW, Hi, CLM, Th1, Th2, Sz, BSz, BL, KNR, Fa]).

In [BHMV], a combinatorial-topological construction of TQFT-functors was described, based on a particularly simple construction of the Witten-Reshetikhin-Turaev 3-manifold invariant [RT] in the SU(2)-case through the Kauffman bracket. In that paper one also constructed certain involutions on the TQFT-vector spaces which were then used to decompose the vector spaces into direct summands. These involutions are associated to simple closed curves on the underlying smooth surface of Σ, and they generate a kind of Heisenberg group presented in terms of the mod 4 intersection form. These ideas were developed further in [BM] to construct spin-refined TQFT’s.

The starting point for the present paper was the idea that the involutions of [BHMV] should correspond on the algebraic-geometric side to the involutions on moduli space naturally associated to 2-torsion points on the Jacobian of Σ. More precisely, the 2-torsion points should correspond to the mod 2 homology classes of the simple closed curves. Note that the involutions on the spaces of holomorphic sections require a choice of lift to the determinant line bundle L. It is easy to see that these lifts generate a central extension of the group of 2-torsion points, whose cocycle is given by the order 2 Weil pairing. It follows that this extension is indeed abstractly isomorphic to the one that appeared in [BHMV]. The ambiguity in the choice of lift is reflected on the topological side by the choice of a simple closed curve within its mod 2 homology class.

One of our motivations in this paper is to establish a more precise correspondance between the two theories. The key idea is to define, for every lift of a 2-torsion point α to a 4-torsion point a, an involution ρ_{a} on L which covers the action of α on the moduli space M. The sign of this lift ρ_{a} is fixed by requiring it to act as the identity over a certain component of the fixed point set of the involution acting on M; this component is simply the one containing the class of the semi-stable bundle L_{a} ⊕ L_{a}^{−1} (see section 1 for more details). We will see in section 3 that the action of this lift ρ_{a} on holomorphic sections corresponds precisely to the involution τ_{γ} which is associated in the [BHMV]-theory to a simple closed curve γ whose homology class is Poincaré dual to a. In this way, we obtain an explicit isomorphism (as group representations) with the TQFT-vector spaces of [BHMV]. We believe this constitutes a nice confirmation that there should be a natural correspondence between the two theories.

The main work in this paper is, however, algebraic-geometric in nature and consists of a detailed study of our lifts ρ_{a} and their action on the Verlinde vector spaces. Our main results are as follows (see section 2 for more complete statements). We show in theorem 2.1 that our lifts satisfy

ρ_{a} ρ_{b} = λ_{4}(a, b) ρ_{a+b},

where λ_{4} is the order 4 Weil pairing (which is the algebraic-geometric analogue of the mod 4 intersection form). Along the way, we also determine in theorem 2.2 the triple intersections of the components of the fixed point sets of the action on the various moduli spaces. The rôle played by the order 4 Weil pairing in this context doesn’t seem to have been observed before. We then compute in theorem 2.3 the trace of the induced involutions ρ_{a}^{⊗k} on the spaces of holomorphic sections of the k-th tensor power of L. This determines the characters of these vector spaces as representations of the group generated by our lifts. We also obtain similar results in the twisted (i.e., degree one) case, where the situation is somewhat simpler, as only the order 2 Weil pairing is needed.

A corollary of our results is a ‘brick decomposition’ of the spaces of holomorphic sections, the structure of which depends on the value of the level k modulo 4. This is, of course, analogous to the decomposition in [BHMV], but we will derive it, as well as explicit dimension formulas, directly from the character of the representation. At low levels, similar decompositions have appeared previously in the work of Beauville [Be1] (see also Laszlo [L]) for k = 2, and of van Geemen and Previato [vGP1, vGP2], Oxbury and Pauly [OP], and Pauly [P] for k = 4 (in our notation). Their approach is based on the relationship with abelian theta-functions and seems quite different from ours.

This paper is organized as follows. After giving the basic definitions in section 1, we state our main results in section 2. The relationship with the [BHMV]-theory is discussed in more detail in section 3, and the ‘brick decompositions’ are described in section 4. The remainder of the paper is devoted to the proofs. The reader interested only in the algebraic geometry may skip section 3, and no familiarity with [BHMV] is necessary to understand the results and their proofs.

Acknowledgment. We would like to thank Ch. Sorger for discussions on this project.

1. Basic definitions and notation.

Let Σ be a complete, non-singular curve over the complex numbers of genus g ≥ 2. Let M_{d} be the moduli space of semi-stable bundles of rank 2 and degree d ∈ ℤ on Σ. There is a natural algebraic action of the degree zero Picard group Pic_{0}(Σ) on M_{d} gotten by tensoring. In this paper, we will use the standard identification of Pic_{0}(Σ) with the Jacobian J(Σ) and speak of an action of J(Σ) on M_{d}.

Fix a point p ∈ Σ and let [p] be the associated line bundle. We have the determinant morphism det : M_{d} → Pic_{d}(Σ). We put M = det^{−1}(O_{Σ}) and M′ = det^{−1}([p]).

As it is explained in [DN] there is a natural construction of a determinant line bundle L_{d} over M_{d}. We will denote the restriction of L_{0} to M by L and the restriction of L_{1} to M′ by L′. These two determinant bundles are generators respectively of the Picard group of M and of M′ (see [DN]).

Let k ≥ 1 be an integer called the level. Put

Z_{k}(Σ) = H^{0}(M, L^{k})

Z′_{k}(Σ) = \{ \begin{matrix}H^{0}(M′, L′^{k/2}) & if k ≡ 0 mod 2 \\ 0 & if k ≡ 1 mod 2 \end{matrix}

It is by now well-known that the dimensions of these vector spaces are given by the celebrated Verlinde formulas (to be recalled in section 4). Following Thaddeus [Th1], we will refer to Z′_{k}(Σ) as the twisted case, and to Z_{k}(Σ) as the untwisted case.

Let J^{(r)} be the subgroup of order r points on J(Σ). We denote by L_{α} (resp. L_{a}) the line bundle on Σ corresponding to α ∈ J^{(2)} (resp. a ∈ J^{(4)}). Note that the group J^{(r)} is identified with H^{1}(Σ; μ_{r}), where μ_{r} ⊂ ℂ is the group of r-th roots of unity.

The action of α ∈ J^{(2)} on M_{d} is given by tensoring with L_{α}. Since L_{α}^{⊗2} ≅ O_{Σ}, this preserves M and M′. By abuse of notation, the automorphisms of M and M′ induced by α ∈ J^{(2)} will again be denoted by α.

The lifts ρ_{a} and ρ′_{α}. By a lift of α to L we mean an invertible bundle map from L to itself covering α. For a lift to exist it suffices that α^{∗}L ≅ L. This is the case for every α ∈ J^{(2)}, since the Picard group of M is isomorphic to ℤ, and L is ample, so that α must act trivially on the Picard group. Therefore the action of any α ∈ J^{(2)} can be lifted to L, and also to L′, for the same reason. Since the only algebraic functions on M and M′ are the constant ones, it follows that we can actually choose involutive lifts of each element of J^{(2)}. Any two involutive lifts of α agree up to sign.

To fix the signs of the lifts of α to L and L′, we use the fact that the sign can be read off in the fiber over a fixed point of α. Given 0 ≠ α ∈ J^{(2)}, we use the notation |X|_{α} for the fixed point variety of the automorphism induced by α on the various moduli spaces X.

It is well-known that |M′|_{α} is isomorphic to the Prym variety P_{α} associated to α, and that |M|_{α} is isomorphic to two copies of the Kummer variety P_{α}/⟨ ± 1⟩. This result is due to Narasimhan and Ramanan [NR]. In particular, |M′|_{α} is connected and non-empty, and |M|_{α} has two components. For any a ∈ J^{(4)} such that 2a = α, we denote by |M|_{a}^{+} the component of |M|_{α} containing the S-equivalence class of the semistable bundle L_{a} ⊕ L_{a}^{−1}. (Note that this is indeed fixed under tensoring with L_{α}, since L_{a} ⊗ L_{α} ≅ L_{a}^{−1}.) The other component of |M|_{α} is denoted by |M|_{a}^{−}.

Definition 1.1. For 0 ≠ α ∈ J^{(2)}, we define ρ′_{α} to be the involutive lift to L′ of α acting on M′ such that ρ′_{α} acts by minus the identity on the restriction of L′ to |M′|_{α}.

Definition 1.2. For a ∈ J^{(4)} such that 2a = α ≠ 0, we define ρ_{a} to be the involutive lift to L of α acting on M such that ρ_{a} acts by the identity on the restriction of L to the component |M|_{a}^{+} of |M|_{α} specified by a.

It will be convenient to extend this definition by letting ρ′_{α} (resp. ρ_{a}) be the identity if α = 0 (resp. 2a = 0).

Note. According to Theorem F in [DN], we have that L′^{−2} ≅ K, where K is the canonical bundle of M′. The natural action of α on K coincides with the one induced by both possible involutive lifts ±ρ′_{α}. This is because α acts as the identity on the fiber of K over the fixed point set |M′|_{α}, since |M′|_{α} has even codimension (see section 5). Similar comments apply in the untwisted case.

The Weil pairing. (See e.g. [Ho].^{1}) Let M(Σ) be the field of meromorphic functions on Σ. The divisor of f ∈ M(Σ) is denoted by (f). As already mentioned, we consider J^{(r)} to be the r-torsion points on the group Pic_{0}(Σ) which is naturally identified with Div_{0}(Σ)/ Div^{pr}(Σ). (Here, Div_{d}(Σ) is the group of degree d divisors, and Div^{pr}(Σ) are the principal divisors.) If D = ∑ n_{j}x_{j} is a divisor and f ∈ M(Σ) a meromorphic function, we put f(D) = ∏ f(x_{j})^{n_{j}}. The Weil pairing

λ_{r} : J^{(r)} × J^{(r)} → μ_{r}

is defined as follows. Given a, b ∈ J^{(r)}, represent them by divisors D_{a}, D_{b} with disjoint support, and pick f, g ∈ M(Σ) such that (f) = rD_{a} and (g) = rD_{b}. Then

λ_{r}(a, b) = \frac{g(D_{a})}{f(D_{b})}.

The Weil pairing is antisymmetric and non-degenerate. The fact that it takes values in μ_{r} follows from Weil reciprocity (see [GH], p. 242).

2. Statement of the main results.

Let E be the group generated by the involutions ρ_{a} acting on L, and let E′ the group generated by the involutions ρ′_{α} acting on L′ . By general theory, these groups are central extensions of J^{(2)}. Our first main result gives a presentation of these groups in terms of our lifts ρ_{a} and ρ′_{α}.

Theorem 2.1. The involutions ρ_{a} and ρ′_{α} satisfy the following relations:

ρ_{a} ρ_{b} = λ_{4}(a, b) ρ_{a+b} (1)

ρ′_{α} ρ′_{β} = λ_{2}(α, β) ρ′_{α+β} (2)

1We thank A. Beauville for pointing out this reference.

It follows that the group E is a non-trivial extension of J^{(2)} by μ_{4}. The cocycle c(α, β) of this extension is the order 2 Weil pairing λ_{2}; indeed:

c(α, β) = ρ_{a}ρ_{b}ρ_{a}^{−1}ρ_{b}^{−1} = λ_{4}(a, b)^{2} = λ_{2}(α, β).

The group E′ is a trivial extension of J^{(2)}, i.e., it is isomorphic to μ_{2} × J^{(2)}.

Note. The cocycle of the extensions E was known before; e.g. it is deduced in [BLS] from the isomorphism of Z_{1}(Σ) with the space of abelian theta-functions due to Beauville, Narasimhan, and Ramanan [BNR]. Given the cocycle, it follows immediately that for any choice of involutive lifts ρ̃_{α}, ρ̃_{β}, and ρ̃_{α+β}, say, of α, β, α+β to the determinant line bundle L, one has ρ̃_{α}ρ̃_{β} = ±ρ̃_{α+β} if λ_{2}(α, β) = 1, and ρ̃_{α}ρ̃_{β} = ±i ρ̃_{α+β} if λ_{2}(α, β) = −1. The contribution of theorem 2.1 is to show that with our precise choice of lifts, the prefactors ±1 and ±i are given by the order 4 Weil pairing. Similar remarks apply in the twisted case.

The above result is related to the triple intersections of the components of the fixed point set on M. In fact, a key step in the proof is the following result which does not seem to have been observed before.

Theorem 2.2. Assume that α and β are distinct non-zero elements of J^{(2)} such that λ_{2}(α, β) = 1 ∈ μ_{2}. Let a, b ∈ J^{(4)} such that 2a = α and 2b = β. Then

|M|_{a}^{+} ∩ |M|_{b}^{+} ∩ |M|_{a+b}^{+} ≠ ∅ ⇔ λ_{4}(a, b) = 1 ∈ μ_{4}
|M|_{a}^{+} ∩ |M|_{b}^{+} ∩ |M|_{a+b}^{−} ≠ ∅ ⇔ λ_{4}(a, b) = −1 ∈ μ_{4}

Note. Given theorem 2.2, theorem 2.1 in the case λ_{4}(a, b) = 1 follows immediately. Indeed, if the triple intersection |M|_{a}^{+} ∩ |M|_{b}^{+} ∩ |M|_{a+b}^{+} is non-empty, it follows from the definition of the lifts that ρ_{a+b} = ρ_{a}ρ_{b} (since ρ_{a}ρ_{b}ρ_{a+b} must be a constant, and this constant can be computed in the fiber of L over a triple intersection point.) The proof in the general case, however, requires some further arguments. The most interesting case is when λ_{4}(a, b) = ±i. In this situation, the fixed point sets |M|_{α}, |M|_{β}, |M|_{α+β} don’t intersect, but there is a ℙ^{1} intersecting each one of the six components |M|_{a}^{+}, |M|_{a}^{−}, |M|_{b}^{+}, |M|_{b}^{−}, |M|_{a+b}^{+}, |M|_{a+b}^{−}, in a point. See section 8.

Our next result describes Z_{k}(Σ) (resp. Z′_{k}(Σ)) as representations of the group E (resp. E′). (Here, E acts on Z_{k}(Σ) via the natural action of ρ_{a}^{⊗k} on L^{⊗k}, and similarly for E′.) This is based on the following result obtained by applying the Lefschetz-Riemann-Roch formula. We assume α ∈ J^{(2)} is non-zero, and 2a = α.

Theorem 2.3. One has

Tr(ρ_{a}^{⊗k}) = \frac{1 + (−1)^{k}}{2} (\frac{k + 2}{2})^{g−1}

and (for even k)

Tr(ρ′_{α}^{⊗k/2}) = (−1)^{k/2} (\frac{k + 2}{2})^{g−1}

Note. In the twisted case and for levels divisible by 4, this result is due to Pantev [Pa]. In the untwisted case and for levels divisible by 4, it is also contained in Beauville’s recent paper [Be2]. Our computation was done independently of his.

The characters of Z_{k}(Σ) and Z′_{k}(Σ) (as representations of the groups E and E′, respectively) are determined by the formulas in theorem 2.3 together with the trace of the identity, given by the Verlinde formulas. Therefore the above result determines these representations up to isomorphism.

A remarkable consequence of this is the following theorem, which was actually the main motivation for the present paper.

Theorem 2.4. The spaces Z_{k}(Σ) and Z′_{k}(Σ) are isomorphic, as representations of the groups E and E′, respectively, to the TQFT-vector spaces constructed in [BHMV].

This will be discussed in more detail in section 3.

Corollary 2.5. If the level k is even, the spaces Z_{k}(Σ) and Z′_{k}(Σ) are decomposed as direct sums of isotypic components (called ‘bricks’ in what follows) for the action of E and E′. If k ≡ 0 mod 4, the bricks are indexed by characters of J^{(2)}. If k ≡ 2 mod 4, the bricks are indexed by θ-characteristics on the curve Σ (or, equivalently, by spin structures on Σ). If the level is odd, Z_{k}(Σ) is isomorphic, as representation of the group E, to a direct sum of copies of Z_{1}(Σ) or to a direct sum of copies of the conjugate representation, \overline{Z_{1}(Σ)}, according to the parity of (k − 1)/2.

Remarks 2.6. (i) Note that formula (1) implies

ρ_{a}^{⊗k} ρ_{b}^{⊗k} = λ_{4}(a, b)^{k} ρ_{a+b}^{⊗k}. (3)

Hence the action of E on Z_{k}(Σ) factors through an action of E′ if k is even, and furthermore through an action of J^{(2)} if k ≡ 0 mod 4. This explains why the bricks are indexed by characters of J^{(2)} if k ≡ 0 mod 4. If k ≡ 2 mod 4, the index set are the characters of E′ which do not factor through J^{(2)}; such characters can be identified with θ-characteristics.

(ii) The dimensions of the bricks are the same for all non-trivial characters of J^{(2)} in the case k ≡ 0 mod 4, and depend only on the parity of the θ-characteristic in the case k ≡ 2 mod 4. In section 4, we will give explicit formulas for their dimensions.

Note. Our brick decomposition can be viewed as a generalisation of an old result of Beauville’s [Be1] (see also Laszlo [L]) at level 2. Beauville constructed bases of Z_{2}(Σ) (resp. Z′_{2}(Σ)) whose basis elements are indexed by even (resp. odd) θ-characteristics. This corresponds from our point of view to the fact that the bricks in level 2 are zero, if the θ-characteristic has the ‘wrong’ parity, and one-dimensional otherwise. Note, however, that the situation in level 2 is very special; in general, the bricks are non-zero for both parities. We would like to mention also the work of van Geemen and Previato [vGP1, vGP2], Oxbury and Pauly [OP], and Pauly [P]. We believe their work contains a description of the bricks in level 4, but a detailed comparison hasn’t been done yet.

3. Relationship with the [BHMV]-theory.

In [BHMV], the Kauffman bracket (at a 2p-th root of unity called A) was used to give a combinatorial-topological construction of TQFT-functors V_{p} on a certain 2+1dimensional cobordism category. It is expected that these functors for p = 2k + 4 correspond (in some natural sense) to Witten’s ones for SU(2) and level k. For example, the dimensions of the vector space V_{p}(Σ) ⊗ ℂ ^{2} associated to a closed oriented surface Σ is also given by the Verlinde formula ([BHMV], cor. 1.16).

The aim of the present section is to explain the following more precise statement of theorem 2.4. (As already mentioned in the introduction, the reader interested only in the algebraic geometry may proceed directly to section 4.)

Theorem 3.1. Let p = 2k + 4. For the ‘right’ choice of 2p-th root of unity A, the vector spaces V_{p}(Σ)⊗ℂ and Z_{k}(Σ) are isomorphic as representations of E (this group is denoted by Γ(Σ) in [BHMV]). This isomorphism sends the involution ρ_{a}^{⊗k} acting on Z_{k}(Σ) to the involution τ_{γ} acting on V_{p}(Σ) ⊗ ℂ, where γ is a simple closed curve on Σ representing the Poincaré dual of a. A similar statement holds in the twisted case.

Note. For simplicity of exposition, a few technical details related to framing issues will be suppressed from the discussion here. The reader is also referred to the survey article [MV].

By definition, elements of V_{p}(Σ) are represented by linear combinations of compact oriented 3-manifolds M with boundary equal to Σ. (No complex structure on Σ is needed here.) The 3-manifolds may also contain colored links or, more generally, colored trivalent graphs. In the following, we assume p is even and put p = 2k + 4. Then a color is just an element of {0, 1, 2, …, k}.

In the [BHMV]-theory, one has involutions τ_{γ} of V_{p}(Σ) associated to unoriented simple closed curves γ on Σ. They are defined as follows. Consider a vector represented by some (M, L), where M is a 3-manifold with boundary Σ, and L stands for some colored link inside M. Then the action of τ_{γ} consists of adding to the link L already present in M, an additional component consisting of the curve γ pushed slightly inside M in a neighborhood of Σ = ∂M, where γ is colored by k (the last color). It was shown in section 7 of [BHMV] that (because of the relations which hold in V_{p}(Σ)) this endomorphism τ_{γ} is an involution.

2By definition, V_{p}(Σ) is a module over a certain abstract cyclotomic ring k_{p}. By V_{p}(Σ) ⊗ ℂ, we mean the vector space obtained by extending coefficients from k_{p} to ℂ; this depends on a choice of a 2p-th root of unity A in ℂ.

Following [BHMV], p. 917, the groups generated by these involutions acting on V_{p}(Σ) can be described as follows. Let Γ(Σ) be the group with one generator [a] for each a ∈ H_{1}(Σ; ℤ/4) plus one additional generator u, and the following relations: the element u is central, u^{4} = 1, [a]^{2} = 1 for all a ∈ H_{1}(Σ; ℤ/4), and

[a] [b] = u^{a·b} [a + b] (4)

for all a, b ∈ H_{1}(Σ; ℤ/4). Here, a · b ∈ ℤ/4 denotes the mod 4 intersection form determined by the orientation of Σ.

If a mod 2 class α ∈ H_{1}(Σ; ℤ/2) is given, every lift of α to a mod 4 class a ∈ H_{1}(Σ; ℤ/4) determines an element [a] ∈ Γ(Σ). However, up to multiplication by u^{2}, this element [a] depends only on α. (Indeed, if a_{1}, a_{2} are two such lifts, then [a_{1}] = u^{2x}[a_{2}] where a_{1} − a_{2} = 2x.) From this it follows easily that the group Γ(Σ) is a central extension of H_{1}(Σ; ℤ/2) by ℤ/4. (It can be viewed as some kind of ‘reduced’ Heisenberg group associated to twice the intersection form.)

Here is the relationship of the group Γ(Σ) with the involutions τ_{γ}. Given α ∈ H_{1}(Σ; ℤ/2), a lift of α to an element of Γ(Σ) can also be specified by an unoriented simple closed curve γ representing α, as follows. The curve γ determines an element a ∈ H_{1}(Σ; ℤ/4) up to sign, and the induced element [a] ∈ Γ(Σ) is well-defined, since a · a = 0 and hence [−a] = [a].

It is clear from the geometric description of the τ_{γ}’s given above that the commutation properties of these involutions are related to the intersection properties of the associated curves; for example two such involutions obviously commute if the corresponding simple closed curves on Σ don’t intersect. The fact that the τ_{γ}’s satisfy precisely the relations in Γ(Σ) is shown in prop. 7.5 of [BHMV]. Moreover, it is shown there that the central element u acts on V_{2k+4}(Σ) as multiplication by (−1)^{k+1}A^{(k+2)^{2}}.

Now let us identify ℤ/4 = μ_{4} such that 1 ∈ ℤ/4 corresponds to i ∈ μ_{4}. Also, let us use Poincaré duality to identify J^{(4)}(Σ) = H^{1}(Σ; μ_{4}) with H_{1}(Σ; ℤ/4). By a well-known folk theorem, there exists a sign ε = ±1 such that the Weil pairing is related to the intersection form as follows: ^{3}

λ_{4}(a, b) = (εi)^{a·b}.

We now choose the primitive root of unity A of order 2p = 4k + 8 such that (−1)^{k+1}A^{(k+2)^{2}} = (εi)^{k}. For instance, we may choose A = −εe^{2iπ/(4k+8)}. Comparing (1) and (4), we see that the assignment

τ_{γ} ↦ ρ_{a}^{⊗k}

3The value of ε should be known, but we have been unable to locate it in the literature.

(where the curve γ represents the Poincaré dual of a) defines an isomorphism from the image of Γ(Σ) in the general linear group of V_{2k+4}(Σ) ⊗ ℂ to the image of the group E in the general linear group of Z_{k}(Σ). (In particular, for k = 1 we have an isomorphism Γ(Σ) ≅ E.)

Using the dimension formulas given in section 7 of [BHMV], one can check that the traces of the involutions τ_{γ} acting on V_{p}(Σ) coincide precisely with the traces computed in our theorem 2.3. This verification will be omitted.

Since representations of finite groups are determined by their characters, this proves theorem 3.1 in the untwisted case.

The twisted case, where Z_{k}(Σ) is replaced with Z′_{k}(Σ), is similar. Here we must replace V_{p}(Σ) with V_{p}(Σ′), where Σ′ is Σ with one puncture colored by k (the last color). (We maintain the convention p = 2k+4.) Then we have again an isomorphism of representations (the group corresponding to E′ is called Γ′(Σ) in [BHMV]). The dimension of V_{p}(Σ′) is given by the ‘twisted Verlinde formula’, see [BHMV], Remark 5.11. This space is zero if k is odd, and this is why we define Z′_{k}(Σ) to be zero for odd k. (See also Thaddeus [Th1].) (N.b., the vector spaces V_{p} are also defined for surfaces with colored punctures; they are called ‘surfaces with colored structure’ in [BHMV]. The theory of the involutions τ_{γ} works the same for these, with the caveat that the curves γ must avoid the punctures. We expect analogues of our results to hold in the case of general colored punctures, using moduli spaces of parabolic bundles.)

Remark. Although it has been known for some time that V_{p}(Σ) and Z_{k}(Σ) have the same dimensions, it seems, to the best of our knowledge, that a natural isomorphism between the two theories is still missing. Of course, the fact that these two spaces are isomorphic also as representations (of canonically isomorphic groups) gives further evidence that there should be a natural isomorphism between the two theories.

4. Brick decomposition and dimension formulas.

In this section, we describe Z_{k}(Σ) and Z′_{k}(Σ) as representations of the groups E and E′, respectively. We also give explicit dimension formulas. The results of this section are immediate consequences of the isomorphism of theorem 3.1 and the computations in [BHMV] and [BM]. In order to make this paper self-contained, we will, however, derive them directly from theorems 2.1 and 2.3.

We first recall the Verlinde formula and its twisted counterpart. Put d_{g}(k) = dim Z_{k}(Σ_{g}) and d′_{g}(k) = dim Z′_{k}(Σ_{g}), where the subscript g indicates the genus of the curve Σ_{g}. ^{4} Then one has

d_{g}(k) = (\frac{k + 2}{2})^{g−1} ∑_{j=1}^{k+1} (sin \frac{πj}{k + 2})^{2−2g}

4These numbers are denoted by d_{g}(p) and d̂_{g}(p) in [BHMV, MV, BM], where p = 2k + 4.

d′_{g}(k) = (\frac{k + 2}{2})^{g−1} ∑_{j=1}^{k+1} (−1)^{j+1} (sin \frac{πj}{k + 2})^{2−2g}

The case k ≡ 0 mod 4.

In view of formula (3) in section 2, the action of the group E on Z_{k}(Σ_{g}) factors through an action of J^{(2)}(Σ_{g}) = H^{1}(Σ_{g}; μ_{2}). We have a direct sum decomposition

Z_{k}(Σ_{g}) = ⨁_{h} Z_{k}(Σ_{g}; h)

where h runs through the characters of J^{(2)}, and Z_{k}(Σ_{g}; h) is the subspace of Z_{k}(Σ_{g}) where α acts as multiplication by h(α) ∈ μ_{2} = ±1, for all α ∈ J^{(2)}. We will refer to Z_{k}(Σ_{g}; h) as the brick associated to h.

By theorem 2.3, the character of the representation Z_{k}(Σ_{g}) takes the same value on all non-trivial elements of J^{(2)}, and is therefore invariant under the automorphism group of J^{(2)}. It follows that the dimension of the brick Z_{k}(Σ_{g}; h) is the same for all non-trivial characters h of the group, since the automorphism group acts transitively on the set of non-trivial characters. We denote this dimension by d_{g}^{(1)}(k), and we put d_{g}^{(0)}(k) = dim Z_{k}(Σ_{g}; 0), where 0 denotes the trivial character. (Thus, the space Z_{k}(Σ_{g}; 0) is the fixed point set of the action of the group J^{(2)} on Z_{k}(Σ_{g}).) These numbers can be computed from the following two formulas:

d_{g}(k) = d_{g}^{(0)}(k) + (2^{2g} − 1) d_{g}^{(1)}(k)

d_{g}^{(0)}(k) − d_{g}^{(1)}(k) = tr(α) = (\frac{k + 2}{2})^{g−1} , α ≠ 0

For example, one has

d_{g}^{(0)}(k) = \frac{1}{2^{2g}} (d_{g}(k) + (2^{2g} − 1) (\frac{k + 2}{2})^{g−1})

Similarly, Z′_{k}(Σ_{g}) is the direct sum of the bricks Z′_{k}(Σ_{g}; h), and the dimensions d′_{g}^{(0)}(k) = dim Z′_{k}(Σ_{g}; 0) and d′_{g}^{(1)}(k) = dim Z′_{k}(Σ_{g}; h) for h ≠ 0, can be computed as before (just replace d_{g}(k) with d′_{g}(k) and d_{g}^{(ε)}(k) with d′_{g}^{(ε)}(k) in the above).

Example: The case k = 4. The numbers d_{g}^{(1)}(4) and d′_{g}^{(1)}(4) are equal to (3^{g−1} + 1)/2 and (3^{g−1}−1)/2, respectively, and the numbers d_{g}^{(0)}(4) and d′_{g}^{(0)}(4) are obtained by adding 3^{g−1}. These numbers have appeared in Oxbury and Pauly [OP] and Pauly [P].

The case k ≡ 2 mod 4.

In this case, the action of E factors through an action of the group E′ but not through an action of J^{(2)}. Indeed, the involutions ρ_{a}^{⊗2} acting on L^{2} depend only on α, and satisfy the same relations as the ρ′_{α}’s (see formulas (2) and (3) in section 2). Therefore one has a direct sum decomposition

Z_{k}(Σ_{g}) = ⨁_{q} Z_{k}(Σ_{g}; q)

where q runs through the characters of the group E′ which do not factor through J^{(2)}, i.e., such that q takes the value −1 on the central element −1 ∈ μ_{2} ⊂ E′. Such characters are in 1-to-1 correspondence with functions q: J^{(2)} → μ_{2} such that

q(α + β) = q(α) q(β) λ_{2}(α, β).

In other words, q runs through the set of quadratic forms on J^{(2)} ≅ H^{1}(Σ_{g}; ℤ/2) inducing the Weil pairing, i.e., the mod 2 intersection form. It is well-known that such quadratic forms correspond to spin structures, or equivalently, θ-characteristics, on Σ. (See Atiyah [At1], Johnson [Jo].)

Let G ≅ Sp(2g; 𝔽_{2}) be the group of automorphisms of J^{(2)} preserving the order 2 Weil pairing λ_{2}. The group G acts on E′. As in the case k ≡ 0 mod 4, it follows from theorem 2.3 that the character of the representation Z_{k}(Σ_{g}) is invariant under the action of G. It is well-known that the induced action of G on quadratic forms has two orbits which are characterized by the Arf invariant, i.e., the action is transitive on forms q with the same Arf invariant Arf(q) ∈ ℤ/2. (The Arf invariant of the quadratic form corresponds to the parity of the θ-characteristic.) This shows that the dimension of the brick Z_{k}(Σ_{g}; q) depends only on Arf(q). Put d_{g}^{(ε)}(k) = dim Z_{k}(Σ_{g}; q_{ε}), where q_{ε} has Arf invariant ε ∈ ℤ/2. These dimensions can be computed from the following two formulas:

d_{g}(k) = 2^{g−1}(2^{g} + 1) d_{g}^{(0)}(k) + 2^{g−1}(2^{g} − 1) d_{g}^{(1)}(k)

2^{g−1}(d_{g}^{(0)}(k) − d_{g}^{(1)}(k)) = tr(ρ_{a}^{⊗k}) = (\frac{k + 2}{2})^{g−1} , α ≠ 0

The first formula follows from the fact that the number of quadratic forms with zero Arf invariant is equal to 2^{g−1}(2^{g} + 1). The second formula follows from the fact^{5} that for α ≠ 0, one has

♯{q | q(α) = −1, Arf(q) = 0} = 2^{2g−2} = ♯{q | q(α) = −1, Arf(q) = 1}

Similarly, Z′_{k}(Σ_{g}) is the direct sum of the bricks Z′_{k}(Σ_{g}; q), and the dimensions d′_{g}^{(ε)}(k) = dim Z′_{k}(Σ_{g}; q_{ε}) can be computed as before (just replace d_{g}(k) with d′_{g}(k) and d_{g}^{(ε)}(k) with d′_{g}^{(ε)}(k) in the above, but notice that tr(ρ′_{α}^{⊗k/2}) is now equal to − ((k + 2)/2)^{g−1}.)

5A nice way to think about this is to observe that there is a natural bijection between quadratic forms q with fixed Arf invariant and such that q(α) = −1, and quadratic forms with arbitrary Arf invariant on the 2g − 2-dimensional space ⟨α⟩^{⊥}/⟨α⟩.

Here are explicit formulas for the dimensions. They are equivalent to the formulas on p. 264 of [BM] ^{6}.

d_{g}^{(ε)}(k) = \frac{1}{2^{2g}} (d_{g}(k) + (\frac{k + 2}{2})^{g−1} ((−1)^{ε}2^{g} − 1))

d′_{g}^{(ε)}(k) = \frac{1}{2^{2g}} (d′_{g}(k) + (\frac{k + 2}{2})^{g−1} (1 − (−1)^{ε}2^{g}))

Example: The case k = 2. The numbers d_{g}^{(0)}(2) and d′_{g}^{(1)}(2) are equal to 1, and the numbers d_{g}^{(1)}(2) and d′_{g}^{(0)}(2) are zero. Therefore d_{g}(2) = 2^{g−1}(2^{g} + 1) and d′_{g}(2) = 2^{g−1}(2^{g} − 1) (see Beauville [Be1]).

Note. In [BM], a ℤ/2-graded TQFT-functor is constructed on a cobordism category of surfaces equipped with spin structures (and other things). Given a connected surface Σ with a spin structure σ, the even (resp. odd) part of this functor is isomorphic to Z_{k}(Σ_{g}; q_{σ}) (resp. Z′_{k}(Σ_{g}; q_{σ})), where q_{σ} is the quadratic form corresponding to σ.

Remark 4.1. The action of the symplectic group G ≅ Sp(2g; 𝔽_{2}) permuting the bricks has the following geometric interpretation. Recall that all elements of G can be represented by diffeomorphisms of Σ. In the [BHMV]-theory, one has, more or less by definition, a natural action of a certain extended mapping class group on V_{2k+4}(Σ). On the geometric side, there is also a (projective-linear) action of the mapping class group of Σ on Z_{k}(Σ); this action is constructed using Hitchin’s projectively-flat connection [Hi]. It is, of course, expected that V_{2k+4}(Σ) ⊗ ℂ and Z_{k}(Σ) are isomorphic as representations of the extended mapping class group. In any case, it is easy to see in both theories that the action of a diffeomorphism f takes the brick associated to a character h (resp. a quadratic form q) to the brick associated to f^{∗}(h) (resp. f^{∗}(q)). On the geometric side, the main reason for this is that Hitchin’s connection is (projectively) invariant under the actions of both the mapping class group and the group E.

The odd-level case.

Proposition 4.2. If k ≡ 1 mod 2, Z_{k}(Σ) is isomorphic, as representation of the group E, to a direct sum of copies of Z_{1}(Σ), if k ≡ 1 mod 4, and to a direct sum of copies of the conjugate representation, \overline{Z_{1}(Σ)}, if k ≡ 3 mod 4.

6Warning: Note that k does not denote the level in [BM]; one has p = 8k in [BM] while in the present paper p = 2k + 4.

Proof. Note that the character of the representation Z_{1}(Σ) is the function χ: E → ℂ which is zero on all group elements not in the central subgroup μ_{4} ⊂ E, while the trace of a central element λ ∈ μ_{4} is χ(λ) = 2^{g} λ. The character of the conjugate representation, \overline{Z_{1}(Σ)}, is of course the conjugate character \overline{χ}. Now it follows from theorems 2.1 and 2.3 that the character of Z_{k}(Σ) is a multiple of χ if k ≡ 1 mod 4, and a multiple of \overline{χ}, if k ≡ 3 mod 4. It is however well-known that χ is an irreducible character, and this proves the proposition.

Note. This result corresponds, via the isomorphism Z_{k}(Σ) ≅ V_{2k+4}(Σ) ⊗ ℂ, to theorems 1.5 and 1.6 of [BHMV]. It is shown there that (for odd k)

V_{2k+4}(Σ) ≅ V′_{2}(Σ) ⊗ V_{k+2}(Σ)

as representations of the group E, where V′_{2}(Σ) and V_{k+2}(Σ) are defined in [BHMV]. Moreover, the group E acts trivially on V_{k+2}(Σ), and V′_{2}(Σ_{g}) is isomorphic to V_{6}(Σ), and hence to Z_{1}(Σ) or to \overline{Z_{1}(Σ)}, after a change of coefficients. It would be interesting to have an algebro-geometric interpretation of the space V_{k+2}(Σ), which in the [BHMV]-theory can be interpreted as a SO(3)-TQFT vector space. Here, the name SO(3)-TQFT just means that the allowed colors in this TQFT are even, or in other words, are SU(2)-representations which lift to SO(3).

5. The fixed point varieties.

We need to analyze the action of J^{(2)} on M and M′ in order to obtain information about our lifts. In this section, we describe the various fixed point varieties, mainly following Narasimhan and Ramanan [NR]. We also discuss when two lifts of α ∈ J^{(2)} to J^{(4)} determine the same component of the fixed point set |M|_{α}.

Notation. Throughout this paper, we denote by L_{α} (resp. L_{a}) the line bundle on Σ corresponding to α ∈ J^{(2)} (resp. a ∈ J^{(4)}).

Let α ∈ J^{(2)} be nonzero. Let π_{α} : Σ^{α} → Σ be the 2-sheeted unramified covering of Σ corresponding to α, and let ϕ_{α} be the covering transformation of π_{α}. Using the line bundle L_{α}, we can explicitly construct Σ^{α} as

Σ^{α} = {ξ ∈ L_{α}|ξ ⊗ ξ = 1}

using an isomorphism L_{α}^{2} ≅ O_{Σ}. The involution ϕ_{α} is then induced by multiplication by −1 on L_{α}.

Given a line bundle L over Σ^{α}, the push-down E = π_{α∗}(L) can be obtained by descending L ⊕ ϕ_{α}^{∗}(L) (which is naturally an equivariant bundle) to Σ:

E = (L ⊕ ϕ_{α}^{∗}(L))/⟨ϕ_{α}⟩.

Here, the natural involution of L ⊕ ϕ_{α}^{∗}(L) covering ϕ_{α} is again denoted by ϕ_{α}. The fundamental observation is that E ⊗ L_{α} is isomorphic to E. This follows formally from the pull-push formula π_{α∗}(L) ⊗ L′ ≅ π_{α∗}(L ⊗ π_{α}^{∗}L′) and the fact that L_{α} pulls back to the trivial bundle on Σ^{α}.

Remark 5.1. We will later need the following explicit isomorphism from E = π_{α∗}(L) to E⊗L_{α}. Note that π_{α}^{∗}L_{α}, as an equivariant bundle, is isomorphic to O_{Σ^{α}}^{−}, that is, the trivial line bundle Σ^{α} ×ℂ, but with non-trivial action, given by (x, z) ↦ (ϕ_{α}(x), −z). Therefore

E ⊗ L_{α} = (L ⊕ ϕ_{α}^{∗}(L))/⟨−ϕ_{α}⟩.

It follows that the diagonal automorphism 1 ⊕ (−1) of L ⊕ ϕ_{α}^{∗}(L) descends to an isomorphism from E to E ⊗ L_{α}.

Let Nm_{α}: Pic(Σ^{α}) → Pic(Σ) be the classical albanese or norm map induced by the following map on divisors. If D = ∑ n_{j}x_{j} ∈ Div(Σ^{α}) then Nm_{α}(D) = ∑ n_{j}π_{α}(x_{j}) ∈ Div(Σ).

The Prym variety P_{α} associated to α is by definition the connected component of Nm_{α}^{−1}(O_{Σ}) containing O_{Σ^{α}}. It is a principally polarized abelian variety of dimension g − 1. The quotient variety P_{α}/⟨±1⟩ is called the Kummer variety.

We define θ_{α} : Pic(Σ^{α}) → Pic(Σ) by

θ_{α}(L) = det(π_{α∗}(L)) = Nm_{α}(L) ⊗ L_{α}

(see [NR] for the second equality). Note that ϕ_{α} acts on Pic(Σ^{α}) by sending L to ϕ_{α}^{∗}(L).

Proposition 5.2 (Narasimhan and Ramanan [NR]). (i) The map L ↦ (π_{α})_{∗}(L) induces isomorphisms

θ_{α}^{−1}(O_{Σ})/⟨ϕ_{α}⟩ ⟶^{∼} |M|_{α} (5)

θ_{α}^{−1}([p])/⟨ϕ_{α}⟩ ⟶^{∼} |M′|_{α} (6)

(ii) Moreover, θ_{α}^{−1}(O_{Σ})/⟨ϕ_{α}⟩ is isomorphic to two copies of P_{α}/⟨±1⟩, while θ_{α}^{−1}([p])/⟨ϕ_{α}⟩ is isomorphic to P_{α}.

Remark 5.3. In the twisted case, π_{α∗}: θ_{α}^{−1}([p]) → |M′|_{α} is a double covering, with covering transformation ϕ_{α}. In the untwisted case, the same holds on the open subvariety of θ_{α}^{−1}(O_{Σ}) where ϕ_{α} acts freely, while the fixed points of ϕ_{α} are sent by the map π_{α∗} bijectively to the points in |M|_{α} represented by semi-stable but not stable bundles. (See remark 5.6 below.)

For later use, we need the following more explicit description of θ_{α}^{−1}(O_{Σ}) and θ_{α}^{−1}([p]). For d = 0, 1, define Φ_{α}^{d} : Pic_{d}(Σ^{α}) → Pic_{0}(Σ^{α}) = J(Σ^{α}) by

Φ_{α}^{d}(L) = L ⊗ ϕ_{α}^{∗}L^{−1}.

The following two properties (7) and (8) of the maps Φ_{α}^{d} are elementary facts from the classical theory of line bundles on curves, see e.g. App. B. in [ACGH].

First, one has the disjoint union

(7) Nm_{α}^{−1}(O_{Σ}) = Im Φ_{α}^{0} ∪ Im Φ_{α}^{1}

and the Prym variety P_{α} is equal to the component Im Φ_{α}^{0}.

Second, note that Nm_{α}(π_{α}^{∗}(L_{β})) = L_{β}^{2} = O_{Σ} for all β ∈ J^{(2)}. Moreover, one has

π_{α}^{∗}L_{β} ∈ Im Φ_{α}^{d} ⇔ λ_{2}(α, β) = (−1)^{d}. (8)

Here, λ_{2} : J^{(2)} × J^{(2)} → μ_{2} is the order 2 Weil pairing.

Now pick a ∈ J^{(4)} such that 2a = α, and β ∈ J^{(2)} such that λ_{2}(α, β) = −1. Note that a′ = a + β is another element of J^{(4)} such that 2a′ = α. Also, pick a point p_{α} ∈ π_{α}^{−1}(p) ⊂ Σ^{α}. Note that θ_{α}^{−1}(O_{Σ}) and θ_{α}^{−1}([p]) are both isomorphic to Nm_{α}^{−1}(O_{Σ}). From (7) and (8), we have the following description of θ_{α}^{−1}(O_{Σ}) and θ_{α}^{−1}([p]) as disjoint unions:

(9)  θ_{α}^{−1}(O_{Σ}) = π_{α}^{∗}L_{a} ⊗ Im Φ_{α}^{0} ∪ π_{α}^{∗}L_{a′} ⊗ Im Φ_{α}^{0}
(10)  θ_{α}^{−1}(O_{Σ}) = π_{α}^{∗}L_{a} ⊗ Im Φ_{α}^{0} ∪ π_{α}^{∗}L_{a} ⊗ Im Φ_{α}^{1}
(11)  θ_{α}^{−1}([p]) = π_{α}^{∗}L_{a} ⊗ [p_{α}] ⊗ Im Φ_{α}^{0} ∪ π_{α}^{∗}L_{a} ⊗ [ϕ_{α}(p_{α})] ⊗ Im Φ_{α}^{0}

Remark 5.4. It follows from (10) and (11) that the action of ϕ_{α} preserves the two components of θ_{α}^{−1}(O_{Σ}), while it exchanges the two components of θ_{α}^{−1}([p]). By (7), the action of ϕ_{α} on Pic(Σ^{α}) restricts to multiplication by −1 (that is, the map L ↦ L^{−1}) on Nm_{α}^{−1}(O_{Σ}). This proves 5.2(ii).

We can now describe when two lifts of α ∈ J^{(2)} to J^{(4)} determine the same component of the fixed point set |M|_{α}. Recall that |M|_{a}^{+} was defined to be the component of |M|_{α} containing the S-equivalence class of the semi-stable bundle L_{a} ⊕ L_{a}^{−1}.

Proposition 5.5. (i) Let a_{1}, a_{2} ∈ J^{(4)} such that 2a_{1} = 2a_{2} = α. Then

|M|_{a_{1}}^{+} = |M|_{a_{2}}^{+} ⇔ λ_{2}(a_{1} − a_{2}, α) = 1 ∈ μ_{2}.

(ii) For β ∈ J^{(2)}, the action of β on |M|_{α} interchanges the two components of |M|_{α} if and only if λ_{2}(α, β) = −1.

Proof. Note that π_{α∗}(π_{α}^{∗}(L_{a_{i}})) ≅ L_{a_{i}} ⊕ L_{a_{i}} ⊗ L_{α} ≅ L_{a_{i}} ⊕ L_{a_{i}}^{−1}. Therefore |M|_{a_{i}}^{+} is the component π_{α∗}(π_{α}^{∗}L_{a_{i}} ⊗Im Φ_{α}^{0}), and (i) follows from formula (9). Now part (ii) follows from (8), since the action of β on |M|_{α} lifts to tensoring with π_{α}^{∗}L_{β} on θ_{α}^{−1}(O_{Σ}). This completes the proof.

Note. Translating the ‘multiplicative’ notation of the Weil pairing into the ‘additive’ notation in section 3, the condition λ_{2}(a_{1} − a_{2}, α) = 1 ∈ μ_{2} becomes the condition ((a_{1} − a_{2})/2) · α = 0 ∈ ℤ/2. Thus, prop. 5.5(i) implies that |M|_{a_{1}}^{+} = |M|_{a_{2}}^{+} if and only if [a_{1}] = [a_{2}], where [a_{i}] is the lift of α to the group Γ(Σ) defined in section 3.

Remark 5.6. Let M^{sing} denote the set of points of M represented by semistable, but not stable, bundles. A semi-stable bundle E with Gr(E) ≅ L_{1} ⊕L_{2} represents a point in M^{sing} if and only if L_{2} ≅ L_{1}^{−1}, and this point lies in |M|_{α} if and only if L_{1}⊗L_{α} ≅ L_{2}. This shows that M^{sing} ∩ |M|_{α} is precisely the set of points represented by bundles of the form L_{a} ⊕ L_{a}^{−1} with a ∈ J^{(4)} and 2a = α. Since L_{a} ⊕ L_{a}^{−1} ≅ π_{α∗}(π_{α}^{∗}(L_{a})), and π_{α}^{∗}({L ∈ Pic_{0}(Σ)|L^{2} ≅ L_{α}}) is precisely the fixed point set of ϕ_{α} on θ_{α}^{−1}(O_{Σ}), we see that π_{α∗} induces a bijection from that fixed point set to M^{sing} ∩ |M|_{α}, as asserted above.

6. Intersections and triple intersections.

In this section, we describe how the various fixed point varieties intersect. The triple intersections will be used in the proof of theorem 2.1.

Note. The intersection properties of the fixed point sets in relation to the order 2 Weil pairing are well-known; they are used for example in van Geemen and Previato [vGP1]. On the other hand, the relationship of the triple intersections of their individual components (in the untwisted case) with the order 4 Weil pairing seems to be new.

Proposition 6.1. For α, β ∈ J^{(2)} non-zero distinct elements, we have that

|M|_{α} ∩ |M|_{β} ≠ ∅ ⇔ λ_{2}(α, β) = 1 ∈ μ_{2}
|M′|_{α} ∩ |M′|_{β} ≠ ∅ ⇔ λ_{2}(α, β) = −1 ∈ μ_{2}

Proof. The morphism π_{α∗}: Pic_{d}(Σ^{α}) → |M_{d}|_{α} is surjective and J^{(2)}-equivariant. (Here, β ∈ J^{(2)} acts on Pic_{d}(Σ^{α}) by L ↦ L ⊗ π_{α}^{∗}L_{β}.) Hence we get the following description of the intersection

|M_{d}|_{α} ∩ |M_{d}|_{β} = π_{α∗}({L ∈ Pic_{d}(Σ^{α})|L ⊗ π_{α}^{∗}L_{β} ≅ ϕ_{α}^{∗}L or L}).

Hence we see that

|M_{d}|_{α} ∩ |M_{d}|_{β} ≠ ∅

if and only if

π_{α}^{∗}L_{β} ∈ Im Φ_{α}^{d}.

From (8), this is the case if and only if λ_{2}(α, β) = (−1)^{d}. Using the action of J(Σ) on |M_{d}|_{α} ∩ |M_{d}|_{β}, the results for |M|_{α} ∩ |M|_{β} and |M′|_{α} ∩ |M′|_{β} follow from this.

Proposition 6.2. If λ_{2}(α, β) = 1, the quotient group J^{(2)}/⟨α, β⟩ acts simply transitively on |M|_{α} ∩ |M|_{β}. In particular, this intersection has 2^{2g−2} elements. If λ_{2}(α, β) = −1, the same holds for |M′|_{α} ∩ |M′|_{β}.

Proof. Put I_{α,β} = {L ∈ θ_{α}^{−1}(O_{Σ}) | π_{α}^{∗}(L_{β}) ≅ L ⊗ ϕ_{α}^{∗}L^{−1}}. Then π_{α∗}: I_{α,β} → |M|_{α} ∩ |M|_{β} is a double covering. Note that J^{(2)}/⟨α⟩ acts simply transitively on I_{α,β}. The involution ϕ_{α} is on I_{α,β} the same as tensoring with π_{α}^{∗}L_{β}, in other words, the action of β. This proves the result in the untwisted case. The twisted case is proved similarly.

Note. In view of remark 5.6, this description shows that |M|_{α} ∩ |M|_{β} is contained in the stable part of M.

Proposition 6.3. |M|_{α} ∩ |M|_{β} is the disjoint union of the sets |M|_{a}^{ε} ∩ |M|_{b}^{μ}, where ε = ± and μ = ±, each of which sets has 2^{2g−4} elements.

Proof. Recall from proposition 5.5(ii) that the action of γ ∈ J^{(2)} exchanges the components of |M|_{α} if and only if λ_{2}(α, γ) = −1. Thus, the result follows by exploiting the fact that for every choice of signs ε = ±1 and μ = ±1, there exists γ such that λ_{2}(α, γ) = ε and λ_{2}(β, γ) = μ.

We now turn to the triple intersections. The first observation is the following easy lemma.

Lemma 6.4. Let α, β, γ be distinct non-zero elements of J^{(2)} such that the triple intersection |M|_{α} ∩ |M|_{β} ∩ |M|_{γ} is non-empty. Then γ = α + β. The same holds for the triple intersections in the twisted case.

Proof. Indeed, it follows from the description in prop. 6.2 that the triple intersection can only be non-empty if π_{α}^{∗}(L_{β}) = π_{α}^{∗}(L_{γ}) which implies γ = α + β.

Note. Since the group J^{(2)} is commutative, we have

|M|_{α} ∩ |M|_{β} = |M|_{α} ∩ |M|_{β} ∩ |M|_{α+β},

and similarly in the twisted case.

In the untwisted case, the fixed point sets have two components each, and we may ask about the triple intersections of the individual components. Our answer was already stated in theorem 2.2. We will prove it in the following equivalent form.

Theorem 6.5. Assume that α and β are distinct non-zero elements of J^{(2)} such that λ_{2}(α, β) = 1 ∈ μ_{2}. Let a, b ∈ J^{(4)} such that 2a = α and 2b = β. Let ε, μ, ν = ±1 be three signs. Then

|M|_{a}^{ε} ∩ |M|_{b}^{μ} ∩ |M|_{a+b}^{ν} ≠ ∅ ⇔ λ_{4}(a, b) = εμν

Note. It follows that if |M|_{a}^{ε} ∩ |M|_{b}^{μ} ∩ |M|_{a+b}^{ν} is non-empty, then it is equal to all of |M|_{a}^{ε} ∩ |M|_{b}^{μ}. This fact can of course be seen directly using prop. 6.3. The important information in the theorem is that it tells us when |M|_{a}^{ε} ∩ |M|_{b}^{μ} intersects |M|_{a+b}^{+} and when |M|_{a+b}^{−}.

Proof. To simplify notation, we put γ = α + β and c = a + b. Let E represent a point in |M|_{α} ∩ |M|_{β} ∩ |M|_{γ}. The description of |M|_{α} in section 5 tells us that there is a unique L_{a} ∈ π_{α}^{∗}(L_{a}) ⊗ Im Φ_{α}^{d} ⊂ Pic_{0}(Σ^{α}) such that E ≅ π_{α∗}L_{a}; moreover E lies in |M|_{a}^{+} if and only d is even. (See formula (10).) Similarly we have E ≅ π_{β∗}L_{b} ≅ π_{γ∗}L_{c}, where L_{b} ∈ π_{β}^{∗}(L_{b}) ⊗ Im Φ_{β}^{d′} ⊂ Pic_{0}(Σ^{β}) and L_{c} ∈ π_{γ}^{∗}(L_{c}) ⊗ Im Φ_{γ}^{d′′} ⊂ Pic_{0}(Σ^{γ}). Thus, the theorem is equivalent to the following lemma.

Lemma 6.6. One has λ_{4}(a, b) = (−1)^{d+d′+d′′}.

The proof of this lemma will occupy the remainder of this section.

There is a curve Σ̃ naturally double covering Σ^{α}, Σ^{β} and Σ^{γ}:

Σ̃ = {(ξ, η) ∈ L_{α} ⊕ L_{β} | ξ^{2} = 1 = η^{2}} .

The projections onto the two factors induce projections π^{α}: Σ̃ → Σ^{α} and π^{β}: Σ̃ → Σ^{β}. The bilinear map L_{α} ⊕ L_{β} → L_{α} ⊗ L_{β} induces the projection π^{γ}: Σ̃ → Σ^{γ}.

Σ̃

π^{α}↙ ↓ π^{β} ↘π^{γ}
Σ^{α} Σ^{β} Σ^{γ}
π_{α}↘ ↓ π_{β} ↙π_{γ}

Σ

The deck-transformations of the coverings π^{α}, π^{β}, π^{γ} will be denoted respectively by ϕ^{α}, ϕ^{β}, ϕ^{γ}. Note that ϕ^{α} (resp. ϕ^{β}) is induced by multiplication by −1 in the fibers of L_{β} (resp. L_{α}), and that ϕ^{γ} = ϕ^{α} ∘ ϕ^{β}. We denote the projection Σ̃ → Σ by π̃, so that

π̃ = π_{α} ∘ π^{α} = π_{β} ∘ π^{β} = π_{γ} ∘ π^{γ}.

Notice also that the involution ϕ_{α} of Σ^{α} is covered by both ϕ^{β} and ϕ^{γ} (but of course not by ϕ^{α}, since Σ^{α} = Σ̃/⟨ϕ^{α}⟩). Similar comments apply to ϕ_{β} and ϕ_{γ}.

Lemma 6.7. One has π^{α∗}(L_{a}) ≅ π^{β∗}(L_{b}) ≅ π^{γ∗}(L_{c}).

Proof. Since E ≅ π_{α∗}(L_{a}) lies in |M|_{α} ∩ |M|_{β}, we have ϕ_{α}^{∗}(L_{a}) ≅ L_{a} ⊗ π_{α}^{∗}(L_{β}) (see the proof of prop. 6.2). Since π^{α∗}π_{α}^{∗}(L_{β}) = π̃^{∗}(L_{β}) is trivial, it follows that

π̃^{∗}(E) ≅ π̃^{∗}π_{α∗}(L_{a}) ≅ π^{α∗}(L_{a} ⊕ ϕ_{α}^{∗}(L_{a})) ≅ π^{α∗}(L_{a}) ⊕ π^{α∗}(L_{a}).

Similarly π̃^{∗}(E) ≅ π^{β∗}(L_{b}) ⊕ π^{β∗}(L_{b}) ≅ π^{γ∗}(L_{c}) ⊕ π^{γ∗}(L_{c}). Since line bundles are simple, the lemma follows.

We now turn to the computation of the Weil pairing λ_{4}(a, b). Represent a, b ∈ J^{(4)} by divisors D_{a}, D_{b} ∈ Div_{0}(Σ) with disjoint support, and put D_{c} = D_{a} + D_{b}. Pick D ∈ Div_{d}(Σ^{α}) (resp. D′ ∈ Div_{d′}(Σ^{β}), resp. D′′ ∈ Div_{d′′}(Σ^{γ})) such that π_{α}^{∗}(D_{a}) + (1 − ϕ_{α}^{∗})(D) (resp. π_{β}^{∗}(D_{b}) + (1 − ϕ_{β}^{∗})(D′), resp. π_{γ}^{∗}(D_{c}) + (1 − ϕ_{γ}^{∗})(D′′)) represents L_{a} (resp. L_{b}, resp. L_{c}). Pulling everything up to Σ̃, we get divisors

F_{a} = π̃^{∗}(D_{a}) + π^{α∗}(1 − ϕ_{α}^{∗})(D)
F_{b} = π̃^{∗}(D_{b}) + π^{β∗}(1 − ϕ_{β}^{∗})(D′)
F_{c} = π̃^{∗}(D_{c}) + π^{γ∗}(1 − ϕ_{γ}^{∗})(D′′)

such that F_{a} represents π^{α∗}(L_{a}), F_{b} represents π^{β∗}(L_{b}), and F_{c} represents π^{γ∗}(L_{c}). Since these three bundles are isomorphic by lemma 6.7, there exist meromorphic functions h_{1}, h_{2} ∈ M(Σ̃) such that

(h_{2}) + F_{a} = F_{c} = (h_{1}) + F_{b}.

Let Nm^{α}: M(Σ̃) → M(Σ^{α}) be the norm map on meromorphic functions associated to the covering π^{α}. The norm maps associated to the various other coverings will similarly be denoted by Nm^{β}, Nm^{γ}, Nm_{α}, Nm_{β}, Nm_{γ}, and Ñm.

Lemma 6.8. (i) Define f, g ∈ M(Σ) by f = Ñm(h_{1}), g = Ñm(h_{2}). Then

(f) = 4 D_{a}, (g) = 4 D_{b}.

(ii) Define f_{α} = Nm^{α}(h_{1}) ∈ M(Σ^{α}), f_{β} = Nm^{β}(h_{2}) ∈ M(Σ^{β}), f_{γ} = Nm^{γ}(h_{1}/h_{2}) ∈ M(Σ^{γ}). Then

f_{α} ∘ ϕ_{α} = −f_{α}, f_{β} ∘ ϕ_{β} = −f_{β}, f_{γ} ∘ ϕ_{γ} = −f_{γ}.

Proof. Using that ϕ^{α} covers ϕ_{β} and ϕ_{γ}, one computes that

π^{α∗}((f_{α})) = (h_{1}) + ϕ^{α∗}(h_{1}) = 2 π̃^{∗}(D_{a}).

It follows that (f_{α}) = 2π_{α}^{∗}(D_{a}) and hence (f) = (Nm_{α}(f_{α})) = 4D_{a}, as asserted. This also shows that the divisor (f_{α}) is ϕ_{α}-invariant. Therefore one has f_{α} ∘ ϕ_{α} = ±f_{α}. But f_{α} itself cannot be ϕ_{α}-invariant, because if it were, it would descend to a function h ∈ M(Σ) such that (h) = 2D_{a}, which is impossible since a has order 4. Therefore f_{α} ∘ ϕ_{α} = −f_{α}. The other assertions of the lemma are proved similarly.

By lemma 6.8(i), we can compute the Weil pairing λ_{4}(a, b) using the functions f and g (see the definition in section 1). Note that by lemma 6.8(ii), we have that

h_{1}(π^{α∗}(1 − ϕ_{α}^{∗})(D)) = f_{α}((1 − ϕ_{α}^{∗})(D)) = \frac{f_{α}(D)}{(f_{α} ∘ ϕ_{α})(D)} = (−1)^{deg D}.

Thus

λ_{4}(a, b) = \frac{g(D_{a})}{f(D_{b})} = \frac{f(−D_{b})}{g(−D_{a})} = \frac{h_{1}(−π̃^{∗}(D_{b}))}{h_{2}(−π̃^{∗}(D_{a}))}
= \frac{h_{1}(−π̃^{∗}(D_{b}) + (h_{2}))}{h_{2}(−π̃^{∗}(D_{a}) + (h_{1}))}
= \frac{h_{1}(π^{γ∗}(1 − ϕ_{γ}^{∗})(D′′) − π^{α∗}(1 − ϕ_{α}^{∗})(D))}{h_{2}(π^{γ∗}(1 − ϕ_{γ}^{∗})(D′′) − π^{β∗}(1 − ϕ_{β}^{∗})(D′))}

= f_{γ}((1 − ϕ_{γ}^{∗})(D′′))\frac{f_{β}((1 − ϕ_{β}^{∗})(D′))}{f_{α}((1 − ϕ_{α}^{∗})(D))}
= (−1)^{deg(D)+deg(D′)+deg(D′′)} = (−1)^{d+d′+d′′}

where we have used Weil reciprocity in the fourth equality. This proves lemma 6.6 and hence theorem 6.5.

7. The action of J^{(2)} on the Hecke correspondence.

We will make use of the Hecke correspondence in our analysis of the involutions in section 8. This is a pair of morphisms

[figure: P; M; M′; q; q′]

which allows one to ‘transfer’ information from M to M′. In this section, we describe the fixed point varieties |P|_{α} of the action of the various α ∈ J^{(2)} on P.

Notation. Given a bundle E over Σ, we denote by E_{x} the fiber of E at the point x ∈ Σ. Also, for a bundle E representing a point in the moduli spaces M or M′, we use the notation [E] for that point.

We briefly review the construction of P. (See e.g. Bertram and Szenes [BSz].) Let U be a Poincaré bundle over Σ × M′. Thus, if [E′] ∈ M′, the restriction of U to Σ×{[E′]} is isomorphic to E′. We can uniquely fix U by requiring that det(U|_{{p}×M′}) is an ample generator of Pic(M′).

We put P = ℙ(U|_{{p}×M′}) and let q′ : P → M′ be the projection. Note that q′ is a ℙ^{1}-fibration and for [E′] ∈ M′, the fiber (q′)^{−1}([E′]) is isomorphic to the projective space ℙ(E′_{p}). In fact, P can be viewed as the moduli space of pairs (E′, F) where E′ is a stable rank 2 bundle with det(E′) = [p], and F ⊂ E′_{p} is a one-dimensional subspace, i.e., P is a moduli space of semi-stable parabolic bundles. We will refer to F as a line in E′_{p}. Points in P will be denoted as [(E′, F)], and we have q′([(E′, F)]) = [E′].

The map q is obtained by the operation of elementary modification at p. This means that we have q([(E′, F)]) = [E] if and only if there is a short exact sequence (of sheaves)

0 → E → E′⟶^{λ}ℂ_{p} → 0

such that ker_{p}(λ) = F ⊂ E′_{p}. Here, ℂ_{p} is the skyscraper sheaf at p.

The group J^{(2)} acts naturally on P. The action of α ∈ J^{(2)} on P sends [(E′, F)] to [(E′ ⊗ L_{α}, F ⊗ L_{α})]. The morphisms q and q′ are J^{(2)}-equivariant.

Let α ∈ J^{(2)} be non-zero. We now describe the fixed point variety |P|_{α}. Recall that π_{α∗}: θ_{α}^{−1}([p]) → |M′|_{α} is a double covering, where θ_{α}^{−1}([p]) ⊂ Pic_{1}(Σ^{α}) consists of two ‘translates’ of the Prym variety P_{α}. Let p_{α} ∈ Σ^{α} be such that π_{α}(p_{α}) = p. If L ∈ Pic_{1}(Σ^{α}), the projection gives a canonical isomorphism

(L ⊕ ϕ_{α}^{∗}(L))_{p_{α}} ⟶^{∼}(π_{α∗}(L))_{p}.

Proposition 7.1. We have an isomorphism

j_{p_{α}}: θ_{α}^{−1}([p]) ⟶^{∼} |P|_{α}

defined by j_{p_{α}}(L) = [(π_{α∗}(L), (L ⊕ 0)_{p_{a}})].

Proof. Put E′ = π_{α∗}(L). It is clear that |P|_{α} ⊂ (q′)^{−1}(|M′|_{α}). Therefore the only question is which lines in E′_{p} correspond to fixed points of α acting on ℙ(E′_{p}) ≅ (q′)^{−1}([E′]). Let ψ: E′⟶^{∼}E′ ⊗ L_{α} be the isomorphism described in remark 5.1. It is covered by the diagonal automorphism ψ̃ = 1⊕(−1) of L⊕ϕ_{α}^{∗}(L). Since E′ is stable, it is simple, hence any other isomorphism is a non-zero multiple of ψ. Therefore (E′, F) represents a point in |P|_{α} if and only if ψ(F) = F ⊗ L_{α}. Letting F̃ denote the line in (L ⊕ ϕ_{α}^{∗}(L))_{p_{α}} projecting down to F ⊂ E′_{p}, this condition is equivalent to ψ̃(F̃) = F̃. The only lines in (L ⊕ ϕ_{α}^{∗}(L))_{p_{α}} that ψ̃ preserves are (L ⊕ 0)_{p_{α}} and (0⊕ϕ_{α}^{∗}(L))_{p_{α}}. The first line defines the point j_{p_{α}}(L) in P, and the second line defines the point j_{p_{α}}(ϕ_{α}^{∗}(L)). This shows that j_{p_{α}} is bijective. It is clearly an algebraic morphism, and since its domain is smooth, this shows j_{p_{α}} is an isomorphism.

Remark 7.2. One has q′ ∘ j_{p_{α}} = π_{α∗}. In other words, j_{p_{α}} is an isomorphism of coverings over the identity of |M′|_{α}. Note also that j_{ϕ_{α}(p_{α})} = j_{p_{α}} ∘ ϕ_{α}^{∗}.

Notation. Given a ∈ J^{(4)} such that 2a = α, we denote by |P|_{a}^{+} the component of |P|_{α} containing the point j_{p_{α}}(π_{α}^{∗}(L_{a}) ⊗ [p_{α}]). (See formula (11).) Note that this point, and hence the definition of the component |P|_{a}^{+}, depends only on a, not on the choice of p_{α}.

Proposition 7.3. One has q(|P|_{a}^{+}) = |M|_{a}^{−}.

Proof. Put L = π_{α}^{∗}(L_{a}) and L′ = L ⊗ [p_{α}]. Then E = π_{α∗}(L) represents a point in |M|_{a}^{+}, and E′ = π_{α∗}(L′) represents a point in |M′|_{α}. The short exact sequence of sheaves

0 → L → L′ → ℂ_{p_{α}} → 0

induces the short exact sequence

0 → E → E′ ⟶^{λ} ℂ_{p} → 0. (12)

We need to determine the line F = ker_{p}(λ) ⊂ E′_{p}. Pulling (12) back to Σ^{α} and restricting to the fiber at p_{α}, the map λ becomes

(L′ ⊕ ϕ_{α}^{∗}(L′))_{p_{α}} → (ℂ_{p_{α}} ⊕ ϕ_{α}^{∗}(ℂ_{p_{α}}))_{p_{α}} = ℂ_{p_{α}} ⊕ 0.

This shows that F = ker_{p}(λ) is the projection of the line (0 ⊕ ϕ_{α}^{∗}(L′))_{p_{α}}. Hence

[(E′, F)] = j_{p_{α}}(ϕ_{α}^{∗}(L′)) = j_{ϕ_{α}(p_{α})}(L′).

Since j_{p_{α}}(L′) ∈ |P|_{a}^{+}, this shows that [(E′, F)] lies in |P|_{a}^{−} (see remark 7.2). Recalling that q([(E′, F)] = [E], it follows that q(|P|_{a}^{−}) = |M|_{a}^{+}, and also that q(|P|_{a}^{+}) = |M|_{a}^{−}.

The following observation will be used in section 9.

Proposition 7.4. Let ν be the relative cotangent sheaf of q′: P → M′. Then the restriction of ν to |P|_{α} is numerically trivial.

Proof. Note that ν|_{|P|_{α}} is the dual of the normal bundle, N, say, of the inclusion |P|_{α} ⊂ (q′)^{−1}(|M′|_{α}). By prop. 7.1, it suffices to show that j_{p_{α}}^{∗}(N) is numerically trivial. Let Λ be a Poincaré bundle over Pic_{1}(Σ^{α}) × Σ^{α}. For x ∈ Σ^{α}, let Λ_{x} denote its restriction to θ_{α}^{−1}([p]) × {x}. We have a commutative diagram

ℙ(Λ_{p_{α}} ⊕ Λ_{ϕ_{α}(p_{α})}) ⟶^{Π_{α∗}} (q′)^{−1}(|M′|_{α}) ⊂ P
↓  ↓ q′
θ_{α}^{−1}([p])  ⟶^{π_{α∗}}  |M′|_{α}

Here, Π_{α∗} is the obvious map covering π_{α∗}. (A point in ℙ(Λ_{p_{α}} ⊕ Λ_{ϕ_{α}(p_{α})}) is a point [L] ∈ θ_{α}^{−1}([p]) together with a line in L_{p_{α}} ⊕ L_{ϕ_{α}(p_{α})} = (L ⊕ ϕ_{α}^{∗}(L))_{p_{α}}. This is sent by Π_{α∗} to the point represented by π_{α∗}(L) and the induced line in (π_{α∗}(L))_{p}.)

Let s_{p_{α}} be the section of the fibration on the left defined by s_{p_{α}}(L) = L_{p_{α}} ⊕ 0, for [L] ∈ θ_{α}^{−1}([p]). Then j_{p_{α}} = Π_{α∗} ∘ s_{p_{α}}, and hence the inclusion |P|_{α} ⊂ (q′)^{−1}(|M′|_{α}) corresponds to the inclusion of the image of s_{p_{α}} in ℙ(Λ_{p_{α}} ⊕ Λ_{ϕ_{α}(p_{α})}). This shows

j_{p_{α}}^{∗}(N) ≅ Λ_{p_{α}}^{∗} ⊗ Λ_{ϕ_{α}(p_{α})}.

Tensoring Λ by the pull-back of a bundle over Pic_{1}(Σ^{α}) if necessary, we may assume Λ_{p_{α}} is trivial. Hence j_{p_{α}}^{∗}(N) is numerically trivial, proving the proposition.

8. Investigation of the involutions ρ_{a} and ρ′_{α}.

Let a ∈ J^{(4)} such that 2a = α ≠ 0. Recall that the involution ρ_{a} is the lift of α to L which acts as the identity over the fixed point component |M|_{a}^{+}.

Proposition 8.1. The involution ρ_{a} acts as minus the identity over the fixed point component |M|_{a}^{−}.

Proof. Let E′ represent a point [E′] in |M′|_{α}. Now α acts on the fiber of q′ over [E′] and from our description of |P|_{α}, we have that α has exactly two fixed points on (q′)^{−1}([E′]). Consider now q^{∗}L|_{(q′)^{−1}([E′])} with its lift of α induced by ρ_{a}. Let s_{1}, s_{2} be the signs by which ρ_{a} acts over the two fixed points. By lemma 2.1 in [BSz] we have that q^{∗}L|_{(q′)^{−1}([E′])} is isomorphic to O(1) over (q′)^{−1}([E′]) ≅ ℙ(E′_{p}). From this we conclude that s_{1}s_{2} = −1, proving the proposition.

Remark 8.2. At this point, it follows easily that the cocycle of the extension E generated by the involutions ρ_{a} is equal to the order 2 Weil pairing λ_{2}. Indeed, recall that the cocycle is defined by c(α, β) = ρ_{a}ρ_{b}ρ_{a}^{−1}ρ_{b}^{−1} where a is a lift of α and b is a lift of β. Assume first that λ_{2}(α, β) = 1. Let us then evaluate ρ_{a}ρ_{b}ρ_{a}^{−1}ρ_{b}^{−1} in a point in |M|_{a}^{+}. Recalling from 5.5(ii) that β preserves this component of |M|_{α}, we get that

c(α, β) = 1 ρ_{b} (1)^{−1} ρ_{b}^{−1} = 1.

If however λ_{2}(α, β) = −1, then β exchanges the two components, and

c(α, β) = 1 ρ_{b} (−1)^{−1} ρ_{b}^{−1} = −1.

Thus c = λ_{2}, as asserted.

Theorem 8.3. We have ρ′_{α}ρ′_{β} = λ_{2}(α, β)ρ′_{α+β}.

Proof. We may assume none of the classes α, β, α+β, is zero, the result being obvious otherwise. Consider first the case λ_{2}(α, β) = −1. Then by prop. 6.1 we have that the triple intersection |M′|_{α} ∩ |M′|_{β} ∩ |M′|_{α+β} is non-empty. Since by definition ρ′_{α} acts as minus the identity on the fiber over |M′|_{α}, it follows that

ρ′_{α}ρ′_{β} = −ρ′_{α+β},

proving the result in this case.

Now consider the case λ_{2}(α, β) = 1. Pick a, b ∈ J^{(4)} such that 2a = α and 2b = β, and consider the involutions ρ_{a}^{⊗2}, ρ_{b}^{⊗2}, and ρ_{a+b}^{⊗2}, acting on L^{2}. In fact, those involutions depend only on α and β, and not on the choice of a and b. By prop. 6.1 we have that the triple intersection |M|_{α} ∩ |M|_{β} ∩ |M|_{α+β} is non-empty. Note that ρ_{a}^{⊗2} acts as the identity over both components of |M|_{α}. Hence

ρ_{a}^{⊗2}ρ_{b}^{⊗2} = ρ_{a+b}^{⊗2}. (13)

Now consider the Hecke correspondence. From Corollary 2.2 in [BSz] (see also Lemma 10.3 in [BLS]) we have that

K_{P} ≅ (q′)^{∗}(L′^{−1}) ⊗ q^{∗}(L^{−2}). (14)

From Proposition 7.1 we see that |P|_{α} has odd codimension, hence α acts by −1 on the restriction of K_{P} to |P|_{α}. Our lifts ρ_{a}^{⊗2} and ρ′_{α} thus make the isomorphism (14) a J^{(2)}-equivariant isomorphism. The action of J^{(2)} on K_{P} is obviously a group action. This enables us to compare the lift ρ′_{α} acting on L′ and the lift ρ_{a}^{⊗2} acting on L^{2}. Thus (13) implies

ρ′_{α}ρ′_{β} = ρ′_{α+β},

proving the result in the case λ_{2}(α, β) = 1. This completes the proof.

Note. The equivariance of the isomorphism (14) is the reason why we defined ρ′_{α} to be the lift which acts as minus the identity over the fixed point set.

Theorem 8.4. We have ρ_{a}ρ_{b} = λ_{4}(a, b)ρ_{a+b}.

Proof. We first deal with the case where λ_{4}(a, b) = ±1, or, equivalently, λ_{2}(α, β) = 1. If α = β = 0, there is nothing to show. If α = β ≠ 0, then ρ_{a} and ρ_{b} are lifts of the same class, hence ρ_{a} = ±ρ_{b}. By prop. 8.1, we have ρ_{a} = ρ_{b} if and only if a and b define the same component of |M|_{α}, which in turn is equivalent, by prop. 5.5(i), to λ_{2}(b − a, α) = 1. But if α = β then λ_{2}(b − a, α) = λ_{4}(a, b) and ρ_{a+b} is the identity (by definition). This proves the result in the case α = β. Finally, if α, β, and α + β are all three non-zero, the triple intersection |M|_{α} ∩ |M|_{β} ∩ |M|_{α+β} is non-empty, and we can compute ρ_{a}ρ_{b}ρ_{a+b} in the fiber over an intersection point. By theorem 2.2 and prop. 8.1, it follows that ρ_{a}ρ_{b}ρ_{a+b} = λ_{4}(a, b), completing the proof in the case λ_{4}(a, b) = ±1.

The remainder of this section is devoted to the proof in the case where λ_{4}(a, b) = ±i. We will again use the notations γ = α + β and c = a + b.

Let the bundle E′ represent a point [E′] in |M′|_{α} ∩ |M′|_{β} ∩ |M′|_{γ}. The three involutions α, β, γ induce involutions on (q′)^{−1}([E′]) ⊂ P; recall that (q′)^{−1}([E′]) is identified with the projective space ℙ(E′_{p}). Each of these involutions has two fixed points on ℙ(E′_{p}); these are precisely the intersection points of ℙ(E′_{p}) with the fixed point varieties |P|_{α}, |P|_{β}, and |P|_{γ}. Note that ρ_{a} acts as ∓1 on the fiber of q^{∗}L at the intersection point of ℙ(E′_{p}) with the component |P|_{a}^{±}, since q(|P|_{a}^{+}) = |M|_{a}^{−} by prop. 7.3. Let F_{a}^{±}, F_{b}^{±}, F_{c}^{±} be the lines in E′_{p} corresponding to the intersection points of ℙ(E′_{p}) with the components |P|_{a}^{±}, |P|_{b}^{±}, and |P|_{c}^{±}. As already used in the proof of prop. 8.1, the restriction of q^{∗}L to ℙ(E′_{p}) is the bundle O(1). It will be convenient to transfer the calculation to the tautological bundle O(−1), whose fiber over a point represented by a line F is that line. Of course, O(−1) is the restriction of q^{∗}L^{−1} to ℙ(E′_{p}). For a ∈ J^{(4)}, let us denote by ρ̂_{a} the involution ρ_{a}^{⊗(−1)} acting on L^{−1}. Then ρ̂_{a} acts as ∓1 on the line F_{a}^{±}, and similarly for ρ̂_{b} and ρ̂_{c} on the lines F_{b}^{±} and F_{c}^{±}.

It follows easily from this description (or from the computation of the cocycle in remark 8.2) that one has ρ̂_{a}ρ̂_{b} = ερ̂_{c} where ε ∈ {±i}. (In fact, the involutions ρ̂_{a}, ρ̂_{b} generate a quaternion subgroup Q_{8} ⊂ Sl_{2}(ℂ), covering the commutative subgroup ℤ/2 × ℤ/2 ⊂ P Sl_{2}(ℂ) generated by α, β.) Of course, the sign of ε is determined by the relative position of the six lines. The following lemma computes this relative position in terms of the Weil pairing λ_{4}(a, b).

Lemma 8.5. Let λ = λ_{4}(b, a) ∈ {±i}. There is an isomorphism of E′_{p} with ℂ^{2} sending the six lines F_{a}^{+}, F_{a}^{−}, F_{b}^{+}, F_{b}^{−}, F_{c}^{+}, F_{c}^{−}, to the lines generated by the vectors

( \begin{matrix}1 \\ 0 \end{matrix} ) , ( \begin{matrix}0 \\ 1 \end{matrix} ) , ( \begin{matrix}1 \\ 1 \end{matrix} ) , ( \begin{matrix}1 \\ −1 \end{matrix} ) , ( \begin{matrix}1 \\ −λ \end{matrix} ) , ( \begin{matrix}1 \\ λ \end{matrix} ) .

The proof will be given later. Assuming lemma 8.5 for the moment, we see that ρ̂_{a}, ρ̂_{b} and ρ̂_{c} correspond to the matrices

T_{a} = ( \begin{matrix}−1 & 0 \\ 0 & 1 \end{matrix} ) , T_{b} = ( \begin{matrix}0 & −1 \\ −1 & 0 \end{matrix} ) , T_{c} = ( \begin{matrix}0 & −λ \\ λ & 0 \end{matrix} ) .

Note that T_{a}T_{b} = λT_{c}, and hence ρ̂_{a}ρ̂_{b} = λρ̂_{c} and ρ_{a}ρ_{b} = λ^{−1}ρ_{c}. Thus the remaining case of theorem 8.4 follows directly from lemma 8.5.

Now let us prove lemma 8.5. The proof uses again the coverings Σ^{α}, Σ^{β}, Σ^{γ}, and their common covering Σ̃ (see section 6).

Choose a point p̃ ∈ π̃^{−1}(p) ⊂ Σ̃ and put

p_{α} = π^{α}(p̃), p_{β} = π^{β}(p̃), p_{γ} = π^{γ}(p̃). (15)

Since [E′] ∈ |M′|_{α}∩|M′|_{β} ∩|M′|_{γ}, there exist line bundles L_{a} over Σ^{α}, L_{b} over Σ^{β}, and L_{c} over Σ^{γ}, such that E′ ≅ π_{α∗}(L_{a}) ≅ π_{β∗}(L_{b}) ≅ π_{γ∗}(L_{c}). We can fix L_{a} (resp. L_{b}, resp. L_{c}) uniquely up to isomorphism by requiring that L_{a} ∈ π_{α}^{∗}(L_{a}) ⊗ [p_{α}] ⊗ Im Φ_{α}^{0} (resp. L_{b} ∈ π_{β}^{∗}(L_{b}) ⊗ [p_{β}] ⊗ Im Φ_{β}^{0}, resp. L_{c} ∈ π_{γ}^{∗}(L_{c}) ⊗ [p_{γ}] ⊗ Im Φ_{γ}^{0}) (see formula (11)).

Let us denote the bundle π^{γ∗}(L_{c}) on Σ̃ by L. Being a pull-back bundle, L has a canonical involution, C, say, covering the involution ϕ^{γ} on Σ̃, and such that L/⟨C⟩ is the bundle L_{c} on Σ^{γ} = Σ̃/⟨ϕ^{γ}⟩. Proceeding as in lemma 6.7, it is easy to check that the bundles π^{α∗}(L_{a}) and π^{β∗}(L_{b}) are isomorphic to L = π^{γ∗}(L_{c}). Therefore L also has canonical involutions A and B, covering ϕ^{α} and ϕ^{β}, respectively, such that L/⟨A⟩ ≅ L_{a} and L/⟨B⟩ ≅ L_{b}.

Lemma 8.6. One has AB = λ_{4}(b, a)C.

Proof. As in the proof of lemma 6.6, represent π^{α∗}(L_{a}), π^{β∗}(L_{b}), and L = π^{γ∗}(L_{c}), by divisors F_{a}, F_{b}, F_{c}, respectively, such that

F_{a} = π̃^{∗}(D_{a}) + π^{α∗}(p_{α} + (1 − ϕ_{α}^{∗})(D))

F_{b} = π̃^{∗}(D_{b}) + π^{β∗}(p_{β} + (1 − ϕ_{β}^{∗})(D′))

F_{c} = π̃^{∗}(D_{c}) + π^{γ∗}(p_{γ} + (1 − ϕ_{γ}^{∗})(D′′))

where D_{a}, D_{b} ∈ Div_{0}(Σ) represent a, b ∈ J^{(4)}, D_{c} = D_{a} + D_{b}, D ∈ Div_{0}(Σ^{α}), D′ ∈ Div_{0}(Σ^{β}), and D′′ ∈ Div_{0}(Σ^{γ}). As before, since the three bundles are isomorphic, there exist meromorphic functions h_{1}, h_{2} ∈ M(Σ̃) such that

(h_{2}) + F_{a} = F_{c} = (h_{1}) + F_{b}.

The action of our involutions A, B and C on L can be described on local sections as follows. Since L = O(F_{c}), a local section over some open set U ⊂ Σ̃ is just a meromorphic function s on U such that (s) + F_{c}|_{U} ≥ 0. The action of C is simply given by

C : s ↦ s ∘ ϕ^{γ},

since the divisor F_{c} was pulled back from Σ^{γ}, and C is the canonical involution of the pull-back bundle.

The involution A is nothing but the canonical involution of the pull-back bundle π^{α∗}(L_{a}) = O(F_{a}), conjugated by an isomorphism with L = O(F_{c}). Since (h_{2}) = F_{c} − F_{a}, multiplication by h_{2} gives such an isomorphism O(F_{c}) ⟶^{∼} O(F_{a}). Therefore the action of A on local sections of L = O(F_{c}) is

A : s ↦ ((sh_{2}) ∘ ϕ^{α})h_{2}^{−1} = (s ∘ ϕ^{α}) k_{A} = (s k_{A}^{−1}) ∘ ϕ^{α}

where we have put k_{A} = (h_{2} ∘ ϕ^{α})/h_{2}. (N.b., one may think about this as follows: s ∘ ϕ^{α} is a local section of ϕ^{α∗}L = O(ϕ^{α∗}F_{c}), and multiplication by k_{A} describes an isomorphism O(ϕ^{α∗}F_{c}) ⟶^{∼} O(F_{c}).)

Similarly, B acts on local sections of O(F_{c}) as

B : s ↦ ((sh_{1}) ∘ ϕ^{β})h_{1}^{−1} = (s ∘ ϕ^{β}) k_{B} = (s k_{B}^{−1}) ∘ ϕ^{β}

where k_{B} = (h_{1} ∘ ϕ^{β})/h_{1}.

Put

λ = \frac{k_{B}}{k_{A}} = \frac{h_{2}}{h_{1}} \frac{h_{1} ∘ ϕ^{β}}{h_{2} ∘ ϕ^{α}} (16)

Note that λ is a constant, since (k_{B}) = (k_{A}). Since AB acts on local sections by

AB : s ↦ (((s ∘ ϕ^{β}) k_{B}) ∘ ϕ^{α}) k_{A} = ((s ∘ ϕ^{β}) k_{B} k_{A}^{−1}) ∘ ϕ^{α} = λ s ∘ ϕ^{γ},

we have AB = λC.

Now let us show that λ = λ_{4}(b, a). As in section 6, we use the functions f, g ∈ M(Σ) defined by f = Ñm(h_{1}) and g = Ñm(h_{2}). A computation shows that the statements of lemma 6.8 hold word for word. We can therefore compute λ_{4}(a, b) = g(D_{a})/f(D_{b}) exactly as before. Note that the divisors D, D′, D′′ have degree zero, so that the terms involving the functions f_{α}, f_{β}, f_{γ} are now equal to 1. We thus obtain

λ_{4}(a, b) = \frac{h_{1}(π^{γ∗}(p_{γ}) − π^{α∗}(p_{α}))}{h_{2}(π^{γ∗}(p_{γ}) − π^{β∗}(p_{β}))} = \frac{h_{1}(ϕ^{γ}(p̃) − ϕ^{α}(p̃))}{h_{2}(ϕ^{γ}(p̃) − ϕ^{β}(p̃))}
= \frac{h_{1}}{h_{2}} \frac{h_{2} ∘ ϕ^{α}}{h_{1} ∘ ϕ^{β}} (ϕ^{γ}(p̃))

where we have used (15) in the last but one step. Comparing this with formula (16), we have λ = λ_{4}(a, b)^{−1} = λ_{4}(b, a), proving lemma 8.6.

We return to the proof of lemma 8.5. Recall that

E′ ≅ π_{α∗}(L_{a}) = (L_{a} ⊕ ϕ_{α}^{∗}L_{a})/⟨ϕ_{α}⟩. (17)

Since L_{a} ∈ π_{α}^{∗}(L_{a}) ⊗ [p_{α}] ⊗ Im Φ_{α}^{0}, we have

j_{p_{α}}(L_{a}) ∈ |P|_{a}^{+},

where j_{p_{α}} is the isomorphism of prop. 7.1. Thus, the line F_{a}^{+} ⊂ E′_{p} (representing the unique intersection point in ℙ(E′_{p}) ∩ |P|_{a}^{+}) is the image of the line (L_{a} ⊕ 0)_{p_{α}} ⊂ (L_{a} ⊕ ϕ_{α}^{∗}L_{a})_{p_{α}} under the natural projection. Also, the line F_{a}^{−} is the image of (0⊕ϕ_{α}^{∗}(L_{a}))_{p_{α}}. Note that since E′ is stable, the isomorphism (17) is unique up to scalar multiples, hence the lines F_{a}^{±} are well-determined by this description.

Similarly, using the isomorphism of E′ with π_{β∗}(L_{b}) and with π_{γ∗}(L_{c}), the lines F_{b}^{+} and F_{b}^{−} correspond to the projections of the lines (L_{b} ⊕ 0)_{p_{β}} and (0 ⊕ ϕ_{β}^{∗}(L_{b}))_{p_{β}}, and the lines F_{c}^{+} and F_{c}^{−} correspond to the projections of the lines (L_{c} ⊕ 0)_{p_{γ}} and (0 ⊕ ϕ_{γ}^{∗}(L_{c}))_{p_{γ}}.

Let us now understand the relative position of the six lines in E′_{p}.

Note that since AB = −BA, the involution B of L covering the involution ϕ^{β} on Σ̃ induces a map

L/⟨ − A⟩ ⟶ L/⟨A⟩

covering the involution ϕ_{α} on Σ^{α}. We may choose isomorphisms L_{a} ≅ L/⟨A⟩ and ϕ_{α}^{∗}L_{a} ≅ L/⟨−A⟩ such that this map becomes the canonical map ϕ_{α}^{∗}L_{a} → L_{a} covering ϕ_{α}. Therefore π_{α∗}(L_{a}) is isomorphic to the bundle E_{a} defined by

E_{a} = L ⊕ L/⟨( \begin{matrix}A & 0 \\ 0 & −A \end{matrix} ) , ( \begin{matrix}0 & B \\ B & 0 \end{matrix} )⟩.

(Notice that the two matrices commute, so that E_{a} is indeed a well-defined bundle on Σ = Σ̃/⟨ϕ^{α}, ϕ^{β}⟩.) Moreover, the lines F_{a}^{+} and F_{a}^{−} in E′_{p} correspond, via an isomorphism E′ ≅ E_{a} (which is unique up to scalar multiples), to the images of the lines (L ⊕ 0)p̃ and (0 ⊕ L)p̃ under the projection from L ⊕ L to E_{a}. Here, we have used that π^{α}(p̃) = p_{α} by our choice of p_{α} in (15).

Similarly, π_{β∗}(L_{b}) is isomorphic to the bundle E_{b} defined by

E_{b} = L ⊕ L/⟨( \begin{matrix}B & 0 \\ 0 & −B \end{matrix} ) , ( \begin{matrix}0 & A \\ A & 0 \end{matrix} )⟩,

and since π^{β}(p̃) = p_{β}, the lines F_{b}^{+} and F_{b}^{−} in E′_{p} correspond to the images of the same lines (L ⊕ 0)p̃ and (0 ⊕ L)p̃, but now projected from L ⊕ L to E_{b}.

Lastly, π_{γ∗}(L_{c}) is isomorphic to the bundle E_{c} defined by

E_{c} = L ⊕ L/⟨( \begin{matrix}C & 0 \\ 0 & −C \end{matrix} ) , ( \begin{matrix}0 & A \\ A & 0 \end{matrix} )⟩,

and as before, since π^{γ}(p̃) = p_{γ}, the lines F_{c}^{+} and F_{c}^{−} in E′_{p} correspond to the images of the lines (L ⊕ 0)p̃ and (0 ⊕ L)p̃ under the projection from L ⊕ L to E_{c}.

The three bundles E_{a},E_{b}, E_{c} are all isomorphic to E′. In fact, we have isomorphisms ψ_{X}: E_{b}⟶^{∼}E_{a} and ψ_{Y} : E_{c}⟶^{∼}E_{a} induced by the endomorphisms of L ⊕ L defined by the matrices

X = ( \begin{matrix}1 & 1 \\ 1 & −1 \end{matrix} ) , Y = ( \begin{matrix}1 & 1 \\ −λ & λ \end{matrix} ) .

The verification, which uses that AB = λC by lemma 8.6, is left to the reader.

Let us identify the fiber (L ⊕ L)p̃ in the obvious way with ℂ ⊕ ℂ and consider the isomorphism

E′_{p} ⟶^{∼} (E_{a})_{p} ⟵^{∼} (L ⊕ L)p̃ = ℂ ⊕ ℂ,

where the first map is induced by an isomorphism E′ ≅ E_{a} and the second map is the projection. Then the six lines F_{a}^{+}, F_{a}^{−}, F_{b}^{+}, F_{b}^{−}, F_{c}^{+}, F_{c}^{−}, correspond to the lines in ℂ ⊕ ℂ generated by the vectors

( \begin{matrix}1 \\ 0 \end{matrix} ) , ( \begin{matrix}0 \\ 1 \end{matrix} ) , X ( \begin{matrix}1 \\ 0 \end{matrix} ) , X ( \begin{matrix}0 \\ 1 \end{matrix} ) , Y ( \begin{matrix}1 \\ 0 \end{matrix} ) , Y ( \begin{matrix}0 \\ 1 \end{matrix} ) .

These vectors are precisely the ones in the statement of lemma 8.5. This proves lemma 8.5, and completes the proof of theorem 8.4.

9. The trace computation.

In this section, we prove theorem 2.3 by computing the trace of ρ′_{α}^{⊗k/2} and ρ_{a}^{⊗k} using the Lefschetz-Riemann-Roch fixed point formula. In the twisted case, the computation is rather straightforward, since M′ is smooth, and the relevant cohomology classes on the fixed point set are given in [NR]. This computation has been done in a different context by Pantev [Pa]. We repeat the calculation here and in the process we correct a misprint in his formula. In the untwisted case, the moduli space and the fixed point sets are not smooth. We circumvent this problem by transferring the computation to P, using some results of [BSz].

Note. Beauville [Be2] has recently computed the traces in the untwisted case in a different way by transferring the calculation to M′. ^{7} Beauville considers, more generally, rank r bundles, and his formula agrees with ours in the case r = 2. He does not, however, choose lifts to the line bundle L, and his result (for r = 2) concerns only the case k ≡ 0 mod 4, where one has a group action of J^{(2)}.

Proposition 9.1. The trace of the involution ρ′_{α}^{⊗k/2} is given by

Tr(ρ′_{α}^{⊗k/2}) = (−1)^{k/2} (\frac{k + 2}{2})^{g−1} .

Proof. The Lefschetz-Riemann-Roch fixed point formula [AS] states that

Tr(ρ′_{α}^{⊗k/2}) = C̃h(L′|_{|M′|_{α}})^{k/2} C̃h(λ_{−1}N_{α})^{−1} Td(|M′|_{α}) ∩ [|M′|_{α}].

Here Td(|M′|_{α}) is the Todd class, N_{α} is the conormal bundle of |M′|_{α}, λ_{t} is the operation defined by λ_{t}E = ∑ t^{i}Λ^{i}E, and

C̃h(E) = Ch(E_{+} − E_{−}),

where Ch is the Chern character, and, for any ℤ/2-equivariant bundle E over the fixed point set |M′|_{α}, E_{+} and E_{−} are the ±1-eigenbundles.

Since the fixed point set |M′|_{α} is isomorphic to the Prym variety P_{α}, its Todd class is 1. The cohomology class C̃h(λ_{−1}N_{α}) was computed in Proposition 4.2 in [NR]. Let Θ be the restriction to P_{α} of the principal polarization of J_{0}(Σ^{α}). The result of Narasimhan and Ramanan then states that

C̃h(λ_{−1}N_{α}) = 2^{2(g−1)}e^{−2Θ}. (18)

By the construction of L′ in [DN], we have the following relation between Θ and Ch(L′|_{|M′|_{α}}).

Ch(L′|_{|M′|_{α}}) = e^{2Θ}. (19)

Note that C̃h(L′|_{|M′|_{α}}) = −Ch(L′|_{|M′|_{α}}), by our definition of ρ′_{α}. Hence we get that

Tr(ρ′_{α}^{⊗k/2}) = (−1)^{k/2}2^{−2(g−1)}e^{(k+2)Θ} ∩ [P_{α}].

Using Corollary 4.16 in [NR], which states that

Θ^{g−1} ∩ [P_{α}] = (g − 1)!2^{g−1}, (20)

the result follows.

Proposition 9.2. The trace of the involution ρ_{a}^{⊗k} is given by

Tr(ρ_{a}^{⊗k}) = \frac{1 + (−1)^{k}}{2} (\frac{k + 2}{2})^{g−1} .

7Our computation was done independently of his.

Proof. The morphism q induces a J^{(2)}-equivariant morphism

q^{∗} : H^{0}(M, L^{k}) → H^{0}(P, q^{∗}L^{k}).

According to [BSz] we have that this morphism is an isomorphism and that

H^{i}(P, q^{∗}L^{k}) = 0,

for i > 0. Hence we just need to apply the Lefschetz-Riemann-Roch fixed point theorem to (P, q^{∗}L^{k}). The fixed point set |P|_{α} has two components, |P|_{a}^{+} and |P|_{a}^{−}, each of which is isomorphic to the Prym variety P_{α}, and hence has trivial Todd class. In order to understand q^{∗}L|_{|P|_{α}} and the conormal bundle N(|P|_{α}), consider the following exact sequence

0 → (q′)^{∗}T_{M′}^{∗} → T_{P}^{∗} → ν → 0,

where ν is the relative cotangent sheaf of q′. We conclude that

K_{P} ≅ ν ⊗ (q′)^{∗}K_{M′} ≅ ν ⊗ (q′)^{∗}(L′)^{−2},

since K_{M′} ≅ (L′)^{−2} [DN]. By cor. 2.2 in [BSz] (see equation (14) in the proof of theorem 8.3), we get that

q^{∗}L^{2} ≅ (q′)^{∗}L′ ⊗ ν^{−1}.

Now recall from prop. 7.4 that the line bundle ν|_{|P|_{α}} is numerically trivial. This shows

Ch(q^{∗}L|_{|P|_{a}^{±}}) = (Ch((q′)^{∗}L′|_{|P|_{a}^{±}}))^{1/2} = e^{Θ}

where in the last equality we have used formula (19), after identifying both |P|_{a}^{+} and |P|_{a}^{−} with the Prym variety P_{α}. Next, observe that α acts as −1 on ν|_{|P|_{α}}, since one has an exact sequence (of bundles over |P|_{α})

0 → (q′)^{∗}N_{α} → N(|P|_{α}) → ν|_{|P|_{α}} → 0. (21)

where N_{α} = N(|M′|_{α}) as before. Therefore

C̃h(λ_{−1}ν|_{|P|_{a}^{±}}) = C̃h(1 − ν|_{|P|_{a}^{±}}) = Ch(1 + 1) = 2,

and the exact sequence (21) gives us that

C̃h(λ_{−1}(N(|P|_{a}^{±}))) = 2 C̃h(λ_{−1}(N_{α})) = 2 · 2^{2(g−1)}e^{−2Θ},

where we have used formula (18) in the last equality, after again identifying |P|_{a}^{+} and |P|_{a}^{−} with P_{α}.

Now recall that ρ_{a} acts with opposite signs on the restriction of q^{∗}L to the two components |P|_{a}^{+} and |P|_{a}^{−}. In fact, it acts as ∓1 over |P|_{a}^{±}, by prop. 7.3. Therefore

C̃h(q^{∗}L|_{|P|_{a}^{±}}) = ∓Ch(q^{∗}L|_{|P|_{a}^{±}}) = ∓e^{Θ}.

Putting everything together, the fixed point formula gives

Tr(ρ_{a}^{⊗k}) = (1 + (−1)^{k})e^{kΘ} 2^{−1}2^{−2(g−1)}e^{2Θ} ∩ [P_{α}].

The proposition now follows as in the twisted case from formula (20).

References

[ACGH] E. Arbarello, M. Cornalba, P.A. Griffiths & J. Harris. Geometry of Algebraic Curves. Vol. I. Springer-Verlag New York 1985.

[At1] M. F. Atiyah. Riemann surfaces and spin structures. Ann. Scien. Ec. Norm. Sup. 4^{e} série 4 (1971), 47-62.

[At2] M. F. Atiyah. The geometry and physics of knots. Cambridge University Press, Cambridge 1990.

[AS] M. F. Atiyah, I. M. Singer. The index of elliptic operators III. Ann. of Math. 87, 546-604 (1968).

[ADW] S. Axelrod, S. Della Pietra & E. Witten. Geometric quantization of Chern Simons gauge theory. J. Diff. Geom. 33 (1991) 787-902.

[Be1] A. Beauville. Fibrés de rang 2 sur une courbe, fibré déterminant et fonctions thêta II. Bull. Soc. Math. France 119 (1991) 259-291.

[Be2] A. Beauville. The Verlinde formula for PGL_{p}. Preprint alg-geom/9609017.

[BL] A. Beauville, Y. Laszlo. Conformal blocks and generalized theta functions. Comm. Math. Phys. 164 (1994) 385-419.

[BLS] A. Beauville, Y. Laszlo, Ch. Sorger. The Picard Group of the Moduli of G-bundles on a Curve. Preprint alg-geom/9608002

[BNR] A. Beauville, M. S. Narasimhan, S. Ramanan. Spectral curves and the generalised theta divisor. J. Reine angew. Math. 398 (1989) 169-179.

[BSz] A. Bertram & A. Szenes. Hilbert polynomials of moduli spaces of rank 2 vector bundles II. Topology 32 (1993) 599-609.

[BM] C. Blanchet, G. Masbaum. Topological quantum field theories for surfaces with spin structure. Duke Math. J. 82 (1996) 229-267.

[BHMV] C. Blanchet, N. Habegger, G. Masbaum, P. Vogel. Topological Quantum Field Theories derived from the Kauffman bracket. Topology 34 (1995), 883-927.

[CLM] S.E. Cappell, R. Lee & E.Y. Miller. Invariants of 3-manifolds from conformal field theory, Topology 90, Ohio state University Mathematical Research Institute Publications 1.

[DN] J.-M. Drezet et M.S. Narasimhan. Groupe de Picard des variétés de modules de fibrés semi-stables sur les courbes algébriques. Invent. Math. 97 (1989), 53-94.

[Fa] G. Faltings. Stable G-bundles and projective connections. J.Alg.Geom. 2 (1993) 507-568.

[vGP1] B. van Geemen, E. Previato. Prym varieties and the Verlinde formula. Math. Ann. 294 (1992) 741-754.

[vGP2] B. van Geemen, E. Previato. Heisenberg action and Verlinde Formulas. In Verdier Memorial Conf. (O. Babelon et al., Ed.) Birkhäuser 1994.

[GH] P. Griffiths, J. Harris. Principles of algebraic gometry. Wiley-Interscience (1978).

[Hi] N. Hitchin. Flat connections and geometric quantization. Comm. Math. Phys. 131 (1990) 347-380.

[Ho] E. W. Howe. The Weil pairing and the Hilbert symbol. Math. Ann. 305 (1996) 387-392.

[Jo] D. Johnson. Spin structures and quadratic forms on surfaces. J. London Math. Soc. (2) 22 (1980) 365-377.

[KNR] S. Kumar, M.S. Narasimhan, A. Ramanathan. Infinite Grassmannians and moduli spaces of G-bundles Math. Ann. 300 (1994) 41-75.

[L] Y. Laszlo. Dimension de l’espace des sections du diviseur thêta généralisé, Bull. Soc. Math. France 119 (1991) 293-306.

[MS] G. Moore & N. Seiberg. Classical and quantum Conformal Field Theory. Comm. Math. Phys. 123 (1989) 177-254.

[MV] G. Masbaum, P. Vogel. Verlinde Formulae for Surfaces with Spin Structure. Contemp. Math. 164 (1994) 119-137.

[NR] M. S. Narasimhan, S. Ramanan. Generalised Prym varieties as fixed points. J. Indian Math. Soc. (N.S.) 39 (1975) 1-19.

[OP] W. Oxbury, Ch. Pauly. SU(2)-Verlinde spaces as Theta-spaces on Pryms. Int. J. Math. 7 (1996) 393-410.

[Pa] T. Pantev. Comparison of generalized theta functions. Duke Math. J. 76 (1994), 509-539.

[P] Ch. Pauly. On Pryms, rank 2 bundles and nonabelian theta functions. Int. J. Math. 8 (1997) 267-287.

[RSW] T.R. Ramadas, I.M. Singer and J. Weitsman. Some Comments on Chern - Simons Gauge Theory. Commun. Math. Phys. 126 (1989) 409-420.

[RT] N. Yu. Reshetikhin, V. G. Turaev. Invariants of 3-manifolds via link polynomials and quantum groups, Invent. Math. 103 (1991), 547-597.

[S] G. Segal. The Definition of Conformal Field Theory. Preprint.

[Sz] A. Szenes. Hilbert polynomial of moduli spaces of rank 2 vector bundles. Topology 32 (1993) 587-597.

[Th1] M. Thaddeus. Conformal field theory and the cohomology of the moduli spaces of stable bundles. J. of Diff. Geom. 35 (1992), 131-150.

[Th2] M. Thaddeus. Stable pairs, linear systems and the Verlinde formula. Invent. Math. 117 (1994) 317-353.

[Ve] E. Verlinde. Fusion rules and modular transformations in 2d conformal field theory. Nuclear Phys. B 300 (1988) 360-376.

[Wi] E. Witten. Quantum field theory and the Jones polynomial. Commun. Math. Phys 121 (1989) 351-98.

Dept. of Math., Univ. of Aarhus, DK-8000 Aarhus C, Denmark

  E-mail address: andersen@mi.aau.dk

Institut de Mathématiques de Jussieu, Equipe ‘Théories Géométriques’, Case 7012, Université Paris VII, 75251 Paris Cedex 05, France

  E-mail address: masbaum@math.jussieu.fr