Observation of the in-plane magnetic field-induced phase transitions in FeSe
Jong Mok Ok Chang Il Kwon Yoshimitsu Kohama Jung Sang You Sun Kyu Park Ji-hye Kim Y. J. Jo E. S. Choi Koichi Kindo Woun Kang Ki Seok Kim E. G. Moon A. Gurevich Jun Sung Kim
cond-mat.supr-con cond-mat.mtrl-sci cond-mat.str-el








Department of Physics, Pohang University of Science and Technology, Pohang 37673, Korea
Center for Artificial Low Dimensional Electronic Systems, Institute for Basic Science, Pohang 37673, Republic of Korea

Department of Physics, Pohang University of Science and Technology, Pohang 37673, Korea
Center for Artificial Low Dimensional Electronic Systems, Institute for Basic Science, Pohang 37673, Republic of Korea

ISSP, University of Tokyo, Kashiwa, Chiba 277-8581, Japan

Department of Physics, Pohang University of Science and Technology, Pohang 37673, Korea
Center for Artificial Low Dimensional Electronic Systems, Institute for Basic Science, Pohang 37673, Republic of Korea

Department of Physics, Pohang University of Science and Technology, Pohang 37673, Korea
Center for Artificial Low Dimensional Electronic Systems, Institute for Basic Science, Pohang 37673, Republic of Korea

Department of Physics, Kyungpook National University, Daegu 702-701, Republic of Korea

Department of Physics, Kyungpook National University, Daegu 702-701, Republic of Korea

National High Magnetic Field Laboratory, Florida State University, Tallahassee, Florida 32310, USA

ISSP, University of Tokyo, Kashiwa, Chiba 277-8581, Japan

Department of Physics, Ewha Womans University, Seoul 03760, Republic of Korea

Department of Physics, Pohang University of Science and Technology, Pohang 37673, Korea

Department of Physics, Korea Academy Institute of Science and Technology, Daejeon 34141, Republic of Korea

Department of Physics, Old Dominion University, Norfolk, VA 23529, USA

js.kim@postech.ac.kr
Department of Physics, Pohang University of Science and Technology, Pohang 37673, Korea
Center for Artificial Low Dimensional Electronic Systems, Institute for Basic Science, Pohang 37673, Republic of Korea


We investigate thermodynamic properties of FeSe under the in-plane magnetic fields using torque magnetometry, specific heat, magnetocaloric measurements. Below the upper critical field H_c2, we observed the field-induced anomalies at H_1 ∼ 15 T and H_2 ∼ 22 T near H ∥ ab and below a characteristic temperature T^* ∼ 2 K. The transition magnetic fields H_1 and H_2 exhibit negligible dependence on both temperature and field orientation. This contrasts to the strong temperature and angle dependence of H_c2, suggesting that these anomalies are attributed to the field-induced phase transitions, originating from the inherent spin-density-wave instability of quasipaticles near the superconducting gap minima or possible Flude-Ferrell-Larkin-Ovchinnikov state in the highly spin-polarized Fermi surfaces. Our observations imply that FeSe, an atypical multiband superconductor with extremely small Fermi energies, represents a unique model system for stabilizing unusual superconducting orders beyond the Pauli limit.





Observation of the in-plane magnetic field-induced phase transitions in FeSe
    Jun Sung Kim
    December 30, 2023
============================================================================





§ INTRODUCTION




In conventional superconductors 

spin polarization can destroy superconductivity when Zeeman energy surpasses the binding energy of Cooper pairs, known as the Pauli paramagnetic limit. In unconventional superconductors, however, exotic superconducting phases have been observed beyond the Pauli limit <cit.>, often coexisting with complex magnetic orders. For example,
the Fulde-Ferrel-Larkin-Ovchinnikov (FFLO) state was proposed a half century ago, <cit.> in which s-wave Cooper pairs acquire a finite center-of-mass momentum to form a spatially modulated superconducting order. <cit.> Alternatively, spin-triplet pairing of field-induced quasiparticles can occur particularly in unconventional d-wave superconductors, inducing novel phases with mixture of spin-singlet and spin-triplet pair density waves. <cit.> Growing evidence has revealed that the former and the latter are realized in highly anisotropic organic superconductors and a heavy fermion superconductor CeCoIn_5, respectively. <cit.>
In both systems, however, 
the typical field-induced spin imbalance σ = (n_↑-n_↓)/(n_↑+n_↓), where ↑ and ↓ refer to the two spin components with densities n_↑,↓, is at most a few %. 
Realization of the highly spin-imbalanced superconductivity and possible field-induced exotic phases have 
been remained elusive so far. 

Usually in single-band superconductors, Cooper pairs are broken by the orbital pair breaking effect or by the Pauli paramagnetic effect  <cit.>. The orbital pair breaking effect destroys superconductivity at the relatively low upper critical fields H_c2 with negligible spin imbalance. Even the orbital effect is largely suppressed, the Pauli paramagnetic effect limits the spin imbalance, typically σ ∼ 10^-2. This is because the maximum Zeeman energy μ_B H_c2 (μ_B, Bohr magneton) is set by the superconducting gap Δ_ SC and usually much smaller than the Fermi energy E_ F. Multiband superconductors however may host the strongly spin imbalanced state. If the E_ F is small in one band with a small Δ_ SC and the superconductivity is maintained by the other band with a relatively larger Δ_ SC  <cit.>, spin polarization can be significantly enhanced under high magnetic fields .

Such a candidate is FeSe with σ ∼ 10^-1. FeSe is
known to be in the so-called BCS-BEC (Bose-Einstein condensate) crossover regime (Δ_SC∼ E_F) <cit.>, and also in
the Pauli limiting regime (μ_B H_c2∼Δ_SC) <cit.> under in-plane magnetic fields.
Possible magnetic field induced phase transitions have been suggested under the out-of-plane fields <cit.>, and very recently, also in the in-plane fields <cit.>. Therefore FeSe can be a model system to study whether, and if so how, competing magnetic or superconducting instabilities trigger exotic field-induced phases in multiband superconductors <cit.> or in the large spin imbalance regime <cit.>.
In this work, using torque magnetometry, specific heat, and magnetocaloric measurements, we identified successive anomalies with increasing in-plane magnetic field, below H_c2 at low temperatures and in the clean limit. These results evidence that unusual superconducting phases are indeed stabilized in FeSe at high in-plane magnetic fields.



§ EXPERIMENTS


Single crystals of FeSe were grown by a KCl-AlCl_3 flux technique <cit.>. The resistivity of single crystals
were measured using the standard four-probe method in a Physical Property Measurement
System (PPMS-14T, Quantum Design) in order to determine the residual resistivity ratio
(RRR). Single crystals with the RRR value higher than 35 were used in the high magnetic
field experiments unless otherwise noted (Supplementary Fig. S1).

 The upper critical field H_c2 of FeSe was determined using the resistivity and the tunnel diode oscillator (TDO) frequency measurements at high magnetic fields. We measured the resistive transition under magnetic field up to 30 T in National High Magnetic Field Laboratory (NHMFL), Tallahassee. For resistivity measurements, a room temperature cured silver paste/epoxy was used to connect gold wires to the sample. The TDO measurements up to 30 T were performed in International MegaGauss Science Laboratory in Institute for Solid State Physics (ISSP), University of Tokyo. In the TDO measurements, we recorded the change of the radio frequency in an LC circuit with a crystal wound in a copper coil. For pressure experiments, quasi-hydrostatic pressure was applied up to 35 kbar using a home-made indenter-type pressure cell. The superconducting transition of lead, mounted next to the FeSe crystal, was used to determine pressure inside the cell at low temperatures (Supplementary Fig. S3).

 The thermodynamic anomalies in the superconducting state were observed using the measurements on torque magnetometry, magnetocaloric effect, and specific heat under magnetic fields up to 30 T at NHMFL. For torque magnetometry measurements, we used a miniature Seiko piezoresistive cantilever. On top of the cantilever we mounted a small single crystal, typically 100×100×20 μm^3, extracted from the crystal whose RRR value was determined from the preceding resistivity measurements. During the up- and down-sweeps of the applied magnetic field, the resistance of the piezoelectric cantilever was monitored, which is proportional to the magnetic torque acting on the cantilever. The in-plane magnetic field is applied along the sample edge (Supplementary Fig. S5), which is nearly along the diagonal direction of the orthorhombic lattice. For magneto-caloric effect and specific heat measurements, the samples were mounted on top of a bare chip Cernox or RuO_2 thermometer, suspended by manganin wires (diameter ∼ 10 μm) as shown in Supplementary Fig. S6. Detailed information on experiments are available in Supplementary Materials.



§ RESULTS



 §.§ Characteristics of FeSe





FeSe is the simplest but an atypical member of FeSCs, which has T_c ≈ 9 K and is composed of edge-shared FeSe_4 tetrahedra layers in van der Waals (vdW) structure <cit.>. The observed in-plane upper critical field H_c2^ab≈ 25 T is well above the Pauli limiting field H_P ≈ 15.6 T at the BCS limit (H_P= 1.84T_c), which indicates that FeSe is in the Pauli limit under in-plane magnetic fields (μ_B H_c2∼Δ_SC). In addition, high-quality FeSe single crystals with a high residual resistive ratio (RRR) ≥ 35 are in the clean limit, as confirmed by magnetic quantum oscillation studies <cit.> (Supplementary Fig. S1). More importantly, FeSe has exceptionally small Fermi energies <cit.>. In FeSe, three Fermi surfaces (FSs) exist; one hole FS is centered at the Γ point and denoted as α, two electron FSs are located at the X and Y points and denoted as δ and ϵ, respectively (Figs. 1b and 1c) <cit.>. The low energy structure of each FS can be described by H=ħ^2 k_x^2/2m_x+ħ^2 k_y^2/2m_y+2tcos(k_z), where m_x,y is the effective mass along the in-plane momentum k_x,y directions, and t is the interlayer hopping parameter. We note that this model does not capture the detailed FS shapes, such as a bow-tie shape of ϵ FS, but provide a qualitative picture of FSs under high magnetic fields. Here the x axis (y axis) is defined to be parallel to the a axis (b axis) of the Fe orthorhombic lattice (Fig. 1a). Based on recent quantum oscillations and angle-resolved photoemission studies <cit.>, the parameters for the low energy structures can be obtained as summarized in Supplementary Table S1. The Fermi energies for these bands are E_F ≈ 7.9 meV, 6.1 meV, and 2.9 meV for α, δ and ϵ pockets, respectively. These values are comparable with their superconducting energies, Δ_ SC= 1.64 meV, 1.53 meV, and 0.39 meV for α, δ and ϵ pockets, respectively <cit.>. The ratio Δ_SC/E_F is ∼ 0.2 for all the FSs, confirming that FeSe is in the BCS-BEC crossover regime.

Under magnetic fields, the Zeeman effect induces imbalance of spin population as well as momentum mismatch between FSs of ↑ and ↓ spins. In FeSe with three FSs, spin imbalance and momentum mismatch are different at each FS, as shown in Figs. 1e and 1f. The spin imbalance in three FSs increases with magnetic fields and reaches up to  15% for α and δ FSs and  40% for ϵ at H_c2^ab≈ 25 T s, assuming the g-factor of 2. These are much larger than the typical value of a few % in other superconductors <cit.>. The momentum mismatch is also significant and is quantified by the size difference of the FS, Δ A = A_↑-A_↓, where A denotes the cross-sectional area of each FS in the k_z planes. The ratio Δ A(H)/A(0) in the k_z=0 plane becomes 10%-50% at H_c2^ab≈ 25 T and is particularly large for the α-FS having a significant k_z warping (Fig.1f and Supplementary Fig. S8). The α-FS with ↑ spin in the k_z=0 plane may disappear at ∼ 40 T, resulting in a magnetic Lifshitz transition <cit.> well above the in-plane H_c2^ab.




 §.§ Upper critical fields





Having established that FeSe is simultaneously in the clean limit, in the BCS-BEC crossover regime, and in the Pauli limit under in-plane magnetic fields, we now discuss its magnetic phase diagram. Using the tunnel-diode oscillator frequency and the resistivity measurements, we obtained upper critical field H_c2 as a function of temperature (Fig. 2a) and field orientation (Fig. 2b), which are consistent with each other and independent of the types of measurements or criteria for determining H_c2 (Supplementary Fig. S2). For H ∥ c, the zero-temperature upper critical field H_c2(0) is estimated to be H_c2^c ≈ 15 T. At low temperatures, H_c2^c (T) data deviate from the conventional Werthamer-Helfand-Hohenberg (WHH) prediction, which is well explained by the two-band model, as found in other FeSCs with significant interband interactions <cit.>. We note that the field-induced phase transition <cit.> observed at H ∼ 15 T for H∥ c does not affect the H_c2^c(T) behavior.

In contrast, near H ∥ ab, we observe unusual behaviors of H_c2 in both the temperature- and the field angle-dependences. Here the in-plane magnetic field is applied along the diagonal direction of the orthogonal lattice (Fig. 1a) and thus denoted as H∥ ab. In this case, the quasi-2D nature of FeSe suppresses the orbital pair breaking effect and enhances H_c2^ab(0) up to ∼ 25 T, larger than H_P (Fig. 2a). Usually in FeSCs, the Pauli-limiting effect is more important than the multiband orbital pair breaking effect under in-plane magnetic fields at low temperatures <cit.>. With a large Maki parameter α_M=√(2) H_c2^orb/H_P > 1, where H_c2^orb is the orbital-limiting upper critical field (H_c2^orb =0.69T_c |dH_c2/dT|_T_c), H_c2^ab(T) curves of FeSCs, such as LiFeAs <cit.>, KFe_2As_2 <cit.>, exhibit a saturation behavior at low temperatures and are well described by the single-band WHH model <cit.>. In FeSe, however, we observed a weak but discernable kink in H_c2^ab(T) curve at T^* ≈ 2 K. Consistently the single-band WHH model with α_M= 1.5 reproduces the measured H_c2^ab (T) above T^*, but not its unusual upturn below T^*≈ 2 K. Furthermore, at T= 0.88 K, just below T^*, H_c2(θ) as a function of field angle (θ) clearly deviates from the GL model near H∥ ab with θ≤ 15^∘. 
This contrasts the H_c2(θ) data taken at T = 2.5 K, just above the T^*, which follows nicely the anisotropic Ginzburg-Landau (GL) equation, H_c2(θ)=H_c2^c/√((cos^2θ+Γ_H^-2sin^2θ)), where Γ_H is the mass anisotropy m_ab/m_c and θ is the tilting angle of the field from the in-plane orientation. The deviations of H_c2^ab (T) and H_c2(θ) from the WWH and GL models can in principle be related to the multiband effect <cit.>. This, however, cannot explain the kink of H_c2^ab (T) at T^* suggesting possible field-induced phase transitions, as discuss in the following section.

The pressure dependence of the low-temperature upturn in H_c2^ab(T) suggests that the observed upturn in H_c2^ab (T) is intimately related to the Pauli-limiting effect. It has been known that FS reconstruction occurs at a critical pressure P_c ∼ 8 kbar <cit.>, suggested by quantum oscillations and upper critical field studies <cit.>. With external pressures below and above P_c, we found a systematic variation of H_c2(T). For H ∥ c, H_c2^c (T) curves, taken at different pressures and by H_c2^orb, follow the same temperature dependence as clearly shown in Fig. 2c. This implies that the relative ratios in the sizes or the diffusivity between dominant hole and electron pockets of FeSe are not significantly changed across P_c. On contrary, the Maki parameter α_M, taken from the fit of H_c2^ab (T) curves to the WHH model, is reduced systematically with pressure (Fig. 2d). At high pressures, therefore, the orbital-limiting effect becomes more important for determining H_c2^ab than the Pauli limiting effect. This is consistent with the fact that the slops of H_c2^ab at T_c, i.e. dH_c2^ab/dT|_T_c for H∥ ab, that is proportional to the Fermi velocity along the c axis, is reduced with increasing pressure (Fig. 2e) <cit.>.
Concomitantly, the unusual upturn in H_c2^ab (T) at low temperatures is systematically suppressed and eventually disappears for α_M < 1 (Fig. 2c). These observations suggest the observed upturn in H_c2^ab (T) is not simply due to the multiband effect, but may reflect formation of field-induced phase beyond the Pauli limit. In fact, this low temperature behavior in H_c2^ab (T) resembles those of the unconventional Pauli-limiting superconductors, κ-(BEDT-TTF)_2Cu(NCS)_2 and CeCoIn_5 (Supplementary Fig. S2) <cit.>, and is consistent with the recent work on FeSe <cit.>.



 §.§ Magnetic field-induced phase transitions



The signatures of unusual phase transitions below H_c2^ab and below T^* ≈ 2 K are observed in the field-dependent torque magnetometry τ(H). The torque magnetometry is known to be extremely sensitive to the small changes in the magnetic susceptibility and usually exhibits a large magnetic hysteresis due to vortex pinning in superconductors <cit.> (Figs. 3a and 3c). This produces a typical saw-tow shaped curve of the field derivative τ(H), dτ/dH, for up- and down-sweeps of magnetic fields. In FeSe, however, we observed two additional anomalies at H_1 ∼ 15 T and H_2 ∼ 22 T, as indicated by arrows in Figs. 3a and 3b, well below the irreversible field H_ irr at which the hysteresis in τ(H) and dτ(H)/dH starts to develop. The transition fields H_1 and H_2 follow the distinct dependence on temperature or field angle from  those of H_ irr ≈ H_c2. Upon increasing temperature or the tilting angle of the magnetic field (θ) from H∥ ab (Figs. 3a and 3c), H_ irr systematically decreases, similar to H_c2 (Figs. 2a and 2b). In contrast, the anomalies are pronounced only at low temperatures below T^* and near H ∥ ab with θ≤ 15^∘. Also the transition fields H_1 and H_2 remain almost the same with variation of temperature or field angle. 


These observations allow us to rule out the vortex-related phenomena as a possible origin for the anomalies at H_1 and H_2.
It has been well known that the vortex phase transitions, such as melting or modification of the vortex structure and the peak effect, usually share the similar dependence on temperature and field orientation with the upper critical field H_c2 <cit.>, particularly in highly anisotropic superconductors <cit.>, like FeSe <cit.>. In contrast, the transition fields observed in FeSe are nearly independent of field angle. Instead, this is one of the characteristic features of the field-induced exotic phases in e.g. κ-(BEDT-TTF)_2Cu(NCS)_2 and CeCoIn_5 <cit.>.
Furthermore, we found that the anomalies at H_1 and H_2 are very sensitive to disorder, whose amount is quantified by the residual resistivity ratio, RRR = ρ(300   K)/ρ(T_c). As shown in Supplementary Fig. S5, the anomalies at H_1 and H_2 in dτ(H)/dH are present for the crystals with a large RRR ≥ 35. In the crystal with a small RRR ≈ 10, we obtained the typical saw-tow shaped  curves of dτ(H)/dH without additional anomalies. The strong sensitivity to disorder is also consistent with the field-induced phase transition in the Pauli-limiting superconductors <cit.>.

The magnetocaloric effect (MCE) provides further thermodynamic evidence regarding the field-induced phase transitions. Once a superconducting sample is placed in a changing magnetic field, difference between the sample temperature (T) and the bath temperature (T_ bath) is determined by the entropy (S) change and the specific heat C_s contribution and is described by (∂ S/∂ H)_T=-(C_s/T)(dT/dH)-κ (T-T_ bath)/TH-(1/T)dQ_ loss/dH. Here, κ is the thermal conductivity between the sample and the environments <cit.>, and Q_ loss corresponds to an additional heat due to the irreversible effect. In our case, the sample is weakly coupled to the bath, due to a relatively low κ, i.e. being close to the adiabatic condition.
Then the field dependent temperature T(H) can be considered as an isentropic curve where the decrease (increase) of temperature implies the increase (decrease) of entropy.

With increasing magnetic field, we observed weak but clear anomalies in R_T (H) (Figs. 3d and 3e), the resistance of a temperature sensor attached to the crystal, which roughly proportional to the inverse of the sample temperature and becomes similar to the shape of the field dependent entropy, S(H). In conventional superconductors the monotonous field dependence of R_T (H) is usually expected with an anomaly only at H_c2. However in our experiments we observed additional anomalies in both up- and down-sweeps of magnetic fields at similar H_1 and H_2 (Supplementary Fig. S6). With variation of temperature and field orientation, these MCE anomalies appears only below T^* and near H∥ ab with θ≤ 15^∘. The transition fields H_1 and H_2 are nearly independent of temperature and field angle, in good agreement with the τ(H) results (Figs. 3a and 3c), as summarized in the magnetic phase diagrams as a function of temperature and field-orientation (Figs. 4a and 4b).



The magnetic field dependent electronic specific heat C_s/T of FeSe also shows consistent results. In Fig. 3f, we plot C_s/T, taken at 0.35 K, as a function of the normalized magnetic field h = H/H_c2 for H ∥ c and H ∥ ab. In both cases, C_s/T increases rapidly at low magnetic fields below h < 0.3, which indicates the field-induced closing of smaller SC gaps mostly in the δ- and the ϵ-FSs, consistent with the previous report <cit.>. Thus at higher magnetic fields, the field dependence of C_s/T is dominated by the contribution from the α-FS and is expected to follow a nearly linear field dependence below H_c2 for both H ∥ c and H ∥ ab. This would make C_s(h)/T curves collapsed into a single curve, once plotted as a function of the normalized h. However in FeSe at higher fields, C_s(h)/T exhibit different behaviour depending on the field orientations. For H ∥ c, C_s(h)/T increases almost linearly until it is saturated at H_c2, as expected. In contrast, for H ∥ ab, C_s(h)/T deviates from the linear field dependence at h ≈ 0.5 with a slope change, and then recovers its normal state value at H_c2. The difference Δ C_s(h)/T = C_s(h)/T|_H∥ ab-C_s(h)/T|_H∥ c reveals the anomaly more clearly. The drop of Δ C_s(h)/T by ∼ 0.4 mJ/mol K^2 occurs clearly at h ≈ 0.5, corresponding to H_1 ∼ 15 T, while the anomaly H_2 ∼ 22 T appears to be smeared out near H_c2.

Combined with anomalies seen in torque and magnetocaloric measurements, the specific heat anomaly confirms the presence of field-induced phases near the in-plane magnetic fields below H_c2^ab.



§ DISCUSSION


Now we discuss possible scenarios for the in-plane field-induced phase transitions in FeSe. One candidate is the magnetic Lifshitz transition, involving the field-induced change of the FS topology. This transition has been proposed to occur in FeSe due to the small E_ F <cit.>, which is however unlikely because the following reasons. Firstly, in order to have a magnetic Lifshitz transition at H ∼ 20 T, e.g. in the α-FS having the smallest A at k_z=0, the g-factor of FeSe must be ∼5-14, unreasonably large for FeSCs <cit.>. Secondly, the transition field for magnetic Lifshitz transition is set by electronic structures and is not directly related to the presence of the superconducting order. It is thus expected to occur even in the normal state, which is not the case in experiments <cit.>. Experimentally the field-induced phase transitions are only observed in the narrow ranges of temperature and field angle within the superconducting phase, and their phase boundaries are never extended to the normal state (Figs. 4a and 4b).

One possible candidate is then the FFLO phase <cit.>. In this case, the Cooper-pair state (k ↑, -k+q ↓) is formed with a momentum mismatch (q) between the spin-split FSs, inducing a spatially modulated superconducting order with a characteristic length of q^-1. In multiband superconductors, each FS has its own FFLO instability with a different modulation length q_i^-1 (i, band index), and these instabilities compete with each other, inducing the multiple FFLO orders at different magnetic fields <cit.>. It is however rather unlikely that both transitions observed at H_1 and H_2 in FeSe are due to the multiple FFLO transitions. Such a multiple FFLO ordering, if exists, is fragile to the interband coupling, because the FFLO instabilities with unequal q's would average the interband pairing energy ∼ Δ_1 Δ_2cos[(q_1-q_2)r] to zero (Δ_i, the superconducting gap at a different band i), and strongly reduces T_c under magnetic fields. Therefore, multiple FFLO transitions are allowed only if the interband coupling are drastically suppressed. As discussed below, however, the nesting between spin-split FSs in FeSe are expected to be enhanced near H_1 and H_2 (Figs. 4c and 4d). Furthermore, we found that the entropy is reduced across H_1 and H_2 for the up-sweep (Figs. 3d and 3e). 
For typical FFLO transitions, entering to the FFLO phase in the up-sweep usually increases the entropy <cit.> because the additional quasiparticles are introduced due to the spatially-inhomogeneous superconducting gap.

Another promising candidate is the spin-density-wave (SDW) phase of field-induced quasiparticles. 
As it has been discussed for a d-wave superconductor CeCoIn_5 <cit.>, the SDW order can be triggered by the nesting effect in the k-space near the superconducting gap nodes, where the superconductivity is suppressed by Pauli pair breaking. This nesting effect together with the residual AFM fluctuations is enhanced only in the superconducting state <cit.>, allowing the coexisting SDW and superconducting phase.
Recent experimental and theoretical studies <cit.> revealed that the hole (α) and the electron (δ and ϵ) FSs have the opposite in-plane momentum anisotropy in their superconducting gap as shown in Fig. 1c. Furthermore, two different types of spin fluctuations are dominant in FeSe, the stripe-type spin fluctuation with q=(π,0) and the Neel-type one with q=(π,π) <cit.>. Therefore, if the incipient spin fluctuations with q=(π,0) or q=(π,π) strongly couple the field-induced quasiparticles near the superconducting gap nodes or minima, the field-induced SDW orders can coexist with the superconducting state and induce the SDW transitions in FeSe. Note that such a SDW order is stable only within the superconducting phase, which is consistent with our phase diagram (Fig. 4a and 4b) and compatible with the absence of FS reconstruction above H_c2 <cit.>.

In order to consider the nesting instability of quasiparticles near the superconducting gap nodes or minima, we estimate the magnetic field dependent Fermi vectors k_F in the planes of k_z=0 (Figs. 4c, 4d) and k_z=π (Supplementary Fig. S9) along k_x and k_y directions. Unlike other superconductors, FeSe has an exceptionally small E_ F, and the Zeeman effect with a typical g-factor of 2 <cit.> results in the spin-split FSs much different in size. This produces a large spin imbalance (Fig. 1d) and also significantly affects the nesting condition. We found that the nesting via q=(π,π) nicely matches the ϵ-FS of ↑ spins with the δ-FS of ↓ spins along the k_x direction at H ∼ 16 T (Fig. 4c), in good agreement with the observed transition field H_1 ∼ 15 T. This nesting effect with q=(π,π) and the resulting interband scattering would suppress the FFLO instability. However, this can couple the field-induced quasiparticles near the nodal superconducting gap regions of the δ and the ϵ FSs (the inset of Fig. 4b) through the Neel-type spin fluctuation <cit.>, leading to the phase transition at H_1. In this case, the entropy is expected to be reduced upon entering the SDW phase in the up-sweep due to decrease of the number of degrees of freedom, in agreement with the experiments (Figs. 3d and 3e). Moreover the specific heat is also expected to be reduced by the SDW gap formation of quasi-particles, again consistent with experiments (Figs. 3f). The observed reduction of C_s/T by ∼ 0.4 mJ/mol K^2 (Fig. 3f) is ∼ 10% of the total density of states of δ- and ϵ-FSs, indicating that only quasiparticles near the nodal gap region participate the SDW formation.

The higher field phase transition at H_2 ∼22 T appears to be more intriguing than one at H_1. We found another nesting condition via q=(0,π) at H ∼24 T, close to H_2, at which the α-FS of ↓ spins matches well with δ-FS of ↓ spins along the k_y direction at k_z=0 (Fig. 4d) and also along the k_x direction at k_z=0 (Supplementary Fig. S9). In this case, however, this nesting condition involves 
the states of α-FSs with the superconducting gap maxima 
(the inset of Fig. 4b), which does not favor the field-induced SDW order.
Instead, the FFLO phase due to the intraband superconducting coupling can be a candidate, dominantly in the α-FS, which has the largest superconducting gap and is not involved in the possible SDW ordering at H_1. Our calculations taking into account the multiband Pauli paramagnetic effect and the FFLO instability, but not the superconducting gap anisotropy, show that the hidden FFLO phase can stabilized at low temperatures (Supplementary Fig. S7), supporting this possibility. We also note that recent thermal conductivity study under the in-plane magnetic field reveal a clear kink at H_2, which is attributed to the FFLO phase transition <cit.>. The detailed nature of the field induced phase transitions, including possible coexistence of the SDW and the FFLO orders hosted by different FSs, remains to be clarified. Nevertheless these observations clearly demonstrate that FeSe offers a unique system, in which field-induced phase transitions occur in the superconducting state.



§ CONCLUSION

 We reported that FeSe undergoes multiple phase transitions under high in-plane magnetic fields below the upper critical fields.
 The extremely small Fermi energy, comparable with the energy scales of the superconducting gap and the field-induced Zeeman effect, is found to be essential to trigger exotic orders with a large spin imbalance, related with either the spin-density-wave or the FFLO phases. These findings add another intriguing aspect to the unique iron-based superconductor FeSe <cit.>, and also pose a challenge to our understanding on the complex interplay between anisotropic superconducting order, incipient magnetic instabilities, and the multi-band effect in the largely spin-imbalanced superconducting systems.






§ ACKNOWLEDGMENTS


The authors thank Y. Matsuda and D.J.Jang for fruitful discussion. We also thank H. G. Kim in Pohang Accelerator Laboratory (PAL) for the technical support. This work was supported by the Institute for Basic Science (IBS) through the Center for Artificial Low Dimensional Electronic Systems (no. IBS-R014-D1) and also by the National Research Foundation of Korea (NRF) through SRC (Grant No. 2018R1A5A6075964) and the Max Planck-POSTECH Center for Complex Phase Materials
(Grant No. 2016K1A4A4A01922028). W.K. acknowledges the support by NRF (No. 2018R1D1A1B07050087, 2018R1A6A1A03025340), and Y.J.J. was supported by NRF (No. 2016R1A2B4016656).



FF  P. Fulde, and R. A. Ferrell, Phys. Rev. 135, A550, (1964).
LO A. I. Larkin, and Yu. N. Ovchinnikov,  Sov. Phys. JETP 20, 762 (1965).
ffloreview Y. Matsuda, and H. Shimahara, J. Phys. Soc. Jpn. 76, 051005 (2007).
organicreview R. Beyer, and J. Wosnitza, Low Temp. Phys. 39, 3 (2013).
offlo4 H. Mayaffre et al., Nat. Phys. 10, 928-932 (2014).
hfq1 H. A. Radovan et al., Nature 425, 51-55 (2003).
kenzelmann1  M. Kenzelmannet al., Science 312, 1652-4 (2008).
kenzelmann2 M. Kenzelmann et al., Phys. Rev. Lett. 104, 127001 (2010).
QSDW S. Gerber et al., Nat. Phys. 10, 126-129 (2014).
kenzelmannreview M. Kenzelmann, Rep. Prog. Phys. 80, 034501 (2017).
WHHN. R.  Werthamer, E. Helfand, and P. C. Hohenberg, Phys. Rev. 147, 295 (1966).
review1  A. Gurevich, Rep. Prog. Phys. 74, 124501 (2011).
review2 J. -L. Zhang, et al., Front. Phys. 6, 463-473 (2011).
FeSe:BCS S. Kasahara, et al., Proc. Nat. Acad. Sci. USA 111 16309-16313 (2014).
FeSe:BCS2 S.  Kasahara et al., Nat. Commun. 7, 12843 (2016).
FeSeHc2_1 S. I. Vedeneev, B. A. Piot, D. K. Maude, A. V. Sadakov, Phys. Rev. B 87 134512 (2013).
FeSe_matsuda S. Kasahara et al., Phys. Rev. Lett. 124 107001 (2020).
FFLO0 A. Gurevich, Phys. Rev. B 82, 184504 (2010).
FFLO1 A. Ptok, J. Phys.: Condens. Matter 27, 482001 (2015).
FFLO2  M. Takahashi, T. Mizushima, K. Machida, Phys. Rev. B. 89, 064505 (2014).
spinim1 Y.  Shin et al., Nature 451, 689-693 (2008).
spinim2 Y. Liao et al., Nature 467, 567-569 (2010).
spinim3 F. Chevy et al., Rep. Prog. Rhys. 73, 112401 (2010).
FeSe:growth D. Chareev et al. Cryst. Eng. Commun. 15, 1989-1993 (2013).
FeSeNMR S.-H. Baek, et al., Nat. Mat. 14, 210-214 (2015).
FeSereview  A. I. Coldea, and  M. D. Watson, Annu. Rev. Condens. Matter Phys. 9 125-46 (2018).
FeSe:QO1 T. Terashima, N. Kikugawa, A. Kiswandhi, E. S. Choi, J. S. Brooks, S. Kasahara et al., Phys. Rev. B 90, 144517 (2014).
FeSe:QO2 M. D. Watson et al., Phys. Rev. B 91, 155106 (2015).
FeSe:QO3 T. Terashima, N. Kikugawa, A. Kiswandhi, D. Graf, E. S. Choi, J. S. Brooks et al., Phys. Rev. B 93,  93, 094505 (2016).
FeSe:ARPES Y. S. Kushnirenko et al., Phys. Rev. B 97, 180501(R) (2018).
FeSe:QPI P. O. Sprau  et al., Science 357, 75-80 (2017).
FeSe:Cp Y. Sun, S. Kittaka, S. Nakamura, T. Sakakibara, K. Irie, T. Nomoto, K. Machida, J. Chen, T. Tamegai, Phys. Rev. B 96, 220505(R) (2017).
ptok  A. Ptok et al., Sci. Rep. 7, 41979 (2017).
Li111:FFLO  K. Cho, H. Kim, M. A. Tanatar, Y. J. Song, Y. S. Kwon, W. A. Coniglio, C. C. Agosta, A. Gurevich, R. Prozorov, Phys. Rev. B 83, 060502(R) (2011).
K122:FFLO D. A. Zocco et al., Phys. Rev. Lett. 111, 057007 (2013).
Hc2slope V. G. Kogan, and R. Prozorov, Rep. Prog. Phys. 75, 114502 (2012).
torque L. Li et al., Nat. Phys. 3, 311-314 (2007).
torque2 B. Bergk, A. Demuer, I. Sheikin, Y. Wang, J. Wosnitza, Y. Nakazawa, R. Lortz, Phys. Rev. B 83, 064506 (2011).
vortex_BSCCO S. Ooi, T. Shibauchi, N. Okuda, and T. Tamegai, Phys. Rev. Lett. 82, 4308 (1999).
vortex_MgB2 D.-J. Jang, H.-S. Lee, B. Kang, H.-G. Lee, and S. I. Lee, Supercond. Sci. Technol. 21 125004 (2008).
FeSevortex Y. Sun, S. Pyon, T. Tamegai, R. Kobayashi, T. Watashige, S. Kasahara, Y. Matsuda, T. Shibauchi, Phys. Rev. B. 92, 144509 (2015).
MCE V. Zapf, M. Jaime, and C. D. Batista, Rev. Mod. Phys. 86, 563 (2014).
gfactor P. F. S. Rosa et al., Sci. Rep. 4, 6543 (2014).
entropy1 Y. Tokiwa, E. D. Bauer, and P. Gegenwart, Phys. Rev. Lett. 109, 116402 (2012).
entropy2 C. C. Agosta, N. A. Fortune, S. T. Hannahs, S. Gu, L. Liang, J. H. Park, J. A. Schleuter, Phys. Rev. Lett. 118, 267001 (2017).
Q_SDW1 Y. Kato, C. D. Batista, and  I. Vekhter, Phys. Rev. Lett. 107, 096401 (2011); Phys. Rev. B 86, 174517 (2012).
Q_SDW2 Y. Hatakeyama and R. Ikeda, Phys. Rev. B 83 224518 (2011).
FeSeGap1 G. -Y. Chen, X. Zhu, H. Yang, H. H. Wen, Phys. Rev. B 96, 064524 (2017).
FeSeGap2 J. -H. She, M. J. Lawler, E. A. Kim, Phys. Rev. Lett. 121, 237002 (2018).
FeSepipi2 Q.  Wang et al., Nat. Commun. 7, 12182 (2016).



