<s> } else { V = LI . getVNInfoAt ( UseIdx ) ; } if ( ! V ) return nullptr ; DefIdx = V -> def ; } else { for ( MCRegUnitIterator Units ( Reg , this ) ; Units . isValid ( ) ; ++ Units ) { LiveRange & LR = LIS -> getRegUnit ( * Units ) ; if ( VNInfo * V = LR . getVNInfoAt ( UseIdx ) ) { if ( ! DefIdx . isValid ( ) || MDT . dominates ( LIS -> getInstructionFromIndex ( DefIdx ) , LIS -> getInstructionFromIndex ( V -> def ) ) ) DefIdx = V -> def ; } else { return nullptr ; } } } MachineInstr * Def = LIS -> getInstructionFromIndex ( DefIdx ) ; if ( ! Def || ! MDT . dominates ( Def , & Use ) ) return nullptr ;

assert ( Def -> modifiesRegister ( Reg , this ) ) ;


<s> if ( BitWidth <= <NUM_LIT> ) return & <STR_LIT> ; if ( BitWidth <= <NUM_LIT> ) return & <STR_LIT> ; if ( BitWidth <= <NUM_LIT> ) return & <STR_LIT> ; if ( BitWidth <= <NUM_LIT> ) return & <STR_LIT> ; if ( BitWidth <= <NUM_LIT> ) return & <STR_LIT> ; if ( BitWidth <= <NUM_LIT> ) return & <STR_LIT> ; if ( BitWidth <= <NUM_LIT> ) return & <STR_LIT> ;


<s> ArrayRef < MCPhysReg > <ISA_LIT>RegisterInfo :: getAllVGPR32 ( const MachineFunction & MF ) const {

return makeArrayRef ( <STR_LIT> . begin ( ) , ST . getMaxNumVGPRs ( MF ) ) ;


<s> & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , } ; for ( const TargetRegisterClass * BaseClass : BaseClasses ) { if ( BaseClass -> contains ( Reg ) ) { return BaseClass ; } } return nullptr ;


<s> const TargetRegisterClass * <ISA_LIT>RegisterInfo :: getSGPRClassForBitWidth ( unsigned BitWidth ) { if ( BitWidth <= <NUM_LIT> ) return & <STR_LIT> ; if ( BitWidth <= <NUM_LIT> ) return & <STR_LIT> ; if ( BitWidth <= <NUM_LIT> ) return & <STR_LIT> ; if ( BitWidth <= <NUM_LIT> ) return & <STR_LIT> ; if ( BitWidth <= <NUM_LIT> ) return & <STR_LIT> ; if ( BitWidth <= <NUM_LIT> ) return & <STR_LIT> ;


<s> if ( BitWidth <= <NUM_LIT> ) return & <STR_LIT> ; if ( BitWidth <= <NUM_LIT> ) return & <STR_LIT> ; if ( BitWidth <= <NUM_LIT> ) return & <STR_LIT> ; if ( BitWidth <= <NUM_LIT> ) return & <STR_LIT> ; if ( BitWidth <= <NUM_LIT> ) return & <STR_LIT> ; return nullptr ;


<s> if ( BitWidth <= <NUM_LIT> ) return & <STR_LIT> ; if ( BitWidth <= <NUM_LIT> ) return & <STR_LIT> ; if ( BitWidth <= <NUM_LIT> ) return & <STR_LIT> ; if ( BitWidth <= <NUM_LIT> ) return & <STR_LIT> ;


<s> const TargetRegisterClass * ARC = getAGPRClassForBitWidth ( Size ) ; if ( ! ARC ) { assert ( getVGPRClassForBitWidth ( Size ) && <STR_LIT> ) ; return false ; }

return getCommonSubClass ( ARC , RC ) != nullptr ;


<s> const MachineFrameInfo & MFI = MF . getFrameInfo ( ) ; return MFI . getNumFixedObjects ( ) && needsStackRealignment ( MF ) ;


<s> if ( Size == <NUM_LIT> ) { return getCommonSubClass ( & <STR_LIT> , RC ) != nullptr || getCommonSubClass ( & <STR_LIT> , RC ) != nullptr ; } const TargetRegisterClass * VRC = getVGPRClassForBitWidth ( Size ) ; if ( ! VRC ) { assert ( Size < <NUM_LIT> && <STR_LIT> ) ; return false ; } return getCommonSubClass ( VRC , RC ) != nullptr ;


<s> bool <ISA_LIT>RegisterInfo :: isAGPR ( const MachineRegisterInfo & MRI , Register Reg ) const { const TargetRegisterClass * RC = getRegClassForReg ( MRI , Reg ) ;


<s> if ( ! <ISA_LIT>InstrInfo :: isMUBUF ( * MI ) ) return false ; int64_t NewOffset = Offset + getMUBUFInstrOffset ( MI ) ;

return isUInt < <NUM_LIT> > ( NewOffset ) ;


<s> const TargetRegisterClass * RC = getRegClassForReg ( MRI , Reg ) ; return RC && hasVGPRs ( RC ) ;


<s> void <ISA_LIT>RegisterInfo :: materializeFrameBaseRegister ( MachineBasicBlock * MBB , Register BaseReg , int FrameIdx , int64_t Offset ) const { MachineBasicBlock :: iterator Ins = MBB -> begin ( ) ; DebugLoc DL ; if ( Ins != MBB -> end ( ) ) DL = Ins -> getDebugLoc ( ) ; MachineFunction * MF = MBB -> getParent ( ) ; const <ISA_LIT>InstrInfo * TII = ST . getInstrInfo ( ) ; if ( Offset == <NUM_LIT> ) { BuildMI ( * MBB , Ins , DL , TII -> get ( <STR_LIT> ) , BaseReg ) . addFrameIndex ( FrameIdx ) ; return ; } MachineRegisterInfo & MRI = MF -> getRegInfo ( ) ;

Register OffsetReg = MRI . createVirtualRegister ( & <STR_LIT> ) ;


<s> assert ( FIOp && FIOp -> isFI ( ) && <STR_LIT> ) ; assert ( TII -> isMUBUF ( MI ) ) ; assert ( TII -> getNamedOperand ( MI , <STR_LIT> :: soffset ) -> getReg ( ) == MF -> getInfo < <ISA_LIT>MachineFunctionInfo > ( ) -> getStackPtrOffsetReg ( ) && <STR_LIT> ) ; MachineOperand * OffsetOp = TII -> getNamedOperand ( MI , <STR_LIT> :: offset ) ; int64_t NewOffset = OffsetOp -> getImm ( ) + Offset ; assert ( isUInt < <NUM_LIT> > ( NewOffset ) && <STR_LIT> ) ;

FIOp -> ChangeToRegister ( BaseReg , false ) ;


<s> if ( SpillToVGPR ) { for ( unsigned i = <NUM_LIT> , e = NumSubRegs ; i < e ; ++ i ) { Register SubReg = NumSubRegs == <NUM_LIT> ? SuperReg : getSubReg ( SuperReg , SplitParts [ i ] ) ; <ISA_LIT>MachineFunctionInfo :: SpilledReg Spill = VGPRSpills [ i ] ; auto MIB = BuildMI ( * MBB , MI , DL , TII -> getMCOpcodeFromPseudo ( <STR_LIT> ) , SubReg ) . addReg ( Spill . VGPR ) . addImm ( Spill . Lane ) ; if ( NumSubRegs > <NUM_LIT> && i == <NUM_LIT> ) MIB . addReg ( SuperReg , RegState :: ImplicitDefine ) ; } } else { Register TmpVGPR = RS -> scavengeRegister ( & <STR_LIT> , MI , <NUM_LIT> ) ; RS -> setRegUsed ( TmpVGPR ) ; unsigned PerVGPR = <NUM_LIT> ; unsigned NumVGPRs = ( NumSubRegs + ( PerVGPR - <NUM_LIT> ) ) / PerVGPR ; int64_t VGPRLanes = ( <NUM_LIT> << std :: min ( PerVGPR , NumSubRegs ) ) - <NUM_LIT> ; for ( unsigned Offset = <NUM_LIT> ; Offset < NumVGPRs ; ++ Offset ) { buildSGPRSpillLoadStore ( MI , Index , Offset , EltSize , TmpVGPR , VGPRLanes , RS , true ) ; for ( unsigned i = Offset * PerVGPR , e = std :: min ( ( Offset + <NUM_LIT> ) * PerVGPR , NumSubRegs ) ; i < e ; ++ i ) { Register SubReg = NumSubRegs == <NUM_LIT> ? SuperReg : getSubReg ( SuperReg , SplitParts [ i ] ) ; bool LastSubReg = ( i + <NUM_LIT> == e ) ; auto MIB = BuildMI ( * MBB , MI , DL , TII -> getMCOpcodeFromPseudo ( <STR_LIT> ) , SubReg ) . addReg ( TmpVGPR , getKillRegState ( LastSubReg ) ) . addImm ( i ) ; if ( NumSubRegs > <NUM_LIT> && i == <NUM_LIT> ) MIB . addReg ( SuperReg , RegState :: ImplicitDefine ) ; } }


<s> bool IsKill = MI -> getOperand ( <NUM_LIT> ) . isKill ( ) ; const DebugLoc & DL = MI -> getDebugLoc ( ) ; assert ( SpillToVGPR || ( SuperReg != MFI -> getStackPtrOffsetReg ( ) && SuperReg != MFI -> getFrameOffsetReg ( ) ) ) ; assert ( SuperReg != <STR_LIT> && <STR_LIT> ) ; assert ( SuperReg != <STR_LIT> && SuperReg != <STR_LIT> && SuperReg != <STR_LIT> && <STR_LIT> ) ; unsigned EltSize = <NUM_LIT> ; const TargetRegisterClass * RC = getPhysRegClass ( SuperReg ) ; ArrayRef < int16_t > SplitParts = getRegSplitParts ( RC , EltSize ) ; unsigned NumSubRegs = SplitParts . empty ( ) ? <NUM_LIT> : SplitParts . size ( ) ; if ( SpillToVGPR ) { for ( unsigned i = <NUM_LIT> , e = NumSubRegs ; i < e ; ++ i ) { Register SubReg = NumSubRegs == <NUM_LIT> ? SuperReg : getSubReg ( SuperReg , SplitParts [ i ] ) ; <ISA_LIT>MachineFunctionInfo :: SpilledReg Spill = VGPRSpills [ i ] ; bool VGPRDefined = true ; if ( MBB == & MF -> front ( ) ) VGPRDefined = ! SGPRSpillVGPRDefinedSet . insert ( Spill . VGPR ) . second ; BuildMI ( * MBB , MI , DL , TII -> getMCOpcodeFromPseudo ( <STR_LIT> ) , Spill . VGPR ) . addReg ( SubReg , getKillRegState ( IsKill ) ) . addImm ( Spill . Lane ) . addReg ( Spill . VGPR , VGPRDefined ? <NUM_LIT> : RegState :: Undef ) ; } } else { Register TmpVGPR = RS -> scavengeRegister ( & <STR_LIT> , MI , <NUM_LIT> ) ; RS -> setRegUsed ( TmpVGPR ) ; unsigned SubKillState = getKillRegState ( ( NumSubRegs == <NUM_LIT> ) && IsKill ) ; unsigned PerVGPR = <NUM_LIT> ; unsigned NumVGPRs = ( NumSubRegs + ( PerVGPR - <NUM_LIT> ) ) / PerVGPR ; int64_t VGPRLanes = ( <NUM_LIT> << std :: min ( PerVGPR , NumSubRegs ) ) - <NUM_LIT> ;

for ( unsigned Offset = <NUM_LIT> ; Offset < NumVGPRs ; ++ Offset ) {


<s> MachineFunction * MF = MBB -> getParent ( ) ; <ISA_LIT>MachineFunctionInfo * MFI = MF -> getInfo < <ISA_LIT>MachineFunctionInfo > ( ) ; const <ISA_LIT>InstrInfo * TII = ST . getInstrInfo ( ) ; Register SuperReg = MI -> getOperand ( <NUM_LIT> ) . getReg ( ) ; const TargetRegisterClass * RC = getPhysRegClass ( SuperReg ) ; ArrayRef < int16_t > SplitParts = getRegSplitParts ( RC , EltSize ) ; unsigned NumSubRegs = SplitParts . empty ( ) ? <NUM_LIT> : SplitParts . size ( ) ; unsigned FirstPart = Offset * <NUM_LIT> ; unsigned ExecLane = <NUM_LIT> ; bool IsKill = MI -> getOperand ( <NUM_LIT> ) . isKill ( ) ; const DebugLoc & DL = MI -> getDebugLoc ( ) ; assert ( SuperReg != <STR_LIT> && SuperReg != <STR_LIT> && SuperReg != <STR_LIT> && <STR_LIT> ) ; bool OnlyExecLo = isWave32 || NumSubRegs == <NUM_LIT> || SuperReg == <STR_LIT> ; unsigned ExecMovOpc = OnlyExecLo ? <STR_LIT> : <STR_LIT> ; Register ExecReg = OnlyExecLo ? <STR_LIT> : <STR_LIT> ; Register SavedExecReg ; if ( OnlyExecLo ) { SavedExecReg = NumSubRegs == <NUM_LIT> ? SuperReg : Register ( getSubReg ( SuperReg , SplitParts [ FirstPart + ExecLane ] ) ) ; } else { for ( ; ExecLane < ( NumSubRegs - <NUM_LIT> ) ; ++ ExecLane ) { SavedExecReg = getMatchingSuperReg ( getSubReg ( SuperReg , SplitParts [ FirstPart + ExecLane ] ) , <STR_LIT> , & <STR_LIT> ) ; if ( SavedExecReg ) break ; }


<s> switch ( CC ) { case CallingConv :: C : case CallingConv :: Fast : case CallingConv :: Cold : case CallingConv :: <ISA_LIT>_Gfx : return CSR_<ISA_LIT>_HighRegs_SaveList ;


<s> case <STR_LIT> : return <NUM_LIT> ; case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : return <NUM_LIT> ; case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : return <NUM_LIT> ; case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : return <NUM_LIT> ; case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : return <NUM_LIT> ; case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : return <NUM_LIT> ; case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : return <NUM_LIT> ; case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : return <NUM_LIT> ; case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : return <NUM_LIT> ; default : llvm_unreachable ( <STR_LIT> ) ; }


<s> reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; if ( isWave32 ) { Reserved . set ( <STR_LIT> ) ; Reserved . set ( <STR_LIT> ) ; } unsigned MaxNumSGPRs = ST . getMaxNumSGPRs ( MF ) ; unsigned TotalNumSGPRs = <STR_LIT> . getNumRegs ( ) ; for ( unsigned i = MaxNumSGPRs ; i < TotalNumSGPRs ; ++ i ) { unsigned Reg = <STR_LIT> . getRegister ( i ) ; reserveRegisterTuples ( Reserved , Reg ) ; } unsigned MaxNumVGPRs = ST . getMaxNumVGPRs ( MF ) ; unsigned TotalNumVGPRs = <STR_LIT> . getNumRegs ( ) ; for ( unsigned i = MaxNumVGPRs ; i < TotalNumVGPRs ; ++ i ) { unsigned Reg = <STR_LIT> . getRegister ( i ) ; reserveRegisterTuples ( Reserved , Reg ) ; Reg = <STR_LIT> . getRegister ( i ) ; reserveRegisterTuples ( Reserved , Reg ) ; } for ( auto Reg : <STR_LIT> ) { Reserved . set ( getSubReg ( Reg , <STR_LIT> ) ) ; Register Low = getSubReg ( Reg , <STR_LIT> ) ; if ( ! <STR_LIT> . contains ( Low ) ) Reserved . set ( Low ) ; } for ( auto Reg : <STR_LIT> ) { Reserved . set ( getSubReg ( Reg , <STR_LIT> ) ) ; } if ( ! ST . hasMAIInsts ( ) ) { for ( unsigned i = <NUM_LIT> ; i < MaxNumVGPRs ; ++ i ) { unsigned Reg = <STR_LIT> . getRegister ( i ) ; reserveRegisterTuples ( Reserved , Reg ) ; } } const <ISA_LIT>MachineFunctionInfo * MFI = MF . getInfo < <ISA_LIT>MachineFunctionInfo > ( ) ; Register ScratchRSrcReg = MFI -> getScratchRSrcReg ( ) ; if ( ScratchRSrcReg != <STR_LIT> ) { reserveRegisterTuples ( Reserved , ScratchRSrcReg ) ; } MCRegister StackPtrReg = MFI -> getStackPtrOffsetReg ( ) ; if ( StackPtrReg ) { reserveRegisterTuples ( Reserved , StackPtrReg ) ; assert ( ! isSubRegister ( ScratchRSrcReg , StackPtrReg ) ) ;


<s> bool <ISA_LIT>RegisterInfo :: isFrameOffsetLegal ( const MachineInstr * MI , Register BaseReg , int64_t Offset ) const {

if ( ! <ISA_LIT>InstrInfo :: isMUBUF ( * MI ) && ! ! <ISA_LIT>InstrInfo :: isFLATScratch ( * MI ) ) return false ;


<s> int64_t FullOffset = Offset + getScratchInstrOffset ( MI ) ; if ( <ISA_LIT>InstrInfo :: isMUBUF ( * MI ) ) return ! <ISA_LIT>InstrInfo :: isLegalMUBUFImmOffset ( FullOffset ) ; const <ISA_LIT>InstrInfo * TII = ST . getInstrInfo ( ) ; return TII -> isLegalFLATOffset ( FullOffset , <ISA_LIT>AS :: PRIVATE_ADDRESS , true ) ;


<s> for ( const MachineOperand & MO : MI . operands ( ) ) { if ( MO . isFI ( ) ) { if ( SeenFI ) llvm_unreachable ( <STR_LIT> ) ; SeenFI = true ; } } MachineOperand * FIOp = TII -> getNamedOperand ( MI , IsFlat ? <STR_LIT> :: saddr : <STR_LIT> :: vaddr ) ; MachineOperand * OffsetOp = TII -> getNamedOperand ( MI , <STR_LIT> :: offset ) ; int64_t NewOffset = OffsetOp -> getImm ( ) + Offset ; MachineBasicBlock * MBB = MI . getParent ( ) ; MachineFunction * MF = MBB -> getParent ( ) ; assert ( FIOp && FIOp -> isFI ( ) && <STR_LIT> ) ; assert ( TII -> isMUBUF ( MI ) || TII -> isFLATScratch ( MI ) ) ; if ( IsFlat ) { assert ( TII -> isLegalFLATOffset ( NewOffset , <ISA_LIT>AS :: PRIVATE_ADDRESS , true ) && <STR_LIT> ) ; FIOp -> ChangeToRegister ( BaseReg , false ) ; OffsetOp -> setImm ( NewOffset ) ; return ;


<s> Register SubReg = NumSubRegs == <NUM_LIT> ? SuperReg : Register ( getSubReg ( SuperReg , SplitParts [ i ] ) ) ; <ISA_LIT>MachineFunctionInfo :: SpilledReg Spill = VGPRSpills [ i ] ; auto MIB = BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) , SubReg ) . addReg ( Spill . VGPR ) . addImm ( Spill . Lane ) ; if ( NumSubRegs > <NUM_LIT> && i == <NUM_LIT> ) MIB . addReg ( SuperReg , RegState :: ImplicitDefine ) ; } } else { Register TmpVGPR = RS -> scavengeRegister ( & <STR_LIT> , MI , <NUM_LIT> ) ; RS -> setRegUsed ( TmpVGPR ) ; unsigned PerVGPR = <NUM_LIT> ; unsigned NumVGPRs = ( NumSubRegs + ( PerVGPR - <NUM_LIT> ) ) / PerVGPR ; int64_t VGPRLanes = ( <NUM_LIT> << std :: min ( PerVGPR , NumSubRegs ) ) - <NUM_LIT> ; for ( unsigned Offset = <NUM_LIT> ; Offset < NumVGPRs ; ++ Offset ) { buildSGPRSpillLoadStore ( MI , Index , Offset , EltSize , TmpVGPR , VGPRLanes , RS , true ) ; for ( unsigned i = Offset * PerVGPR , e = std :: min ( ( Offset + <NUM_LIT> ) * PerVGPR , NumSubRegs ) ; i < e ; ++ i ) { Register SubReg = NumSubRegs == <NUM_LIT> ? SuperReg : Register ( getSubReg ( SuperReg , SplitParts [ i ] ) ) ;

bool LastSubReg = ( i + <NUM_LIT> == e ) ;


<s> } else { Register TmpVGPR = RS -> scavengeRegister ( & <STR_LIT> , MI , <NUM_LIT> ) ; RS -> setRegUsed ( TmpVGPR ) ; unsigned PerVGPR = <NUM_LIT> ; unsigned NumVGPRs = ( NumSubRegs + ( PerVGPR - <NUM_LIT> ) ) / PerVGPR ; int64_t VGPRLanes = ( <NUM_LIT> << std :: min ( PerVGPR , NumSubRegs ) ) - <NUM_LIT> ; for ( unsigned Offset = <NUM_LIT> ; Offset < NumVGPRs ; ++ Offset ) { buildSGPRSpillLoadStore ( MI , Index , Offset , EltSize , TmpVGPR , VGPRLanes , RS , true ) ; for ( unsigned i = Offset * PerVGPR , e = std :: min ( ( Offset + <NUM_LIT> ) * PerVGPR , NumSubRegs ) ; i < e ; ++ i ) { Register SubReg = NumSubRegs == <NUM_LIT> ? SuperReg : Register ( getSubReg ( SuperReg , SplitParts [ i ] ) ) ; bool LastSubReg = ( i + <NUM_LIT> == e ) ; auto MIB = BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) , SubReg ) . addReg ( TmpVGPR , getKillRegState ( LastSubReg ) ) . addImm ( i ) ;

if ( NumSubRegs > <NUM_LIT> && i == <NUM_LIT> ) MIB . addReg ( SuperReg , RegState :: ImplicitDefine ) ;


<s> } if ( NumSubRegs > <NUM_LIT> ) MIB . addReg ( SuperReg , getKillRegState ( UseKill ) | RegState :: Implicit ) ; } } else { Register TmpVGPR = RS -> scavengeRegister ( & <STR_LIT> , MI , <NUM_LIT> ) ; RS -> setRegUsed ( TmpVGPR ) ; unsigned SubKillState = getKillRegState ( ( NumSubRegs == <NUM_LIT> ) && IsKill ) ; unsigned PerVGPR = <NUM_LIT> ; unsigned NumVGPRs = ( NumSubRegs + ( PerVGPR - <NUM_LIT> ) ) / PerVGPR ; int64_t VGPRLanes = ( <NUM_LIT> << std :: min ( PerVGPR , NumSubRegs ) ) - <NUM_LIT> ; for ( unsigned Offset = <NUM_LIT> ; Offset < NumVGPRs ; ++ Offset ) { unsigned TmpVGPRFlags = RegState :: Undef ; for ( unsigned i = Offset * PerVGPR , e = std :: min ( ( Offset + <NUM_LIT> ) * PerVGPR , NumSubRegs ) ; i < e ; ++ i ) { Register SubReg = NumSubRegs == <NUM_LIT> ? SuperReg : Register ( getSubReg ( SuperReg , SplitParts [ i ] ) ) ; MachineInstrBuilder WriteLane = BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) , TmpVGPR ) . addReg ( SubReg , SubKillState ) . addImm ( i % PerVGPR ) . addReg ( TmpVGPR , TmpVGPRFlags ) ; TmpVGPRFlags = <NUM_LIT> ; if ( NumSubRegs > <NUM_LIT> ) { unsigned SuperKillState = <NUM_LIT> ; if ( i + <NUM_LIT> == NumSubRegs ) SuperKillState |= getKillRegState ( IsKill ) ; WriteLane . addReg ( SuperReg , RegState :: Implicit | SuperKillState ) ; } } buildSGPRSpillLoadStore ( MI , Index , Offset , EltSize , TmpVGPR , VGPRLanes , RS , false ) ; } }


<s> unsigned Dst = IsStore ? Reg : ValueReg ; unsigned Src = IsStore ? ValueReg : Reg ; unsigned Opc = ( IsStore ^ TRI -> isVGPR ( MRI , Reg ) ) ? <STR_LIT> : <STR_LIT> ; auto MIB = BuildMI ( * MBB , MI , MI -> getDebugLoc ( ) , TII -> get ( Opc ) , Dst ) . addReg ( Src , getKillRegState ( IsKill ) ) ; MIB -> setAsmPrinterFlag ( MachineInstr :: ReloadReuse ) ;


<s> auto * TRI = static_cast < const <ISA_LIT>RegisterInfo * > ( MRI . getTargetRegisterInfo ( ) ) ; unsigned Dst = IsStore ? Reg : ValueReg ; unsigned Src = IsStore ? ValueReg : Reg ; unsigned Opc = ( IsStore ^ TRI -> isVGPR ( MRI , Reg ) ) ? <STR_LIT> : <STR_LIT> ; auto MIB = BuildMI ( * MBB , MI , MI -> getDebugLoc ( ) , TII -> get ( Opc ) , Dst ) . addReg ( Src , getKillRegState ( IsKill ) ) ;

MIB -> setAsmPrinterFlag ( MachineInstr :: ReloadReuse ) ;


<s> } case <NUM_LIT> : return RB . getID ( ) == <STR_LIT> ? & <STR_LIT> : & <STR_LIT> ; case <NUM_LIT> : return RB . getID ( ) == <STR_LIT> ? & <STR_LIT> : & <STR_LIT> ; case <NUM_LIT> : return RB . getID ( ) == <STR_LIT> ? & <STR_LIT> : & <STR_LIT> ; case <NUM_LIT> : return RB . getID ( ) == <STR_LIT> ? & <STR_LIT> : & <STR_LIT> ; case <NUM_LIT> : return RB . getID ( ) == <STR_LIT> ? & <STR_LIT> : & <STR_LIT> ; case <NUM_LIT> : return RB . getID ( ) == <STR_LIT> ? & <STR_LIT> : & <STR_LIT> ; case <NUM_LIT> : return RB . getID ( ) == <STR_LIT> ? & <STR_LIT> : & <STR_LIT> ;


<s> case <STR_LIT> : return std :: min ( ST . getMaxNumVGPRs ( Occupancy ) , ST . getMaxNumVGPRs ( MF ) ) ; case <STR_LIT> :

return std :: min ( ST . getMaxNumSGPRs ( Occupancy , true ) , ST . getMaxNumSGPRs ( MF ) ) ;


<s> if ( Idx == getVGPRPressureSet ( ) || Idx == getAGPRPressureSet ( ) ) return getRegPressureLimit ( & <STR_LIT> , const_cast < MachineFunction & > ( MF ) ) ; if ( Idx == getSGPRPressureSet ( ) ) return getRegPressureLimit ( & <STR_LIT> , const_cast < MachineFunction & > ( MF ) ) ;

return <ISA_LIT>GenRegisterInfo :: getRegPressureSetLimit ( MF , Idx ) ;


<s> if ( Idx == getSGPRPressureSet ( ) ) return getRegPressureLimit ( & <STR_LIT> , const_cast < MachineFunction & > ( MF ) ) ;


<s> static const int Empty [ ] = { - <NUM_LIT> } ; if ( hasRegUnit ( <STR_LIT> , RegUnit ) ) return Empty ;

return <ISA_LIT>GenRegisterInfo :: getRegUnitPressureSets ( RegUnit ) ;


<s> const int * <ISA_LIT>RegisterInfo :: getRegUnitPressureSets ( unsigned RegUnit ) const { static const int Empty [ ] = { - <NUM_LIT> } ; if ( hasRegUnit ( <STR_LIT> , RegUnit ) ) return Empty ;


<s> case <NUM_LIT> : return & <STR_LIT> ; case <NUM_LIT> : return & <STR_LIT> ; case <NUM_LIT> : default : llvm_unreachable ( <STR_LIT> ) ; } } else if ( hasAGPRs ( RC ) ) { switch ( Count ) { case <NUM_LIT> : return & <STR_LIT> ; case <NUM_LIT> : return & <STR_LIT> ; case <NUM_LIT> : return & <STR_LIT> ; case <NUM_LIT> : return & <STR_LIT> ; case <NUM_LIT> : default : llvm_unreachable ( <STR_LIT> ) ; }


<s> for ( unsigned i = <NUM_LIT> ; i < NumRegPressureSets ; ++ i ) { classifyPressureSet ( i , <STR_LIT> , SGPRPressureSets ) ; classifyPressureSet ( i , <STR_LIT> , VGPRPressureSets ) ; classifyPressureSet ( i , <STR_LIT> , AGPRPressureSets ) ; } std :: vector < unsigned > PressureSetRegUnits ( NumRegPressureSets , <NUM_LIT> ) ; for ( unsigned i = <NUM_LIT> , e = getNumRegUnits ( ) ; i != e ; ++ i ) { const int * PSets = getRegUnitPressureSets ( i ) ; for ( unsigned j = <NUM_LIT> ; PSets [ j ] != - <NUM_LIT> ; ++ j ) { ++ PressureSetRegUnits [ PSets [ j ] ] ; } } unsigned VGPRMax = <NUM_LIT> , SGPRMax = <NUM_LIT> , AGPRMax = <NUM_LIT> ;


<s> case <NUM_LIT> : { switch ( RB . getID ( ) ) { case <STR_LIT> : return & <STR_LIT> ; case <STR_LIT> : return isWave32 ? & <STR_LIT> : & <STR_LIT> ; case <STR_LIT> : return & <STR_LIT> ; case <STR_LIT> : return & <STR_LIT> ; default : llvm_unreachable ( <STR_LIT> ) ; } } case <NUM_LIT> : return RB . getID ( ) == <STR_LIT> ? & <STR_LIT> : & <STR_LIT> ; case <NUM_LIT> : return RB . getID ( ) == <STR_LIT> ? & <STR_LIT> : & <STR_LIT> ; case <NUM_LIT> : return RB . getID ( ) == <STR_LIT> ? & <STR_LIT> : & <STR_LIT> ; case <NUM_LIT> : return RB . getID ( ) == <STR_LIT> ? & <STR_LIT> : & <STR_LIT> ; case <NUM_LIT> : return RB . getID ( ) == <STR_LIT> ? & <STR_LIT> : & <STR_LIT> ; case <NUM_LIT> : return RB . getID ( ) == <STR_LIT> ? & <STR_LIT> : & <STR_LIT> ; case <NUM_LIT> : return RB . getID ( ) == <STR_LIT> ? & <STR_LIT> :


<s> static bool buildMUBUFOffsetLoadStore ( const <ISA_LIT>InstrInfo * TII , MachineFrameInfo & MFI , MachineBasicBlock :: iterator MI , int Index , int64_t Offset ) { MachineBasicBlock * MBB = MI -> getParent ( ) ; const DebugLoc & DL = MI -> getDebugLoc ( ) ; bool IsStore = MI -> mayStore ( ) ; unsigned Opc = MI -> getOpcode ( ) ; int LoadStoreOp = IsStore ? getOffsetMUBUFStore ( Opc ) : getOffsetMUBUFLoad ( Opc ) ; if ( LoadStoreOp == - <NUM_LIT> ) return false ; const MachineOperand * Reg = TII -> getNamedOperand ( * MI , <STR_LIT> :: vdata ) ;

MachineInstrBuilder NewMI = BuildMI ( * MBB , MI , DL , TII -> get ( LoadStoreOp ) ) . add ( * Reg ) . add ( * TII -> getNamedOperand ( * MI , <STR_LIT> :: srsrc ) ) . add ( * TII -> getNamedOperand ( * MI , <STR_LIT> :: soffset ) ) . addImm ( Offset ) . addImm ( <NUM_LIT> ) . addImm ( <NUM_LIT> ) . addImm ( <NUM_LIT> ) . cloneMemRefs ( * MI ) ;


<s> reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; const GCNSubtarget & ST = MF . getSubtarget < GCNSubtarget > ( ) ; unsigned MaxNumSGPRs = ST . getMaxNumSGPRs ( MF ) ; unsigned TotalNumSGPRs = <STR_LIT> . getNumRegs ( ) ; for ( unsigned i = MaxNumSGPRs ; i < TotalNumSGPRs ; ++ i ) { unsigned Reg = <STR_LIT> . getRegister ( i ) ; reserveRegisterTuples ( Reserved , Reg ) ; } unsigned MaxNumVGPRs = ST . getMaxNumVGPRs ( MF ) ; unsigned TotalNumVGPRs = <STR_LIT> . getNumRegs ( ) ; for ( unsigned i = MaxNumVGPRs ; i < TotalNumVGPRs ; ++ i ) { unsigned Reg = <STR_LIT> . getRegister ( i ) ; reserveRegisterTuples ( Reserved , Reg ) ; } const <ISA_LIT>MachineFunctionInfo * MFI = MF . getInfo < <ISA_LIT>MachineFunctionInfo > ( ) ;

unsigned ScratchWaveOffsetReg = MFI -> getScratchWaveOffsetReg ( ) ;


<s> struct <ISA_LIT>MachineFunctionInfo :: SpilledReg Spill = MFI -> getSpilledReg ( MF , Index , i ) ; if ( Spill . hasReg ( ) ) { BuildMI ( * MBB , MI , DL , TII -> getMCOpcodeFromPseudo ( <STR_LIT> ) , Spill . VGPR ) . addReg ( SubReg , getKillRegState ( IsKill ) ) . addImm ( Spill . Lane ) ; } else { MachineInstrBuilder Mov = BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) , TmpReg ) . addReg ( SubReg , SubKillState ) ; if ( NumSubRegs > <NUM_LIT> ) { unsigned SuperKillState = <NUM_LIT> ; if ( i + <NUM_LIT> == e ) SuperKillState |= getKillRegState ( IsKill ) ; Mov . addReg ( SuperReg , RegState :: Implicit | SuperKillState ) ; } unsigned Size = FrameInfo -> getObjectSize ( Index ) ; unsigned Align = FrameInfo -> getObjectAlignment ( Index ) ; MachinePointerInfo PtrInfo = MachinePointerInfo :: getFixedStack ( * MF , Index ) ; MachineMemOperand * MMO = MF -> getMachineMemOperand ( PtrInfo , MachineMemOperand :: MOStore , Size , Align ) ; BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) ) . addReg ( TmpReg , RegState :: Kill ) . addFrameIndex ( Index ) . addReg ( MFI -> getScratchRSrcReg ( ) ) . addReg ( MFI -> getScratchWaveOffsetReg ( ) ) . addImm ( i * <NUM_LIT> ) . addMemOperand ( MMO ) ; } } MI -> eraseFromParent ( ) ; MFI -> addToSpilledSGPRs ( NumSubRegs ) ; break ; } case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : { unsigned NumSubRegs = getNumSubRegsForSpillOp ( MI -> getOpcode ( ) ) ; unsigned TmpReg = MRI . createVirtualRegister ( & <STR_LIT> ) ; for ( unsigned i = <NUM_LIT> , e = NumSubRegs ; i < e ; ++ i ) { unsigned SubReg = getPhysRegSubReg ( MI -> getOperand ( <NUM_LIT> ) . getReg ( ) , & <STR_LIT> , i ) ; struct <ISA_LIT>MachineFunctionInfo :: SpilledReg Spill = MFI -> getSpilledReg ( MF , Index , i ) ; if ( Spill . hasReg ( ) ) { BuildMI ( * MBB , MI , DL , TII -> getMCOpcodeFromPseudo ( <STR_LIT> ) , SubReg ) . addReg ( Spill . VGPR ) . addImm ( Spill . Lane ) . addReg ( MI -> getOperand ( <NUM_LIT> ) . getReg ( ) , RegState :: ImplicitDefine ) ; } else {


<s> DebugLoc DL ; if ( Ins != MBB -> end ( ) ) DL = Ins -> getDebugLoc ( ) ; MachineFunction * MF = MBB -> getParent ( ) ; const <ISA_LIT>Subtarget & Subtarget = MF -> getSubtarget < <ISA_LIT>Subtarget > ( ) ; const <ISA_LIT>InstrInfo * TII = Subtarget . getInstrInfo ( ) ; if ( Offset == <NUM_LIT> ) { BuildMI ( * MBB , Ins , DL , TII -> get ( <STR_LIT> ) , BaseReg ) . addFrameIndex ( FrameIdx ) ; return ; } MachineRegisterInfo & MRI = MF -> getRegInfo ( ) ; unsigned UnusedCarry = MRI . createVirtualRegister ( & <STR_LIT> ) ;

unsigned OffsetReg = MRI . createVirtualRegister ( & <STR_LIT> ) ;


<s> assert ( FIOp && FIOp -> isFI ( ) && <STR_LIT> ) ; assert ( TII -> isMUBUF ( MI ) ) ; MachineOperand * OffsetOp = TII -> getNamedOperand ( MI , <STR_LIT> :: offset ) ; int64_t NewOffset = OffsetOp -> getImm ( ) + Offset ; if ( isUInt < <NUM_LIT> > ( NewOffset ) ) { FIOp -> ChangeToRegister ( BaseReg , false ) ; OffsetOp -> setImm ( NewOffset ) ; return ; } MachineRegisterInfo & MRI = MF -> getRegInfo ( ) ; unsigned NewReg = MRI . createVirtualRegister ( & <STR_LIT> ) ; DebugLoc DL = MI . getDebugLoc ( ) ; assert ( Offset != <NUM_LIT> && <STR_LIT> ) ; unsigned UnusedCarry = MRI . createVirtualRegister ( & <STR_LIT> ) ; unsigned OffsetReg = MRI . createVirtualRegister ( & <STR_LIT> ) ; BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) , OffsetReg ) . addImm ( Offset ) ;

BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) , NewReg ) . addReg ( UnusedCarry , RegState :: Define | RegState :: Dead ) . addReg ( OffsetReg , RegState :: Kill ) . addReg ( BaseReg ) ;


<s> const <ISA_LIT>Subtarget & Subtarget = MF -> getSubtarget < <ISA_LIT>Subtarget > ( ) ; const <ISA_LIT>InstrInfo * TII = Subtarget . getInstrInfo ( ) ; bool SeenFI = false ; for ( const MachineOperand & MO : MI . operands ( ) ) { if ( MO . isFI ( ) ) { if ( SeenFI ) llvm_unreachable ( <STR_LIT> ) ; SeenFI = true ; } } MachineOperand * FIOp = TII -> getNamedOperand ( MI , <STR_LIT> :: vaddr ) ; assert ( FIOp && FIOp -> isFI ( ) && <STR_LIT> ) ; assert ( TII -> isMUBUF ( MI ) ) ;

MachineOperand * OffsetOp = TII -> getNamedOperand ( MI , <STR_LIT> :: offset ) ;


<s> MachineOperand & FIOp = MI -> getOperand ( FIOperandNum ) ; int Index = MI -> getOperand ( FIOperandNum ) . getIndex ( ) ; switch ( MI -> getOpcode ( ) ) { case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : { spillSGPR ( MI , Index , RS ) ; break ; } case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : { restoreSGPR ( MI , Index , RS ) ; break ; } case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : { const MachineOperand * VData = TII -> getNamedOperand ( * MI , <STR_LIT> :: vdata ) ; buildSpillLoadStore ( MI , <STR_LIT> , Index , VData -> getReg ( ) , VData -> isKill ( ) , TII -> getNamedOperand ( * MI , <STR_LIT> :: srsrc ) -> getReg ( ) , TII -> getNamedOperand ( * MI , <STR_LIT> :: soffset ) -> getReg ( ) , TII -> getNamedOperand ( * MI , <STR_LIT> :: offset ) -> getImm ( ) , * MI -> memoperands_begin ( ) , RS ) ; MFI -> addToSpilledVGPRs ( getNumSubRegsForSpillOp ( MI -> getOpcode ( ) ) ) ; MI -> eraseFromParent ( ) ; break ; }


<s> BitVector Reserved ( getNumRegs ( ) ) ; Reserved . set ( <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; const <ISA_LIT>Subtarget & ST = MF . getSubtarget < <ISA_LIT>Subtarget > ( ) ; unsigned MaxNumSGPRs = ST . getMaxNumSGPRs ( MF ) ; unsigned TotalNumSGPRs = <STR_LIT> . getNumRegs ( ) ; for ( unsigned i = MaxNumSGPRs ; i < TotalNumSGPRs ; ++ i ) { unsigned Reg = <STR_LIT> . getRegister ( i ) ; reserveRegisterTuples ( Reserved , Reg ) ; } unsigned MaxNumVGPRs = ST . getMaxNumVGPRs ( MF ) ; unsigned TotalNumVGPRs = <STR_LIT> . getNumRegs ( ) ; for ( unsigned i = MaxNumVGPRs ; i < TotalNumVGPRs ; ++ i ) { unsigned Reg = <STR_LIT> . getRegister ( i ) ; reserveRegisterTuples ( Reserved , Reg ) ; } const <ISA_LIT>MachineFunctionInfo * MFI = MF . getInfo < <ISA_LIT>MachineFunctionInfo > ( ) ; unsigned ScratchWaveOffsetReg = MFI -> getScratchWaveOffsetReg ( ) ; if ( ScratchWaveOffsetReg != <STR_LIT> ) { reserveRegisterTuples ( Reserved , ScratchWaveOffsetReg ) ; } unsigned ScratchRSrcReg = MFI -> getScratchRSrcReg ( ) ; if ( ScratchRSrcReg != <STR_LIT> ) { reserveRegisterTuples ( Reserved , ScratchRSrcReg ) ; assert ( ! isSubRegister ( ScratchRSrcReg , ScratchWaveOffsetReg ) ) ; }


<s> MachineBasicBlock :: iterator Ins = MBB -> begin ( ) ; DebugLoc DL ; if ( Ins != MBB -> end ( ) ) DL = Ins -> getDebugLoc ( ) ; MachineFunction * MF = MBB -> getParent ( ) ; const <ISA_LIT>Subtarget & Subtarget = MF -> getSubtarget < <ISA_LIT>Subtarget > ( ) ; const <ISA_LIT>InstrInfo * TII = Subtarget . getInstrInfo ( ) ; if ( Offset == <NUM_LIT> ) { BuildMI ( * MBB , Ins , DL , TII -> get ( <STR_LIT> ) , BaseReg ) . addFrameIndex ( FrameIdx ) ; return ;


<s> bool <ISA_LIT>RegisterInfo :: requiresRegisterScavenging ( const MachineFunction & Fn ) const {

return Fn . getFrameInfo ( ) . hasStackObjects ( ) ;


<s> unsigned RegCount = ST . getMaxNumSGPRs ( MF ) ; unsigned Reg ; if ( RegCount & <NUM_LIT> ) { Reg = RegCount - <NUM_LIT> ; } else { Reg = RegCount - <NUM_LIT> ; } return <STR_LIT> . getRegister ( Reg ) ;


<s> MachineBasicBlock * MBB = MI . getParent ( ) ; MachineFunction * MF = MBB -> getParent ( ) ; const <ISA_LIT>Subtarget & Subtarget = MF -> getSubtarget < <ISA_LIT>Subtarget > ( ) ; const <ISA_LIT>InstrInfo * TII = Subtarget . getInstrInfo ( ) ; bool SeenFI = false ; for ( const MachineOperand & MO : MI . operands ( ) ) { if ( MO . isFI ( ) ) { if ( SeenFI ) llvm_unreachable ( <STR_LIT> ) ; SeenFI = true ; } } MachineOperand * FIOp = TII -> getNamedOperand ( MI , <STR_LIT> :: vaddr ) ;

assert ( FIOp && FIOp -> isFI ( ) && <STR_LIT> ) ;


<s> assert ( TII -> isMUBUF ( MI ) ) ; MachineOperand * OffsetOp = TII -> getNamedOperand ( MI , <STR_LIT> :: offset ) ; int64_t NewOffset = OffsetOp -> getImm ( ) + Offset ; assert ( isUInt < <NUM_LIT> > ( NewOffset ) && <STR_LIT> ) ; FIOp -> ChangeToRegister ( BaseReg , false ) ;


<s> const <ISA_LIT>Subtarget & ST = MF -> getSubtarget < <ISA_LIT>Subtarget > ( ) ; const <ISA_LIT>InstrInfo * TII = ST . getInstrInfo ( ) ; const DebugLoc & DL = MI -> getDebugLoc ( ) ; unsigned SuperReg = MI -> getOperand ( <NUM_LIT> ) . getReg ( ) ; bool SpillToSMEM = ST . hasScalarStores ( ) && EnableSpillSGPRToSMEM ; assert ( SuperReg != <STR_LIT> && <STR_LIT> ) ; unsigned OffsetReg = <STR_LIT> ; unsigned M0CopyReg = <STR_LIT> ; if ( SpillToSMEM ) { if ( RS -> isRegUsed ( <STR_LIT> ) ) { M0CopyReg = MRI . createVirtualRegister ( & <STR_LIT> ) ; BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) , M0CopyReg ) . addReg ( <STR_LIT> ) ; } } unsigned EltSize = <NUM_LIT> ; unsigned ScalarLoadOp ; const TargetRegisterClass * RC = getPhysRegClass ( SuperReg ) ; if ( SpillToSMEM && isSGPRClass ( RC ) ) { std :: tie ( EltSize , ScalarLoadOp ) = getSpillEltSize ( RC -> getSize ( ) , false ) ; } ArrayRef < int16_t > SplitParts = getRegSplitParts ( RC , EltSize ) ; unsigned NumSubRegs = SplitParts . empty ( ) ? <NUM_LIT> : SplitParts . size ( ) ; int64_t FrOffset = FrameInfo . getObjectOffset ( Index ) ; for ( unsigned i = <NUM_LIT> , e = NumSubRegs ; i < e ; ++ i ) { unsigned SubReg = NumSubRegs == <NUM_LIT> ? SuperReg : getSubReg ( SuperReg , SplitParts [ i ] ) ; if ( SpillToSMEM ) { unsigned Align = FrameInfo . getObjectAlignment ( Index ) ; MachinePointerInfo PtrInfo = MachinePointerInfo :: getFixedStack ( * MF , Index , EltSize * i ) ; MachineMemOperand * MMO = MF -> getMachineMemOperand ( PtrInfo , MachineMemOperand :: MOLoad , EltSize , MinAlign ( Align , EltSize * i ) ) ; int64_t Offset = ( ST . getWavefrontSize ( ) * FrOffset ) + ( EltSize * i ) ; if ( Offset != <NUM_LIT> ) { BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) , OffsetReg ) . addReg ( MFI -> getScratchWaveOffsetReg ( ) ) . addImm ( Offset ) ;


<s> unsigned DstSize = DstRC -> getSize ( ) ; unsigned NewSize = NewRC -> getSize ( ) ; if ( SrcSize <= <NUM_LIT> || DstSize <= <NUM_LIT> ) return true ; return NewSize <= DstSize || NewSize <= SrcSize ;


<s> bool <ISA_LIT>RegisterInfo :: shouldCoalesce ( MachineInstr * MI , const TargetRegisterClass * SrcRC , unsigned SubReg , const TargetRegisterClass * DstRC , unsigned DstSubReg , const TargetRegisterClass * NewRC ) const { unsigned SrcSize = SrcRC -> getSize ( ) ;

unsigned DstSize = DstRC -> getSize ( ) ;


<s> std :: vector < unsigned > PressureSetRegUnits ( NumRegPressureSets , <NUM_LIT> ) ; for ( unsigned i = <NUM_LIT> , e = getNumRegUnits ( ) ; i != e ; ++ i ) { const int * PSets = getRegUnitPressureSets ( i ) ; for ( unsigned j = <NUM_LIT> ; PSets [ j ] != - <NUM_LIT> ; ++ j ) { ++ PressureSetRegUnits [ PSets [ j ] ] ; } } unsigned VGPRMax = <NUM_LIT> , SGPRMax = <NUM_LIT> ; for ( unsigned i = <NUM_LIT> ; i < NumRegPressureSets ; ++ i ) { if ( isVGPRPressureSet ( i ) && PressureSetRegUnits [ i ] > VGPRMax ) { VGPRSetID = i ; VGPRMax = PressureSetRegUnits [ i ] ; continue ; } if ( isSGPRPressureSet ( i ) && PressureSetRegUnits [ i ] > SGPRMax ) { SGPRSetID = i ; SGPRMax = PressureSetRegUnits [ i ] ;


<s> MachineBasicBlock * MBB = MI -> getParent ( ) ; const DebugLoc & DL = MI -> getDebugLoc ( ) ; bool IsStore = MI -> mayStore ( ) ; unsigned Opc = MI -> getOpcode ( ) ; int LoadStoreOp = IsStore ? getOffsetMUBUFStore ( Opc ) : getOffsetMUBUFLoad ( Opc ) ; if ( LoadStoreOp == - <NUM_LIT> ) return false ; unsigned Reg = TII -> getNamedOperand ( * MI , <STR_LIT> :: vdata ) -> getReg ( ) ;

BuildMI ( * MBB , MI , DL , TII -> get ( LoadStoreOp ) ) . addReg ( Reg , getDefRegState ( ! IsStore ) ) . add ( * TII -> getNamedOperand ( * MI , <STR_LIT> :: srsrc ) ) . add ( * TII -> getNamedOperand ( * MI , <STR_LIT> :: soffset ) ) . addImm ( Offset ) . addImm ( <NUM_LIT> ) . addImm ( <NUM_LIT> ) . addImm ( <NUM_LIT> ) . setMemRefs ( MI -> memoperands_begin ( ) , MI -> memoperands_end ( ) ) ;


<s> const DebugLoc & DL = MI -> getDebugLoc ( ) ; bool IsStore = MI -> mayStore ( ) ; unsigned Opc = MI -> getOpcode ( ) ; int LoadStoreOp = IsStore ? getOffsetMUBUFStore ( Opc ) : getOffsetMUBUFLoad ( Opc ) ; if ( LoadStoreOp == - <NUM_LIT> ) return false ;

unsigned Reg = TII -> getNamedOperand ( * MI , <STR_LIT> :: vdata ) -> getReg ( ) ;


<s> unsigned DiffReg = MRI . createVirtualRegister ( & <STR_LIT> ) ; bool IsCopy = MI -> getOpcode ( ) == <STR_LIT> ; unsigned ResultReg = IsCopy ? MI -> getOperand ( <NUM_LIT> ) . getReg ( ) : MRI . createVirtualRegister ( & <STR_LIT> ) ; BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) , DiffReg ) . addReg ( MFI -> getFrameOffsetReg ( ) ) . addReg ( MFI -> getScratchWaveOffsetReg ( ) ) ; int64_t Offset = FrameInfo . getObjectOffset ( Index ) ; if ( Offset == <NUM_LIT> ) { BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) , ResultReg ) . addImm ( Log2_32 ( ST . getWavefrontSize ( ) ) ) . addReg ( DiffReg ) ; } else { unsigned CarryOut = MRI . createVirtualRegister ( & <STR_LIT> ) ; unsigned ScaledReg = MRI . createVirtualRegister ( & <STR_LIT> ) ; BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) , ScaledReg ) . addImm ( Log2_32 ( ST . getWavefrontSize ( ) ) ) . addReg ( DiffReg , RegState :: Kill ) ; if ( <STR_LIT> ( Offset , ST . hasInv2PiInlineImm ( ) ) ) { BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) , ResultReg ) . addReg ( CarryOut , RegState :: Define | RegState :: Dead ) . addImm ( Offset ) . addReg ( ScaledReg , RegState :: Kill ) ; } else { unsigned ConstOffsetReg = MRI . createVirtualRegister ( & <STR_LIT> ) ; BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) , ConstOffsetReg ) . addImm ( Offset ) ; BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) , ResultReg ) . addReg ( CarryOut , RegState :: Define | RegState :: Dead ) . addReg ( ConstOffsetReg , RegState :: Kill ) . addReg ( ScaledReg , RegState :: Kill ) ; } MRI . setRegAllocationHint ( CarryOut , <NUM_LIT> , <STR_LIT> ) ; } if ( IsCopy ) MI -> eraseFromParent ( ) ; else FIOp . ChangeToRegister ( ResultReg , false , false , true ) ; return ; } if ( IsMUBUF ) { assert ( static_cast < int > ( FIOperandNum ) == <STR_LIT> ( MI -> getOpcode ( ) , <STR_LIT> :: vaddr ) ) ; assert ( TII -> getNamedOperand ( * MI , <STR_LIT> :: soffset ) -> getReg ( ) == MFI -> getFrameOffsetReg ( ) ) ; int64_t Offset = FrameInfo . getObjectOffset ( Index ) ; int64_t OldImm = TII -> getNamedOperand ( * MI , <STR_LIT> :: offset ) -> getImm ( ) ;


<s> return isWave32 ? & <STR_LIT> : & <STR_LIT> ;


<s> return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ;


<s> return MFI -> WorkGroupIDYSystemSGPR ; case <ISA_LIT>RegisterInfo :: WORKGROUP_ID_Z : assert ( MFI -> hasWorkGroupIDZ ( ) ) ; return MFI -> WorkGroupIDZSystemSGPR ; case <ISA_LIT>RegisterInfo :: PRIVATE_SEGMENT_WAVE_BYTE_OFFSET : return MFI -> PrivateSegmentWaveByteOffsetSystemSGPR ; case <ISA_LIT>RegisterInfo :: PRIVATE_SEGMENT_BUFFER : if ( ST . isAmdCodeObjectV2 ( MF ) ) { assert ( MFI -> hasPrivateSegmentBuffer ( ) ) ; return MFI -> PrivateSegmentBufferUserSGPR ; } assert ( MFI -> hasPrivateMemoryInputPtr ( ) ) ; return MFI -> PrivateMemoryPtrUserSGPR ; case <ISA_LIT>RegisterInfo :: KERNARG_SEGMENT_PTR : assert ( MFI -> hasKernargSegmentPtr ( ) ) ; return MFI -> KernargSegmentPtrUserSGPR ; case <ISA_LIT>RegisterInfo :: DISPATCH_ID : assert ( MFI -> hasDispatchID ( ) ) ; return MFI -> DispatchIDUserSGPR ; case <ISA_LIT>RegisterInfo :: FLAT_SCRATCH_INIT :

assert ( MFI -> hasFlatScratchInit ( ) ) ;


<s> const <ISA_LIT>Subtarget & ST = MF . getSubtarget < <ISA_LIT>Subtarget > ( ) ; const <ISA_LIT>MachineFunctionInfo * MFI = MF . getInfo < <ISA_LIT>MachineFunctionInfo > ( ) ; unsigned Occupancy = ST . getOccupancyWithLocalMemSize ( MFI -> getLDSSize ( ) , * MF . getFunction ( ) ) ; switch ( RC -> getID ( ) ) { default : return <ISA_LIT>RegisterInfo :: getRegPressureLimit ( RC , MF ) ;


<s> return std :: min ( ST . getMaxNumVGPRs ( Occupancy ) , ST . getMaxNumVGPRs ( MF ) ) ; case <STR_LIT> : return std :: min ( ST . getMaxNumSGPRs ( Occupancy , true ) , ST . getMaxNumSGPRs ( MF ) ) ;


<s> BitVector <ISA_LIT>RegisterInfo :: getReservedRegs ( const MachineFunction & MF ) const { BitVector Reserved ( getNumRegs ( ) ) ; Reserved . set ( <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; const <ISA_LIT>Subtarget & ST = MF . getSubtarget < <ISA_LIT>Subtarget > ( ) ; unsigned MaxNumSGPRs = ST . getMaxNumSGPRs ( MF ) ; unsigned TotalNumSGPRs = <STR_LIT> . getNumRegs ( ) ; for ( unsigned i = MaxNumSGPRs ; i < TotalNumSGPRs ; ++ i ) { unsigned Reg = <STR_LIT> . getRegister ( i ) ; reserveRegisterTuples ( Reserved , Reg ) ; } unsigned MaxNumVGPRs = ST . getMaxNumVGPRs ( MF ) ; unsigned TotalNumVGPRs = <STR_LIT> . getNumRegs ( ) ; for ( unsigned i = MaxNumVGPRs ; i < TotalNumVGPRs ; ++ i ) { unsigned Reg = <STR_LIT> . getRegister ( i ) ; reserveRegisterTuples ( Reserved , Reg ) ; } const <ISA_LIT>MachineFunctionInfo * MFI = MF . getInfo < <ISA_LIT>MachineFunctionInfo > ( ) ; unsigned ScratchWaveOffsetReg = MFI -> getScratchWaveOffsetReg ( ) ; if ( ScratchWaveOffsetReg != <STR_LIT> ) { reserveRegisterTuples ( Reserved , ScratchWaveOffsetReg ) ; } unsigned ScratchRSrcReg = MFI -> getScratchRSrcReg ( ) ; if ( ScratchRSrcReg != <STR_LIT> ) {

reserveRegisterTuples ( Reserved , ScratchRSrcReg ) ;


<s> if ( SubIdx == <STR_LIT> ) return RC ; LaneBitmask :: Type Mask = getSubRegIndexLaneMask ( SubIdx ) . getAsInteger ( ) ; unsigned Count = countPopulation ( Mask ) ; if ( isSGPRClass ( RC ) ) { switch ( Count ) { case <NUM_LIT> : return & <STR_LIT> ; case <NUM_LIT> : return & <STR_LIT> ; case <NUM_LIT> :


<s> virtual bool isDivergentRegClass ( const TargetRegisterClass * RC ) const override {


<s> bool <ISA_LIT>RegisterInfo :: isVGPR ( const MachineRegisterInfo & MRI , unsigned Reg ) const {

return hasVGPRs ( getRegClassForReg ( MRI , Reg ) ) ;


<s> return hasVGPRs ( getRegClassForReg ( MRI , Reg ) ) ;


<s> bool <ISA_LIT>RegisterInfo :: requiresFrameIndexScavenging ( const MachineFunction & MF ) const { return MF . getFrameInfo ( ) . hasStackObjects ( ) ;


<s> unsigned SrcSize = getRegSizeInBits ( * SrcRC ) ; unsigned DstSize = getRegSizeInBits ( * DstRC ) ; unsigned NewSize = getRegSizeInBits ( * NewRC ) ; if ( SrcSize <= <NUM_LIT> || DstSize <= <NUM_LIT> ) return true ; return NewSize <= DstSize || NewSize <= SrcSize ;


<s> } else { Scavenged = true ; } BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) , SOffset ) . addReg ( ScratchOffset ) . addImm ( Offset ) ; Offset = <NUM_LIT> ; } if ( RanOutOfSGPRs ) Ctx . emitError ( <STR_LIT> ) ; for ( unsigned i = <NUM_LIT> , e = NumSubRegs ; i != e ; ++ i , Offset += <NUM_LIT> ) { unsigned SubReg = NumSubRegs > <NUM_LIT> ? getPhysRegSubReg ( Value , & <STR_LIT> , i ) : Value ; unsigned SOffsetRegState = <NUM_LIT> ; if ( i + <NUM_LIT> == e && Scavenged ) SOffsetRegState |= RegState :: Kill ; BuildMI ( * MBB , MI , DL , TII -> get ( LoadStoreOp ) ) . addReg ( SubReg , getDefRegState ( ! IsStore ) ) . addReg ( ScratchRsrcReg ) . addReg ( SOffset , SOffsetRegState ) . addImm ( Offset ) . addImm ( <NUM_LIT> ) . addImm ( <NUM_LIT> ) . addImm ( <NUM_LIT> ) . addReg ( Value , RegState :: Implicit | getDefRegState ( ! IsStore ) ) . setMemRefs ( MI -> memoperands_begin ( ) , MI -> memoperands_end ( ) ) ;


<s> unsigned NumSubRegs = getNumSubRegsForSpillOp ( MI -> getOpcode ( ) ) ; unsigned Size = NumSubRegs * <NUM_LIT> ; if ( ! isUInt < <NUM_LIT> > ( Offset + Size ) ) { SOffset = MRI . createVirtualRegister ( & <STR_LIT> ) ; if ( SOffset == <STR_LIT> ) { RanOutOfSGPRs = true ; SOffset = <STR_LIT> ; } else { Scavenged = true ; } BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) , SOffset ) . addReg ( ScratchOffset ) . addImm ( Offset ) ; Offset = <NUM_LIT> ; } if ( RanOutOfSGPRs ) Ctx . emitError ( <STR_LIT> ) ; for ( unsigned i = <NUM_LIT> , e = NumSubRegs ; i != e ; ++ i , Offset += <NUM_LIT> ) { unsigned SubReg = NumSubRegs > <NUM_LIT> ? getPhysRegSubReg ( Value , & <STR_LIT> , i ) :


<s> case <ISA_LIT>RegisterInfo :: PRIVATE_SEGMENT_BUFFER : assert ( ST . isAmdHsaOS ( ) && <STR_LIT> ) ; assert ( MFI -> hasPrivateSegmentBuffer ( ) ) ; return MFI -> PrivateSegmentBufferUserSGPR ; case <ISA_LIT>RegisterInfo :: KERNARG_SEGMENT_PTR : assert ( MFI -> hasKernargSegmentPtr ( ) ) ; return MFI -> KernargSegmentPtrUserSGPR ; case <ISA_LIT>RegisterInfo :: DISPATCH_ID : llvm_unreachable ( <STR_LIT> ) ; case <ISA_LIT>RegisterInfo :: FLAT_SCRATCH_INIT : assert ( MFI -> hasFlatScratchInit ( ) ) ; return MFI -> FlatScratchInitUserSGPR ; case <ISA_LIT>RegisterInfo :: DISPATCH_PTR : assert ( MFI -> hasDispatchPtr ( ) ) ; return MFI -> DispatchPtrUserSGPR ;


<s> reserveRegisterTuples ( Reserved , Reg ) ; } } const <ISA_LIT>MachineFunctionInfo * MFI = MF . getInfo < <ISA_LIT>MachineFunctionInfo > ( ) ; unsigned ScratchWaveOffsetReg = MFI -> getScratchWaveOffsetReg ( ) ; if ( ScratchWaveOffsetReg != <STR_LIT> ) { reserveRegisterTuples ( Reserved , ScratchWaveOffsetReg ) ; } unsigned ScratchRSrcReg = MFI -> getScratchRSrcReg ( ) ; if ( ScratchRSrcReg != <STR_LIT> ) { reserveRegisterTuples ( Reserved , ScratchRSrcReg ) ; assert ( ! isSubRegister ( ScratchRSrcReg , ScratchWaveOffsetReg ) ) ; }


<s> bool hasVectorRegisters ( const TargetRegisterClass * RC ) const {

return hasVGPRs ( RC ) || hasAGPRs ( RC ) ;


<s> bool isAGPRPressureSet ( unsigned SetID ) const { return AGPRPressureSets . test ( SetID ) && ! SGPRPressureSets . test ( SetID ) && ! VGPRPressureSets . test ( SetID ) ;


<s> return ! hasVGPRs ( RC ) && ! hasAGPRs ( RC ) ;


<s> bool isSGPRPressureSet ( unsigned SetID ) const {

return SGPRPressureSets . test ( SetID ) && ! VGPRPressureSets . test ( SetID ) && ! AGPRPressureSets . test ( SetID ) ;


<s> return SGPRPressureSets . test ( SetID ) && ! VGPRPressureSets . test ( SetID ) && ! AGPRPressureSets . test ( SetID ) ;


<s> bool isVectorRegister ( const MachineRegisterInfo & MRI , unsigned Reg ) const { return isVGPR ( MRI , Reg ) || isAGPR ( MRI , Reg ) ;


<s> bool isVGPRPressureSet ( unsigned SetID ) const { return VGPRPressureSets . test ( SetID ) && ! SGPRPressureSets . test ( SetID ) && ! AGPRPressureSets . test ( SetID ) ;


<s> return VGPRPressureSets . test ( SetID ) && ! SGPRPressureSets . test ( SetID ) && ! AGPRPressureSets . test ( SetID ) ;


<s> unsigned BaseIdx = <ISA_LIT>Subtarget :: FIXED_SGPR_COUNT_FOR_INIT_BUG - <NUM_LIT> - <NUM_LIT> ; unsigned BaseReg ( <STR_LIT> . getRegister ( BaseIdx ) ) ; return getMatchingSuperReg ( BaseReg , <STR_LIT> , & <STR_LIT> ) ; } if ( ST . getGeneration ( ) >= <ISA_LIT>Subtarget :: VOLCANIC_ISLANDS ) { return <STR_LIT> ;


<s> const <ISA_LIT>Subtarget & ST = MF . getSubtarget < <ISA_LIT>Subtarget > ( ) ; if ( ST . hasSGPRInitBug ( ) ) { unsigned Idx = <ISA_LIT>Subtarget :: FIXED_SGPR_COUNT_FOR_INIT_BUG - <NUM_LIT> - <NUM_LIT> ; return <STR_LIT> . getRegister ( Idx ) ; }

if ( ST . getGeneration ( ) >= <ISA_LIT>Subtarget :: VOLCANIC_ISLANDS ) {


<s> unsigned NumRegPressureSets = getNumRegPressureSets ( ) ; SGPR32SetID = NumRegPressureSets ; VGPR32SetID = NumRegPressureSets ; for ( unsigned i = <NUM_LIT> ; i < NumRegPressureSets ; ++ i ) { if ( strncmp ( <STR_LIT> , getRegPressureSetName ( i ) , <NUM_LIT> ) == <NUM_LIT> ) SGPR32SetID = i ; else if ( strncmp ( <STR_LIT> , getRegPressureSetName ( i ) , <NUM_LIT> ) == <NUM_LIT> ) VGPR32SetID = i ; } assert ( SGPR32SetID < NumRegPressureSets && VGPR32SetID < NumRegPressureSets ) ;


<s> <ISA_LIT>RegisterInfo :: <ISA_LIT>RegisterInfo ( ) : <ISA_LIT>RegisterInfo ( ) { unsigned NumRegPressureSets = getNumRegPressureSets ( ) ; SGPR32SetID = NumRegPressureSets ; VGPR32SetID = NumRegPressureSets ; for ( unsigned i = <NUM_LIT> ; i < NumRegPressureSets ; ++ i ) { if ( strncmp ( <STR_LIT> , getRegPressureSetName ( i ) , <NUM_LIT> ) == <NUM_LIT> ) SGPR32SetID = i ; else if ( strncmp ( <STR_LIT> , getRegPressureSetName ( i ) , <NUM_LIT> ) == <NUM_LIT> ) VGPR32SetID = i ;


<s> } if ( IsStore ) BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) , TmpReg ) . addReg ( SubReg , getKillRegState ( IsKill ) ) ; SubReg = TmpReg ; } MachinePointerInfo PInfo = BasePtrInfo . getWithOffset ( EltSize * i ) ; MachineMemOperand * NewMMO = MF -> getMachineMemOperand ( PInfo , MMO -> getFlags ( ) , EltSize , commonAlignment ( Alignment , EltSize * i ) ) ; MIB = BuildMI ( * MBB , MI , DL , Desc ) . addReg ( SubReg , getDefRegState ( ! IsStore ) | getKillRegState ( IsKill ) ) . addReg ( ScratchRsrcReg ) ; if ( SOffset == <STR_LIT> ) { MIB . addImm ( <NUM_LIT> ) ; } else { MIB . addReg ( SOffset , SOffsetRegState ) ; } MIB . addImm ( Offset ) . addImm ( <NUM_LIT> ) . addImm ( <NUM_LIT> ) . addImm ( <NUM_LIT> ) . addImm ( <NUM_LIT> ) . addImm ( <NUM_LIT> ) . addMemOperand ( NewMMO ) ; if ( ! IsStore && TmpReg != <STR_LIT> ) MIB = BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) , FinalReg ) . addReg ( TmpReg , RegState :: Kill ) ; } if ( NumSubRegs > <NUM_LIT> ) MIB . addReg ( ValueReg , RegState :: Implicit | SrcDstRegState ) ; } if ( ScratchOffsetRegDelta != <NUM_LIT> ) { BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) , SOffset ) . addReg ( SOffset ) . addImm ( ScratchOffsetRegDelta ) ;


<s> reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; if ( isWave32 ) { Reserved . set ( <STR_LIT> ) ; Reserved . set ( <STR_LIT> ) ; } unsigned MaxNumSGPRs = ST . getMaxNumSGPRs ( MF ) ; unsigned TotalNumSGPRs = <STR_LIT> . getNumRegs ( ) ; for ( unsigned i = MaxNumSGPRs ; i < TotalNumSGPRs ; ++ i ) { unsigned Reg = <STR_LIT> . getRegister ( i ) ; reserveRegisterTuples ( Reserved , Reg ) ; } unsigned MaxNumVGPRs = ST . getMaxNumVGPRs ( MF ) ; unsigned TotalNumVGPRs = <STR_LIT> . getNumRegs ( ) ; for ( unsigned i = MaxNumVGPRs ; i < TotalNumVGPRs ; ++ i ) { unsigned Reg = <STR_LIT> . getRegister ( i ) ; reserveRegisterTuples ( Reserved , Reg ) ; Reg = <STR_LIT> . getRegister ( i ) ; reserveRegisterTuples ( Reserved , Reg ) ; } for ( auto Reg : <STR_LIT> ) { Reserved . set ( getSubReg ( Reg , <STR_LIT> ) ) ; Register Low = getSubReg ( Reg , <STR_LIT> ) ; if ( ! <STR_LIT> . contains ( Low ) ) Reserved . set ( Low ) ; } for ( auto Reg : <STR_LIT> ) { Reserved . set ( getSubReg ( Reg , <STR_LIT> ) ) ; } if ( ! ST . hasMAIInsts ( ) ) { for ( unsigned i = <NUM_LIT> ; i < MaxNumVGPRs ; ++ i ) { unsigned Reg = <STR_LIT> . getRegister ( i ) ; reserveRegisterTuples ( Reserved , Reg ) ; } } const <ISA_LIT>MachineFunctionInfo * MFI = MF . getInfo < <ISA_LIT>MachineFunctionInfo > ( ) ; Register ScratchRSrcReg = MFI -> getScratchRSrcReg ( ) ; if ( ScratchRSrcReg != <STR_LIT> ) { reserveRegisterTuples ( Reserved , ScratchRSrcReg ) ; } MCRegister StackPtrReg = MFI -> getStackPtrOffsetReg ( ) ; if ( StackPtrReg ) { reserveRegisterTuples ( Reserved , StackPtrReg ) ; assert ( ! isSubRegister ( ScratchRSrcReg , StackPtrReg ) ) ; } MCRegister FrameReg = MFI -> getFrameOffsetReg ( ) ; if ( FrameReg ) { reserveRegisterTuples ( Reserved , FrameReg ) ; assert ( ! isSubRegister ( ScratchRSrcReg , FrameReg ) ) ; } if ( hasBasePointer ( MF ) ) { MCRegister BasePtrReg = getBaseRegister ( ) ; reserveRegisterTuples ( Reserved , BasePtrReg ) ; assert ( ! isSubRegister ( ScratchRSrcReg , BasePtrReg ) ) ; } for ( MCRegister Reg : MFI -> WWMReservedRegs ) { reserveRegisterTuples ( Reserved , Reg ) ;


<s> bool <ISA_LIT>RegisterInfo :: isFrameOffsetLegal ( const MachineInstr * MI , Register BaseReg , int64_t Offset ) const { if ( ! <ISA_LIT>InstrInfo :: isMUBUF ( * MI ) ) return false ; int64_t NewOffset = Offset + getMUBUFInstrOffset ( MI ) ;

return <ISA_LIT>InstrInfo :: isLegalMUBUFImmOffset ( NewOffset ) ;


<s> bool <ISA_LIT>RegisterInfo :: needsFrameBaseReg ( MachineInstr * MI , int64_t Offset ) const { if ( ! MI -> mayLoadOrStore ( ) ) return false ;

int64_t FullOffset = Offset + getMUBUFInstrOffset ( MI ) ;


<s> int64_t FullOffset = Offset + getMUBUFInstrOffset ( MI ) ; return ! <ISA_LIT>InstrInfo :: isLegalMUBUFImmOffset ( FullOffset ) ;


<s> assert ( TII -> isMUBUF ( MI ) ) ; MachineOperand * SOffset = TII -> getNamedOperand ( MI , <STR_LIT> :: soffset ) ; assert ( SOffset -> getReg ( ) == MF -> getInfo < <ISA_LIT>MachineFunctionInfo > ( ) -> getStackPtrOffsetReg ( ) && <STR_LIT> ) ; MachineOperand * OffsetOp = TII -> getNamedOperand ( MI , <STR_LIT> :: offset ) ; int64_t NewOffset = OffsetOp -> getImm ( ) + Offset ; assert ( <ISA_LIT>InstrInfo :: isLegalMUBUFImmOffset ( NewOffset ) && <STR_LIT> ) ; FIOp -> ChangeToRegister ( BaseReg , false ) ;


<s> unsigned EltSize = <NUM_LIT> ; const TargetRegisterClass * RC = getPhysRegClass ( SuperReg ) ; ArrayRef < int16_t > SplitParts = getRegSplitParts ( RC , EltSize ) ; unsigned NumSubRegs = SplitParts . empty ( ) ? <NUM_LIT> : SplitParts . size ( ) ; if ( SpillToVGPR ) { for ( unsigned i = <NUM_LIT> , e = NumSubRegs ; i < e ; ++ i ) { Register SubReg = NumSubRegs == <NUM_LIT> ? SuperReg : getSubReg ( SuperReg , SplitParts [ i ] ) ; <ISA_LIT>MachineFunctionInfo :: SpilledReg Spill = VGPRSpills [ i ] ; bool UseKill = IsKill && i == NumSubRegs - <NUM_LIT> ; bool VGPRDefined = true ; if ( MBB == & MF -> front ( ) ) VGPRDefined = ! SGPRSpillVGPRDefinedSet . insert ( Spill . VGPR ) . second ; auto MIB = BuildMI ( * MBB , MI , DL , TII -> getMCOpcodeFromPseudo ( <STR_LIT> ) , Spill . VGPR ) . addReg ( SubReg , getKillRegState ( UseKill ) ) . addImm ( Spill . Lane ) . addReg ( Spill . VGPR , VGPRDefined ? <NUM_LIT> : RegState :: Undef ) ; if ( i == <NUM_LIT> && NumSubRegs > <NUM_LIT> ) { MIB . addReg ( SuperReg , RegState :: ImplicitDefine ) ; } if ( NumSubRegs > <NUM_LIT> ) MIB . addReg ( SuperReg , getKillRegState ( UseKill ) | RegState :: Implicit ) ; } } else { Register TmpVGPR = RS -> scavengeRegister ( & <STR_LIT> , MI , <NUM_LIT> ) ; RS -> setRegUsed ( TmpVGPR ) ; unsigned SubKillState = getKillRegState ( ( NumSubRegs == <NUM_LIT> ) && IsKill ) ; unsigned PerVGPR = <NUM_LIT> ; unsigned NumVGPRs = ( NumSubRegs + ( PerVGPR - <NUM_LIT> ) ) / PerVGPR ; int64_t VGPRLanes = ( <NUM_LIT> << std :: min ( PerVGPR , NumSubRegs ) ) - <NUM_LIT> ; for ( unsigned Offset = <NUM_LIT> ; Offset < NumVGPRs ; ++ Offset ) { unsigned TmpVGPRFlags = RegState :: Undef ; for ( unsigned i = Offset * PerVGPR , e = std :: min ( ( Offset + <NUM_LIT> ) * PerVGPR , NumSubRegs ) ; i < e ; ++ i ) { Register SubReg = NumSubRegs == <NUM_LIT> ? SuperReg : getSubReg ( SuperReg , SplitParts [ i ] ) ; MachineInstrBuilder WriteLane = BuildMI ( * MBB , MI , DL , TII -> getMCOpcodeFromPseudo ( <STR_LIT> ) , TmpVGPR ) . addReg ( SubReg , SubKillState ) . addImm ( i % PerVGPR ) . addReg ( TmpVGPR , TmpVGPRFlags ) ; TmpVGPRFlags = <NUM_LIT> ; if ( NumSubRegs > <NUM_LIT> ) { unsigned SuperKillState = <NUM_LIT> ; if ( i + <NUM_LIT> == NumSubRegs ) SuperKillState |= getKillRegState ( IsKill ) ;

WriteLane . addReg ( SuperReg , RegState :: Implicit | SuperKillState ) ;


<s> const DebugLoc & DL = MI -> getDebugLoc ( ) ; bool IsStore = Desc . mayStore ( ) ; bool Scavenged = false ; MCRegister SOffset = ScratchOffsetReg ; const unsigned EltSize = <NUM_LIT> ; const TargetRegisterClass * RC = getRegClassForReg ( MF -> getRegInfo ( ) , ValueReg ) ; unsigned NumSubRegs = <STR_LIT> ( RC -> getID ( ) ) / ( EltSize * CHAR_BIT ) ; unsigned Size = NumSubRegs * EltSize ; int64_t Offset = InstOffset + MFI . getObjectOffset ( Index ) ; int64_t ScratchOffsetRegDelta = <NUM_LIT> ; Align Alignment = MFI . getObjectAlign ( Index ) ; const MachinePointerInfo & BasePtrInfo = MMO -> getPointerInfo ( ) ; Register TmpReg = hasAGPRs ( RC ) ? TII -> getNamedOperand ( * MI , <STR_LIT> :: tmp ) -> getReg ( ) : Register ( ) ; assert ( ( Offset % EltSize ) == <NUM_LIT> && <STR_LIT> ) ; if ( ! isUInt < <NUM_LIT> > ( Offset + Size - EltSize ) ) { SOffset = MCRegister ( ) ; Offset *= ST . getWavefrontSize ( ) ; if ( RS ) SOffset = RS -> scavengeRegister ( & <STR_LIT> , MI , <NUM_LIT> , false ) ; if ( ! SOffset ) { if ( ! ScratchOffsetReg ) { report_fatal_error ( <STR_LIT> ) ; } SOffset = ScratchOffsetReg ; ScratchOffsetRegDelta = Offset ; } else { Scavenged = true ; } if ( ScratchOffsetReg == <STR_LIT> ) { BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) , SOffset ) . addImm ( Offset ) ; } else { BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) , SOffset ) . addReg ( ScratchOffsetReg ) . addImm ( Offset ) ; } Offset = <NUM_LIT> ; } for ( unsigned i = <NUM_LIT> , e = NumSubRegs ; i != e ; ++ i , Offset += EltSize ) { Register SubReg = NumSubRegs == <NUM_LIT> ? Register ( ValueReg ) : getSubReg ( ValueReg , getSubRegFromChannel ( i ) ) ; unsigned SOffsetRegState = <NUM_LIT> ; unsigned SrcDstRegState = getDefRegState ( ! IsStore ) ; if ( i + <NUM_LIT> == e ) { SOffsetRegState |= getKillRegState ( Scavenged ) ; SrcDstRegState |= getKillRegState ( IsKill ) ; } auto MIB = spillVGPRtoAGPR ( ST , MI , Index , i , SubReg , IsKill ) ; if ( ! MIB . getInstr ( ) ) { unsigned FinalReg = SubReg ; if ( TmpReg != <STR_LIT> ) {

if ( IsStore ) BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) , TmpReg ) . addReg ( SubReg , getKillRegState ( IsKill ) ) ;


<s> MCRegister <ISA_LIT>RegisterInfo :: findUnusedRegister ( const MachineRegisterInfo & MRI , const TargetRegisterClass * RC , const MachineFunction & MF ) const { for ( MCRegister Reg : * RC ) if ( MRI . isAllocatable ( Reg ) && ! MRI . isPhysRegUsed ( Reg ) ) return Reg ;


<s> static const TargetRegisterClass * const BaseClasses [ ] = { & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , & <STR_LIT> , } ; for ( const TargetRegisterClass * BaseClass : BaseClasses ) { if ( BaseClass -> contains ( Reg ) ) { return BaseClass ; }


<s> unsigned Occupancy = ST . getOccupancyWithLocalMemSize ( MFI -> getLDSSize ( ) , MF . getFunction ( ) ) ; switch ( RC -> getID ( ) ) { default : return <ISA_LIT>GenRegisterInfo :: getRegPressureLimit ( RC , MF ) ; case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : return std :: min ( ST . getMaxNumVGPRs ( Occupancy ) , ST . getMaxNumVGPRs ( MF ) ) ; case <STR_LIT> : return std :: min ( ST . getMaxNumSGPRs ( Occupancy , true ) , ST . getMaxNumSGPRs ( MF ) ) ;


<s> const <ISA_LIT>MachineFunctionInfo * MFI = MF . getInfo < <ISA_LIT>MachineFunctionInfo > ( ) ; unsigned Occupancy = ST . getOccupancyWithLocalMemSize ( MFI -> getLDSSize ( ) , MF . getFunction ( ) ) ; switch ( RC -> getID ( ) ) { default : return <ISA_LIT>GenRegisterInfo :: getRegPressureLimit ( RC , MF ) ; case <STR_LIT> : case <STR_LIT> :


<s> reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; if ( isWave32 ) { Reserved . set ( <STR_LIT> ) ; Reserved . set ( <STR_LIT> ) ; } unsigned MaxNumSGPRs = ST . getMaxNumSGPRs ( MF ) ; unsigned TotalNumSGPRs = <STR_LIT> . getNumRegs ( ) ; for ( unsigned i = MaxNumSGPRs ; i < TotalNumSGPRs ; ++ i ) { unsigned Reg = <STR_LIT> . getRegister ( i ) ; reserveRegisterTuples ( Reserved , Reg ) ; } unsigned MaxNumVGPRs = ST . getMaxNumVGPRs ( MF ) ; unsigned TotalNumVGPRs = <STR_LIT> . getNumRegs ( ) ; for ( unsigned i = MaxNumVGPRs ; i < TotalNumVGPRs ; ++ i ) { unsigned Reg = <STR_LIT> . getRegister ( i ) ; reserveRegisterTuples ( Reserved , Reg ) ; Reg = <STR_LIT> . getRegister ( i ) ; reserveRegisterTuples ( Reserved , Reg ) ; } if ( ! ST . hasMAIInsts ( ) ) { for ( unsigned i = <NUM_LIT> ; i < MaxNumVGPRs ; ++ i ) { unsigned Reg = <STR_LIT> . getRegister ( i ) ; reserveRegisterTuples ( Reserved , Reg ) ; } } const <ISA_LIT>MachineFunctionInfo * MFI = MF . getInfo < <ISA_LIT>MachineFunctionInfo > ( ) ; unsigned ScratchRSrcReg = MFI -> getScratchRSrcReg ( ) ; if ( ScratchRSrcReg != <STR_LIT> ) { reserveRegisterTuples ( Reserved , ScratchRSrcReg ) ; } MCRegister StackPtrReg = MFI -> getStackPtrOffsetReg ( ) ; if ( StackPtrReg ) { reserveRegisterTuples ( Reserved , StackPtrReg ) ; assert ( ! isSubRegister ( ScratchRSrcReg , StackPtrReg ) ) ; } MCRegister FrameReg = MFI -> getFrameOffsetReg ( ) ; if ( FrameReg ) { reserveRegisterTuples ( Reserved , FrameReg ) ; assert ( ! isSubRegister ( ScratchRSrcReg , FrameReg ) ) ; } for ( MCRegister Reg : MFI -> WWMReservedRegs ) { reserveRegisterTuples ( Reserved , Reg ) ;


<s> static bool hasSGPRs ( const TargetRegisterClass * RC ) {


<s> return getCommonSubClass ( & <STR_LIT> , RC ) != nullptr ; case <NUM_LIT> : return getCommonSubClass ( & <STR_LIT> , RC ) != nullptr ; case <NUM_LIT> : return getCommonSubClass ( & <STR_LIT> , RC ) != nullptr ; case <NUM_LIT> : return getCommonSubClass ( & <STR_LIT> , RC ) != nullptr ; case <NUM_LIT> : return getCommonSubClass ( & <STR_LIT> , RC ) != nullptr ;


<s> assert ( RC && <STR_LIT> ) ; return hasAGPRs ( RC ) ;


<s> const TargetRegisterClass * RC = getRegClassForReg ( MRI , Reg ) ; assert ( RC && <STR_LIT> ) ; return hasAGPRs ( RC ) ;


<s> return hasAGPRs ( RC ) && ! hasVGPRs ( RC ) && ! hasSGPRs ( RC ) ;


<s> return hasSGPRs ( RC ) && ! hasVGPRs ( RC ) && ! hasAGPRs ( RC ) ;


<s> return hasVGPRs ( RC ) && hasAGPRs ( RC ) && ! hasSGPRs ( RC ) ;


<s> assert ( RC && <STR_LIT> ) ; return hasVGPRs ( RC ) ;


<s> const TargetRegisterClass * RC = getRegClassForReg ( MRI , Reg ) ; assert ( RC && <STR_LIT> ) ; return hasVGPRs ( RC ) ;


<s> static bool isVGPRClass ( const TargetRegisterClass * RC ) {

return hasVGPRs ( RC ) && ! hasAGPRs ( RC ) && ! hasSGPRs ( RC ) ;


<s> bool isVSSuperClass ( const TargetRegisterClass * RC ) const { return hasVGPRs ( RC ) && hasSGPRs ( RC ) && ! hasAGPRs ( RC ) ;


<s> const <ISA_LIT>InstrInfo * TII = ST . getInstrInfo ( ) ; const DebugLoc & DL = MI -> getDebugLoc ( ) ; Register SuperReg = MI -> getOperand ( <NUM_LIT> ) . getReg ( ) ; assert ( SuperReg != <STR_LIT> && <STR_LIT> ) ; unsigned EltSize = <NUM_LIT> ; const TargetRegisterClass * RC = getPhysRegClass ( SuperReg ) ; ArrayRef < int16_t > SplitParts = getRegSplitParts ( RC , EltSize ) ; unsigned NumSubRegs = SplitParts . empty ( ) ? <NUM_LIT> : SplitParts . size ( ) ; Register TmpVGPR ; for ( unsigned i = <NUM_LIT> , e = NumSubRegs ; i < e ; ++ i ) { Register SubReg = NumSubRegs == <NUM_LIT> ? SuperReg : getSubReg ( SuperReg , SplitParts [ i ] ) ; if ( SpillToVGPR ) { <ISA_LIT>MachineFunctionInfo :: SpilledReg Spill = VGPRSpills [ i ] ;

auto MIB = BuildMI ( * MBB , MI , DL , TII -> getMCOpcodeFromPseudo ( <STR_LIT> ) , SubReg ) . addReg ( Spill . VGPR ) . addImm ( Spill . Lane ) ;


<s> <ISA_LIT>MachineFunctionInfo * MFI = MF -> getInfo < <ISA_LIT>MachineFunctionInfo > ( ) ; ArrayRef < <ISA_LIT>MachineFunctionInfo :: SpilledReg > VGPRSpills = MFI -> getSGPRToVGPRSpills ( Index ) ; bool SpillToVGPR = ! VGPRSpills . empty ( ) ; if ( OnlyToVGPR && ! SpillToVGPR ) return false ; MachineFrameInfo & FrameInfo = MF -> getFrameInfo ( ) ; const <ISA_LIT>InstrInfo * TII = ST . getInstrInfo ( ) ; const DebugLoc & DL = MI -> getDebugLoc ( ) ; Register SuperReg = MI -> getOperand ( <NUM_LIT> ) . getReg ( ) ; assert ( SuperReg != <STR_LIT> && <STR_LIT> ) ; unsigned EltSize = <NUM_LIT> ; const TargetRegisterClass * RC = getPhysRegClass ( SuperReg ) ; ArrayRef < int16_t > SplitParts = getRegSplitParts ( RC , EltSize ) ; unsigned NumSubRegs = SplitParts . empty ( ) ? <NUM_LIT> : SplitParts . size ( ) ; Register TmpVGPR ; for ( unsigned i = <NUM_LIT> , e = NumSubRegs ; i < e ; ++ i ) { Register SubReg = NumSubRegs == <NUM_LIT> ? SuperReg : getSubReg ( SuperReg , SplitParts [ i ] ) ; if ( SpillToVGPR ) { <ISA_LIT>MachineFunctionInfo :: SpilledReg Spill = VGPRSpills [ i ] ; auto MIB = BuildMI ( * MBB , MI , DL , TII -> getMCOpcodeFromPseudo ( <STR_LIT> ) , SubReg ) . addReg ( Spill . VGPR ) . addImm ( Spill . Lane ) ; if ( NumSubRegs > <NUM_LIT> && i == <NUM_LIT> ) MIB . addReg ( SuperReg , RegState :: ImplicitDefine ) ; } else { if ( OnlyToVGPR ) return false ; if ( ! TmpVGPR . isValid ( ) ) TmpVGPR = RS -> scavengeRegister ( & <STR_LIT> , MI , <NUM_LIT> ) ; Align Alignment = FrameInfo . getObjectAlign ( Index ) ; MachinePointerInfo PtrInfo = MachinePointerInfo :: getFixedStack ( * MF , Index , EltSize * i ) ; MachineMemOperand * MMO = MF -> getMachineMemOperand ( PtrInfo , MachineMemOperand :: MOLoad , EltSize , commonAlignment ( Alignment , EltSize * i ) ) ; BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) , TmpVGPR ) . addFrameIndex ( Index ) . addReg ( MFI -> getScratchRSrcReg ( ) ) . addReg ( MFI -> getStackPtrOffsetReg ( ) ) . addImm ( i * <NUM_LIT> ) . addMemOperand ( MMO ) ; auto MIB = BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) , SubReg ) . addReg ( TmpVGPR , RegState :: Kill ) ;

if ( NumSubRegs > <NUM_LIT> ) MIB . addReg ( MI -> getOperand ( <NUM_LIT> ) . getReg ( ) , RegState :: ImplicitDefine ) ;


<s> assert ( getSubRegIndexLaneMask ( <STR_LIT> ) . getAsInteger ( ) == <NUM_LIT> && getSubRegIndexLaneMask ( <STR_LIT> ) . getAsInteger ( ) == ( <NUM_LIT> << <NUM_LIT> ) && ( getSubRegIndexLaneMask ( <STR_LIT> ) | getSubRegIndexLaneMask ( <STR_LIT> ) ) . getAsInteger ( ) == getSubRegIndexLaneMask ( <STR_LIT> ) . getAsInteger ( ) && <STR_LIT> ) ; RegPressureIgnoredUnits . resize ( getNumRegUnits ( ) ) ; RegPressureIgnoredUnits . set ( * MCRegUnitIterator ( <STR_LIT> , this ) ) ; for ( auto Reg : <STR_LIT> ) RegPressureIgnoredUnits . set ( * MCRegUnitIterator ( Reg , this ) ) ;


<s> MachineFunction * MF = MBB -> getParent ( ) ; <ISA_LIT>MachineFunctionInfo * MFI = MF -> getInfo < <ISA_LIT>MachineFunctionInfo > ( ) ; DenseSet < unsigned > SGPRSpillVGPRDefinedSet ; ArrayRef < <ISA_LIT>MachineFunctionInfo :: SpilledReg > VGPRSpills = MFI -> getSGPRToVGPRSpills ( Index ) ; bool SpillToVGPR = ! VGPRSpills . empty ( ) ; if ( OnlyToVGPR && ! SpillToVGPR ) return false ; const <ISA_LIT>InstrInfo * TII = ST . getInstrInfo ( ) ; Register SuperReg = MI -> getOperand ( <NUM_LIT> ) . getReg ( ) ; bool IsKill = MI -> getOperand ( <NUM_LIT> ) . isKill ( ) ; const DebugLoc & DL = MI -> getDebugLoc ( ) ; MachineFrameInfo & FrameInfo = MF -> getFrameInfo ( ) ; assert ( SpillToVGPR || ( SuperReg != MFI -> getStackPtrOffsetReg ( ) && SuperReg != MFI -> getFrameOffsetReg ( ) ) ) ; assert ( SuperReg != <STR_LIT> && <STR_LIT> ) ; unsigned EltSize = <NUM_LIT> ; const TargetRegisterClass * RC = getPhysRegClass ( SuperReg ) ; ArrayRef < int16_t > SplitParts = getRegSplitParts ( RC , EltSize ) ; unsigned NumSubRegs = SplitParts . empty ( ) ? <NUM_LIT> : SplitParts . size ( ) ; Register TmpVGPR ; unsigned SubKillState = getKillRegState ( ( NumSubRegs == <NUM_LIT> ) && IsKill ) ; for ( unsigned i = <NUM_LIT> , e = NumSubRegs ; i < e ; ++ i ) { Register SubReg = NumSubRegs == <NUM_LIT> ? SuperReg : getSubReg ( SuperReg , SplitParts [ i ] ) ; if ( SpillToVGPR ) { <ISA_LIT>MachineFunctionInfo :: SpilledReg Spill = VGPRSpills [ i ] ; bool VGPRDefined = true ; if ( MBB == & MF -> front ( ) ) VGPRDefined = ! SGPRSpillVGPRDefinedSet . insert ( Spill . VGPR ) . second ; BuildMI ( * MBB , MI , DL , TII -> getMCOpcodeFromPseudo ( <STR_LIT> ) , Spill . VGPR ) . addReg ( SubReg , getKillRegState ( IsKill ) ) . addImm ( Spill . Lane ) . addReg ( Spill . VGPR , VGPRDefined ? <NUM_LIT> : RegState :: Undef ) ; } else { if ( OnlyToVGPR ) return false ; if ( ! TmpVGPR . isValid ( ) ) TmpVGPR = RS -> scavengeRegister ( & <STR_LIT> , MI , <NUM_LIT> ) ; MachineInstrBuilder Mov = BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) , TmpVGPR ) . addReg ( SubReg , SubKillState ) ; if ( NumSubRegs > <NUM_LIT> ) { unsigned SuperKillState = <NUM_LIT> ;

if ( i + <NUM_LIT> == e ) SuperKillState |= getKillRegState ( IsKill ) ;


<s> bool isVectorSuperClass ( const TargetRegisterClass * RC ) const { return hasVGPRs ( RC ) && hasAGPRs ( RC ) ;


<s> MFI -> addToSpilledVGPRs ( getNumSubRegsForSpillOp ( MI -> getOpcode ( ) ) ) ; MI -> eraseFromParent ( ) ; break ; } case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : { const MachineOperand * VData = TII -> getNamedOperand ( * MI , <STR_LIT> :: vdata ) ; buildSpillLoadStore ( MI , <STR_LIT> , Index , VData -> getReg ( ) , VData -> isKill ( ) , TII -> getNamedOperand ( * MI , <STR_LIT> :: srsrc ) -> getReg ( ) , TII -> getNamedOperand ( * MI , <STR_LIT> :: soffset ) -> getReg ( ) , TII -> getNamedOperand ( * MI , <STR_LIT> :: offset ) -> getImm ( ) , * MI -> memoperands_begin ( ) , RS ) ; MI -> eraseFromParent ( ) ; break ; } default : { const DebugLoc & DL = MI -> getDebugLoc ( ) ; bool IsMUBUF = TII -> isMUBUF ( * MI ) ; if ( ! IsMUBUF && MFI -> getFrameOffsetReg ( ) != MFI -> getScratchWaveOffsetReg ( ) ) { unsigned DiffReg = MRI . createVirtualRegister ( & <STR_LIT> ) ; bool IsCopy = MI -> getOpcode ( ) == <STR_LIT> ; unsigned ResultReg = IsCopy ? MI -> getOperand ( <NUM_LIT> ) . getReg ( ) : MRI . createVirtualRegister ( & <STR_LIT> ) ; BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) , DiffReg ) . addReg ( MFI -> getFrameOffsetReg ( ) ) . addReg ( MFI -> getScratchWaveOffsetReg ( ) ) ; int64_t Offset = FrameInfo . getObjectOffset ( Index ) ; if ( Offset == <NUM_LIT> ) { BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) , ResultReg ) . addImm ( Log2_32 ( ST . getWavefrontSize ( ) ) ) . addReg ( DiffReg ) ; } else { unsigned CarryOut = MRI . createVirtualRegister ( & <STR_LIT> ) ; unsigned ScaledReg = MRI . createVirtualRegister ( & <STR_LIT> ) ; BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) , ScaledReg ) . addReg ( DiffReg , RegState :: Kill ) . addImm ( Log2_32 ( ST . getWavefrontSize ( ) ) ) ;

BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) , ResultReg ) . addReg ( CarryOut , RegState :: Define | RegState :: Dead ) . addImm ( Offset ) . addReg ( ScaledReg , RegState :: Kill ) ;


<s> if ( LoadStoreOp == - <NUM_LIT> ) return false ; const MachineOperand * Reg = TII -> getNamedOperand ( * MI , <STR_LIT> :: vdata ) ; if ( spillVGPRtoAGPR ( ST , * MBB , MI , Index , <NUM_LIT> , Reg -> getReg ( ) , false , false ) . getInstr ( ) ) return true ;

MachineInstrBuilder NewMI = BuildMI ( * MBB , MI , DL , TII -> get ( LoadStoreOp ) ) . add ( * Reg ) . add ( * TII -> getNamedOperand ( * MI , <STR_LIT> :: srsrc ) ) . add ( * TII -> getNamedOperand ( * MI , <STR_LIT> :: soffset ) ) . addImm ( Offset ) . addImm ( <NUM_LIT> ) . addImm ( <NUM_LIT> ) . addImm ( <NUM_LIT> ) . cloneMemRefs ( * MI ) ;


<s> Register FrameReg = FrameInfo . isFixedObjectIndex ( Index ) && hasBasePointer ( SB . MF ) ? getBaseRegister ( ) : getFrameRegister ( SB . MF ) ; Align Alignment = FrameInfo . getObjectAlign ( Index ) ; MachinePointerInfo PtrInfo = MachinePointerInfo :: getFixedStack ( SB . MF , Index ) ; MachineMemOperand * MMO = SB . MF . getMachineMemOperand ( PtrInfo , IsLoad ? MachineMemOperand :: MOLoad : MachineMemOperand :: MOStore , SB . EltSize , Alignment ) ; if ( IsLoad ) {

unsigned Opc = ST . enableFlatScratch ( ) ? <STR_LIT> : <STR_LIT> ;


<s> case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : return spillSGPR ( MI , FI , RS , LIS , true , NeedsCFI ) ; case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : return restoreSGPR ( MI , FI , RS , LIS , true ) ; default : llvm_unreachable ( <STR_LIT> ) ; }


<s> const uint32_t * <ISA_LIT>RegisterInfo :: getAllAGPRRegMask ( ) const {


<s> const uint32_t * <ISA_LIT>RegisterInfo :: getAllVectorRegMask ( ) const { return CSR_<ISA_LIT>_AllVectorRegs_RegMask ;


<s> unsigned getChannelFromSubReg ( unsigned SubReg ) const {

return SubReg ? ( getSubRegIdxOffset ( SubReg ) + <NUM_LIT> ) / <NUM_LIT> : <NUM_LIT> ;


<s> const TargetRegisterClass * <ISA_LIT>RegisterInfo :: getConstrainedRegClassForOperand ( const MachineOperand & MO , const MachineRegisterInfo & MRI ) const { const RegClassOrRegBank & RCOrRB = MRI . getRegClassOrRegBank ( MO . getReg ( ) ) ; if ( const RegisterBank * RB = RCOrRB . dyn_cast < const RegisterBank * > ( ) ) return getRegClassForTypeOnBank ( MRI . getType ( MO . getReg ( ) ) , * RB , MRI ) ; const TargetRegisterClass * RC = RCOrRB . get < const TargetRegisterClass * > ( ) ;


<s> Register <ISA_LIT>RegisterInfo :: getFrameRegister ( const MachineFunction & MF ) const { const <ISA_LIT>FrameLowering * TFI = MF . getSubtarget < GCNSubtarget > ( ) . getFrameLowering ( ) ; const <ISA_LIT>MachineFunctionInfo * FuncInfo = MF . getInfo < <ISA_LIT>MachineFunctionInfo > ( ) ;

if ( FuncInfo -> isEntryFunction ( ) ) {


<s> const <ISA_LIT>MachineFunctionInfo * FuncInfo = MF . getInfo < <ISA_LIT>MachineFunctionInfo > ( ) ; if ( FuncInfo -> isEntryFunction ( ) ) { return TFI -> hasFP ( MF ) ? FuncInfo -> getFrameOffsetReg ( ) : Register ( ) ;


<s> unsigned getHWRegIndex ( MCRegister Reg ) const { return getEncodingValue ( Reg ) & <NUM_LIT> ;


<s> return getNumCoveredRegs ( getSubRegIndexLaneMask ( SubReg ) ) ;


<s> uint64_t Mask = LM . getAsInteger ( ) ; uint64_t Even = Mask & <NUM_LIT> ; Mask = ( Even >> <NUM_LIT> ) | Mask ; uint64_t Odd = Mask & <NUM_LIT> ;


<s> uint64_t Even = Mask & <NUM_LIT> ; Mask = ( Even >> <NUM_LIT> ) | Mask ; uint64_t Odd = Mask & <NUM_LIT> ; return countPopulation ( Odd ) ;


<s> case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : return <NUM_LIT> ; case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : return <NUM_LIT> ; case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : return <NUM_LIT> ; case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : return <NUM_LIT> ; case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : return <NUM_LIT> ; case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : return <NUM_LIT> ; case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : return <NUM_LIT> ; case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : return <NUM_LIT> ; case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : return <NUM_LIT> ; case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> :


<s> const TargetRegisterClass * getRegClassForTypeOnBank ( LLT Ty , const RegisterBank & Bank , const MachineRegisterInfo & MRI ) const { return getRegClassForSizeOnBank ( Ty . getSizeInBits ( ) , Bank , MRI ) ;


<s> reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; if ( isWave32 ) { Reserved . set ( <STR_LIT> ) ; Reserved . set ( <STR_LIT> ) ; } unsigned MaxNumSGPRs = ST . getMaxNumSGPRs ( MF ) ; unsigned TotalNumSGPRs = <STR_LIT> . getNumRegs ( ) ; for ( unsigned i = MaxNumSGPRs ; i < TotalNumSGPRs ; ++ i ) { unsigned Reg = <STR_LIT> . getRegister ( i ) ; reserveRegisterTuples ( Reserved , Reg ) ; } const <ISA_LIT>MachineFunctionInfo * MFI = MF . getInfo < <ISA_LIT>MachineFunctionInfo > ( ) ; unsigned MaxNumVGPRs = ST . getMaxNumVGPRs ( MF ) ; unsigned MaxNumAGPRs = MaxNumVGPRs ; unsigned TotalNumVGPRs = <STR_LIT> . getNumRegs ( ) ; if ( ST . hasGFX90AInsts ( ) ) { if ( MFI -> usesAGPRs ( MF ) ) { MaxNumVGPRs /= <NUM_LIT> ; MaxNumAGPRs = MaxNumVGPRs ; } else { if ( MaxNumVGPRs > TotalNumVGPRs ) { MaxNumAGPRs = MaxNumVGPRs - TotalNumVGPRs ; MaxNumVGPRs = TotalNumVGPRs ; } else MaxNumAGPRs = <NUM_LIT> ; } } for ( unsigned i = MaxNumVGPRs ; i < TotalNumVGPRs ; ++ i ) { unsigned Reg = <STR_LIT> . getRegister ( i ) ; reserveRegisterTuples ( Reserved , Reg ) ; } for ( unsigned i = MaxNumAGPRs ; i < TotalNumVGPRs ; ++ i ) { unsigned Reg = <STR_LIT> . getRegister ( i ) ; reserveRegisterTuples ( Reserved , Reg ) ; } for ( auto Reg : <STR_LIT> ) { Reserved . set ( getSubReg ( Reg , <STR_LIT> ) ) ; Register Low = getSubReg ( Reg , <STR_LIT> ) ; if ( ! <STR_LIT> . contains ( Low ) ) Reserved . set ( Low ) ; } for ( auto Reg : <STR_LIT> ) { Reserved . set ( getSubReg ( Reg , <STR_LIT> ) ) ; } if ( ! ST . hasMAIInsts ( ) ) { for ( unsigned i = <NUM_LIT> ; i < MaxNumVGPRs ; ++ i ) { unsigned Reg = <STR_LIT> . getRegister ( i ) ; reserveRegisterTuples ( Reserved , Reg ) ;


<s> } else if ( hasAGPRs ( RC ) ) { RC = getAGPRClassForBitWidth ( Size ) ; } else { RC = getVGPRClassForBitWidth ( Size ) ; } assert ( RC && <STR_LIT> ) ; return RC ;


<s> const TargetRegisterClass * getWaveMaskRegClass ( ) const { return isWave32 ? & <STR_LIT> : & <STR_LIT> ;


<s> const TargetRegisterClass * getWaveMaskRegClass ( ) const {


<s> static bool hasAGPRs ( const TargetRegisterClass * RC ) {


<s> static bool hasVectorRegisters ( const TargetRegisterClass * RC ) {


<s> return hasAGPRs ( RC ) && ! hasVGPRs ( RC ) ;


<s> bool <ISA_LIT>RegisterInfo :: isCFISavedRegsSpillEnabled ( ) const { return EnableSpillCFISavedRegs ;


<s> bool isDivergentRegClass ( const TargetRegisterClass * RC ) const override {


<s> bool isDivergentRegClass ( const TargetRegisterClass * RC ) const override { return ! isSGPRClass ( RC ) ;


<s> if ( hasAGPRs ( & RC ) ) return RC . hasSuperClassEq ( getAGPRClassForBitWidth ( getRegSizeInBits ( RC ) ) ) ; return true ;


<s> static bool isSGPRClass ( const TargetRegisterClass * RC ) { return ! hasVGPRs ( RC ) && ! hasAGPRs ( RC ) ;


<s> bool isVectorRegister ( const MachineRegisterInfo & MRI , Register Reg ) const { return isVGPR ( MRI , Reg ) || isAGPR ( MRI , Reg ) ;


<s> static bool isVGPRClass ( const TargetRegisterClass * RC ) { return hasVGPRs ( RC ) && ! hasAGPRs ( RC ) ;


<s> } assert ( ! SavedExecReg && <STR_LIT> ) ; const TargetRegisterClass & RC = IsWave32 ? <STR_LIT> : <STR_LIT> ; RS -> setRegUsed ( SuperReg ) ; SavedExecReg = RS -> scavengeRegister ( & RC , MI , <NUM_LIT> , false ) ; int64_t VGPRLanes = getPerVGPRData ( ) . VGPRLanes ; if ( SavedExecReg ) { RS -> setRegUsed ( SavedExecReg ) ; BuildMI ( MBB , MI , DL , TII . get ( MovOpc ) , SavedExecReg ) . addReg ( ExecReg ) ; auto I = BuildMI ( MBB , MI , DL , TII . get ( MovOpc ) , ExecReg ) . addImm ( VGPRLanes ) ; if ( ! TmpVGPRLive ) I . addReg ( TmpVGPR , RegState :: ImplicitDefine ) ; TRI . buildVGPRSpillLoadStore ( * this , TmpVGPRIndex , <NUM_LIT> , false ) ;


<s> RS -> setRegUsed ( SuperReg ) ; SavedExecReg = RS -> scavengeRegister ( & RC , MI , <NUM_LIT> , false ) ; int64_t VGPRLanes = getPerVGPRData ( ) . VGPRLanes ; if ( SavedExecReg ) { RS -> setRegUsed ( SavedExecReg ) ; BuildMI ( MBB , MI , DL , TII . get ( MovOpc ) , SavedExecReg ) . addReg ( ExecReg ) ; auto I = BuildMI ( MBB , MI , DL , TII . get ( MovOpc ) , ExecReg ) . addImm ( VGPRLanes ) ; if ( ! TmpVGPRLive ) I . addReg ( TmpVGPR , RegState :: ImplicitDefine ) ; TRI . buildVGPRSpillLoadStore ( * this , TmpVGPRIndex , <NUM_LIT> , false ) ; } else { if ( TmpVGPRLive ) TRI . buildVGPRSpillLoadStore ( * this , TmpVGPRIndex , <NUM_LIT> , false , false ) ; auto I = BuildMI ( MBB , MI , DL , TII . get ( NotOpc ) , ExecReg ) . addReg ( ExecReg ) ; if ( ! TmpVGPRLive ) I . addReg ( TmpVGPR , RegState :: ImplicitDefine ) ;


<s> if ( SavedExecReg ) { TRI . buildVGPRSpillLoadStore ( * this , Index , Offset , IsLoad ) ; } else { TRI . buildVGPRSpillLoadStore ( * this , Index , Offset , IsLoad , false ) ; BuildMI ( MBB , MI , DL , TII . get ( NotOpc ) , ExecReg ) . addReg ( ExecReg ) ; TRI . buildVGPRSpillLoadStore ( * this , Index , Offset , IsLoad ) ;


<s> } } else { TRI . buildVGPRSpillLoadStore ( * this , TmpVGPRIndex , <NUM_LIT> , true , false ) ; auto I = BuildMI ( MBB , MI , DL , TII . get ( NotOpc ) , ExecReg ) . addReg ( ExecReg ) ; if ( ! TmpVGPRLive ) { I . addReg ( TmpVGPR , RegState :: ImplicitKill ) ; } if ( TmpVGPRLive ) TRI . buildVGPRSpillLoadStore ( * this , TmpVGPRIndex , <NUM_LIT> , true ) ; }


<s> if ( SpillToVGPR ) { for ( unsigned i = <NUM_LIT> , e = SB . NumSubRegs ; i < e ; ++ i ) { Register SubReg = SB . NumSubRegs == <NUM_LIT> ? SB . SuperReg : Register ( getSubReg ( SB . SuperReg , SB . SplitParts [ i ] ) ) ; <ISA_LIT>MachineFunctionInfo :: SpilledReg Spill = VGPRSpills [ i ] ; auto MIB = BuildMI ( SB . MBB , MI , SB . DL , SB . TII . get ( <STR_LIT> ) , SubReg ) . addReg ( Spill . VGPR ) . addImm ( Spill . Lane ) ; if ( SB . NumSubRegs > <NUM_LIT> && i == <NUM_LIT> ) MIB . addReg ( SB . SuperReg , RegState :: ImplicitDefine ) ; if ( LIS ) { if ( i == e - <NUM_LIT> ) LIS -> ReplaceMachineInstrInMaps ( * MI , * MIB ) ; else LIS -> InsertMachineInstrInMaps ( * MIB ) ; } } } else { SB . prepare ( ) ; auto PVD = SB . getPerVGPRData ( ) ; for ( unsigned Offset = <NUM_LIT> ; Offset < PVD . NumVGPRs ; ++ Offset ) { SB . readWriteTmpVGPR ( Offset , true ) ; for ( unsigned i = Offset * PVD . PerVGPR , e = std :: min ( ( Offset + <NUM_LIT> ) * PVD . PerVGPR , SB . NumSubRegs ) ; i < e ; ++ i ) { Register SubReg = SB . NumSubRegs == <NUM_LIT> ? SB . SuperReg : Register ( getSubReg ( SB . SuperReg , SB . SplitParts [ i ] ) ) ; bool LastSubReg = ( i + <NUM_LIT> == e ) ; auto MIB = BuildMI ( SB . MBB , MI , SB . DL , SB . TII . get ( <STR_LIT> ) , SubReg ) . addReg ( SB . TmpVGPR , getKillRegState ( LastSubReg ) ) . addImm ( i ) ; if ( SB . NumSubRegs > <NUM_LIT> && i == <NUM_LIT> ) MIB . addReg ( SB . SuperReg , RegState :: ImplicitDefine ) ; if ( LIS ) { if ( i == e - <NUM_LIT> ) LIS -> ReplaceMachineInstrInMaps ( * MI , * MIB ) ; else LIS -> InsertMachineInstrInMaps ( * MIB ) ; } } }


<s> if ( Vec . empty ( ) ) { unsigned MaxNumParts = <NUM_LIT> / Size ; Vec . resize ( MaxNumParts ) ; } Vec [ Pos ] = Idx ; } } ; static llvm :: once_flag InitializeSubRegFromChannelTableFlag ; static auto InitializeSubRegFromChannelTableOnce = [ this ] ( ) { for ( auto & Row : SubRegFromChannelTable ) Row . fill ( <STR_LIT> ) ; for ( uint16_t Idx = <NUM_LIT> ; Idx < getNumSubRegIndices ( ) ; ++ Idx ) { unsigned Width = <ISA_LIT>SubRegIdxRanges [ Idx ] . Size / <NUM_LIT> ; unsigned Offset = <ISA_LIT>SubRegIdxRanges [ Idx ] . Offset / <NUM_LIT> ; assert ( Width < SubRegFromChannelTableWidthMap . size ( ) ) ; Width = SubRegFromChannelTableWidthMap [ Width ] ; if ( Width == <NUM_LIT> ) continue ; unsigned TableIdx = Width - <NUM_LIT> ; assert ( TableIdx < SubRegFromChannelTable . size ( ) ) ;

assert ( Offset < SubRegFromChannelTable [ TableIdx ] . size ( ) ) ;


<s> auto MIB = BuildMI ( SB . MBB , MI , SB . DL , SB . TII . get ( <STR_LIT> ) , Spill . VGPR ) . addReg ( SubReg , getKillRegState ( UseKill ) ) . addImm ( Spill . Lane ) . addReg ( Spill . VGPR ) ; MachineInstr * CFI = nullptr ; if ( NeedsCFI ) { CFI = TFL -> buildCFIForSGPRToVGPRSpill ( SB . MBB , MI , DebugLoc ( ) , SubReg , Spill . VGPR , Spill . Lane ) ; } if ( LIS ) { if ( i == <NUM_LIT> ) LIS -> ReplaceMachineInstrInMaps ( * MI , * MIB ) ; else LIS -> InsertMachineInstrInMaps ( * MIB ) ; if ( CFI ) LIS -> InsertMachineInstrInMaps ( * CFI ) ; } if ( i == <NUM_LIT> && SB . NumSubRegs > <NUM_LIT> ) { MIB . addReg ( SB . SuperReg , RegState :: ImplicitDefine ) ; } if ( SB . NumSubRegs > <NUM_LIT> ) MIB . addReg ( SB . SuperReg , getKillRegState ( UseKill ) | RegState :: Implicit ) ; } } else { SB . prepare ( ) ; unsigned SubKillState = getKillRegState ( ( SB . NumSubRegs == <NUM_LIT> ) && SB . IsKill ) ; auto PVD = SB . getPerVGPRData ( ) ; for ( unsigned Offset = <NUM_LIT> ; Offset < PVD . NumVGPRs ; ++ Offset ) { unsigned TmpVGPRFlags = RegState :: Undef ; for ( unsigned i = Offset * PVD . PerVGPR , e = std :: min ( ( Offset + <NUM_LIT> ) * PVD . PerVGPR , SB . NumSubRegs ) ; i < e ; ++ i ) { Register SubReg = SB . NumSubRegs == <NUM_LIT> ? SB . SuperReg : Register ( getSubReg ( SB . SuperReg , SB . SplitParts [ i ] ) ) ; MachineInstrBuilder WriteLane = BuildMI ( SB . MBB , MI , SB . DL , SB . TII . get ( <STR_LIT> ) , SB . TmpVGPR ) . addReg ( SubReg , SubKillState ) . addImm ( i % PVD . PerVGPR ) . addReg ( SB . TmpVGPR , TmpVGPRFlags ) ; TmpVGPRFlags = <NUM_LIT> ; if ( LIS ) { if ( i == <NUM_LIT> ) LIS -> ReplaceMachineInstrInMaps ( * MI , * WriteLane ) ; else LIS -> InsertMachineInstrInMaps ( * WriteLane ) ; } if ( SB . NumSubRegs > <NUM_LIT> ) { unsigned SuperKillState = <NUM_LIT> ; if ( i + <NUM_LIT> == SB . NumSubRegs ) SuperKillState |= getKillRegState ( SB . IsKill ) ; WriteLane . addReg ( SB . SuperReg , RegState :: Implicit | SuperKillState ) ; } } SB . readWriteTmpVGPR ( Offset , false ) ; } SB . restore ( ) ; } MI -> eraseFromParent ( ) ; SB . MFI . addToSpilledSGPRs ( SB . NumSubRegs ) ;

if ( LIS ) LIS -> removeAllRegUnitsForPhysReg ( SB . SuperReg ) ;


<s> MCPhysReg Reg = MFI -> getVGPRToAGPRSpill ( Index , Lane ) ; if ( Reg == <STR_LIT> ) return MachineInstrBuilder ( ) ; bool IsStore = MI -> mayStore ( ) ; MachineRegisterInfo & MRI = MF -> getRegInfo ( ) ; auto * TRI = static_cast < const <ISA_LIT>RegisterInfo * > ( MRI . getTargetRegisterInfo ( ) ) ; unsigned Dst = IsStore ? Reg : ValueReg ;

unsigned Src = IsStore ? ValueReg : Reg ;


<s> Register FrameReg = getFrameRegister ( * MF ) ; switch ( MI -> getOpcode ( ) ) { case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : { spillSGPR ( MI , Index , RS ) ; break ; } case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : { restoreSGPR ( MI , Index , RS ) ; break ; } case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : { const MachineOperand * VData = TII -> getNamedOperand ( * MI , <STR_LIT> :: vdata ) ; assert ( TII -> getNamedOperand ( * MI , <STR_LIT> :: soffset ) -> getReg ( ) == MFI -> getStackPtrOffsetReg ( ) ) ; buildSpillLoadStore ( MI , <STR_LIT> , Index , VData -> getReg ( ) , VData -> isKill ( ) , TII -> getNamedOperand ( * MI , <STR_LIT> :: srsrc ) -> getReg ( ) , FrameReg , TII -> getNamedOperand ( * MI , <STR_LIT> :: offset ) -> getImm ( ) , * MI -> memoperands_begin ( ) , RS ) ; MFI -> addToSpilledVGPRs ( getNumSubRegsForSpillOp ( MI -> getOpcode ( ) ) ) ; MI -> eraseFromParent ( ) ; break ; } case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : { const MachineOperand * VData = TII -> getNamedOperand ( * MI , <STR_LIT> :: vdata ) ; assert ( TII -> getNamedOperand ( * MI , <STR_LIT> :: soffset ) -> getReg ( ) == MFI -> getStackPtrOffsetReg ( ) ) ; buildSpillLoadStore ( MI , <STR_LIT> , Index , VData -> getReg ( ) , VData -> isKill ( ) , TII -> getNamedOperand ( * MI , <STR_LIT> :: srsrc ) -> getReg ( ) , FrameReg , TII -> getNamedOperand ( * MI , <STR_LIT> :: offset ) -> getImm ( ) , * MI -> memoperands_begin ( ) , RS ) ; MI -> eraseFromParent ( ) ; break ; } default : { const DebugLoc & DL = MI -> getDebugLoc ( ) ; bool IsMUBUF = TII -> isMUBUF ( * MI ) ; if ( ! IsMUBUF && ! MFI -> isEntryFunction ( ) ) { unsigned DiffReg = MRI . createVirtualRegister ( & <STR_LIT> ) ; bool IsCopy = MI -> getOpcode ( ) == <STR_LIT> ; Register ResultReg = IsCopy ? MI -> getOperand ( <NUM_LIT> ) . getReg ( ) : MRI . createVirtualRegister ( & <STR_LIT> ) ; BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) , DiffReg ) . addReg ( FrameReg ) . addReg ( MFI -> getScratchWaveOffsetReg ( ) ) ; int64_t Offset = FrameInfo . getObjectOffset ( Index ) ; if ( Offset == <NUM_LIT> ) {

BuildMI ( * MBB , MI , DL , TII -> get ( <STR_LIT> ) , ResultReg ) . addImm ( Log2_32 ( ST . getWavefrontSize ( ) ) ) . addReg ( DiffReg ) ;


<s> if ( Reg >= BeginReg && Reg <= EndReg ) { \ unsigned Index = Reg - BeginReg ; \ assert ( Index < array_lengthof ( RegTable ) ) ; \ return RegTable [ Index ] ; \ } REG_RANGE ( <STR_LIT> , <STR_LIT> , VGPR32RegNames ) ; REG_RANGE ( <STR_LIT> , <STR_LIT> , SGPR32RegNames ) ; REG_RANGE ( <STR_LIT> , <STR_LIT> , AGPR32RegNames ) ; REG_RANGE ( <STR_LIT> , <STR_LIT> , VGPR64RegNames ) ; REG_RANGE ( <STR_LIT> , <STR_LIT> , SGPR64RegNames ) ; REG_RANGE ( <STR_LIT> , <STR_LIT> , AGPR64RegNames ) ; REG_RANGE ( <STR_LIT> , <STR_LIT> , VGPR96RegNames ) ;


<s> llvm_unreachable ( <STR_LIT> ) ; } } if ( EltSize == <NUM_LIT> ) { static const int16_t Sub0_15_64 [ ] = { <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> } ; static const int16_t Sub0_7_64 [ ] = { <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> } ; static const int16_t Sub0_3_64 [ ] = { <STR_LIT> , <STR_LIT> } ; switch ( <STR_LIT> ( * RC -> MC ) ) { case <NUM_LIT> : return { } ; case <NUM_LIT> : return makeArrayRef ( Sub0_3_64 ) ; case <NUM_LIT> : return makeArrayRef ( Sub0_7_64 ) ; case <NUM_LIT> : return makeArrayRef ( Sub0_15_64 ) ; default : llvm_unreachable ( <STR_LIT> ) ; } } if ( EltSize == <NUM_LIT> ) { static const int16_t Sub0_15_128 [ ] = { <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> } ; static const int16_t Sub0_7_128 [ ] = { <STR_LIT> , <STR_LIT> } ; switch ( <STR_LIT> ( * RC -> MC ) ) { case <NUM_LIT> : return { } ; case <NUM_LIT> : return makeArrayRef ( Sub0_7_128 ) ; case <NUM_LIT> : return makeArrayRef ( Sub0_15_128 ) ;


<s> reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; if ( isWave32 ) { Reserved . set ( <STR_LIT> ) ; Reserved . set ( <STR_LIT> ) ; } const GCNSubtarget & ST = MF . getSubtarget < GCNSubtarget > ( ) ; unsigned MaxNumSGPRs = ST . getMaxNumSGPRs ( MF ) ; unsigned TotalNumSGPRs = <STR_LIT> . getNumRegs ( ) ; for ( unsigned i = MaxNumSGPRs ; i < TotalNumSGPRs ; ++ i ) { unsigned Reg = <STR_LIT> . getRegister ( i ) ; reserveRegisterTuples ( Reserved , Reg ) ; } unsigned MaxNumVGPRs = ST . getMaxNumVGPRs ( MF ) ; unsigned TotalNumVGPRs = <STR_LIT> . getNumRegs ( ) ;

for ( unsigned i = MaxNumVGPRs ; i < TotalNumVGPRs ; ++ i ) {


<s> Reserved . set ( <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; if ( MF . getSubtarget < <ISA_LIT>Subtarget > ( ) . hasSGPRInitBug ( ) ) { unsigned NumSGPRs = <STR_LIT> . getNumRegs ( ) ; unsigned Limit = <ISA_LIT>Subtarget :: FIXED_SGPR_COUNT_FOR_INIT_BUG - <NUM_LIT> ; for ( unsigned i = Limit ; i < NumSGPRs ; ++ i ) {

unsigned Reg = <STR_LIT> . getRegister ( i ) ;


<s> reserveRegisterTuples ( Reserved , <STR_LIT> ) ; reserveRegisterTuples ( Reserved , <STR_LIT> ) ; if ( MF . getSubtarget < <ISA_LIT>Subtarget > ( ) . hasSGPRInitBug ( ) ) { unsigned NumSGPRs = <STR_LIT> . getNumRegs ( ) ;


<s> const TargetRegisterClass * RC ; if ( Reg . isVirtual ( ) ) RC = MRI . getRegClass ( Reg ) ;


<s> const MCPhysReg * <ISA_LIT>RegisterInfo :: getCalleeSavedRegs ( const MachineFunction * MF ) const { CallingConv :: ID CC = MF -> getFunction ( ) . getCallingConv ( ) ; switch ( CC ) { case CallingConv :: C : case CallingConv :: Fast :


<s> case CallingConv :: Fast : case CallingConv :: Cold :


<s> const uint32_t * <ISA_LIT>RegisterInfo :: getCallPreservedMask ( const MachineFunction & MF , CallingConv :: ID CC ) const { switch ( CC ) { case CallingConv :: C : case CallingConv :: Fast : case CallingConv :: Cold :


<s> unsigned getSGPRAllocGranule ( ) const {


<s> unsigned getTotalNumVGPRs ( ) const { return <NUM_LIT> ;


<s> if ( SavedExecReg ) { RS -> setRegUsed ( SavedExecReg ) ; BuildMI ( * MBB , MI , DL , TII . get ( MovOpc ) , SavedExecReg ) . addReg ( ExecReg ) ; auto I = BuildMI ( * MBB , MI , DL , TII . get ( MovOpc ) , ExecReg ) . addImm ( VGPRLanes ) ; if ( ! TmpVGPRLive ) I . addReg ( TmpVGPR , RegState :: ImplicitDefine ) ; TRI . buildVGPRSpillLoadStore ( * this , TmpVGPRIndex , <NUM_LIT> , false ) ; } else { if ( TmpVGPRLive ) TRI . buildVGPRSpillLoadStore ( * this , TmpVGPRIndex , <NUM_LIT> , false , false ) ; auto I = BuildMI ( * MBB , MI , DL , TII . get ( NotOpc ) , ExecReg ) . addReg ( ExecReg ) ; if ( ! TmpVGPRLive ) I . addReg ( TmpVGPR , RegState :: ImplicitDefine ) ; TRI . buildVGPRSpillLoadStore ( * this , TmpVGPRIndex , <NUM_LIT> , false ) ;


<s> void prepare ( ) { assert ( RS && <STR_LIT> ) ; TmpVGPR = RS -> scavengeRegister ( & <STR_LIT> , MI , <NUM_LIT> , false ) ; TmpVGPRIndex = MFI . getScavengeFI ( MF . getFrameInfo ( ) , TRI ) ; if ( TmpVGPR ) { TmpVGPRLive = false ; } else { TmpVGPR = <STR_LIT> ; TmpVGPRLive = true ; } assert ( ! SavedExecReg && <STR_LIT> ) ; const TargetRegisterClass & RC = IsWave32 ? <STR_LIT> : <STR_LIT> ; RS -> setRegUsed ( SuperReg ) ; SavedExecReg = RS -> scavengeRegister ( & RC , MI , <NUM_LIT> , false ) ; int64_t VGPRLanes = getPerVGPRData ( ) . VGPRLanes ; if ( SavedExecReg ) { RS -> setRegUsed ( SavedExecReg ) ; BuildMI ( * MBB , MI , DL , TII . get ( MovOpc ) , SavedExecReg ) . addReg ( ExecReg ) ; auto I = BuildMI ( * MBB , MI , DL , TII . get ( MovOpc ) , ExecReg ) . addImm ( VGPRLanes ) ; if ( ! TmpVGPRLive ) I . addReg ( TmpVGPR , RegState :: ImplicitDefine ) ; TRI . buildVGPRSpillLoadStore ( * this , TmpVGPRIndex , <NUM_LIT> , false ) ; } else { if ( TmpVGPRLive ) TRI . buildVGPRSpillLoadStore ( * this , TmpVGPRIndex , <NUM_LIT> , false , false ) ;

auto I = BuildMI ( * MBB , MI , DL , TII . get ( NotOpc ) , ExecReg ) . addReg ( ExecReg ) ;


<s> void readWriteTmpVGPR ( unsigned Offset , bool IsLoad ) { if ( SavedExecReg ) { TRI . buildVGPRSpillLoadStore ( * this , Index , Offset , IsLoad ) ; } else {

TRI . buildVGPRSpillLoadStore ( * this , Index , Offset , IsLoad , false ) ;


<s> } else { TRI . buildVGPRSpillLoadStore ( * this , Index , Offset , IsLoad , false ) ;

BuildMI ( * MBB , MI , DL , TII . get ( NotOpc ) , ExecReg ) . addReg ( ExecReg ) ;


<s> if ( ! TmpVGPRLive ) { I . addReg ( TmpVGPR , RegState :: ImplicitKill ) ; } } else { TRI . buildVGPRSpillLoadStore ( * this , TmpVGPRIndex , <NUM_LIT> , true , false ) ;


<s> void restore ( ) { if ( SavedExecReg ) { TRI . buildVGPRSpillLoadStore ( * this , TmpVGPRIndex , <NUM_LIT> , true , false ) ; auto I = BuildMI ( * MBB , MI , DL , TII . get ( MovOpc ) , ExecReg ) . addReg ( SavedExecReg , RegState :: Kill ) ; if ( ! TmpVGPRLive ) { I . addReg ( TmpVGPR , RegState :: ImplicitKill ) ; } } else { TRI . buildVGPRSpillLoadStore ( * this , TmpVGPRIndex , <NUM_LIT> , true , false ) ; auto I = BuildMI ( * MBB , MI , DL , TII . get ( NotOpc ) , ExecReg ) . addReg ( ExecReg ) ; if ( ! TmpVGPRLive ) { I . addReg ( TmpVGPR , RegState :: ImplicitKill ) ; }

if ( TmpVGPRLive ) TRI . buildVGPRSpillLoadStore ( * this , TmpVGPRIndex , <NUM_LIT> , true ) ;


<s> MachineRegisterInfo & MRI = MF -> getRegInfo ( ) ; auto * TRI = static_cast < const <ISA_LIT>RegisterInfo * > ( MRI . getTargetRegisterInfo ( ) ) ; unsigned Dst = IsStore ? Reg : ValueReg ; unsigned Src = IsStore ? ValueReg : Reg ;

unsigned Opc = ( IsStore ^ TRI -> isVGPR ( MRI , Reg ) ) ? <STR_LIT> : <STR_LIT> ;


<s> if ( Reg == <STR_LIT> ) return MachineInstrBuilder ( ) ; bool IsStore = MI -> mayStore ( ) ; MachineRegisterInfo & MRI = MF -> getRegInfo ( ) ; auto * TRI = static_cast < const <ISA_LIT>RegisterInfo * > ( MRI . getTargetRegisterInfo ( ) ) ; unsigned Dst = IsStore ? Reg : ValueReg ; unsigned Src = IsStore ? ValueReg : Reg ; unsigned Opc = ( IsStore ^ TRI -> isVGPR ( MRI , Reg ) ) ? <STR_LIT> : <STR_LIT> ;

auto MIB = BuildMI ( MBB , MI , MI -> getDebugLoc ( ) , TII -> get ( Opc ) , Dst ) . addReg ( Src , getKillRegState ( IsKill ) ) ;