<s> if ( isInlineConstantIfFolded ( TII , * UseMI , OpNo , OpToFold ) ) { foldOperand ( OpToFold , UseMI , OpNo , FoldList , CopiesToReplace ) ; } else { if ( ++ NumLiteralUses == <NUM_LIT> ) { NonInlineUse = & * Use ; NonInlineUseOpNo = OpNo ; } } } if ( NumLiteralUses == <NUM_LIT> ) { MachineInstr * UseMI = NonInlineUse -> getParent ( ) ; foldOperand ( OpToFold , UseMI , NonInlineUseOpNo , FoldList , CopiesToReplace ) ; } } else { for ( MachineRegisterInfo :: use_iterator Use = MRI -> use_begin ( Dst . getReg ( ) ) , E = MRI -> use_end ( ) ; Use != E ; ++ Use ) { MachineInstr * UseMI = Use -> getParent ( ) ; foldOperand ( OpToFold , UseMI , Use . getOperandNo ( ) , FoldList , CopiesToReplace ) ; } } MachineFunction * MF = MI . getParent


<s> for ( MachineRegisterInfo :: use_iterator Use = MRI -> use_begin ( Dst . getReg ( ) ) , E = MRI -> use_end ( ) ; Use != E ; Use = NextUse ) { NextUse = std :: next ( Use ) ; MachineInstr * UseMI = Use -> getParent ( ) ; unsigned OpNo = Use . getOperandNo ( ) ; if ( OpToFold . isImm ( ) && tryConstantFoldOp ( * MRI , TII , UseMI , & OpToFold ) ) { LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << * UseMI << '\n' ) ; NextUse = MRI -> use_begin ( Dst . getReg ( ) ) ; FoldList . clear ( ) ; continue ; } if ( isInlineConstantIfFolded ( TII , * UseMI , OpNo , OpToFold ) ) { foldOperand ( OpToFold , UseMI , OpNo , FoldList , CopiesToReplace ) ; } else { if ( ++ NumLiteralUses == <NUM_LIT> ) { NonInlineUse = & * Use ; NonInlineUseOpNo = OpNo ; } } } if ( NumLiteralUses == <NUM_LIT> ) { MachineInstr * UseMI = NonInlineUse -> getParent ( ) ; foldOperand ( OpToFold , UseMI , NonInlineUseOpNo , FoldList , CopiesToReplace ) ; } } else { for ( MachineRegisterInfo :: use_iterator Use = MRI -> use_begin ( Dst . getReg ( ) ) , E = MRI -> use_end ( ) ; Use != E ; ++ Use ) { MachineInstr * UseMI = Use -> getParent ( ) ; foldOperand ( OpToFold , UseMI , Use . getOperandNo ( ) , FoldList , CopiesToReplace ) ; } } MachineFunction * MF = MI . getParent ( ) -> getParent ( ) ; for ( MachineInstr * Copy : CopiesToReplace )

Copy -> addImplicitDefUseOperands ( * MF ) ;


<s> static int getOModValue ( unsigned Opc , int64_t Val ) { switch ( Opc ) { case <STR_LIT> : { switch ( static_cast < uint32_t > ( Val ) ) { case <NUM_LIT> : return <STR_LIT> ; case <NUM_LIT> : return <STR_LIT> ; case <NUM_LIT> : return <STR_LIT> ; default : return <STR_LIT> ; } } case <STR_LIT> : { switch ( static_cast < uint16_t > ( Val ) ) { case <NUM_LIT> : return <STR_LIT> ; case <NUM_LIT> : return <STR_LIT> ; case


<s> static bool hasOneNonDBGUseInst ( const MachineRegisterInfo & MRI , unsigned Reg ) { int Count = <NUM_LIT> ; for ( auto I = MRI . use_instr_nodbg_begin ( Reg ) , E = MRI . use_instr_nodbg_end ( ) ; I != E ; ++ I ) { if ( ++ Count > <NUM_LIT> ) return false ; } return true


<s> unsigned Op = MI . getOpcode ( ) ; switch ( Op ) { case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : { if ( ! TII -> getNamedOperand ( MI , <STR_LIT> :: clamp ) -> getImm ( ) ) return nullptr ; const MachineOperand * Src0 = TII -> getNamedOperand ( MI , <STR_LIT> :: src0 ) ; const MachineOperand * Src1 = TII -> getNamedOperand ( MI , <STR_LIT> :: src1 ) ; if ( ! Src0 -> isReg ( ) || ! Src1 -> isReg ( ) || Src0 -> getReg ( ) != Src1 -> getReg ( ) || Src0 -> getSubReg ( ) != Src1 -> getSubReg ( ) || Src0 -> getSubReg ( ) != <STR_LIT> ) return nullptr ; if ( TII -> hasModifiersSet ( MI , <STR_LIT> :: omod ) ) return nullptr ; unsigned Src0Mods = TII -> getNamedOperand ( MI , <STR_LIT> :: src0_modifiers ) -> getImm ( ) ; unsigned Src1Mods = TII -> getNamedOperand ( MI ,

<STR_LIT> :: src1_modifiers ) -> getImm ( ) ;


<s> bool isCommuted ( ) const { return Commuted


<s> bool isFI ( ) const { return Kind == MachineOperand


<s> case <STR_LIT> : case <STR_LIT> : { int Src2Idx = <STR_LIT> ( Opc , <STR_LIT> :: src2 ) ; if ( static_cast < int > ( OpNo ) == Src2Idx ) { bool IsFMA = Opc == <STR_LIT> ; bool IsF32 = Opc == <STR_LIT> ; unsigned Opc = IsFMA ? <STR_LIT> : ( IsF32 ? <STR_LIT> : <STR_LIT> ) ; const MCInstrDesc & MadDesc = TII -> get ( Opc ) ; return TII -> isInlineConstant ( OpToFold , MadDesc . OpInfo [ OpNo ] . OperandType ) ; } return


<s> std :: pair < const MachineOperand * , int > <ISA_LIT>FoldOperands :: isOMod ( const MachineInstr & MI ) const { unsigned Op = MI . getOpcode ( ) ; switch ( Op ) { case <STR_LIT> : case <STR_LIT> : { if ( ( Op == <STR_LIT> && ST -> hasFP32Denormals ( ) ) || ( Op == <STR_LIT> && ST -> hasFP16Denormals ( ) ) ) return std :: make_pair ( nullptr , <STR_LIT> ) ; const MachineOperand * RegOp = nullptr ; const MachineOperand * ImmOp = nullptr ; const MachineOperand * Src0 = TII -> getNamedOperand ( MI , <STR_LIT> :: src0 ) ; const MachineOperand * Src1 = TII -> getNamedOperand ( MI , <STR_LIT> :: src1 ) ; if ( Src0 -> isImm ( ) ) { ImmOp = Src0 ; RegOp = Src1 ; } else if ( Src1 -> isImm ( ) ) { ImmOp = Src1 ; RegOp = Src0 ; } else return std :: make_pair ( nullptr , <STR_LIT> ) ; int OMod = getOModValue ( Op , ImmOp -> getImm ( ) ) ; if ( OMod == <STR_LIT> || TII -> hasModifiersSet ( MI , <STR_LIT> :: src0_modifiers ) || TII -> hasModifiersSet ( MI , <STR_LIT> :: src1_modifiers ) || TII -> hasModifiersSet ( MI , <STR_LIT> :: omod ) || TII -> hasModifiersSet ( MI , <STR_LIT> :: clamp ) ) return std :: make_pair ( nullptr , <STR_LIT> ) ; return std :: make_pair ( RegOp , OMod ) ; } case <STR_LIT> : case <STR_LIT> :


<s> static bool isUseMIInFoldList ( ArrayRef < FoldCandidate > FoldList , const MachineInstr * MI ) { for ( auto Candidate : FoldList


<s> static bool isUseMIInFoldList ( ArrayRef < FoldCandidate > FoldList , const MachineInstr * MI ) { for ( auto Candidate : FoldList ) { if ( Candidate . UseMI


<s> static void mutateCopyOp ( MachineInstr & MI , const MCInstrDesc & NewDesc ) { MI . setDesc (


<s> static void mutateCopyOp ( MachineInstr & MI , const MCInstrDesc & NewDesc


<s> TRI = & TII -> getRegisterInfo ( ) ; const <ISA_LIT>MachineFunctionInfo * MFI = MF . getInfo < <ISA_LIT>MachineFunctionInfo > ( ) ; bool IsIEEEMode = ST -> enableIEEEBit ( MF ) || ! MFI -> hasNoSignedZerosFPMath ( ) ; for ( MachineBasicBlock * MBB : depth_first ( & MF ) ) { MachineBasicBlock :: iterator I , Next ; for ( I = MBB -> begin ( ) ; I != MBB -> end ( ) ; I = Next ) { Next = std :: next ( I ) ; MachineInstr & MI = * I ; tryFoldInst ( TII , & MI ) ; if ( ! TII -> isFoldableCopy ( MI ) ) { if ( IsIEEEMode || ! tryFoldOMod ( MI ) ) tryFoldClamp ( MI ) ; continue ; } MachineOperand & OpToFold = MI . getOperand ( <NUM_LIT> ) ; bool FoldingImm = OpToFold . isImm ( ) || OpToFold . isFI ( ) ; if ( ! FoldingImm && ! OpToFold . isReg ( ) ) continue ; if ( OpToFold . isReg ( ) && ! TargetRegisterInfo ::

isVirtualRegister ( OpToFold . getReg ( ) ) ) continue ;


<s> MachineBasicBlock :: iterator I , Next ; for ( I = MBB -> begin ( ) ; I != MBB -> end ( ) ; I = Next ) { Next = std :: next ( I ) ; MachineInstr & MI = * I ; tryFoldInst ( TII , & MI ) ; if ( ! TII -> isFoldableCopy ( MI ) ) { if ( IsIEEEMode || ! tryFoldOMod ( MI ) ) tryFoldClamp ( MI ) ; continue ; } MachineOperand & OpToFold = MI . getOperand ( <NUM_LIT> ) ; bool FoldingImm = OpToFold . isImm ( ) || OpToFold . isFI ( ) ; if ( ! FoldingImm && ! OpToFold . isReg ( ) ) continue ; if ( OpToFold . isReg ( ) && ! TargetRegisterInfo :: isVirtualRegister ( OpToFold


<s> const MCInstrDesc & Desc = MI . getDesc ( ) ; unsigned NumOps = Desc . getNumOperands ( ) + Desc . getNumImplicitUses ( ) + Desc .


<s> bool FoldAsMAD = tryAddToFoldList ( FoldList , MI , OpNo , OpToFold , TII ) ; if ( FoldAsMAD ) { MI -> untieRegOperand ( OpNo ) ; return true ; } MI -> setDesc ( TII -> get ( Opc ) ) ; } if ( Opc == <STR_LIT> && OpToFold -> isImm ( ) ) { MI -> setDesc ( TII -> get ( <STR_LIT> ) ) ; FoldList . push_back ( FoldCandidate ( MI , OpNo , OpToFold ) ) ; return true ; } if ( isUseMIInFoldList ( FoldList , MI ) ) return false ; unsigned CommuteIdx0 = TargetInstrInfo :: CommuteAnyOperandIndex ; unsigned CommuteIdx1 = TargetInstrInfo :: CommuteAnyOperandIndex ; bool CanCommute = TII -> findCommutedOpIndices ( * MI , CommuteIdx0 , CommuteIdx1 ) ; if ( CanCommute ) { if ( CommuteIdx0 == OpNo ) OpNo = CommuteIdx1 ; else if ( CommuteIdx1 == OpNo ) OpNo = CommuteIdx0 ; } if ( CanCommute && ( ! MI -> getOperand ( CommuteIdx0 ) . isReg ( ) || ! MI -> getOperand ( CommuteIdx1 ) . isReg ( ) ) ) return false ; if ( ! CanCommute || ! TII -> commuteInstruction ( * MI , false , CommuteIdx0 , CommuteIdx1 ) ) return false ; if ( ! TII -> isOperandLegal ( * MI , OpNo , OpToFold ) ) { TII -> commuteInstruction ( * MI , false , CommuteIdx0 , CommuteIdx1 ) ; return false ; } FoldList . push_back ( FoldCandidate ( MI ,


<s> if ( FoldAsMAD ) { MI -> untieRegOperand ( OpNo ) ; return true ; } MI -> setDesc ( TII -> get ( Opc ) ) ; } if ( Opc == <STR_LIT> && OpToFold -> isImm ( ) ) { MI -> setDesc ( TII -> get ( <STR_LIT> ) ) ; FoldList . push_back ( FoldCandidate ( MI , OpNo , OpToFold ) ) ; return true ; } if ( isUseMIInFoldList ( FoldList , MI ) ) return false ; unsigned CommuteIdx0 = TargetInstrInfo :: CommuteAnyOperandIndex ; unsigned CommuteIdx1 = TargetInstrInfo :: CommuteAnyOperandIndex ; bool CanCommute = TII -> findCommutedOpIndices ( * MI , CommuteIdx0 , CommuteIdx1 ) ; if ( CanCommute ) { if ( CommuteIdx0 == OpNo ) OpNo = CommuteIdx1 ; else if ( CommuteIdx1 == OpNo ) OpNo = CommuteIdx0 ; } if ( CanCommute && ( ! MI -> getOperand ( CommuteIdx0 ) . isReg ( ) || ! MI -> getOperand ( CommuteIdx1 ) . isReg ( ) ) ) return false ; if ( ! CanCommute || ! TII -> commuteInstruction ( * MI , false , CommuteIdx0 , CommuteIdx1 ) ) return false ; if ( ! TII -> isOperandLegal ( * MI , OpNo , OpToFold ) ) { TII -> commuteInstruction ( * MI ,


<s> MI -> setDesc ( TII -> get ( UseCopy ? <STR_LIT> : <STR_LIT> ) ) ; return true ; } } if ( Src0 -> isImm ( ) && Src1 -> isImm ( ) ) { int32_t NewImm ; if ( ! evalBinaryInstruction ( Opc , NewImm , Src0 -> getImm ( ) , Src1 -> getImm ( ) ) ) return false ; const <ISA_LIT>RegisterInfo & TRI = TII -> getRegisterInfo ( ) ; bool IsSGPR = TRI . isSGPRReg ( MRI , MI -> getOperand ( <NUM_LIT> ) . getReg ( ) ) ; MI -> getOperand ( Src0Idx ) . ChangeToImmediate ( NewImm ) ; MI -> RemoveOperand ( Src1Idx ) ; mutateCopyOp ( * MI , TII -> get ( getMovOpc ( IsSGPR ) ) ) ; return true ; } if ( ! MI -> isCommutable ( ) ) return false ; if ( Src0 -> isImm ( ) && ! Src1 -> isImm ( ) ) { std :: swap ( Src0 , Src1 ) ; std :: swap ( Src0Idx , Src1Idx ) ; } int32_t Src1Val = static_cast < int32_t > ( Src1 -> getImm ( ) ) ; if ( Opc == <STR_LIT> || Opc == <STR_LIT> || Opc == <STR_LIT> ) { if ( Src1Val == <NUM_LIT> ) { MI -> RemoveOperand ( Src1Idx ) ; mutateCopyOp ( * MI , TII -> get ( <STR_LIT> ) ) ; } else if ( Src1Val == - <NUM_LIT> ) { MI -> RemoveOperand ( Src1Idx ) ; mutateCopyOp ( * MI , TII -> get ( getMovOpc ( Opc == <STR_LIT> ) ) ) ; } else return false ; return true ; } if ( MI -> getOpcode ( ) == <STR_LIT> || MI -> getOpcode ( ) == <STR_LIT> || MI -> getOpcode ( ) == <STR_LIT> ) { if ( Src1Val == <NUM_LIT> ) { MI -> RemoveOperand ( Src0Idx ) ; mutateCopyOp ( * MI , TII -> get ( getMovOpc ( Opc


<s> LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << * DefClamp << <STR_LIT> << * Def << '\n' ) ; DefClamp -> setImm ( <NUM_LIT> ) ; MRI -> replaceRegWith ( MI . getOperand ( <NUM_LIT> ) . getReg ( ) , Def -> getOperand ( <NUM_LIT> ) . getReg ( ) ) ; MI . eraseFromParent ( ) ; return true


<s> unsigned Opc = MI -> getOpcode ( ) ; if ( Opc == <STR_LIT> || Opc == <STR_LIT> || Opc == <STR_LIT> ) { const MachineOperand * Src0 = TII -> getNamedOperand ( * MI , <STR_LIT> :: src0 ) ; const MachineOperand * Src1 = TII -> getNamedOperand ( * MI , <STR_LIT> :: src1 ) ; if ( Src1 -> isIdenticalTo ( * Src0 ) ) { LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << * MI << <STR_LIT> ) ; int Src2Idx = <STR_LIT> ( Opc , <STR_LIT> :: src2 ) ; if ( Src2Idx != - <NUM_LIT> ) MI -> RemoveOperand ( Src2Idx ) ; MI -> RemoveOperand ( <STR_LIT> ( Opc , <STR_LIT> :: src1 ) ) ; mutateCopyOp ( * MI , TII -> get ( Src0 -> isReg ( ) ? ( unsigned ) <STR_LIT> :


<s> if ( TII -> hasModifiersSet ( * Def , <STR_LIT> :: clamp ) ) return false ; LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << MI << <STR_LIT> << * Def << '\n' ) ; DefOMod -> setImm ( OMod ) ; MRI -> replaceRegWith ( MI . getOperand ( <NUM_LIT> ) . getReg ( ) , Def -> getOperand ( <NUM_LIT> ) .


<s> static bool updateOperand ( FoldCandidate & Fold , const TargetRegisterInfo & TRI ) { MachineInstr * MI = Fold . UseMI ; MachineOperand & Old = MI -> getOperand ( Fold . UseOpNo ) ; assert ( Old . isReg ( ) ) ; if ( Fold . isImm ( ) ) { if ( MI -> getDesc ( ) . TSFlags & <STR_LIT> ) { unsigned Opcode = MI -> getOpcode ( ) ; int OpNo = MI -> getOperandNo ( & Old ) ; int ModIdx = - <NUM_LIT> ; if ( OpNo == <STR_LIT> ( Opcode , <STR_LIT> :: src0 ) ) ModIdx = <STR_LIT> :: src0_modifiers ; else if ( OpNo == <STR_LIT> ( Opcode , <STR_LIT> :: src1 ) ) ModIdx = <STR_LIT> :: src1_modifiers ; else if ( OpNo == <STR_LIT> ( Opcode , <STR_LIT> :: src2 ) ) ModIdx = <STR_LIT> :: src2_modifiers ; assert ( ModIdx != - <NUM_LIT> ) ; ModIdx = <STR_LIT> ( Opcode , ModIdx ) ; MachineOperand & Mod = MI -> getOperand ( ModIdx ) ; unsigned Val = Mod . getImm ( ) ; if ( ( Val & <STR_LIT> ) || ! ( Val & <STR_LIT> ) ) return false ; if ( ! isUInt < <NUM_LIT> >


<s> bool IsVMEM = isVMEMClauseInst ( MI ) ; if ( ! isValidClauseInst ( MI , IsVMEM ) ) continue ; RegUse Defs , Uses ; GCNDownwardRPTracker RPT ( * LIS ) ; RPT . reset ( MI ) ; if ( ! processRegUses ( MI , Defs , Uses , RPT ) ) continue ; unsigned Length = <NUM_LIT> ; for ( ; Next != E && Length < MaxClause ; ++ Next ) { if ( ! isValidClauseInst ( * Next , IsVMEM ) ) break ; if ( ! processRegUses ( * Next , Defs , Uses , RPT ) ) break ; ++ Length ; } if ( Length < <NUM_LIT> ) continue ; Changed = true ; MFI -> limitOccupancy ( LastRecordedOccupancy ) ; auto B = BuildMI ( MBB , I , DebugLoc ( ) , TII -> get ( TargetOpcode :: BUNDLE ) ) ; Ind -> insertMachineInstrInMaps ( * B ) ; for ( auto BI = I ; BI != Next ; ++ BI ) { BI -> bundleWithPred ( ) ; Ind -> removeSingleMachineInstrFromMaps ( * BI ) ; for ( MachineOperand & MO : BI -> defs ( ) ) if ( MO . readsReg ( ) ) MO . setIsInternalRead ( true ) ; } for ( auto && R : Defs ) { forAllLanes ( R . first , R . second . second , [ & R , & B ] ( unsigned SubReg ) { unsigned S = R . second . first | RegState :: EarlyClobber ; if ( ! SubReg ) S &= ~ ( RegState :: Undef | RegState :: Dead ) ; B . addDef ( R . first , S , SubReg ) ; } ) ; } for ( auto && R : Uses


<s> if ( LaneMask . all ( ) || TargetRegisterInfo :: isPhysicalRegister ( Reg ) || LaneMask == MRI -> getMaxLaneMaskForVReg ( Reg ) ) { Func ( <NUM_LIT> ) ; return ; } const TargetRegisterClass * RC = MRI -> getRegClass ( Reg ) ; unsigned E = TRI -> getNumSubRegIndices ( ) ; SmallVector < unsigned , <STR_LIT> > CoveringSubregs ; for ( unsigned Idx = <NUM_LIT> ; Idx < E ; ++ Idx ) { if ( TRI -> getSubClassWithSubReg ( RC , Idx ) != RC ) continue ; LaneBitmask SubRegMask = TRI -> getSubRegIndexLaneMask ( Idx ) ; if ( SubRegMask == LaneMask ) { Func ( Idx ) ; return ; } if ( ( SubRegMask & ~ LaneMask ) . any ( ) || ( SubRegMask & LaneMask ) . none ( ) ) continue ; CoveringSubregs . push_back ( Idx ) ; } llvm :: sort ( CoveringSubregs , [ this ] ( unsigned A , unsigned B ) { LaneBitmask MaskA = TRI -> getSubRegIndexLaneMask ( A ) ; LaneBitmask MaskB = TRI -> getSubRegIndexLaneMask ( B ) ; unsigned NA = MaskA . getNumLanes ( ) ; unsigned NB = MaskB . getNumLanes ( ) ; if ( NA != NB ) return NA > NB ; return MaskA . getHighestLane ( ) > MaskB . getHighestLane ( ) ; } )


<s> if ( LaneMask . all ( ) || TargetRegisterInfo :: isPhysicalRegister ( Reg ) || LaneMask == MRI -> getMaxLaneMaskForVReg ( Reg ) ) { Func ( <NUM_LIT> ) ; return ; } const TargetRegisterClass * RC = MRI -> getRegClass ( Reg ) ; unsigned E = TRI -> getNumSubRegIndices ( ) ; SmallVector < unsigned , <STR_LIT> > CoveringSubregs ; for ( unsigned Idx = <NUM_LIT> ; Idx < E ; ++ Idx ) { if ( TRI -> getSubClassWithSubReg ( RC , Idx ) != RC ) continue ; LaneBitmask SubRegMask = TRI -> getSubRegIndexLaneMask ( Idx ) ; if ( SubRegMask == LaneMask ) { Func ( Idx ) ; return ; } if ( ( SubRegMask & ~ LaneMask ) . any ( ) || ( SubRegMask & LaneMask ) . none ( ) ) continue ; CoveringSubregs . push_back ( Idx ) ; } llvm :: sort ( CoveringSubregs , [ this ] ( unsigned A , unsigned B ) { LaneBitmask MaskA = TRI -> getSubRegIndexLaneMask ( A ) ; LaneBitmask MaskB = TRI -> getSubRegIndexLaneMask ( B ) ; unsigned NA = MaskA . getNumLanes ( ) ; unsigned NB = MaskB . getNumLanes ( ) ; if ( NA != NB ) return NA > NB ; return MaskA . getHighestLane ( ) > MaskB . getHighestLane ( ) ; } ) ; for ( unsigned Idx


<s> for ( const MachineOperand & MO : MI . operands ( ) ) { if ( MO . isFI ( ) ) return false ; if ( ! MO . isReg ( ) ) continue ; Register Reg = MO . getReg ( ) ; if ( MO . isTied ( ) ) return false ; RegUse & Map = MO . isDef ( ) ? Uses : Defs ; auto Conflict = Map . find ( Reg ) ; if ( Conflict == Map . end (


<s> void <ISA_LIT>FormMemoryClauses :: collectRegUses ( const MachineInstr & MI , RegUse & Defs , RegUse & Uses ) const { for ( const MachineOperand & MO : MI . operands ( ) ) { if ( ! MO . isReg ( ) ) continue ; Register Reg = MO . getReg ( ) ; if ( ! Reg ) continue ; LaneBitmask Mask = Reg . isVirtual ( ) ? TRI -> getSubRegIndexLaneMask ( MO . getSubReg


<s> Func ( <NUM_LIT> ) ; return ; } const TargetRegisterClass * RC = MRI -> getRegClass ( Reg ) ; unsigned E = TRI -> getNumSubRegIndices ( ) ; SmallVector < unsigned , <STR_LIT> > CoveringSubregs ; for ( unsigned Idx = <NUM_LIT> ; Idx < E ; ++ Idx ) { if ( TRI -> getSubClassWithSubReg ( RC , Idx ) != RC ) continue ; LaneBitmask SubRegMask = TRI -> getSubRegIndexLaneMask ( Idx ) ; if ( SubRegMask == LaneMask ) { Func ( Idx ) ; return ; } if ( ( SubRegMask & ~ LaneMask ) . any ( ) || ( SubRegMask & LaneMask ) . none ( ) ) continue ; CoveringSubregs . push_back ( Idx ) ; } llvm :: sort ( CoveringSubregs , [ this ] ( unsigned A , unsigned B ) { LaneBitmask MaskA = TRI -> getSubRegIndexLaneMask ( A ) ; LaneBitmask MaskB = TRI -> getSubRegIndexLaneMask ( B ) ; unsigned NA = MaskA . getNumLanes ( ) ; unsigned NB = MaskB . getNumLanes ( ) ; if ( NA != NB ) return NA > NB ; return MaskA . getHighestLane ( ) >


<s> SmallVector < unsigned , <STR_LIT> > CoveringSubregs ; for ( unsigned Idx = <NUM_LIT> ; Idx < E ; ++ Idx ) { if ( TRI -> getSubClassWithSubReg ( RC , Idx ) != RC ) continue ; LaneBitmask SubRegMask = TRI -> getSubRegIndexLaneMask ( Idx ) ; if ( SubRegMask == LaneMask ) { Func ( Idx ) ; return ; } if ( ( SubRegMask & ~ LaneMask ) . any ( ) || ( SubRegMask & LaneMask ) . none ( ) ) continue ; CoveringSubregs . push_back ( Idx ) ; } llvm :: sort ( CoveringSubregs , [ this ] ( unsigned A , unsigned B ) { LaneBitmask MaskA = TRI -> getSubRegIndexLaneMask ( A ) ; LaneBitmask MaskB = TRI -> getSubRegIndexLaneMask ( B ) ; unsigned NA = MaskA . getNumLanes ( ) ; unsigned NB = MaskB . getNumLanes ( ) ; if ( NA != NB ) return NA > NB ; return MaskA . getHighestLane ( ) > MaskB . getHighestLane ( ) ; } ) ; for ( unsigned Idx : CoveringSubregs ) { LaneBitmask SubRegMask = TRI -> getSubRegIndexLaneMask ( Idx ) ; if ( ( SubRegMask & ~ LaneMask ) . any ( ) || (


<s> for ( MachineBasicBlock & MBB : MF ) { MachineBasicBlock :: instr_iterator Next ; for ( auto I = MBB . instr_begin ( ) , E = MBB . instr_end ( ) ; I != E ; I = Next ) { MachineInstr & MI = * I ; Next = std :: next ( I ) ; bool IsVMEM = isVMEMClauseInst ( MI ) ; if ( ! isValidClauseInst ( MI , IsVMEM ) ) continue ; RegUse Defs , Uses ; GCNDownwardRPTracker RPT ( * LIS ) ; RPT . reset ( MI ) ; if ( ! processRegUses ( MI , Defs , Uses , RPT ) ) continue ; unsigned Length = <NUM_LIT> ; for ( ; Next != E && Length < FuncMaxClause ; ++ Next ) { if ( ! isValidClauseInst ( * Next , IsVMEM ) ) break ; if ( ! processRegUses ( * Next , Defs , Uses , RPT ) ) break ; ++ Length ; } if ( Length < <NUM_LIT> ) continue ; Changed = true ; MFI -> limitOccupancy ( LastRecordedOccupancy ) ; auto B = BuildMI ( MBB , I , DebugLoc ( ) , TII -> get ( TargetOpcode :: BUNDLE ) ) ; Ind -> insertMachineInstrInMaps ( * B ) ; for ( auto BI = I ; BI !=


<s> Register Reg = MO . getReg ( ) ; if ( MO . isTied ( ) ) return false ; RegUse & Map = MO . isDef ( ) ? Uses : Defs ; auto Conflict = Map . find ( Reg ) ; if ( Conflict == Map . end ( ) ) continue ; if ( Register :: isPhysicalRegister (


<s> if ( MO . isTied ( ) ) return false ; RegUse & Map = MO . isDef ( ) ? Uses : Defs ; auto Conflict = Map . find ( Reg ) ; if ( Conflict == Map . end ( ) ) continue ; if ( Register :: isPhysicalRegister ( Reg


<s> void <ISA_LIT>FormMemoryClauses :: collectRegUses ( const MachineInstr & MI , RegUse & Defs , RegUse & Uses ) const { for ( const MachineOperand & MO : MI . operands ( ) ) { if ( ! MO . isReg ( ) ) continue ; Register Reg = MO . getReg ( ) ; if ( ! Reg ) continue ; LaneBitmask Mask = Register :: isVirtualRegister ( Reg ) ? TRI -> getSubRegIndexLaneMask ( MO . getSubReg ( ) ) : LaneBitmask :: getAll ( ) ; RegUse & Map = MO . isDef ( ) ? Defs : Uses ; auto Loc = Map . find ( Reg ) ; unsigned State = getMopState ( MO ) ; if ( Loc == Map . end ( ) ) { Map [ Reg ] = std :: make_pair ( State , Mask ) ; } else { Loc -> second .


<s> void <ISA_LIT>FormMemoryClauses :: forAllLanes ( unsigned Reg , LaneBitmask LaneMask , Callable Func ) const { if ( LaneMask . all ( ) || Register :: isPhysicalRegister ( Reg ) || LaneMask == MRI -> getMaxLaneMaskForVReg ( Reg ) ) { Func ( <NUM_LIT> ) ; return ; } const TargetRegisterClass * RC = MRI -> getRegClass ( Reg ) ; unsigned E = TRI -> getNumSubRegIndices ( ) ; SmallVector < unsigned , <STR_LIT> > CoveringSubregs ; for ( unsigned Idx = <NUM_LIT> ; Idx < E ; ++ Idx ) { if ( TRI -> getSubClassWithSubReg ( RC , Idx ) != RC ) continue ; LaneBitmask SubRegMask = TRI -> getSubRegIndexLaneMask ( Idx ) ; if ( SubRegMask == LaneMask ) { Func ( Idx


<s> static unsigned getMopState ( const MachineOperand & MO ) { unsigned S = <NUM_LIT> ; if ( MO . isImplicit ( ) ) S |= RegState :: Implicit ; if ( MO . isDead ( ) ) S |= RegState :: Dead ; if ( MO . isUndef ( ) ) S |= RegState :: Undef ; if ( MO . isKill ( ) ) S |= RegState ::


<s> if ( MO . isEarlyClobber ( ) ) S |= RegState :: EarlyClobber ; if ( Register :: isPhysicalRegister ( MO . getReg ( ) ) && MO . isRenamable (


<s> if ( ! isValidClauseInst ( MI , IsVMEM ) ) continue ; RegUse Defs , Uses ; GCNDownwardRPTracker RPT ( * LIS ) ; RPT . reset ( MI ) ; if ( ! processRegUses ( MI , Defs , Uses , RPT ) ) continue ; unsigned Length = <NUM_LIT> ; for ( ; Next != E && Length < FuncMaxClause ; ++ Next ) { if ( ! isValidClauseInst ( * Next , IsVMEM ) ) break ; if ( ! processRegUses ( * Next , Defs , Uses , RPT ) ) break ; ++ Length ; } if ( Length < <NUM_LIT> ) continue ; Changed = true ; MFI -> limitOccupancy ( LastRecordedOccupancy ) ; auto B = BuildMI ( MBB , I , DebugLoc ( ) , TII -> get ( TargetOpcode :: BUNDLE ) ) ; Ind -> insertMachineInstrInMaps ( * B ) ; for ( auto BI = I ; BI != Next ; ++ BI ) { BI -> bundleWithPred ( ) ; Ind -> removeSingleMachineInstrFromMaps ( * BI ) ; for ( MachineOperand & MO : BI -> defs ( ) ) if ( MO . readsReg ( ) ) MO . setIsInternalRead ( true ) ; } for ( auto && R : Defs ) { forAllLanes ( R . first , R . second . second , [ & R , & B ] ( unsigned SubReg ) { unsigned S = R . second . first | RegState :: EarlyClobber ; if ( ! SubReg ) S &= ~ ( RegState :: Undef | RegState :: Dead ) ; B . addDef ( R . first , S , SubReg ) ; } ) ; } for ( auto && R : Uses ) { forAllLanes ( R . first , R . second . second , [ & R , & B ] ( unsigned SubReg ) { B . addUse ( R . first , R . second . first & ~ RegState :: Kill , SubReg ) ; } ) ; } for ( auto && R : Defs ) { unsigned Reg = R . first ; Uses . erase ( Reg ) ; if ( Register :: isPhysicalRegister ( Reg ) ) continue ; LIS -> removeInterval ( Reg ) ; LIS -> createAndComputeVirtRegInterval ( Reg ) ; } for ( auto && R : Uses ) { unsigned Reg = R . first ; if ( Register :: isPhysicalRegister ( Reg )


<s> if ( ! isValidClauseInst ( * Next , IsVMEM ) ) break ; if ( ! processRegUses ( * Next , Defs , Uses , RPT ) ) break ; ++ Length ; } if ( Length < <NUM_LIT> ) continue ; Changed = true ; MFI -> limitOccupancy ( LastRecordedOccupancy ) ; auto B = BuildMI ( MBB , I , DebugLoc ( ) , TII -> get ( TargetOpcode :: BUNDLE ) ) ; Ind -> insertMachineInstrInMaps ( * B ) ; for ( auto BI = I ; BI != Next ; ++ BI ) { BI -> bundleWithPred ( ) ; Ind -> removeSingleMachineInstrFromMaps ( * BI ) ; for ( MachineOperand & MO : BI -> defs ( ) ) if ( MO . readsReg ( ) ) MO . setIsInternalRead ( true ) ; } for ( auto && R : Defs ) { forAllLanes ( R . first , R . second . second , [ & R , & B ] ( unsigned SubReg ) { unsigned S = R . second . first | RegState :: EarlyClobber ; if ( ! SubReg ) S &= ~ ( RegState :: Undef | RegState :: Dead ) ; B . addDef ( R . first , S , SubReg ) ; } ) ; } for ( auto && R : Uses ) { forAllLanes ( R . first , R . second . second , [ & R , & B ] ( unsigned SubReg ) { B . addUse ( R . first , R . second . first & ~ RegState :: Kill , SubReg ) ; } ) ; } for ( auto && R : Defs ) { unsigned Reg = R . first ; Uses . erase ( Reg ) ; if ( Register :: isPhysicalRegister ( Reg ) ) continue ; LIS -> removeInterval ( Reg ) ; LIS -> createAndComputeVirtRegInterval ( Reg ) ; } for ( auto && R : Uses ) { unsigned Reg = R . first ; if ( Register :: isPhysicalRegister ( Reg )


<s> MFI -> limitOccupancy ( LastRecordedOccupancy ) ; auto B = BuildMI ( MBB , I , DebugLoc ( ) , TII -> get ( TargetOpcode :: BUNDLE ) ) ; Ind -> insertMachineInstrInMaps ( * B ) ; for ( auto BI = I ; BI != Next ; ++ BI ) { BI -> bundleWithPred ( ) ; Ind -> removeSingleMachineInstrFromMaps ( * BI ) ; for ( MachineOperand & MO : BI -> defs ( ) ) if ( MO . readsReg ( ) ) MO . setIsInternalRead ( true ) ; } for ( auto && R : Defs ) { forAllLanes ( R . first , R . second . second , [ & R , & B ] ( unsigned SubReg ) { unsigned S = R . second . first | RegState :: EarlyClobber ; if ( ! SubReg ) S &= ~ ( RegState :: Undef | RegState :: Dead ) ; B . addDef ( R . first , S , SubReg ) ; } ) ; } for ( auto && R : Uses ) { forAllLanes ( R . first , R . second . second , [ & R , & B ] ( unsigned SubReg ) { B . addUse ( R . first , R . second . first & ~ RegState :: Kill , SubReg ) ; } ) ; } for ( auto && R : Defs ) { unsigned Reg = R


<s> if ( ! MO . isReg ( ) ) continue ; Register Reg = MO . getReg ( ) ; if ( MO . isTied ( ) ) return false ; const RegUse & Map = MO . isDef ( ) ? Uses : Defs ; auto Conflict = Map . find ( Reg ) ; if ( Conflict == Map . end ( ) ) continue ; if ( Reg . isPhysical ( ) ) return false ; LaneBitmask Mask = TRI -> getSubRegIndexLaneMask ( MO . getSubReg ( ) ) ; if ( ( Conflict -> second . second & Mask ) . any ( )


<s> bool <ISA_LIT>FormMemoryClauses :: canBundle ( const MachineInstr & MI , const RegUse & Defs , const RegUse & Uses ) const { for ( const MachineOperand & MO : MI . operands ( ) ) { if ( MO . isFI ( ) ) return false ; if ( ! MO . isReg ( ) ) continue ; Register Reg = MO . getReg ( ) ; if ( MO . isTied ( ) ) return false ; const RegUse & Map = MO . isDef ( ) ? Uses : Defs ; auto Conflict = Map . find ( Reg ) ; if ( Conflict == Map . end


<s> RPT . advanceToNext ( ) ; GCNRegPressure MaxPressure = RPT . moveMaxPressure ( ) ; unsigned Occupancy = MaxPressure . getOccupancy ( * ST ) ; if ( Occupancy >= MFI -> getMinAllowedOccupancy ( ) && MaxPressure . getVGPRNum ( ST -> hasGFX90AInsts ( ) ) <= MaxVGPRs / <NUM_LIT> && MaxPressure . getSGPRNum ( ) <= MaxSGPRs / <NUM_LIT> ) { LastRecordedOccupancy = Occupancy ; return true


<s> MachineFunctionProperties getClearedProperties ( ) const


<s> if ( MI . isBundled ( ) ) return false ; if ( ! MI . mayLoad ( ) || MI . mayStore ( ) ) return false ; if ( <ISA_LIT>InstrInfo :: isAtomic ( MI ) ) return false ; if ( IsVMEMClause && ! isVMEMClauseInst ( MI ) ) return false ; if ( ! IsVMEMClause && ! isSMEMClauseInst ( MI )


<s> RPT . advanceBeforeNext ( ) ; } const GCNRPTracker :: LiveRegSet LiveRegsCopy ( RPT . getLiveRegs ( ) ) ; RegUse Defs , Uses ; if ( ! processRegUses ( MI , Defs , Uses , RPT ) ) { RPT . reset ( MI , & LiveRegsCopy ) ; continue ; } MachineBasicBlock :: iterator LastClauseInst = Next ; unsigned Length = <NUM_LIT> ; for ( ; Next != E && Length < FuncMaxClause ; ++ Next ) { if ( Next -> isMetaInstruction ( ) ) continue ; if ( ! isValidClauseInst ( * Next , IsVMEM ) ) break ; if ( ! processRegUses ( * Next , Defs , Uses , RPT ) ) break ; LastClauseInst = Next ; ++ Length ; } if ( Length < <NUM_LIT> ) { RPT . reset ( MI , & LiveRegsCopy ) ; continue ; } Changed = true ; MFI -> limitOccupancy ( LastRecordedOccupancy ) ; assert ( ! LastClauseInst -> isMetaInstruction ( ) ) ; SlotIndex ClauseLiveInIdx = LIS -> getInstructionIndex ( MI ) ; SlotIndex ClauseLiveOutIdx = LIS -> getInstructionIndex ( * LastClauseInst ) . getNextIndex ( ) ; MachineInstrBuilder Kill ; for ( auto && R : Uses ) { Register Reg = R . first ; if ( Reg . isPhysical ( ) ) continue ; SmallVector < std :: tuple < unsigned , unsigned >> KillOps ; const LiveInterval & LI = LIS -> getInterval ( R . first ) ; if ( ! LI . hasSubRanges ( ) ) { if ( ! LI . liveAt ( ClauseLiveOutIdx ) ) { KillOps . emplace_back ( R . second . first | RegState :: Kill , <STR_LIT> ) ; } } else { LaneBitmask KilledMask ; for ( const LiveInterval :: SubRange & SR : LI . subranges ( ) ) { if ( SR . liveAt ( ClauseLiveInIdx ) && ! SR . liveAt ( ClauseLiveOutIdx ) ) KilledMask |= SR . LaneMask ; } if ( KilledMask . none ( ) )


<s> } if ( Length < <NUM_LIT> ) { RPT . reset ( MI , & LiveRegsCopy ) ; continue ; } Changed = true ; MFI -> limitOccupancy ( LastRecordedOccupancy ) ; auto B = BuildMI ( MBB , I , DebugLoc ( ) , TII -> get ( TargetOpcode :: BUNDLE ) ) ; Ind -> insertMachineInstrInMaps ( * B ) ; RPT . reset ( * B , & LiveRegsCopy ) ; for ( auto BI = I ; BI != Next ; ++ BI ) { BI -> bundleWithPred ( ) ; Ind -> removeSingleMachineInstrFromMaps ( * BI ) ; for ( MachineOperand & MO : BI -> defs ( ) ) if ( MO . readsReg ( ) ) MO . setIsInternalRead ( true ) ; } for ( auto && R : Defs ) { forAllLanes ( R . first , R . second . second , [ & R , & B ] ( unsigned SubReg ) { unsigned S = R . second . first | RegState :: EarlyClobber ; if ( ! SubReg ) S &= ~ ( RegState :: Undef | RegState :: Dead ) ; B . addDef ( R . first , S , SubReg ) ; } ) ; } for ( auto && R : Uses ) { forAllLanes ( R . first , R . second . second , [ & R , & B ] ( unsigned SubReg ) { B . addUse ( R . first , R . second . first & ~ RegState :: Kill , SubReg ) ; } ) ; } for ( auto && R : Defs ) { Register Reg = R . first ; Uses . erase ( Reg ) ; if ( Reg . isPhysical ( ) ) continue ; LIS -> removeInterval ( Reg ) ; LIS -> createAndComputeVirtRegInterval ( Reg ) ; } for ( auto && R : Uses ) { Register Reg = R . first ; if ( Reg . isPhysical ( ) ) continue ; LIS -> removeInterval ( Reg ) ; LIS -> createAndComputeVirtRegInterval ( Reg ) ; } } } return Changed


<s> bool <ISA_LIT>FormMemoryClauses :: canBundle ( const MachineInstr & MI , RegUse & Defs , RegUse & Uses ) const { for ( const MachineOperand & MO : MI . operands ( ) ) { if ( MO . isFI ( ) ) return false ; if ( ! MO . isReg ( ) ) continue ; unsigned Reg = MO . getReg ( ) ; if ( MO . isTied ( ) ) return false ; RegUse & Map = MO . isDef


<s> if ( Occupancy >= MFI -> getMinAllowedOccupancy ( ) && MaxPressure . getVGPRNum ( ) <= MaxVGPRs && MaxPressure . getSGPRNum ( ) <= MaxSGPRs ) { LastRecordedOccupancy =


<s> FunctionPass * llvm :: create<ISA_LIT>FormMemoryClausesPass ( ) { return new <ISA_LIT>FormMemoryClauses (


<s> if ( LaneMask . all ( ) || TargetRegisterInfo :: isPhysicalRegister ( Reg ) || LaneMask == MRI -> getMaxLaneMaskForVReg ( Reg ) ) { Func ( <NUM_LIT> ) ; return ; } const TargetRegisterClass * RC = MRI -> getRegClass ( Reg ) ; unsigned E = TRI -> getNumSubRegIndices ( ) ; SmallVector < unsigned , <STR_LIT> > CoveringSubregs ; for ( unsigned Idx = <NUM_LIT> ; Idx < E ; ++ Idx ) { if ( TRI -> getSubClassWithSubReg ( RC , Idx ) != RC ) continue ; LaneBitmask SubRegMask = TRI -> getSubRegIndexLaneMask ( Idx ) ; if ( SubRegMask == LaneMask ) { Func ( Idx ) ; return ; } if ( ( SubRegMask & ~ LaneMask ) . any ( ) || ( SubRegMask & LaneMask ) . none ( ) ) continue ; CoveringSubregs . push_back ( Idx ) ; } llvm :: sort ( CoveringSubregs . begin ( ) , CoveringSubregs . end ( ) , [ this ] ( unsigned A , unsigned B ) { LaneBitmask MaskA = TRI -> getSubRegIndexLaneMask ( A ) ; LaneBitmask MaskB = TRI -> getSubRegIndexLaneMask ( B ) ; unsigned NA = MaskA . getNumLanes ( ) ; unsigned NB = MaskB . getNumLanes ( ) ; if ( NA != NB ) return NA > NB ; return MaskA . getHighestLane ( ) > MaskB . getHighestLane ( ) ; } ) ; for ( unsigned Idx : CoveringSubregs ) { LaneBitmask SubRegMask = TRI -> getSubRegIndexLaneMask ( Idx ) ; if ( ( SubRegMask & ~ LaneMask ) . any ( ) || ( SubRegMask & LaneMask )


<s> if ( SubRegMask == LaneMask ) { Func ( Idx ) ; return ; } if ( ( SubRegMask & ~ LaneMask ) . any ( ) || ( SubRegMask & LaneMask ) . none ( ) ) continue ; CoveringSubregs . push_back ( Idx ) ; } llvm :: sort ( CoveringSubregs . begin ( ) , CoveringSubregs . end ( ) , [ this ] ( unsigned A , unsigned B ) { LaneBitmask MaskA = TRI -> getSubRegIndexLaneMask ( A ) ; LaneBitmask MaskB = TRI -> getSubRegIndexLaneMask ( B ) ; unsigned NA = MaskA . getNumLanes ( ) ; unsigned NB = MaskB . getNumLanes ( ) ; if ( NA != NB ) return NA > NB ; return MaskA . getHighestLane ( ) > MaskB . getHighestLane ( ) ; }


<s> StringRef getPassName ( ) const override { return


<s> static bool isValidClauseInst ( const MachineInstr & MI , bool IsVMEMClause ) { if ( MI . isDebugValue ( ) || MI . isBundled ( ) ) return false ; if ( ! MI . mayLoad ( ) || MI . mayStore


<s> if ( MI . isDebugValue ( ) || MI . isBundled ( ) ) return false ; if ( ! MI . mayLoad ( ) || MI . mayStore ( ) ) return false ; if ( <STR_LIT> ( MI . getOpcode ( ) ) != - <NUM_LIT> || <STR_LIT> ( MI . getOpcode ( ) ) != - <NUM_LIT> ) return false ; if ( IsVMEMClause && ! isVMEMClauseInst ( MI ) )


<s> static bool isVMEMClauseInst ( const MachineInstr & MI


<s> return <ISA_LIT>InstrInfo :: isFLAT ( MI ) ||


<s> bool <ISA_LIT>FormMemoryClauses :: processRegUses ( const MachineInstr & MI , RegUse & Defs , RegUse & Uses , GCNDownwardRPTracker & RPT ) { if ( ! canBundle ( MI , Defs , Uses )


<s> if ( ! canBundle ( MI , Defs , Uses ) ) return false ; if ( ! checkPressure ( MI , RPT ) ) return false ; collectRegUses ( MI , Defs


<s> GCNDownwardRPTracker RPT ( * LIS ) ; RPT . reset ( MI ) ; if ( ! processRegUses ( MI , Defs , Uses , RPT ) ) continue ; unsigned Length = <NUM_LIT> ; for ( ; Next != E && Length < MaxClause ; ++ Next ) { if ( ! isValidClauseInst ( * Next , IsVMEM ) ) break ; if ( ! processRegUses ( * Next , Defs , Uses , RPT ) ) break ; ++ Length ; } if ( Length < <NUM_LIT> ) continue ; Changed = true ; MFI -> limitOccupancy ( LastRecordedOccupancy ) ; auto B = BuildMI ( MBB , I , DebugLoc ( ) , TII -> get ( TargetOpcode :: BUNDLE ) ) ; Ind -> insertMachineInstrInMaps ( * B ) ; for ( auto BI = I ; BI != Next ; ++ BI ) { BI -> bundleWithPred ( ) ; Ind -> removeSingleMachineInstrFromMaps ( * BI ) ; for ( MachineOperand & MO : BI -> defs ( ) ) if ( MO . readsReg ( ) ) MO . setIsInternalRead ( true ) ; } for ( auto && R : Defs ) { forAllLanes ( R . first , R . second . second , [ & R , & B ] ( unsigned SubReg ) { unsigned S = R . second . first | RegState :: EarlyClobber ; if ( ! SubReg ) S &= ~ ( RegState :: Undef | RegState :: Dead ) ; B . addDef ( R . first , S , SubReg ) ; } ) ; } for ( auto && R : Uses ) { forAllLanes ( R . first , R . second . second , [ & R , & B ] ( unsigned SubReg ) { B . addUse ( R . first , R . second . first & ~ RegState :: Kill , SubReg ) ; } ) ; } for ( auto && R : Defs ) { unsigned Reg = R . first ; Uses . erase ( Reg ) ; if ( TargetRegisterInfo :: isPhysicalRegister ( Reg ) ) continue ; LIS -> removeInterval ( Reg ) ; LIS -> createAndComputeVirtRegInterval ( Reg ) ; } for ( auto && R : Uses ) { unsigned Reg = R . first ; if ( TargetRegisterInfo :: isPhysicalRegister ( Reg ) ) continue ; LIS -> removeInterval ( Reg ) ; LIS -> createAndComputeVirtRegInterval ( Reg ) ; } } } return Changed


<s> if ( ST . isAmdHsaOS ( ) ) { MRI . addLiveIn ( PreloadedPrivateBufferReg ) ; MBB . addLiveIn ( PreloadedPrivateBufferReg ) ; } if ( ScratchRsrcReg == TRI -> reservedPrivateSegmentBufferReg ( MF ) ) { MachineRegisterInfo & MRI = MF . getRegInfo ( ) ; unsigned NumPreloaded = MFI -> getNumPreloadedSGPRs ( ) / <NUM_LIT> ; for ( MCPhysReg Reg : getAllSGPR128 ( ) . drop_back ( <NUM_LIT> ) . slice ( NumPreloaded ) ) { if ( ! MRI . isPhysRegUsed ( Reg ) ) { assert ( MRI . isAllocatable ( Reg ) ) ; MRI . replaceRegWith ( ScratchRsrcReg , Reg ) ; ScratchRsrcReg = Reg ; MFI -> setScratchRSrcReg ( ScratchRsrcReg ) ; break ; } } } if ( ScratchWaveOffsetReg == TRI -> reservedPrivateSegmentWaveByteOffsetReg ( MF ) ) { MachineRegisterInfo & MRI = MF . getRegInfo ( ) ; unsigned NumPreloaded = MFI -> getNumPreloadedSGPRs ( ) ; for ( MCPhysReg Reg : getAllSGPRs ( ) . drop_back ( <NUM_LIT> ) . slice ( NumPreloaded ) ) { if ( ! MRI . isPhysRegUsed ( Reg ) ) { assert ( MRI . isAllocatable ( Reg ) && ! TRI -> isSubRegisterEq ( ScratchRsrcReg , Reg ) ) ; MRI . replaceRegWith ( ScratchWaveOffsetReg , Reg ) ; ScratchWaveOffsetReg = Reg ; MFI -> setScratchWaveOffsetReg ( ScratchWaveOffsetReg ) ; break ; } } } assert ( ! TRI -> isSubRegister ( ScratchRsrcReg , ScratchWaveOffsetReg ) ) ; const MCInstrDesc & SMovB32 = TII -> get ( <STR_LIT> ) ; MachineBasicBlock :: iterator I = MBB . begin ( ) ; DebugLoc DL ; if ( PreloadedScratchWaveOffsetReg != ScratchWaveOffsetReg ) { BuildMI ( MBB , I , DL , SMovB32 , ScratchWaveOffsetReg ) . addReg ( PreloadedScratchWaveOffsetReg , RegState :: Kill ) ; } if ( ST . isAmdHsaOS ( ) ) { assert ( ! TRI -> isSubRegisterEq ( PreloadedPrivateBufferReg , ScratchRsrcReg ) && ! TRI -> isSubRegisterEq ( PreloadedPrivateBufferReg , ScratchWaveOffsetReg ) ) ; unsigned Rsrc01 = TRI -> getSubReg ( ScratchRsrcReg , <STR_LIT> ) ; unsigned Rsrc23 = TRI -> getSubReg ( ScratchRsrcReg , <STR_LIT> ) ; unsigned Lo = TRI -> getSubReg ( PreloadedPrivateBufferReg , <STR_LIT> ) ; unsigned Hi = TRI -> getSubReg ( PreloadedPrivateBufferReg , <STR_LIT> ) ; const MCInstrDesc & SMovB64 = TII -> get ( <STR_LIT> ) ; BuildMI ( MBB , I , DL , SMovB64 , Rsrc01 ) . addReg ( Lo , RegState :: Kill ) ; BuildMI ( MBB , I , DL , SMovB64 , Rsrc23 ) . addReg ( Hi , RegState :: Kill ) ; } else { unsigned Rsrc0 = TRI -> getSubReg ( ScratchRsrcReg , <STR_LIT> ) ; unsigned Rsrc1 = TRI -> getSubReg ( ScratchRsrcReg , <STR_LIT> ) ; unsigned Rsrc2 = TRI -> getSubReg ( ScratchRsrcReg , <STR_LIT>


<s> return makeArrayRef ( <STR_LIT> . begin ( ) , <STR_LIT> . getNumRegs (


<s> return makeArrayRef ( <STR_LIT> . begin ( ) , <STR_LIT> . getNumRegs ( )


<s> if ( FuncInfo -> hasSpilledVGPRs ( ) ) return false ; for ( int I = FrameInfo -> getObjectIndexBegin ( ) , E = FrameInfo -> getObjectIndexEnd ( ) ; I != E


<s> MachineBasicBlock :: iterator I = MBB . begin ( ) ; DebugLoc DL ; for ( unsigned i = <NUM_LIT> ; i < <NUM_LIT> ; ++ i ) { unsigned WorkGroupIDSGPR = MFI -> getWorkGroupIDSGPR ( i ) ; MF . getRegInfo ( ) . addLiveIn ( WorkGroupIDSGPR ) ; MBB . addLiveIn ( WorkGroupIDSGPR ) ; unsigned WorkGroupIDVGPR = MF . getRegInfo ( ) . createVirtualRegister ( & <STR_LIT> ) ; BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , WorkGroupIDVGPR ) . addReg ( WorkGroupIDSGPR ) ; int WorkGroupIDObjectIdx = MFI -> getDebuggerWorkGroupIDStackObjectIndex ( i ) ; TII -> storeRegToStackSlot ( MBB , I , WorkGroupIDVGPR , false , WorkGroupIDObjectIdx , & <STR_LIT> , TRI ) ; unsigned WorkItemIDVGPR = MFI -> getWorkItemIDVGPR ( i ) ; MF . getRegInfo ( ) . addLiveIn ( WorkItemIDVGPR ) ; MBB . addLiveIn ( WorkItemIDVGPR


<s> MRI . addLiveIn ( PreloadedScratchWaveOffsetReg ) ; MBB . addLiveIn ( PreloadedScratchWaveOffsetReg ) ; } if ( ResourceRegUsed && PreloadedPrivateBufferReg != <STR_LIT> ) { assert ( ST . isAmdCodeObjectV2 ( MF ) || ST . isMesaGfxShader ( MF ) ) ; MRI . addLiveIn ( PreloadedPrivateBufferReg ) ; MBB . addLiveIn ( PreloadedPrivateBufferReg ) ; } for ( MachineBasicBlock & OtherBB : MF ) { if ( & OtherBB == & MBB ) continue ; if ( OffsetRegUsed ) OtherBB . addLiveIn ( ScratchWaveOffsetReg ) ; if ( ResourceRegUsed ) OtherBB . addLiveIn ( ScratchRsrcReg ) ; } DebugLoc DL ; MachineBasicBlock :: iterator I = MBB . begin ( ) ; bool CopyBuffer = ResourceRegUsed && PreloadedPrivateBufferReg != <STR_LIT> && ST . isAmdCodeObjectV2 ( MF ) && ScratchRsrcReg != PreloadedPrivateBufferReg ; bool CopyBufferFirst = TRI -> isSubRegisterEq ( PreloadedPrivateBufferReg , ScratchWaveOffsetReg ) ; if ( CopyBuffer && CopyBufferFirst ) { BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , ScratchRsrcReg ) . addReg ( PreloadedPrivateBufferReg , RegState :: Kill ) ; } if ( OffsetRegUsed && PreloadedScratchWaveOffsetReg != ScratchWaveOffsetReg ) { BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , ScratchWaveOffsetReg ) . addReg ( PreloadedScratchWaveOffsetReg , MRI . isPhysRegUsed ( ScratchWaveOffsetReg ) ? <NUM_LIT> : RegState :: Kill ) ; } if ( CopyBuffer && ! CopyBufferFirst ) { BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , ScratchRsrcReg ) . addReg ( PreloadedPrivateBufferReg , RegState :: Kill ) ; } if ( ResourceRegUsed && ( ST . isMesaGfxShader ( MF ) || ( PreloadedPrivateBufferReg == <STR_LIT> ) ) ) { assert ( ! ST . isAmdCodeObjectV2 ( MF ) ) ; const MCInstrDesc & SMovB32 = TII -> get ( <STR_LIT> ) ; unsigned Rsrc2 = TRI -> getSubReg ( ScratchRsrcReg , <STR_LIT> ) ; unsigned Rsrc3 = TRI -> getSubReg ( ScratchRsrcReg , <STR_LIT> ) ; uint64_t Rsrc23 = TII -> getScratchRsrcWords23 ( ) ; if ( MFI -> hasImplicitBufferPtr ( ) ) { unsigned Rsrc01 = TRI -> getSubReg ( ScratchRsrcReg , <STR_LIT> ) ; if ( <STR_LIT> ( MF . getFunction ( ) -> getCallingConv ( ) ) ) { const MCInstrDesc & Mov64 = TII -> get ( <STR_LIT> ) ; BuildMI ( MBB , I , DL , Mov64 , Rsrc01 ) . addReg ( MFI -> getImplicitBufferPtrUserSGPR ( ) ) . addReg ( ScratchRsrcReg , RegState :: ImplicitDefine ) ; } else


<s> if ( NumBytes != <NUM_LIT> && hasSP ( MF ) ) { BuildMI ( MBB , MBBI , DL , TII -> get ( <STR_LIT> ) , StackPtrReg ) . addReg ( StackPtrReg ) . addImm ( NumBytes * ST . getWavefrontSize ( ) ) . setMIFlag


<s> if ( NeedFP ) { BuildMI ( MBB , MBBI , DL , TII -> get ( <STR_LIT> ) , FramePtrReg ) . addReg ( StackPtrReg ) . setMIFlag ( MachineInstr :: FrameSetup ) ; } uint32_t NumBytes = MFI . getStackSize ( ) ; if ( NumBytes != <NUM_LIT> && hasSP ( MF ) ) { BuildMI ( MBB , MBBI , DL , TII -> get ( <STR_LIT> ) , StackPtrReg ) . addReg ( StackPtrReg ) . addImm ( NumBytes * ST . getWavefrontSize ( ) ) . setMIFlag


<s> static ArrayRef < MCPhysReg > getAllSGPRs ( const <ISA_LIT>Subtarget


<s> int <ISA_LIT>FrameLowering :: getFrameIndexReference ( const MachineFunction & MF , int FI , unsigned &


<s> const <ISA_LIT>RegisterInfo * RI = MF . getSubtarget < <ISA_LIT>Subtarget > ( ) . getRegisterInfo ( ) ; FrameReg = RI -> getFrameRegister (


<s> unsigned ScratchRsrcReg = MFI -> getScratchRSrcReg ( ) ; if ( ScratchRsrcReg == <STR_LIT> || ! MRI . isPhysRegUsed ( ScratchRsrcReg ) ) return <STR_LIT> ; if ( ST . hasSGPRInitBug ( ) || ScratchRsrcReg != TRI -> reservedPrivateSegmentBufferReg ( MF ) ) return ScratchRsrcReg ; unsigned NumPreloaded = ( MFI -> getNumPreloadedSGPRs ( ) + <NUM_LIT> ) / <NUM_LIT> ; ArrayRef < MCPhysReg > AllSGPR128s = getAllSGPR128 ( ST , MF ) ; AllSGPR128s = AllSGPR128s . slice ( std :: min ( static_cast < unsigned > ( AllSGPR128s . size ( ) ) , NumPreloaded ) ) ; for ( MCPhysReg Reg : AllSGPR128s ) { if ( ! MRI . isPhysRegUsed ( Reg ) && MRI . isAllocatable ( Reg ) ) { MRI . replaceRegWith ( ScratchRsrcReg , Reg ) ; MFI -> setScratchRSrcReg ( Reg ) ; return Reg ; } } return ScratchRsrcReg


<s> return MFI . hasStackObjects ( ) && ! allStackObjectsAreDead ( MFI


<s> bool <ISA_LIT>FrameLowering :: hasSP ( const MachineFunction


<s> if ( ! MFI . hasStackObjects ( ) ) return ; const <ISA_LIT>Subtarget & ST = MF . getSubtarget < <ISA_LIT>Subtarget > ( ) ; const <ISA_LIT>InstrInfo * TII = ST . getInstrInfo ( ) ; const <ISA_LIT>RegisterInfo & TRI = TII -> getRegisterInfo ( ) ; <ISA_LIT>MachineFunctionInfo * FuncInfo = MF . getInfo < <ISA_LIT>MachineFunctionInfo > ( ) ; bool AllSGPRSpilledToVGPRs = false ; if ( TRI . spillSGPRToVGPR ( ) && FuncInfo -> hasSpilledSGPRs ( ) ) { AllSGPRSpilledToVGPRs = true ; for ( MachineBasicBlock & MBB : MF ) { MachineBasicBlock :: iterator Next ; for ( auto I = MBB . begin ( ) , E = MBB . end ( ) ; I != E ; I = Next ) { MachineInstr & MI = * I ; Next = std :: next ( I ) ; if ( TII -> isSGPRSpill ( MI ) ) { int FI = TII -> getNamedOperand ( MI , <STR_LIT> :: addr ) -> getIndex ( ) ; if ( FuncInfo -> allocateSGPRSpillToVGPR ( MF , FI ) ) { bool Spilled = TRI . eliminateSGPRToVGPRSpillFrameIndex ( MI , FI , RS ) ; ( void ) Spilled ; assert ( Spilled && <STR_LIT> ) ; } else AllSGPRSpilledToVGPRs = false ; } } } FuncInfo -> removeSGPRToVGPRFrameIndices ( MFI ) ; } if ( FuncInfo -> hasNonSpillStackObjects ( ) || FuncInfo -> hasSpilledVGPRs ( ) || ! AllSGPRSpilledToVGPRs


<s> I != E ; ++ I ) { if ( ! MFI . isDeadObjectIndex ( I ) && MFI . getStackID ( I ) == TargetStackID :: SGPRSpill && ( I != FuncInfo -> FramePointerSaveIndex && I != FuncInfo -> BasePointerSaveIndex ) ) { return false ; } } return true


<s> MachinePointerInfo PtrInfo = MachinePointerInfo :: getFixedStack ( MF , FI ) ; MachineMemOperand * MMO = MF . getMachineMemOperand ( PtrInfo , MachineMemOperand :: MOLoad , FrameInfo . getObjectSize ( FI ) , FrameInfo . getObjectAlign ( FI ) ) ; TRI . buildSpillLoadStore ( MBB , I , DL , Opc , FI , SpillReg , false , FuncInfo . getStackPtrOffsetReg ( ) , <NUM_LIT> , MMO ,


<s> static void buildPrologSpill ( const GCNSubtarget & ST , const <ISA_LIT>RegisterInfo & TRI , const <ISA_LIT>MachineFunctionInfo & FuncInfo , LivePhysRegs & LiveRegs , MachineFunction & MF , MachineBasicBlock & MBB , MachineBasicBlock :: iterator I , const DebugLoc & DL , Register SpillReg , int FI ) { unsigned Opc = ST . enableFlatScratch ( ) ? <STR_LIT> : <STR_LIT> ; MachineFrameInfo & FrameInfo = MF . getFrameInfo ( ) ; MachinePointerInfo PtrInfo = MachinePointerInfo :: getFixedStack ( MF , FI ) ; MachineMemOperand * MMO = MF . getMachineMemOperand ( PtrInfo , MachineMemOperand :: MOStore , FrameInfo . getObjectSize ( FI ) , FrameInfo . getObjectAlign ( FI ) ) ; LiveRegs . addReg ( SpillReg


<s> MachineFrameInfo & FrameInfo = MF . getFrameInfo ( ) ; MachinePointerInfo PtrInfo = MachinePointerInfo :: getFixedStack ( MF , FI ) ; MachineMemOperand * MMO = MF . getMachineMemOperand ( PtrInfo , MachineMemOperand :: MOStore , FrameInfo . getObjectSize ( FI ) , FrameInfo . getObjectAlign ( FI ) ) ; LiveRegs . addReg


<s> Register ScratchExecCopy ; MachineRegisterInfo & MRI = MF . getRegInfo ( ) ; const GCNSubtarget & ST = MF . getSubtarget < GCNSubtarget > ( ) ; const <ISA_LIT>InstrInfo * TII = ST . getInstrInfo ( ) ; const <ISA_LIT>RegisterInfo & TRI = TII -> getRegisterInfo ( ) ; <ISA_LIT>MachineFunctionInfo * FuncInfo = MF . getInfo < <ISA_LIT>MachineFunctionInfo > ( ) ; DebugLoc DL ; initLiveRegs ( LiveRegs , TRI , FuncInfo , MF , MBB , MBBI , IsProlog ) ; ScratchExecCopy = findScratchNonCalleeSaveRegister ( MRI , LiveRegs , * TRI . getWaveMaskRegClass ( ) ) ; if ( ! ScratchExecCopy ) report_fatal_error ( <STR_LIT> ) ; LiveRegs . addReg ( ScratchExecCopy ) ; const unsigned OrSaveExec = ST . isWave32 ( ) ? <STR_LIT> : <STR_LIT> ; auto SaveExec = BuildMI ( MBB , MBBI , DL , TII -> get ( OrSaveExec

) , ScratchExecCopy ) . addImm ( - <NUM_LIT> ) ;


<s> SavedVGPRs . clearBitsNotInMask ( TRI -> getAllVectorRegMask ( ) ) ; if ( ! ST . hasGFX90AInsts ( ) ) SavedVGPRs . clearBitsInMask ( TRI -> getAllAGPRRegMask ( ) ) ; const bool WillHaveFP = FrameInfo . hasCalls ( ) && ( SavedVGPRs . any ( ) || ! allStackObjectsAreDead ( FrameInfo ) ) ; for ( auto SSpill : MFI -> getSGPRSpillVGPRs ( ) ) SavedVGPRs . reset ( SSpill . VGPR ) ; LivePhysRegs LiveRegs ; LiveRegs . init ( * TRI ) ; if ( WillHaveFP || hasFP ( MF ) ) { assert ( ! MFI -> SGPRForFPSaveRestoreCopy && !

MFI -> FramePointerSaveIndex && <STR_LIT> ) ;


<s> const GCNSubtarget & ST = MF . getSubtarget < GCNSubtarget > ( ) ; const <ISA_LIT>InstrInfo * TII = ST . getInstrInfo ( ) ; const DebugLoc & DL = I -> getDebugLoc ( ) ; unsigned Opc = I -> getOpcode ( ) ; bool IsDestroy = Opc == TII -> getCallFrameDestroyOpcode ( ) ; uint64_t CalleePopAmount = IsDestroy ? I -> getOperand ( <NUM_LIT> ) . getImm ( ) : <NUM_LIT> ; if ( ! hasReservedCallFrame ( MF ) ) { Amount = alignTo ( Amount , getStackAlign ( ) ) ; assert ( isUInt < <NUM_LIT> > ( Amount ) && <STR_LIT> ) ; const <ISA_LIT>MachineFunctionInfo * MFI = MF . getInfo < <ISA_LIT>MachineFunctionInfo > ( ) ; Register SPReg = MFI -> getStackPtrOffsetReg ( ) ; Amount *= getScratchScaleFactor ( ST ) ; if ( IsDestroy ) Amount = - Amount ; auto Add = BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , SPReg ) . addReg ( SPReg ) . addImm ( Amount ) ; Add -> getOperand ( <NUM_LIT> ) . setIsDead ( ) ; } else if ( CalleePopAmount != <NUM_LIT> ) { llvm_unreachable ( <STR_LIT> )


<s> if ( Amount == <NUM_LIT> ) return MBB . erase ( I ) ; const GCNSubtarget & ST = MF . getSubtarget < GCNSubtarget > ( ) ; const <ISA_LIT>InstrInfo * TII = ST . getInstrInfo ( ) ; const DebugLoc & DL = I -> getDebugLoc ( ) ; unsigned Opc = I -> getOpcode ( ) ; bool IsDestroy = Opc == TII -> getCallFrameDestroyOpcode ( ) ; uint64_t CalleePopAmount = IsDestroy ? I -> getOperand ( <NUM_LIT> ) . getImm ( ) : <NUM_LIT> ; if ( ! hasReservedCallFrame ( MF ) ) { Amount = alignTo ( Amount , getStackAlign ( ) ) ; assert ( isUInt < <NUM_LIT> > ( Amount ) &&


<s> const MCInstrDesc & LoadDwordX2 = TII -> get ( <STR_LIT> ) ; auto * MMO = MF . getMachineMemOperand ( PtrInfo , MachineMemOperand :: MOLoad | MachineMemOperand :: MOInvariant | MachineMemOperand :: MODereferenceable , <NUM_LIT> , Align ( <NUM_LIT> ) ) ; unsigned Offset = MF . getFunction ( ) . getCallingConv ( ) == CallingConv :: <ISA_LIT>_CS ? <NUM_LIT> : <NUM_LIT> ; const GCNSubtarget & Subtarget = MF . getSubtarget < GCNSubtarget > ( ) ; unsigned EncodedOffset = <STR_LIT> ( Subtarget , Offset ) ; BuildMI ( MBB , I , DL , LoadDwordX2 , FlatScrInit ) . addReg ( FlatScrInit ) . addImm ( EncodedOffset ) . addImm ( <NUM_LIT> ) . addMemOperand ( MMO ) ; const MCInstrDesc & SAndB32 = TII -> get ( <STR_LIT> ) ; auto And = BuildMI ( MBB , I , DL , SAndB32 , FlatScrInitHi ) . addReg ( FlatScrInitHi ) . addImm ( <NUM_LIT> ) ; And -> getOperand ( <NUM_LIT> ) . setIsDead ( ) ; } else { Register FlatScratchInitReg = MFI -> getPreloadedReg ( <STR_LIT> ) ; assert ( FlatScratchInitReg ) ; MachineRegisterInfo & MRI = MF . getRegInfo ( ) ; MRI . addLiveIn ( FlatScratchInitReg ) ; MBB . addLiveIn ( FlatScratchInitReg ) ; FlatScrInitLo = TRI -> getSubReg ( FlatScratchInitReg , <STR_LIT> ) ; FlatScrInitHi = TRI -> getSubReg ( FlatScratchInitReg , <STR_LIT> ) ; } if ( ST . flatScratchIsPointer ( ) ) { if ( ST . getGeneration ( ) >= <ISA_LIT>Subtarget :: GFX10 ) { BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , FlatScrInitLo ) . addReg ( FlatScrInitLo ) . addReg ( ScratchWaveOffsetReg ) ; auto Addc = BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , FlatScrInitHi ) . addReg ( FlatScrInitHi ) . addImm ( <NUM_LIT> ) ; Addc -> getOperand ( <NUM_LIT> ) . setIsDead ( ) ; BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) ) . addReg ( FlatScrInitLo ) . addImm ( int16_t ( <STR_LIT> :: ID_FLAT_SCR_LO | ( <NUM_LIT> << <STR_LIT> :: WIDTH_M1_SHIFT_ ) ) ) ; BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) ) . addReg ( FlatScrInitHi ) . addImm ( int16_t ( <STR_LIT> :: ID_FLAT_SCR_HI | ( <NUM_LIT> << <STR_LIT> :: WIDTH_M1_SHIFT_ ) ) ) ; return ; } BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , <STR_LIT> ) . addReg ( FlatScrInitLo ) . addReg ( ScratchWaveOffsetReg ) ; auto Addc = BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , <STR_LIT> ) . addReg ( FlatScrInitHi ) . addImm


<s> assert ( FlatScrInit && <STR_LIT> ) ; FlatScrInitLo = TRI -> getSubReg ( FlatScrInit , <STR_LIT> ) ; FlatScrInitHi = TRI -> getSubReg ( FlatScrInit , <STR_LIT> ) ; buildGitPtr ( MBB , I , DL , TII , FlatScrInit ) ; MachinePointerInfo PtrInfo ( <ISA_LIT>AS :: CONSTANT_ADDRESS ) ; const MCInstrDesc & LoadDwordX2 = TII -> get ( <STR_LIT> ) ; auto * MMO = MF . getMachineMemOperand ( PtrInfo , MachineMemOperand :: MOLoad | MachineMemOperand :: MOInvariant | MachineMemOperand :: MODereferenceable , <NUM_LIT> , Align ( <NUM_LIT> ) ) ; unsigned Offset = MF . getFunction ( ) . getCallingConv ( ) == CallingConv :: <ISA_LIT>_CS ? <NUM_LIT> : <NUM_LIT> ; const GCNSubtarget & Subtarget = MF . getSubtarget < GCNSubtarget > ( ) ; unsigned EncodedOffset = <STR_LIT> ( Subtarget , Offset ) ; BuildMI ( MBB , I , DL , LoadDwordX2 , FlatScrInit ) . addReg ( FlatScrInit ) . addImm ( EncodedOffset ) . addImm ( <NUM_LIT> ) . addMemOperand ( MMO ) ; const MCInstrDesc & SAndB32 = TII -> get ( <STR_LIT> ) ; auto And = BuildMI ( MBB , I , DL , SAndB32 , FlatScrInitHi ) . addReg ( FlatScrInitHi ) . addImm ( <NUM_LIT> ) ; And -> getOperand ( <NUM_LIT> ) . setIsDead ( ) ; } else { Register FlatScratchInitReg = MFI -> getPreloadedReg ( <STR_LIT> ) ; assert ( FlatScratchInitReg ) ; MachineRegisterInfo & MRI = MF . getRegInfo ( ) ; MRI . addLiveIn ( FlatScratchInitReg ) ; MBB . addLiveIn ( FlatScratchInitReg ) ; FlatScrInitLo = TRI -> getSubReg ( FlatScratchInitReg , <STR_LIT> ) ; FlatScrInitHi = TRI -> getSubReg ( FlatScratchInitReg , <STR_LIT> ) ; } if ( ST . flatScratchIsPointer ( ) ) { if ( ST . getGeneration ( ) >= <ISA_LIT>Subtarget :: GFX10 ) { BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , FlatScrInitLo ) . addReg ( FlatScrInitLo ) . addReg ( ScratchWaveOffsetReg ) ; auto Addc = BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , FlatScrInitHi ) . addReg ( FlatScrInitHi


<s> void <ISA_LIT>FrameLowering :: emitEntryFunctionPrologue ( MachineFunction & MF , MachineBasicBlock & MBB ) const { assert ( & MF . front ( ) == & MBB && <STR_LIT> ) ; <ISA_LIT>MachineFunctionInfo * MFI = MF . getInfo < <ISA_LIT>MachineFunctionInfo > ( ) ; const GCNSubtarget & ST = MF . getSubtarget < GCNSubtarget > ( ) ; const <ISA_LIT>InstrInfo * TII = ST . getInstrInfo ( ) ; const <ISA_LIT>RegisterInfo * TRI = & TII -> getRegisterInfo ( ) ; MachineRegisterInfo & MRI = MF . getRegInfo ( ) ; const Function & F = MF . getFunction ( ) ; MachineFrameInfo & FrameInfo = MF . getFrameInfo ( ) ; assert ( MFI -> isEntryFunction ( ) ) ; Register PreloadedScratchWaveOffsetReg = MFI -> getPreloadedReg ( <STR_LIT> ) ; Register ScratchRsrcReg ; if ( ! ST . enableFlatScratch ( ) ) ScratchRsrcReg = getEntryFunctionReservedScratchRsrcReg ( MF ) ; if ( ScratchRsrcReg ) { for ( MachineBasicBlock & OtherBB : MF ) { if ( & OtherBB != & MBB ) { OtherBB . addLiveIn ( ScratchRsrcReg ) ; } } } Register PreloadedScratchRsrcReg ; if ( ST . isAmdHsaOrMesa ( F ) ) { PreloadedScratchRsrcReg = MFI -> getPreloadedReg ( <STR_LIT> ) ; if ( ScratchRsrcReg && PreloadedScratchRsrcReg ) { MRI . addLiveIn ( PreloadedScratchRsrcReg ) ; MBB . addLiveIn ( PreloadedScratchRsrcReg ) ; } } DebugLoc DL ; MachineBasicBlock :: iterator I = MBB . begin ( ) ; Register ScratchWaveOffsetReg ; if ( PreloadedScratchWaveOffsetReg && TRI -> isSubRegisterEq ( ScratchRsrcReg , PreloadedScratchWaveOffsetReg ) ) { ArrayRef < MCPhysReg > AllSGPRs = TRI -> getAllSGPR32 ( MF ) ; unsigned NumPreloaded = MFI -> getNumPreloadedSGPRs ( ) ; AllSGPRs = AllSGPRs . slice ( std :: min ( static_cast < unsigned > ( AllSGPRs . size ( ) ) , NumPreloaded ) ) ; Register GITPtrLoReg = MFI -> getGITPtrLoReg ( MF ) ; for ( MCPhysReg Reg : AllSGPRs ) { if ( ! MRI . isPhysRegUsed ( Reg ) && MRI . isAllocatable ( Reg ) && ! TRI -> isSubRegisterEq ( ScratchRsrcReg , Reg ) && GITPtrLoReg != Reg ) { ScratchWaveOffsetReg = Reg ; BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , ScratchWaveOffsetReg ) . addReg ( PreloadedScratchWaveOffsetReg , RegState :: Kill ) ; break ; } } } else { ScratchWaveOffsetReg = PreloadedScratchWaveOffsetReg ; } assert ( ScratchWaveOffsetReg || ! PreloadedScratchWaveOffsetReg ) ; if ( requiresStackPointerReference ( MF ) ) { Register SPReg = MFI -> getStackPtrOffsetReg ( ) ; assert ( SPReg != <STR_LIT> ) ; BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , SPReg

) . addImm ( FrameInfo . getStackSize ( ) * getScratchScaleFactor ( ST ) ) ;


<s> MachineFrameInfo & FrameInfo = MF . getFrameInfo ( ) ; assert ( MFI -> isEntryFunction ( ) ) ; Register PreloadedScratchWaveOffsetReg = MFI -> getPreloadedReg ( <STR_LIT> ) ; Register ScratchRsrcReg ; if ( ! ST . enableFlatScratch ( ) ) ScratchRsrcReg = getEntryFunctionReservedScratchRsrcReg ( MF ) ; if ( ScratchRsrcReg ) { for ( MachineBasicBlock & OtherBB : MF ) { if ( & OtherBB != & MBB ) { OtherBB . addLiveIn ( ScratchRsrcReg ) ; } } } Register PreloadedScratchRsrcReg ; if ( ST . isAmdHsaOrMesa ( F ) ) { PreloadedScratchRsrcReg = MFI -> getPreloadedReg ( <STR_LIT> ) ; if ( ScratchRsrcReg && PreloadedScratchRsrcReg ) { MRI . addLiveIn ( PreloadedScratchRsrcReg ) ; MBB . addLiveIn ( PreloadedScratchRsrcReg ) ; } } DebugLoc DL ; MachineBasicBlock :: iterator I = MBB . begin ( ) ; Register ScratchWaveOffsetReg ; if ( PreloadedScratchWaveOffsetReg && TRI -> isSubRegisterEq ( ScratchRsrcReg , PreloadedScratchWaveOffsetReg ) ) { ArrayRef < MCPhysReg > AllSGPRs = TRI -> getAllSGPR32 ( MF ) ; unsigned NumPreloaded = MFI -> getNumPreloadedSGPRs ( ) ; AllSGPRs = AllSGPRs . slice ( std :: min ( static_cast < unsigned > ( AllSGPRs . size ( ) ) , NumPreloaded ) ) ; Register GITPtrLoReg = MFI -> getGITPtrLoReg ( MF ) ; for ( MCPhysReg Reg : AllSGPRs ) { if ( ! MRI . isPhysRegUsed ( Reg ) && MRI . isAllocatable ( Reg ) && ! TRI -> isSubRegisterEq ( ScratchRsrcReg , Reg ) && GITPtrLoReg != Reg ) { ScratchWaveOffsetReg = Reg ; BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , ScratchWaveOffsetReg ) . addReg ( PreloadedScratchWaveOffsetReg


<s> } } else if ( ST . isMesaGfxShader ( Fn ) || ! PreloadedScratchRsrcReg ) { assert ( ! ST . isAmdHsaOrMesa ( Fn ) ) ; const MCInstrDesc & SMovB32 = TII -> get ( <STR_LIT> ) ; Register Rsrc2 = TRI -> getSubReg ( ScratchRsrcReg , <STR_LIT> ) ; Register Rsrc3 = TRI -> getSubReg ( ScratchRsrcReg , <STR_LIT> ) ; uint64_t Rsrc23 = TII -> getScratchRsrcWords23 ( ) ; if ( MFI -> hasImplicitBufferPtr ( ) ) { Register Rsrc01 = TRI -> getSubReg ( ScratchRsrcReg , <STR_LIT> ) ; if ( <STR_LIT> ( MF . getFunction ( ) . getCallingConv ( ) ) ) { const MCInstrDesc & Mov64 = TII -> get ( <STR_LIT> ) ; BuildMI ( MBB , I , DL , Mov64 , Rsrc01 ) . addReg ( MFI -> getImplicitBufferPtrUserSGPR ( ) ) . addReg ( ScratchRsrcReg , RegState :: ImplicitDefine ) ; } else { const MCInstrDesc & LoadDwordX2 = TII -> get ( <STR_LIT> ) ; MachinePointerInfo PtrInfo ( <ISA_LIT>AS :: CONSTANT_ADDRESS ) ; auto MMO = MF . getMachineMemOperand ( PtrInfo , MachineMemOperand :: MOLoad | MachineMemOperand :: MOInvariant | MachineMemOperand :: MODereferenceable , <NUM_LIT> , Align ( <NUM_LIT> ) ) ; BuildMI ( MBB , I , DL , LoadDwordX2 , Rsrc01 ) . addReg ( MFI -> getImplicitBufferPtrUserSGPR ( ) ) . addImm ( <NUM_LIT> ) . addImm ( <NUM_LIT> ) . addMemOperand ( MMO ) . addReg ( ScratchRsrcReg , RegState :: ImplicitDefine ) ; MF . getRegInfo ( ) . addLiveIn ( MFI -> getImplicitBufferPtrUserSGPR ( ) ) ; MBB . addLiveIn ( MFI -> getImplicitBufferPtrUserSGPR ( ) ) ; } } else { Register Rsrc0 = TRI -> getSubReg ( ScratchRsrcReg , <STR_LIT> ) ; Register Rsrc1 = TRI -> getSubReg ( ScratchRsrcReg , <STR_LIT> ) ; BuildMI ( MBB , I , DL , SMovB32 , Rsrc0 ) . addExternalSymbol ( <STR_LIT> ) . addReg ( ScratchRsrcReg , RegState :: ImplicitDefine ) ; BuildMI ( MBB , I , DL , SMovB32 , Rsrc1 ) . addExternalSymbol ( <STR_LIT> ) . addReg ( ScratchRsrcReg , RegState :: ImplicitDefine ) ; } BuildMI ( MBB , I , DL , SMovB32 , Rsrc2 ) . addImm ( Rsrc23 & <NUM_LIT> ) . addReg ( ScratchRsrcReg , RegState :: ImplicitDefine ) ; BuildMI ( MBB , I , DL , SMovB32 , Rsrc3 ) . addImm ( Rsrc23 >> <NUM_LIT> ) . addReg ( ScratchRsrcReg , RegState :: ImplicitDefine ) ; } else if ( ST . isAmdHsaOrMesa ( Fn ) ) { assert ( PreloadedScratchRsrcReg ) ; if ( ScratchRsrcReg != PreloadedScratchRsrcReg ) { BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , ScratchRsrcReg ) . addReg ( PreloadedScratchRsrcReg , RegState :: Kill ) ; } } Register ScratchRsrcSub0 = TRI -> getSubReg (


<s> const GCNSubtarget & ST = MF . getSubtarget < GCNSubtarget > ( ) ; const <ISA_LIT>InstrInfo * TII = ST . getInstrInfo ( ) ; const <ISA_LIT>RegisterInfo * TRI = & TII -> getRegisterInfo ( ) ; MachineRegisterInfo & MRI = MF . getRegInfo ( ) ; <ISA_LIT>MachineFunctionInfo * MFI = MF . getInfo < <ISA_LIT>MachineFunctionInfo > ( ) ; assert ( MFI -> isEntryFunction ( ) ) ; Register ScratchRsrcReg = MFI -> getScratchRSrcReg ( ) ; if ( ! ScratchRsrcReg || ( ! MRI . isPhysRegUsed ( ScratchRsrcReg ) && allStackObjectsAreDead ( MF . getFrameInfo ( ) ) ) ) return Register ( ) ; if ( ST . hasSGPRInitBug ( ) || ScratchRsrcReg != TRI -> reservedPrivateSegmentBufferReg ( MF ) ) return ScratchRsrcReg ; unsigned NumPreloaded = ( MFI -> getNumPreloadedSGPRs ( ) + <NUM_LIT> ) / <NUM_LIT> ; ArrayRef < MCPhysReg > AllSGPR128s = TRI -> getAllSGPR128 ( MF ) ; AllSGPR128s = AllSGPR128s . slice ( std :: min ( static_cast < unsigned > ( AllSGPR128s . size ( ) ) , NumPreloaded ) ) ; Register GITPtrLoReg = MFI -> getGITPtrLoReg ( MF ) ; for ( MCPhysReg Reg : AllSGPR128s ) { if ( ! MRI . isPhysRegUsed ( Reg ) && MRI . isAllocatable ( Reg ) && ! TRI -> isSubRegisterEq ( Reg , GITPtrLoReg ) ) { MRI . replaceRegWith ( ScratchRsrcReg , Reg ) ; MFI -> setScratchRSrcReg ( Reg ) ; return Reg ; } } return


<s> void <ISA_LIT>FrameLowering :: determineCalleeSaves ( MachineFunction & MF , BitVector & SavedRegs , RegScavenger * RS ) const { TargetFrameLowering :: determineCalleeSaves ( MF , SavedRegs , RS ) ; const <ISA_LIT>MachineFunctionInfo * MFI = MF . getInfo <


<s> Amount = alignTo ( Amount , Align ) ; assert ( isUInt < <NUM_LIT> > ( Amount ) && <STR_LIT> ) ; const <ISA_LIT>MachineFunctionInfo * MFI = MF . getInfo < <ISA_LIT>MachineFunctionInfo > ( ) ; unsigned SPReg = MFI -> getStackPtrOffsetReg ( ) ; unsigned Op = IsDestroy ? <STR_LIT> : <STR_LIT> ; BuildMI ( MBB , I , DL , TII -> get ( Op ) , SPReg ) . addReg ( SPReg ) . addImm ( Amount * ST


<s> int64_t Amount = I -> getOperand ( <NUM_LIT> ) . getImm ( ) ; if ( Amount == <NUM_LIT> ) return MBB . erase ( I ) ; const <ISA_LIT>Subtarget & ST = MF . getSubtarget < <ISA_LIT>Subtarget > ( ) ; const <ISA_LIT>InstrInfo * TII = ST . getInstrInfo ( ) ; const DebugLoc & DL = I -> getDebugLoc ( ) ; unsigned Opc = I -> getOpcode ( ) ; bool IsDestroy = Opc == TII -> getCallFrameDestroyOpcode ( ) ; uint64_t CalleePopAmount = IsDestroy ? I -> getOperand ( <NUM_LIT> ) . getImm ( ) : <NUM_LIT> ; const TargetFrameLowering * TFI = MF . getSubtarget


<s> const <ISA_LIT>MachineFunctionInfo * MFI = MF . getInfo < <ISA_LIT>MachineFunctionInfo > ( ) ; DebugLoc DL ; MachineBasicBlock :: iterator I = MBB . begin ( ) ; unsigned FlatScratchInitReg = MFI -> getPreloadedReg ( <STR_LIT> ) ; MachineRegisterInfo & MRI = MF . getRegInfo ( ) ; MRI . addLiveIn ( FlatScratchInitReg ) ; MBB . addLiveIn ( FlatScratchInitReg ) ; unsigned FlatScrInitLo = TRI -> getSubReg ( FlatScratchInitReg , <STR_LIT> ) ; unsigned FlatScrInitHi = TRI -> getSubReg ( FlatScratchInitReg , <STR_LIT> ) ; unsigned ScratchWaveOffsetReg = MFI -> getScratchWaveOffsetReg ( ) ; if ( ST . flatScratchIsPointer ( ) ) { BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , <STR_LIT> ) . addReg ( FlatScrInitLo ) . addReg ( ScratchWaveOffsetReg ) ; BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , <STR_LIT> ) . addReg ( FlatScrInitHi ) . addImm ( <NUM_LIT> ) ; return ; } BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , <STR_LIT> ) . addReg ( FlatScrInitHi


<s> const MachineFrameInfo & MFI = MF . getFrameInfo ( ) ; const <ISA_LIT>Subtarget & ST = MF . getSubtarget < <ISA_LIT>Subtarget > ( ) ; const <ISA_LIT>InstrInfo * TII = ST . getInstrInfo ( ) ; unsigned StackPtrReg = FuncInfo -> getStackPtrOffsetReg ( ) ; unsigned FramePtrReg = FuncInfo -> getFrameOffsetReg ( ) ; MachineBasicBlock :: iterator MBBI = MBB . begin ( ) ; DebugLoc DL ; bool NeedFP = hasFP ( MF ) ; if ( NeedFP ) { BuildMI ( MBB , MBBI , DL , TII -> get ( <STR_LIT> ) , FramePtrReg ) . addReg ( StackPtrReg ) . setMIFlag ( MachineInstr :: FrameSetup ) ; } uint32_t NumBytes = MFI . getStackSize ( ) ; if ( NumBytes != <NUM_LIT> && hasSP ( MF ) ) { BuildMI ( MBB , MBBI , DL , TII -> get ( <STR_LIT> ) , StackPtrReg ) . addReg ( StackPtrReg ) . addImm ( NumBytes * ST . getWavefrontSize ( ) ) . setMIFlag ( MachineInstr :: FrameSetup ) ; } for ( const <ISA_LIT>MachineFunctionInfo :: SGPRSpillVGPRCSR & Reg : FuncInfo -> getSGPRSpillVGPRs ( ) ) { if ( ! Reg . FI . hasValue ( ) ) continue ; TII -> storeRegToStackSlot ( MBB , MBBI , Reg . VGPR , true , Reg . FI . getValue ( ) , & <STR_LIT> , & TII


<s> MachineRegisterInfo & MRI = MF . getRegInfo ( ) ; unsigned ScratchWaveOffsetReg = MFI -> getScratchWaveOffsetReg ( ) ; if ( ScratchWaveOffsetReg == <STR_LIT> || ! MRI . isPhysRegUsed ( ScratchWaveOffsetReg ) ) { assert ( MFI -> getStackPtrOffsetReg ( ) == <STR_LIT> ) ; return std :: make_pair ( <STR_LIT> , <STR_LIT> ) ; } unsigned SPReg = MFI -> getStackPtrOffsetReg ( ) ; if ( ST . hasSGPRInitBug ( ) ) return std :: make_pair ( ScratchWaveOffsetReg , SPReg ) ; unsigned NumPreloaded = MFI -> getNumPreloadedSGPRs ( ) ; ArrayRef < MCPhysReg > AllSGPRs = getAllSGPRs ( ST , MF ) ; if ( NumPreloaded > AllSGPRs . size ( ) ) return std :: make_pair ( ScratchWaveOffsetReg , SPReg ) ; AllSGPRs = AllSGPRs . slice ( NumPreloaded ) ; unsigned ReservedRegCount = <NUM_LIT> ; if ( AllSGPRs . size ( ) < ReservedRegCount ) return std :: make_pair ( ScratchWaveOffsetReg , SPReg ) ; bool HandledScratchWaveOffsetReg = ScratchWaveOffsetReg != TRI -> reservedPrivateSegmentWaveByteOffsetReg ( MF ) ; for ( MCPhysReg Reg : AllSGPRs . drop_back ( ReservedRegCount ) ) { if ( ! MRI . isPhysRegUsed ( Reg ) && MRI . isAllocatable ( Reg ) ) { if ( ! HandledScratchWaveOffsetReg ) { HandledScratchWaveOffsetReg = true ; MRI . replaceRegWith ( ScratchWaveOffsetReg , Reg ) ; MFI -> setScratchWaveOffsetReg ( Reg ) ; ScratchWaveOffsetReg = Reg ; break ; } } } return std :: make_pair ( ScratchWaveOffsetReg ,


<s> bool CopyBuffer = ResourceRegUsed && PreloadedPrivateBufferReg != <STR_LIT> && ST . isAmdCodeObjectV2 ( MF ) && ScratchRsrcReg != PreloadedPrivateBufferReg ; bool CopyBufferFirst = TRI -> isSubRegisterEq ( PreloadedPrivateBufferReg , ScratchWaveOffsetReg ) ; if ( CopyBuffer && CopyBufferFirst ) { BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , ScratchRsrcReg ) . addReg ( PreloadedPrivateBufferReg , RegState :: Kill ) ; } if ( OffsetRegUsed && PreloadedScratchWaveOffsetReg != ScratchWaveOffsetReg ) { BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , ScratchWaveOffsetReg ) . addReg ( PreloadedScratchWaveOffsetReg , RegState :: Kill ) ; } if ( CopyBuffer && ! CopyBufferFirst ) { BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , ScratchRsrcReg ) . addReg ( PreloadedPrivateBufferReg , RegState :: Kill ) ; } if ( ResourceRegUsed && ( ST . isMesaGfxShader ( MF ) || ( PreloadedPrivateBufferReg == <STR_LIT> ) ) ) { assert ( ! ST . isAmdCodeObjectV2 ( MF ) ) ; const MCInstrDesc & SMovB32 = TII -> get ( <STR_LIT> ) ; unsigned Rsrc2 = TRI -> getSubReg ( ScratchRsrcReg , <STR_LIT> ) ; unsigned Rsrc3 = TRI -> getSubReg ( ScratchRsrcReg , <STR_LIT> ) ; uint64_t Rsrc23 = TII -> getScratchRsrcWords23 ( ) ; if ( MFI -> hasPrivateMemoryInputPtr ( ) ) { unsigned Rsrc01 = TRI -> getSubReg ( ScratchRsrcReg , <STR_LIT> ) ; if ( <STR_LIT> ( MF . getFunction ( ) -> getCallingConv ( ) ) ) { const MCInstrDesc & Mov64 = TII -> get ( <STR_LIT> ) ; BuildMI ( MBB , I , DL , Mov64 , Rsrc01 ) . addReg ( PreloadedPrivateBufferReg ) . addReg ( ScratchRsrcReg , RegState :: ImplicitDefine ) ; } else { const MCInstrDesc & LoadDwordX2 = TII -> get ( <STR_LIT> ) ; PointerType * PtrTy = PointerType :: get ( Type :: getInt64Ty ( MF . getFunction ( ) -> getContext ( ) ) , AMDGPUA<ISA_LIT> . CONSTANT_ADDRESS ) ; MachinePointerInfo PtrInfo ( UndefValue :: get ( PtrTy ) ) ; auto MMO = MF . getMachineMemOperand ( PtrInfo , MachineMemOperand :: MOLoad | MachineMemOperand :: MOInvariant | MachineMemOperand :: MODereferenceable , <NUM_LIT> , <NUM_LIT> ) ; BuildMI ( MBB , I , DL , LoadDwordX2 , Rsrc01 ) . addReg ( PreloadedPrivateBufferReg ) . addImm ( <NUM_LIT> ) . addImm ( <NUM_LIT> ) . addMemOperand ( MMO ) . addReg ( ScratchRsrcReg , RegState :: ImplicitDefine ) ; } } else { unsigned Rsrc0 = TRI -> getSubReg (


<s> } } MF . getRegInfo ( ) . addLiveIn ( GitPtrLo ) ; MF . front ( ) . addLiveIn ( GitPtrLo ) ; BuildMI ( MBB , I , DL , SMovB32 , RsrcLo ) . addReg ( GitPtrLo ) . addReg ( ScratchRsrcReg , RegState :: ImplicitDefine ) ; PointerType * PtrTy = PointerType :: get ( Type :: getInt64Ty ( MF . getFunction ( ) . getContext ( ) ) , <ISA_LIT>AS :: CONSTANT_ADDRESS ) ; MachinePointerInfo PtrInfo ( UndefValue :: get ( PtrTy ) ) ; const MCInstrDesc & LoadDwordX4 = TII -> get ( <STR_LIT> ) ; auto MMO = MF . getMachineMemOperand ( PtrInfo , MachineMemOperand :: MOLoad | MachineMemOperand :: MOInvariant | MachineMemOperand :: MODereferenceable , <NUM_LIT> , <NUM_LIT> ) ; unsigned Offset = MF . getFunction ( ) . getCallingConv ( ) == CallingConv :: <ISA_LIT>_CS ? <NUM_LIT> : <NUM_LIT> ; BuildMI ( MBB , I , DL , LoadDwordX4 , ScratchRsrcReg ) . addReg ( Rsrc01 ) . addImm ( Offset ) . addImm ( <NUM_LIT> ) . addReg ( ScratchRsrcReg , RegState :: ImplicitDefine ) . addMemOperand ( MMO ) ; return ; } if ( ST . isMesaGfxShader ( MF ) || ( PreloadedPrivateBufferReg == <STR_LIT> ) ) { assert ( ! ST . isAmdCodeObjectV2 ( MF ) ) ; const MCInstrDesc & SMovB32 = TII -> get ( <STR_LIT> ) ; unsigned Rsrc2 = TRI -> getSubReg ( ScratchRsrcReg , <STR_LIT> ) ; unsigned Rsrc3 = TRI -> getSubReg ( ScratchRsrcReg , <STR_LIT> ) ; uint64_t Rsrc23 = TII -> getScratchRsrcWords23 ( ) ; if ( MFI -> hasImplicitBufferPtr ( ) ) { unsigned Rsrc01 = TRI -> getSubReg ( ScratchRsrcReg , <STR_LIT> ) ; if ( <STR_LIT> ( MF . getFunction ( ) . getCallingConv ( ) ) ) { const MCInstrDesc & Mov64 = TII -> get ( <STR_LIT> ) ; BuildMI ( MBB , I , DL , Mov64 , Rsrc01 ) . addReg ( MFI -> getImplicitBufferPtrUserSGPR ( ) ) . addReg ( ScratchRsrcReg , RegState :: ImplicitDefine ) ; } else { const MCInstrDesc & LoadDwordX2 = TII -> get ( <STR_LIT> ) ; PointerType * PtrTy = PointerType :: get ( Type :: getInt64Ty ( MF . getFunction ( ) . getContext ( ) ) , <ISA_LIT>AS :: CONSTANT_ADDRESS ) ; MachinePointerInfo PtrInfo ( UndefValue :: get ( PtrTy ) ) ; auto MMO = MF . getMachineMemOperand ( PtrInfo , MachineMemOperand :: MOLoad | MachineMemOperand :: MOInvariant | MachineMemOperand :: MODereferenceable , <NUM_LIT> , <NUM_LIT> ) ; BuildMI ( MBB , I , DL , LoadDwordX2 , Rsrc01 ) . addReg ( MFI -> getImplicitBufferPtrUserSGPR ( ) ) . addImm ( <NUM_LIT> ) . addImm ( <NUM_LIT> ) . addMemOperand ( MMO ) . addReg ( ScratchRsrcReg , RegState :: ImplicitDefine ) ; } } else


<s> const <ISA_LIT>Subtarget & ST = MF . getSubtarget < <ISA_LIT>Subtarget > ( ) ; const <ISA_LIT>InstrInfo * TII = ST . getInstrInfo ( ) ; MachineBasicBlock :: iterator MBBI = MBB . getFirstTerminator ( ) ; for ( const <ISA_LIT>MachineFunctionInfo :: SGPRSpillVGPRCSR & Reg : FuncInfo -> getSGPRSpillVGPRs ( ) ) { if ( ! Reg . FI . hasValue ( ) ) continue ; TII -> loadRegFromStackSlot ( MBB , MBBI , Reg . VGPR , Reg . FI . getValue ( ) , & <STR_LIT> , & TII -> getRegisterInfo ( ) ) ; } unsigned StackPtrReg = FuncInfo -> getStackPtrOffsetReg ( ) ; if ( StackPtrReg == <STR_LIT> ) return ; const MachineFrameInfo & MFI =


<s> unsigned StackPtrReg = FuncInfo -> getStackPtrOffsetReg ( ) ; if ( StackPtrReg == <STR_LIT> ) return ; const MachineFrameInfo & MFI = MF . getFrameInfo ( ) ; uint32_t NumBytes = MFI . getStackSize ( ) ; DebugLoc DL ; if ( NumBytes != <NUM_LIT> && hasSP ( MF ) ) { uint32_t RoundedSize = FuncInfo -> isStackRealigned ( ) ? NumBytes + MFI . getMaxAlignment ( ) : NumBytes ; BuildMI ( MBB , MBBI , DL , TII -> get ( <STR_LIT> ) , StackPtrReg ) . addReg ( StackPtrReg ) . addImm ( RoundedSize * ST


<s> } const MachineFrameInfo & MFI = MF . getFrameInfo ( ) ; const <ISA_LIT>Subtarget & ST = MF . getSubtarget < <ISA_LIT>Subtarget > ( ) ; const <ISA_LIT>InstrInfo * TII = ST . getInstrInfo ( ) ; const <ISA_LIT>RegisterInfo & TRI = TII -> getRegisterInfo ( ) ; unsigned StackPtrReg = FuncInfo -> getStackPtrOffsetReg ( ) ; unsigned FramePtrReg = FuncInfo -> getFrameOffsetReg ( ) ; MachineBasicBlock :: iterator MBBI = MBB . begin ( ) ; DebugLoc DL ; bool NeedFP = hasFP ( MF ) ; uint32_t NumBytes = MFI . getStackSize ( ) ; uint32_t RoundedSize = NumBytes ; const bool NeedsRealignment = TRI . needsStackRealignment ( MF ) ; if ( NeedsRealignment ) { assert ( NeedFP ) ; const unsigned Alignment = MFI . getMaxAlignment ( ) ; RoundedSize += Alignment ; unsigned ScratchSPReg = findScratchNonCalleeSaveRegister ( MBB ) ; assert ( ScratchSPReg != <STR_LIT> ) ; BuildMI ( MBB , MBBI , DL , TII -> get ( <STR_LIT> ) , ScratchSPReg ) . addReg ( StackPtrReg ) . addImm ( ( Alignment - <NUM_LIT> ) * ST . getWavefrontSize ( ) ) . setMIFlag ( MachineInstr :: FrameSetup ) ; BuildMI ( MBB , MBBI , DL , TII -> get ( <STR_LIT> ) , FramePtrReg ) . addReg ( ScratchSPReg , RegState :: Kill ) . addImm (

- Alignment * ST . getWavefrontSize ( ) ) . setMIFlag ( MachineInstr :: FrameSetup ) ;


<s> MachineFunction * MF = MBB . getParent ( ) ; const <ISA_LIT>Subtarget & Subtarget = MF -> getSubtarget < <ISA_LIT>Subtarget > ( ) ; const <ISA_LIT>RegisterInfo & TRI = * Subtarget . getRegisterInfo ( ) ; LivePhysRegs LiveRegs (


<s> <ISA_LIT>MachineFunctionInfo * FuncInfo = MF . getInfo < <ISA_LIT>MachineFunctionInfo > ( ) ; bool AllSGPRSpilledToVGPRs = false ; if ( TRI . spillSGPRToVGPR ( ) && FuncInfo -> hasSpilledSGPRs ( ) ) { AllSGPRSpilledToVGPRs = true ; for ( MachineBasicBlock & MBB : MF ) { MachineBasicBlock :: iterator Next ; for ( auto I = MBB . begin ( ) , E = MBB . end ( ) ; I != E ; I = Next ) { MachineInstr & MI = * I ; Next = std :: next ( I ) ; if ( TII -> isSGPRSpill ( MI ) ) { int FI = TII -> getNamedOperand ( MI , <STR_LIT> :: addr ) -> getIndex ( ) ; assert ( MFI . getStackID ( FI ) == <STR_LIT> ) ; if ( FuncInfo -> allocateSGPRSpillToVGPR ( MF , FI ) ) { bool Spilled = TRI . eliminateSGPRToVGPRSpillFrameIndex ( MI , FI , RS ) ; ( void ) Spilled ; assert ( Spilled && <STR_LIT> ) ; } else AllSGPRSpilledToVGPRs = false ; } } } FuncInfo -> removeSGPRToVGPRFrameIndices ( MFI ) ; } if ( FuncInfo -> hasNonSpillStackObjects ( ) || FuncInfo -> hasSpilledVGPRs ( ) || ! AllSGPRSpilledToVGPRs || ! allStackObjectsAreDead ( MFI ) ) { assert ( RS && <STR_LIT> ) ; int ScavengeFI = MFI . CreateFixedObject ( TRI . getSpillSize ( <STR_LIT> ) , <NUM_LIT> , false ) ; RS -> addScavengingFrameIndex


<s> Next = std :: next ( I ) ; if ( TII -> isSGPRSpill ( MI ) ) { int FI = TII -> getNamedOperand ( MI , <STR_LIT> :: addr ) -> getIndex ( ) ; assert ( MFI . getStackID ( FI ) == <STR_LIT> ) ; if ( FuncInfo -> allocateSGPRSpillToVGPR ( MF , FI ) ) { bool Spilled = TRI . eliminateSGPRToVGPRSpillFrameIndex ( MI , FI , RS ) ; ( void ) Spilled ; assert ( Spilled && <STR_LIT> ) ; } else AllSGPRSpilledToVGPRs = false ; } } } FuncInfo -> removeSGPRToVGPRFrameIndices ( MFI )


<s> const <ISA_LIT>MachineFunctionInfo * MFI = MF . getInfo < <ISA_LIT>MachineFunctionInfo > ( ) ; unsigned SPReg = MFI -> getStackPtrOffsetReg ( ) ; unsigned Op = IsDestroy ? <STR_LIT> : <STR_LIT> ; BuildMI ( MBB , I , DL , TII -> get ( Op ) , SPReg ) . addReg ( SPReg ) . addImm ( Amount * ST . getWavefrontSize ( ) ) ; } else if ( CalleePopAmount != <NUM_LIT> ) { llvm_unreachable ( <STR_LIT> ) ; } return MBB . erase ( I


<s> MachineBasicBlock :: iterator <ISA_LIT>FrameLowering :: eliminateCallFramePseudoInstr ( MachineFunction & MF , MachineBasicBlock & MBB , MachineBasicBlock :: iterator I ) const { int64_t Amount = I -> getOperand ( <NUM_LIT> ) . getImm ( ) ; if ( Amount == <NUM_LIT> ) return MBB . erase ( I ) ; const GCNSubtarget & ST = MF . getSubtarget < GCNSubtarget > ( ) ; const <ISA_LIT>InstrInfo * TII = ST . getInstrInfo ( ) ; const DebugLoc & DL = I -> getDebugLoc ( ) ; unsigned Opc = I -> getOpcode ( ) ; bool IsDestroy = Opc == TII -> getCallFrameDestroyOpcode ( ) ; uint64_t CalleePopAmount = IsDestroy ? I -> getOperand ( <NUM_LIT> ) . getImm ( ) : <NUM_LIT> ; const TargetFrameLowering * TFI = MF . getSubtarget ( ) . getFrameLowering ( ) ; if ( ! TFI -> hasReservedCallFrame ( MF ) ) { unsigned Align = getStackAlignment ( ) ; Amount = alignTo ( Amount , Align ) ; assert ( isUInt < <NUM_LIT> > ( Amount ) && <STR_LIT> ) ; const <ISA_LIT>MachineFunctionInfo * MFI = MF . getInfo < <ISA_LIT>MachineFunctionInfo > ( ) ; unsigned SPReg = MFI -> getStackPtrOffsetReg ( ) ; unsigned Op = IsDestroy ? <STR_LIT> : <STR_LIT> ; BuildMI ( MBB , I , DL , TII -> get ( Op ) , SPReg ) . addReg

( SPReg ) . addImm ( Amount * ST . getWavefrontSize ( ) ) ;


<s> const <ISA_LIT>MachineFunctionInfo * MFI = MF . getInfo < <ISA_LIT>MachineFunctionInfo > ( ) ; MachineBasicBlock :: iterator I = MBB . begin ( ) ; DebugLoc DL ; for ( unsigned i = <NUM_LIT> ; i < <NUM_LIT> ; ++ i ) { unsigned WorkGroupIDSGPR = MFI -> getWorkGroupIDSGPR ( i ) ; MF . getRegInfo ( ) . addLiveIn ( WorkGroupIDSGPR ) ; MBB . addLiveIn ( WorkGroupIDSGPR ) ; unsigned WorkGroupIDVGPR = MF . getRegInfo ( ) . createVirtualRegister ( & <STR_LIT> ) ; BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , WorkGroupIDVGPR ) . addReg ( WorkGroupIDSGPR ) ; int WorkGroupIDObjectIdx = MFI -> getDebuggerWorkGroupIDStackObjectIndex ( i ) ; TII -> storeRegToStackSlot ( MBB , I , WorkGroupIDVGPR , false , WorkGroupIDObjectIdx , & <STR_LIT> , TRI ) ; unsigned WorkItemIDVGPR = MFI -> getWorkItemIDVGPR ( i ) ; MF . getRegInfo ( ) . addLiveIn ( WorkItemIDVGPR ) ; MBB . addLiveIn ( WorkItemIDVGPR ) ; int WorkItemIDObjectIdx = MFI -> getDebuggerWorkItemIDStackObjectIndex (


<s> MF . getRegInfo ( ) . addLiveIn ( WorkGroupIDSGPR ) ; MBB . addLiveIn ( WorkGroupIDSGPR ) ; unsigned WorkGroupIDVGPR = MF . getRegInfo ( ) . createVirtualRegister ( & <STR_LIT> ) ; BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , WorkGroupIDVGPR ) . addReg ( WorkGroupIDSGPR ) ; int WorkGroupIDObjectIdx = MFI -> getDebuggerWorkGroupIDStackObjectIndex ( i ) ; TII -> storeRegToStackSlot ( MBB , I , WorkGroupIDVGPR , false , WorkGroupIDObjectIdx , & <STR_LIT> , TRI ) ; unsigned WorkItemIDVGPR = MFI -> getWorkItemIDVGPR ( i ) ; MF . getRegInfo ( )


<s> const GCNSubtarget & ST = MF . getSubtarget < GCNSubtarget > ( ) ; const <ISA_LIT>InstrInfo * TII = ST . getInstrInfo ( ) ; MachineBasicBlock :: iterator MBBI = MBB . getFirstTerminator ( ) ; for ( const <ISA_LIT>MachineFunctionInfo :: SGPRSpillVGPRCSR & Reg : FuncInfo -> getSGPRSpillVGPRs ( ) ) { if ( ! Reg . FI . hasValue ( ) ) continue ; TII -> loadRegFromStackSlot ( MBB , MBBI , Reg . VGPR , Reg . FI . getValue ( ) , & <STR_LIT> , & TII -> getRegisterInfo ( ) ) ; } unsigned StackPtrReg = FuncInfo -> getStackPtrOffsetReg ( ) ; if ( StackPtrReg ==


<s> unsigned FramePtrReg = FuncInfo -> getFrameOffsetReg ( ) ; MachineBasicBlock :: iterator MBBI = MBB . begin ( ) ; DebugLoc DL ; bool NeedFP = hasFP ( MF ) ; uint32_t NumBytes = MFI . getStackSize ( ) ; uint32_t RoundedSize = NumBytes ; const bool NeedsRealignment = TRI . needsStackRealignment ( MF ) ; if ( NeedsRealignment ) { assert ( NeedFP ) ; const unsigned Alignment = MFI . getMaxAlignment ( ) ; RoundedSize += Alignment ; unsigned ScratchSPReg = findScratchNonCalleeSaveRegister ( MBB ) ; assert ( ScratchSPReg != <STR_LIT> ) ; BuildMI ( MBB , MBBI , DL , TII -> get ( <STR_LIT> ) , ScratchSPReg ) . addReg ( StackPtrReg ) . addImm ( ( Alignment - <NUM_LIT> ) * ST . getWavefrontSize ( ) ) . setMIFlag ( MachineInstr :: FrameSetup ) ; BuildMI ( MBB , MBBI , DL , TII -> get ( <STR_LIT> ) , FramePtrReg ) . addReg ( ScratchSPReg , RegState :: Kill ) . addImm ( - Alignment *


<s> static ArrayRef < MCPhysReg > getAllSGPR128 ( const GCNSubtarget &


<s> return makeArrayRef ( <STR_LIT> . begin ( ) , ST . getMaxNumSGPRs ( MF ) /


<s> static ArrayRef < MCPhysReg > getAllSGPRs ( const GCNSubtarget & ST , const


<s> FrameReg = RI -> getFrameRegister ( MF ) ; return MF . getFrameInfo ( ) .


<s> FrameReg = RI -> getFrameRegister ( MF ) ; return MF . getFrameInfo ( ) . getObjectOffset


<s> if ( ScratchRsrcReg == <STR_LIT> || ! MRI . isPhysRegUsed ( ScratchRsrcReg ) ) return <STR_LIT> ; if ( ST . hasSGPRInitBug ( ) || ScratchRsrcReg != TRI -> reservedPrivateSegmentBufferReg ( MF ) ) return ScratchRsrcReg ; unsigned NumPreloaded = ( MFI -> getNumPreloadedSGPRs ( ) + <NUM_LIT> ) / <NUM_LIT> ; ArrayRef < MCPhysReg > AllSGPR128s = getAllSGPR128 ( ST , MF ) ; AllSGPR128s = AllSGPR128s . slice ( std :: min ( static_cast < unsigned > ( AllSGPR128s . size ( ) ) , NumPreloaded ) ) ; for ( MCPhysReg Reg : AllSGPR128s ) { if ( ! MRI . isPhysRegUsed ( Reg ) && MRI . isAllocatable ( Reg ) ) { MRI . replaceRegWith ( ScratchRsrcReg


<s> unsigned ReservedRegCount = <NUM_LIT> ; if ( AllSGPRs . size ( ) < ReservedRegCount ) return std :: make_pair ( ScratchWaveOffsetReg , SPReg ) ; bool HandledScratchWaveOffsetReg = ScratchWaveOffsetReg != TRI -> reservedPrivateSegmentWaveByteOffsetReg ( MF ) ; for ( MCPhysReg Reg : AllSGPRs . drop_back ( ReservedRegCount ) ) { if ( ! MRI . isPhysRegUsed ( Reg ) && MRI . isAllocatable ( Reg ) ) { if ( ! HandledScratchWaveOffsetReg ) { HandledScratchWaveOffsetReg = true ; MRI . replaceRegWith ( ScratchWaveOffsetReg , Reg ) ; MFI -> setScratchWaveOffsetReg ( Reg ) ; ScratchWaveOffsetReg = Reg ; break ; } } } return std :: make_pair ( ScratchWaveOffsetReg ,


<s> const MachineFrameInfo & MFI = MF . getFrameInfo ( ) ; return MFI . hasCalls ( ) || MFI . hasVarSizedObjects ( ) || TRI


<s> if ( <STR_LIT> ( MF . getFunction ( ) . getCallingConv ( ) ) ) { const MCInstrDesc & Mov64 = TII -> get ( <STR_LIT> ) ; BuildMI ( MBB , I , DL , Mov64 , Rsrc01 ) . addReg ( MFI -> getImplicitBufferPtrUserSGPR ( ) ) . addReg ( ScratchRsrcReg , RegState :: ImplicitDefine ) ; } else { const MCInstrDesc & LoadDwordX2 = TII -> get ( <STR_LIT> ) ; PointerType * PtrTy = PointerType :: get ( Type :: getInt64Ty ( MF . getFunction ( ) . getContext ( ) ) , <ISA_LIT>AS :: CONSTANT_ADDRESS ) ; MachinePointerInfo PtrInfo ( UndefValue :: get ( PtrTy ) ) ; auto MMO = MF . getMachineMemOperand ( PtrInfo , MachineMemOperand :: MOLoad | MachineMemOperand :: MOInvariant | MachineMemOperand :: MODereferenceable , <NUM_LIT> , <NUM_LIT> ) ; BuildMI ( MBB , I , DL , LoadDwordX2 , Rsrc01 ) . addReg ( MFI -> getImplicitBufferPtrUserSGPR ( ) ) . addImm ( <NUM_LIT> ) . addImm ( <NUM_LIT> ) . addMemOperand ( MMO ) . addReg ( ScratchRsrcReg , RegState :: ImplicitDefine ) ; } } else { unsigned Rsrc0 = TRI -> getSubReg ( ScratchRsrcReg , <STR_LIT> ) ; unsigned Rsrc1 = TRI -> getSubReg ( ScratchRsrcReg , <STR_LIT> ) ; BuildMI ( MBB , I , DL , SMovB32 , Rsrc0 ) . addExternalSymbol ( <STR_LIT> ) . addReg ( ScratchRsrcReg , RegState :: ImplicitDefine ) ; BuildMI ( MBB , I , DL , SMovB32 , Rsrc1 ) . addExternalSymbol ( <STR_LIT> ) . addReg ( ScratchRsrcReg , RegState :: ImplicitDefine ) ; } BuildMI ( MBB , I , DL , SMovB32 , Rsrc2 ) . addImm ( Rsrc23 & <NUM_LIT>


<s> } Register PreloadedScratchRsrcReg ; if ( ST . isAmdHsaOrMesa ( F ) ) { PreloadedScratchRsrcReg = MFI -> getPreloadedReg ( <STR_LIT> ) ; if ( ScratchRsrcReg && PreloadedScratchRsrcReg ) { MRI . addLiveIn ( PreloadedScratchRsrcReg ) ; MBB . addLiveIn ( PreloadedScratchRsrcReg ) ; } } DebugLoc DL ; MachineBasicBlock :: iterator I = MBB . begin ( ) ; Register ScratchWaveOffsetReg ; if ( TRI -> isSubRegisterEq ( ScratchRsrcReg , PreloadedScratchWaveOffsetReg ) ) { ArrayRef < MCPhysReg > AllSGPRs = getAllSGPRs ( ST , MF ) ; unsigned NumPreloaded = MFI -> getNumPreloadedSGPRs ( ) ; AllSGPRs = AllSGPRs . slice ( std :: min ( static_cast < unsigned > ( AllSGPRs . size ( ) ) , NumPreloaded ) ) ; Register GITPtrLoReg = MFI -> getGITPtrLoReg ( MF ) ; for ( MCPhysReg Reg : AllSGPRs ) { if ( ! MRI . isPhysRegUsed ( Reg ) && MRI . isAllocatable ( Reg ) && ! TRI -> isSubRegisterEq ( ScratchRsrcReg , Reg ) && GITPtrLoReg != Reg ) { ScratchWaveOffsetReg = Reg ; BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , ScratchWaveOffsetReg ) . addReg ( PreloadedScratchWaveOffsetReg , RegState :: Kill ) ; break ; } } } else { ScratchWaveOffsetReg = PreloadedScratchWaveOffsetReg ; } assert ( ScratchWaveOffsetReg )


<s> if ( ! MFI . isDeadObjectIndex ( I ) && MFI . getStackID ( I ) == TargetStackID :: SGPRSpill && ( ( FramePointerSaveIndex && I != FramePointerSaveIndex ) || ( BasePointerSaveIndex && I != BasePointerSaveIndex ) ) ) { return


<s> if ( ! MFI . isDeadObjectIndex ( I ) ) return false ; } return true


<s> int64_t Offset = MFI . getObjectOffset ( FI ) ; MachineMemOperand * MMO = MF -> getMachineMemOperand ( MachinePointerInfo :: getFixedStack ( * MF , FI ) , MachineMemOperand :: MOLoad , <NUM_LIT> , MFI . getObjectAlign ( FI ) ) ; if ( isUInt < <NUM_LIT> > ( Offset ) ) { BuildMI ( MBB , I , DebugLoc ( ) , TII -> get ( <STR_LIT> ) , SpillReg ) . addReg ( ScratchRsrcReg ) . addReg ( SPReg ) . addImm ( Offset ) . addImm ( <NUM_LIT> ) . addImm ( <NUM_LIT> ) . addImm ( <NUM_LIT> ) . addImm ( <NUM_LIT> ) . addImm


<s> static void buildPrologSpill ( LivePhysRegs & LiveRegs , MachineBasicBlock & MBB , MachineBasicBlock :: iterator I , const <ISA_LIT>InstrInfo * TII , Register SpillReg , Register ScratchRsrcReg , Register SPReg , int FI ) { MachineFunction * MF = MBB . getParent ( ) ; MachineFrameInfo & MFI = MF -> getFrameInfo ( ) ; int64_t Offset = MFI . getObjectOffset ( FI ) ; MachineMemOperand * MMO = MF -> getMachineMemOperand ( MachinePointerInfo :: getFixedStack ( * MF , FI ) ,

MachineMemOperand :: MOStore , <NUM_LIT> , MFI . getObjectAlign ( FI ) ) ;


<s> if ( LiveRegs . empty ( ) ) { if ( IsProlog ) { LiveRegs . init ( TRI ) ; LiveRegs . addLiveIns ( MBB ) ; if ( FuncInfo -> SGPRForFPSaveRestoreCopy ) LiveRegs . removeReg ( FuncInfo -> SGPRForFPSaveRestoreCopy ) ; if ( FuncInfo -> SGPRForBPSaveRestoreCopy ) LiveRegs . removeReg ( FuncInfo -> SGPRForBPSaveRestoreCopy ) ; } else { LiveRegs . init ( * ST . getRegisterInfo ( ) ) ; LiveRegs . addLiveOuts (


<s> LiveRegs . init ( * ST . getRegisterInfo ( ) ) ; LiveRegs . addLiveOuts ( MBB ) ; LiveRegs . stepBackward ( * MBBI ) ; } } ScratchExecCopy = findScratchNonCalleeSaveRegister ( MRI , LiveRegs , * TRI . getWaveMaskRegClass ( ) ) ; if ( ! IsProlog ) LiveRegs . removeReg ( ScratchExecCopy ) ; const unsigned OrSaveExec = ST . isWave32 ( ) ? <STR_LIT> : <STR_LIT> ; BuildMI ( MBB , MBBI , DL , TII -> get ( OrSaveExec ) , ScratchExecCopy ) . addImm ( - <NUM_LIT> ) ; return


<s> SavedVGPRs . clearBitsNotInMask ( TRI -> getAllVGPRRegMask ( ) ) ; const bool WillHaveFP = FrameInfo . hasCalls ( ) && ( SavedVGPRs . any ( ) || ! allStackObjectsAreDead ( FrameInfo ) ) ; for ( auto SSpill : MFI -> getSGPRSpillVGPRs ( ) ) SavedVGPRs . reset ( SSpill . VGPR ) ; LivePhysRegs LiveRegs ; LiveRegs . init ( * TRI ) ; if ( WillHaveFP || hasFP ( MF


<s> const <ISA_LIT>MachineFunctionInfo * MFI = MF . getInfo < <ISA_LIT>MachineFunctionInfo > ( ) ; if ( MFI -> isEntryFunction ( ) ) return ; const GCNSubtarget & ST = MF . getSubtarget < GCNSubtarget > ( ) ; const <ISA_LIT>RegisterInfo * TRI = ST . getRegisterInfo ( ) ; SavedRegs . reset ( MFI ->


<s> MachineBasicBlock :: iterator <ISA_LIT>FrameLowering :: eliminateCallFramePseudoInstr ( MachineFunction & MF , MachineBasicBlock & MBB , MachineBasicBlock :: iterator I ) const { int64_t Amount = I -> getOperand ( <NUM_LIT> ) . getImm ( ) ; if ( Amount == <NUM_LIT> ) return MBB . erase ( I ) ; const GCNSubtarget & ST = MF . getSubtarget < GCNSubtarget > ( ) ; const <ISA_LIT>InstrInfo * TII = ST . getInstrInfo ( ) ; const DebugLoc & DL = I -> getDebugLoc ( ) ; unsigned Opc = I -> getOpcode ( ) ; bool IsDestroy = Opc == TII -> getCallFrameDestroyOpcode ( ) ; uint64_t CalleePopAmount = IsDestroy ? I -> getOperand ( <NUM_LIT> ) . getImm ( ) : <NUM_LIT> ; if ( ! hasReservedCallFrame ( MF


<s> const <ISA_LIT>InstrInfo * TII = ST . getInstrInfo ( ) ; const DebugLoc & DL = I -> getDebugLoc ( ) ; unsigned Opc = I -> getOpcode ( ) ; bool IsDestroy = Opc == TII -> getCallFrameDestroyOpcode ( ) ; uint64_t CalleePopAmount = IsDestroy ? I -> getOperand ( <NUM_LIT> ) . getImm ( ) : <NUM_LIT> ; if ( ! hasReservedCallFrame ( MF ) ) { Amount = alignTo ( Amount , getStackAlign ( ) ) ; assert ( isUInt < <NUM_LIT> > ( Amount ) && <STR_LIT> ) ; const <ISA_LIT>MachineFunctionInfo * MFI = MF . getInfo < <ISA_LIT>MachineFunctionInfo > ( ) ; Register SPReg = MFI -> getStackPtrOffsetReg ( ) ; unsigned Op = IsDestroy ? <STR_LIT> :


<s> BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) ) . addReg ( FlatScrInitLo ) . addImm ( int16_t ( <STR_LIT> :: ID_FLAT_SCR_LO | ( <NUM_LIT> << <STR_LIT> :: WIDTH_M1_SHIFT_ ) ) ) ; BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) ) . addReg ( FlatScrInitHi ) . addImm ( int16_t ( <STR_LIT> :: ID_FLAT_SCR_HI | ( <NUM_LIT> << <STR_LIT> :: WIDTH_M1_SHIFT_ ) ) ) ; return ; } BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , <STR_LIT> ) . addReg ( FlatScrInitLo ) . addReg ( ScratchWaveOffsetReg ) ; BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , <STR_LIT> ) . addReg ( FlatScrInitHi ) . addImm ( <NUM_LIT> ) ; return ; } assert ( ST . getGeneration ( ) < <ISA_LIT>Subtarget :: GFX10 ) ; BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , <STR_LIT> ) . addReg ( FlatScrInitHi , RegState :: Kill ) ; BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , FlatScrInitLo ) . addReg


<s> BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , FlatScrInitLo ) . addReg ( FlatScrInitLo ) . addReg ( ScratchWaveOffsetReg ) ; BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , FlatScrInitHi ) . addReg ( FlatScrInitHi ) . addImm ( <NUM_LIT> ) ; BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) ) . addReg ( FlatScrInitLo ) . addImm ( int16_t ( <STR_LIT> :: ID_FLAT_SCR_LO | ( <NUM_LIT> << <STR_LIT> :: WIDTH_M1_SHIFT_ ) ) ) ; BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) ) . addReg ( FlatScrInitHi ) . addImm ( int16_t ( <STR_LIT> :: ID_FLAT_SCR_HI | ( <NUM_LIT> << <STR_LIT> :: WIDTH_M1_SHIFT_ ) ) ) ; return ; } BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , <STR_LIT> ) . addReg ( FlatScrInitLo ) . addReg ( ScratchWaveOffsetReg ) ; BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , <STR_LIT> ) . addReg ( FlatScrInitHi ) . addImm ( <NUM_LIT> ) ; return ; } assert ( ST . getGeneration ( ) < <ISA_LIT>Subtarget :: GFX10 ) ; BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , <STR_LIT> )

. addReg ( FlatScrInitHi , RegState :: Kill ) ;


<s> assert ( MFI -> isEntryFunction ( ) ) ; Register PreloadedScratchWaveOffsetReg = MFI -> getPreloadedReg ( <STR_LIT> ) ; if ( ! PreloadedScratchWaveOffsetReg ) return ; Register ScratchRsrcReg = getEntryFunctionReservedScratchRsrcReg ( MF ) ; if ( ScratchRsrcReg ) { for ( MachineBasicBlock & OtherBB : MF ) { if ( & OtherBB != & MBB ) { OtherBB . addLiveIn ( ScratchRsrcReg ) ; } } } Register PreloadedScratchRsrcReg ; if ( ST . isAmdHsaOrMesa ( F ) ) { PreloadedScratchRsrcReg = MFI -> getPreloadedReg ( <STR_LIT> ) ; if ( ScratchRsrcReg && PreloadedScratchRsrcReg ) { MRI . addLiveIn ( PreloadedScratchRsrcReg ) ; MBB . addLiveIn ( PreloadedScratchRsrcReg ) ; } } DebugLoc DL ; MachineBasicBlock :: iterator I = MBB . begin ( ) ; Register ScratchWaveOffsetReg ; if ( TRI -> isSubRegisterEq ( ScratchRsrcReg , PreloadedScratchWaveOffsetReg ) ) { ArrayRef < MCPhysReg > AllSGPRs = TRI -> getAllSGPR32 ( MF ) ; unsigned NumPreloaded = MFI -> getNumPreloadedSGPRs ( ) ; AllSGPRs = AllSGPRs . slice ( std :: min ( static_cast < unsigned > ( AllSGPRs . size ( ) ) , NumPreloaded ) ) ; Register GITPtrLoReg = MFI -> getGITPtrLoReg ( MF ) ; for ( MCPhysReg Reg : AllSGPRs ) { if ( ! MRI . isPhysRegUsed ( Reg ) && MRI . isAllocatable ( Reg ) && ! TRI -> isSubRegisterEq ( ScratchRsrcReg , Reg ) && GITPtrLoReg != Reg ) { ScratchWaveOffsetReg = Reg ; BuildMI ( MBB , I , DL , TII -> get ( <STR_LIT> ) , ScratchWaveOffsetReg ) . addReg ( PreloadedScratchWaveOffsetReg , RegState :: Kill ) ; break ; } } } else { ScratchWaveOffsetReg = PreloadedScratchWaveOffsetReg ; } assert ( ScratchWaveOffsetReg ) ; if ( MF . getFrameInfo ( ) . hasCalls ( )


<s> const Function & F = MF . getFunction ( ) ; assert ( MFI -> isEntryFunction ( ) ) ; Register PreloadedScratchWaveOffsetReg = MFI -> getPreloadedReg ( <STR_LIT> ) ; if ( ! PreloadedScratchWaveOffsetReg ) return ; Register ScratchRsrcReg = getEntryFunctionReservedScratchRsrcReg ( MF ) ; if ( ScratchRsrcReg ) { for ( MachineBasicBlock & OtherBB : MF ) { if ( & OtherBB != & MBB ) { OtherBB . addLiveIn ( ScratchRsrcReg ) ; } } } Register PreloadedScratchRsrcReg ; if ( ST . isAmdHsaOrMesa ( F ) ) { PreloadedScratchRsrcReg = MFI -> getPreloadedReg ( <STR_LIT> ) ; if ( ScratchRsrcReg && PreloadedScratchRsrcReg ) { MRI . addLiveIn ( PreloadedScratchRsrcReg ) ; MBB . addLiveIn ( PreloadedScratchRsrcReg ) ; } } DebugLoc DL ; MachineBasicBlock :: iterator I = MBB . begin ( ) ; Register ScratchWaveOffsetReg ; if ( TRI -> isSubRegisterEq ( ScratchRsrcReg , PreloadedScratchWaveOffsetReg ) ) { ArrayRef < MCPhysReg > AllSGPRs = TRI -> getAllSGPR32 ( MF ) ; unsigned NumPreloaded = MFI -> getNumPreloadedSGPRs ( ) ; AllSGPRs = AllSGPRs . slice ( std :: min ( static_cast < unsigned > ( AllSGPRs . size ( ) ) , NumPreloaded ) ) ; Register GITPtrLoReg = MFI ->


<s> if ( FuncInfo -> SGPRForBPSaveRestoreCopy ) { BuildMI ( MBB , MBBI , DL , TII -> get ( <STR_LIT> ) , BasePtrReg ) . addReg ( FuncInfo -> SGPRForBPSaveRestoreCopy ) . setMIFlag ( MachineInstr :: FrameSetup ) ; } Register ScratchExecCopy ; if ( HasFPSaveIndex ) { const int FI = FuncInfo -> FramePointerSaveIndex . getValue ( ) ; assert ( ! MFI . isDeadObjectIndex ( FI ) ) ; if ( SpillFPToMemory ) { if ( ! ScratchExecCopy ) ScratchExecCopy = buildScratchExecCopy ( LiveRegs , MF , MBB , MBBI , false ) ; MCPhysReg TempVGPR = findScratchNonCalleeSaveRegister ( MRI , LiveRegs , <STR_LIT> ) ; buildEpilogReload ( LiveRegs , MBB , MBBI , TII , TempVGPR , FuncInfo -> getScratchRSrcReg ( ) , StackPtrReg , FI ) ; BuildMI ( MBB , MBBI , DL , TII -> get ( <STR_LIT> ) , FramePtrReg ) . addReg ( TempVGPR , RegState :: Kill ) ; } else { assert ( MFI . getStackID ( FI ) == TargetStackID :: SGPRSpill ) ; ArrayRef < <ISA_LIT>MachineFunctionInfo :: SpilledReg > Spill = FuncInfo -> getSGPRToVGPRSpills ( FI ) ; assert ( Spill . size ( ) == <NUM_LIT> ) ; BuildMI ( MBB , MBBI , DL , TII -> getMCOpcodeFromPseudo ( <STR_LIT> ) , FramePtrReg ) . addReg ( Spill [ <NUM_LIT> ] . VGPR ) . addImm ( Spill [ <NUM_LIT> ] . Lane ) ; } } if ( HasBPSaveIndex ) { const int BasePtrFI = * FuncInfo -> BasePointerSaveIndex ; assert ( ! MFI . isDeadObjectIndex ( BasePtrFI ) ) ; if ( SpillBPToMemory ) { if ( ! ScratchExecCopy ) ScratchExecCopy = buildScratchExecCopy ( LiveRegs , MF , MBB , MBBI , false ) ; MCPhysReg TempVGPR = findScratchNonCalleeSaveRegister ( MRI , LiveRegs ,


<s> } else { for ( MCRegister Reg : RC ) { if ( LiveRegs . available ( MRI , Reg ) ) return Reg ; } } if ( ! Unused ) report_fatal_error ( <STR_LIT> ) ; return MCRegister


<s> Register <ISA_LIT>FrameLowering :: getEntryFunctionReservedScratchRsrcReg ( MachineFunction & MF ) const { const GCNSubtarget & ST = MF . getSubtarget < GCNSubtarget > ( ) ; const <ISA_LIT>InstrInfo * TII = ST . getInstrInfo ( ) ; const <ISA_LIT>RegisterInfo * TRI = & TII -> getRegisterInfo ( ) ; MachineRegisterInfo & MRI = MF . getRegInfo ( ) ; <ISA_LIT>MachineFunctionInfo * MFI = MF . getInfo < <ISA_LIT>MachineFunctionInfo > ( ) ; assert ( MFI ->


<s> int <ISA_LIT>FrameLowering :: getFrameIndexReference ( const MachineFunction & MF , int FI , Register & FrameReg ) const { const <ISA_LIT>RegisterInfo * RI = MF . getSubtarget < GCNSubtarget > ( ) . getRegisterInfo ( ) ; FrameReg = RI -> getFrameRegister ( MF


<s> int <ISA_LIT>FrameLowering :: getFrameIndexReference ( const MachineFunction & MF , int FI , Register & FrameReg


<s> MachineFrameInfo & FrameInfo = MF . getFrameInfo ( ) ; const GCNSubtarget & ST = MF . getSubtarget < GCNSubtarget > ( ) ; const <ISA_LIT>RegisterInfo * TRI = ST . getRegisterInfo ( ) ; if ( MFI -> haveFreeLanesForSGPRSpill ( MF , <NUM_LIT> ) ) { int NewFI = FrameInfo . CreateStackObject ( <NUM_LIT> , <NUM_LIT> , true , nullptr , TargetStackID :: SGPRSpill ) ; if ( ! MFI -> allocateSGPRSpillToVGPR ( MF , NewFI ) ) llvm_unreachable ( <STR_LIT> ) ; FrameIndex = NewFI ; LLVM_DEBUG ( auto Spill = MFI -> getSGPRToVGPRSpills ( NewFI ) . front ( ) ; dbgs ( ) << <STR_LIT> << ( IsFP ? <STR_LIT> : <STR_LIT> ) << <STR_LIT> << printReg ( Spill . VGPR , TRI ) << ':' << Spill . Lane << '\n' ) ; return ; } TempSGPR = findScratchNonCalleeSaveRegister ( MF . getRegInfo ( ) , LiveRegs , <STR_LIT> , true ) ; if ( ! TempSGPR ) { int NewFI = FrameInfo . CreateStackObject ( <NUM_LIT> , <NUM_LIT> , true , nullptr , TargetStackID :: SGPRSpill ) ; if ( MFI -> allocateSGPRSpillToVGPR ( MF , NewFI ) ) { FrameIndex = NewFI ; } else { FrameIndex = FrameInfo . CreateSpillStackObject ( <NUM_LIT> , Align (


<s> static bool allSGPRSpillsAreDead ( const MachineFrameInfo & MFI , Optional < int > FramePointerSaveIndex ) { for ( int I = MFI . getObjectIndexBegin ( ) , E = MFI . getObjectIndexEnd ( ) ; I != E ; ++ I ) { if ( ! MFI . isDeadObjectIndex ( I ) && MFI . getStackID ( I ) == TargetStackID :: SGPRSpill && FramePointerSaveIndex && I != FramePointerSaveIndex ) { return false


<s> if ( isUInt < <NUM_LIT> > ( Offset ) ) { BuildMI ( MBB , I , DebugLoc ( ) , TII -> get ( <STR_LIT> ) ) . addReg ( SpillReg , RegState :: Kill ) . addReg ( ScratchRsrcReg ) . addReg ( SPReg ) . addImm ( Offset ) . addImm ( <NUM_LIT> ) . addImm ( <NUM_LIT> ) . addImm ( <NUM_LIT> ) . addImm ( <NUM_LIT> ) . addMemOperand ( MMO ) ; return ; } MCPhysReg OffsetReg = findScratchNonCalleeSaveRegister ( MF -> getRegInfo ( ) , LiveRegs , <STR_LIT> ) ; BuildMI ( MBB , I , DebugLoc ( ) , TII -> get (

<STR_LIT> ) , OffsetReg ) . addImm ( Offset ) ;


<s> const GCNSubtarget & ST = MF . getSubtarget < GCNSubtarget > ( ) ; const <ISA_LIT>RegisterInfo * TRI = ST . getRegisterInfo ( ) ; SavedVGPRs . clearBitsNotInMask ( TRI -> getAllVGPRRegMask ( ) ) ; const bool WillHaveFP = FrameInfo . hasCalls ( ) && ( SavedVGPRs . any ( ) || ! allStackObjectsAreDead ( FrameInfo ) ) ; for ( auto SSpill : MFI -> getSGPRSpillVGPRs ( ) ) SavedVGPRs . reset ( SSpill . VGPR ) ; const bool HasFP = WillHaveFP || hasFP ( MF ) ; if ( ! HasFP ) return ; if ( MFI -> haveFreeLanesForSGPRSpill ( MF , <NUM_LIT> ) ) { int NewFI = MF . getFrameInfo ( ) . CreateStackObject ( <NUM_LIT> , <NUM_LIT> , true , nullptr , TargetStackID :: SGPRSpill ) ; if ( ! MFI -> allocateSGPRSpillToVGPR ( MF , NewFI ) ) llvm_unreachable ( <STR_LIT> ) ; MFI -> FramePointerSaveIndex =


<s> assert ( isUInt < <NUM_LIT> > ( Amount ) && <STR_LIT> ) ; const <ISA_LIT>MachineFunctionInfo * MFI = MF . getInfo < <ISA_LIT>MachineFunctionInfo > ( ) ; unsigned SPReg = MFI -> getStackPtrOffsetReg ( ) ; unsigned Op = IsDestroy ? <STR_LIT> : <STR_LIT> ; BuildMI ( MBB , I , DL , TII -> get ( Op ) , SPReg ) . addReg ( SPReg ) . addImm ( Amount * ST . getWavefrontSize ( ) ) ; } else if ( CalleePopAmount != <NUM_LIT> ) { llvm_unreachable ( <STR_LIT> )