<s> if ( Indices . size ( ) != Factor || UseMaskForCond || UseMaskForGaps ) return BaseT :: getInterleavedMemoryOpCost ( Opcode , VecTy , Factor , Indices , Alignment , AddressSpace , UseMaskForCond , UseMaskForGaps ) ; return getMemoryOpCost ( Opcode , VecTy , MaybeAlign (

Alignment ) , AddressSpace , nullptr ) ;


<s> if ( ID == <STR_LIT> ) { std :: pair < int , MVT > LT = TLI . getTypeLegalizationCost ( DL , RetTy ) ; return LT . first +


<s> unsigned <ISA_LIT>TTIImpl :: getMemoryOpCost ( unsigned Opcode , Type * Src , MaybeAlign Alignment , unsigned AddressSpace , const Instruction * I ) { assert ( Opcode == Instruction :: Load || Opcode == Instruction :: Store ) ; if ( Opcode == Instruction :: Store ) return BaseT :: getMemoryOpCost ( Opcode , Src , Alignment , AddressSpace , I ) ; if ( Src -> isVectorTy ( ) ) { VectorType * VecTy = cast < VectorType > ( Src ) ; unsigned VecWidth = VecTy -> getBitWidth ( ) ; if ( useHVX ( ) && isTypeForHVX ( VecTy ) ) { unsigned RegWidth = getRegisterBitWidth ( true ) ; assert ( RegWidth && <STR_LIT> ) ; if ( VecWidth % RegWidth == <NUM_LIT> ) return VecWidth / RegWidth ; const Align RegAlign ( RegWidth / <NUM_LIT> ) ; if ( ! Alignment || * Alignment > RegAlign ) Alignment = RegAlign ; assert ( Alignment ) ; unsigned AlignWidth = <NUM_LIT> * Alignment -> value ( ) ; unsigned NumLoads = alignTo ( VecWidth , AlignWidth )


<s> bool <ISA_LIT>TTIImpl :: isLegalMaskedLoad ( Type * DataType , Align ) { return <ISA_LIT>MaskedVMem && isTypeForHVX ( DataType


<s> return <ISA_LIT>MaskedVMem && isTypeForHVX


<s> return <ISA_LIT>MaskedVMem && isTypeForHVX ( DataType


<s> return <ISA_LIT>MaskedVMem && isTypeForHVX (


<s> EVT VecVT = EVT :: getEVT ( VecTy ) ; if ( ! VecVT . isSimple ( ) || VecVT . getSizeInBits ( ) <= <NUM_LIT> ) return false ; if ( ST . isHVXVectorType ( VecVT . getSimpleVT ( ) ) ) return true ; auto Action = TLI . getPreferredVectorAction (


<s> unsigned <ISA_LIT>TTIImpl :: getScalarizationOverhead ( Type * Ty , const

APInt & DemandedElts , bool Insert , bool Extract ) {


<s> InstructionCost <ISA_LIT>TTIImpl :: getShuffleCost ( TTI :: ShuffleKind Kind , Type * Tp , ArrayRef < int > Mask , int Index , Type * SubTp ) { return <NUM_LIT>


<s> unsigned <ISA_LIT>TTIImpl :: getMaxInterleaveFactor ( unsigned


<s> return <ISA_LIT>AutoHVX && getST ( ) -> useHVXOps ( )


<s> return getST ( ) -> useHVXOps ( ) ? getST ( ) -> getVectorLength


<s> unsigned <ISA_LIT>TTIImpl :: getNumberOfRegisters ( bool Vector ) const { if ( Vector ) return <ISA_LIT>AutoHVX && getST ( ) -> useHVXOps ( )


<s> unsigned <ISA_LIT>TTIImpl :: getNumberOfRegisters ( bool Vector ) const { if ( Vector ) return <ISA_LIT>AutoHVX && getST ( ) -> useHVXOps ( ) ? <NUM_LIT> : <NUM_LIT> ; return <NUM_LIT>


<s> int <ISA_LIT>TTIImpl :: getUserCost ( const User * U ) { auto isCastFoldedIntoLoad = [ ] ( const CastInst * CI ) -> bool { if ( ! CI -> isIntegerCast ( ) ) return false ; const LoadInst * LI = dyn_cast < const LoadInst > ( CI -> getOperand ( <NUM_LIT> ) ) ; if ( ! LI || ! LI -> hasOneUse ( ) ) return false ; unsigned SBW = CI -> getSrcTy ( ) -> getIntegerBitWidth ( ) ; unsigned DBW = CI -> getDestTy ( ) -> getIntegerBitWidth


<s> std :: pair < int , MVT > SrcLT = TLI . getTypeLegalizationCost ( DL , SrcTy ) ; std :: pair < int , MVT > DstLT = TLI . getTypeLegalizationCost ( DL , DstTy ) ; return std :: max ( SrcLT . first , DstLT . first ) +


<s> unsigned <ISA_LIT>TTIImpl :: getInterleavedMemoryOpCost ( unsigned Opcode , Type * VecTy , unsigned Factor , ArrayRef < unsigned > Indices , unsigned Alignment , unsigned AddressSpace ) { if ( Indices . size ( ) != Factor ) return BaseT :: getInterleavedMemoryOpCost ( Opcode , VecTy , Factor , Indices , Alignment , AddressSpace ) ; return getMemoryOpCost ( Opcode , VecTy , Alignment


<s> unsigned <ISA_LIT>TTIImpl :: getAddressComputationCost ( Type * Tp , ScalarEvolution * SE , const SCEV *


<s> unsigned <ISA_LIT>TTIImpl :: getArithmeticInstrCost ( unsigned Opcode , Type * Ty , TTI :: TargetCostKind CostKind , TTI :: OperandValueKind Opd1Info , TTI :: OperandValueKind Opd2Info , TTI :: OperandValueProperties Opd1PropInfo , TTI :: OperandValueProperties Opd2PropInfo , ArrayRef < const Value * > Args , const Instruction * CxtI ) { if ( CostKind != TTI :: TCK_RecipThroughput ) return BaseT :: getArithmeticInstrCost ( Opcode , Ty ,

CostKind , Opd1Info , Opd2Info , Opd1PropInfo , Opd2PropInfo , Args , CxtI ) ;


<s> unsigned <ISA_LIT>TTIImpl :: getCacheLineSize ( ) const { return ST . getL1CacheLineSize (


<s> unsigned <ISA_LIT>TTIImpl :: getCallInstrCost ( Function * F , Type * RetTy , ArrayRef < Type * > Tys , TTI :: TargetCostKind CostKind ) { return BaseT :: getCallInstrCost ( F , RetTy , Tys , CostKind


<s> std :: pair < int , MVT > LT = TLI . getTypeLegalizationCost ( DL , ValTy ) ; if ( Opcode == Instruction :: FCmp ) return LT . first + FloatFactor * getTypeNumElements


<s> unsigned <ISA_LIT>TTIImpl :: getGatherScatterOpCost ( unsigned Opcode , Type * DataTy , Value * Ptr , bool VariableMask , unsigned Alignment , TTI :: TargetCostKind CostKind , const Instruction * I ) { return BaseT :: getGatherScatterOpCost ( Opcode , DataTy , Ptr , VariableMask , Alignment


<s> if ( ICA . getID ( ) == <STR_LIT> ) { std :: pair < int , MVT > LT = TLI . getTypeLegalizationCost ( DL , ICA . getReturnType (


<s> return BaseT :: getMaskedMemoryOpCost ( Opcode , Src , Alignment ,


<s> unsigned <ISA_LIT>TTIImpl :: getMaxInterleaveFactor (


<s> assert ( RegWidth && <STR_LIT> ) ; if ( VecWidth % RegWidth == <NUM_LIT> ) return VecWidth / RegWidth ; const Align RegAlign ( RegWidth / <NUM_LIT> ) ; if ( ! Alignment || * Alignment > RegAlign ) Alignment = RegAlign ; assert ( Alignment ) ; unsigned AlignWidth = <NUM_LIT> * Alignment -> value ( ) ; unsigned NumLoads = alignTo ( VecWidth , AlignWidth ) / AlignWidth ; return <NUM_LIT> * NumLoads ; } unsigned Cost = VecTy -> getElementType ( ) -> isFloatingPointTy ( ) ? FloatFactor : <NUM_LIT> ; const Align BoundAlignment = std :: min ( Alignment . valueOrOne ( ) , Align ( <NUM_LIT> ) ) ; unsigned AlignWidth = <NUM_LIT> * BoundAlignment . value ( ) ; unsigned NumLoads = alignTo ( VecWidth , AlignWidth ) / AlignWidth ; if ( Alignment == Align ( <NUM_LIT> ) || Alignment == Align ( <NUM_LIT> ) ) return Cost * NumLoads ; assert ( BoundAlignment <= Align (


<s> unsigned <ISA_LIT>TTIImpl :: getMinimumVF ( unsigned ElemWidth ) const { return ( <NUM_LIT> * ST . getVectorLength ( ) )


<s> return useHVX ( ) ? ST . getVectorLength (


<s> unsigned <ISA_LIT>TTIImpl :: getMinVectorRegisterBitWidth ( ) const { return useHVX ( ) ? ST . getVectorLength ( )


<s> if ( Vector ) return useHVX ( ) ? <NUM_LIT> : <NUM_LIT> ; return


<s> unsigned <ISA_LIT>TTIImpl :: getOperandsScalarizationOverhead ( ArrayRef < const Value * > Args , unsigned VF ) { return BaseT :: getOperandsScalarizationOverhead ( Args , VF


<s> TargetTransformInfo :: PopcntSupportKind <ISA_LIT>TTIImpl :: getPopcntSupport ( unsigned IntTyWidthInBit ) const { return TargetTransformInfo


<s> return ST . getL1PrefetchDistance ( )


<s> unsigned <ISA_LIT>TTIImpl :: getPrefetchDistance ( ) const { return ST .


<s> unsigned <ISA_LIT>TTIImpl :: getScalarizationOverhead ( VectorType * Ty , const APInt & DemandedElts , bool Insert


<s> unsigned <ISA_LIT>TTIImpl :: getShuffleCost ( TTI :: ShuffleKind Kind , Type * Tp , int Index , Type


<s> if ( auto * VTy = dyn_cast < FixedVectorType > ( Ty ) ) return VTy -> getNumElements ( ) ; assert ( ( Ty -> isIntegerTy ( ) || Ty -> isFloatingPointTy ( ) ) && <STR_LIT> ) ; return <NUM_LIT>


<s> unsigned <ISA_LIT>TTIImpl :: getTypeNumElements ( Type * Ty ) const { if ( auto * VTy = dyn_cast < FixedVectorType > ( Ty ) )


<s> if ( L && L -> empty ( ) && canPeel ( L ) && SE . getSmallConstantTripCount ( L ) == <NUM_LIT> && SE . getSmallConstantMaxTripCount ( L ) > <NUM_LIT> && SE . getSmallConstantMaxTripCount ( L ) <= <NUM_LIT> ) { UP . PeelCount


<s> auto isCastFoldedIntoLoad = [ this ] ( const CastInst * CI ) -> bool { if ( ! CI -> isIntegerCast ( ) ) return false ; const DataLayout & DL = getDataLayout ( ) ; unsigned SBW = DL . getTypeSizeInBits ( CI -> getSrcTy ( ) ) ; unsigned DBW = DL . getTypeSizeInBits ( CI -> getDestTy ( ) ) ; if ( DBW != <NUM_LIT> || SBW >= DBW ) return false ; const LoadInst * LI = dyn_cast < const LoadInst > ( CI -> getOperand ( <NUM_LIT> ) ) ; return LI && LI -> hasOneUse ( ) ; } ; if ( const CastInst * CI = dyn_cast < const CastInst > ( U ) ) if ( isCastFoldedIntoLoad ( CI ) ) return TargetTransformInfo :: TCC_Free ; return BaseT :: getUserCost ( U


<s> Type * ElemTy = Val -> isVectorTy ( ) ? cast < VectorType > ( Val ) -> getElementType ( ) : Val ; if ( Opcode == Instruction :: InsertElement ) { unsigned Cost = ( Index != <NUM_LIT> ) ? <NUM_LIT> : <NUM_LIT> ; if ( ElemTy -> isIntegerTy ( <NUM_LIT> ) ) return


<s> if ( ! cast < VectorType > ( VecTy ) -> getElementType ( ) -> isIntegerTy ( ) ) return false ; EVT VecVT = EVT :: getEVT ( VecTy ) ; if ( ! VecVT . isSimple ( ) || VecVT . getSizeInBits ( ) <=


<s> bool <ISA_LIT>TTIImpl :: shouldBuildLookupTables ( ) const { return


<s> bool <ISA_LIT>TTIImpl :: shouldFavorPostInc


<s> return ST . useHVXOps ( ) && <ISA_LIT>AutoHVX


<s> State . addLoc ( CCValAssign :: getReg ( ValNo , ValVT . getSimpleVT ( ) , Reg , LocVT . getSimpleVT ( ) , LocInfo ) ) ; return false ; } } } const Type * ArgTy = LocVT . getTypeForEVT ( State . getContext ( ) ) ; unsigned Alignment = State . getTarget ( ) . getSubtargetImpl ( ) -> getDataLayout ( ) -> getABITypeAlignment ( ArgTy ) ; unsigned Size = State . getTarget ( ) . getSubtargetImpl ( ) -> getDataLayout ( ) -> getTypeSizeInBits ( ArgTy ) / <NUM_LIT> ; if ( ArgFlags . isByVal ( ) ) { Size = ByValSize ; Alignment = <NUM_LIT> ; } unsigned Offset3 = State . AllocateStack ( Size , Alignment ) ; State . addLoc ( CCValAssign :: getMem ( ValNo , ValVT . getSimpleVT ( ) , Offset3 , LocVT . getSimpleVT ( ) , LocInfo ) ) ; return


<s> <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> } ; if ( unsigned Reg = State . AllocateReg ( RegList1 , <NUM_LIT> ) ) { State . addLoc ( CCValAssign :: getReg ( ValNo , ValVT . getSimpleVT ( ) , Reg , LocVT . getSimpleVT ( ) , LocInfo ) ) ; return false ; } } if ( LocVT == <STR_LIT> || LocVT == <STR_LIT> ) { static const unsigned RegList2 [ ] = { <STR_LIT> , <STR_LIT> , <STR_LIT> } ; if ( unsigned Reg = State . AllocateReg ( RegList2 , <NUM_LIT> ) ) { State . addLoc ( CCValAssign :: getReg ( ValNo , ValVT . getSimpleVT ( ) , Reg , LocVT . getSimpleVT ( ) , LocInfo ) ) ; return false ; } } const Type * ArgTy = LocVT . getTypeForEVT ( State . getContext ( ) ) ; unsigned Alignment = State . getTarget ( ) . getSubtargetImpl ( ) -> getDataLayout ( ) -> getABITypeAlignment ( ArgTy ) ; unsigned Size = State . getTarget ( ) . getSubtargetImpl ( ) -> getDataLayout ( ) -> getTypeSizeInBits ( ArgTy ) / <NUM_LIT> ; unsigned Offset3 = State . AllocateStack ( Size , Alignment ) ; State . addLoc ( CCValAssign :: getMem ( ValNo , ValVT . getSimpleVT ( ) ,

Offset3 , LocVT . getSimpleVT ( ) , LocInfo ) ) ;


<s> static bool CC_<ISA_LIT>32_VarArgs ( unsigned ValNo , EVT ValVT , EVT LocVT , CCValAssign :: LocInfo LocInfo , <STR_LIT> ArgFlags , <ISA_LIT>_CCState & State , int NonVarArgsParams , int CurrentParam , bool ForceMem ) { unsigned ByValSize = <NUM_LIT> ; if ( ArgFlags . isByVal ( ) && ( ( ByValSize = ArgFlags . getByValSize ( ) ) > ( MVT ( <STR_LIT> ) . getSizeInBits ( ) / <NUM_LIT> ) ) ) { ForceMem = true ; } if ( ! ForceMem && ( ( NonVarArgsParams == - <NUM_LIT> ) || ( CurrentParam <= NonVarArgsParams ) ) ) { if ( LocVT == <STR_LIT> || LocVT == <STR_LIT> || LocVT == <STR_LIT> || LocVT == <STR_LIT> ) { static const unsigned RegList1 [ ] = { <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> } ; if ( unsigned Reg = State . AllocateReg ( RegList1 , <NUM_LIT> ) ) { State . addLoc ( CCValAssign :: getReg ( ValNo , ValVT . getSimpleVT ( ) , Reg , LocVT . getSimpleVT ( ) , LocInfo ) ) ; return false ; } } if ( LocVT == <STR_LIT> ||


<s> static bool CC_<ISA_LIT>32_VarArgs ( unsigned ValNo , EVT ValVT , EVT LocVT , CCValAssign :: LocInfo LocInfo , <STR_LIT> ArgFlags , <ISA_LIT>_CCState & State , int NonVarArgsParams , int CurrentParam , bool ForceMem ) { unsigned ByValSize = <NUM_LIT> ; if ( ArgFlags . isByVal ( ) && ( ( ByValSize = ArgFlags . getByValSize ( ) ) > ( MVT ( <STR_LIT> ) . getSizeInBits ( ) / <NUM_LIT> ) ) ) { ForceMem = true ; } if ( ! ForceMem && ( ( NonVarArgsParams == - <NUM_LIT> ) || ( CurrentParam <= NonVarArgsParams ) ) ) { if ( LocVT == <STR_LIT> || LocVT == <STR_LIT> || LocVT == <STR_LIT> || LocVT == <STR_LIT> ) { static const unsigned RegList1 [ ] = { <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> } ; if ( unsigned Reg = State . AllocateReg ( RegList1 , <NUM_LIT> ) ) { State . addLoc ( CCValAssign :: getReg ( ValNo , ValVT . getSimpleVT ( ) , Reg , LocVT . getSimpleVT ( ) , LocInfo ) ) ; return false ; } } if ( LocVT == <STR_LIT> || LocVT == <STR_LIT> ) { static const unsigned RegList2 [ ] = { <STR_LIT> , <STR_LIT> , <STR_LIT> } ; if ( unsigned Reg = State . AllocateReg ( RegList2 , <NUM_LIT> ) ) { State . addLoc ( CCValAssign :: getReg ( ValNo , ValVT . getSimpleVT ( ) , Reg , LocVT . getSimpleVT ( ) , LocInfo ) ) ; return false ; } } } const Type * ArgTy = LocVT . getTypeForEVT ( State . getContext ( ) ) ; unsigned Alignment = State . getTarget ( ) . getDataLayout ( ) -> getABITypeAlignment ( ArgTy ) ; unsigned Size = State . getTarget ( ) . getDataLayout ( ) -> getTypeSizeInBits ( ArgTy ) / <NUM_LIT> ; if ( ArgFlags . isByVal ( ) ) { Size = ByValSize ; Alignment = <NUM_LIT> ; } unsigned Offset3 = State . AllocateStack


<s> static const unsigned RegList2 [ ] = { <STR_LIT> , <STR_LIT> , <STR_LIT> } ; if ( unsigned Reg = State . AllocateReg ( RegList2 , <NUM_LIT> ) ) { State . addLoc ( CCValAssign :: getReg ( ValNo , ValVT . getSimpleVT ( ) , Reg , LocVT . getSimpleVT ( ) , LocInfo ) ) ; return false ; } } } const Type * ArgTy = LocVT . getTypeForEVT ( State . getContext ( ) ) ; unsigned Alignment = State . getTarget ( ) . getDataLayout ( ) -> getABITypeAlignment ( ArgTy ) ; unsigned Size = State . getTarget ( ) . getDataLayout ( ) -> getTypeSizeInBits ( ArgTy ) / <NUM_LIT> ; if ( ArgFlags . isByVal ( ) ) { Size = ByValSize ; Alignment = <NUM_LIT> ; } unsigned Offset3 = State . AllocateStack ( Size , Alignment ) ; State . addLoc ( CCValAssign :: getMem ( ValNo , ValVT . getSimpleVT ( ) , Offset3 , LocVT . getSimpleVT ( ) ,


<s> static const unsigned RegList1 [ ] = { <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> } ; if ( unsigned Reg = State . AllocateReg ( RegList1 , <NUM_LIT> ) ) { State . addLoc ( CCValAssign :: getReg ( ValNo , ValVT . getSimpleVT ( ) , Reg , LocVT . getSimpleVT ( ) , LocInfo ) ) ; return false ; } } if ( LocVT == <STR_LIT> || LocVT == <STR_LIT> ) { static const unsigned RegList2 [ ] = { <STR_LIT> , <STR_LIT> , <STR_LIT> } ; if ( unsigned Reg = State . AllocateReg ( RegList2 , <NUM_LIT> ) ) { State . addLoc ( CCValAssign :: getReg ( ValNo , ValVT . getSimpleVT ( ) , Reg , LocVT . getSimpleVT ( ) , LocInfo ) ) ; return false ; } } const Type * ArgTy = LocVT . getTypeForEVT ( State . getContext ( ) ) ; unsigned Alignment = State . getTarget ( ) .

getDataLayout ( ) -> getABITypeAlignment ( ArgTy ) ;


<s> if ( ! AI ) continue ; auto F = findBaseAndOffset ( * AI ) ; Instruction * GroupInst ; if ( Instruction * BI = F . first ) { AI -> Offset = F . second ; GroupInst = BI ; } else { WorkStack . push_back ( * AI ) ; GroupInst = AI -> Inst ; } AddrGroups [ GroupInst ] . push_back ( * AI ) ; } for ( DomTreeNode * C : DomN -> children ( ) ) Visit ( C , Visit ) ; while ( ! WorkStack . empty ( ) && WorkStack . back ( ) . Inst -> getParent ( ) == & Block ) WorkStack . pop_back ( ) ; } ; traverseBlock ( HVC . DT . getRootNode ( ) , traverseBlock ) ; assert ( WorkStack . empty ( ) ) ; erase_if ( AddrGroups , [ ] ( auto & G ) { return G . second . size ( ) == <NUM_LIT> ; } ) ; erase_if ( AddrGroups , [ & ] ( auto & G ) { return ! llvm :: any_of ( G . second , [ & ] ( auto & I ) { return HVC . HST . isTypeForHVX ( I . ValTy ) ; } ) ; } ) ; erase_if ( AddrGroups , [ & ] ( auto & G ) { return llvm :: all_of ( G . second , [ & ] ( auto & I ) { return


<s> if ( Instruction * BI = F . first ) { AI -> Offset = F . second ; GroupInst = BI ; } else { WorkStack . push_back ( * AI ) ; GroupInst = AI -> Inst ; } AddrGroups [ GroupInst ] . push_back ( * AI ) ; } for ( DomTreeNode * C : DomN -> children ( ) ) Visit ( C , Visit ) ; while ( ! WorkStack . empty ( ) && WorkStack . back ( ) . Inst -> getParent ( ) == & Block ) WorkStack . pop_back ( ) ; } ; traverseBlock ( HVC . DT . getRootNode ( ) , traverseBlock ) ; assert ( WorkStack . empty ( ) ) ; erase_if ( AddrGroups , [ ] ( auto & G ) { return G . second . size ( ) == <NUM_LIT> ; } ) ; erase_if ( AddrGroups , [ & ] ( auto & G ) { return ! llvm :: any_of ( G . second , [ & ] ( auto & I ) { return HVC . HST . isTypeForHVX ( I . ValTy ) ; } ) ; } ) ; erase_if ( AddrGroups , [ & ] ( auto & G ) { return llvm :: all_of ( G . second , [ & ] ( auto & I ) { return I . HaveAlign >= I . NeedAlign ; } ) ; } )


<s> if ( Adjust % ElemSize == <NUM_LIT> ) { Value * Tmp0 = Builder . CreateGEP ( Ptr , HVC . getConstInt ( Adjust / ElemSize ) ) ; return Builder . CreatePointerCast ( Tmp0 , ValTy -> getPointerTo ( ) ) ; } PointerType * CharPtrTy = Type :: getInt8PtrTy ( HVC . F


<s> return Builder . CreatePointerCast ( Tmp0 , ValTy -> getPointerTo ( ) ) ; } PointerType * CharPtrTy = Type :: getInt8PtrTy ( HVC . F . getContext ( ) ) ; Value * Tmp0 = Builder . CreatePointerCast ( Ptr , CharPtrTy ) ; Value * Tmp1 = Builder . CreateGEP ( Tmp0 , HVC . getConstInt ( Adjust ) ) ; return Builder . CreatePointerCast ( Tmp1 , ValTy -> getPointerTo


<s> auto AlignVectors :: createAlignedLoad ( IRBuilder < > & Builder , Type * ValTy , Value * Ptr , int Alignment , Value * Mask , Value * PassThru ) const -> Value * { assert ( ! HVC . isUndef ( Mask ) ) ; if ( HVC . isZero ( Mask ) ) return PassThru ; if ( Mask == ConstantInt :: getTrue ( Mask -> getType ( ) ) ) return Builder . CreateAlignedLoad


<s> auto getLocOrNone = [ this ] ( const Instruction & I ) -> Optional < MemoryLocation > { if ( const auto * II = dyn_cast < IntrinsicInst > ( & I ) ) { switch ( II -> getIntrinsicID ( ) ) { case <STR_LIT> : return MemoryLocation :: getForArgument ( II , <NUM_LIT> , TLI ) ; case <STR_LIT> : return MemoryLocation :: getForArgument ( II , <NUM_LIT> , TLI ) ; } } return MemoryLocation :: getOrNone ( & I ) ; } ; const BasicBlock & Block = * In . getParent ( ) ; assert ( Block . begin ( ) == To || Block . end ( ) == To || To -> getParent ( ) == & Block ) ; if ( isa < PHINode > ( In ) || ( To != Block . end ( ) && isa < PHINode > ( * To ) ) ) return false ; if ( ! mayBeMemoryDependent ( In ) ) return true ; bool MayWrite = In . mayWriteToMemory ( ) ; auto MaybeLoc = getLocOrNone ( In ) ; auto From = In . getIterator (


<s> auto & TM = getAnalysis < TargetPassConfig > ( ) . getTM < <ISA_LIT>TargetMachine > ( ) ; <ISA_LIT>VectorCombine HVC ( F , AA , AC ,


<s> } else { WorkStack . push_back ( * AI ) ; GroupInst = AI -> Inst ; } AddrGroups [ GroupInst ] . push_back ( * AI ) ; } for ( DomTreeNode * C : DomN -> children ( ) ) Visit ( C , Visit ) ; while ( ! WorkStack . empty ( ) && WorkStack . back ( ) . Inst -> getParent ( ) == & Block ) WorkStack . pop_back ( ) ; } ; traverseBlock ( HVC . DT . getRootNode ( ) , traverseBlock ) ; assert ( WorkStack . empty ( ) ) ; erase_if ( AddrGroups , [ ] ( auto & G ) { return G . second . size ( ) == <NUM_LIT> ; } ) ; erase_if ( AddrGroups , [ & ] ( auto & G ) { return llvm :: none_of ( G . second , [ & ] ( auto & I ) { return HVC . HST . isTypeForHVX ( I . ValTy ) ; } ) ; } ) ; return ! AddrGroups . empty (


<s> int ElemSize = HVC . getAllocSizeOf ( ElemTy ) ; if ( Adjust % ElemSize == <NUM_LIT> && Adjust != <NUM_LIT> ) { Value * Tmp0 = Builder . CreateGEP ( ElemTy , Ptr , HVC . getConstInt ( Adjust / ElemSize ) ) ; return Builder . CreatePointerCast ( Tmp0 , ValTy -> getPointerTo ( ) ) ; } } PointerType * CharPtrTy = Type :: getInt8PtrTy ( HVC . F . getContext ( ) ) ; Value * Tmp0 = Builder . CreatePointerCast ( Ptr , CharPtrTy ) ; Value * Tmp1 = Builder . CreateGEP ( Type :: getInt8Ty ( HVC . F . getContext ( ) ) , Tmp0 , HVC . getConstInt ( Adjust ) ) ; return Builder . CreatePointerCast ( Tmp1 , ValTy -> getPointerTo


<s> if ( isa < PHINode > ( In ) || ( To != Block . end ( ) && isa < PHINode > ( * To ) ) ) return false ; if ( ! mayHaveNonDefUseDependency ( In ) ) return true ; bool MayWrite = In . mayWriteToMemory ( ) ; auto MaybeLoc = getLocOrNone ( In ) ; auto From = In . getIterator ( ) ; if ( From == To ) return true ; bool MoveUp = ( To != Block . end ( ) && To -> comesBefore ( & In ) ) ; auto Range = MoveUp ? std :: make_pair ( To , From ) : std :: make_pair ( std :: next ( From ) , To ) ; for ( auto It = Range . first ; It != Range . second ; ++ It ) { const Instruction & I = * It ; if ( llvm :: is_contained ( Ignore , & I ) ) continue ; if ( auto * II = dyn_cast < IntrinsicInst > ( & I ) ) { if ( II -> getIntrinsicID ( ) == <STR_LIT> ) continue ; } if ( I . mayThrow ( ) ) return false ; if ( auto * CB = dyn_cast < CallBase > ( & I ) ) { if ( ! CB -> hasFnAttr ( Attribute :: WillReturn ) ) return false ; if ( ! CB -> hasFnAttr ( Attribute :: NoSync ) ) return false ; } if ( I . mayReadOrWriteMemory ( ) ) { auto MaybeLocI = getLocOrNone ( I ) ; if ( MayWrite || I . mayWriteToMemory ( ) ) { if ( ! MaybeLoc || ! MaybeLocI ) return false ; if ( ! AA . isNoAlias ( * MaybeLoc , * MaybeLocI )


<s> int FromSize = getSizeOf ( FromSTy ) ; int ToSize = getSizeOf ( ToSTy ) ; assert ( FromSize % ToSize == <NUM_LIT> || ToSize % FromSize == <NUM_LIT> ) ; auto * MaskTy = cast < VectorType > ( Mask -> getType ( ) ) ; int FromCount = MaskTy -> getElementCount ( ) . getFixedValue ( ) ; int ToCount = ( FromCount * FromSize ) / ToSize ; assert ( ( FromCount * FromSize ) % ToSize == <NUM_LIT> ) ; auto * FromITy = IntegerType :: get ( F .

getContext ( ) , FromSize * <NUM_LIT> ) ;


<s> assert ( ( FromCount * FromSize ) % ToSize == <NUM_LIT> ) ; auto * FromITy = IntegerType :: get ( F . getContext ( ) , FromSize * <NUM_LIT> ) ; auto * ToITy = IntegerType :: get ( F . getContext ( ) , ToSize * <NUM_LIT> ) ; Value * Ext = Builder . CreateSExt ( Mask , VectorType :: get ( FromITy , FromCount , false ) ) ; Value * Cast = Builder . CreateBitCast ( Ext , VectorType :: get ( ToITy , ToCount , false ) ) ; return Builder . CreateTrunc ( Cast , VectorType :: get ( getBoolTy (


<s> std :: vector < Value * > Work [ <NUM_LIT> ] ; int ThisW = <NUM_LIT> , OtherW = <NUM_LIT> ; Work [ ThisW ] . assign ( Vecs . begin ( ) , Vecs . end ( ) ) ; while ( Work [ ThisW ] . size ( ) > <NUM_LIT> ) { auto * Ty = cast < VectorType > ( Work [ ThisW ] . front ( ) -> getType ( ) ) ; int ElemCount = Ty -> getElementCount ( ) . getFixedValue ( ) ; SMask . resize ( ElemCount * <NUM_LIT> ) ; std :: iota ( SMask . begin ( ) , SMask . end ( ) , <NUM_LIT> ) ; Work [ OtherW ] . clear ( ) ; if ( Work [ ThisW ] . size ( ) % <NUM_LIT> != <NUM_LIT> )

Work [ ThisW ] . push_back ( UndefValue :: get ( Ty ) ) ;


<s> std :: iota ( SMask . begin ( ) , SMask . end ( ) , <NUM_LIT> ) ; Work [ OtherW ] . clear ( ) ; if ( Work [ ThisW ] . size ( ) % <NUM_LIT> != <NUM_LIT> ) Work [ ThisW ] . push_back ( UndefValue :: get ( Ty ) ) ; for ( int i = <NUM_LIT> , e = Work [ ThisW ] . size ( ) ; i < e ; i += <NUM_LIT> ) { Value * Joined = Builder . CreateShuffleVector ( Work [ ThisW ] [ i ] , Work [ ThisW ] [ i + <NUM_LIT> ] , SMask ) ; Work [ OtherW ] . push_back ( Joined ) ; } std :: swap ( ThisW , OtherW ) ; } SMask . resize ( Vecs . size ( ) * getSizeOf ( Vecs . front ( ) -> getType ( ) ) ) ; std :: iota ( SMask . begin ( ) , SMask . end ( ) , <NUM_LIT> ) ; Value * Total = Work [ OtherW ] . front ( ) ; return Builder . CreateShuffleVector ( Total , UndefValue :: get ( Total -> getType (


<s> if ( auto * I = dyn_cast < Instruction > ( Op ) ) { if ( I -> getParent ( ) == Parent && Base -> comesBefore ( I ) ) WorkQ . push_back ( I ) ; } } } return Deps ; } ; auto tryAddTo = [ & ] ( const AddrInfo & Info , MoveGroup & Move ) { assert ( ! Move . Main . empty ( ) && <STR_LIT> ) ; if ( Move . IsHvx != isHvx ( Info ) ) return false ; Instruction * Base = Move . Main . front ( ) ; if ( Base -> getParent ( ) != Info . Inst -> getParent ( ) ) return false ; auto isSafeToMoveToBase = [ & ] ( const Instruction * I ) { return HVC . isSafeToMoveBeforeInBB ( * I , Base -> getIterator ( ) ) ; } ; DepList Deps = getUpwardDeps ( Info .


<s> auto * PtrTy = cast < PointerType > ( Ptr -> getType ( ) ) ; if ( ! PtrTy -> isOpaque ( ) ) { Type * ElemTy = PtrTy -> getElementType ( ) ; int ElemSize = HVC . getSizeOf ( ElemTy ) ; if ( Adjust % ElemSize == <NUM_LIT> ) { Value * Tmp0 = Builder . CreateGEP ( ElemTy , Ptr , HVC . getConstInt ( Adjust /


<s> for ( Instruction & I : Block ) { auto AI = this -> getAddrInfo ( I ) ; if ( ! AI ) continue ; auto F = findBaseAndOffset ( * AI ) ; Instruction * GroupInst ; if ( Instruction * BI = F . first ) { AI -> Offset = F . second ; GroupInst = BI ; } else { WorkStack . push_back ( * AI ) ; GroupInst = AI -> Inst ; } AddrGroups [ GroupInst ] . push_back ( * AI ) ; } for ( DomTreeNode * C : DomN -> children ( ) ) Visit ( C , Visit ) ; while ( ! WorkStack . empty ( ) && WorkStack . back ( ) . Inst -> getParent ( ) == & Block ) WorkStack . pop_back ( ) ; } ; traverseBlock ( HVC . DT . getRootNode ( ) , traverseBlock ) ; assert ( WorkStack . empty ( ) ) ; erase_if ( AddrGroups , [ ] ( auto & G ) { return G . second . size ( ) == <NUM_LIT> ; } ) ; erase_if ( AddrGroups , [ & ] ( auto & G ) { return ! llvm :: any_of ( G . second , [ & ] ( auto & I ) { return HVC . HST . isTypeForHVX ( I . ValTy ) ; } ) ; } ) ; return ! AddrGroups . empty (


<s> if ( Mask == ConstantInt :: getTrue ( Mask -> getType ( ) ) ) return Builder . CreateAlignedLoad ( ValTy , Ptr , Align ( Alignment ) ) ; return Builder . CreateMaskedLoad ( ValTy , Ptr , Align ( Alignment ) ,


<s> auto AlignVectors :: createAlignedLoad ( IRBuilder < > & Builder , Type * ValTy , Value * Ptr , int Alignment , Value * Mask , Value * PassThru ) const -> Value * { assert ( ! HVC . isUndef ( Mask ) ) ; if ( HVC . isZero ( Mask ) ) return PassThru ; if ( Mask == ConstantInt :: getTrue ( Mask -> getType ( ) ) ) return Builder . CreateAlignedLoad ( ValTy


<s> if ( HVC . isZero ( Mask ) || HVC . isUndef ( Val ) || HVC . isUndef ( Mask ) ) return UndefValue :: get ( Val -> getType ( ) ) ; if ( Mask == ConstantInt :: getTrue ( Mask -> getType ( ) ) ) return Builder

. CreateAlignedStore ( Val , Ptr , Align ( Alignment ) ) ;


<s> FunctionPass * llvm :: create<ISA_LIT>VectorCombineLegacyPass (


<s> return new <ISA_LIT>VectorCombineLegacy


<s> if ( SrcTy == DestTy ) return Val ; if ( HST . isTypeForHVX ( SrcTy , true ) ) { if ( cast < VectorType > ( SrcTy ) -> getElementType ( ) == BoolTy ) { <STR_LIT> TC = HwLen == <NUM_LIT> ? <STR_LIT> : <STR_LIT> ; Function * FI = <STR_LIT> ( F . getParent ( ) , TC , { DestTy , Val -> getType ( ) } ) ; return Builder . CreateCall ( FI , { Val } ) ; } return Builder . CreateBitCast ( Val , DestTy ) ; } llvm_unreachable ( <STR_LIT> ) ; } ; SmallVector < Value * , <NUM_LIT> > IntOps ; for ( Value * A : Args ) IntOps . push_back ( getCast ( Builder , A , getTypeForIntrin ( A -> getType ( ) ) ) ) ; Function * FI = <STR_LIT> ( F . getParent ( ) , IntID ) ; Value * Call = Builder . CreateCall ( FI , IntOps ) ; Type * CallTy = Call -> getType ( ) ; if ( CallTy == RetTy ) return Call ; assert ( HST . isTypeForHVX ( CallTy , true ) ) ; if ( cast < VectorType > ( CallTy ) -> getElementType ( ) == BoolTy ) return getCast ( Builder , Call , RetTy ) ; return Builder . CreateBitCast ( Call , RetTy


<s> for ( Value * Op : D -> operands ( ) ) { if ( auto * I = dyn_cast < Instruction > ( Op ) ) { if ( I -> getParent ( ) == Parent && Base -> comesBefore ( I ) ) WorkQ . push_back ( I ) ; } } } return Deps ; } ; auto tryAddTo = [ & ] ( const AddrInfo & Info , MoveGroup & Move ) { assert ( ! Move . Main . empty ( ) && <STR_LIT> ) ; if ( Move . IsHvx != isHvx ( Info ) ) return false ; Instruction * Base = Move . Main . front ( ) ; if ( Base -> getParent ( ) != Info . Inst -> getParent ( ) ) return false ; auto isSafeToMoveToBase = [ & ] ( const Instruction * I ) { return HVC . isSafeToMoveBeforeInBB ( * I , Base -> getIterator ( ) ) ; } ; DepList Deps = getUpwardDeps ( Info . Inst , Base ) ; if ( ! llvm :: all_of ( Deps , isSafeToMoveToBase ) ) return false ; Deps . erase ( Info . Inst ) ; auto inAddrMap = [ & ] ( Instruction * I ) { return AddrGroups . count ( I ) > <NUM_LIT> ; } ; if ( llvm :: any_of ( Deps , inAddrMap ) ) return false ; Move . Main . push_back ( Info . Inst ) ; llvm :: append_range ( Move . Deps , Deps ) ; return true


<s> } } return Deps ; } ; auto tryAddTo = [ & ] ( const AddrInfo & Info , MoveGroup & Move ) { assert ( ! Move . Main . empty ( ) && <STR_LIT> ) ; if ( Move . IsHvx != isHvx ( Info ) ) return false ; Instruction * Base = Move . Main . front ( ) ; if ( Base -> getParent ( ) != Info . Inst -> getParent ( ) ) return false ; auto isSafeToMoveToBase = [ & ] ( const Instruction * I ) { return HVC . isSafeToMoveBeforeInBB ( * I , Base -> getIterator ( ) ) ; } ; DepList Deps = getUpwardDeps ( Info . Inst , Base ) ; if ( ! llvm :: all_of ( Deps , isSafeToMoveToBase ) ) return false ; Deps . erase ( Info . Inst ) ; auto inAddrMap = [ & ] ( Instruction * I ) { return AddrGroups . count ( I ) > <NUM_LIT> ; } ; if ( llvm :: any_of ( Deps , inAddrMap ) ) return false ; Move . Main . push_back ( Info .


<s> const AddrInfo & Info = * I ; if ( ! Info . Inst -> mayWriteToMemory ( ) ) continue ; if ( StoreGroups . empty ( ) || ! tryAddTo ( Info , StoreGroups . back ( ) ) ) StoreGroups . emplace_back ( Info , Group . front ( ) . Inst , isHvx ( Info ) , false ) ; } erase_if ( StoreGroups , [ ] ( const MoveGroup & G ) { return G . Main . size (


<s> const_iterator end ( ) const { return Blocks . end


<s> void erase_if ( T && container , Pred p ) { llvm :: erase_if ( std :: forward < T


<s> void erase_if ( T && container , Pred p ) { llvm :: erase_if ( std :: forward < T > (


<s> auto AlignVectors :: getAddrInfo ( Instruction & In ) const -> Optional < AddrInfo > { if ( auto * L = isCandidate < LoadInst > ( & In ) ) return AddrInfo ( HVC , L , L -> getPointerOperand ( ) , L -> getType ( ) , L -> getAlign ( ) ) ; if ( auto * S = isCandidate < StoreInst > ( & In ) ) return AddrInfo ( HVC , S , S -> getPointerOperand ( ) , S -> getValueOperand ( ) -> getType ( ) , S -> getAlign ( ) ) ; if ( auto * II = isCandidate < IntrinsicInst > ( & In ) ) { <STR_LIT> ID = II -> getIntrinsicID ( ) ; switch ( ID


<s> auto AlignVectors :: getAlignFromValue ( const Value * V ) const -> Align { const auto * C = dyn_cast < ConstantInt > (


<s> const auto * C = dyn_cast < ConstantInt > ( V ) ; assert ( C && <STR_LIT> ) ; return C -> getAlignValue (


<s> AU . addRequired < AssumptionCacheTracker > ( ) ; AU . addRequired < DominatorTreeWrapperPass > ( ) ; AU . addRequired < TargetLibraryInfoWrapperPass > ( ) ; AU . addRequired < TargetPassConfig > ( ) ; FunctionPass :: getAnalysisUsage


<s> AU . addRequired < AssumptionCacheTracker > ( ) ; AU . addRequired < DominatorTreeWrapperPass > ( ) ; AU . addRequired < TargetLibraryInfoWrapperPass > ( ) ; AU . addRequired < TargetPassConfig > (


<s> assert ( ElemCount >= <NUM_LIT> ) ; IntegerType * BoolTy = Type :: getInt1Ty ( F . getContext ( ) ) ; if ( ElemCount == <NUM_LIT> ) return


<s> IntegerType * ByteTy = Type :: getInt8Ty ( F . getContext ( ) ) ; if ( ElemCount == <NUM_LIT> ) return ByteTy ; return VectorType :: get ( ByteTy , ElemCount


<s> assert ( <NUM_LIT> <= Start && Start < Length ) ; SmallVector < int , <NUM_LIT> > SMask ( Length ) ; std :: iota ( SMask . begin ( ) , SMask . end ( ) , Start ) ; return Builder . CreateShuffleVector ( Lo


<s> if ( auto * VecTy = dyn_cast < VectorType > ( Ty ) ) return ConstantVector :: getSplat ( VecTy -> getElementCount ( ) , Minus1 ) ; return Minus1


<s> } Type * ValTy = getPayload ( Val ) -> getType ( ) ; if ( auto * VecTy = dyn_cast < VectorType > ( ValTy ) ) { int ElemCount = VecTy -> getElementCount ( ) . getFixedValue


<s> return II -> getArgOperand ( <NUM_LIT> ) ; case <STR_LIT> : return II -> getArgOperand ( <NUM_LIT> ) ; } } Type * ValTy = getPayload ( Val ) -> getType ( ) ; if ( auto * VecTy = dyn_cast <


<s> if ( auto * VecTy = dyn_cast < VectorType > ( Ty ) ) return ConstantVector :: getSplat ( VecTy -> getElementCount ( ) , Zero ) ; return Zero


<s> auto <STR_LIT> ( const Type * Ty ) const -> int { return DL . getTypeStoreSize ( const_cast < Type * > ( Ty ) ) .


<s> if ( auto * VecTy = dyn_cast < VectorType > ( Ty ) ) return VecTy -> getElementType ( ) == getByteTy ( ) ; return false


<s> auto AlignVectors :: isHvx ( const AddrInfo & AI )


<s> int Size = getSizeOf ( Ty ) ; if ( HST . isTypeForHVX ( Ty ) ) return Size == static_cast < int > ( HST . getVectorLength (


<s> auto <STR_LIT> ( const Value * Val ) const -> bool { if ( auto * C = dyn_cast < Constant > ( Val ) ) return C -> isZeroValue ( ) ; return


<s> if ( Move . IsLoad ) { for ( Instruction * D : Move . Deps ) D -> moveBefore ( Where ) ; ArrayRef < Instruction * > Main ( Move . Main ) ; for ( Instruction * M : Main . drop_front ( <NUM_LIT> ) ) { M -> moveAfter ( Where ) ; Where = M ; } } else { assert ( Move . Deps .


<s> AI . Inst , IsHvx ( Hvx ) , IsLoad ( Load


<s> int FromSize = getSizeOf ( FromSTy ) ; int ToSize = getSizeOf ( ToSTy ) ; assert ( FromSize % ToSize == <NUM_LIT> || ToSize % FromSize == <NUM_LIT> ) ; auto * MaskTy = cast < VectorType > ( Mask -> getType ( ) ) ; int FromCount = MaskTy -> getElementCount ( ) . getFixedValue ( ) ; int ToCount = ( FromCount * FromSize ) / ToSize ; assert ( ( FromCount * FromSize ) %


<s> auto <STR_LIT> ( IRBuilder < > & Builder , Value * Mask , Type * FromTy , Type * ToTy ) const -> Value * { assert ( isa < VectorType > ( Mask -> getType ( ) ) ) ; Type * FromSTy = FromTy -> getScalarType ( ) ; Type * ToSTy = ToTy -> getScalarType ( ) ; if ( FromSTy == ToSTy ) return Mask ; int FromSize = getSizeOf ( FromSTy ) ; int ToSize = getSizeOf ( ToSTy ) ; assert ( FromSize % ToSize == <NUM_LIT> ||


<s> if ( ! HST . useHVXOps ( ) ) return false ; bool Changed = AlignVectors ( * this ) .


<s> if ( skipFunction ( F ) ) return false ; AliasAnalysis & AA = getAnalysis < AAResultsWrapperPass > ( ) . getAAResults ( ) ; AssumptionCache & AC = getAnalysis < AssumptionCacheTracker > ( ) . getAssumptionCache ( F ) ; DominatorTree & DT = getAnalysis < DominatorTreeWrapperPass


<s> auto <STR_LIT> ( IRBuilder < > & Builder , Value * Val ) const -> Value * { Type * ScalarTy = Val -> getType ( ) -> getScalarType ( ) ; if ( ScalarTy == getByteTy ( ) ) return Val ; if ( ScalarTy != getBoolTy ( ) ) return Builder . CreateBitCast

( Val , getByteTy ( getSizeOf ( Val ) ) ) ;


<s> Value * Pair = concat ( Builder , { Lo , Hi } ) ; Value * Shift = Builder . CreateLShr ( Builder . CreateShl ( Pair , Amt ) , <NUM_LIT> ) ; Value * Trunc = Builder . CreateTrunc ( Shift , Type :: getInt32Ty ( F . getContext ( ) ) ) ; return Builder . CreateBitCast ( Trunc , Hi -> getType ( ) ) ; } if ( VecLen == <NUM_LIT> ) { Value * Sub = Builder . CreateSub ( getConstInt ( VecLen ) , Amt ) ; return vralignb ( Builder , Lo , Hi , Sub ) ; } llvm_unreachable ( <STR_LIT>


<s> auto <STR_LIT> ( IRBuilder < > & Builder , Value * Val ) const -> Value * { Type * ScalarTy = Val -> getType ( ) -> getScalarType ( ) ; if ( ScalarTy == getBoolTy ( ) ) return Val ; Value * Bytes = vbytes ( Builder , Val ) ; if ( auto * VecTy = dyn_cast < VectorType > ( Bytes -> getType ( ) ) ) return Builder

. CreateTrunc ( Bytes , getBoolTy ( getSizeOf ( VecTy ) ) ) ;


<s> auto <STR_LIT> ( IRBuilder < > & Builder , Value * Val ) const -> Value * { Type * ScalarTy = Val -> getType ( ) -> getScalarType ( ) ; if ( ScalarTy == getBoolTy (


<s> assert ( ValTy -> getElementType ( ) == Pad -> getType ( ) ) ; int CurSize = ValTy -> getElementCount ( ) . getFixedValue ( ) ; if ( CurSize == NewSize ) return Val ; if ( CurSize > NewSize ) return getElementRange ( Builder , Val , Val , <NUM_LIT> , NewSize ) ; SmallVector < int , <NUM_LIT> > SMask ( NewSize ) ; std :: iota ( SMask . begin ( ) , SMask . begin ( ) + CurSize


<s> assert ( isa < VectorType > ( Val -> getType ( ) ) ) ; auto * ValTy = cast < VectorType > ( Val -> getType ( ) ) ; assert ( ValTy -> getElementType ( ) == Pad -> getType ( ) ) ; int CurSize = ValTy -> getElementCount ( ) . getFixedValue ( ) ; if ( CurSize == NewSize ) return Val ; if ( CurSize > NewSize ) return getElementRange ( Builder , Val , Val , <NUM_LIT>


<s> for ( Instruction * I : llvm :: reverse ( ToErase ) )


<s> if ( II && ( II -> getIntrinsicID ( ) == <STR_LIT> ||

II -> getIntrinsicID ( ) == <STR_LIT> ) ) {


<s> if ( II && ( II -> getIntrinsicID ( ) == <STR_LIT> || II -> getIntrinsicID ( ) == <STR_LIT> ) ) { DEBUG ( dbgs ( ) << <STR_LIT> << * II <<


<s> findLoopCarriedDeps ( ) ; findValueToReuse ( ) ; if ( ReuseCandidate . isDefined ( ) ) { reuseValue ( ) ; Changed = true ; Continue = true ; } llvm :: for_each ( Dependences , std :: default_delete < DepChain > ( ) ) ; } while ( Continue ) ; return


<s> if ( User -> getParent ( ) != BB ) continue ; if ( ReplacedInsts . count ( User ) ) { DEBUG ( dbgs ( ) << * User << <STR_LIT> ) ; continue ; } if ( isa < PHINode > ( User ) ) continue ; if ( User -> mayHaveSideEffects ( ) ) continue ; if ( ! canReplace ( User ) ) continue ; PNUsers . push_back ( User ) ; } DEBUG ( dbgs ( ) << PNUsers . size ( ) << <STR_LIT> ) ; for ( Instruction * I : PNUsers ) { for ( auto UI = BEInst -> use_begin ( ) , E = BEInst -> use_end ( ) ; UI != E ; ++ UI ) { Use & U = * UI ; Instruction * BEUser = cast < Instruction > ( U . getUser ( ) ) ; if ( BEUser -> getParent ( ) != BB ) continue ; if ( ! isEquivalentOperation ( I , BEUser ) ) continue ; int NumOperands = I -> getNumOperands ( ) ; for ( int OpNo = <NUM_LIT> ; OpNo < NumOperands ; ++ OpNo ) { Value * Op = I -> getOperand (


<s> DEBUG ( dbgs ( ) << ReuseCandidate ) ; Instruction * Inst2Replace = ReuseCandidate . Inst2Replace ; Instruction * BEInst = ReuseCandidate . BackedgeInst ; int NumOperands = Inst2Replace -> getNumOperands ( ) ; std :: map < Instruction * , DepChain * > DepChains ; int Iterations = - <NUM_LIT> ; BasicBlock * LoopPH = CurLoop -> getLoopPreheader ( ) ; for ( int i = <NUM_LIT> ; i < NumOperands ; ++ i ) { Instruction * I = dyn_cast < Instruction > ( Inst2Replace -> getOperand ( i ) ) ; if ( ! I ) continue ; else { Instruction * J = cast < Instruction > ( BEInst -> getOperand ( i ) ) ; DepChain * D = getDepChainBtwn ( I , J ) ; assert ( D && <STR_LIT> ) ; if ( Iterations == - <NUM_LIT> ) Iterations = D -> iterations ( ) ; assert ( Iterations == D -> iterations ( ) && <STR_LIT> ) ; DepChains [ I ] = D ; } } DEBUG ( dbgs ( ) << <STR_LIT> ) ; SmallVector < Instruction * , <NUM_LIT> > InstsInPreheader ; for ( int i = <NUM_LIT> ; i < Iterations ; ++ i ) { Instruction * InstInPreheader = Inst2Replace -> clone ( ) ; SmallVector < Value * , <NUM_LIT> > Ops ; for ( int j = <NUM_LIT> ; j < NumOperands ; ++ j ) { Instruction * I = dyn_cast < Instruction > ( Inst2Replace -> getOperand ( j ) ) ; if ( ! I ) continue ; DepChain & D = * DepChains [ I ] ; Value * ValInPreheader = findValueInBlock ( D [ i ] , LoopPH ) ; InstInPreheader -> setOperand ( j , ValInPreheader ) ; } InstsInPreheader . push_back ( InstInPreheader ) ; InstInPreheader -> setName ( Inst2Replace -> getName ( ) + <STR_LIT> ) ; InstInPreheader -> insertBefore ( LoopPH ->


<s> continue ; } PHINode * PN = cast < PHINode > ( D -> front ( ) ) ; Instruction * BEInst = D -> back ( ) ; int Iters = D -> iterations ( ) ; BasicBlock * BB = PN -> getParent ( ) ; LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << * PN << <STR_LIT> ) ; SmallVector < Instruction * , <NUM_LIT> > PNUsers ; for ( auto UI = PN -> use_begin ( ) , E = PN -> use_end ( ) ; UI != E ; ++ UI ) { Use & U = * UI ; Instruction * User = cast < Instruction > ( U . getUser ( ) ) ; if ( User -> getParent ( ) != BB ) continue ; if ( ReplacedInsts . count ( User ) ) { LLVM_DEBUG ( dbgs ( ) << * User << <STR_LIT> ) ; continue ; } if ( isa < PHINode > ( User ) ) continue ; if ( User -> mayHaveSideEffects ( ) ) continue ; if ( ! canReplace ( User ) ) continue ; PNUsers . push_back


<s> void <ISA_LIT>VectorLoopCarriedReuse :: findValueToReuse ( ) { for ( auto * D : Dependences ) { LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << * ( D -> front ( ) ) << <STR_LIT> ) ; if ( D -> iterations ( ) > <ISA_LIT>VLCRIterationLim ) { LLVM_DEBUG ( dbgs ( ) << <STR_LIT> ) ; continue ; } PHINode * PN = cast < PHINode > ( D -> front ( ) ) ; Instruction * BEInst = D -> back ( ) ; int Iters = D -> iterations ( ) ; BasicBlock * BB = PN -> getParent ( ) ; LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << * PN << <STR_LIT> ) ; SmallVector < Instruction * , <NUM_LIT> > PNUsers ; for ( auto UI = PN -> use_begin ( ) , E = PN -> use_end ( ) ; UI != E ; ++ UI ) { Use & U = * UI ; Instruction * User = cast < Instruction > ( U . getUser ( ) ) ; if ( User -> getParent ( ) != BB ) continue ; if ( ReplacedInsts . count ( User ) ) { LLVM_DEBUG ( dbgs ( ) << * User << <STR_LIT> ) ; continue ; } if ( isa < PHINode > ( User ) ) continue ; if ( User -> mayHaveSideEffects ( ) ) continue ; if ( ! canReplace ( User ) ) continue ; PNUsers . push_back ( User ) ; } LLVM_DEBUG ( dbgs ( ) << PNUsers . size ( ) << <STR_LIT> ) ; for ( Instruction * I : PNUsers ) { for ( auto UI = BEInst -> use_begin ( ) , E = BEInst -> use_end ( ) ; UI != E ; ++ UI ) { Use & U = * UI ; Instruction * BEUser = cast < Instruction > ( U . getUser ( ) ) ; if ( BEUser -> getParent ( ) != BB ) continue ; if ( ! isEquivalentOperation ( I , BEUser ) ) continue ; int NumOperands = I -> getNumOperands ( ) ; for ( int OpNo = <NUM_LIT> ; OpNo < NumOperands ; ++ OpNo ) { Value * Op = I -> getOperand ( OpNo ) ; Value * BEOp = BEUser -> getOperand ( OpNo ) ; Instruction * OpInst = dyn_cast < Instruction > ( Op ) ; if ( ! OpInst ) { if ( Op == BEOp ) continue ; BEUser = nullptr ; break ; } Instruction * BEOpInst = dyn_cast < Instruction > ( BEOp ) ; if ( ! isDepChainBtwn ( OpInst , BEOpInst , Iters ) ) { BEUser = nullptr ; break ; } } if ( BEUser ) { LLVM_DEBUG ( dbgs ( ) << <STR_LIT> ) ; ReuseCandidate . Inst2Replace = I ; ReuseCandidate . BackedgeInst


<s> PNUsers . push_back ( User ) ; } LLVM_DEBUG ( dbgs ( ) << PNUsers . size ( ) << <STR_LIT> ) ; for ( Instruction * I : PNUsers ) { for ( auto UI = BEInst -> use_begin ( ) , E = BEInst -> use_end ( ) ; UI != E ; ++ UI ) { Use & U = * UI ; Instruction * BEUser = cast < Instruction > ( U . getUser ( ) ) ; if ( BEUser -> getParent ( ) != BB ) continue ; if ( ! isEquivalentOperation ( I , BEUser ) ) continue ; int NumOperands = I -> getNumOperands ( ) ; for ( int OpNo = <NUM_LIT> ; OpNo < NumOperands ; ++ OpNo ) { Value * Op = I -> getOperand ( OpNo ) ; Instruction * OpInst = dyn_cast < Instruction > ( Op ) ; if ( ! OpInst ) continue ; Value * BEOp = BEUser -> getOperand ( OpNo ) ; Instruction * BEOpInst = dyn_cast < Instruction > ( BEOp ) ; if ( ! isDepChainBtwn ( OpInst , BEOpInst , Iters ) ) { BEUser = nullptr ; break ; } } if ( BEUser ) { LLVM_DEBUG ( dbgs ( ) << <STR_LIT> ) ; ReuseCandidate . Inst2Replace = I ; ReuseCandidate . BackedgeInst = BEUser ; return ; } else ReuseCandidate . reset


<s> for ( auto * D : Dependences ) { if ( D -> front ( ) == I1 && D -> back ( ) == I2 ) return D ; } return


<s> bool <ISA_LIT>VectorLoopCarriedReuse :: isDepChainBtwn ( Instruction * I1 , Instruction * I2 , int Iters ) { for ( auto * D : Dependences ) { if ( D -> front ( ) == I1 && D -> back ( ) == I2 && D ->


<s> Instruction * Inst2Replace = ReuseCandidate . Inst2Replace ; Instruction * BEInst = ReuseCandidate . BackedgeInst ; int NumOperands = Inst2Replace -> getNumOperands ( ) ; std :: map < Instruction * , DepChain * > DepChains ; int Iterations = - <NUM_LIT> ; BasicBlock * LoopPH = CurLoop -> getLoopPreheader ( ) ; for ( int i = <NUM_LIT> ; i < NumOperands ; ++ i ) { Instruction * I = dyn_cast < Instruction > ( Inst2Replace -> getOperand ( i ) ) ; if ( ! I ) continue ; else { Instruction * J = cast < Instruction > ( BEInst -> getOperand ( i ) ) ; DepChain * D = getDepChainBtwn ( I , J ) ; assert ( D && <STR_LIT> ) ; if ( Iterations == - <NUM_LIT> ) Iterations = D -> iterations ( ) ; assert ( Iterations == D -> iterations ( ) && <STR_LIT> ) ; DepChains [ I ] = D ; } } LLVM_DEBUG ( dbgs ( ) << <STR_LIT> ) ; SmallVector < Instruction * , <NUM_LIT> > InstsInPreheader ; for ( int i = <NUM_LIT> ; i < Iterations ; ++ i ) { Instruction * InstInPreheader = Inst2Replace -> clone ( ) ; SmallVector < Value * , <NUM_LIT> > Ops ; for ( int j = <NUM_LIT> ; j < NumOperands ; ++ j ) { Instruction * I = dyn_cast < Instruction > ( Inst2Replace -> getOperand ( j ) ) ; if ( ! I ) continue ; DepChain & D = * DepChains [ I ] ; Value * ValInPreheader = findValueInBlock ( D [ i ] , LoopPH ) ; InstInPreheader -> setOperand ( j


<s> void getAnalysisUsage ( AnalysisUsage & AU ) const override { AU . addRequiredID ( LoopSimplifyID


<s> <ISA_LIT>VectorLoopCarriedReuse ( Loop * L ) :


<s> PassRegistry * PR = PassRegistry :: getPassRegistry


<s> if ( CurLoop -> getNumBlocks ( ) != <NUM_LIT> ) return false ; return doVLCR ( )


<s> <ISA_LIT>VectorLoopCarriedReuse Vlcr ( L ) ; return Vlcr . run (


<s> DEBUG ( dbgs ( ) << <STR_LIT> << * CurLoop -> getHeader ( ) << <STR_LIT> ) ; do { Dependences . clear ( ) ; Continue = false ; findLoopCarriedDeps ( ) ; findValueToReuse ( ) ; if ( ReuseCandidate . isDefined ( ) ) { reuseValue ( ) ; Changed = true ; Continue = true ; } std :: for_each ( Dependences . begin ( ) , Dependences . end ( ) , std :: default_delete < DepChain > ( ) ) ; } while ( Continue ) ; return Changed


<s> Instruction * back ( ) const { return Chain . back (


<s> Instruction * back ( ) const { return Chain .


<s> Value * <ISA_LIT>VectorLoopCarriedReuse :: findValueInBlock ( Value * Op , BasicBlock * BB ) { PHINode * PN = dyn_cast < PHINode > ( Op ) ; assert ( PN ) ; Value * ValueInBlock = PN -> getIncomingValueForBlock ( BB ) ; return


<s> int Iters = D -> iterations ( ) ; BasicBlock * BB = PN -> getParent ( ) ; LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << * PN << <STR_LIT> ) ; SmallVector < Instruction * , <NUM_LIT> > PNUsers ; for ( auto UI = PN -> use_begin ( ) , E = PN -> use_end ( ) ; UI != E ; ++ UI ) { Use & U = * UI ; Instruction * User = cast < Instruction > ( U . getUser ( ) ) ; if ( User -> getParent ( ) != BB ) continue ; if ( ReplacedInsts . count ( User ) ) { LLVM_DEBUG ( dbgs ( ) << * User << <STR_LIT> ) ; continue ; } if ( isa < PHINode > ( User ) ) continue ; if ( User -> mayHaveSideEffects ( ) ) continue ; if ( ! canReplace ( User ) ) continue ; PNUsers . push_back ( User ) ; } LLVM_DEBUG ( dbgs ( ) << PNUsers . size ( ) << <STR_LIT> ) ; for ( Instruction * I : PNUsers ) { for ( auto UI = BEInst -> use_begin ( ) , E = BEInst -> use_end ( ) ; UI != E ; ++ UI ) { Use & U = * UI ; Instruction * BEUser = cast < Instruction > ( U . getUser ( ) ) ; if ( BEUser -> getParent ( ) != BB ) continue ; if ( ! isEquivalentOperation ( I , BEUser ) ) continue ; int NumOperands = I -> getNumOperands ( ) ; std :: map < Instruction * , DepChain * > DepChains ; CallInst * C1 = dyn_cast < CallInst > ( I ) ; if ( ( I && I -> isCommutative ( ) ) || ( C1 && isCallInstCommutative ( C1 ) ) ) { bool Found = false ; for ( int OpNo = <NUM_LIT> ; OpNo < NumOperands ; ++ OpNo ) { Value * Op = I -> getOperand ( OpNo ) ; Instruction * OpInst = dyn_cast < Instruction > ( Op ) ; Found = false ; for ( int T = <NUM_LIT> ; T < NumOperands ; ++ T ) { Value * BEOp = BEUser -> getOperand ( T ) ; Instruction * BEOpInst = dyn_cast < Instruction > ( BEOp ) ; if ( ! OpInst && ! BEOpInst ) { if ( Op ==


<s> void getAnalysisUsage ( AnalysisUsage & AU ) const override { AU . addRequired <


<s> ChainOfDependences & getChain ( ) { return Chain


<s> void reset ( ) { Inst2Replace = nullptr ; BackedgeInst = nullptr ; DepChains . clear ( )


<s> BackedgeInst = nullptr ; DepChains . clear


<s> BasicBlock * LoopPH = CurLoop -> getLoopPreheader ( ) ; assert ( ! DepChains . empty ( ) && <STR_LIT> ) ; LLVM_DEBUG ( dbgs ( ) << <STR_LIT> ) ; SmallVector < Instruction * , <NUM_LIT> > InstsInPreheader ; for ( int i = <NUM_LIT> ; i < Iterations ; ++ i ) { Instruction * InstInPreheader = Inst2Replace -> clone ( ) ; SmallVector < Value * , <NUM_LIT> > Ops ; for ( int j = <NUM_LIT> ; j < NumOperands ; ++ j ) { Instruction * I = dyn_cast < Instruction > ( Inst2Replace -> getOperand ( j ) ) ; if ( ! I ) continue ; DepChain & D = * DepChains [ I ] ; Value * ValInPreheader = findValueInBlock ( D [ i ] , LoopPH ) ; InstInPreheader -> setOperand ( j , ValInPreheader ) ; } InstsInPreheader . push_back ( InstInPreheader ) ; InstInPreheader -> setName ( Inst2Replace -> getName ( ) + <STR_LIT> ) ; InstInPreheader -> insertBefore ( LoopPH -> getTerminator ( ) ) ; LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << * InstInPreheader << <STR_LIT> << LoopPH -> getName ( ) << <STR_LIT> ) ; } BasicBlock * BB = BEInst -> getParent ( ) ; IRBuilder < > IRB ( BB ) ; IRB . SetInsertPoint ( BB -> getFirstNonPHI ( ) ) ; Value * BEVal = BEInst ; PHINode * NewPhi ; for ( int i = Iterations - <NUM_LIT> ; i >= <NUM_LIT> ; -- i ) { Instruction * InstInPreheader = InstsInPreheader [ i ] ; NewPhi = IRB . CreatePHI ( InstInPreheader -> getType ( )


<s> if ( ! L -> getLoopPreheader ( ) ) return false ; if ( ! L -> getSubLoops ( ) .


<s> bool <ISA_LIT>VectorLoopCarriedReuse :: runOnLoop ( Loop * L , LPPassManager & LPM ) { if ( skipLoop ( L ) ) return false ; if ( ! L -> getLoopPreheader ( ) ) return false ; if ( ! L -> getSubLoops ( ) . empty ( ) ) return false ; if ( L -> getNumBlocks ( ) != <NUM_LIT> ) return false ; CurLoop =


<s> int size ( ) const { return Chain . size (


<s> static void addAsmInstr ( MachineBasicBlock * MBB , unsigned Reg , MachineBasicBlock :: instr_iterator I , const DebugLoc & DL , const <ISA_LIT>InstrInfo * QII , MachineFunction & Fn ) { std :: string VDescStr = <STR_LIT> + getStringReg ( Reg ) ; const char * cstr = Fn . createExternalSymbolName ( VDescStr ) ; unsigned ExtraInfo = InlineAsm :: Extra_HasSideEffects ; BuildMI ( * MBB , I , DL , QII -> get ( TargetOpcode ::

INLINEASM ) ) . addExternalSymbol ( cstr ) . addImm ( ExtraInfo ) ;


<s> const char * cstr = Fn . createExternalSymbolName ( VDescStr ) ; unsigned ExtraInfo = InlineAsm :: Extra_HasSideEffects ; BuildMI ( * MBB , I , DL , QII -> get ( TargetOpcode :: INLINEASM ) ) . addExternalSymbol ( cstr ) .


<s> StringRef getPassName ( ) const override { return <STR_LIT>


<s> initialize<ISA_LIT>VectorPrintPass ( * PassRegistry ::


<s> LLVM_DEBUG ( dbgs ( ) << <STR_LIT> ; MII -> dump ( ) ) ; } } } else { unsigned Reg = <NUM_LIT> ; if ( getInstrVecReg ( MI , Reg ) ) { VecPrintList . push_back ( & MI ) ; LLVM_DEBUG ( dbgs ( ) << <STR_LIT> ; MI . dump ( ) ) ; } } } Changed = ! VecPrintList . empty ( ) ; if ( ! Changed ) return Changed ; for ( auto * I : VecPrintList ) { DebugLoc DL = I -> getDebugLoc ( ) ; MachineBasicBlock * MBB = I -> getParent ( ) ; LLVM_DEBUG ( dbgs ( ) << <STR_LIT> ; I -> dump ( ) ) ; unsigned Reg = <NUM_LIT> ; if ( ! getInstrVecReg ( * I , Reg ) ) llvm_unreachable ( <STR_LIT> ) ; MachineBasicBlock :: instr_iterator MII = I -> getIterator ( ) ; if ( I -> isInsideBundle ( ) ) { LLVM_DEBUG ( dbgs ( ) << <STR_LIT> ; I -> dump ( ) ) ; while ( MBB -> instr_end ( ) != MII && MII -> isInsideBundle ( ) ) MII ++ ; } else


<s> QST = & Fn . getSubtarget < <ISA_LIT>Subtarget > ( ) ; QRI = QST -> getRegisterInfo ( ) ; QII = QST -> getInstrInfo ( ) ; std :: vector < MachineInstr * > VecPrintList ; for ( auto & MBB : Fn ) for ( auto & MI : MBB ) { if ( MI . isBundle ( ) ) { MachineBasicBlock :: instr_iterator MII = MI . getIterator ( ) ; for ( ++ MII ; MII != MBB . instr_end ( ) && MII -> isInsideBundle ( ) ; ++ MII ) { if ( MII -> getNumOperands ( ) < <NUM_LIT> ) continue ; unsigned Reg = <NUM_LIT> ; if ( getInstrVecReg ( * MII , Reg ) ) { VecPrintList . push_back ( ( & * MII ) ) ; DEBUG ( dbgs ( ) << <STR_LIT> ; MII -> dump ( ) ) ; } } } else { unsigned Reg = <NUM_LIT> ; if ( getInstrVecReg ( MI , Reg ) ) { VecPrintList . push_back ( & MI ) ; DEBUG ( dbgs ( ) << <STR_LIT> ; MI . dump ( ) ) ; } } } Changed = ! VecPrintList . empty ( ) ; if ( ! Changed ) return Changed ; for ( auto * I : VecPrintList ) { DebugLoc DL = I -> getDebugLoc ( ) ; MachineBasicBlock * MBB = I -> getParent ( ) ; DEBUG ( dbgs ( ) << <STR_LIT> ; I -> dump ( ) ) ; unsigned Reg = <NUM_LIT> ; if ( ! getInstrVecReg ( * I , Reg ) ) llvm_unreachable ( <STR_LIT> ) ; MachineBasicBlock :: instr_iterator MII = I -> getIterator ( ) ; if ( I -> isInsideBundle ( ) ) { DEBUG ( dbgs ( ) << <STR_LIT> ; I -> dump ( ) ) ; while ( MBB -> instr_end ( ) != MII && MII -> isInsideBundle ( ) ) MII ++ ; } else { DEBUG ( dbgs ( ) << <STR_LIT> ; I -> dump


<s> static bool isVecReg ( unsigned Reg ) { return ( Reg >= <STR_LIT> && Reg <= <STR_LIT> ) || ( Reg >= <STR_LIT> && Reg <= <STR_LIT> ) || ( Reg

>= <STR_LIT> && Reg <= <STR_LIT> ) || ( Reg >= <STR_LIT> && Reg <= <STR_LIT> ) ;


<s> <ISA_LIT>VectorPrint ( ) : MachineFunctionPass ( ID ) , QST ( nullptr ) , QII ( nullptr ) , QRI ( nullptr ) { initialize<ISA_LIT>VectorPrintPass ( * PassRegistry :: getPassRegistry (


<s> std :: string VDescStr = <STR_LIT> + getStringReg ( Reg ) ; const char * cstr = Fn . createExternalSymbolName ( VDescStr . c_str ( ) ) ; unsigned ExtraInfo = InlineAsm ::


<s> BuildMI ( * MBB , I , DL , QII -> get ( TargetOpcode :: INLINEASM ) ) . addExternalSymbol ( cstr )


<s> FunctionPass * llvm :: create<ISA_LIT>VectorPrint


<s> Reg = MI . getOperand ( <NUM_LIT> ) . getReg ( ) ; if ( isVecReg ( Reg ) ) return true ; } if ( MI . mayStore ( ) && MI . getNumOperands ( ) >= <NUM_LIT> && MI . getOperand ( <NUM_LIT> ) . isReg ( ) ) { Reg = MI . getOperand ( <NUM_LIT> ) . getReg (


<s> if ( R >= <STR_LIT> && R <= <STR_LIT> ) { static const char * S [ ] = { <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> } ; return S [ R - <STR_LIT> ] ; } if ( R >= <STR_LIT> && R <= <STR_LIT> ) { static const char * S [ ] = {


<s> return S [ R - <STR_LIT> ] ; } if ( R >= <STR_LIT> && R <= <STR_LIT> ) { static const char * S [ ] = { <STR_LIT> , <STR_LIT> , <STR_LIT> , <STR_LIT> } ; return S [ R - <STR_LIT> ] ; } llvm_unreachable ( <STR_LIT> )


<s> return ( Reg >= <STR_LIT> && Reg <= <STR_LIT> ) || ( Reg >= <STR_LIT> && Reg <= <STR_LIT>

) || ( Reg >= <STR_LIT> && Reg <= <STR_LIT> ) ;


<s> } } } else { unsigned Reg = <NUM_LIT> ; if ( getInstrVecReg ( MI , Reg ) ) { VecPrintList . push_back ( & MI ) ; DEBUG ( dbgs ( ) << <STR_LIT> ; MI . dump ( ) ) ; } } } Changed = VecPrintList . size ( ) > <NUM_LIT> ; if ( ! Changed ) return Changed ; for ( auto * I : VecPrintList ) { DebugLoc DL = I -> getDebugLoc ( ) ; MachineBasicBlock * MBB = I -> getParent ( ) ; DEBUG ( dbgs ( ) << <STR_LIT> ; I -> dump ( ) ) ; unsigned Reg = <NUM_LIT> ; if ( ! getInstrVecReg ( * I , Reg ) ) llvm_unreachable ( <STR_LIT> ) ; MachineBasicBlock :: instr_iterator MII = I -> getIterator ( ) ; if ( I -> isInsideBundle ( ) ) { DEBUG ( dbgs ( ) << <STR_LIT> ; I -> dump ( ) ) ; while ( MII -> isInsideBundle ( ) ) ++ MII ; } else { DEBUG ( dbgs ( ) << <STR_LIT> ; I -> dump ( ) ) ; MII ++ ; } if ( Reg >= <STR_LIT> && Reg <= <STR_LIT> ) { DEBUG ( dbgs ( ) << <STR_LIT> << Reg - <STR_LIT> << '\n' ) ; addAsmInstr ( MBB , Reg , MII , DL , QII , Fn ) ; } else if ( Reg >= <STR_LIT> && Reg <= <STR_LIT> ) { DEBUG ( dbgs ( ) << <STR_LIT> << Reg - <STR_LIT> << '\n' ) ; addAsmInstr ( MBB , <STR_LIT> + ( Reg - <STR_LIT> ) * <NUM_LIT> + <NUM_LIT> , MII , DL , QII , Fn ) ; addAsmInstr ( MBB , <STR_LIT> + ( Reg - <STR_LIT> ) * <NUM_LIT> , MII , DL , QII , Fn ) ; } else if ( Reg >= <STR_LIT> && Reg <= <STR_LIT> ) { DEBUG ( dbgs ( ) << <STR_LIT> << Reg - <STR_LIT> << '\n' ) ; addAsmInstr ( MBB , Reg , MII , DL , QII , Fn ) ; } else llvm_unreachable


<s> unsigned Reg = <NUM_LIT> ; if ( ! getInstrVecReg ( * I , Reg ) ) llvm_unreachable ( <STR_LIT> ) ; MachineBasicBlock :: instr_iterator MII = I -> getIterator ( ) ; if ( I -> isInsideBundle ( ) ) { DEBUG ( dbgs ( ) << <STR_LIT> ; I -> dump ( ) ) ; while ( MII -> isInsideBundle ( ) ) ++ MII ; } else { DEBUG ( dbgs ( ) << <STR_LIT> ; I -> dump ( ) ) ; MII ++ ; } if ( Reg >= <STR_LIT> && Reg <= <STR_LIT> ) { DEBUG ( dbgs ( ) << <STR_LIT> << Reg - <STR_LIT> << '\n' ) ; addAsmInstr ( MBB , Reg , MII , DL , QII , Fn ) ; } else if ( Reg >= <STR_LIT> && Reg <= <STR_LIT> ) { DEBUG ( dbgs ( ) << <STR_LIT> << Reg - <STR_LIT> << '\n' ) ; addAsmInstr ( MBB , <STR_LIT> + ( Reg - <STR_LIT> ) * <NUM_LIT> + <NUM_LIT> , MII , DL , QII , Fn ) ; addAsmInstr ( MBB , <STR_LIT> + ( Reg - <STR_LIT> ) * <NUM_LIT> , MII , DL , QII , Fn ) ; } else if ( Reg >= <STR_LIT> && Reg <= <STR_LIT> ) { DEBUG ( dbgs ( ) << <STR_LIT> << Reg - <STR_LIT> << '\n' ) ; addAsmInstr ( MBB , Reg , MII , DL , QII , Fn ) ; } else llvm_unreachable ( <STR_LIT> ) ; } return


<s> unsigned StoreOpc = VecRC . getID ( ) == <STR_LIT> ? <STR_LIT> : <STR_LIT> ; BuildMI ( DefB , At , DefI -> getDebugLoc ( ) , HII -> get ( StoreOpc ) ) . addFrameIndex ( FI ) . addImm ( <NUM_LIT> ) . addReg ( VecR ) ; unsigned VecSize = HRI . getRegSizeInBits ( VecRC ) / <NUM_LIT> ; for ( MachineInstr * ExtI : P . second ) { assert ( ExtI -> getOpcode ( ) == <STR_LIT> ) ; unsigned SR = ExtI -> getOperand ( <NUM_LIT> ) . getSubReg ( ) ; assert ( ExtI -> getOperand ( <NUM_LIT> ) . getReg ( ) == VecR ) ; MachineBasicBlock & ExtB = * ExtI -> getParent ( ) ; DebugLoc DL = ExtI -> getDebugLoc ( ) ; Register BaseR = MRI . createVirtualRegister ( & <STR_LIT> ) ; BuildMI ( ExtB , ExtI , DL , HII -> get ( <STR_LIT> ) , BaseR ) . addFrameIndex ( FI ) . addImm ( SR == <NUM_LIT> ? <NUM_LIT> : VecSize / <NUM_LIT> ) ; unsigned ElemR = genElemLoad ( ExtI , BaseR , MRI ) ; Register ExtR = ExtI -> getOperand ( <NUM_LIT> ) . getReg ( ) ; MRI . replaceRegWith ( ExtR


<s> HII = HST -> getInstrInfo ( ) ; const auto & HRI = * HST -> getRegisterInfo ( ) ; MachineRegisterInfo & MRI = MF . getRegInfo ( ) ; MachineFrameInfo & MFI = MF . getFrameInfo ( ) ; std :: map < unsigned , SmallVector < MachineInstr * , <NUM_LIT> >> VExtractMap ; bool Changed = false ; for ( MachineBasicBlock & MBB : MF ) { for ( MachineInstr & MI : MBB ) { unsigned Opc = MI . getOpcode ( ) ; if ( Opc != <STR_LIT> ) continue ; Register VecR = MI . getOperand ( <NUM_LIT> ) . getReg ( ) ; VExtractMap [ VecR ] . push_back ( & MI ) ; } } for ( auto & P : VExtractMap ) { unsigned VecR = P . first ; if ( P . second . size ( ) <= VExtractThreshold ) continue ; const auto & VecRC = * MRI . getRegClass ( VecR ) ; int FI = MFI . CreateSpillStackObject ( HRI . getSpillSize ( VecRC ) , HRI . getSpillAlignment ( VecRC ) ) ; MachineInstr * DefI = MRI . getVRegDef ( VecR ) ; MachineBasicBlock :: iterator At = std :: next ( DefI -> getIterator ( ) ) ; MachineBasicBlock & DefB = * DefI -> getParent ( ) ; unsigned StoreOpc = VecRC . getID ( ) == <STR_LIT> ? <STR_LIT> : <STR_LIT> ; BuildMI ( DefB , At , DefI -> getDebugLoc ( ) , HII -> get ( StoreOpc ) ) . addFrameIndex ( FI ) . addImm ( <NUM_LIT> ) . addReg ( VecR ) ; unsigned VecSize = HRI . getRegSizeInBits ( VecRC ) / <NUM_LIT> ; for ( MachineInstr * ExtI : P . second ) { assert ( ExtI -> getOpcode ( ) == <STR_LIT> ) ; unsigned SR = ExtI -> getOperand ( <NUM_LIT> ) . getSubReg ( ) ; assert ( ExtI -> getOperand ( <NUM_LIT> ) . getReg ( ) == VecR ) ; MachineBasicBlock & ExtB = * ExtI -> getParent ( ) ; DebugLoc DL = ExtI -> getDebugLoc ( ) ; Register BaseR = MRI . createVirtualRegister ( & <STR_LIT> ) ; BuildMI ( ExtB , ExtI , DL , HII -> get ( <STR_LIT> ) , BaseR ) . addFrameIndex ( FI ) . addImm ( SR == <NUM_LIT> ? <NUM_LIT> : VecSize / <NUM_LIT> ) ; unsigned ElemR = genElemLoad ( ExtI , BaseR , MRI ) ; Register ExtR = ExtI -> getOperand ( <NUM_LIT> ) . getReg ( ) ; MRI . replaceRegWith ( ExtR , ElemR ) ; ExtB . erase ( ExtI ) ; Changed = true


<s> MachineBasicBlock & ExtB = * ExtI -> getParent ( ) ; DebugLoc DL = ExtI -> getDebugLoc ( ) ; unsigned ElemR = MRI . createVirtualRegister ( & <STR_LIT> ) ; unsigned ExtIdxR = ExtI -> getOperand ( <NUM_LIT> ) . getReg ( ) ; unsigned ExtIdxS = ExtI -> getOperand ( <NUM_LIT> ) . getSubReg ( ) ; if ( ExtIdxS == <NUM_LIT> ) { MachineInstr * DI = MRI . getVRegDef ( ExtIdxR ) ; if ( DI -> getOpcode ( ) == <STR_LIT> ) { unsigned V = DI -> getOperand ( <NUM_LIT> ) . getImm ( ) ; V &= ( HST -> getVectorLength ( ) - <NUM_LIT> ) & - <NUM_LIT> ; BuildMI ( ExtB , ExtI , DL , HII -> get ( <STR_LIT> ) , ElemR ) . addReg ( BaseR ) . addImm ( V ) ; return


<s> Align Alignment = HRI . getSpillAlign ( VecRC ) ; MaxAlign = max ( MaxAlign , Alignment ) ; int FI = MFI . CreateStackObject ( HRI . getSpillSize ( VecRC ) , Alignment , false ) ; MachineInstr * DefI = MRI . getVRegDef ( VecR ) ; MachineBasicBlock :: iterator At = std :: next ( DefI -> getIterator ( ) ) ; MachineBasicBlock & DefB = * DefI -> getParent ( ) ; unsigned StoreOpc = VecRC . getID ( ) == <STR_LIT> ? <STR_LIT> : <STR_LIT> ; Register AddrR = EmitAddr ( DefB , At , DefI -> getDebugLoc ( ) , FI , <NUM_LIT> ) ; BuildMI ( DefB , At , DefI -> getDebugLoc ( ) , HII -> get ( StoreOpc ) ) . addReg ( AddrR ) . addImm ( <NUM_LIT> ) . addReg ( VecR ) ; unsigned VecSize = HRI . getRegSizeInBits ( VecRC ) / <NUM_LIT> ; for ( MachineInstr * ExtI : P . second ) { assert ( ExtI -> getOpcode ( ) == <STR_LIT> ) ; unsigned SR = ExtI -> getOperand ( <NUM_LIT> ) . getSubReg ( ) ; assert ( ExtI -> getOperand ( <NUM_LIT> ) . getReg ( ) == VecR ) ; MachineBasicBlock & ExtB = * ExtI


<s> FunctionPass * llvm :: create<ISA_LIT>VExtract ( ) { return new <ISA_LIT>VExtract


<s> FunctionPass * llvm :: create<ISA_LIT>VExtract ( ) { return new <ISA_LIT>VExtract ( )


<s> StringRef getPassName ( ) const override { return <STR_LIT>


<s> bool <ISA_LIT>PacketizerList :: cannotCoexist ( const MachineInstr & MI , const MachineInstr & MJ ) { return cannotCoexistAsymm ( MI , MJ , * HII ) || cannotCoexistAsymm ( MJ , MI ,


<s> if ( ! MO . isReg ( ) ) continue ; predRegNumDst = MO . getReg ( ) ; predRegClass = HRI -> getMinimalPhysRegClass ( predRegNumDst ) ; if ( predRegClass == & <STR_LIT> ) break ; } assert ( ( predRegClass == & <STR_LIT> ) && <STR_LIT> ) ; if ( predRegNumDst != predRegNumSrc || HII -> isDotNewInst ( PacketMI ) != HII -> isDotNewInst ( MI ) || getPredicateSense ( MI , HII ) != getPredicateSense ( PacketMI , HII ) ) return false ; } unsigned StartCheck = <NUM_LIT> ; for ( auto I : CurrentPacketMIs ) { SUnit * TempSU = MIToSUnit . find ( I ) -> second ; MachineInstr & TempMI = * TempSU -> getInstr ( ) ; if ( & TempMI != & PacketMI && ! StartCheck ) continue ; StartCheck = <NUM_LIT> ; if ( & TempMI == & PacketMI ) continue ; for ( auto & MO : MI . operands ( ) ) if ( MO . isReg ( ) && TempSU -> getInstr ( ) -> modifiesRegister ( MO . getReg ( ) , HRI ) ) return false ; } if ( ! HII -> isPostIncrement ( MI ) ) { for ( unsigned opNum = <NUM_LIT> ; opNum < MI . getNumOperands ( ) - <NUM_LIT> ; opNum ++ ) { const MachineOperand & MO = MI . getOperand ( opNum ) ; if ( MO . isReg ( ) && MO . getReg ( ) == DepReg ) return false ; } } for ( auto & MO : PacketMI . operands ( ) ) { if ( MO . isRegMask ( ) && MO . clobbersPhysReg ( DepReg ) ) return false ; if ( ! MO . isReg ( ) || ! MO . isDef ( ) || ! MO . isImplicit ( ) ) continue ; unsigned R = MO . getReg ( ) ; if ( R == DepReg || HRI -> isSuperRegister ( DepReg , R ) ) return false ; } for ( auto & MO : MI . operands ( ) ) { if ( MO . isReg ( ) && MO . isUse ( ) && MO . isImplicit ( ) && MO . getReg ( ) == DepReg ) return false ; } return true


<s> bool <ISA_LIT>PacketizerList :: canReserveResourcesForConstExt ( ) { return tryAllocateResourcesForConstExt ( false )


<s> MI . setDesc ( HII -> get ( NewOpcode ) ) ; return true


<s> bool <ISA_LIT>PacketizerList :: demoteToDotOld ( MachineInstr & MI ) { int NewOpcode = HII -> getDotOldOp ( MI ) ; MI . setDesc ( HII -> get ( NewOpcode


<s> for ( MachineInstr * MI : CurrentPacketMIs ) { MachineBasicBlock :: instr_iterator NextMI = std :: next ( MI -> getIterator ( ) ) ; for ( auto & I : make_range ( HII -> expandVGatherPseudo ( * MI ) , NextMI ) ) OldPacketMIs . push_back ( & I ) ; } CurrentPacketMIs . clear ( ) ; if ( OldPacketMIs . size ( ) > <NUM_LIT> ) { MachineBasicBlock :: instr_iterator FirstMI ( OldPacketMIs . front ( ) ) ; MachineBasicBlock :: instr_iterator LastMI ( EndMI . getInstrIterator ( ) ) ; finalizeBundle ( * MBB , FirstMI , LastMI ) ; auto BundleMII = std :: prev ( FirstMI ) ; if ( memShufDisabled ) HII -> setBundleNoShuf ( BundleMII ) ; setmemShufDisabled ( false ) ; } ResourceTracker -> clearResources


<s> if ( Opc == <STR_LIT> || Opc == <STR_LIT> ) continue ; if ( HII -> isMemOp ( * MJ ) ) continue ; if ( MJ -> mayLoad ( ) ) FoundLoad = true ; if ( MJ -> mayStore ( ) && ! HII -> isNewValueStore ( * MJ ) ) FoundStore = true ; } return FoundLoad &&


<s> static const MachineOperand & getAbsSetOperand (


<s> static const MachineOperand & getAbsSetOperand ( const MachineInstr & MI ) { assert ( isLoadAbsSet (


<s> AU . addRequired < MachineDominatorTree > ( ) ; AU . addRequired < MachineLoopInfo > ( ) ; AU . addPreserved < MachineDominatorTree >


<s> StringRef getPassName ( ) const override { return


<s> return MI . getOperand ( MI . getNumOperands


<s> if ( isControlFlow ( I ) && isControlFlow ( J ) ) return true ; auto isBadForLoopN = [ this ] ( const MachineInstr & MI ) -> bool { if ( MI . isCall ( ) || HII -> isDeallocRet ( MI ) || HII -> isNewValueJump ( MI ) ) return true ; if ( HII -> isPredicated ( MI ) && HII -> isPredicatedNew ( MI ) && HII -> isJumpR ( MI ) ) return true ; return false ; } ; if ( HII -> isLoopN ( I ) && isBadForLoopN ( J ) ) return true ; if ( HII -> isLoopN ( J ) && isBadForLoopN ( I ) ) return true ; return HII -> isDeallocRet ( I ) && ( J . isBranch ( ) || J . isCall ( )


<s> return false ; } ; if ( HII -> isLoopN ( I ) && isBadForLoopN ( J ) ) return true ; if ( HII -> isLoopN ( J ) && isBadForLoopN ( I ) ) return true ; return HII -> isDeallocRet ( I ) && ( J . isBranch ( ) || J . isCall ( ) ||


<s> BitVector DeadDefs ( <STR_LIT> ) ; for ( auto & MO : I . operands ( ) ) { if ( ! MO . isReg ( ) || ! MO . isDef ( ) || ! MO . isDead ( ) ) continue ; DeadDefs [ MO . getReg ( ) ] = true ; } for ( auto & MO : J . operands ( ) ) { if ( ! MO . isReg ( ) || ! MO . isDef ( ) || ! MO . isDead ( ) ) continue ; unsigned R = MO . getReg ( ) ; if ( R != <STR_LIT> && DeadDefs [ R ] ) return true ; } return false


<s> DeadDefs [ MO . getReg ( ) ] = true ; } for ( auto & MO : J . operands ( ) ) { if ( ! MO . isReg ( ) || ! MO . isDef ( ) || ! MO . isDead ( ) ) continue ; unsigned R = MO . getReg ( ) ; if ( R != <STR_LIT> && DeadDefs [ R ] ) return true ; } return


<s> bool <ISA_LIT>PacketizerList :: hasDualStoreDependence ( const MachineInstr & I , const MachineInstr & J ) { bool SysI = isSystemInstr ( I ) , SysJ = isSystemInstr ( J ) ; bool StoreI = I . mayStore ( ) , StoreJ = J . mayStore ( ) ; if ( ( SysI && StoreJ ) || ( SysJ && StoreI ) ) return true ; if ( StoreI && StoreJ ) { if ( HII -> isNewValueInst ( J ) || HII -> isMemOp ( J ) || HII -> isMemOp ( I ) ) return true ; } else { bool MopStI = HII -> isMemOp ( I ) || StoreI ; bool MopStJ = HII -> isMemOp ( J ) || StoreJ ; if ( MopStI && MopStJ ) return true ; } return ( StoreJ && HII -> isDeallocRet ( I ) ) || (


<s> HII = MF . getSubtarget < <ISA_LIT>Subtarget > ( ) . getInstrInfo ( ) ; HRI = MF . getSubtarget < <ISA_LIT>Subtarget > ( ) . getRegisterInfo ( ) ; addMutation ( llvm :: make_unique < <ISA_LIT>Subtarget :: UsrOverflowMutation >


<s> if ( MI . isCFIInstruction ( ) ) return false ; if ( MI . isInlineAsm ( ) ) return false ; if ( MI . isImplicitDef ( ) ) return false ; const MCInstrDesc & TID = MI . getDesc ( ) ; auto * IS = ResourceTracker -> getInstrItins ( ) -> beginStage ( TID . getSchedClass ( ) ) ; unsigned FuncUnits = IS -> getUnits (


<s> Dependence = false ; PromotedToDotNew = false ; GlueToNewValueJump = false ; GlueAllocframeStore = false ; FoundSequentialDependence = false ; ChangedOffset =


<s> GlueToNewValueJump = false ; GlueAllocframeStore = false ; FoundSequentialDependence = false ; ChangedOffset =


<s> if ( HII -> isDeallocRet ( MI ) ) if ( DepReg == HRI -> getFrameRegister ( ) || DepReg == HRI -> getStackRegister ( ) ) return true ; if ( DepType == SDep


<s> if ( DepReg == HRI -> getRARegister ( ) ) return true ; if ( HII -> isDeallocRet ( MI ) ) if ( DepReg == HRI -> getFrameRegister ( ) || DepReg == HRI -> getStackRegister ( ) ) return true ; if ( DepType == SDep :: Data ) { for ( const MachineOperand MO : MI . operands ( ) ) if ( MO . isReg ( ) && MO . getReg ( ) == DepReg && ! MO . isImplicit ( ) ) return true ; } return false


<s> static bool isControlFlow ( const MachineInstr & MI ) { return MI . getDesc ( ) . isTerminator ( ) || MI . getDesc ( )


<s> static bool isControlFlow ( const MachineInstr & MI ) { return MI . getDesc ( ) . isTerminator ( ) || MI


<s> static bool isDirectJump ( const MachineInstr & MI


<s> static bool isDirectJump ( const MachineInstr & MI ) { return MI . getOpcode ( ) == <STR_LIT>


<s> for ( auto & MO : I . operands ( ) ) { if ( CheckDef && MO . isRegMask ( ) && MO . clobbersPhysReg ( DepReg ) ) return true ; if ( ! MO . isReg ( ) || MO . getReg ( ) != DepReg || ! MO . isImplicit ( ) ) continue ; if ( CheckDef == MO . isDef ( ) ) return true ; } return false


<s> if ( ! MO . isReg ( ) || MO . getReg ( ) != DepReg || ! MO . isImplicit ( ) ) continue ; if ( CheckDef == MO . isDef ( ) ) return


<s> continue ; } if ( isDirectJump ( I ) && ! J . isBranch ( ) && ! J . isCall ( ) && DepType == SDep :: Order ) continue ; if ( I . isConditionalBranch ( ) && DepType != SDep :: Data && DepType != SDep :: Output ) continue ; if ( DepType == SDep :: Output ) { FoundSequentialDependence = true ; break ; } if ( DepType == SDep :: Order ) { if ( ! PacketizeVolatiles ) { bool OrdRefs = I . hasOrderedMemoryRef ( ) || J . hasOrderedMemoryRef ( ) ; if ( OrdRefs ) { FoundSequentialDependence = true ; break ; } } bool LoadJ = J . mayLoad ( ) , StoreJ = J . mayStore ( ) ; bool LoadI = I . mayLoad ( ) , StoreI = I . mayStore ( ) ; bool NVStoreJ = HII -> isNewValueStore ( J ) ; bool NVStoreI = HII -> isNewValueStore ( I ) ; bool IsVecJ = HII -> isHVXVec ( J ) ; bool IsVecI = HII -> isHVXVec ( I ) ; if ( Slot1Store && MF . getSubtarget < <ISA_LIT>Subtarget > ( ) . hasV65Ops ( ) && ( ( LoadJ && StoreI && ! NVStoreI ) || ( StoreJ && LoadI && ! NVStoreJ ) ) && ( J . getOpcode ( ) != <STR_LIT> && I . getOpcode ( ) != <STR_LIT> ) && ( J . getOpcode ( ) != <STR_LIT> && I . getOpcode ( ) != <STR_LIT> ) && ( ! HII -> isMemOp ( J ) && ! HII -> isMemOp ( I ) ) && ( ! IsVecJ && ! IsVecI ) ) setmemShufDisabled ( true ) ; else if ( StoreJ && LoadI && alias ( J , I ) ) { FoundSequentialDependence = true ; break ; } if ( ! StoreJ ) if ( ! LoadJ || ( ! LoadI && ! StoreI ) ) { FoundSequentialDependence = true ; break ; } continue ; } if ( DepType == SDep :: Data && J . getOpcode ( ) == <STR_LIT> ) { unsigned Opc = I . getOpcode ( ) ; switch ( Opc ) { case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : if ( I . getOperand ( <NUM_LIT> ) . getReg ( ) == HRI -> getStackRegister ( ) ) { GlueAllocframeStore = useCallersSP ( I ) ; if ( GlueAllocframeStore ) continue ; } break ; default : break ; } } if ( ( DepType == SDep :: Anti || DepType == SDep :: Output ) && J . isCall ( ) ) { for ( const MachineOperand & Op :


<s> if ( Coexist && ! Dependence ) return true ; if ( PromotedToDotNew ) demoteToDotOld ( I ) ; cleanUpDotCur ( ) ; if ( GlueAllocframeStore ) { useCalleesSP ( I ) ; GlueAllocframeStore = false ; } if ( ChangedOffset != INT64_MAX ) undoChangedOffset ( I ) ; if ( GlueToNewValueJump ) { GlueToNewValueJump =


<s> static bool isRegDependence ( const SDep :: Kind DepType


<s> if ( MI . isEHLabel ( ) || MI . isCFIInstruction ( ) ) return true ; if ( MI . isInlineAsm ( ) && ! ScheduleInlineAsm ) return true ; if ( isSchedBarrier ( MI ) )


<s> static bool isSystemInstr ( const MachineInstr & MI ) { unsigned Opc = MI . getOpcode ( ) ; switch ( Opc ) { case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : return true


<s> MI . clearFlag ( MachineInstr :: BundledSucc ) ; MI . clearFlag ( MachineInstr :: BundledPred ) ; } else { MI . unbundleFromPred ( ) ; } B . splice ( InsertPt , & B , MI . getIterator ( ) ) ; MachineBasicBlock :: const_instr_iterator I = BundleIt . getInstrIterator ( ) ; MachineBasicBlock :: const_instr_iterator E = B . instr_end ( ) ; unsigned Size = <NUM_LIT> ; for ( ++ I ; I != E && I -> isBundledWithPred ( ) ; ++ I ) ++ Size ; if ( Size > <NUM_LIT> ) return BundleIt ; MachineBasicBlock :: iterator NextIt = std ::


<s> MachineBasicBlock & B = * MI . getParent ( ) ; assert ( MI . isBundledWithPred ( ) ) ; if ( MI . isBundledWithSucc ( ) ) { MI . clearFlag ( MachineInstr :: BundledSucc ) ; MI . clearFlag ( MachineInstr :: BundledPred ) ; } else { MI . unbundleFromPred ( ) ; } B . splice ( InsertPt , & B , MI . getIterator ( ) ) ; MachineBasicBlock :: const_instr_iterator I = BundleIt . getInstrIterator ( ) ; MachineBasicBlock :: const_instr_iterator E = B . instr_end ( ) ; unsigned Size = <NUM_LIT> ; for ( ++ I ; I != E && I -> isBundledWithPred ( ) ; ++ I ) ++ Size ; if ( Size > <NUM_LIT> ) return BundleIt ; MachineBasicBlock :: iterator NextIt = std :: next ( BundleIt ) ; MachineInstr & SingleI = * BundleIt -> getNextNode


<s> bool <ISA_LIT>PacketizerList :: promoteToDotCur ( MachineInstr & MI , SDep :: Kind DepType , MachineBasicBlock :: iterator & MII , const TargetRegisterClass * RC ) { assert ( DepType ==


<s> bool <ISA_LIT>PacketizerList :: promoteToDotCur ( MachineInstr & MI , SDep :: Kind DepType , MachineBasicBlock :: iterator & MII , const TargetRegisterClass * RC ) { assert ( DepType == SDep :: Data ) ; int CurOpcode = HII -> getDotCurOp ( MI ) ; MI . setDesc ( HII -> get (


<s> assert ( DepType == SDep :: Data ) ; int NewOpcode ; if ( RC == & <STR_LIT> ) NewOpcode = HII -> getDotNewPredOp ( MI , MBPI ) ; else NewOpcode = HII -> getDotNewOp ( MI ) ; MI . setDesc ( HII -> get (


<s> while ( Begin != End ) { MachineBasicBlock :: iterator RB = Begin ; while ( RB != End && HII -> isSchedulingBoundary ( * RB , & MB , MF ) ) ++ RB ; MachineBasicBlock :: iterator RE = RB ; while ( RE != End && ! HII -> isSchedulingBoundary ( * RE , & MB , MF ) ) ++ RE ; if ( RE != End ) ++ RE ; if ( RB != End ) Packetizer . PacketizeMIs ( & MB , RB , RE ) ; Begin = RE ; } } Packetizer . unpacketizeSoloInstrs ( MF ) ; return


<s> auto * ExtMI = MF . CreateMachineInstr ( HII -> get ( <STR_LIT> ) , DebugLoc ( ) ) ; bool Avail = ResourceTracker -> canReserveResources ( * ExtMI ) ; if ( Reserve && Avail ) ResourceTracker -> reserveResources


<s> void <ISA_LIT>PacketizerList :: undoChangedOffset ( MachineInstr & MI ) { unsigned BP ,


<s> assert ( SUI -> getInstr ( ) && SUJ -> getInstr ( ) ) ; MachineInstr & MI = * SUI -> getInstr ( ) ; MachineInstr & MJ = * SUJ -> getInstr ( ) ; unsigned BPI , OPI ; if ( ! HII -> getBaseAndOffsetPosition ( MI , BPI , OPI ) ) return false ; unsigned BPJ , OPJ ; if ( ! HII -> getBaseAndOffsetPosition ( MJ , BPJ , OPJ ) ) return false ; unsigned Reg = MI . getOperand ( BPI ) . getReg ( ) ; if ( Reg != MJ . getOperand ( BPJ ) . getReg ( ) ) return false ; for ( const auto & PI : SUI -> Preds ) if ( PI . getKind ( ) != SDep :: Anti && ( PI . getKind ( ) != SDep :: Data || PI . getReg ( ) != Reg ) ) return false ; int Incr ; if ( ! HII -> getIncrementValue ( MJ , Incr ) ) return false ; int64_t Offset = MI . getOperand ( OPI ) . getImm ( ) ; if ( ! HII -> isValidOffset ( MI . getOpcode ( ) , Offset + Incr , HRI ) ) return false ; MI . getOperand ( OPI ) . setImm ( Offset + Incr ) ; ChangedOffset = Offset


<s> } unsigned FrameSize = MF . getFrameInfo ( ) . getStackSize ( ) ; MachineOperand & Off = MI . getOperand ( <NUM_LIT> ) ; Off . setImm ( Off . getImm ( )


<s> MachineInstr & NvjMI = * ++ MII ; ResourceTracker -> reserveResources ( MI ) ; if ( ExtMI ) Good = tryAllocateResourcesForConstExt ( true ) ; bool ExtNvjMI = HII -> isExtended ( NvjMI ) || HII -> isConstExtended ( NvjMI ) ; if ( Good ) { if ( ResourceTracker -> canReserveResources ( NvjMI ) ) ResourceTracker -> reserveResources ( NvjMI ) ; else Good = false ; } if ( Good && ExtNvjMI ) Good = tryAllocateResourcesForConstExt ( true ) ; if ( ! Good ) { endPacket ( MBB , MI ) ; assert ( ResourceTracker -> canReserveResources ( MI ) ) ; ResourceTracker -> reserveResources ( MI ) ; if ( ExtMI ) { assert ( canReserveResourcesForConstExt ( ) ) ; tryAllocateResourcesForConstExt ( true ) ; } assert ( ResourceTracker -> canReserveResources ( NvjMI ) ) ; ResourceTracker -> reserveResources ( NvjMI ) ; if ( ExtNvjMI ) { assert ( canReserveResourcesForConstExt ( ) ) ; reserveResourcesForConstExt ( ) ; } } CurrentPacketMIs . push_back ( & MI


<s> if ( ! isNewifiable ( MI , RC ) ) return false ; const MachineInstr & PI = * PacketSU -> getInstr ( ) ; if ( PI . isInlineAsm ( ) ) return false ; if ( PI . isImplicitDef ( ) ) return false ; if ( isImplicitDependency ( PI , true , DepReg ) || isImplicitDependency ( MI , false , DepReg ) ) return false ; const MCInstrDesc & MCID = PI . getDesc ( ) ; const TargetRegisterClass * VecRC = HII -> getRegClass ( MCID , <NUM_LIT> , HRI , MF ) ; if ( DisableVecDblNVStores && VecRC == & <STR_LIT> ) return false ; if ( RC == & <STR_LIT> ) return HII -> predCanBeUsedAsDotNew ( PI ,


<s> } if ( MI ) { for ( auto & MO : BI -> operands ( ) ) if ( MO . isReg ( ) && MO . getReg ( ) == MI -> getOperand ( <NUM_LIT> ) . getReg ( ) ) return ; } } if ( ! MI ) return ; MI -> setDesc ( HII -> get ( HII ->


<s> if ( HII -> isDotCurInst ( * BI ) ) { MI = BI ; continue ; } if ( MI ) { for ( auto & MO : BI -> operands ( ) ) if ( MO . isReg ( ) && MO . getReg (

) == MI -> getOperand ( <NUM_LIT> ) . getReg ( ) ) return ;


<s> OldPacketMIs = CurrentPacketMIs ; VLIWPacketizerList :: endPacket ( MBB ,


<s> HII = MF . getSubtarget < <ISA_LIT>Subtarget > ( ) . getInstrInfo ( ) ; HRI = MF . getSubtarget < <ISA_LIT>Subtarget > ( ) . getRegisterInfo ( ) ; addMutation ( make_unique < <ISA_LIT>Subtarget ::


<s> HII = MF . getSubtarget < <ISA_LIT>Subtarget > ( ) . getInstrInfo ( ) ; HRI = MF . getSubtarget < <ISA_LIT>Subtarget > ( ) . getRegisterInfo ( ) ; addMutation ( make_unique < <ISA_LIT>Subtarget :: <ISA_LIT>DAGMutation > ( )


<s> void <ISA_LIT>PacketizerList :: initPacketizerState ( ) { Dependence = false


<s> bool <ISA_LIT>PacketizerList :: isSoloInstruction ( const MachineInstr & MI ) { if ( MI . isEHLabel ( ) || MI . isCFIInstruction ( ) ) return true ; if ( MI . isInlineAsm ( ) && ! ScheduleInlineAsm ) return true ; if ( isSchedBarrier ( MI ) ) return true ; if ( HII -> isSolo ( MI ) ) return true ; if ( MI . getOpcode ( )


<s> bool <ISA_LIT>PacketizerList :: promoteToDotNew ( MachineInstr & MI , SDep :: Kind DepType , MachineBasicBlock :: iterator & MII , const TargetRegisterClass * RC ) { assert ( DepType == SDep :: Data ) ; int NewOpcode ; if ( RC == & <STR_LIT> ) NewOpcode = HII -> getDotNewPredOp ( MI , MBPI ) ; else NewOpcode = HII -> getDotNewOp ( MI ) ; MI . setDesc ( HII -> get ( NewOpcode ) ) ; return


<s> while ( MI != End ) { auto NextI = std :: next ( MI ) ; if ( MI -> isKill ( ) ) { MB . erase ( MI ) ; End = MB . end ( ) ; } MI = NextI ; } } for ( auto & MB : MF ) { auto Begin = MB . begin ( ) , End = MB . end ( ) ; while ( Begin != End ) { MachineBasicBlock :: iterator RB = Begin ; while ( RB != End && HII -> isSchedulingBoundary ( * RB , & MB


<s> bool <ISA_LIT>PacketizerList :: shouldAddToPacket ( const MachineInstr & MI ) { return ! producesStall


<s> case <STR_LIT> : break ; default : llvm_unreachable ( <STR_LIT> ) ; } unsigned FrameSize = MF . getFrameInfo ( ) . getStackSize ( ) ; MachineOperand & Off = MI . getOperand ( <NUM_LIT> ) ; int64_t NewOff = Off . getImm ( ) - ( FrameSize + HEXAGON_LRFP_SIZE ) ; if ( HII -> isValidOffset ( Opc , NewOff


<s> bool <ISA_LIT>Packetizer :: runOnMachineFunction ( MachineFunction & MF ) { MF . getProperties ( ) . set ( MachineFunctionProperties :: Property :: FailsVerification ) ; auto & HST = MF . getSubtarget < <ISA_LIT>Subtarget > ( ) ; HII = HST . getInstrInfo ( ) ; HRI = HST . getRegisterInfo ( ) ; auto & MLI = getAnalysis < MachineLoopInfo > ( ) ; auto * AA = & getAnalysis < AAResultsWrapperPass > ( ) . getAAResults ( ) ; auto * MBPI = & getAnalysis < MachineBranchProbabilityInfo > ( ) ; if ( EnableGenAllInsnClass ) HII -> genAllInsnTimingClasses ( MF ) ; bool MinOnly = Minimal || DisablePacketizer || ! HST . usePackets ( ) || skipFunction ( MF . getFunction ( ) ) ; <ISA_LIT>PacketizerList Packetizer ( MF , MLI , AA , MBPI , MinOnly ) ; assert ( Packetizer . getResourceTracker ( ) && <STR_LIT> ) ; for ( MachineBasicBlock & MB : MF ) { for ( MachineInstr & MI : llvm :: make_early_inc_range ( MB ) ) if ( MI . isKill ( ) ) MB . erase ( & MI ) ; } if ( HST . isTinyCoreWithDuplex ( ) ) HII -> translateInstrsForDup ( MF , true ) ; for ( auto & MB : MF ) { auto Begin = MB . begin ( ) , End = MB . end ( ) ; while ( Begin != End ) { MachineBasicBlock :: iterator RB = Begin ; while ( RB != End && HII -> isSchedulingBoundary ( * RB , & MB


<s> assert ( Packetizer . getResourceTracker ( ) && <STR_LIT> ) ; for ( MachineBasicBlock & MB : MF ) { for ( MachineInstr & MI : llvm :: make_early_inc_range ( MB ) ) if ( MI . isKill ( ) ) MB . erase ( & MI ) ; } if ( HST . isTinyCoreWithDuplex ( ) ) HII -> translateInstrsForDup ( MF , true ) ; for ( auto & MB : MF ) { auto Begin = MB . begin ( ) , End = MB . end ( ) ; while ( Begin != End ) { MachineBasicBlock :: iterator RB = Begin ; while ( RB != End && HII -> isSchedulingBoundary ( * RB , & MB , MF ) ) ++ RB ; MachineBasicBlock :: iterator RE = RB ; while ( RE != End && ! HII -> isSchedulingBoundary ( * RE , & MB , MF ) ) ++ RE ; if ( RE != End ) ++ RE ; if ( RB != End ) Packetizer . PacketizeMIs ( & MB , RB , RE ) ; Begin = RE ; } } if ( HST . isTinyCoreWithDuplex ( ) ) HII -> translateInstrsForDup ( MF , false ) ; Packetizer . unpacketizeSoloInstrs


<s> HII = MF . getSubtarget < <ISA_LIT>Subtarget > ( ) . getInstrInfo ( ) ; HRI = MF . getSubtarget < <ISA_LIT>Subtarget > ( ) . getRegisterInfo ( ) ; addMutation ( llvm :: make_unique < <ISA_LIT>Subtarget :: UsrOverflowMutation


<s> const MachineFunction * MF = MI . getParent ( ) -> getParent ( ) ; if ( MF -> getSubtarget < <ISA_LIT>Subtarget > ( ) . hasV60TOpsOnly ( ) && HII . isHVXMemWithAIndirect ( MI , MJ ) ) return true ; if ( MI . isInlineAsm ( ) ) return MJ . isInlineAsm ( ) || MJ . isBranch ( ) || MJ . isBarrier ( ) || MJ . isCall ( ) || MJ . isTerminator ( ) ; switch ( MI . getOpcode ( ) ) { case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case


<s> MachineBasicBlock :: instr_iterator FirstMI ( CurrentPacketMIs . front ( ) ) ; MachineBasicBlock :: instr_iterator LastMI ( MI . getInstrIterator ( ) ) ; finalizeBundle ( * MBB , FirstMI , LastMI ) ; auto BundleMII = std :: prev ( FirstMI ) ; if ( memShufDisabled ) HII -> setBundleNoShuf ( BundleMII ) ; setmemShufDisabled ( false ) ; } OldPacketMIs = CurrentPacketMIs ; CurrentPacketMIs . clear ( ) ; ResourceTracker -> clearResources ( ) ; LLVM_DEBUG ( dbgs ( )


<s> auto MI = MB . begin ( ) ; while ( MI != End ) { auto NextI = std :: next ( MI ) ; if ( MI -> isKill ( ) ) { MB . erase ( MI ) ; End = MB . end ( ) ; } MI = NextI ; } } for ( auto & MB : MF ) { auto Begin = MB . begin ( ) , End = MB . end ( ) ; while ( Begin != End ) { MachineBasicBlock :: iterator RB = Begin ; while ( RB != End && HII -> isSchedulingBoundary ( * RB , & MB , MF ) ) ++ RB ; MachineBasicBlock :: iterator RE = RB ; while ( RE != End && ! HII -> isSchedulingBoundary ( * RE , & MB , MF ) ) ++ RE ; if ( RE != End ) ++ RE ; if ( RB != End ) Packetizer . PacketizeMIs ( & MB , RB ,


<s> <ISA_LIT>PacketizerList Packetizer ( MF , MLI , AA , MBPI ) ; assert ( Packetizer . getResourceTracker ( ) && <STR_LIT> ) ; for ( auto & MB : MF ) { auto End = MB . end ( ) ; auto MI = MB . begin ( ) ; while ( MI != End ) { auto NextI = std :: next ( MI ) ; if ( MI -> isKill ( ) ) { MB . erase ( MI ) ; End = MB . end ( ) ; } MI = NextI ; } } for ( auto & MB : MF ) { auto Begin = MB . begin ( ) , End = MB . end ( ) ; while ( Begin != End ) { MachineBasicBlock :: iterator RB = Begin ; while ( RB != End && HII -> isSchedulingBoundary ( * RB , & MB , MF ) ) ++ RB ; MachineBasicBlock :: iterator RE = RB ; while ( RE != End && ! HII -> isSchedulingBoundary ( * RE , & MB , MF ) ) ++ RE ; if ( RE != End ) ++


<s> assert ( ( QII -> isExtended ( MI ) || QII -> isConstExtended ( MI ) ) && <STR_LIT> ) ; MachineFunction * MF = MI -> getParent ( ) -> getParent ( ) ; MachineInstr * PseudoMI = MF -> CreateMachineInstr ( QII -> get ( <STR_LIT> ) , MI -> getDebugLoc ( ) ) ; bool CanReserve = ResourceTracker -> canReserveResources


<s> unsigned SchedClass = TID . getSchedClass ( ) ; const InstrStage * IS = ResourceTracker -> getInstrItins ( ) -> beginStage ( SchedClass ) ; unsigned FuncUnits = IS -> getUnits


<s> const <ISA_LIT>InstrInfo * QII = ( const <ISA_LIT>InstrInfo * ) TII ; if ( isCondInst ( MI ) || QII -> mayBeNewStore ( MI ) ) return true ; else return


<s> if ( isCondInst ( MI ) || QII -> mayBeNewStore ( MI )


<s> if ( MI -> isInlineAsm ( ) ) return true ; if ( MI -> isEHLabel ( ) ) return true ; if ( IsSchedBarrier ( MI ) ) return true ; return


<s> if ( MI -> isInlineAsm ( ) ) return true ; if ( MI -> isEHLabel ( ) ) return true ; if ( IsSchedBarrier ( MI ) ) return true ; return false


<s> MachineFunction * MF = MI -> getParent ( ) -> getParent ( ) ; MachineInstr * PseudoMI = MF -> CreateMachineInstr ( QII -> get ( <STR_LIT> ) , MI -> getDebugLoc ( ) ) ; if ( ResourceTracker -> canReserveResources ( PseudoMI ) ) { ResourceTracker -> reserveResources


<s> MachineFunction * MF = MI -> getParent ( ) -> getParent ( ) ; MachineInstr * PseudoMI = MF -> CreateMachineInstr ( QII -> get ( <STR_LIT> ) , MI -> getDebugLoc ( ) ) ; if ( ResourceTracker -> canReserveResources ( PseudoMI ) ) { ResourceTracker -> reserveResources ( PseudoMI ) ; MI -> getParent ( ) -> getParent ( ) -> DeleteMachineInstr


<s> bool <ISA_LIT>PacketizerList :: tryAllocateResourcesForConstExt ( MachineInstr * MI ) { const <ISA_LIT>InstrInfo * QII = ( const <ISA_LIT>InstrInfo * ) TII ; MachineFunction * MF = MI -> getParent ( ) ->


<s> bool <ISA_LIT>PacketizerList :: CanPromoteToDotNew ( MachineInstr * MI , SUnit * PacketSU , unsigned DepReg , std :: map < MachineInstr * , SUnit * > MIToSUnit , MachineBasicBlock :: iterator & MII , const TargetRegisterClass * RC ) { const <ISA_LIT>InstrInfo * QII = ( const <ISA_LIT>InstrInfo * ) TII ; if ( QII -> isDotNewInst ( MI ) && ! QII -> mayBeNewStore ( MI ) ) return false ; if ( ! isNewifiable ( MI ) ) return false ; if ( RC == & <STR_LIT> && isCondInst ( MI ) ) return true ; else if ( RC != & <STR_LIT> && ! QII -> mayBeNewStore ( MI ) ) return false ; else { int NewOpcode = QII -> GetDotNewOp ( MI ) ; const MCInstrDesc & desc = QII -> get ( NewOpcode ) ; DebugLoc dl ; MachineInstr * NewMI = MI -> getParent ( ) -> getParent ( ) -> CreateMachineInstr ( desc , dl ) ; bool ResourcesAvailable = ResourceTracker -> canReserveResources ( NewMI ) ; MI -> getParent ( ) -> getParent ( ) -> DeleteMachineInstr ( NewMI


<s> else if ( RC != & <STR_LIT> && ! QII -> mayBeNewStore ( MI ) ) return false ; else { int NewOpcode = QII -> GetDotNewOp ( MI ) ; const MCInstrDesc & desc = QII -> get ( NewOpcode ) ; DebugLoc dl ; MachineInstr * NewMI = MI -> getParent ( ) -> getParent ( ) -> CreateMachineInstr ( desc , dl ) ; bool ResourcesAvailable = ResourceTracker -> canReserveResources ( NewMI ) ; MI -> getParent ( ) -> getParent ( ) -> DeleteMachineInstr ( NewMI ) ; if ( ! ResourcesAvailable ) return false ; if ( ! CanPromoteToNewValue ( MI , PacketSU , DepReg , MIToSUnit , MII ) ) { return false ; } } return true


<s> bool <ISA_LIT>PacketizerList :: CanPromoteToNewValue ( MachineInstr * MI , SUnit * PacketSU , unsigned DepReg , std :: map < MachineInstr * , SUnit * > MIToSUnit , MachineBasicBlock :: iterator & MII ) { const <ISA_LIT>InstrInfo * QII = ( const <ISA_LIT>InstrInfo * ) TII ; const <ISA_LIT>RegisterInfo * QRI = ( const <ISA_LIT>RegisterInfo * ) TM . getRegisterInfo ( ) ; if ( ! QRI -> Subtarget . hasV4TOps ( ) || ! QII -> mayBeNewStore ( MI ) ) return false ; MachineInstr * PacketMI = PacketSU


<s> const <ISA_LIT>InstrInfo * QII = ( const <ISA_LIT>InstrInfo * ) TII ; const <ISA_LIT>RegisterInfo * QRI = ( const <ISA_LIT>RegisterInfo * ) TM . getRegisterInfo ( ) ; if ( ! QRI -> Subtarget . hasV4TOps ( ) || ! QII -> mayBeNewStore ( MI ) ) return false ; MachineInstr * PacketMI = PacketSU -> getInstr (


<s> if ( ! QII -> mayBeNewStore ( MI ) ) return false ; if ( GetStoreValueOperand ( MI ) . isReg ( ) && GetStoreValueOperand ( MI ) . getReg ( ) != DepReg ) return false ; const <ISA_LIT>RegisterInfo * QRI = ( const <ISA_LIT>RegisterInfo * ) TM . getRegisterInfo ( ) ; const MCInstrDesc & MCID = PacketMI -> getDesc ( ) ; const TargetRegisterClass * PacketRC = QII -> getRegClass ( MCID , <NUM_LIT> , QRI , MF ) ; for ( std :: vector < MachineInstr * > :: iterator VI = CurrentPacketMIs . begin ( ) , VE = CurrentPacketMIs . end ( ) ; ( VI != VE ) ; ++ VI ) { SUnit * PacketSU = MIToSUnit [ * VI ] ; if ( PacketSU -> getInstr ( ) -> getDesc ( ) . mayStore ( ) || PacketSU -> getInstr ( ) -> getOpcode ( ) == <STR_LIT> || PacketSU -> getInstr ( ) -> getOpcode ( ) == <STR_LIT> ) return false ; } if ( PacketRC == & <STR_LIT> ) { return false ; } if ( QII -> isPostIncrement ( MI ) && MI -> getDesc ( ) . mayStore ( ) && GetPostIncrementOperand ( MI , QII ) . getReg ( ) == DepReg ) { return false ; } if ( QII -> isPostIncrement ( PacketMI ) && PacketMI -> getDesc ( ) . mayLoad ( ) && GetPostIncrementOperand ( PacketMI , QII ) . getReg ( ) == DepReg ) { return false ; } if ( QII -> isPredicated ( PacketMI ) ) { if ( ! QII -> isPredicated ( MI ) ) return false ; unsigned predRegNumSrc = <NUM_LIT> ; unsigned predRegNumDst = <NUM_LIT> ; const TargetRegisterClass * predRegClass = NULL ; for ( unsigned opNum = <NUM_LIT> ; opNum < PacketMI -> getNumOperands ( ) ; opNum ++ ) { if ( PacketMI -> getOperand ( opNum ) . isReg ( ) ) predRegNumSrc = PacketMI -> getOperand ( opNum ) . getReg ( ) ; predRegClass = QRI -> getMinimalPhysRegClass ( predRegNumSrc ) ; if ( predRegClass == & <STR_LIT> ) { break ; } } assert ( ( predRegClass == & <STR_LIT> ) && ( <STR_LIT> ) ) ; for ( unsigned opNum = <NUM_LIT> ; opNum < MI -> getNumOperands ( ) ; opNum ++ ) { if ( MI -> getOperand ( opNum ) . isReg ( ) ) predRegNumDst = MI -> getOperand ( opNum ) . getReg


<s> static bool DoesModifyCalleeSavedReg ( MachineInstr * MI , const TargetRegisterInfo * TRI ) { for ( const uint16_t * CSR = TRI -> getCalleeSavedRegs ( ) ; * CSR ; ++ CSR ) { unsigned CalleeSavedReg = * CSR ; if ( MI -> modifiesRegister ( CalleeSavedReg , TRI ) ) return true ; } return


<s> if ( MI -> modifiesRegister ( CalleeSavedReg , TRI ) ) return true ; } return


<s> AU . addRequired < MachineLoopInfo > ( ) ; AU . addPreserved < MachineLoopInfo > ( ) ; MachineFunctionPass :: getAnalysisUsage (


<s> void getAnalysisUsage ( AnalysisUsage & AU ) const { AU . setPreservesCFG ( ) ; AU . addRequired < MachineDominatorTree > ( ) ; AU . addRequired < MachineBranchProbabilityInfo > (


<s> const char * getPassName ( ) const { return


<s> } if ( QII -> isDeallocRet ( MI ) ) { if ( DepReg == QRI -> getFrameRegister ( ) || DepReg == QRI -> getStackRegister ( ) ) return true ; } const TargetRegisterClass * RC = QRI -> getMinimalPhysRegClass ( DepReg ) ; if ( RC == & <STR_LIT> ) { return true ; } if ( IsIndirectCall ( MI ) && ( DepType == SDep :: Data ) ) { MachineOperand MO = MI -> getOperand ( <NUM_LIT> ) ; if ( MO . isReg ( ) && MO . isUse ( ) && ( MO . getReg ( ) == DepReg ) ) { return true


<s> const <ISA_LIT>InstrInfo * QII = ( const <ISA_LIT>InstrInfo * ) TII ; MachineFunction * MF = MI -> getParent ( ) -> getParent ( ) ; MachineInstr * PseudoMI = MF -> CreateMachineInstr ( QII -> get ( <STR_LIT> ) , MI -> getDebugLoc ( ) ) ; if ( ResourceTracker -> canReserveResources ( PseudoMI ) ) { ResourceTracker -> reserveResources ( PseudoMI ) ; MI -> getParent ( ) -> getParent ( ) -> DeleteMachineInstr ( PseudoMI ) ; }


<s> void <ISA_LIT>PacketizerList :: reserveResourcesForConstExt ( MachineInstr * MI ) { const <ISA_LIT>InstrInfo * QII = ( const <ISA_LIT>InstrInfo * ) TII ; MachineFunction * MF = MI -> getParent ( ) -> getParent ( ) ; MachineInstr * PseudoMI = MF -> CreateMachineInstr ( QII -> get ( <STR_LIT> ) , MI -> getDebugLoc ( ) ) ; if ( ResourceTracker -> canReserveResources ( PseudoMI ) ) { ResourceTracker -> reserveResources ( PseudoMI ) ; MI -> getParent ( ) -> getParent ( ) -> DeleteMachineInstr ( PseudoMI ) ; }


<s> End = MBB -> end ( ) ; continue ; } ++ MI ; } } for ( MachineFunction :: iterator MBB = Fn . begin ( ) , MBBe = Fn . end ( ) ; MBB != MBBe ; ++ MBB ) { unsigned RemainingCount = MBB -> size ( ) ; for ( MachineBasicBlock :: iterator RegionEnd = MBB -> end ( ) ; RegionEnd != MBB -> begin ( ) ; ) { MachineBasicBlock :: iterator I = RegionEnd ; for ( ; I != MBB -> begin ( ) ; -- I , -- RemainingCount ) { if ( TII -> isSchedulingBoundary ( std :: prev ( I ) , MBB , Fn ) ) break ; } I = MBB -> begin ( ) ; if ( I == RegionEnd ) { RegionEnd = std :: prev ( RegionEnd ) ; -- RemainingCount ; continue ; } if ( I == std :: prev ( RegionEnd ) ) { RegionEnd = std :: prev ( RegionEnd ) ; continue ; } Packetizer . PacketizeMIs ( MBB , I , RegionEnd ) ; RegionEnd = I ; } } return true


<s> MachineInstr * PseudoMI = MF -> CreateMachineInstr ( QII -> get ( <STR_LIT> ) , MI -> getDebugLoc ( ) ) ; if ( ResourceTracker -> canReserveResources ( PseudoMI ) ) { ResourceTracker -> reserveResources ( PseudoMI


<s> <ISA_LIT>PacketizerList :: <ISA_LIT>PacketizerList ( MachineFunction & MF , MachineLoopInfo & MLI , AliasAnalysis * AA , const MachineBranchProbabilityInfo * MBPI , bool Minimal ) : VLIWPacketizerList ( MF , MLI , AA ) , MBPI ( MBPI ) , MLI


<s> assert ( ResourceTracker -> canReserveResources ( MI ) ) ; ResourceTracker -> reserveResources ( MI ) ; if ( QII -> isExtended ( MI ) && ! tryAllocateResourcesForConstExt ( MI ) ) { endPacket ( MBB , MI ) ; ResourceTracker -> reserveResources ( MI ) ; assert ( canReserveResourcesForConstExt ( MI ) && <STR_LIT> ) ; reserveResourcesForConstExt ( MI ) ; assert ( canReserveResourcesForConstExt ( MI ) && <STR_LIT> ) ; reserveResourcesForConstExt ( nvjMI ) ; assert ( ResourceTracker -> canReserveResources ( nvjMI ) && <STR_LIT> ) ; } else if ( ( QII -> isExtended ( nvjMI ) && ( ! tryAllocateResourcesForConstExt ( nvjMI ) || ! ResourceTracker -> canReserveResources ( nvjMI ) ) ) || ( ! QII -> isExtended ( nvjMI ) && ! ResourceTracker -> canReserveResources ( nvjMI ) ) ) { endPacket ( MBB , MI ) ; ResourceTracker -> reserveResources ( MI ) ; if ( QII -> isExtended ( nvjMI ) ) reserveResourcesForConstExt ( nvjMI ) ; } ResourceTracker -> reserveResources ( nvjMI ) ; CurrentPacketMIs . push_back ( MI ) ; CurrentPacketMIs . push_back ( nvjMI ) ; } else


<s> else { const <ISA_LIT>InstrInfo * QII = ( const <ISA_LIT>InstrInfo * ) TII ; int NewOpcode = GetDotNewOp ( MI -> getOpcode ( ) ) ; const MCInstrDesc & desc = QII -> get ( NewOpcode ) ; DebugLoc dl ; MachineInstr * NewMI = MI -> getParent ( ) -> getParent ( ) -> CreateMachineInstr ( desc , dl ) ; bool ResourcesAvailable = ResourceTracker -> canReserveResources ( NewMI ) ; MI -> getParent ( ) -> getParent ( ) -> DeleteMachineInstr ( NewMI ) ; if ( ! ResourcesAvailable ) return false ; if ( ! CanPromoteToNewValue ( MI , PacketSU , DepReg , MIToSUnit , MII ) ) { return


<s> bool <ISA_LIT>PacketizerList :: CanPromoteToNewValue ( MachineInstr * MI , SUnit * PacketSU , unsigned DepReg , std :: map < MachineInstr * , SUnit * > MIToSUnit , MachineBasicBlock :: iterator & MII ) { const <ISA_LIT>RegisterInfo * QRI = ( const <ISA_LIT>RegisterInfo * ) TM . getRegisterInfo ( ) ; if ( ! QRI -> Subtarget . hasV4TOps ( ) || ! IsNewifyStore ( MI ) ) return false ; MachineInstr * PacketMI = PacketSU -> getInstr ( ) ; if ( CanPromoteToNewValueStore ( MI , PacketMI , DepReg , MIToSUnit ) ) return true ; return


<s> bool <ISA_LIT>PacketizerList :: canReserveResourcesForConstExt ( MachineInstr * MI ) { const <ISA_LIT>InstrInfo * QII = ( const <ISA_LIT>InstrInfo * ) TII ; assert ( QII -> isExtended ( MI ) && <STR_LIT> ) ; MachineFunction * MF = MI -> getParent ( ) -> getParent ( ) ; MachineInstr * PseudoMI = MF -> CreateMachineInstr ( QII -> get ( <STR_LIT> ) , MI -> getDebugLoc ( ) ) ; bool CanReserve = ResourceTracker -> canReserveResources ( PseudoMI ) ; MF -> DeleteMachineInstr ( PseudoMI ) ; return CanReserve


<s> MachineFunction * MF = MI -> getParent ( ) -> getParent ( ) ; MachineInstr * PseudoMI = MF -> CreateMachineInstr ( QII -> get ( <STR_LIT> )


<s> bool <ISA_LIT>PacketizerList :: DemoteToDotOld ( MachineInstr * MI ) { const <ISA_LIT>InstrInfo * QII = ( const <ISA_LIT>InstrInfo * ) TII ; int NewOpcode = GetDotOldOp ( MI -> getOpcode ( )


<s> AU . addRequired < MachineDominatorTree > ( ) ; AU . addPreserved < MachineDominatorTree > ( ) ; AU . addRequired < MachineLoopInfo >


<s> AU . addRequired < MachineDominatorTree > ( ) ; AU . addPreserved < MachineDominatorTree > ( ) ; AU . addRequired < MachineLoopInfo > ( ) ; AU . addPreserved < MachineLoopInfo > ( ) ; MachineFunctionPass :: getAnalysisUsage ( AU