<s> } } Constant * C = nullptr ; if ( hasPtr ) { C = M -> getOrInsertFunction ( FuncName , FuncTy ) ; } else { AttributeList Attr ; LLVMContext & Ctx = M -> getContext ( ) ; Attr . addAttribute ( Ctx , AttributeList :: FunctionIndex , Attribute :: ReadOnly ) ; Attr . addAttribute ( Ctx , AttributeList :: FunctionIndex , Attribute :: NoUnwind ) ; C = M -> getOrInsertFunction ( FuncName , FuncTy , Attr ) ; } return cast < Function >


<s> std :: string const FuncName = fInfo . mangle ( ) ; Function * F = dyn_cast_or_null < Function > ( M -> getValueSymbolTable ( ) . lookup ( FuncName ) ) ; if ( F && ! F -> isDeclaration ( ) && ! F -> isVarArg ( ) && F -> arg_size ( ) == fInfo . getNumArgs ( ) ) { return F ; } FunctionType * FuncTy = fInfo . getFunctionType ( * M ) ; bool hasPtr = false ; for ( FunctionType :: param_iterator PI = FuncTy -> param_begin ( ) , PE = FuncTy -> param_end ( ) ; PI != PE ; ++ PI ) { const Type * argTy = static_cast < const Type * > ( * PI ) ; if ( argTy -> isPointerTy ( ) ) { hasPtr = true ; break ; } } Constant * C = nullptr ; if ( hasPtr ) { C = M -> getOrInsertFunction ( FuncName , FuncTy ) ; } else { AttributeList Attr ; LLVMContext & Ctx = M -> getContext ( ) ; Attr = Attr . addAttribute ( Ctx , AttributeList :: FunctionIndex , Attribute :: ReadOnly ) ; Attr = Attr . addAttribute ( Ctx , AttributeList ::

FunctionIndex , Attribute :: NoUnwind ) ;


<s> auto Loc = Map . find ( Name ) ; if ( Loc != Map . end ( ) ) { Id = toFuncId ( Loc -> second ) ; return true ; } Id = <ISA_LIT>LibFunc :: EI_NONE ; return false


<s> ManglingRulesMap ( ) : StringMap < int > ( sizeof ( manglingRules ) / sizeof ( manglingRules [ <NUM_LIT> ] ) ) { int Id


<s> for ( unsigned I = <NUM_LIT> ; I != TableSize ; ++ I ) ( * this ) [ Table [ I ]


<s> FuncId = static_cast < EFuncId > ( manglingRulesMap . lookup ( FullName ) ) ; return FuncId !=


<s> FuncId = static_cast < EFuncId > ( manglingRulesMap . lookup


<s> } else { switch ( param . front ( ) ) { case '1' : res . PtrKind |= <ISA_LIT>LibFunc :: GLOBAL ; break ; case '2' : res . PtrKind |= <ISA_LIT>LibFunc :: READONLY ; break ; case '3' : res . PtrKind |= <ISA_LIT>LibFunc :: LOCAL ; break ; case '4' : res . PtrKind |= <ISA_LIT>LibFunc :: GENERIC ; break ; case '5' : res . PtrKind |= <ISA_LIT>LibFunc :: OTHER ; break ; default : return false ; } drop_front ( param , <NUM_LIT> ) ; } } else { res . PtrKind = <ISA_LIT>LibFunc :: BYVALUE ; } if ( eatTerm ( param , <STR_LIT> ) ) { res . VectorSize = parseVecSize ( param ) ; if ( res . VectorSize == <NUM_LIT> || ! eatTerm ( param , '_' ) ) return false ; } char const TC = param . front ( ) ; if ( :: isDigit ( TC ) ) { res . ArgType = StringSwitch < <ISA_LIT>LibFunc :: EType > ( eatLengthPrefixedName ( param ) ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG1DA ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG1DB ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG2DA ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG1D ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG2D ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG3D ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: DUMMY ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: DUMMY ) . Default ( <ISA_LIT>LibFunc :: DUMMY ) ; } else { drop_front ( param ) ; switch ( TC ) { case 'h' : res . ArgType = <ISA_LIT>LibFunc :: U8 ; break ; case 't' : res . ArgType = <ISA_LIT>LibFunc :: U16 ; break ; case 'j' : res . ArgType = <ISA_LIT>LibFunc :: U32 ; break ; case 'm' : res . ArgType = <ISA_LIT>LibFunc :: U64 ; break ; case 'c' : res . ArgType = <ISA_LIT>LibFunc :: I8 ; break ; case 's' : res . ArgType = <ISA_LIT>LibFunc :: I16 ; break ; case 'i' : res . ArgType = <ISA_LIT>LibFunc :: I32 ; break ; case 'l' : res . ArgType = <ISA_LIT>LibFunc :: I64 ; break ; case 'f' : res . ArgType = <ISA_LIT>LibFunc :: F32 ; break ; case 'd' : res . ArgType =


<s> void setName ( StringRef N ) { Name = std :: string ( N


<s> case <ISA_LIT>LibFunc :: U16 : case <ISA_LIT>LibFunc :: I16 : T = Type :: getInt16Ty ( C ) ; break ; case <ISA_LIT>LibFunc :: U32 : case <ISA_LIT>LibFunc :: I32 : T = Type :: getInt32Ty ( C ) ; break ; case <ISA_LIT>LibFunc :: U64 : case <ISA_LIT>LibFunc :: I64 : T = Type :: getInt64Ty ( C ) ; break ; case <ISA_LIT>LibFunc :: F16 : T = Type :: getHalfTy ( C ) ; break ; case <ISA_LIT>LibFunc :: F32 : T = Type :: getFloatTy ( C ) ; break ; case <ISA_LIT>LibFunc :: F64 : T = Type :: getDoubleTy ( C ) ; break ; case <ISA_LIT>LibFunc :: IMG1DA : case <ISA_LIT>LibFunc :: IMG1DB : case <ISA_LIT>LibFunc :: IMG2DA : case <ISA_LIT>LibFunc :: IMG1D : case <ISA_LIT>LibFunc :: IMG2D : case <ISA_LIT>LibFunc :: IMG3D : T = StructType :: create ( C , <STR_LIT> ) -> getPointerTo ( ) ; break ; case <ISA_LIT>LibFunc :: SAMPLER : T = StructType :: create ( C , <STR_LIT> ) -> getPointerTo ( ) ; break ; case <ISA_LIT>LibFunc :: EVENT : T = StructType :: create ( C , <STR_LIT> ) -> getPointerTo ( ) ; break ; default : llvm_unreachable ( <STR_LIT> ) ; return nullptr ; } if ( P . VectorSize > <NUM_LIT> ) T = VectorType :: get ( T , P . VectorSize ) ; if ( P . PtrKind != <ISA_LIT>LibFunc :: BYVALUE ) T = useAddrSpace ? T -> getPointerTo ( ( P . PtrKind & <ISA_LIT>LibFunc :: ADDR_SPACE ) - <NUM_LIT> ) : T -> getPointerTo ( ) ; return


<s> std :: string <ISA_LIT>MangledLibFunc :: getName ( ) const { SmallString < <NUM_LIT>


<s> } FunctionCallee C ; if ( hasPtr ) { C = M -> getOrInsertFunction ( FuncName , FuncTy ) ; } else { AttributeList Attr ; LLVMContext & Ctx = M -> getContext ( ) ; Attr = Attr . addAttribute ( Ctx , AttributeList :: FunctionIndex , Attribute :: ReadOnly ) ; Attr = Attr . addAttribute ( Ctx , AttributeList :: FunctionIndex , Attribute :: NoUnwind ) ; C = M -> getOrInsertFunction ( FuncName , FuncTy ,


<s> ItaniumMangler Mangler ( true ) ; ParamIterator I ( Leads , manglingRules [ FuncId ] ) ; Param P ; while ( ( P = I . getNextParam ( ) ) . ArgType != <NUM_LIT> ) Mangler ( S , P ) ; return S . str


<s> if ( eatTerm ( FuncName , <STR_LIT> ) ) F . Impl = make_unique < <ISA_LIT>MangledLibFunc > ( ) ; else F . Impl = make_unique < <ISA_LIT>UnmangledLibFunc > ( ) ; if ( F . Impl -> parseFuncName ( FuncName )


<s> if ( eatTerm ( param , 'K' ) ) res . PtrKind |= <ISA_LIT>LibFunc :: CONST ; if ( eatTerm ( param , 'V' ) ) res . PtrKind |= <ISA_LIT>LibFunc :: VOLATILE ; unsigned AS ; if ( ! eatTerm ( param , <STR_LIT> ) ) { AS = <NUM_LIT> ; } else { AS = param . front ( ) - '0' ; drop_front ( param , <NUM_LIT> ) ; } res . PtrKind |= <STR_LIT> ( AS ) ; } else { res . PtrKind = <ISA_LIT>LibFunc :: BYVALUE ; } if ( eatTerm ( param , <STR_LIT> ) ) { res . VectorSize = parseVecSize ( param ) ; if ( res . VectorSize == <NUM_LIT> || ! eatTerm ( param , '_' ) ) return false ; } char const TC = param . front ( ) ; if ( :: isDigit ( TC ) ) { res . ArgType = StringSwitch < <ISA_LIT>LibFunc :: EType > ( eatLengthPrefixedName ( param ) ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG1DA ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG1DB ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG2DA ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG1D ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG2D ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG3D ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: DUMMY ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: DUMMY ) . Default ( <ISA_LIT>LibFunc :: DUMMY ) ; } else { drop_front ( param ) ; switch ( TC ) { case 'h' : res . ArgType = <ISA_LIT>LibFunc :: U8 ; break ; case 't' : res . ArgType = <ISA_LIT>LibFunc ::


<s> if ( eatTerm ( param , 'V' ) ) res . PtrKind |= <ISA_LIT>LibFunc :: VOLATILE ; unsigned AS ; if ( ! eatTerm ( param , <STR_LIT> ) ) { AS = <NUM_LIT> ; } else { AS = param . front ( ) - '0' ; drop_front ( param , <NUM_LIT> ) ; } res . PtrKind |= <STR_LIT> ( AS ) ; } else { res . PtrKind = <ISA_LIT>LibFunc :: BYVALUE ; } if ( eatTerm ( param , <STR_LIT> ) ) { res . VectorSize = parseVecSize ( param ) ; if ( res . VectorSize == <NUM_LIT> || ! eatTerm ( param , '_' ) ) return false ; } char const TC = param . front ( ) ; if ( :: isDigit ( TC ) ) { res . ArgType = StringSwitch < <ISA_LIT>LibFunc :: EType > ( eatLengthPrefixedName ( param ) ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG1DA ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG1DB ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG2DA ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG1D ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG2D ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG3D ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: DUMMY ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: DUMMY ) . Default ( <ISA_LIT>LibFunc :: DUMMY ) ; } else { drop_front ( param ) ; switch ( TC ) { case 'h' : res . ArgType = <ISA_LIT>LibFunc :: U8 ; break ; case 't' : res . ArgType = <ISA_LIT>LibFunc :: U16 ; break ; case 'j' : res . ArgType = <ISA_LIT>LibFunc :: U32 ; break ; case 'm' : res . ArgType = <ISA_LIT>LibFunc :: U64 ; break ; case 'c' : res . ArgType = <ISA_LIT>LibFunc :: I8 ; break ; case 's' : res . ArgType = <ISA_LIT>LibFunc :: I16 ; break ; case 'i' : res . ArgType = <ISA_LIT>LibFunc :: I32 ; break ; case 'l' : res . ArgType = <ISA_LIT>LibFunc :: I64 ; break ; case 'f' : res . ArgType = <ISA_LIT>LibFunc :: F32 ; break ; case 'd' : res . ArgType = <ISA_LIT>LibFunc :: F64 ; break ; case 'D' : if ( ! eatTerm ( param , 'h' ) ) return false ; res . ArgType = <ISA_LIT>LibFunc :: F16 ; break ; case 'S' : if ( ! eatTerm ( param , '_' ) ) { eatNumber ( param ) ; if ( ! eatTerm ( param , '_' ) ) return false ; } res . VectorSize = Prev . VectorSize ; res . ArgType = Prev . ArgType ; break ; default : ; } } if ( res . ArgType == <NUM_LIT> ) return false ; Prev . VectorSize = res . VectorSize ; Prev . ArgType = res .


<s> Function * ParentFunc = Memcpy -> getParent ( ) -> getParent ( ) ; const TargetTransformInfo & TTI = getAnalysis < TargetTransformInfoWrapperPass > ( ) . getTTI ( * ParentFunc ) ; expandMemCpyAsLoop ( Memcpy , TTI ) ; Changed = true ; Memcpy -> eraseFromParent ( ) ; } break ; } case <STR_LIT> : { auto * Memmove = cast < MemMoveInst > ( Inst ) ; if ( shouldExpandOperationWithSize ( Memmove -> getLength ( ) ) ) { expandMemMoveAsLoop ( Memmove ) ; Changed = true ; Memmove -> eraseFromParent


<s> AU . addRequired < TargetTransformInfoWrapperPass > (


<s> return new <ISA_LIT>LowerIntrinsics ( TM


<s> bool <ISA_LIT>LowerIntrinsics :: makeLIDRangeMetadata ( Function & F ) const { auto * TPC = getAnalysisIfAvailable < TargetPassConfig > ( ) ; if ( ! TPC ) return false ; const TargetMachine & TM = TPC -> getTM < TargetMachine > ( ) ; bool Changed = false ; for ( auto * U : F . users ( ) ) { auto * CI = dyn_cast < CallInst > ( U ) ; if ( ! CI ) continue ; Changed |= <ISA_LIT>Subtarget :: get ( TM , F ) . makeLIDRangeMetadata ( CI ) ; } return


<s> if ( ! TPC ) return false ; const TargetMachine & TM = TPC -> getTM < TargetMachine > ( ) ; bool Changed = false ; for ( auto * U : F . users ( ) ) { auto * CI = dyn_cast < CallInst > ( U ) ; if ( ! CI ) continue ; Function * Caller = CI -> getParent ( ) -> getParent ( ) ; const <ISA_LIT>Subtarget & ST = <ISA_LIT>Subtarget :: get ( TM , * Caller ) ; Changed |= ST . makeLIDRangeMetadata ( CI ) ; } return Changed


<s> switch ( F . getIntrinsicID ( ) ) { case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : if ( expandMemIntrinsicUses ( F ) ) Changed = true ; break ; case <STR_LIT> : case <STR_LIT> : case


<s> ModulePass * llvm :: create<ISA_LIT>LowerIntrinsicsPass ( ) { return new <ISA_LIT>LowerIntrinsics (


<s> ModulePass * llvm :: create<ISA_LIT>LowerIntrinsicsPass ( ) { return new <ISA_LIT>LowerIntrinsics


<s> bool Changed = false ; for ( auto * U : F . users ( ) ) { auto * CI = dyn_cast < CallInst > ( U ) ; if ( ! CI ) continue ; Changed |= ST . makeLIDRangeMetadata ( CI ) ; } return


<s> return ! CI || ( CI -> getZExtValue ( )


<s> AdjustedArgTy = V4Ty ; } ArgPtr = Builder . CreateBitCast ( ArgPtr , AdjustedArgTy -> getPointerTo ( AS ) , ArgPtr -> getName ( ) + <STR_LIT> ) ; LoadInst * Load = Builder . CreateAlignedLoad ( AdjustedArgTy , ArgPtr , AdjustedAlign . value ( ) ) ; Load -> setMetadata ( LLVMContext :: MD_invariant_load , MDNode :: get ( Ctx , { } ) ) ; MDBuilder MDB ( Ctx ) ; if ( isa < PointerType > ( ArgTy ) ) { if ( Arg . hasNonNullAttr ( ) ) Load -> setMetadata ( LLVMContext :: MD_nonnull , MDNode :: get ( Ctx , { } ) ) ; uint64_t DerefBytes = Arg . getDereferenceableBytes ( ) ; if ( DerefBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefBytes ) ) ) ) ; } uint64_t DerefOrNullBytes = Arg . getDereferenceableOrNullBytes ( ) ; if ( DerefOrNullBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable_or_null , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefOrNullBytes ) ) ) ) ; } unsigned ParamAlign = Arg . getParamAlignment ( ) ; if ( ParamAlign != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_align , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , ParamAlign ) ) ) ) ; } } if ( DoShiftOpt ) { Value * ExtractBits = OffsetDiff == <NUM_LIT> ? Load : Builder . CreateLShr ( Load , OffsetDiff * <NUM_LIT> ) ; IntegerType * ArgIntTy = Builder . getIntNTy ( Size ) ; Value * Trunc = Builder . CreateTrunc ( ExtractBits , ArgIntTy ) ; Value * NewVal = Builder . CreateBitCast ( Trunc , ArgTy , Arg . getName ( ) + <STR_LIT> ) ; Arg . replaceAllUsesWith ( NewVal ) ; } else if ( IsV3 ) { Value * Shuf = Builder . CreateShuffleVector ( Load , UndefValue :: get ( V4Ty ) , { <NUM_LIT> , <NUM_LIT> , <NUM_LIT> } , Arg . getName ( ) + <STR_LIT> ) ; Arg . replaceAllUsesWith ( Shuf


<s> if ( CC != CallingConv :: <ISA_LIT>_KERNEL || F . arg_empty ( ) ) return false ; auto & TPC = getAnalysis < TargetPassConfig > ( ) ; const TargetMachine & TM = TPC . getTM < TargetMachine > ( ) ; const GCNSubtarget & ST = TM . getSubtarget < GCNSubtarget > ( F ) ; LLVMContext & Ctx = F . getParent ( ) -> getContext ( ) ; const DataLayout & DL = F . getParent ( ) -> getDataLayout ( ) ; BasicBlock & EntryBlock = * F . begin ( ) ; IRBuilder < > Builder ( & * EntryBlock . begin ( ) ) ; const unsigned KernArgBaseAlign = <NUM_LIT> ; const uint64_t BaseOffset = ST . getExplicitKernelArgOffset ( F ) ; unsigned MaxAlign ; const uint64_t TotalKernArgSize = ST . getKernArgSegmentSize ( F , MaxAlign ) ; if ( TotalKernArgSize == <NUM_LIT> ) return false ; CallInst * KernArgSegment = Builder . CreateIntrinsic ( <STR_LIT> , nullptr , F . getName ( ) + <STR_LIT> ) ; KernArgSegment -> addAttribute ( AttributeList :: ReturnIndex , Attribute :: NonNull ) ; KernArgSegment -> addAttribute ( AttributeList :: ReturnIndex , Attribute :: getWithDereferenceableBytes ( Ctx , TotalKernArgSize ) ) ; unsigned AS = KernArgSegment -> getType ( ) -> getPointerAddressSpace ( ) ; uint64_t ExplicitArgOffset = <NUM_LIT> ; for ( Argument & Arg : F . args ( ) ) { Type * ArgTy = Arg . getType ( ) ; unsigned Align = DL . getABITypeAlignment ( ArgTy ) ; unsigned Size = DL . getTypeSizeInBits ( ArgTy ) ; unsigned AllocSize = DL . getTypeAllocSize ( ArgTy ) ; bool IsExtArg = AllocSize < <NUM_LIT> && ( Arg . hasZExtAttr ( ) || Arg . hasSExtAttr ( ) ) && ! ST . isAmdHsaOS ( ) ; if ( IsExtArg ) AllocSize = <NUM_LIT> ; uint64_t EltOffset = alignTo ( ExplicitArgOffset , Align ) + BaseOffset ; ExplicitArgOffset = alignTo ( ExplicitArgOffset , Align ) + AllocSize ; if ( Arg . use_empty ( ) ) continue ; if ( PointerType * PT = dyn_cast < PointerType > ( ArgTy ) ) { if ( PT -> getAddressSpace ( ) == <ISA_LIT>AS :: LOCAL_ADDRESS && ST . getGeneration ( ) == <ISA_LIT>Subtarget :: SOUTHERN_ISLANDS ) continue ; if ( Arg . hasNoAliasAttr ( ) ) continue ; } VectorType * VT = dyn_cast < VectorType > ( ArgTy ) ; bool IsV3 = VT && VT


<s> int64_t OffsetDiff = EltOffset - AlignDownOffset ; unsigned AdjustedAlign = MinAlign ( KernArgBaseAlign , AlignDownOffset ) ; Value * ArgPtr ; if ( Size < <NUM_LIT> && ! ArgTy -> isAggregateType ( ) ) { ArgPtr = Builder . CreateConstInBoundsGEP1_64 ( KernArgSegment , AlignDownOffset , Arg . getName ( ) + <STR_LIT> ) ; ArgPtr = Builder . CreateBitCast ( ArgPtr , Builder . getInt32Ty ( ) -> getPointerTo ( AS ) , ArgPtr -> getName ( ) + <STR_LIT> ) ; } else { ArgPtr = Builder . CreateConstInBoundsGEP1_64 ( KernArgSegment , AlignDownOffset , Arg . getName ( ) + <STR_LIT> ) ; ArgPtr = Builder . CreateBitCast ( ArgPtr , ArgTy -> getPointerTo ( AS ) , ArgPtr -> getName ( ) + <STR_LIT> ) ; } assert ( ( ! IsExtArg || ! IsV3 ) && <STR_LIT> ) ; if ( IsV3 && Size >= <NUM_LIT> ) { V4Ty = VectorType :: get ( VT -> getVectorElementType ( ) , <NUM_LIT> ) ; ArgPtr = Builder . CreateBitCast ( ArgPtr , V4Ty -> getPointerTo ( AS ) ) ; } LoadInst * Load = Builder . CreateAlignedLoad ( ArgPtr , AdjustedAlign ) ; Load -> setMetadata ( LLVMContext :: MD_invariant_load , MDNode :: get ( Ctx , { } ) ) ; MDBuilder MDB ( Ctx ) ; if ( isa < PointerType > ( ArgTy ) ) { if ( Arg . hasNonNullAttr ( ) ) Load -> setMetadata ( LLVMContext :: MD_nonnull , MDNode :: get ( Ctx , { } ) ) ; uint64_t DerefBytes = Arg . getDereferenceableBytes ( ) ; if ( DerefBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefBytes ) ) ) ) ; } uint64_t DerefOrNullBytes = Arg . getDereferenceableOrNullBytes ( ) ; if ( DerefOrNullBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable_or_null , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefOrNullBytes ) ) ) ) ; } unsigned ParamAlign = Arg . getParamAlignment ( ) ; if ( ParamAlign != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_align , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , ParamAlign ) ) ) ) ; } } if ( Size < <NUM_LIT> && ! ArgTy -> isAggregateType ( ) ) { if ( IsExtArg && OffsetDiff == <NUM_LIT> ) { Type * I32Ty = Builder . getInt32Ty ( ) ; bool IsSext = Arg . hasSExtAttr ( ) ; Metadata * LowAndHigh [ ] = { ConstantAsMetadata :: get ( ConstantInt :: get ( I32Ty , IsSext ? minIntN ( Size ) : <NUM_LIT> ) ) , ConstantAsMetadata :: get ( ConstantInt ::

get ( I32Ty , IsSext ? maxIntN ( Size ) + <NUM_LIT> : maxUIntN ( Size ) + <NUM_LIT> ) ) } ;


<s> if ( Arg . hasNoAliasAttr ( ) ) continue ; } VectorType * VT = dyn_cast < VectorType > ( ArgTy ) ; bool IsV3 = VT && VT -> getNumElements ( ) == <NUM_LIT> ; bool DoShiftOpt = Size < <NUM_LIT> && ! ArgTy -> isAggregateType ( ) ; VectorType * V4Ty = nullptr ; int64_t AlignDownOffset = alignDown ( EltOffset , <NUM_LIT> ) ; int64_t OffsetDiff = EltOffset - AlignDownOffset ; Align AdjustedAlign = commonAlignment ( KernArgBaseAlign , DoShiftOpt ? AlignDownOffset : EltOffset ) ; Value * ArgPtr ; Type * AdjustedArgTy ; if ( DoShiftOpt ) { ArgPtr = Builder . CreateConstInBoundsGEP1_64 ( Builder . getInt8Ty ( ) , KernArgSegment , AlignDownOffset , Arg . getName ( ) + <STR_LIT> ) ; AdjustedArgTy = Builder . getInt32Ty ( ) ; } else { ArgPtr = Builder . CreateConstInBoundsGEP1_64 ( Builder . getInt8Ty ( ) , KernArgSegment , EltOffset , Arg . getName ( ) + <STR_LIT> ) ; AdjustedArgTy = ArgTy ; } if ( IsV3 && Size >= <NUM_LIT> ) { V4Ty = VectorType :: get ( VT -> getVectorElementType ( ) , <NUM_LIT> ) ; AdjustedArgTy = V4Ty ; } ArgPtr = Builder . CreateBitCast ( ArgPtr , AdjustedArgTy -> getPointerTo ( AS ) , ArgPtr -> getName ( ) + <STR_LIT> ) ; LoadInst * Load = Builder . CreateAlignedLoad ( AdjustedArgTy , ArgPtr , AdjustedAlign ) ; Load -> setMetadata ( LLVMContext :: MD_invariant_load , MDNode :: get ( Ctx , { } ) ) ; MDBuilder MDB ( Ctx ) ; if ( isa < PointerType > ( ArgTy ) ) { if ( Arg . hasNonNullAttr ( ) ) Load -> setMetadata ( LLVMContext :: MD_nonnull , MDNode :: get ( Ctx , { } ) ) ; uint64_t DerefBytes = Arg . getDereferenceableBytes ( ) ; if ( DerefBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefBytes ) ) ) ) ; } uint64_t DerefOrNullBytes = Arg . getDereferenceableOrNullBytes ( ) ; if ( DerefOrNullBytes !=


<s> IRBuilder < > Builder ( & * getInsertPt ( EntryBlock ) ) ; const Align KernArgBaseAlign ( <NUM_LIT> ) ; const uint64_t BaseOffset = ST . getExplicitKernelArgOffset ( F ) ; Align MaxAlign ; const uint64_t TotalKernArgSize = ST . getKernArgSegmentSize ( F , MaxAlign ) ; if ( TotalKernArgSize == <NUM_LIT> ) return false ; CallInst * KernArgSegment = Builder . CreateIntrinsic ( <STR_LIT> , { } , { } , nullptr , F . getName ( ) + <STR_LIT> ) ; KernArgSegment -> addAttribute ( AttributeList :: ReturnIndex , Attribute :: NonNull ) ; KernArgSegment -> addAttribute ( AttributeList :: ReturnIndex , Attribute :: getWithDereferenceableBytes ( Ctx , TotalKernArgSize ) ) ; unsigned AS = KernArgSegment -> getType ( ) -> getPointerAddressSpace ( ) ; uint64_t ExplicitArgOffset = <NUM_LIT> ; for ( Argument & Arg : F . args ( ) ) { const bool IsByRef = Arg . hasByRefAttr ( ) ; Type * ArgTy = IsByRef ? Arg . getParamByRefType ( ) : Arg . getType ( ) ; MaybeAlign ABITypeAlign = IsByRef ? Arg . getParamAlign ( ) : None ; if ( ! ABITypeAlign ) ABITypeAlign = DL . getABITypeAlign ( ArgTy ) ; uint64_t Size = DL . getTypeSizeInBits ( ArgTy ) ; uint64_t AllocSize = DL . getTypeAllocSize ( ArgTy ) ; uint64_t EltOffset = alignTo ( ExplicitArgOffset , ABITypeAlign ) + BaseOffset ; ExplicitArgOffset = alignTo ( ExplicitArgOffset , ABITypeAlign ) + AllocSize ; if ( Arg . use_empty ( ) ) continue ; if ( IsByRef ) { Value * ArgOffsetPtr = Builder . CreateConstInBoundsGEP1_64 ( Builder . getInt8Ty ( ) , KernArgSegment , EltOffset , Arg . getName ( ) + <STR_LIT> ) ; Value * CastOffsetPtr = Builder . CreatePointerBitCastOrAddrSpaceCast ( ArgOffsetPtr , Arg . getType ( ) ) ; Arg . replaceAllUsesWith ( CastOffsetPtr ) ; continue ; } if ( PointerType * PT = dyn_cast < PointerType > ( ArgTy ) ) { if ( ( PT -> getAddressSpace ( ) == <ISA_LIT>AS :: LOCAL_ADDRESS || PT -> getAddressSpace ( ) == <ISA_LIT>AS :: REGION_ADDRESS ) && ! ST . hasUsableDSOffset ( ) ) continue ; if ( Arg . hasNoAliasAttr ( ) ) continue ; } auto * VT = dyn_cast < FixedVectorType > ( ArgTy ) ; bool IsV3 = VT && VT -> getNumElements ( ) == <NUM_LIT> ; bool DoShiftOpt = Size < <NUM_LIT> && ! ArgTy -> isAggregateType ( ) ; VectorType * V4Ty = nullptr ; int64_t AlignDownOffset = alignDown ( EltOffset , <NUM_LIT> ) ; int64_t OffsetDiff = EltOffset - AlignDownOffset ; Align AdjustedAlign = commonAlignment ( KernArgBaseAlign , DoShiftOpt ? AlignDownOffset : EltOffset ) ; Value * ArgPtr


<s> AllocaInst * AI = dyn_cast < AllocaInst > ( & * InsPt ) ; if ( ! AI || ! AI -> isStaticAlloca ( ) ) break ; } return InsPt


<s> } if ( IsV3 && Size >= <NUM_LIT> ) { V4Ty = VectorType :: get ( VT -> getVectorElementType ( ) , <NUM_LIT> ) ; AdjustedArgTy = V4Ty ; } ArgPtr = Builder . CreateBitCast ( ArgPtr , AdjustedArgTy -> getPointerTo ( AS ) , ArgPtr -> getName ( ) + <STR_LIT> ) ; LoadInst * Load = Builder . CreateAlignedLoad ( AdjustedArgTy , ArgPtr , AdjustedAlign ) ; Load -> setMetadata ( LLVMContext :: MD_invariant_load , MDNode :: get ( Ctx , { } ) ) ; MDBuilder MDB ( Ctx ) ; if ( isa < PointerType > ( ArgTy ) ) { if ( Arg . hasNonNullAttr ( ) ) Load -> setMetadata ( LLVMContext :: MD_nonnull , MDNode :: get ( Ctx , { } ) ) ; uint64_t DerefBytes = Arg . getDereferenceableBytes ( ) ; if ( DerefBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefBytes ) ) ) ) ; } uint64_t DerefOrNullBytes = Arg . getDereferenceableOrNullBytes ( ) ; if ( DerefOrNullBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable_or_null , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefOrNullBytes ) ) ) ) ; } unsigned ParamAlign = Arg . getParamAlignment ( ) ; if ( ParamAlign != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_align , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , ParamAlign ) ) ) ) ; } } if ( DoShiftOpt ) { Value * ExtractBits = OffsetDiff == <NUM_LIT> ? Load : Builder . CreateLShr ( Load , OffsetDiff * <NUM_LIT> ) ; IntegerType * ArgIntTy = Builder . getIntNTy ( Size ) ; Value * Trunc = Builder . CreateTrunc ( ExtractBits , ArgIntTy ) ; Value * NewVal = Builder . CreateBitCast ( Trunc , ArgTy , Arg


<s> if ( IsV3 && Size >= <NUM_LIT> ) { V4Ty = VectorType :: get ( VT -> getVectorElementType ( ) , <NUM_LIT> ) ; ArgPtr = Builder . CreateBitCast ( ArgPtr , V4Ty -> getPointerTo ( AS ) ) ; } LoadInst * Load = Builder . CreateAlignedLoad ( ArgPtr , AdjustedAlign ) ; Load -> setMetadata ( LLVMContext :: MD_invariant_load , MDNode :: get ( Ctx , { } ) ) ; MDBuilder MDB ( Ctx ) ; if ( isa < PointerType > ( ArgTy ) ) { if ( Arg . hasNonNullAttr ( ) ) Load -> setMetadata ( LLVMContext :: MD_nonnull , MDNode :: get ( Ctx , { } ) ) ; uint64_t DerefBytes = Arg . getDereferenceableBytes ( ) ; if ( DerefBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefBytes ) ) ) ) ; } uint64_t DerefOrNullBytes = Arg . getDereferenceableOrNullBytes ( ) ; if ( DerefOrNullBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable_or_null , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefOrNullBytes ) ) ) ) ; } unsigned ParamAlign = Arg . getParamAlignment ( ) ; if ( ParamAlign != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_align , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , ParamAlign ) ) ) ) ; } } if ( Size < <NUM_LIT> && ! ArgTy -> isAggregateType ( ) ) { Value * ExtractBits = OffsetDiff == <NUM_LIT> ? Load : Builder . CreateLShr ( Load , OffsetDiff * <NUM_LIT> ) ; IntegerType * ArgIntTy = Builder . getIntNTy ( Size ) ; Value * Trunc = Builder . CreateTrunc ( ExtractBits , ArgIntTy ) ; Value * NewVal = Builder . CreateBitCast ( Trunc , ArgTy , Arg . getName ( ) + <STR_LIT> ) ; Arg . replaceAllUsesWith ( NewVal ) ; } else if ( IsV3 ) { Value * Shuf = Builder . CreateShuffleVector ( Load , UndefValue :: get ( V4Ty ) ,

{ <NUM_LIT> , <NUM_LIT> , <NUM_LIT> } , Arg . getName ( ) + <STR_LIT> ) ;


<s> } else { ArgPtr = Builder . CreateConstInBoundsGEP1_64 ( Builder . getInt8Ty ( ) , KernArgSegment , EltOffset , Arg . getName ( ) + <STR_LIT> ) ; AdjustedArgTy = ArgTy ; } if ( IsV3 && Size >= <NUM_LIT> ) { V4Ty = VectorType :: get ( VT -> getVectorElementType ( ) , <NUM_LIT> ) ; AdjustedArgTy = V4Ty ; } ArgPtr = Builder . CreateBitCast ( ArgPtr , AdjustedArgTy -> getPointerTo ( AS ) , ArgPtr -> getName ( ) + <STR_LIT> ) ; LoadInst * Load = Builder . CreateAlignedLoad ( AdjustedArgTy , ArgPtr , AdjustedAlign ) ; Load -> setMetadata ( LLVMContext :: MD_invariant_load , MDNode :: get ( Ctx , { } ) ) ; MDBuilder MDB ( Ctx ) ; if ( isa < PointerType > ( ArgTy ) ) { if ( Arg . hasNonNullAttr ( ) ) Load -> setMetadata ( LLVMContext :: MD_nonnull , MDNode :: get ( Ctx , { } ) ) ; uint64_t DerefBytes = Arg . getDereferenceableBytes ( ) ; if ( DerefBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefBytes ) ) ) ) ; } uint64_t DerefOrNullBytes = Arg . getDereferenceableOrNullBytes ( ) ; if ( DerefOrNullBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable_or_null , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefOrNullBytes ) ) ) ) ; } unsigned ParamAlign = Arg . getParamAlignment ( ) ; if ( ParamAlign != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_align , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , ParamAlign ) ) ) ) ; } } if ( DoShiftOpt ) { Value * ExtractBits = OffsetDiff == <NUM_LIT> ? Load : Builder . CreateLShr ( Load , OffsetDiff * <NUM_LIT> ) ; IntegerType * ArgIntTy = Builder . getIntNTy ( Size ) ; Value * Trunc = Builder . CreateTrunc ( ExtractBits , ArgIntTy ) ; Value * NewVal = Builder . CreateBitCast ( Trunc , ArgTy , Arg . getName ( ) + <STR_LIT> ) ; Arg . replaceAllUsesWith ( NewVal ) ; } else if


<s> for ( Argument & Arg : F . args ( ) ) { Type * ArgTy = Arg . getType ( ) ; Align ABITypeAlign = DL . getABITypeAlign ( ArgTy ) ; unsigned Size = DL . getTypeSizeInBits ( ArgTy ) ; unsigned AllocSize = DL . getTypeAllocSize ( ArgTy ) ; uint64_t EltOffset = alignTo ( ExplicitArgOffset , ABITypeAlign ) + BaseOffset ; ExplicitArgOffset = alignTo ( ExplicitArgOffset , ABITypeAlign ) + AllocSize ; if ( Arg . use_empty ( ) ) continue ; if ( PointerType * PT = dyn_cast < PointerType > ( ArgTy ) ) { if ( ( PT -> getAddressSpace ( ) == <ISA_LIT>AS :: LOCAL_ADDRESS || PT -> getAddressSpace ( ) == <ISA_LIT>AS :: REGION_ADDRESS ) && ! ST . hasUsableDSOffset ( ) ) continue ; if ( Arg . hasNoAliasAttr ( ) ) continue ; } auto * VT = dyn_cast < FixedVectorType > ( ArgTy ) ; bool IsV3 = VT && VT -> getNumElements ( ) == <NUM_LIT> ; bool DoShiftOpt = Size < <NUM_LIT> && ! ArgTy -> isAggregateType ( ) ; VectorType * V4Ty = nullptr ; int64_t AlignDownOffset = alignDown ( EltOffset , <NUM_LIT> ) ; int64_t OffsetDiff = EltOffset - AlignDownOffset ; Align AdjustedAlign = commonAlignment ( KernArgBaseAlign , DoShiftOpt ? AlignDownOffset : EltOffset ) ; Value * ArgPtr ; Type * AdjustedArgTy ; if ( DoShiftOpt ) { ArgPtr = Builder . CreateConstInBoundsGEP1_64 ( Builder . getInt8Ty ( ) , KernArgSegment , AlignDownOffset , Arg . getName ( ) + <STR_LIT> ) ; AdjustedArgTy = Builder . getInt32Ty ( ) ; } else { ArgPtr = Builder . CreateConstInBoundsGEP1_64 ( Builder . getInt8Ty ( ) , KernArgSegment , EltOffset , Arg . getName ( ) + <STR_LIT> ) ; AdjustedArgTy = ArgTy ; } if ( IsV3 && Size >= <NUM_LIT> ) { V4Ty = FixedVectorType :: get ( VT -> getElementType ( ) , <NUM_LIT> ) ; AdjustedArgTy = V4Ty ; } ArgPtr = Builder . CreateBitCast ( ArgPtr , AdjustedArgTy -> getPointerTo ( AS ) , ArgPtr -> getName ( ) + <STR_LIT> ) ; LoadInst * Load = Builder . CreateAlignedLoad ( AdjustedArgTy , ArgPtr , AdjustedAlign ) ; Load -> setMetadata ( LLVMContext :: MD_invariant_load , MDNode :: get ( Ctx , { } ) ) ; MDBuilder MDB ( Ctx ) ; if ( isa < PointerType > ( ArgTy ) ) { if ( Arg . hasNonNullAttr ( ) ) Load -> setMetadata ( LLVMContext :: MD_nonnull , MDNode :: get ( Ctx , { } ) ) ; uint64_t DerefBytes = Arg . getDereferenceableBytes ( ) ; if ( DerefBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefBytes ) ) ) ) ; } uint64_t DerefOrNullBytes = Arg . getDereferenceableOrNullBytes ( ) ; if ( DerefOrNullBytes !=


<s> FunctionPass * llvm :: create<ISA_LIT>LowerKernelArgumentsPass ( )


<s> FunctionPass * llvm :: create<ISA_LIT>LowerKernelArgumentsPass ( ) { return new <ISA_LIT>LowerKernelArguments ( )


<s> AU . addRequired < TargetPassConfig > ( ) ; AU . setPreservesAll


<s> Value * ArgPtr ; if ( DoShiftOpt ) { ArgPtr = Builder . CreateConstInBoundsGEP1_64 ( KernArgSegment , AlignDownOffset , Arg . getName ( ) + <STR_LIT> ) ; ArgPtr = Builder . CreateBitCast ( ArgPtr , Builder . getInt32Ty ( ) -> getPointerTo ( AS ) , ArgPtr -> getName ( ) + <STR_LIT> ) ; } else { ArgPtr = Builder . CreateConstInBoundsGEP1_64 ( KernArgSegment , EltOffset , Arg . getName ( ) + <STR_LIT> ) ; ArgPtr = Builder . CreateBitCast ( ArgPtr , ArgTy -> getPointerTo ( AS ) , ArgPtr -> getName ( ) + <STR_LIT> ) ; } if ( IsV3 && Size >= <NUM_LIT> ) { V4Ty = VectorType :: get ( VT -> getVectorElementType ( ) , <NUM_LIT> ) ; ArgPtr = Builder . CreateBitCast ( ArgPtr , V4Ty -> getPointerTo ( AS ) ) ; } LoadInst * Load = Builder . CreateAlignedLoad ( ArgPtr , AdjustedAlign ) ; Load -> setMetadata ( LLVMContext :: MD_invariant_load , MDNode :: get ( Ctx , { } ) ) ; MDBuilder MDB ( Ctx ) ; if ( isa < PointerType > ( ArgTy ) ) { if ( Arg . hasNonNullAttr ( ) ) Load -> setMetadata ( LLVMContext :: MD_nonnull , MDNode :: get ( Ctx , { } ) ) ; uint64_t DerefBytes = Arg . getDereferenceableBytes ( ) ; if ( DerefBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefBytes ) ) ) ) ; } uint64_t DerefOrNullBytes = Arg . getDereferenceableOrNullBytes ( ) ; if ( DerefOrNullBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable_or_null , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefOrNullBytes ) ) ) ) ; } unsigned ParamAlign = Arg . getParamAlignment ( ) ; if ( ParamAlign != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_align , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , ParamAlign ) ) ) ) ; } } if ( DoShiftOpt ) { Value * ExtractBits = OffsetDiff == <NUM_LIT> ? Load : Builder . CreateLShr ( Load , OffsetDiff * <NUM_LIT> ) ; IntegerType * ArgIntTy = Builder . getIntNTy ( Size ) ; Value * Trunc = Builder . CreateTrunc ( ExtractBits , ArgIntTy ) ; Value * NewVal = Builder . CreateBitCast ( Trunc , ArgTy , Arg . getName ( ) + <STR_LIT> ) ; Arg . replaceAllUsesWith ( NewVal ) ; } else if ( IsV3 ) { Value * Shuf = Builder . CreateShuffleVector ( Load , UndefValue :: get ( V4Ty ) , { <NUM_LIT> , <NUM_LIT> , <NUM_LIT> } , Arg . getName ( ) + <STR_LIT> ) ; Arg . replaceAllUsesWith (


<s> Function * DispatchPtr = M . getFunction ( DispatchPtrName ) ; if ( ! DispatchPtr ) return false ; bool MadeChange = false ; SmallPtrSet < Instruction * , <NUM_LIT> > HandledUses ; for ( auto * U : DispatchPtr -> users ( ) ) { CallInst * CI = cast < CallInst > (


<s> CallInst * CI = cast < CallInst > ( U ) ; if ( HandledUses . insert ( CI ) . second ) { if ( processUse ( CI


<s> const bool HasReqdWorkGroupSize = MD && MD -> getNumOperands ( ) == <NUM_LIT> ; const bool HasUniformWorkGroupSize = F -> getFnAttribute ( <STR_LIT> ) . getValueAsBool ( ) ; if ( ! HasReqdWorkGroupSize && ! HasUniformWorkGroupSize ) return false ; Value * WorkGroupSizeX = nullptr ; Value * WorkGroupSizeY = nullptr ; Value * WorkGroupSizeZ = nullptr ; Value * GridSizeX = nullptr ; Value * GridSizeY = nullptr ; Value * GridSizeZ = nullptr ; const DataLayout & DL = F -> getParent ( ) -> getDataLayout ( ) ; for ( User * U : CI -> users ( ) ) { if ( ! U -> hasOneUse ( ) ) continue ; int64_t Offset = <NUM_LIT> ; if ( GetPointerBaseWithConstantOffset ( U , Offset , DL ) != CI ) continue ; auto * BCI = dyn_cast < BitCastInst > ( * U -> user_begin ( ) ) ; if ( ! BCI || ! BCI -> hasOneUse ( ) ) continue ; auto * Load = dyn_cast < LoadInst > ( * BCI -> user_begin ( ) ) ; if ( ! Load || ! Load -> isSimple ( ) ) continue ; unsigned LoadSize = DL . getTypeStoreSize ( Load -> getType ( ) ) ; switch ( Offset ) { case WORKGROUP_<ISA_LIT>ZE_X : if ( LoadSize == <NUM_LIT> ) WorkGroupSizeX = Load ; break ; case WORKGROUP_<ISA_LIT>ZE_Y : if ( LoadSize == <NUM_LIT> ) WorkGroupSizeY = Load ; break ; case WORKGROUP_<ISA_LIT>ZE_Z : if ( LoadSize == <NUM_LIT> ) WorkGroupSizeZ = Load ; break ; case GRID_<ISA_LIT>ZE_X : if ( LoadSize == <NUM_LIT> ) GridSizeX = Load ; break ; case GRID_<ISA_LIT>ZE_Y : if ( LoadSize == <NUM_LIT> )


<s> if ( LoadSize == <NUM_LIT> ) GridSizeX = Load ; break ; case GRID_<ISA_LIT>ZE_Y : if ( LoadSize == <NUM_LIT> ) GridSizeY = Load ; break ; case GRID_<ISA_LIT>ZE_Z : if ( LoadSize == <NUM_LIT> ) GridSizeZ = Load ; break ; default : break ; } } bool MadeChange = false ; Value * WorkGroupSizes [ <NUM_LIT> ] = { WorkGroupSizeX , WorkGroupSizeY , WorkGroupSizeZ } ; Value * GridSizes [ <NUM_LIT> ] = { GridSizeX , GridSizeY , GridSizeZ } ; for ( int I = <NUM_LIT> ; HasUniformWorkGroupSize && I < <NUM_LIT> ; ++ I ) { Value * GroupSize = WorkGroupSizes [ I ] ; Value * GridSize = GridSizes [ I ] ; if ( ! GroupSize || ! GridSize ) continue ; for ( User * U : GroupSize -> users ( ) ) { auto * ZextGroupSize = dyn_cast < ZExtInst > ( U ) ; if ( ! ZextGroupSize ) continue ; for ( User * ZextUser : ZextGroupSize -> users ( ) ) { auto * <ISA_LIT> = dyn_cast < SelectInst > ( ZextUser ) ; if ( ! <ISA_LIT> ) continue ; using namespace llvm :: PatternMatch ; auto GroupIDIntrin = I == <NUM_LIT> ? m_Intrinsic < <STR_LIT> > ( ) : ( I == <NUM_LIT> ? m_Intrinsic < <STR_LIT> > ( ) : m_Intrinsic < <STR_LIT> > ( ) ) ; auto SubExpr = m_Sub ( m_Specific ( GridSize ) , m_Mul ( GroupIDIntrin , m_Specific ( ZextGroupSize ) ) ) ; ICmpInst :: Predicate Pred ; if ( match ( <ISA_LIT> , m_Select ( m_ICmp ( Pred , SubExpr , m_Specific ( ZextGroupSize ) ) , SubExpr , m_Specific ( ZextGroupSize ) ) ) && Pred == ICmpInst :: ICMP_ULT ) { if ( HasReqdWorkGroupSize ) { ConstantInt * KnownSize = mdconst :: extract < ConstantInt > ( MD -> getOperand ( I ) ) ; <ISA_LIT> -> replaceAllUsesWith ( ConstantExpr :: getIntegerCast ( KnownSize , <ISA_LIT> -> getType ( ) , false ) ) ; } else { <ISA_LIT> -> replaceAllUsesWith (


<s> bool <ISA_LIT>LowerKernelAttributes :: doInitialization (


<s> StringRef getPassName ( ) const override


<s> if ( ! U -> hasOneUse ( ) ) continue ; int64_t Offset = <NUM_LIT> ; if ( GetPointerBaseWithConstantOffset ( U , Offset , DL ) != CI ) continue ; auto * BCI = dyn_cast < BitCastInst > ( * U -> user_begin ( ) ) ; if ( ! BCI || ! BCI -> hasOneUse ( ) ) continue ; auto * Load = dyn_cast < LoadInst > ( * BCI -> user_begin ( ) ) ; if ( ! Load || ! Load -> isSimple ( ) ) continue ; unsigned LoadSize = DL . getTypeStoreSize ( Load -> getType ( ) ) ; switch ( Offset ) { case WORKGROUP_<ISA_LIT>ZE_X : if ( LoadSize == <NUM_LIT> ) WorkGroupSizeX = Load ; break ; case WORKGROUP_<ISA_LIT>ZE_Y : if ( LoadSize == <NUM_LIT> ) WorkGroupSizeY = Load ; break ; case WORKGROUP_<ISA_LIT>ZE_Z : if ( LoadSize == <NUM_LIT> ) WorkGroupSizeZ = Load ; break ; case GRID_<ISA_LIT>ZE_X : if ( LoadSize == <NUM_LIT> ) GridSizeX = Load ; break ; case GRID_<ISA_LIT>ZE_Y : if ( LoadSize == <NUM_LIT> ) GridSizeY = Load ; break ; case GRID_<ISA_LIT>ZE_Z : if ( LoadSize == <NUM_LIT> ) GridSizeZ = Load ; break ; default : break ; } } bool MadeChange = false ; Value * WorkGroupSizes [ <NUM_LIT> ] = { WorkGroupSizeX , WorkGroupSizeY , WorkGroupSizeZ } ; Value * GridSizes [ <NUM_LIT> ] = { GridSizeX , GridSizeY , GridSizeZ } ; for ( int I = <NUM_LIT> ; HasUniformWorkGroupSize && I < <NUM_LIT> ; ++ I ) { Value * GroupSize = WorkGroupSizes [ I ] ; Value * GridSize = GridSizes [ I ] ; if ( ! GroupSize || ! GridSize ) continue ; for ( User * U : GroupSize -> users ( ) ) { auto * ZextGroupSize = dyn_cast < ZExtInst > ( U ) ; if ( ! ZextGroupSize ) continue ; for ( User * ZextUser : ZextGroupSize -> users ( ) ) { auto * <ISA_LIT> = dyn_cast < SelectInst > ( ZextUser ) ; if ( ! <ISA_LIT> )


<s> bool <ISA_LIT>LowerKernelAttributes :: processUse ( CallInst * CI ) { Function * F = CI -> getParent ( ) -> getParent ( ) ; auto MD = F -> getMetadata ( <STR_LIT> ) ; const bool HasReqdWorkGroupSize = MD && MD -> getNumOperands ( ) == <NUM_LIT> ; const bool HasUniformWorkGroupSize = F -> getFnAttribute ( <STR_LIT> ) . getValueAsString ( ) == <STR_LIT> ; if ( ! HasReqdWorkGroupSize && ! HasUniformWorkGroupSize ) return false ; Value * WorkGroupSizeX = nullptr ; Value * WorkGroupSizeY = nullptr ; Value * WorkGroupSizeZ = nullptr ; Value * GridSizeX = nullptr ; Value * GridSizeY = nullptr ; Value * GridSizeZ = nullptr ; const DataLayout & DL = Mod -> getDataLayout ( ) ; for ( User * U : CI -> users ( ) ) { if ( ! U -> hasOneUse ( ) ) continue ; int64_t Offset = <NUM_LIT> ; if ( GetPointerBaseWithConstantOffset ( U , Offset , DL ) != CI ) continue ; auto * BCI = dyn_cast < BitCastInst > ( * U -> user_begin ( ) ) ; if ( ! BCI || ! BCI -> hasOneUse ( ) ) continue ; auto * Load = dyn_cast < LoadInst > ( * BCI -> user_begin ( ) ) ; if ( ! Load || ! Load -> isSimple ( ) ) continue ; unsigned LoadSize = DL . getTypeStoreSize ( Load -> getType ( ) ) ; switch ( Offset ) { case WORKGROUP_<ISA_LIT>ZE_X : if ( LoadSize == <NUM_LIT> ) WorkGroupSizeX = Load ; break ; case WORKGROUP_<ISA_LIT>ZE_Y : if ( LoadSize == <NUM_LIT> ) WorkGroupSizeY = Load ; break ; case WORKGROUP_<ISA_LIT>ZE_Z : if ( LoadSize == <NUM_LIT> ) WorkGroupSizeZ = Load ; break ; case GRID_<ISA_LIT>ZE_X : if ( LoadSize == <NUM_LIT> ) GridSizeX = Load ; break ; case


<s> ValueToValueMapTy ignored ; Function * NewF = F . empty ( ) ? Function :: Create ( F . getFunctionType ( ) , Function :: ExternalLinkage , <STR_LIT> , F . getParent ( ) ) : CloneFunction ( & F , ignored ) ; NewF -> setCallingConv ( CallingConv :: C ) ; if ( ! NewF -> isDeclaration ( ) ) { NewF -> setVisibility ( GlobalValue :: DefaultVisibility ) ; NewF -> setLinkage ( GlobalValue ::


<s> if ( ! NewF -> isDeclaration ( ) ) { NewF -> setVisibility ( GlobalValue :: DefaultVisibility ) ; NewF -> setLinkage ( GlobalValue :: InternalLinkage ) ; } setNameForBody ( NewF


<s> ModulePass * create<ISA_LIT>LowerKernelCallsPass ( ) { return new <ISA_LIT>LowerKernelCalls (


<s> StringRef getPassName ( ) const override { return <STR_LIT>


<s> CallInst * CI = dyn_cast < CallInst > ( * UI ++ ) ; if ( ! CI ) continue ; if ( ! FBody ) FBody = cloneKernel ( F ) ; CI -> setCalledFunction (


<s> UI != UE ; ) { CallInst * CI = dyn_cast < CallInst > ( * UI ++ ) ; if ( ! CI ) continue ; if ( ! FBody ) FBody = cloneKernel ( F ) ; CI -> setCalledFunction (


<s> SmallString < <NUM_LIT> > NewName ( <STR_LIT> ) ; NewName += Name ; NewName += <STR_LIT> ; FBody -> setName ( NewName . str


<s> for ( Function & F : M . functions ( ) ) { if ( F . isDeclaration ( ) ) continue ; if ( ! <STR_LIT> ( F . getCallingConv ( ) ) ) continue ; Changed |= processUsedLDS ( M , & F ) ; } UsedList . clear (


<s> for ( size_t I = <NUM_LIT> ; I < LocalVars . size ( ) ; I ++ ) { GlobalVariable * GV = LocalVars [ I ] ; Constant * GEPIdx [ ] = { ConstantInt :: get ( I32 , <NUM_LIT> ) , ConstantInt :: get ( I32 , I ) } ; Constant * GEP = ConstantExpr :: getGetElementPtr ( LDSTy , SGV , GEPIdx ) ; if ( F ) { for ( User * U : make_early_inc_range ( GV -> users ( ) ) ) { if ( ConstantExpr * C = dyn_cast < ConstantExpr > ( U ) ) <STR_LIT> ( C , F ) ; } GV -> removeDeadConstantUsers ( ) ; GV -> replaceUsesWithIf ( GEP , [ F ] ( Use & U ) { Instruction * I = dyn_cast < Instruction > ( U . getUser ( ) ) ; return I && I -> getFunction ( ) == F ; } ) ; } else { GV -> replaceAllUsesWith ( GEP ) ; } if ( GV -> use_empty ( ) ) { UsedList . erase ( GV ) ; GV -> eraseFromParent ( ) ; } uint64_t Off = DL . getStructLayout ( LDSTy ) -> getElementOffset ( I ) ; Align A = commonAlignment ( StructAlign , Off ) ; if ( I ) NoAliasList [ I - <NUM_LIT> ] = AliasScopes [ I - <NUM_LIT> ] ; MDNode * NoAlias = NoAliasList . empty ( ) ? nullptr : MDNode :: get ( Ctx , NoAliasList ) ; MDNode * AliasScope = AliasScopes . empty ( ) ? nullptr : MDNode :: get ( Ctx , { AliasScopes [ I ] } ) ; refineUsesAlignmentAndAA ( GEP , A , DL , AliasScope , NoAlias ) ; } if ( ! F ) { IRBuilder < > Builder ( Ctx ) ; for ( Function & Func : M .


<s> bool runOnModule ( Module & M ) override { CallGraph CG = CallGraph ( M ) ; UsedList = getUsedList ( M ) ; bool Changed = superAlignLDSGlobals ( M ) ; Changed |= processUsedLDS ( CG , M ) ; for ( Function & F : M . functions ( ) ) { if ( F . isDeclaration ( ) ) continue ; if ( ! <STR_LIT> ( F . getCallingConv ( ) ) ) continue ; Changed |= processUsedLDS ( CG , M , & F ) ; } UsedList . clear ( ) ; return


<s> for ( size_t I = <NUM_LIT> ; I < FoundLocalVars . size ( ) ; I ++ ) { GlobalVariable * FGV = FoundLocalVars [ I ] ; Align DataAlign = <STR_LIT> ( DL , FGV ) ; uint64_t DataAlignV = DataAlign . value ( ) ; if ( uint64_t Rem = CurrentOffset % DataAlignV ) { uint64_t Padding = DataAlignV - Rem ; Type * ATy = ArrayType :: get ( Type :: getInt8Ty ( Ctx ) , Padding ) ; LocalVars . push_back ( new GlobalVariable ( M , ATy , false , GlobalValue :: InternalLinkage , UndefValue :: get ( ATy ) , <STR_LIT> , nullptr , GlobalValue :: NotThreadLocal , <ISA_LIT>AS :: LOCAL_ADDRESS , false ) ) ; CurrentOffset += Padding ; } LocalVars . push_back ( FGV ) ; CurrentOffset += DL . getTypeAllocSize ( FGV -> getValueType ( ) ) ; } } std :: vector < Type * > LocalVarTypes ; LocalVarTypes . reserve ( LocalVars . size ( ) ) ; std :: transform ( LocalVars . cbegin ( ) , LocalVars . cend ( ) , std :: back_inserter ( LocalVarTypes ) , [ ] ( const GlobalVariable * V ) -> Type * { return V -> getValueType ( ) ; } ) ; std :: string VarName ( F ? ( Twine ( <STR_LIT> ) + F -> getName ( ) + <STR_LIT> ) . str ( ) : <STR_LIT> ) ; StructType * LDSTy = StructType :: create ( Ctx , LocalVarTypes , VarName + <STR_LIT> ) ; Align MaxAlign = <STR_LIT> ( DL , LocalVars [ <NUM_LIT> ] ) ; GlobalVariable * SGV = new GlobalVariable ( M , LDSTy , false , GlobalValue :: InternalLinkage , UndefValue :: get ( LDSTy ) , VarName , nullptr , GlobalValue :: NotThreadLocal , <ISA_LIT>AS :: LOCAL_ADDRESS , false ) ; SGV -> setAlignment ( MaxAlign ) ; if ( ! F ) { appendToCompilerUsed ( M , { static_cast < GlobalValue * > ( ConstantExpr :: getPointerBitCastOrAddrSpaceCast ( cast < Constant > ( SGV ) , Type :: getInt8PtrTy ( Ctx ) ) ) } ) ; } removeFromUsedLists ( M , LocalVars ) ; Type * I32 = Type :: getInt32Ty ( Ctx ) ; for ( size_t I = <NUM_LIT> ; I < LocalVars . size ( ) ; I ++ ) { GlobalVariable * GV = LocalVars [ I ] ; Constant * GEPIdx [ ] = { ConstantInt :: get ( I32 , <NUM_LIT> ) , ConstantInt :: get ( I32 , I ) } ; Constant * GEP = ConstantExpr :: getGetElementPtr ( LDSTy , SGV , GEPIdx ) ; if ( F ) { GV -> replaceUsesWithIf ( GEP , [ F ] ( Use & U ) { return <STR_LIT> ( U . getUser ( ) , F ) ; } ) ; } else { GV -> replaceAllUsesWith ( GEP


<s> UsedList = <STR_LIT> ( M ) ; bool Changed = processUsedLDS ( M ) ; for ( Function & F : M . functions ( ) ) { if ( ! <STR_LIT> ( & F ) ) continue ; Changed |= processUsedLDS ( M , & F


<s> LocalVars . push_back ( FGV ) ; CurrentOffset += LayoutFields [ I ] . Size ; } } std :: vector < Type * > LocalVarTypes ; LocalVarTypes . reserve ( LocalVars . size ( ) ) ; std :: transform ( LocalVars . cbegin ( ) , LocalVars . cend ( ) , std :: back_inserter ( LocalVarTypes ) , [ ] ( const GlobalVariable * V ) -> Type * { return V -> getValueType ( ) ; } ) ; std :: string VarName ( F ? ( Twine ( <STR_LIT> ) + F -> getName ( ) + <STR_LIT> ) . str ( ) : <STR_LIT> ) ; StructType * LDSTy = StructType :: create ( Ctx , LocalVarTypes , VarName + <STR_LIT> ) ; Align StructAlign = <STR_LIT> ( DL , LocalVars [ <NUM_LIT> ] ) ; GlobalVariable * SGV = new GlobalVariable ( M , LDSTy , false , GlobalValue :: InternalLinkage , UndefValue :: get ( LDSTy ) , VarName , nullptr , GlobalValue :: NotThreadLocal , <ISA_LIT>AS :: LOCAL_ADDRESS , false ) ; SGV -> setAlignment ( StructAlign ) ; if ( ! F ) { appendToCompilerUsed ( M , { static_cast < GlobalValue * > ( ConstantExpr :: getPointerBitCastOrAddrSpaceCast ( cast < Constant > ( SGV ) , Type :: getInt8PtrTy ( Ctx ) ) ) } ) ; } removeFromUsedLists ( M , LocalVars ) ; SmallVector < MDNode * > AliasScopes ; SmallVector < Metadata * > NoAliasList ; if ( LocalVars . size ( ) > <NUM_LIT> ) { MDBuilder MDB ( Ctx ) ; AliasScopes . reserve ( LocalVars . size ( ) ) ; MDNode * Domain = MDB . createAnonymousAliasScopeDomain ( ) ; for ( size_t I = <NUM_LIT> ; I < LocalVars . size ( ) ; I ++ ) { MDNode * Scope = MDB . createAnonymousAliasScope ( Domain ) ; AliasScopes . push_back ( Scope ) ; } NoAliasList . append ( & AliasScopes [ <NUM_LIT> ] , AliasScopes . end ( ) ) ; } Type * I32 = Type :: getInt32Ty ( Ctx ) ; for ( size_t I = <NUM_LIT> ; I < LocalVars . size ( ) ; I ++ ) { GlobalVariable * GV = LocalVars [ I


<s> bool Changed = processUsedLDS ( M ) ; for ( Function & F : M . functions ( ) ) { if ( ! <STR_LIT> ( F . getCallingConv


<s> std :: vector < Type * > LocalVarTypes ; LocalVarTypes . reserve ( LocalVars . size ( ) ) ; std :: transform ( LocalVars . cbegin ( ) , LocalVars . cend ( ) , std :: back_inserter ( LocalVarTypes ) , [ ] ( const GlobalVariable * V ) -> Type * { return V -> getValueType ( ) ; } ) ; StructType * LDSTy = StructType :: create ( Ctx , LocalVarTypes , llvm :: StringRef ( <STR_LIT> ) ) ; Align MaxAlign = <STR_LIT> ( DL , LocalVars [ <NUM_LIT> ] ) ; Constant * InstanceAddress = Constant :: getIntegerValue ( PointerType :: get ( LDSTy , <ISA_LIT>AS :: LOCAL_ADDRESS ) , APInt ( <NUM_LIT> , <NUM_LIT> ) ) ; GlobalVariable * SGV = new GlobalVariable ( M , LDSTy , false , GlobalValue :: InternalLinkage , UndefValue :: get ( LDSTy ) , <STR_LIT> , nullptr , GlobalValue :: NotThreadLocal , <ISA_LIT>AS :: LOCAL_ADDRESS , false ) ; SGV -> setAlignment ( MaxAlign ) ; appendToCompilerUsed ( M , { static_cast < GlobalValue * > ( ConstantExpr :: getPointerBitCastOrAddrSpaceCast ( cast < Constant > ( SGV ) , Type :: getInt8PtrTy ( Ctx ) ) ) } ) ; removeFromUsedLists ( M , LocalVars ) ; Type * I32 = Type :: getInt32Ty ( Ctx ) ; for ( size_t I = <NUM_LIT> ; I < LocalVars . size ( ) ; I ++ ) { GlobalVariable * GV = LocalVars [ I ] ; Constant * GEPIdx [ ] = { ConstantInt :: get ( I32 , <NUM_LIT> ) , ConstantInt :: get ( I32 , I ) } ; GV -> replaceAllUsesWith ( ConstantExpr :: getGetElementPtr ( LDSTy , InstanceAddress , GEPIdx ) ) ; GV -> eraseFromParent ( ) ; } { IRBuilder < > Builder ( Ctx ) ; SmallPtrSet < Function * , <NUM_LIT> > Kernels ; for ( auto & I : M . functions (


<s> static void removeFromUsedLists ( Module & M , const std :: vector < GlobalVariable * > & LocalVars ) { SmallPtrSet < Constant * , <NUM_LIT> > LocalVarsSet ; for ( GlobalVariable * LocalVar : LocalVars ) if ( Constant * C = dyn_cast < Constant > ( LocalVar -> stripPointerCasts ( ) ) ) LocalVarsSet . insert ( C ) ; removeFromUsedList ( M , <STR_LIT> , LocalVarsSet ) ; removeFromUsedList ( M , <STR_LIT> ,


<s> static void removeFromUsedLists ( Module & M , const std :: vector < GlobalVariable * > & LocalVars ) { SmallPtrSet < Constant * , <NUM_LIT> > LocalVarsSet ; for ( GlobalVariable * LocalVar : LocalVars ) if ( Constant * C = dyn_cast < Constant > ( LocalVar -> stripPointerCasts ( ) ) ) LocalVarsSet . insert ( C ) ; removeFromUsedList ( M , <STR_LIT> , LocalVarsSet ) ; removeFromUsedList ( M , <STR_LIT>


<s> PreservedAnalyses <ISA_LIT>LowerModuleLDSPass :: run ( Module & M , ModuleAnalysisManager & ) { return <ISA_LIT>LowerModuleLDS ( ) . runOnModule ( M ) ? PreservedAnalyses :: none


<s> } if ( ! GV . hasInitializer ( ) ) { continue ; } Align Alignment = <STR_LIT> ( DL , & GV ) ; TypeSize GVSize = DL . getTypeAllocSize ( GV . getValueType ( ) ) ; if ( GVSize > <NUM_LIT> ) { Alignment = std :: max ( Alignment , Align ( <NUM_LIT> ) ) ; } else if ( GVSize > <NUM_LIT> ) { Alignment = std :: max ( Alignment , Align ( <NUM_LIT> ) ) ; } else if ( GVSize > <NUM_LIT> ) { Alignment = std :: max ( Alignment , Align ( <NUM_LIT> ) ) ; } else if ( GVSize > <NUM_LIT> ) { Alignment = std :: max ( Alignment , Align (


<s> static bool isPHIRegionIndex ( SmallVector < unsigned , <NUM_LIT>


<s> createEntryPHIs ( CurrentRegion ) ; removeExternalCFGSuccessors ( CodeBB ) ; CodeBB -> addSuccessor ( MergeBB ) ; CurrentRegion -> addMBB ( CodeBB ) ; return nullptr ; } else { const TargetRegisterClass * RegClass = MRI -> getRegClass ( BBSelectRegIn ) ; Register CodeBBSelectReg = MRI -> createVirtualRegister ( RegClass ) ; rewriteCodeBBTerminator ( CodeBB , MergeBB , CodeBBSelectReg ) ; bool IsRegionEntryBB = CurrentRegion -> getEntry ( ) == CodeBB ; MachineBasicBlock * IfBB = createIfBlock ( MergeBB , CodeBB , CodeBB , CodeBB , BBSelectRegIn , IsRegionEntryBB ) ; CurrentRegion -> addMBB ( IfBB ) ; if ( IsRegionEntryBB ) { CurrentRegion -> setEntry ( IfBB ) ; if ( CurrentRegion -> getHasLoop ( ) ) { MachineBasicBlock * RegionExit = CurrentRegion -> getExit ( ) ; MachineBasicBlock * ETrueBB = nullptr ; MachineBasicBlock * EFalseBB = nullptr ; SmallVector < MachineOperand , <NUM_LIT> > ECond ; const DebugLoc & DL = DebugLoc ( ) ; TII -> analyzeBranch ( * RegionExit , ETrueBB , EFalseBB , ECond ) ; TII -> removeBranch (


<s> LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << printReg ( Reg , MRI -> getTargetRegisterInfo ( ) ) << <STR_LIT> ) ; } assert ( MRI -> hasOneDef ( Reg ) && <STR_LIT> ) ; MachineOperand * Def = & ( * ( MRI -> def_begin ( Reg ) ) ) ; MachineOperand * UseOperand = & ( RI ) ; bool UseIsOutsideDefMBB = Def -> getParent ( ) -> getParent ( ) != MBB ; if ( UseIsOutsideDefMBB && UseOperand -> isKill ( ) ) { LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << printReg ( Reg , TRI ) << <STR_LIT> ) ; UseOperand -> setIsKill ( false


<s> PHIInfoElementT * NewElement = new PHIInfoElementT ( ) ; NewElement -> DestReg = DestReg ; NewElement -> DL = DL ; NewElement -> Sources =


<s> assert ( findPHIInfoElement ( DestReg ) == nullptr && <STR_LIT> ) ; PHISourcesT EmptySet ; PHIInfoElementT * NewElement = new PHIInfoElementT ( ) ; NewElement -> DestReg = DestReg ; NewElement -> DL = DL ; NewElement -> Sources = EmptySet ; PHIInfo . insert ( NewElement )


<s> if ( Instr . isPHI ( ) ) { unsigned PHIDestReg = getPHIDestReg ( Instr ) ; LLVM_DEBUG ( dbgs ( ) << <STR_LIT> ) ; LLVM_DEBUG ( Instr . dump ( ) ) ; PHIs . insert ( & Instr


<s> MachineBasicBlock * result = nullptr ; for ( auto & MFI : MF ) { if ( MFI . succ_empty ( ) ) { if ( result == nullptr ) { result = & MFI ; } else { return nullptr ; } } } return


<s> MachineBasicBlock * MRT :: initializeMRT ( MachineFunction & MF , const MachineRegionInfo * RegionInfo , DenseMap < MachineRegion * , RegionMRT * > & RegionMap ) { for ( auto & MFI : MF ) { MachineBasicBlock * ExitMBB = & MFI ; if ( ExitMBB -> succ_empty ( ) ) { return


<s> MachineBasicBlock * MRT :: initializeMRT ( MachineFunction & MF , const MachineRegionInfo * RegionInfo , DenseMap < MachineRegion * , RegionMRT * > & RegionMap ) { for ( auto & MFI : MF ) { MachineBasicBlock * ExitMBB = & MFI ; if ( ExitMBB -> succ_empty ( ) ) { return ExitMBB ; } } llvm_unreachable ( <STR_LIT> ) ; return nullptr


<s> if ( ReplaceOutside && ( isLiveOut ( Register ) || this -> getParent ( ) -> isLiveOut ( Register ) ) ) { LinearizedRegion * Current = this ; while ( Current != nullptr && Current -> getEntry ( ) != nullptr ) { LLVM_DEBUG ( dbgs ( ) << <STR_LIT> ) ; LLVM_DEBUG ( Current -> print ( dbgs ( ) , MRI -> getTargetRegisterInfo ( ) ) ) ; Current -> replaceLiveOut ( Register , NewRegister ) ; LLVM_DEBUG ( dbgs ( ) << <STR_LIT> ) ; LLVM_DEBUG ( Current -> print ( dbgs ( ) , MRI -> getTargetRegisterInfo ( ) ) ) ; Current = Current -> getParent ( ) ; } } for ( MachineRegisterInfo :: reg_iterator I = MRI -> reg_begin ( Register ) , E = MRI -> reg_end ( ) ; I != E ; ) { MachineOperand & O = * I ; ++


<s> RegionMRT * MRT :: buildMRT ( MachineFunction & MF , const MachineRegionInfo * RegionInfo , const <ISA_LIT>InstrInfo * TII , MachineRegisterInfo * MRI ) { SmallPtrSet < MachineRegion * , <NUM_LIT> > PlacedRegions ; DenseMap < MachineRegion * , RegionMRT * > RegionMap ; MachineRegion * TopLevelRegion = RegionInfo -> getTopLevelRegion ( ) ; RegionMRT * Result = new RegionMRT ( TopLevelRegion ) ; RegionMap [ TopLevelRegion ] = Result ; MachineBasicBlock * Exit = initializeMRT ( MF , RegionInfo , RegionMap ) ; unsigned BBSelectRegIn = createBBSelectReg ( TII , MRI ) ; MBBMRT * ExitMRT = new MBBMRT ( Exit ) ; RegionMap [ RegionInfo -> getRegionFor ( Exit ) ] -> addChild ( ExitMRT ) ; ExitMRT -> setBBSelectRegIn ( BBSelectRegIn ) ; for ( auto MBBI : post_order ( & ( MF . front ( ) ) ) ) { MachineBasicBlock * MBB = & ( * MBBI ) ; if ( MBB == Exit ) { continue ; } DEBUG ( dbgs ( ) << <STR_LIT> << MBB -> getNumber ( ) << <STR_LIT> ) ; MBBMRT * NewMBB = new MBBMRT ( MBB ) ; MachineRegion * Region = RegionInfo -> getRegionFor ( MBB ) ; if ( RegionMap . count ( Region ) ==


<s> } else { MachineInstr * PHIDefInstr = getDefInstr ( SourceReg ) ; MachineBasicBlock * PHIDefMBB = PHIDefInstr -> getParent ( ) ; const TargetRegisterClass * RegClass = MRI -> getRegClass ( CurrentBackedgeReg ) ; unsigned NewBackedgeReg = MRI -> createVirtualRegister ( RegClass ) ; MachineInstrBuilder BackedgePHI = BuildMI ( * PHIDefMBB , PHIDefMBB -> instr_begin ( ) , DL , TII -> get ( TargetOpcode :: PHI ) , NewBackedgeReg ) ; BackedgePHI . addReg ( CurrentBackedgeReg ) ; BackedgePHI . addMBB ( getPHIPred ( * PHIDefInstr , <NUM_LIT> ) ) ; BackedgePHI . addReg ( getPHISourceReg ( * PHIDefInstr , <NUM_LIT> ) ) ; BackedgePHI . addMBB ( ( * SRI ) . second ) ; CurrentBackedgeReg = NewBackedgeReg ; DEBUG ( dbgs ( ) << <STR_LIT> << PrintReg ( NewBackedgeReg , TRI ) << <STR_LIT> << PrintReg ( CurrentBackedgeReg , TRI ) << <STR_LIT> << getPHIPred ( * PHIDefInstr , <NUM_LIT> ) -> getNumber ( ) << <STR_LIT> << PrintReg ( getPHISourceReg ( * PHIDefInstr , <NUM_LIT> ) , TRI ) << <STR_LIT> << ( * SRI ) . second -> getNumber ( ) ) ; } } else


<s> for ( MachineBasicBlock :: pred_iterator PI = CodeBBStart -> pred_begin ( ) , E = CodeBBStart -> pred_end ( ) ; PI != E ; ++ PI ) { if ( ( * PI ) != CodeBBEnd ) { MachineBasicBlock * Pred = ( * PI ) ; Pred -> addSuccessor ( IfBB ) ; } } } removeExternalCFGEdges ( CodeBBStart , CodeBBEnd ) ; auto CodeBBStartI = CodeBBStart -> getIterator ( ) ; auto CodeBBEndI = CodeBBEnd -> getIterator ( ) ; auto MergeIter = MergeBB -> getIterator ( ) ; MF -> insert ( MergeIter , IfBB ) ; MF -> splice ( MergeIter , CodeBBStartI , ++ CodeBBEndI ) ; IfBB -> addSuccessor ( MergeBB ) ; IfBB -> addSuccessor ( CodeBBStart ) ; DEBUG ( dbgs ( ) << <STR_LIT> << IfBB -> getNumber ( ) << <STR_LIT> ) ; if ( ! CodeBBEnd -> isSuccessor ( MergeBB ) ) CodeBBEnd -> addSuccessor ( MergeBB ) ; DEBUG ( dbgs ( ) << <STR_LIT> << CodeBBStart -> getNumber ( ) << <STR_LIT> << CodeBBEnd -> getNumber ( ) << <STR_LIT> ) ; MachineBasicBlock * SinglePred = CodeBBStart -> pred_size ( ) == <NUM_LIT> ? * ( CodeBBStart -> pred_begin ( ) ) : nullptr ; const DebugLoc & DL = SinglePred ? SinglePred -> findDebugLoc ( SinglePred -> getFirstTerminator ( ) ) : DebugLoc ( ) ; unsigned Reg = TII -> insertEQ ( IfBB , IfBB -> begin ( ) , DL , IfReg , SelectBB -> getNumber ( ) ) ; if ( & ( * ( IfBB -> getParent ( ) -> begin ( ) ) )


<s> } } removeExternalCFGEdges ( CodeBBStart , CodeBBEnd ) ; auto CodeBBStartI = CodeBBStart -> getIterator ( ) ; auto CodeBBEndI = CodeBBEnd -> getIterator ( ) ; auto MergeIter = MergeBB -> getIterator ( ) ; MF -> insert ( MergeIter , IfBB ) ; MF -> splice ( MergeIter , CodeBBStartI , ++ CodeBBEndI ) ; IfBB -> addSuccessor ( MergeBB ) ; IfBB -> addSuccessor ( CodeBBStart ) ; DEBUG ( dbgs ( ) << <STR_LIT> << IfBB -> getNumber ( ) << <STR_LIT> ) ; if ( ! CodeBBEnd -> isSuccessor ( MergeBB ) ) CodeBBEnd -> addSuccessor ( MergeBB ) ; DEBUG ( dbgs ( ) << <STR_LIT> << CodeBBStart -> getNumber ( ) << <STR_LIT> << CodeBBEnd -> getNumber ( ) << <STR_LIT> ) ; MachineBasicBlock * SinglePred = CodeBBStart -> pred_size ( ) == <NUM_LIT> ? * ( CodeBBStart -> pred_begin ( ) ) : nullptr ; const DebugLoc & DL = SinglePred ? SinglePred -> findDebugLoc ( SinglePred -> getFirstTerminator ( ) ) : DebugLoc ( ) ; unsigned Reg = TII -> insertEQ ( IfBB , IfBB -> begin ( ) , DL , IfReg , SelectBB -> getNumber ( ) ) ; if ( & ( * ( IfBB -> getParent ( ) -> begin ( ) ) ) == IfBB ) { TII -> materializeImmediate ( * IfBB , IfBB -> begin ( ) , DL , IfReg , CodeBBStart -> getNumber (


<s> unsigned CodeBBSelectReg = InnerRegion -> getRegionMRT ( ) -> getInnerOutputRegister ( ) ; MachineBasicBlock * CodeEntryBB = InnerRegion -> getEntry ( ) ; MachineBasicBlock * CodeExitBB = InnerRegion -> getExit ( ) ; MachineBasicBlock * IfBB = createIfBlock ( MergeBB , CodeEntryBB , CodeExitBB , SelectBB , BBSelectRegIn , true ) ; CurrentRegion -> addMBB ( IfBB ) ; bool isEntry = CurrentRegion -> getEntry ( ) == InnerRegion -> getEntry ( ) ; if ( isEntry ) { if ( CurrentRegion -> getHasLoop ( ) ) { MachineBasicBlock * RegionExit = CurrentRegion -> getExit ( ) ; MachineBasicBlock * ETrueBB = nullptr ; MachineBasicBlock * EFalseBB = nullptr ; SmallVector < MachineOperand , <NUM_LIT> > ECond ; const DebugLoc & DL = DebugLoc ( ) ; TII -> analyzeBranch ( * RegionExit , ETrueBB , EFalseBB , ECond ) ; TII -> removeBranch ( * RegionExit ) ; unsigned Reg = TII -> insertNE ( RegionExit , RegionExit -> instr_end ( ) , DL , CurrentRegion -> getRegionMRT ( ) -> getInnerOutputRegister ( ) , CurrentRegion -> getRegionMRT

( ) -> getEntry ( ) -> getNumber ( ) ) ;


<s> if ( Exit == nullptr && hasOneExitNode ( * MF ) ) { return & ( * ( -- ( Region -> getEntry ( ) -> getParent ( ) -> end ( ) ) ) ) ; } MachineBasicBlock * LastMerge = MF -> CreateMachineBasicBlock ( ) ; if ( Exit == nullptr ) { MachineFunction :: iterator ExitIter = MF -> end ( ) ; MF -> insert ( ExitIter , LastMerge ) ; } else { MachineFunction :: iterator ExitIter = Exit -> getIterator ( ) ; MF -> insert ( ExitIter , LastMerge ) ; LastMerge -> addSuccessor ( Exit ) ; insertUnconditionalBranch ( LastMerge , Exit ) ; DEBUG ( dbgs ( ) << <STR_LIT> << LastMerge -> getNumber ( ) << <STR_LIT> ) ; } return LastMerge


<s> LRegion -> setRegionMRT ( Region ) ; Region -> setLinearizedRegion ( LRegion ) ; LRegion -> setParent ( Region -> getParent ( ) ? Region -> getParent (

) -> getLinearizedRegion ( ) : nullptr ) ;


<s> dbgs ( ) << <STR_LIT> << PrintReg ( getBBSelectRegIn ( ) , TRI ) ; dbgs ( ) << <STR_LIT> << PrintReg ( getBBSelectRegOut ( ) , TRI ) << <STR_LIT> ; dumpDepth ( depth ) ; if ( getSucc ( ) ) dbgs ( ) << <STR_LIT> << getSucc ( ) -> getNumber ( ) << <STR_LIT> ; else dbgs ( ) << <STR_LIT> ; for ( auto MRTI : Children ) { MRTI -> dump ( TRI ,


<s> if ( MRI -> def_begin ( Reg ) == MRI -> def_end ( ) ) { DEBUG ( dbgs ( ) << <STR_LIT> << PrintReg ( Reg , MRI -> getTargetRegisterInfo ( ) ) << <STR_LIT> ) ; } else if ( ! MRI -> hasOneDef ( Reg ) ) { DEBUG ( dbgs ( ) << <STR_LIT> << PrintReg ( Reg , MRI -> getTargetRegisterInfo ( ) ) << <STR_LIT> ) ; DEBUG ( dbgs ( ) << <STR_LIT> ) ; for ( auto DI = MRI -> def_begin ( Reg ) , DE = MRI -> def_end ( ) ; DI != DE


<s> DEBUG ( dbgs ( ) << <STR_LIT> << MBBI . getNumber ( ) << <STR_LIT> << MBB -> getNumber ( ) << <STR_LIT> ) ; } FallthroughMap [ & MBBI ] =


<s> MachineInstrBuilder MIB = BuildMI ( * MergeBB , MergeBB -> instr_begin ( ) , DL , TII -> get ( TargetOpcode :: PHI ) , DestRegister ) ; if ( IsUndefIfSource && false ) { MIB . addReg ( IfSourceRegister , RegState :: Undef ) ; } else { MIB . addReg ( IfSourceRegister ) ; } MIB . addMBB (


<s> } OS << <STR_LIT> << Entry -> getNumber ( ) << <STR_LIT> << ( Exit == nullptr ? - <NUM_LIT> : Exit -> getNumber ( ) ) << <STR_LIT> << PrintReg ( getBBSelectRegIn ( ) , TRI ) << <STR_LIT> << PrintReg ( getBBSelectRegOut ( ) , TRI ) << <STR_LIT> ; for ( auto & LI : LiveOuts ) { OS << PrintReg ( LI , TRI )


<s> unsigned SuccSize = StartMBB -> succ_size ( ) ; if ( SuccSize > <NUM_LIT> ) { MachineBasicBlock * StartMBBSucc = * ( StartMBB -> succ_begin ( ) ) ; for ( MachineBasicBlock :: succ_iterator PI = EndMBB -> succ_begin ( ) , E = EndMBB -> succ_end ( ) ; PI != E ; ++ PI ) { if ( ( * PI ) != StartMBB && ! ( ( * PI ) == StartMBBSucc && StartMBB != EndMBB && SuccSize == <NUM_LIT> ) ) { Succs . insert ( std :: pair < MachineBasicBlock * , MachineBasicBlock * > ( EndMBB , * PI ) ) ; } } } for ( MachineBasicBlock :: pred_iterator PI = StartMBB -> pred_begin ( ) , E = StartMBB -> pred_end ( ) ; PI != E ; ++ PI


<s> for ( unsigned i = <NUM_LIT> ; i < NumInputs ; ++ i ) { if ( isPHIRegionIndex ( PHIRegionIndices , i ) ) { NumNonRegionInputs -- ; } } if ( NumNonRegionInputs == <NUM_LIT> ) { auto DestReg = getPHIDestReg ( PHI ) ; replaceRegisterWith ( DestReg , CombinedSourceReg ) ; DEBUG ( dbgs ( ) << <STR_LIT> << PrintReg ( CombinedSourceReg , TRI ) << <STR_LIT> ) ; PHI . eraseFromParent ( ) ; } else { DEBUG ( dbgs ( ) << PrintReg ( getPHIDestReg ( PHI ) , TRI ) << <STR_LIT> ) ; MachineBasicBlock * MBB = PHI . getParent ( ) ; MachineInstrBuilder MIB = BuildMI ( * MBB , PHI , PHI . getDebugLoc ( ) , TII -> get ( TargetOpcode :: PHI ) , getPHIDestReg ( PHI ) ) ; MIB . addReg ( CombinedSourceReg ) ; MIB . addMBB ( IfMBB ) ; DEBUG ( dbgs ( ) << PrintReg ( CombinedSourceReg , TRI ) << <STR_LIT> << IfMBB -> getNumber ( ) ) ; unsigned NumInputs = getPHINumInputs ( PHI ) ; for ( unsigned i = <NUM_LIT> ; i < NumInputs


<s> Current -> replaceLiveOut ( Register , NewRegister ) ; DEBUG ( dbgs ( ) << <STR_LIT> ) ; DEBUG ( Current -> print ( dbgs ( ) , MRI -> getTargetRegisterInfo ( ) ) ) ; Current = Current -> getParent ( ) ; } } for ( MachineRegisterInfo :: reg_iterator I = MRI -> reg_begin ( Register ) , E = MRI -> reg_end ( ) ; I != E ; ) { MachineOperand & O = * I ; ++ I ; if ( O . isDef ( ) ) continue ; bool IsInside = contains ( O . getParent ( ) -> getParent ( ) ) ; bool IsLoopPHI = IsInside && ( O . getParent ( ) -> isPHI ( ) && O . getParent ( ) -> getParent ( ) == getEntry ( ) ) ; bool ShouldReplace = ( IsInside && ReplaceInside ) || ( ! IsInside && ReplaceOutside ) || ( IncludeLoopPHI && IsLoopPHI ) ; if ( ShouldReplace ) { if ( TargetRegisterInfo :: isPhysicalRegister ( NewRegister ) ) { DEBUG ( dbgs ( ) << <STR_LIT> << PrintReg ( NewRegister , MRI -> getTargetRegisterInfo ( ) ) << <STR_LIT> ) ; llvm_unreachable ( <STR_LIT> ) ; } else { DEBUG ( dbgs ( ) << <STR_LIT> << PrintReg ( Register , MRI -> getTargetRegisterInfo ( ) ) << <STR_LIT>


<s> ++ I ; if ( TargetRegisterInfo :: isPhysicalRegister ( NewRegister ) ) { DEBUG ( dbgs ( ) << <STR_LIT> << PrintReg ( NewRegister , MRI -> getTargetRegisterInfo ( ) ) << <STR_LIT> ) ; llvm_unreachable ( <STR_LIT> ) ; } else { DEBUG ( dbgs ( ) << <STR_LIT> << PrintReg ( Register , MRI -> getTargetRegisterInfo ( ) ) << <STR_LIT> << PrintReg ( NewRegister , MRI -> getTargetRegisterInfo ( ) ) << <STR_LIT> ) ; O . setReg ( NewRegister ) ; } } PHIInfo . deleteDef ( Register ) ; getRegionMRT ( ) -> replaceLiveOutReg ( Register , NewRegister ) ; DEBUG ( PHIInfo . dump (


<s> auto SRI = PHIInfo . sources_begin ( DestReg ) ; unsigned SourceReg = ( * SRI ) . first ; DEBUG ( dbgs ( ) << <STR_LIT> << PrintReg ( DestReg , TRI ) << <STR_LIT> << PrintReg ( SourceReg , TRI ) << <STR_LIT> ) ; assert ( PHIInfo . sources_end ( DestReg ) == ++ SRI &&


<s> DenseSet < unsigned > * LiveOuts = InnerRegion -> getLiveOuts ( ) ; SmallVector < unsigned , <NUM_LIT> > OldLiveOuts ; bool IsSingleBB = InnerRegion -> getEntry ( ) == InnerRegion -> getExit ( ) ; for ( auto OLI : * LiveOuts ) { OldLiveOuts . push_back ( OLI ) ; } for ( auto LI : OldLiveOuts ) { DEBUG ( dbgs ( ) << <STR_LIT> << PrintReg ( LI , TRI ) ) ; if ( ! containsDef ( CodeBB , InnerRegion , LI ) || ( ! IsSingleBB && ( getDefInstr ( LI ) -> getParent ( ) == LRegion -> getExit ( ) ) ) ) { DEBUG ( dbgs ( ) << <STR_LIT> ) ; continue ; } DEBUG ( dbgs ( ) << <STR_LIT> ) ; unsigned Reg = LI ; if ( Reg != InnerRegion -> getBBSelectRegOut


<s> Regions = & ( getAnalysis < MachineRegionInfoPass > ( ) . getRegionInfo ( ) ) ; DEBUG ( Regions -> dump ( ) ) ; RegionMRT * RTree = MRT :: buildMRT ( MF , Regions , TII , MRI ) ; setRegionMRT ( RTree ) ; initializeSelectRegisters ( RTree , <NUM_LIT> , MRI , TII ) ; DEBUG ( RTree -> dump ( TRI ) ) ; bool result = structurizeRegions ( RTree , true ) ; delete RTree ; DEBUG ( dbgs ( ) << <STR_LIT> ) ; initFallthroughMap (


<s> MachineBasicBlock * EntrySucc = split ( Entry -> getFirstNonPHI ( ) ) ; MachineBasicBlock * Exit = LRegion -> getExit ( ) ; DEBUG ( dbgs ( ) << <STR_LIT> << Entry -> getNumber ( ) << <STR_LIT> << Entry -> getNumber ( ) << <STR_LIT> << EntrySucc -> getNumber ( ) << <STR_LIT> ) ; LRegion -> addMBB ( EntrySucc ) ; if ( Exit -> isSuccessor ( Entry ) ) { Exit -> removeSuccessor ( Entry ) ; } Exit -> addSuccessor ( EntrySucc ) ; MachineInstr & Branch = * ( Exit -> instr_rbegin ( ) ) ; for ( auto & UI : Branch


<s> } const TargetRegisterClass * RegClass = MRI -> getRegClass ( PHIDest ) ; unsigned NewDestReg = MRI -> createVirtualRegister ( RegClass ) ; LRegion -> replaceRegisterInsideRegion ( PHIDest , NewDestReg , false , MRI ) ; MachineInstrBuilder MIB = BuildMI ( * EntrySucc , EntrySucc -> instr_begin ( ) , PHI . getDebugLoc ( ) , TII -> get ( TargetOpcode :: PHI ) , NewDestReg ) ; DEBUG ( dbgs ( ) << <STR_LIT> << PrintReg ( NewDestReg , TRI ) << <STR_LIT> ) ; MIB . addReg ( PHISource ) ; MIB . addMBB ( Entry ) ; DEBUG ( dbgs ( ) << PrintReg ( PHISource , TRI ) << <STR_LIT> << Entry -> getNumber ( ) ) ; MIB . addReg ( RegionSourceReg ) ; MIB . addMBB (


<s> MachineBasicBlock * Exit = Region -> getSucc ( ) ; RegionMRT * TopRegion = CurrentTopRegion == nullptr ? Region : CurrentTopRegion ; if ( Exit == nullptr ) return ; auto Children = Region -> getChildren ( ) ; for ( auto CI : * Children ) { if ( CI -> isMBB ( ) ) { auto MBB = CI -> getMBBMRT ( ) -> getMBB ( ) ; storeMBBLiveOuts ( MBB , MRI , TRI , PHIInfo , TopRegion ) ; } else { LinearizedRegion * SubRegion = CI -> getRegionMRT ( ) -> getLinearizedRegion ( ) ; for ( auto MBBI : SubRegion -> MBBs ) { storeMBBLiveOuts ( MBBI , MRI , TRI , PHIInfo , TopRegion ) ; } } } if ( CurrentTopRegion == nullptr ) { auto Succ = Region -> getSucc ( ) ; for ( auto & II : * Succ ) { if ( II . isPHI ( ) ) { MachineInstr & PHI = II ; int numPreds = getPHINumInputs ( PHI ) ; for ( int i = <NUM_LIT> ;


<s> auto CNI = CI ; ++ CNI ; MRT * Child = ( * CI ) ; if ( Child -> isRegion ( ) ) { LinearizedRegion * InnerLRegion = Child -> getRegionMRT ( ) -> getLinearizedRegion ( ) ; DEBUG ( dbgs ( ) << <STR_LIT> ) ; DEBUG ( InnerLRegion -> print ( dbgs ( ) , TRI ) ) ; DEBUG ( dbgs ( ) << <STR_LIT> ) ; MachineBasicBlock * InnerEntry = InnerLRegion -> getEntry ( ) ; if ( ( & ( * ( InnerEntry -> getParent ( ) -> begin ( ) ) ) ) == InnerEntry ) { unsigned OuterSelect = InnerLRegion -> getBBSelectRegOut ( ) ; unsigned InnerSelectReg = InnerLRegion -> getRegionMRT ( ) -> getInnerOutputRegister ( ) ; replaceRegisterWith ( InnerSelectReg , OuterSelect ) , resolvePHIInfos ( InnerEntry ) ; if ( ! InnerLRegion -> getExit ( ) -> isSuccessor ( CurrentMerge ) ) InnerLRegion -> getExit ( ) -> addSuccessor ( CurrentMerge ) ; continue ; } BBSelectRegOut = Child -> getBBSelectRegOut ( ) ; BBSelectRegIn = Child -> getBBSelectRegIn ( ) ; DEBUG ( dbgs ( ) << <STR_LIT> << PrintReg ( BBSelectRegIn , TRI ) << <STR_LIT> ) ; DEBUG ( dbgs ( ) << <STR_LIT> << PrintReg ( BBSelectRegOut , TRI ) << <STR_LIT> ) ; MachineBasicBlock * IfEnd = CurrentMerge ; CurrentMerge = createIfRegion ( CurrentMerge , InnerLRegion , LRegion , Child -> getRegionMRT ( ) -> getEntry ( ) , BBSelectRegIn , BBSelectRegOut ) ; TII -> convertNonUniformIfRegion ( CurrentMerge , IfEnd ) ; } else { MachineBasicBlock * MBB = Child -> getMBBMRT ( ) -> getMBB ( ) ; DEBUG ( dbgs ( ) << <STR_LIT> << MBB -> getNumber ( ) << <STR_LIT> ) ; if ( MBB == getSingleExitNode ( * ( MBB -> getParent ( ) ) ) ) { continue ; } BBSelectRegOut = Child -> getBBSelectRegOut ( ) ; BBSelectRegIn = Child -> getBBSelectRegIn ( ) ; DEBUG ( dbgs ( ) << <STR_LIT> << PrintReg ( BBSelectRegIn , TRI ) << <STR_LIT> ) ; DEBUG ( dbgs ( ) << <STR_LIT> << PrintReg ( BBSelectRegOut , TRI ) << <STR_LIT> ) ; MachineBasicBlock * IfEnd = CurrentMerge ; CurrentMerge = createIfRegion ( CurrentMerge , MBB , LRegion , BBSelectRegIn , BBSelectRegOut ) ; if ( CurrentMerge ) { TII -> convertNonUniformIfRegion ( CurrentMerge , IfEnd ) ; } DEBUG ( PHIInfo . dump ( MRI ) ) ; } } LRegion -> removeFalseRegisterKills ( MRI ) ; if ( LRegion -> getHasLoop ( ) ) { MachineBasicBlock * NewSucc = splitEntry ( LRegion ) ; if ( isFunctionEntryBlock ( LRegion ->


<s> LLVM_DEBUG ( Regions -> dump ( ) ) ; RegionMRT * RTree = MRT :: buildMRT ( MF , Regions , TII , MRI ) ; setRegionMRT ( RTree ) ; initializeSelectRegisters ( RTree , <NUM_LIT> , MRI , TII ) ; LLVM_DEBUG ( RTree -> dump ( TRI ) ) ; bool result = structurizeRegions ( RTree , true ) ; delete


<s> if ( MRI -> def_begin ( Reg ) == MRI -> def_end ( ) ) { DEBUG ( dbgs ( ) << <STR_LIT> << printReg ( Reg , MRI -> getTargetRegisterInfo ( ) ) << <STR_LIT> ) ; } else if ( ! MRI -> hasOneDef ( Reg ) ) { DEBUG ( dbgs ( ) << <STR_LIT> << printReg ( Reg , MRI -> getTargetRegisterInfo ( ) ) << <STR_LIT> ) ; DEBUG ( dbgs ( ) << <STR_LIT> ) ; for ( auto DI = MRI -> def_begin ( Reg ) , DE = MRI -> def_end ( ) ; DI != DE ; ++ DI ) { DEBUG ( DI -> getParent ( )


<s> PI != E ; ++ PI ) { if ( ( * PI ) != EndMBB ) { Succs . insert ( std :: pair < MachineBasicBlock * , MachineBasicBlock * > ( * PI , StartMBB ) ) ; } } for ( auto <ISA_LIT> : Succs ) { std :: pair < MachineBasicBlock * , MachineBasicBlock * > Edge = <ISA_LIT> ; DEBUG ( dbgs ( ) << <STR_LIT> << printMBBReference ( * Edge . first ) << <STR_LIT> << printMBBReference ( * Edge . second


<s> PI != E ; ++ PI ) { if ( ( * PI ) != EndMBB ) { Succs . insert ( std :: pair < MachineBasicBlock * , MachineBasicBlock * > ( * PI , StartMBB ) ) ; } } for ( auto <ISA_LIT> : Succs ) { std :: pair < MachineBasicBlock * , MachineBasicBlock * > Edge = <ISA_LIT> ; DEBUG ( dbgs ( ) << <STR_LIT> << printMBBReference ( * Edge . first

) << <STR_LIT> << printMBBReference ( * Edge . second ) << <STR_LIT> ) ;


<s> if ( MRI -> def_begin ( Reg ) == MRI -> def_end ( ) ) { DEBUG ( dbgs ( ) << <STR_LIT> << printReg ( Reg , MRI -> getTargetRegisterInfo ( ) ) << <STR_LIT> ) ; } else if ( ! MRI -> hasOneDef ( Reg ) ) { DEBUG ( dbgs ( ) << <STR_LIT> << printReg ( Reg , MRI -> getTargetRegisterInfo ( ) ) << <STR_LIT> ) ; } assert ( MRI -> hasOneDef ( Reg ) && <STR_LIT> ) ; MachineOperand * Def = & ( * ( MRI -> def_begin ( Reg )


<s> DEBUG ( dbgs ( ) << printReg ( Reg , TRI ) << <STR_LIT> ) ; } if ( MRI -> def_begin ( Reg ) == MRI -> def_end ( ) ) { DEBUG ( dbgs ( ) << <STR_LIT> << printReg ( Reg , MRI -> getTargetRegisterInfo ( ) ) << <STR_LIT> ) ; } else if ( ! MRI -> hasOneDef ( Reg ) ) { DEBUG ( dbgs ( ) << <STR_LIT> << printReg ( Reg , MRI -> getTargetRegisterInfo ( ) ) << <STR_LIT> ) ; } assert ( MRI -> hasOneDef ( Reg ) && <STR_LIT> ) ; MachineOperand * Def = & ( * ( MRI -> def_begin ( Reg ) ) ) ; MachineOperand * UseOperand = & ( RI ) ; bool UseIsOutsideDefMBB = Def -> getParent ( ) -> getParent ( ) != MBB ; if ( UseIsOutsideDefMBB && UseOperand -> isKill ( ) ) { DEBUG ( dbgs ( ) << <STR_LIT> << printReg ( Reg , TRI ) << <STR_LIT> ) ; UseOperand -> setIsKill ( false


<s> MachineInstrBuilder MIB = BuildMI ( * MBB , PHI , PHI . getDebugLoc ( ) , TII -> get ( TargetOpcode :: PHI ) , getPHIDestReg ( PHI ) ) ; MIB . addReg ( CombinedSourceReg ) ; MIB . addMBB ( LastMerge ) ; DEBUG ( dbgs ( ) << printReg ( CombinedSourceReg , TRI ) << <STR_LIT> << printMBBReference ( * LastMerge ) ) ; for ( unsigned i = <NUM_LIT> ; i < NumInputs ; ++ i ) { if ( isPHIRegionIndex ( PHIRegionIndices , i ) ) { continue ; } unsigned SourceReg = getPHISourceReg ( PHI , i ) ; MachineBasicBlock * SourcePred = getPHIPred ( PHI , i ) ; MIB . addReg ( SourceReg ) ; MIB . addMBB ( SourcePred ) ; DEBUG ( dbgs ( ) << printReg ( SourceReg , TRI ) << <STR_LIT> << printMBBReference ( * SourcePred ) ) ; } DEBUG ( dbgs ( )


<s> } else { DEBUG ( dbgs ( ) << <STR_LIT> << printReg ( Register , MRI -> getTargetRegisterInfo ( ) ) << <STR_LIT> << printReg ( NewRegister , MRI -> getTargetRegisterInfo ( ) ) << <STR_LIT> ) ; O . setReg ( NewRegister ) ; } } PHIInfo . deleteDef ( Register ) ; getRegionMRT ( ) -> replaceLiveOutReg ( Register


<s> I != E ; ) { MachineOperand & O = * I ; ++ I ; if ( TargetRegisterInfo :: isPhysicalRegister ( NewRegister ) ) { DEBUG ( dbgs ( ) << <STR_LIT> << printReg ( NewRegister , MRI -> getTargetRegisterInfo ( ) ) << <STR_LIT> ) ; llvm_unreachable ( <STR_LIT> ) ; } else { DEBUG ( dbgs ( ) << <STR_LIT> << printReg ( Register , MRI -> getTargetRegisterInfo ( ) ) << <STR_LIT> << printReg ( NewRegister , MRI -> getTargetRegisterInfo ( ) ) << <STR_LIT> ) ; O . setReg ( NewRegister ) ; } } PHIInfo . deleteDef


<s> DEBUG ( dbgs ( ) << <STR_LIT> ) ; DEBUG ( PHIInfo . dump ( MRI ) ) ; for ( auto DRI = PHIInfo . dests_begin ( ) , DE = PHIInfo . dests_end ( ) ; DRI != DE ; ++ DRI ) { unsigned DestReg = * DRI ; DEBUG ( dbgs ( ) << <STR_LIT> << printReg ( DestReg , TRI ) << <STR_LIT> ) ; auto SRI = PHIInfo . sources_begin ( DestReg


<s> if ( ! containsDef ( CodeBB , InnerRegion , LI ) || ( ! IsSingleBB && ( getDefInstr ( LI ) -> getParent ( ) == LRegion -> getExit ( ) ) ) ) { DEBUG ( dbgs ( ) << <STR_LIT> ) ; continue ; } DEBUG ( dbgs ( ) << <STR_LIT> ) ; unsigned Reg = LI ; if ( Reg != InnerRegion -> getBBSelectRegOut ( ) ) { const TargetRegisterClass * RegClass = MRI -> getRegClass ( Reg ) ; unsigned PHIDestReg = MRI -> createVirtualRegister ( RegClass ) ; unsigned IfSourceReg = MRI -> createVirtualRegister ( RegClass ) ; DEBUG ( dbgs ( ) << <STR_LIT> << printReg ( Reg ) << <STR_LIT> ) ; TII -> materializeImmediate ( * IfBB , IfBB -> getFirstTerminator ( ) , DebugLoc ( ) , IfSourceReg , <NUM_LIT> ) ; InnerRegion -> replaceRegisterOutsideRegion ( Reg , PHIDestReg , true , MRI ) ; DEBUG ( dbgs ( ) << <STR_LIT>


<s> MachineBasicBlock * <ISA_LIT>MachineCFGStructurizer :: splitEntry ( LinearizedRegion * LRegion ) { MachineBasicBlock * Entry = LRegion -> getEntry ( ) ; MachineBasicBlock * EntrySucc = split ( Entry -> getFirstNonPHI ( ) ) ; MachineBasicBlock * Exit = LRegion -> getExit ( ) ; DEBUG ( dbgs ( ) << <STR_LIT> << printMBBReference ( * Entry ) << <STR_LIT> << printMBBReference ( * Entry ) << <STR_LIT> << printMBBReference ( * EntrySucc


<s> unsigned RegionIndex = PHIRegionIndices [ <NUM_LIT> ] ; unsigned RegionSourceReg = getPHISourceReg ( PHI , RegionIndex ) ; MachineBasicBlock * RegionSourceMBB = getPHIPred ( PHI , RegionIndex ) ; unsigned PHIDest = getPHIDestReg ( PHI ) ; unsigned PHISource = PHIDest ; unsigned ReplaceReg ; if ( shrinkPHI ( PHI , PHIRegionIndices , & ReplaceReg ) ) { PHISource = ReplaceReg ; } const TargetRegisterClass * RegClass = MRI -> getRegClass ( PHIDest ) ; unsigned NewDestReg = MRI -> createVirtualRegister ( RegClass ) ; LRegion -> replaceRegisterInsideRegion ( PHIDest , NewDestReg , false , MRI ) ; MachineInstrBuilder MIB = BuildMI ( * EntrySucc , EntrySucc -> instr_begin ( ) , PHI . getDebugLoc ( ) , TII -> get ( TargetOpcode :: PHI ) , NewDestReg ) ; DEBUG ( dbgs ( ) << <STR_LIT> << printReg ( NewDestReg , TRI ) << <STR_LIT> ) ; MIB . addReg ( PHISource


<s> MachineBasicBlock * RegionSourceMBB = getPHIPred ( PHI , RegionIndex ) ; unsigned PHIDest = getPHIDestReg ( PHI ) ; unsigned PHISource = PHIDest ; unsigned ReplaceReg ; if ( shrinkPHI ( PHI , PHIRegionIndices , & ReplaceReg ) ) { PHISource = ReplaceReg ; } const TargetRegisterClass * RegClass = MRI -> getRegClass ( PHIDest ) ; unsigned NewDestReg = MRI -> createVirtualRegister ( RegClass ) ; LRegion -> replaceRegisterInsideRegion ( PHIDest , NewDestReg , false , MRI ) ; MachineInstrBuilder MIB = BuildMI ( * EntrySucc , EntrySucc -> instr_begin ( ) , PHI . getDebugLoc ( ) , TII -> get ( TargetOpcode :: PHI


<s> if ( PHIInfo . isSource ( Reg ) ) { DEBUG ( dbgs ( ) << <STR_LIT> << printReg ( Reg , TRI ) << <STR_LIT> ) ; addLiveOut ( Reg ) ; } else { for ( auto & UI : MRI -> use_operands ( Reg ) ) { if ( UI . getParent ( ) -> getParent ( ) != MBB ) { DEBUG ( dbgs ( ) << <STR_LIT> << printMBBReference ( * MBB ) << <STR_LIT> << printReg ( Reg , TRI ) << <STR_LIT> ) ; addLiveOut ( Reg ) ; } else { MachineInstr * UseInstr = UI


<s> TII -> convertNonUniformIfRegion ( CurrentMerge , IfEnd ) ; } DEBUG ( PHIInfo . dump ( MRI ) ) ; } } LRegion -> removeFalseRegisterKills ( MRI ) ; if ( LRegion -> getHasLoop ( ) ) { MachineBasicBlock * NewSucc = splitEntry ( LRegion ) ; if ( isFunctionEntryBlock ( LRegion -> getEntry ( ) ) ) { resolvePHIInfos ( LRegion -> getEntry ( ) ) ; } const DebugLoc & DL = NewSucc -> findDebugLoc ( NewSucc -> getFirstNonPHI ( ) ) ; unsigned InReg = LRegion -> getBBSelectRegIn ( ) ; unsigned InnerSelectReg = MRI -> createVirtualRegister ( MRI -> getRegClass ( InReg ) ) ; unsigned NewInReg = MRI -> createVirtualRegister ( MRI -> getRegClass ( InReg ) ) ; TII -> materializeImmediate ( * ( LRegion -> getEntry ( ) ) , LRegion -> getEntry ( ) -> getFirstTerminator ( ) , DL , NewInReg , Region -> getEntry ( ) -> getNumber ( ) ) ; LRegion -> replaceRegisterInsideRegion ( InReg , InnerSelectReg , false , MRI ) ; DEBUG ( dbgs ( ) << <STR_LIT> ) ; insertMergePHI ( LRegion -> getEntry ( ) , LRegion -> getExit ( ) , NewSucc , InnerSelectReg , NewInReg , LRegion -> getRegionMRT ( ) -> getInnerOutputRegister ( ) ) ; splitExit ( LRegion ) ; TII -> convertNonUniformLoopRegion ( NewSucc , LastMerge ) ; } if ( Region -> isRoot ( ) ) { TII -> insertReturn ( * LastMerge ) ; } DEBUG ( Region -> getEntry ( ) -> getParent ( ) -> dump ( ) ) ; DEBUG ( LRegion -> print ( dbgs ( ) , TRI ) ) ; DEBUG ( PHIInfo . dump ( MRI ) ) ; DEBUG ( dbgs ( ) << <STR_LIT> ) ; Region -> setLinearizedRegion (


<s> } else { RegionMRT * Region = Tree -> getRegionMRT ( ) ; for ( MRT * C : llvm :: reverse ( * Region -> getChildren ( ) ) ) if ( containsNewBackedge ( C , MBBs ) ) return true ; } return false


<s> LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << printMBBReference ( * CodeBBStart ) << <STR_LIT> << printMBBReference ( * CodeBBEnd ) << <STR_LIT> ) ; MachineBasicBlock * SinglePred = CodeBBStart -> pred_size ( ) == <NUM_LIT> ? * ( CodeBBStart -> pred_begin ( ) ) : nullptr ; const DebugLoc & DL = SinglePred ? SinglePred -> findDebugLoc ( SinglePred -> getFirstTerminator ( ) ) : DebugLoc ( ) ; Register Reg = TII -> insertEQ ( IfBB , IfBB -> begin ( ) , DL , IfReg , SelectBB -> getNumber ( ) ) ; if ( & ( * ( IfBB -> getParent ( ) -> begin ( ) ) ) == IfBB ) { TII -> materializeImmediate ( * IfBB , IfBB -> begin ( ) , DL , IfReg , CodeBBStart -> getNumber ( ) ) ; } MachineOperand RegOp = MachineOperand :: CreateReg ( Reg , false , false , true ) ; ArrayRef < MachineOperand > Cond ( RegOp ) ; TII -> insertBranch ( * IfBB , MergeBB ,


<s> if ( ! Cond [ <NUM_LIT> ] . isReg ( ) ) return ; Register CondReg = Cond [ <NUM_LIT> ]


<s> SmallPtrSet < MachineBasicBlock * , <NUM_LIT> > Successors ; for ( MachineBasicBlock * Succ : Exit -> successors ( ) ) { if ( LRegion -> contains ( Succ ) ) { assert ( InternalSucc == nullptr ) ; InternalSucc = Succ ; } else { assert ( ExternalSucc == nullptr ) ; ExternalSucc = Succ ; } } for ( auto & TI : Exit -> terminators ( ) ) { for ( auto & UI : TI . uses ( ) ) { if ( UI . isMBB ( ) ) { auto Target = UI . getMBB ( ) ; if ( Target != InternalSucc && Target != ExternalSucc


<s> for ( MachineBasicBlock * Succ : MBB -> successors ( ) ) if ( MBBs . contains ( Succ ) ) return true ; return false


<s> for ( MachineBasicBlock * Pred : StartMBB -> predecessors ( ) ) if ( Pred != EndMBB ) Succs . insert ( std :: make_pair ( Pred , StartMBB ) ) ; for ( auto <ISA_LIT> : Succs ) { std :: pair < MachineBasicBlock * , MachineBasicBlock * > Edge = <ISA_LIT> ; LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << printMBBReference ( * Edge . first ) << <STR_LIT> << printMBBReference ( * Edge . second ) << <STR_LIT> ) ; Edge . first -> removeSuccessor ( Edge .


<s> bool IsDead = true ; MachineBasicBlock * DefMBB = getDefInstr ( Reg ) -> getParent ( ) ; for ( const MachineOperand & MO : MRI -> use_operands ( Reg ) ) if ( MO . getParent ( ) -> getParent ( ) != DefMBB ) IsDead = false ; LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << printReg ( Reg , TRI ) << <STR_LIT> << ( IsDead ? <STR_LIT> :


<s> for ( auto MBB : InnerRegion -> MBBs ) { addMBB ( MBB


<s> bool LinearizedRegion :: contains ( MachineBasicBlock * MBB ) { return MBBs . contains


<s> NumSources ++ ; } if ( NumSources == <NUM_LIT> ) { auto SRI = PHIInfo . sources_begin ( DestReg ) ; unsigned SourceReg = ( * SRI ) . first ; replaceRegisterWith ( DestReg , SourceReg ) ; } else { const DebugLoc & DL = Entry -> findDebugLoc ( Entry -> begin ( ) ) ; MachineInstrBuilder MIB = BuildMI ( * Entry , Entry -> instr_begin ( ) , DL , TII -> get ( TargetOpcode :: PHI ) , DestReg ) ; LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << printReg ( DestReg , TRI ) << <STR_LIT> ) ; unsigned CurrentBackedgeReg = <NUM_LIT> ; for ( auto SRI = PHIInfo . sources_begin ( DestReg ) ; SRI != SE ; ++ SRI ) { unsigned SourceReg = ( * SRI ) . first ; if ( CurrentRegion -> contains ( ( * SRI ) . second ) ) { if ( CurrentBackedgeReg == <NUM_LIT> ) { CurrentBackedgeReg = SourceReg ; } else { MachineInstr * PHIDefInstr = getDefInstr ( SourceReg ) ; MachineBasicBlock * PHIDefMBB = PHIDefInstr -> getParent ( ) ; const TargetRegisterClass * RegClass = MRI -> getRegClass ( CurrentBackedgeReg ) ; Register NewBackedgeReg = MRI -> createVirtualRegister ( RegClass ) ; MachineInstrBuilder BackedgePHI = BuildMI ( * PHIDefMBB , PHIDefMBB -> instr_begin ( ) , DL , TII -> get ( TargetOpcode :: PHI ) , NewBackedgeReg ) ; BackedgePHI . addReg ( CurrentBackedgeReg ) ; BackedgePHI . addMBB ( getPHIPred ( * PHIDefInstr , <NUM_LIT> ) ) ; BackedgePHI . addReg ( getPHISourceReg ( * PHIDefInstr , <NUM_LIT> ) ) ; BackedgePHI . addMBB ( ( * SRI ) . second ) ; CurrentBackedgeReg = NewBackedgeReg ; LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << printReg ( NewBackedgeReg , TRI ) << <STR_LIT> << printReg ( CurrentBackedgeReg , TRI ) << <STR_LIT> << printMBBReference ( * getPHIPred ( * PHIDefInstr , <NUM_LIT> ) ) << <STR_LIT> << printReg ( getPHISourceReg ( * PHIDefInstr , <NUM_LIT> ) , TRI ) << <STR_LIT> << printMBBReference ( * ( * SRI ) . second ) ) ; } } else { MIB . addReg ( SourceReg ) ; MIB . addMBB ( ( * SRI ) . second


<s> MF -> splice ( MergeIter , CodeBBStartI , ++ CodeBBEndI ) ; IfBB -> addSuccessor ( MergeBB ) ; IfBB -> addSuccessor ( CodeBBStart ) ; LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << IfBB -> getNumber ( ) << <STR_LIT> ) ; if ( ! CodeBBEnd -> isSuccessor ( MergeBB ) ) CodeBBEnd -> addSuccessor ( MergeBB ) ; LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << printMBBReference ( * CodeBBStart ) << <STR_LIT> << printMBBReference ( * CodeBBEnd ) << <STR_LIT> ) ; MachineBasicBlock * SinglePred = CodeBBStart -> pred_size ( ) == <NUM_LIT> ? * ( CodeBBStart -> pred_begin ( ) ) : nullptr ; const DebugLoc & DL = SinglePred ? SinglePred -> findDebugLoc ( SinglePred -> getFirstTerminator ( ) ) : DebugLoc ( ) ; Register Reg = TII -> insertEQ ( IfBB , IfBB -> begin ( ) , DL , IfReg , SelectBB -> getNumber ( ) ) ; if ( & ( * ( IfBB -> getParent ( ) -> begin ( ) ) ) == IfBB ) { TII -> materializeImmediate ( * IfBB , IfBB -> begin ( ) , DL , IfReg , CodeBBStart -> getNumber ( ) ) ; } MachineOperand RegOp = MachineOperand :: CreateReg ( Reg , false , false


<s> void <ISA_LIT>MachineCFGStructurizer :: ensureCondIsNotKilled ( SmallVector < MachineOperand , <NUM_LIT> > Cond ) { if ( Cond . size ( ) != <NUM_LIT> ) return ; if ( ! Cond [ <NUM_LIT> ] . isReg ( ) ) return ; Register CondReg = Cond [ <NUM_LIT> ] . getReg ( ) ; for ( auto UI = MRI -> use_begin ( CondReg ) , E = MRI -> use_end ( ) ; UI != E ; ++ UI ) { ( * UI ) . setIsKill ( false


<s> bool LinearizedRegion :: isLiveOut ( unsigned Reg ) { return LiveOuts . contains (


<s> } if ( MRI -> def_begin ( Reg ) == MRI -> def_end ( ) ) { LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << printReg ( Reg , MRI -> getTargetRegisterInfo ( ) ) << <STR_LIT> ) ; } else if ( ! MRI -> hasOneDef ( Reg ) ) { LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << printReg ( Reg , MRI -> getTargetRegisterInfo ( ) ) << <STR_LIT> ) ; } assert ( MRI -> hasOneDef ( Reg ) && <STR_LIT> ) ; MachineOperand * Def = & ( * ( MRI -> def_begin ( Reg ) ) ) ; MachineOperand * UseOperand = & (


<s> } else if ( ! MRI -> hasOneDef ( Reg ) ) { LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << printReg ( Reg , MRI -> getTargetRegisterInfo ( ) ) << <STR_LIT> ) ; } assert ( MRI -> hasOneDef ( Reg ) && <STR_LIT> ) ; MachineOperand * Def = & ( * ( MRI -> def_begin ( Reg ) ) ) ; MachineOperand * UseOperand = & ( RI ) ; bool UseIsOutsideDefMBB = Def -> getParent ( ) -> getParent ( ) != MBB ; if ( UseIsOutsideDefMBB && UseOperand -> isKill ( )


<s> TII -> materializeImmediate ( * CodeBB , CodeBB -> getFirstTerminator ( ) , DL , BBSelectReg , TrueBB -> getNumber ( ) ) ; } else { const TargetRegisterClass * RegClass = MRI -> getRegClass ( BBSelectReg ) ; Register TrueBBReg = MRI -> createVirtualRegister ( RegClass ) ; Register FalseBBReg = MRI -> createVirtualRegister ( RegClass ) ; TII -> materializeImmediate ( * CodeBB , CodeBB -> getFirstTerminator ( ) , DL , TrueBBReg , TrueBB -> getNumber ( ) ) ; TII -> materializeImmediate ( * CodeBB , CodeBB -> getFirstTerminator ( ) , DL , FalseBBReg , FalseBB -> getNumber


<s> const TargetRegisterClass * RegClass = MRI -> getRegClass ( Reg ) ; Register PHIDestReg = MRI -> createVirtualRegister ( RegClass ) ; Register IfSourceReg = MRI -> createVirtualRegister ( RegClass ) ; LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << printReg ( Reg ) << <STR_LIT> ) ; TII -> materializeImmediate ( * IfBB , IfBB -> getFirstTerminator ( ) , DebugLoc ( ) , IfSourceReg , <NUM_LIT> ) ; InnerRegion -> replaceRegisterOutsideRegion ( Reg , PHIDestReg , true , MRI ) ; LLVM_DEBUG ( dbgs ( ) << <STR_LIT> ) ; insertMergePHI ( IfBB , InnerRegion -> getExit ( ) , MergeBB , PHIDestReg , IfSourceReg , Reg , true ) ; } } SmallVector < unsigned , <NUM_LIT> > Sources ; if ( PHIInfo . findSourcesFromMBB ( CodeBB , Sources ) ) { LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << printMBBReference ( * CodeBB ) << <STR_LIT> ) ; for ( auto <ISA_LIT> : Sources ) { unsigned DestReg


<s> MachineBasicBlock * RegionSourceMBB = getPHIPred ( PHI , RegionIndex ) ; unsigned PHIDest = getPHIDestReg ( PHI ) ; unsigned PHISource = PHIDest ; unsigned ReplaceReg ; if ( shrinkPHI ( PHI , PHIRegionIndices , & ReplaceReg ) ) { PHISource = ReplaceReg ; } const TargetRegisterClass * RegClass = MRI -> getRegClass ( PHIDest ) ; Register NewDestReg = MRI -> createVirtualRegister ( RegClass ) ; LRegion -> replaceRegisterInsideRegion ( PHIDest , NewDestReg , false , MRI ) ; MachineInstrBuilder MIB = BuildMI ( * EntrySucc , EntrySucc -> instr_begin ( ) , PHI . getDebugLoc ( ) , TII -> get ( TargetOpcode :: PHI ) , NewDestReg ) ; LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << printReg ( NewDestReg , TRI ) << <STR_LIT> ) ; MIB . addReg ( PHISource ) ; MIB . addMBB ( Entry ) ; LLVM_DEBUG ( dbgs ( ) << printReg ( PHISource , TRI ) << <STR_LIT> << printMBBReference ( * Entry ) ) ; MIB . addReg


<s> if ( Reg . isVirtual ( ) ) { LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << printReg ( Reg , TRI ) << <STR_LIT> ) ; if ( PHIInfo . isSource ( Reg ) ) { LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << printReg ( Reg , TRI ) << <STR_LIT> ) ; addLiveOut ( Reg ) ; } else { for ( auto & UI : MRI -> use_operands


<s> unsigned <ISA_LIT>MachineCFGStructurizer :: storePHILinearizationInfo ( MachineInstr & PHI , SmallVector < unsigned , <NUM_LIT> > * RegionIndices ) { unsigned DestReg = getPHIDestReg ( PHI ) ; Register LinearizeDestReg = MRI -> createVirtualRegister ( MRI -> getRegClass (


<s> PHIInfo . addDest ( LinearizeDestReg , PHI . getDebugLoc ( ) ) ; storePHILinearizationInfoDest ( LinearizeDestReg , PHI , RegionIndices ) ; return


<s> dbgs ( ) << <STR_LIT> << PrintReg ( getBBSelectRegOut ( ) , TRI ) << <STR_LIT> ; dumpDepth ( depth ) ; if ( getSucc ( ) ) dbgs ( ) << <STR_LIT> << getSucc ( ) -> getNumber ( )


<s> virtual MBBMRT * getMBBMRT ( ) { return this


<s> void LinearizedRegion :: addMBB ( MachineBasicBlock


<s> void LinearizedRegion :: addMBB ( MachineBasicBlock *


<s> void LinearizedRegion :: addMBBs ( LinearizedRegion * InnerRegion ) { for ( const auto & MBB : InnerRegion -> MBBs ) { addMBB (


<s> void PHILinearize :: addSource ( unsigned DestReg , unsigned SourceReg


<s> for ( auto MBBI : post_order ( & ( MF . front ( ) ) ) ) { MachineBasicBlock * MBB = & ( * MBBI ) ; if ( MBB == Exit ) { continue ; } LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << printMBBReference ( * MBB ) << <STR_LIT> ) ; MBBMRT * NewMBB = new MBBMRT ( MBB ) ; MachineRegion * Region = RegionInfo -> getRegionFor ( MBB ) ; if ( RegionMap . count ( Region ) == <NUM_LIT> ) { RegionMRT * NewMRTRegion = new RegionMRT ( Region ) ; RegionMap [ Region ] = NewMRTRegion ; MachineRegion * Parent = Region -> getParent ( ) ; while ( RegionMap . count ( Parent ) == <NUM_LIT> ) { RegionMRT * NewMRTParent = new RegionMRT ( Parent ) ; NewMRTParent -> addChild ( NewMRTRegion ) ; NewMRTRegion -> setParent ( NewMRTParent ) ; RegionMap [ Parent ] = NewMRTParent ; NewMRTRegion = NewMRTParent ; Parent = Parent -> getParent ( ) ; } RegionMap [ Parent ] -> addChild (


<s> bool LinearizedRegion :: contains ( MachineBasicBlock


<s> bool <ISA_LIT>MachineCFGStructurizer :: containsDef ( MachineBasicBlock * MBB , LinearizedRegion


<s> bool <ISA_LIT>MachineCFGStructurizer :: containsDef ( MachineBasicBlock * MBB , LinearizedRegion * InnerRegion , unsigned Register ) { return getDefInstr ( Register ) -> getParent ( ) == MBB || InnerRegion -> contains ( getDefInstr ( Register ) -> getParent


<s> static bool containsNewBackedge ( RegionMRT * Region


<s> return new <ISA_LIT>MachineCFGStructurizer (


<s> static unsigned createBBSelectReg ( const <ISA_LIT>InstrInfo * TII , MachineRegisterInfo *


<s> return MRI -> createVirtualRegister ( TII ->

getPreferredSelectRegClass ( <NUM_LIT> ) ) ;


<s> LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << Exit -> getNumber ( ) << <STR_LIT> << ( * ( Entry -> pred_begin ( ) ) ) -> getNumber ( ) << <STR_LIT> ) ; int NumSources = <NUM_LIT> ; auto SE = PHIInfo . sources_end ( DestReg ) ; for ( auto SRI = PHIInfo . sources_begin ( DestReg ) ; SRI != SE ; ++ SRI ) { NumSources ++ ; } if ( NumSources == <NUM_LIT> ) { auto SRI = PHIInfo . sources_begin ( DestReg ) ; unsigned SourceReg = ( * SRI ) . first ; replaceRegisterWith ( DestReg , SourceReg ) ; } else { const DebugLoc & DL = Entry -> findDebugLoc ( Entry -> begin ( ) ) ; MachineInstrBuilder MIB = BuildMI ( * Entry , Entry -> instr_begin ( ) , DL , TII -> get ( TargetOpcode :: PHI ) , DestReg ) ; LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << printReg ( DestReg , TRI ) << <STR_LIT> ) ; unsigned CurrentBackedgeReg = <NUM_LIT> ; for ( auto SRI = PHIInfo . sources_begin ( DestReg ) ; SRI != SE ; ++ SRI ) { unsigned SourceReg = ( * SRI ) . first ; if ( CurrentRegion -> contains ( ( * SRI ) . second ) ) { if ( CurrentBackedgeReg == <NUM_LIT> ) { CurrentBackedgeReg = SourceReg ; } else { MachineInstr * PHIDefInstr = getDefInstr ( SourceReg ) ; MachineBasicBlock * PHIDefMBB = PHIDefInstr -> getParent ( ) ; const TargetRegisterClass * RegClass = MRI -> getRegClass ( CurrentBackedgeReg ) ; unsigned NewBackedgeReg = MRI -> createVirtualRegister ( RegClass ) ; MachineInstrBuilder BackedgePHI = BuildMI ( * PHIDefMBB , PHIDefMBB -> instr_begin ( ) , DL , TII -> get ( TargetOpcode :: PHI ) , NewBackedgeReg ) ; BackedgePHI . addReg


<s> LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << IfBB -> getNumber ( ) << <STR_LIT> ) ; if ( ! CodeBBEnd -> isSuccessor ( MergeBB ) ) CodeBBEnd -> addSuccessor ( MergeBB ) ; LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << printMBBReference ( * CodeBBStart ) << <STR_LIT> << printMBBReference ( * CodeBBEnd ) << <STR_LIT> ) ; MachineBasicBlock * SinglePred = CodeBBStart -> pred_size ( ) == <NUM_LIT> ? * ( CodeBBStart -> pred_begin ( ) ) : nullptr ; const DebugLoc & DL = SinglePred ? SinglePred -> findDebugLoc ( SinglePred -> getFirstTerminator ( ) ) : DebugLoc ( ) ; unsigned Reg = TII -> insertEQ ( IfBB , IfBB -> begin ( ) , DL , IfReg


<s> MachineFunction * MF = Region -> getEntry ( ) -> getParent ( ) ; if ( Exit == nullptr && hasOneExitNode ( * MF ) ) { return & ( * ( -- ( Region -> getEntry ( ) -> getParent ( ) -> end ( ) ) ) ) ; } MachineBasicBlock * LastMerge = MF -> CreateMachineBasicBlock ( ) ; if ( Exit == nullptr ) { MachineFunction :: iterator ExitIter = MF -> end ( ) ; MF -> insert ( ExitIter , LastMerge ) ; } else { MachineFunction :: iterator ExitIter = Exit -> getIterator


<s> void <ISA_LIT>MachineCFGStructurizer :: createLinearizedRegion ( RegionMRT * Region , unsigned SelectOut ) { LinearizedRegion * LRegion = new LinearizedRegion ( ) ; if ( SelectOut ) { LRegion -> addLiveOut


<s> PHIInfo . erase ( InfoElement ) ; delete InfoElement


<s> PHIInfoElementT * InfoElement = findPHIInfoElement ( DestReg


<s> PHILinearize :: dest_iterator PHILinearize :: dests_begin ( ) { return PHILinearizeDestIterator ( PHIInfo . begin (


<s> PHILinearize :: dest_iterator PHILinearize :: dests_end ( ) { return PHILinearizeDestIterator ( PHIInfo .


<s> PHILinearize :: dest_iterator PHILinearize :: dests_end ( ) { return PHILinearizeDestIterator ( PHIInfo . end ( )


<s> if ( Cond . size ( ) != <NUM_LIT> ) return ; if ( ! Cond [ <NUM_LIT> ] . isReg ( ) ) return ; unsigned CondReg = Cond [ <NUM_LIT> ] . getReg ( ) ; for ( auto UI = MRI -> use_begin ( CondReg ) , E = MRI -> use_end ( ) ; UI != E


<s> bool PHILinearize :: findDest ( unsigned SourceReg , MachineBasicBlock * SourceMBB , unsigned & DestReg ) { PHIInfoElementT * InfoElement = findPHIInfoElementFromSource ( SourceReg , SourceMBB ) ; if ( InfoElement != nullptr ) { DestReg = phiInfoElementGetDest ( InfoElement ) ; return true ; } return false


<s> for ( auto I : PHIInfo ) { if ( phiInfoElementGetDest ( I ) == DestReg ) { return I


<s> PHILinearize :: PHIInfoElementT * PHILinearize :: findPHIInfoElementFromSource ( unsigned SourceReg , MachineBasicBlock * SourceMBB ) { for ( auto I : PHIInfo ) { for ( auto <ISA_LIT> : phiInfoElementGetSources ( I ) ) { if ( <ISA_LIT> . first == SourceReg && ( <ISA_LIT> . second == nullptr || <ISA_LIT> . second == SourceMBB ) ) { return


<s> for ( auto I : PHIInfo ) { for ( auto <ISA_LIT> : phiInfoElementGetSources ( I ) ) { if ( <ISA_LIT> . second == SourceMBB ) { FoundSource = true ; Sources . push_back (


<s> for ( auto & TI : MBB -> terminators ( ) ) { for ( auto & UI : TI . uses ( ) ) { if ( UI . isMBB ( ) && UI . getMBB ( ) != Succ ) { UI . setMBB (


<s> auto Exit = LRegion -> getExit ( ) ; SmallPtrSet < MachineBasicBlock * , <NUM_LIT> > Successors ; for ( MachineBasicBlock :: const_succ_iterator <ISA_LIT> = Exit -> succ_begin ( ) , SE = Exit -> succ_end ( ) ; <ISA_LIT> != SE ; ++ <ISA_LIT> ) { MachineBasicBlock * Succ = * <ISA_LIT> ; if ( LRegion -> contains ( Succ ) ) { assert ( InternalSucc == nullptr ) ; InternalSucc = Succ ; } else { assert ( ExternalSucc == nullptr ) ; ExternalSucc = Succ


<s> AU . addRequired < MachineRegionInfoPass > ( ) ; MachineFunctionPass :: getAnalysisUsage (


<s> unsigned LinearizedRegion :: getBBSelectRegOut


<s> unsigned LinearizedRegion :: getBBSelectRegOut ( ) { return getRegionMRT ( ) ->


<s> MachineBasicBlock * LinearizedRegion :: getEntry ( ) { return Entry


<s> MRT * getEntryTree ( ) { return Children .


<s> MachineBasicBlock * LinearizedRegion :: getExit ( ) { return Exit


<s> MachineBasicBlock * LinearizedRegion :: getExit (


<s> return ( * ( Children . begin ( ) ) ) -> getBBSelectRegOut (


<s> unsigned getInnerOutputRegister ( ) { return ( * ( Children . begin ( ) ) ) -> getBBSelectRegOut


<s> LinearizedRegion * getLinearizedRegion ( ) { return


<s> DenseSet < unsigned > * LinearizedRegion ::


<s> MachineRegion * getMachineRegion ( ) { return Region


<s> MBBMRT * getMBBMRT ( ) override { return


<s> return PHI . getOperand ( <NUM_LIT> ) .


<s> static MachineBasicBlock * getPHIPred ( MachineInstr & PHI , unsigned Index ) { assert ( PHI . isPHI ( )


<s> return PHI . getOperand ( Index * <NUM_LIT> + <NUM_LIT> ) . getReg


<s> bool LinearizedRegion :: hasNoDef ( unsigned Reg , MachineRegisterInfo * MRI ) { return MRI -> def_begin ( Reg ) ==


<s> LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << MBBI . getNumber ( ) << <STR_LIT> << MBB -> getNumber ( ) << <STR_LIT> ) ; } FallthroughMap [ & MBBI ] =


<s> MachineBasicBlock * ExitMBB = & MFI ; if ( ExitMBB -> succ_size ( ) == <NUM_LIT> ) { return ExitMBB ; } } llvm_unreachable (


<s> LinearizedRegion * <ISA_LIT>MachineCFGStructurizer :: initLinearizedRegion ( RegionMRT * Region ) { LinearizedRegion * LRegion = Region ->


<s> void LinearizedRegion :: initLiveOut ( RegionMRT * Region , const MachineRegisterInfo * MRI , const TargetRegisterInfo * TRI ,