<s> } if ( numArgs > <NUM_LIT> ) { if ( ( copr2 = dyn_cast < Constant > ( aCI -> getArgOperand ( <NUM_LIT> ) ) ) == nullptr ) return false ; } double DVal0 [ <NUM_LIT> ] , DVal1 [ <NUM_LIT> ] ; bool hasTwoResults = ( FInfo . getId ( ) == <ISA_LIT>LibFunc :: EI_<ISA_LIT>NCOS ) ; if ( getVecSize ( FInfo ) == <NUM_LIT> ) { if ( ! evaluateScalarMathFunc ( FInfo , DVal0 [ <NUM_LIT> ] , DVal1 [ <NUM_LIT> ] , copr0 , copr1 , copr2 ) ) { return false ; } } else { ConstantDataVector * CDV0 = dyn_cast_or_null < ConstantDataVector > ( copr0 ) ; ConstantDataVector * CDV1 = dyn_cast_or_null < ConstantDataVector > ( copr1 ) ; ConstantDataVector * CDV2 = dyn_cast_or_null < ConstantDataVector > ( copr2 ) ; for ( int i = <NUM_LIT> ; i < getVecSize ( FInfo ) ; ++ i ) { Constant * celt0 = CDV0 ? CDV0 -> getElementAsConstant ( i ) : nullptr ; Constant * celt1 = CDV1 ? CDV1 -> getElementAsConstant ( i ) : nullptr ; Constant * celt2 = CDV2 ? CDV2 -> getElementAsConstant ( i ) : nullptr ; if ( ! evaluateScalarMathFunc ( FInfo , DVal0 [ i ] , DVal1 [ i ] , celt0 , celt1 , celt2 ) ) { return false ; } } } LLVMContext & context = CI -> getParent ( ) -> getParent ( ) -> getContext ( ) ; Constant * nval0 , * nval1 ; if ( getVecSize ( FInfo ) == <NUM_LIT> ) { nval0 = ConstantFP :: get ( CI -> getType ( ) , DVal0 [ <NUM_LIT> ] ) ; if ( hasTwoResults ) nval1 = ConstantFP :: get ( CI -> getType ( ) , DVal1 [ <NUM_LIT> ] ) ; } else { if ( getArgType ( FInfo ) == <ISA_LIT>LibFunc :: F32 ) { SmallVector < float , <NUM_LIT> > FVal0 , FVal1 ; for ( int i = <NUM_LIT> ; i < getVecSize ( FInfo ) ; ++ i ) FVal0 . push_back ( ( float ) DVal0 [ i ] ) ; ArrayRef < float > tmp0 ( FVal0 ) ; nval0 = ConstantDataVector :: get ( context , tmp0 ) ; if ( hasTwoResults ) { for ( int i = <NUM_LIT> ; i < getVecSize ( FInfo ) ; ++ i ) FVal1 . push_back ( ( float ) DVal1 [ i ] ) ; ArrayRef < float > tmp1 ( FVal1 ) ; nval1 = ConstantDataVector :: get ( context , tmp1 ) ; } } else { ArrayRef < double > tmp0 ( DVal0 ) ;


<s> return ( getVecSize ( FInfo ) != <NUM_LIT> ) ? false : fold_recip ( CI , B , FInfo ) ; case <ISA_LIT>LibFunc :: EI_DIVIDE : assert ( ( FInfo . getPrefix ( ) == <ISA_LIT>LibFunc :: NATIVE || FInfo . getPrefix ( ) == <ISA_LIT>LibFunc :: HALF ) && <STR_LIT> ) ; return ( getVecSize ( FInfo ) != <NUM_LIT> ) ? false : fold_divide ( CI , B , FInfo ) ; case <ISA_LIT>LibFunc :: EI_POW : case <ISA_LIT>LibFunc :: EI_POWR : case <ISA_LIT>LibFunc :: EI_POWN : return fold_pow ( CI , B , FInfo ) ; case <ISA_LIT>LibFunc :: EI_ROOTN : return ( getVecSize ( FInfo ) != <NUM_LIT> ) ? false : fold_rootn ( CI , B , FInfo ) ; case <ISA_LIT>LibFunc :: EI_FMA : case <ISA_LIT>LibFunc :: EI_MAD : case <ISA_LIT>LibFunc :: EI_NFMA : return ( getVecSize ( FInfo ) != <NUM_LIT> ) ? false : fold_fma_mad ( CI , B , FInfo ) ; case <ISA_LIT>LibFunc :: EI_SQRT : return isUnsafeMath ( CI ) && fold_sqrt ( CI , B , FInfo ) ; case <ISA_LIT>LibFunc :: EI_COS : case <ISA_LIT>LibFunc :: EI_<ISA_LIT>N : if ( ( getArgType ( FInfo ) == <ISA_LIT>LibFunc :: F32 || getArgType ( FInfo ) == <ISA_LIT>LibFunc :: F64 ) && ( FInfo . getPrefix ( ) == <ISA_LIT>LibFunc :: NOPFX ) ) return fold_sincos ( CI , B , AA ) ;


<s> auto * Callee = CI -> getCalledFunction ( ) ; if ( ! Callee -> isDeclaration ( ) ) return false ; assert ( Callee -> hasName ( ) && <STR_LIT> ) ; auto * M = Callee -> getParent ( ) ; auto & Ctx = M -> getContext ( ) ; std :: string Name = std :: string ( Callee -> getName ( ) ) ; auto NumArg = CI -> arg_size ( ) ; if ( NumArg != <NUM_LIT> && NumArg != <NUM_LIT> ) return false ; auto * PacketSize = CI -> getArgOperand ( NumArg - <NUM_LIT> ) ; auto * PacketAlign = CI -> getArgOperand ( NumArg - <NUM_LIT> ) ; if ( ! isa < ConstantInt > ( PacketSize ) || ! isa < ConstantInt > ( PacketAlign ) ) return false ; unsigned Size = cast < ConstantInt > ( PacketSize ) -> getZExtValue ( ) ; Align Alignment = cast < ConstantInt > ( PacketAlign ) -> getAlignValue ( ) ; if ( Alignment != Size ) return false ; Type * PtrElemTy ; if ( Size <= <NUM_LIT> ) PtrElemTy = Type :: getIntNTy ( Ctx , Size * <NUM_LIT> ) ; else PtrElemTy = FixedVectorType :: get ( Type :: getInt64Ty ( Ctx ) , Size / <NUM_LIT> ) ; unsigned PtrArgLoc = CI -> arg_size ( ) - <NUM_LIT> ; auto PtrArg = CI -> getArgOperand ( PtrArgLoc ) ; unsigned PtrArgAS = PtrArg -> getType ( ) -> getPointerAddressSpace ( ) ; auto * PtrTy = llvm :: PointerType :: get ( PtrElemTy , PtrArgAS ) ; SmallVector < llvm :: Type * , <NUM_LIT> > ArgTys ; for ( unsigned I = <NUM_LIT> ; I != PtrArgLoc ; ++ I ) ArgTys . push_back ( CI -> getArgOperand ( I ) -> getType ( ) ) ; ArgTys . push_back ( PtrTy ) ; Name = Name + <STR_LIT> + std :: to_string ( Size ) ; auto * FTy = FunctionType :: get ( Callee -> getReturnType ( ) , ArrayRef < Type * > ( ArgTys ) , false ) ; <ISA_LIT>LibFunc NewLibFunc ( Name , FTy ) ; FunctionCallee F = <ISA_LIT>LibFunc :: getOrInsertFunction ( M , NewLibFunc ) ; if ( ! F ) return false ; auto * BCast = B . CreatePointerCast ( PtrArg , PtrTy ) ; SmallVector < Value * , <NUM_LIT> > Args ; for ( unsigned I = <NUM_LIT> ; I != PtrArgLoc ; ++ I ) Args . push_back ( CI -> getArgOperand ( I ) ) ; Args . push_back ( BCast ) ; auto * NCI = B . CreateCall ( F , Args ) ;

NCI -> setAttributes ( CI -> getAttributes ( ) ) ;


<s> return false ; } } } LLVMContext & context = CI -> getParent ( ) -> getParent ( ) -> getContext ( ) ; Constant * nval0 , * nval1 ; if ( FuncVecSize == <NUM_LIT> ) { nval0 = ConstantFP :: get ( CI -> getType ( ) , DVal0 [ <NUM_LIT> ] ) ; if ( hasTwoResults ) nval1 = ConstantFP :: get ( CI -> getType ( ) , DVal1 [ <NUM_LIT> ] ) ; } else { if ( getArgType ( FInfo ) == <ISA_LIT>LibFunc :: F32 ) { SmallVector < float , <NUM_LIT> > FVal0 , FVal1 ; for ( int i = <NUM_LIT> ; i < FuncVecSize ; ++ i ) FVal0 . push_back ( ( float ) DVal0 [ i ] ) ; ArrayRef < float > tmp0 ( FVal0 ) ; nval0 = ConstantDataVector :: get ( context , tmp0 ) ; if ( hasTwoResults ) { for ( int i = <NUM_LIT> ; i < FuncVecSize ; ++ i ) FVal1 . push_back ( ( float ) DVal1 [ i ] ) ; ArrayRef < float > tmp1 ( FVal1 ) ; nval1 = ConstantDataVector :: get ( context , tmp1 ) ; }


<s> LLVMContext & Context = CI -> getParent ( ) -> getContext ( ) ; IRBuilder < > B ( Context ) ; B . SetInsertPoint ( BB , CI -> getIterator ( ) ) ; if ( const FPMathOperator * FPOp = dyn_cast < const FPMathOperator > ( CI ) ) B . setFastMathFlags ( FPOp -> getFastMathFlags ( ) ) ; switch ( Callee -> getIntrinsicID ( ) ) { default : break ; case <STR_LIT> : return ! EnablePreLink && fold_wavefrontsize ( CI , B ) ; } FuncInfo FInfo ; if ( ! parseFunctionName ( Callee -> getName ( ) , FInfo ) ) return false ; if ( CI -> arg_size ( ) != FInfo . getNumArgs ( ) ) return false ; if ( TDOFold ( CI , FInfo ) ) return true ; if ( isUnsafeMath ( CI ) && evaluateCall ( CI , FInfo ) ) return true ; switch ( FInfo . getId ( ) ) { case <ISA_LIT>LibFunc :: EI_RECIP : assert ( ( FInfo . getPrefix ( ) == <ISA_LIT>LibFunc :: NATIVE || FInfo . getPrefix ( ) == <ISA_LIT>LibFunc :: HALF ) && <STR_LIT> ) ;

return ( getVecSize ( FInfo ) != <NUM_LIT> ) ? false : fold_recip ( CI , B , FInfo ) ;


<s> if ( Callee == nullptr ) return false ; BasicBlock * BB = CI -> getParent ( ) ; LLVMContext & Context = CI -> getParent ( ) -> getContext ( ) ; IRBuilder < > B ( Context ) ; B . SetInsertPoint ( BB , CI -> getIterator ( ) ) ; if ( const FPMathOperator * FPOp = dyn_cast < const FPMathOperator > ( CI ) ) B . setFastMathFlags ( FPOp -> getFastMathFlags ( ) ) ; switch ( Callee -> getIntrinsicID ( ) ) { default : break ; case <STR_LIT> : return ! EnablePreLink && fold_wavefrontsize ( CI , B ) ; } FuncInfo FInfo ; if ( ! parseFunctionName ( Callee -> getName ( ) , FInfo ) ) return false ; if ( CI -> arg_size ( ) != FInfo . getNumArgs ( ) ) return false ; if ( TDOFold ( CI , FInfo ) ) return true ; if ( isUnsafeMath ( CI ) && evaluateCall ( CI , FInfo ) ) return true ; switch ( FInfo . getId ( ) ) { case <ISA_LIT>LibFunc :: EI_RECIP : assert ( ( FInfo . getPrefix ( ) == <ISA_LIT>LibFunc :: NATIVE || FInfo . getPrefix ( ) == <ISA_LIT>LibFunc :: HALF ) && <STR_LIT> ) ; return ( getVecSize ( FInfo ) != <NUM_LIT> ) ? false : fold_recip ( CI , B , FInfo ) ; case <ISA_LIT>LibFunc :: EI_DIVIDE : assert ( ( FInfo . getPrefix ( ) == <ISA_LIT>LibFunc :: NATIVE || FInfo . getPrefix ( ) == <ISA_LIT>LibFunc :: HALF ) && <STR_LIT> ) ; return ( getVecSize ( FInfo ) != <NUM_LIT> ) ? false : fold_divide ( CI , B , FInfo ) ; case <ISA_LIT>LibFunc :: EI_POW : case <ISA_LIT>LibFunc :: EI_POWR : case <ISA_LIT>LibFunc :: EI_POWN : return fold_pow ( CI , B , FInfo ) ; case <ISA_LIT>LibFunc :: EI_ROOTN : return ( getVecSize ( FInfo ) != <NUM_LIT> ) ? false : fold_rootn ( CI , B , FInfo ) ; case <ISA_LIT>LibFunc :: EI_FMA : case <ISA_LIT>LibFunc :: EI_MAD :


<s> AllocaInst * <ISA_LIT>LibCalls :: insertAlloca ( CallInst * UI , IRBuilder < > & B , const char * prefix ) { BasicBlock :: iterator ItNew = getEntryIns ( UI ) ; Function * UCallee = UI -> getCalledFunction ( ) ; Type * RetType = UCallee -> getReturnType ( ) ; B . SetInsertPoint ( & * ItNew ) ; AllocaInst * Alloc = B . CreateAlloca ( RetType , nullptr , std :: string ( prefix ) + UI -> getName ( ) ) ;

Alloc -> setAlignment ( Align ( UCallee -> getParent ( ) -> getDataLayout ( ) . getTypeAllocSize ( RetType ) ) ) ;


<s> Type * RetType = UCallee -> getReturnType ( ) ; B . SetInsertPoint ( & * ItNew ) ;

AllocaInst * Alloc = B . CreateAlloca ( RetType , nullptr , std :: string ( prefix ) + UI -> getName ( ) ) ;


<s> bool Changed = false ; for ( auto & BB : F ) { for ( BasicBlock :: iterator I = BB . begin ( ) , E = BB . end ( ) ; I != E ; ) { CallInst * CI = dyn_cast < CallInst > ( I ) ; ++ I ; if ( ! CI ) continue ; Function * Callee = CI -> getCalledFunction ( ) ;


<s> for ( BasicBlock :: iterator I = BB . begin ( ) , E = BB . end ( ) ; I != E ; ) { CallInst * CI = dyn_cast < CallInst > ( I ) ; ++ I ; if ( ! CI ) continue ; Function * Callee = CI -> getCalledFunction ( ) ;


<s> Function * UCallee = XI -> getCalledFunction ( ) ; if ( ! UCallee || ! UCallee -> getName ( ) . equals ( PairName ) ) continue ; BasicBlock :: iterator BBI = CI -> getIterator ( ) ; if ( BBI == CI -> getParent ( ) -> begin ( ) ) break ; -- BBI ; for ( int I = MaxScan ; I > <NUM_LIT> && BBI != CBB -> begin ( ) ; -- BBI , -- I ) { if ( cast < Instruction > ( BBI ) == XI ) { UI = XI ; break ; } } if ( UI ) break ; } if ( ! UI ) return false ; <ISA_LIT>LibFunc nf ( <ISA_LIT>LibFunc :: EI_<ISA_LIT>NCOS , fInfo ) ; const <ISA_LIT>AS AS = <STR_LIT> ( * M ) ; nf . getLeads ( ) [ <NUM_LIT> ] . PtrKind = <ISA_LIT>LibFunc :: getEPtrKindFromAddrSpace ( AS . FLAT_ADDRESS ) ; Function * Fsincos = dyn_cast_or_null < Function > ( getFunction ( M , nf ) ) ; if ( ! Fsincos ) return false ; BasicBlock :: iterator ItOld = B . GetInsertPoint ( ) ; AllocaInst * Alloc = insertAlloca ( UI , B , <STR_LIT> ) ; B . SetInsertPoint ( UI ) ; Value * P = Alloc ; Type * PTy = Fsincos -> getFunctionType ( ) -> getParamType ( <NUM_LIT> ) ; if ( PTy -> getPointerAddressSpace ( ) != AS . PRIVATE_ADDRESS ) P = B . CreateAddrSpaceCast ( Alloc , PTy ) ; CallInst * Call = CreateCallEx2 ( B , Fsincos , UI -> getArgOperand ( <NUM_LIT> ) , P ) ; DEBUG ( errs ( ) << <STR_LIT> << * CI << <STR_LIT> << * UI << <STR_LIT> << * Call << <STR_LIT> ) ; if ( ! isSin ) { B . SetInsertPoint ( & * ItOld ) ; UI -> replaceAllUsesWith ( & * Call ) ; Instruction * Reload = B . CreateLoad ( Alloc ) ; CI -> replaceAllUsesWith ( Reload ) ; UI -> eraseFromParent ( ) ; CI -> eraseFromParent ( ) ;


<s> if ( ! UI ) return false ; <ISA_LIT>LibFunc nf ( <ISA_LIT>LibFunc :: EI_<ISA_LIT>NCOS , fInfo ) ; const <ISA_LIT>AS AS = <STR_LIT> ( * M ) ; nf . getLeads ( ) [ <NUM_LIT> ] . PtrKind = <ISA_LIT>LibFunc :: getEPtrKindFromAddrSpace ( AS . FLAT_ADDRESS ) ; Function * Fsincos = dyn_cast_or_null < Function > ( getFunction ( M , nf ) ) ; if ( ! Fsincos ) return false ; BasicBlock :: iterator ItOld = B . GetInsertPoint ( ) ; AllocaInst * Alloc = insertAlloca ( UI , B , <STR_LIT> ) ; B . SetInsertPoint ( UI ) ; Value * P = Alloc ; Type * PTy = Fsincos -> getFunctionType ( ) -> getParamType ( <NUM_LIT> ) ; if ( PTy -> getPointerAddressSpace ( ) != AS . PRIVATE_ADDRESS ) P = B . CreateAddrSpaceCast ( Alloc , PTy ) ; CallInst * Call = CreateCallEx2 ( B , Fsincos , UI -> getArgOperand ( <NUM_LIT> ) , P ) ; LLVM_DEBUG ( errs ( ) << <STR_LIT> << * CI << <STR_LIT> << * UI << <STR_LIT> << * Call << <STR_LIT> ) ; if ( ! isSin ) { B . SetInsertPoint ( & * ItOld ) ; UI -> replaceAllUsesWith ( & * Call ) ; Instruction * Reload = B . CreateLoad ( Alloc ) ;


<s> <ISA_LIT>LibCalls ( const TargetMachine * TM_ = nullptr ) : TM ( TM_ ) {


<s> initialize<ISA_LIT>SimplifyLibCallsPass ( * PassRegistry :: getPassRegistry ( ) ) ;


<s> <ISA_LIT>UseNativeCalls ( ) : FunctionPass ( ID ) { initialize<ISA_LIT>UseNativeCallsPass ( * PassRegistry :: getPassRegistry ( ) ) ;


<s> return new <ISA_LIT>SimplifyLibCalls ( TM ) ;


<s> FunctionPass * llvm :: create<ISA_LIT>SimplifyLibCallsPass ( const TargetMachine * TM ) { return new <ISA_LIT>SimplifyLibCalls ( TM ) ;


<s> FunctionPass * llvm :: create<ISA_LIT>UseNativeCallsPass ( ) {

return new <ISA_LIT>UseNativeCalls ( ) ;


<s> if ( Function * F = dyn_cast < Function > ( Callee . getCallee ( ) ) ) R -> setCallingConv ( F -> getCallingConv ( ) ) ;


<s> static CallInst * CreateCallEx ( IRB & B , FunctionCallee Callee , Value * Arg , const Twine & Name = <STR_LIT> ) { CallInst * R = B . CreateCall ( Callee , Arg , Name ) ; if ( Function * F = dyn_cast < Function > ( Callee . getCallee ( ) ) ) R -> setCallingConv ( F -> getCallingConv ( ) ) ;


<s> static CallInst * CreateCallEx2 ( IRB & B , FunctionCallee Callee , Value * Arg1 , Value * Arg2 , const Twine & Name = <STR_LIT> ) { CallInst * R = B . CreateCall ( Callee , { Arg1 , Arg2 } , Name ) ;

if ( Function * F = dyn_cast < Function > ( Callee . getCallee ( ) ) ) R -> setCallingConv ( F -> getCallingConv ( ) ) ;


<s> if ( Function * F = dyn_cast < Function > ( Callee . getCallee ( ) ) ) R -> setCallingConv ( F -> getCallingConv ( ) ) ; return R ;


<s> return false ; } } else { ConstantDataVector * CDV0 = dyn_cast_or_null < ConstantDataVector > ( copr0 ) ; ConstantDataVector * CDV1 = dyn_cast_or_null < ConstantDataVector > ( copr1 ) ; ConstantDataVector * CDV2 = dyn_cast_or_null < ConstantDataVector > ( copr2 ) ; for ( int i = <NUM_LIT> ; i < getVecSize ( FInfo ) ; ++ i ) { Constant * celt0 = CDV0 ? CDV0 -> getElementAsConstant ( i ) : nullptr ; Constant * celt1 = CDV1 ? CDV1 -> getElementAsConstant ( i ) : nullptr ; Constant * celt2 = CDV2 ? CDV2 -> getElementAsConstant ( i ) : nullptr ; if ( ! evaluateScalarMathFunc ( FInfo , DVal0 [ i ] , DVal1 [ i ] , celt0 , celt1 , celt2 ) ) { return false ; } } } LLVMContext & context = CI -> getParent ( ) -> getParent ( ) -> getContext ( ) ; Constant * nval0 , * nval1 ; if ( getVecSize ( FInfo ) == <NUM_LIT> ) { nval0 = ConstantFP :: get ( CI -> getType ( ) , DVal0 [ <NUM_LIT> ] ) ; if ( hasTwoResults ) nval1 = ConstantFP :: get ( CI -> getType ( ) , DVal1 [ <NUM_LIT> ] ) ; } else { if ( getArgType ( FInfo ) == <ISA_LIT>LibFunc :: F32 ) { SmallVector < float , <NUM_LIT> > FVal0 , FVal1 ; for ( int i = <NUM_LIT> ; i < getVecSize ( FInfo ) ; ++ i ) FVal0 . push_back ( ( float ) DVal0 [ i ] ) ; ArrayRef < float > tmp0 ( FVal0 ) ; nval0 = ConstantDataVector :: get ( context , tmp0 ) ; if ( hasTwoResults ) { for ( int i = <NUM_LIT> ; i < getVecSize ( FInfo ) ; ++ i ) FVal1 . push_back ( ( float ) DVal1 [ i ] ) ; ArrayRef < float > tmp1 ( FVal1 ) ; nval1 = ConstantDataVector :: get ( context , tmp1 ) ; } } else { ArrayRef < double > tmp0 ( DVal0 ) ; nval0 = ConstantDataVector :: get ( context , tmp0 ) ; if ( hasTwoResults ) { ArrayRef < double > tmp1 ( DVal1 ) ; nval1 = ConstantDataVector :: get ( context , tmp1 ) ; } } } if ( hasTwoResults ) { assert ( FInfo . getId ( ) == <ISA_LIT>LibFunc :: EI_<ISA_LIT>NCOS && <STR_LIT> ) ; new StoreInst ( nval1 , aCI -> getArgOperand ( <NUM_LIT> ) , aCI ) ; } replaceCall ( nval0 ) ; return true ;


<s> return true ; case <ISA_LIT>LibFunc :: EI_ATAN : Res0 = atan ( opr0 ) ; return true ; case <ISA_LIT>LibFunc :: EI_ATANH : Res0 = ( log ( opr0 + <NUM_LIT> ) - log ( opr0 - <NUM_LIT> ) ) / <NUM_LIT> ; return true ; case <ISA_LIT>LibFunc :: EI_ATANPI : Res0 = atan ( opr0 ) / MATH_PI ; return true ; case <ISA_LIT>LibFunc :: EI_CBRT : Res0 = ( opr0 < <NUM_LIT> ) ? - pow ( - opr0 , <NUM_LIT> / <NUM_LIT> ) : pow ( opr0 , <NUM_LIT> / <NUM_LIT> ) ; return true ; case <ISA_LIT>LibFunc :: EI_COS : Res0 = cos ( opr0 ) ; return true ; case <ISA_LIT>LibFunc :: EI_COSH : Res0 = cosh ( opr0 ) ; return true ; case <ISA_LIT>LibFunc :: EI_COSPI : Res0 = cos ( MATH_PI * opr0 ) ; return true ; case <ISA_LIT>LibFunc :: EI_EXP : Res0 = exp ( opr0 ) ; return true ; case <ISA_LIT>LibFunc :: EI_EXP2 : Res0 = pow ( <NUM_LIT> , opr0 ) ; return true ; case <ISA_LIT>LibFunc :: EI_EXP10 : Res0 = pow ( <NUM_LIT> , opr0 ) ; return true ; case <ISA_LIT>LibFunc :: EI_EXPM1 : Res0 = exp ( opr0 ) - <NUM_LIT> ; return true ; case <ISA_LIT>LibFunc :: EI_LOG : Res0 = log ( opr0 ) ; return true ; case <ISA_LIT>LibFunc :: EI_LOG2 : Res0 = log ( opr0 ) / log ( <NUM_LIT> ) ; return true ; case <ISA_LIT>LibFunc :: EI_LOG10 : Res0 = log ( opr0 ) / log ( <NUM_LIT> ) ; return true ; case <ISA_LIT>LibFunc :: EI_RSQRT : Res0 = <NUM_LIT> / sqrt ( opr0 ) ; return true ; case <ISA_LIT>LibFunc :: EI_<ISA_LIT>N : Res0 = sin ( opr0 ) ; return true ; case <ISA_LIT>LibFunc :: EI_<ISA_LIT>NH : Res0 = sinh ( opr0 ) ; return true ; case <ISA_LIT>LibFunc :: EI_<ISA_LIT>NPI : Res0 = sin ( MATH_PI * opr0 ) ; return true ; case <ISA_LIT>LibFunc :: EI_SQRT : Res0 = sqrt ( opr0 ) ; return true ; case <ISA_LIT>LibFunc :: EI_TAN : Res0 = tan ( opr0 ) ; return true ; case <ISA_LIT>LibFunc :: EI_TANH : Res0 = tanh ( opr0 ) ; return true ; case <ISA_LIT>LibFunc :: EI_TANPI : Res0 = tan ( MATH_PI * opr0 ) ;


<s> if ( const FPMathOperator * FPOp = dyn_cast < const FPMathOperator > ( CI ) ) B . setFastMathFlags ( FPOp -> getFastMathFlags ( ) ) ; switch ( Callee -> getIntrinsicID ( ) ) { default : break ; case <STR_LIT> : return ! EnablePreLink && fold_wavefrontsize ( CI , B ) ; } FuncInfo FInfo ; if ( ! parseFunctionName ( Callee -> getName ( ) , FInfo ) ) return false ; if ( CI -> arg_size ( ) != FInfo . getNumArgs ( ) ) return false ; if ( TDOFold ( CI , FInfo ) ) return true ; if ( isUnsafeMath ( CI ) && evaluateCall ( CI , FInfo ) ) return true ; switch ( FInfo . getId ( ) ) { case <ISA_LIT>LibFunc :: EI_RECIP : assert ( ( FInfo . getPrefix ( ) == <ISA_LIT>LibFunc :: NATIVE || FInfo . getPrefix ( ) == <ISA_LIT>LibFunc :: HALF ) && <STR_LIT> ) ; return ( getVecSize ( FInfo ) != <NUM_LIT> ) ? false : fold_recip ( CI , B , FInfo ) ; case <ISA_LIT>LibFunc :: EI_DIVIDE : assert ( ( FInfo . getPrefix ( ) == <ISA_LIT>LibFunc :: NATIVE || FInfo . getPrefix ( ) == <ISA_LIT>LibFunc :: HALF ) && <STR_LIT> ) ; return ( getVecSize ( FInfo ) != <NUM_LIT> ) ? false : fold_divide ( CI , B , FInfo ) ; case <ISA_LIT>LibFunc :: EI_POW : case <ISA_LIT>LibFunc :: EI_POWR : case <ISA_LIT>LibFunc :: EI_POWN : return fold_pow ( CI , B , FInfo ) ; case <ISA_LIT>LibFunc :: EI_ROOTN : return ( getVecSize ( FInfo ) != <NUM_LIT> ) ? false : fold_rootn ( CI , B , FInfo ) ; case <ISA_LIT>LibFunc :: EI_FMA : case <ISA_LIT>LibFunc :: EI_MAD :


<s> if ( Callee == <NUM_LIT> ) return false ; BasicBlock * BB = CI -> getParent ( ) ; LLVMContext & Context = CI -> getParent ( ) -> getContext ( ) ; IRBuilder < > B ( Context ) ; B . SetInsertPoint ( BB , CI -> getIterator ( ) ) ; if ( const FPMathOperator * FPOp = dyn_cast < const FPMathOperator > ( CI ) ) B . setFastMathFlags ( FPOp -> getFastMathFlags ( ) ) ; switch ( Callee -> getIntrinsicID ( ) ) { default : break ; case <STR_LIT> : return ! EnablePreLink && fold_wavefrontsize ( CI , B ) ; } FuncInfo FInfo ; if ( ! parseFunctionName ( Callee -> getName ( ) , FInfo ) ) return false ; if ( CI -> arg_size ( ) != FInfo . getNumArgs ( ) ) return false ; if ( TDOFold ( CI , FInfo ) ) return true ; if ( isUnsafeMath ( CI ) && evaluateCall ( CI , FInfo ) ) return true ; switch ( FInfo . getId ( ) ) { case <ISA_LIT>LibFunc :: EI_RECIP : assert ( ( FInfo . getPrefix ( ) == <ISA_LIT>LibFunc :: NATIVE || FInfo . getPrefix ( ) == <ISA_LIT>LibFunc :: HALF ) && <STR_LIT> ) ; return ( getVecSize ( FInfo ) != <NUM_LIT> ) ? false : fold_recip ( CI , B , FInfo ) ; case <ISA_LIT>LibFunc :: EI_DIVIDE : assert ( ( FInfo . getPrefix ( ) == <ISA_LIT>LibFunc :: NATIVE || FInfo . getPrefix ( ) == <ISA_LIT>LibFunc :: HALF ) && <STR_LIT> ) ; return ( getVecSize ( FInfo ) != <NUM_LIT> ) ? false : fold_divide ( CI , B , FInfo ) ; case <ISA_LIT>LibFunc :: EI_POW : case <ISA_LIT>LibFunc :: EI_POWR : case <ISA_LIT>LibFunc :: EI_POWN : return fold_pow ( CI , B , FInfo ) ; case <ISA_LIT>LibFunc :: EI_ROOTN : return ( getVecSize ( FInfo ) != <NUM_LIT> ) ? false : fold_rootn ( CI , B , FInfo ) ; case <ISA_LIT>LibFunc :: EI_FMA : case <ISA_LIT>LibFunc :: EI_MAD : case <ISA_LIT>LibFunc :: EI_NFMA : return ( getVecSize ( FInfo ) != <NUM_LIT> ) ? false : fold_fma_mad ( CI , B , FInfo ) ;


<s> if ( ( CF0 && CF1 ) || ( CF1 && ( getArgType ( FInfo ) == <ISA_LIT>LibFunc :: F32 ) ) ) {

Value * nval1 = B . CreateFDiv ( ConstantFP :: get ( opr1 -> getType ( ) , <NUM_LIT> ) , opr1 , <STR_LIT> ) ;


<s> Value * opr1 = CI -> getArgOperand ( <NUM_LIT> ) ; ConstantFP * CF0 = dyn_cast < ConstantFP > ( opr0 ) ; ConstantFP * CF1 = dyn_cast < ConstantFP > ( opr1 ) ; if ( ( CF0 && CF1 ) || ( CF1 && ( getArgType ( FInfo ) == <ISA_LIT>LibFunc :: F32 ) ) ) { Value * nval1 = B . CreateFDiv ( ConstantFP :: get ( opr1 -> getType ( ) , <NUM_LIT> ) , opr1 , <STR_LIT> ) ; Value * nval = B . CreateFMul ( opr0 , nval1 , <STR_LIT> ) ; replaceCall ( nval ) ; return true ; }


<s> if ( CF1 && CF1 -> isExactlyValue ( <NUM_LIT> ) ) { LLVM_DEBUG ( errs ( ) << <STR_LIT> << * CI << <STR_LIT> << * opr0 << <STR_LIT> << * opr2 << <STR_LIT> ) ; Value * nval = B . CreateFAdd ( opr0 , opr2 , <STR_LIT> ) ; replaceCall ( nval ) ; return true ; } if ( ConstantFP * CF = dyn_cast < ConstantFP > ( opr2 ) ) { if ( CF -> isZero ( ) ) { LLVM_DEBUG ( errs ( ) << <STR_LIT> << * CI << <STR_LIT> << * opr0 << <STR_LIT> << * opr1 << <STR_LIT> ) ; Value * nval = B . CreateFMul ( opr0 , opr1 , <STR_LIT> ) ; replaceCall ( nval ) ; return true ;


<s> auto * M = Callee -> getParent ( ) ; auto & Ctx = M -> getContext ( ) ; std :: string Name = std :: string ( Callee -> getName ( ) ) ; auto NumArg = CI -> arg_size ( ) ; if ( NumArg != <NUM_LIT> && NumArg != <NUM_LIT> ) return false ; auto * PacketSize = CI -> getArgOperand ( NumArg - <NUM_LIT> ) ; auto * PacketAlign = CI -> getArgOperand ( NumArg - <NUM_LIT> ) ; if ( ! isa < ConstantInt > ( PacketSize ) || ! isa < ConstantInt > ( PacketAlign ) ) return false ; unsigned Size = cast < ConstantInt > ( PacketSize ) -> getZExtValue ( ) ; Align Alignment = cast < ConstantInt > ( PacketAlign ) -> getAlignValue ( ) ; if ( Alignment != Size ) return false ; Type * PtrElemTy ; if ( Size <= <NUM_LIT> ) PtrElemTy = Type :: getIntNTy ( Ctx , Size * <NUM_LIT> ) ; else PtrElemTy = FixedVectorType :: get ( Type :: getInt64Ty ( Ctx ) , Size / <NUM_LIT> ) ; unsigned PtrArgLoc = CI -> arg_size ( ) - <NUM_LIT> ; auto PtrArg = CI -> getArgOperand ( PtrArgLoc ) ; unsigned PtrArgAS = PtrArg -> getType ( ) -> getPointerAddressSpace ( ) ; auto * PtrTy = llvm :: PointerType :: get ( PtrElemTy , PtrArgAS ) ; SmallVector < llvm :: Type * , <NUM_LIT> > ArgTys ; for ( unsigned I = <NUM_LIT> ; I != PtrArgLoc ; ++ I ) ArgTys . push_back ( CI -> getArgOperand ( I ) -> getType ( ) ) ; ArgTys . push_back ( PtrTy ) ; Name = Name + <STR_LIT> + std :: to_string ( Size ) ; auto * FTy = FunctionType :: get ( Callee -> getReturnType ( ) , ArrayRef < Type * > ( ArgTys ) , false ) ; <ISA_LIT>LibFunc NewLibFunc ( Name , FTy ) ; FunctionCallee F = <ISA_LIT>LibFunc :: getOrInsertFunction ( M , NewLibFunc ) ; if ( ! F ) return false ; auto * BCast = B . CreatePointerCast ( PtrArg , PtrTy ) ;

SmallVector < Value * , <NUM_LIT> > Args ;


<s> Value * opr0 = CI -> getArgOperand ( <NUM_LIT> ) ; Value * opr1 = CI -> getArgOperand ( <NUM_LIT> ) ; ConstantInt * CINT = dyn_cast < ConstantInt > ( opr1 ) ; if ( ! CINT ) { return false ; } int ci_opr1 = ( int ) CINT -> getSExtValue ( ) ; if ( ci_opr1 == <NUM_LIT> ) { LLVM_DEBUG ( errs ( ) << <STR_LIT> << * CI << <STR_LIT> << * opr0 << <STR_LIT> ) ; replaceCall ( opr0 ) ; return true ; } if ( ci_opr1 == <NUM_LIT> ) { Module * M = CI -> getModule ( ) ; if ( FunctionCallee FPExpr = getFunction ( M , <ISA_LIT>LibFunc ( <ISA_LIT>LibFunc :: EI_SQRT , FInfo ) ) ) { LLVM_DEBUG ( errs ( ) << <STR_LIT> << * CI << <STR_LIT> << * opr0 << <STR_LIT> ) ; Value * nval = CreateCallEx ( B , FPExpr , opr0 , <STR_LIT> ) ; replaceCall ( nval ) ; return true ; } } else if ( ci_opr1 == <NUM_LIT> ) { Module * M = CI -> getModule ( ) ; if ( FunctionCallee FPExpr = getFunction ( M , <ISA_LIT>LibFunc ( <ISA_LIT>LibFunc :: EI_CBRT , FInfo ) ) ) { LLVM_DEBUG ( errs ( ) << <STR_LIT> << * CI << <STR_LIT> << * opr0 << <STR_LIT> ) ; Value * nval = CreateCallEx ( B , FPExpr , opr0 , <STR_LIT> ) ; replaceCall ( nval ) ; return true ; } } else if ( ci_opr1 == - <NUM_LIT> ) { LLVM_DEBUG ( errs ( ) << <STR_LIT> << * CI << <STR_LIT> << * opr0 << <STR_LIT> ) ; Value * nval = B . CreateFDiv ( ConstantFP :: get ( opr0 -> getType ( ) , <NUM_LIT> ) , opr0 , <STR_LIT> ) ; replaceCall ( nval ) ; return true ; } else if ( ci_opr1 == - <NUM_LIT> ) { Module * M = CI -> getModule ( ) ; if ( FunctionCallee FPExpr = getFunction ( M , <ISA_LIT>LibFunc ( <ISA_LIT>LibFunc :: EI_RSQRT , FInfo ) ) ) { LLVM_DEBUG ( errs ( ) << <STR_LIT> << * CI << <STR_LIT> << * opr0 << <STR_LIT> ) ; Value * nval = CreateCallEx ( B , FPExpr , opr0 , <STR_LIT> ) ; replaceCall ( nval ) ; return true ; } }


<s> if ( FunctionCallee FPExpr = getNativeFunction ( CI -> getModule ( ) , <ISA_LIT>LibFunc ( <ISA_LIT>LibFunc :: EI_SQRT , FInfo ) ) ) { Value * opr0 = CI -> getArgOperand ( <NUM_LIT> ) ; LLVM_DEBUG ( errs ( ) << <STR_LIT> << * CI << <STR_LIT> << <STR_LIT> << * opr0 << <STR_LIT> ) ;

Value * nval = CreateCallEx ( B , FPExpr , opr0 , <STR_LIT> ) ;


<s> if ( ( CPU . empty ( ) || CPU . equals_insensitive ( <STR_LIT> ) ) && ( Features . empty ( ) || ! Features . contains_insensitive ( <STR_LIT> ) ) ) return false ;

Function * F = CI -> getParent ( ) -> getParent ( ) ;


<s> Function * F = CI -> getParent ( ) -> getParent ( ) ; const GCNSubtarget & ST = TM -> getSubtarget < GCNSubtarget > ( * F ) ; unsigned N = ST . getWavefrontSize ( ) ; LLVM_DEBUG ( errs ( ) << <STR_LIT> << * CI << <STR_LIT> << N << <STR_LIT> ) ; CI -> replaceAllUsesWith ( ConstantInt :: get ( B . getInt32Ty ( ) , N ) ) ; CI -> eraseFromParent ( ) ;


<s> void getAnalysisUsage ( AnalysisUsage & AU ) const override {

AU . addRequired < AAResultsWrapperPass > ( ) ;


<s> void getAnalysisUsage ( AnalysisUsage & AU ) const override { AU . addRequired < AAResultsWrapperPass > ( ) ;


<s> static inline <ISA_LIT>LibFunc :: EType getArgType ( const <ISA_LIT>LibFunc & FInfo ) {

return ( <ISA_LIT>LibFunc :: EType ) FInfo . getLeads ( ) [ <NUM_LIT> ] . ArgType ;


<s> static inline <ISA_LIT>LibFunc :: EType getArgType ( const <ISA_LIT>LibFunc & FInfo ) { return ( <ISA_LIT>LibFunc :: EType ) FInfo . getLeads ( ) [ <NUM_LIT> ] . ArgType ;


<s> assert ( BB && <STR_LIT> ) ; BasicBlock :: iterator ItNew = BB -> begin ( ) ;


<s> BasicBlock :: iterator <ISA_LIT>LibCalls :: getEntryIns ( CallInst * UI ) {

Function * Func = UI -> getParent ( ) -> getParent ( ) ;


<s> FunctionCallee <ISA_LIT>LibCalls :: getFunction ( Module * M , const FuncInfo & fInfo ) { return EnablePreLink ? <ISA_LIT>LibFunc :: getOrInsertFunction ( M , fInfo ) : <ISA_LIT>LibFunc :: getFunction ( M , fInfo ) ;


<s> FunctionCallee <ISA_LIT>LibCalls :: getNativeFunction ( Module * M , const FuncInfo & FInfo ) { if ( getArgType ( FInfo ) == <ISA_LIT>LibFunc :: F64 || ! HasNative ( FInfo . getId ( ) ) ) return nullptr ; FuncInfo nf = FInfo ; nf . setPrefix ( <ISA_LIT>LibFunc :: NATIVE ) ;


<s> case <ISA_LIT>LibFunc :: EI_ATAN : return TableRef ( tbl_atan ) ; case <ISA_LIT>LibFunc :: EI_ATANH : return TableRef ( tbl_atanh ) ; case <ISA_LIT>LibFunc :: EI_ATANPI : return TableRef ( tbl_atanpi ) ; case <ISA_LIT>LibFunc :: EI_CBRT : return TableRef ( tbl_cbrt ) ; case <ISA_LIT>LibFunc :: EI_NCOS : case <ISA_LIT>LibFunc :: EI_COS : return TableRef ( tbl_cos ) ; case <ISA_LIT>LibFunc :: EI_COSH : return TableRef ( tbl_cosh ) ; case <ISA_LIT>LibFunc :: EI_COSPI : return TableRef ( tbl_cospi ) ; case <ISA_LIT>LibFunc :: EI_ERFC : return TableRef ( tbl_erfc ) ; case <ISA_LIT>LibFunc :: EI_ERF : return TableRef ( tbl_erf ) ; case <ISA_LIT>LibFunc :: EI_EXP : return TableRef ( tbl_exp ) ; case <ISA_LIT>LibFunc :: EI_NEXP2 :

case <ISA_LIT>LibFunc :: EI_EXP2 : return TableRef ( tbl_exp2 ) ;


<s> return FInfo . getLeads ( ) [ <NUM_LIT> ] . VectorSize ;


<s> AllNative = useNativeFunc ( <STR_LIT> ) || ( UseNative . getNumOccurrences ( ) && UseNative . size ( ) == <NUM_LIT> && UseNative . begin ( ) -> empty ( ) ) ;


<s> BasicBlock :: iterator ItNew = getEntryIns ( UI ) ; Function * UCallee = UI -> getCalledFunction ( ) ; Type * RetType = UCallee -> getReturnType ( ) ;


<s> AllocaInst * <ISA_LIT>LibCalls :: insertAlloca ( CallInst * UI , IRBuilder < > & B , const char * prefix ) { BasicBlock :: iterator ItNew = getEntryIns ( UI ) ; Function * UCallee = UI -> getCalledFunction ( ) ;

Type * RetType = UCallee -> getReturnType ( ) ;


<s> bool <ISA_LIT>LibCalls :: isUnsafeMath ( const CallInst * CI ) const { if ( auto Op = dyn_cast < FPMathOperator > ( CI ) ) if ( Op -> isFast ( ) ) return true ; const Function * F = CI -> getParent ( ) -> getParent ( ) ;

Attribute Attr = F -> getFnAttribute ( <STR_LIT> ) ;


<s> return <ISA_LIT>LibFunc :: parse ( FMangledName , FInfo ) ;


<s> if ( getArgType ( FInfo ) != <ISA_LIT>LibFunc :: F32 || FInfo . getPrefix ( ) != <ISA_LIT>LibFunc :: NOPFX || ! HasNative ( FInfo . getId ( ) ) ) return false ; <ISA_LIT>LibFunc nf = FInfo ; nf . setPrefix ( <ISA_LIT>LibFunc :: NATIVE ) ; if ( FunctionCallee FPExpr = getFunction ( M , nf ) ) { LLVM_DEBUG ( dbgs ( ) << <STR_LIT> << * CI << <STR_LIT> ) ; CI -> setCalledFunction ( FPExpr ) ; LLVM_DEBUG ( dbgs ( ) << * CI << '\n' ) ; return true ; } return false ;


<s> Function * Callee = CI -> getCalledFunction ( ) ; if ( Callee == <NUM_LIT> ) continue ; if ( Simplifier . useNative ( CI ) ) Changed = true ; } } return Changed ? PreservedAnalyses :: none ( ) : PreservedAnalyses :: all ( ) ;


<s> if ( UseNative . empty ( ) ) return PreservedAnalyses :: all ( ) ; <ISA_LIT>LibCalls Simplifier ; Simplifier . initNativeFuncs ( ) ; bool Changed = false ; for ( auto & BB : F ) { for ( BasicBlock :: iterator I = BB . begin ( ) , E = BB . end ( ) ; I != E ; ) { CallInst * CI = dyn_cast < CallInst > ( I ) ; ++ I ; if ( ! CI ) continue ; Function * Callee = CI -> getCalledFunction ( ) ; if ( Callee == <NUM_LIT> ) continue ; if ( Simplifier . useNative ( CI ) ) Changed = true ; } }

return Changed ? PreservedAnalyses :: none ( ) : PreservedAnalyses :: all ( ) ;


<s> ++ I ; if ( ! CI ) continue ; Function * Callee = CI -> getCalledFunction ( ) ; if ( Callee == <NUM_LIT> ) continue ; if ( Simplifier . useNative ( CI ) ) Changed = true ; }


<s> bool <ISA_LIT>LibCalls :: sincosUseNative ( CallInst * aCI , const FuncInfo & FInfo ) { bool native_sin = useNativeFunc ( <STR_LIT> ) ; bool native_cos = useNativeFunc ( <STR_LIT> ) ; if ( native_sin && native_cos ) { Module * M = aCI -> getModule ( ) ; Value * opr0 = aCI -> getArgOperand ( <NUM_LIT> ) ; <ISA_LIT>LibFunc nf ; nf . getLeads ( ) [ <NUM_LIT> ] . ArgType = FInfo . getLeads ( ) [ <NUM_LIT> ] . ArgType ; nf . getLeads ( ) [ <NUM_LIT> ] . VectorSize = FInfo . getLeads ( ) [ <NUM_LIT> ] . VectorSize ; nf . setPrefix ( <ISA_LIT>LibFunc :: NATIVE ) ; nf . setId ( <ISA_LIT>LibFunc :: EI_<ISA_LIT>N ) ; FunctionCallee sinExpr = getFunction ( M , nf ) ; nf . setPrefix ( <ISA_LIT>LibFunc :: NATIVE ) ;

nf . setId ( <ISA_LIT>LibFunc :: EI_COS ) ;


<s> TableRef ( const TableEntry ( & tbl ) [ N ] ) : size ( N ) , table ( & tbl [ <NUM_LIT> ] ) {


<s> SmallVector < double , <NUM_LIT> > DVal ; for ( int eltNo = <NUM_LIT> ; eltNo < getVecSize ( FInfo ) ; ++ eltNo ) { ConstantFP * eltval = dyn_cast < ConstantFP > ( CV -> getElementAsConstant ( ( unsigned ) eltNo ) ) ; assert ( eltval && <STR_LIT> ) ; bool found = false ; for ( int i = <NUM_LIT> ; i < sz ; ++ i ) { if ( eltval -> isExactlyValue ( ftbl [ i ] . input ) ) { DVal . push_back ( ftbl [ i ] . result ) ; found = true ; break ; } } if ( ! found ) { return false ; } } LLVMContext & context = CI -> getParent ( ) -> getParent ( ) -> getContext ( ) ; Constant * nval ; if ( getArgType ( FInfo ) == <ISA_LIT>LibFunc :: F32 ) { SmallVector < float , <NUM_LIT> > FVal ; for ( unsigned i = <NUM_LIT> ; i < DVal . size ( ) ; ++ i ) { FVal . push_back ( ( float ) DVal [ i ] ) ; } ArrayRef < float > tmp ( FVal ) ; nval = ConstantDataVector :: get ( context , tmp ) ; } else { ArrayRef < double > tmp ( DVal ) ; nval = ConstantDataVector :: get ( context , tmp ) ; } LLVM_DEBUG ( errs ( ) << <STR_LIT> << * CI << <STR_LIT> << * nval << <STR_LIT> ) ; replaceCall ( nval ) ; return true ; } } else { if ( ConstantFP * CF = dyn_cast < ConstantFP > ( opr0 ) ) { for ( int i = <NUM_LIT> ; i < sz ; ++ i ) { if ( CF -> isExactlyValue ( ftbl [ i ] . input ) ) { Value * nval = ConstantFP :: get ( CF -> getType ( ) , ftbl [ i ] . result ) ; LLVM_DEBUG ( errs ( ) << <STR_LIT> << * CI << <STR_LIT> << * nval << <STR_LIT> ) ; replaceCall ( nval ) ; return true ; } } }


<s> } if ( FInfo . getId ( ) == <ISA_LIT>LibFunc :: EI_<ISA_LIT>NCOS ) return sincosUseNative ( aCI , FInfo ) ; FInfo . setPrefix ( <ISA_LIT>LibFunc :: NATIVE ) ; FunctionCallee F = getFunction ( aCI -> getModule ( ) , FInfo ) ; if ( ! F ) return false ; aCI -> setCalledFunction ( F ) ; DEBUG_WITH_TYPE ( <STR_LIT> , dbgs ( ) << <STR_LIT> << * aCI << <STR_LIT> ) ; return true ;


<s> bool <ISA_LIT>LibCalls :: useNativeFunc ( const StringRef F ) const {

return AllNative || llvm :: is_contained ( UseNative , F ) ;


<s> explicit <ISA_LIT>LibFunc ( ) : Impl ( std :: unique_ptr < <ISA_LIT>LibFuncImpl > ( ) ) {


<s> <ISA_LIT>MangledLibFunc :: <ISA_LIT>MangledLibFunc ( EFuncId id , const <ISA_LIT>MangledLibFunc & copyFrom ) { FuncId = id ; FKind = copyFrom . FKind ; Leads [ <NUM_LIT> ] = copyFrom . Leads [ <NUM_LIT> ] ; Leads [ <NUM_LIT> ] = copyFrom . Leads [ <NUM_LIT> ] ;


<s> StringMap < int > Map ( array_lengthof ( manglingRules ) ) ; int Id = <NUM_LIT> ; for ( auto Rule : manglingRules ) Map . insert ( { Rule . Name , Id ++ } ) ;


for ( unsigned I = <NUM_LIT> ; I != TableSize ; ++ I ) Map [ Table [ I ] . Name ] = I ;


<s> inline static void drop_front ( StringRef & str , size_t n = <NUM_LIT> ) { str = str . drop_front ( n ) ;


<s> StringRef Res = mangledName . substr ( <NUM_LIT> , Len ) ; drop_front ( mangledName , Len ) ; return Res ;


<s> static StringRef eatLengthPrefixedName ( StringRef & mangledName ) { int const Len = eatNumber ( mangledName ) ; if ( Len <= <NUM_LIT> || static_cast < size_t > ( Len ) > mangledName . size ( ) ) return StringRef ( ) ; StringRef Res = mangledName . substr ( <NUM_LIT> , Len ) ; drop_front ( mangledName , Len ) ;


<s> int n = <NUM_LIT> ; while ( ! s . empty ( ) && isDigit ( s . front ( ) ) ) { n = n * <NUM_LIT> + s . front ( ) - '0' ; drop_front ( s ) ; }

return s . size ( ) < savedSize ? n : - <NUM_LIT> ;


<s> drop_front ( mangledName , N - <NUM_LIT> ) ; return true ; }


<s> int findSubst ( const <ISA_LIT>LibFunc :: Param & P ) const {

for ( unsigned I = <NUM_LIT> ; I < Str . size ( ) ; ++ I ) {


<s> Kind = Kind & ADDR_SPACE ; assert ( Kind >= <NUM_LIT> ) ;


<s> static unsigned getAddrSpaceFromEPtrKind ( unsigned Kind ) { Kind = Kind & ADDR_SPACE ; assert ( Kind >= <NUM_LIT> ) ;


<s> assert ( ( ( AS + <NUM_LIT> ) & ~ ADDR_SPACE ) == <NUM_LIT> ) ;


<s> static unsigned getEPtrKindFromAddrSpace ( unsigned AS ) {

assert ( ( ( AS + <NUM_LIT> ) & ~ ADDR_SPACE ) == <NUM_LIT> ) ;


<s> std :: string FuncName = fInfo . mangle ( ) ; Function * F = dyn_cast_or_null < Function > ( M -> getValueSymbolTable ( ) . lookup ( FuncName ) ) ; if ( F && ! F -> isDeclaration ( ) && ! F -> isVarArg ( ) && F -> arg_size ( ) == fInfo . getNumArgs ( ) ) {


<s> case <ISA_LIT>LibFunc :: F32 : T = Type :: getFloatTy ( C ) ; break ; case <ISA_LIT>LibFunc :: F64 : T = Type :: getDoubleTy ( C ) ; break ; case <ISA_LIT>LibFunc :: IMG1DA : case <ISA_LIT>LibFunc :: IMG1DB : case <ISA_LIT>LibFunc :: IMG2DA : case <ISA_LIT>LibFunc :: IMG1D : case <ISA_LIT>LibFunc :: IMG2D : case <ISA_LIT>LibFunc :: IMG3D : T = StructType :: create ( C , <STR_LIT> ) -> getPointerTo ( ) ; break ; case <ISA_LIT>LibFunc :: SAMPLER : T = StructType :: create ( C , <STR_LIT> ) -> getPointerTo ( ) ; break ;


<s> const <ISA_LIT>LibFunc :: Param * <ISA_LIT>LibFunc :: getLeads ( ) const { return cast < const <ISA_LIT>MangledLibFunc > ( Impl . get ( ) ) -> Leads ;


<s> const <ISA_LIT>LibFunc :: Param * <ISA_LIT>LibFunc :: getLeads ( ) const {

return cast < const <ISA_LIT>MangledLibFunc > ( Impl . get ( ) ) -> Leads ;


<s> case EX_FLOAT4 : P . ArgType = <ISA_LIT>LibFunc :: F32 ; P . VectorSize = <NUM_LIT> ; break ; case EX_<ISA_LIT>ZET : P . ArgType = <ISA_LIT>LibFunc :: U64 ; break ; case EX_EVENT : P . ArgType = <ISA_LIT>LibFunc :: EVENT ; break ; case EX_SAMPLER : P . ArgType = <ISA_LIT>LibFunc :: SAMPLER ; break ; case EX_RESERVEDID : break ; default : if ( Index == ( Rule . Lead [ <NUM_LIT> ] - <NUM_LIT> ) ) P = Leads [ <NUM_LIT> ] ; else P = Leads [ <NUM_LIT> ] ; switch ( R ) { case E_ANY : case E_COPY : break ; case E_POINTEE : P . PtrKind = <ISA_LIT>LibFunc :: BYVALUE ; break ; case E_V2_OF_POINTEE : P . VectorSize = <NUM_LIT> ; P . PtrKind = <ISA_LIT>LibFunc :: BYVALUE ; break ; case E_V3_OF_POINTEE : P . VectorSize = <NUM_LIT> ; P . PtrKind = <ISA_LIT>LibFunc :: BYVALUE ; break ; case E_V4_OF_POINTEE : P . VectorSize = <NUM_LIT> ; P . PtrKind = <ISA_LIT>LibFunc :: BYVALUE ; break ; case E_V8_OF_POINTEE : P . VectorSize = <NUM_LIT> ; P . PtrKind = <ISA_LIT>LibFunc :: BYVALUE ; break ; case E_V16_OF_POINTEE : P . VectorSize = <NUM_LIT> ; P . PtrKind = <ISA_LIT>LibFunc :: BYVALUE ; break ; case E_CONSTPTR_ANY : P . PtrKind |= <ISA_LIT>LibFunc :: CONST ; break ; case E_VLTLPTR_ANY : P . PtrKind |= <ISA_LIT>LibFunc :: VOLATILE ; break ; case E_SETBASE_I32 : P . ArgType = <ISA_LIT>LibFunc :: I32 ; break ; case E_SETBASE_U32 :

P . ArgType = <ISA_LIT>LibFunc :: U32 ; break ;


<s> unsigned getNumArgs ( ) const { return Impl -> getNumArgs ( ) ;


return ( Lead [ <NUM_LIT> ] ? <NUM_LIT> : <NUM_LIT> ) + ( Lead [ <NUM_LIT> ] ? <NUM_LIT> : <NUM_LIT> ) ;


<s> int getNumLeads ( ) const { return ( Lead [ <NUM_LIT> ] ? <NUM_LIT> : <NUM_LIT> ) + ( Lead [ <NUM_LIT> ] ? <NUM_LIT> : <NUM_LIT> ) ;


<s> Function * F = dyn_cast_or_null < Function > ( M -> getValueSymbolTable ( ) . lookup ( FuncName ) ) ; if ( F && ! F -> isDeclaration ( ) && ! F -> isVarArg ( ) && F -> arg_size ( ) == fInfo . getNumArgs ( ) ) { return F ; } FunctionType * FuncTy = fInfo . getFunctionType ( * M ) ;


<s> if ( eatTerm ( S , <STR_LIT> ) ) return eatLengthPrefixedName ( S ) ; return StringRef ( ) ;


<s> StringRef S = mangledName ; if ( eatTerm ( S , <STR_LIT> ) ) return eatLengthPrefixedName ( S ) ; return StringRef ( ) ;


<s> bool isMangled ( ) const { return Impl -> isMangled ( ) ;


<s> ItaniumMangler ( bool useAddrSpace ) : UseAddrSpace ( useAddrSpace ) {


<s> if ( Loc != Map . end ( ) ) { Id = toFuncId ( Loc -> second ) ; return true ; } Id = <ISA_LIT>LibFunc :: EI_NONE ; return false ;


<s> std :: string mangle ( ) const override {


<s> ParamIterator I ( Leads , manglingRules [ FuncId ] ) ; Param P ; while ( ( P = I . getNextParam ( ) ) . ArgType != <NUM_LIT> ) Mangler ( S , P ) ; return std :: string ( S . str ( ) ) ;


<s> return ( std :: max ) ( Lead [ <NUM_LIT> ] , Lead [ <NUM_LIT> ] ) ;


return ( std :: max ) ( Lead [ <NUM_LIT> ] , Lead [ <NUM_LIT> ] ) ;


<s> ParamIterator ( const <ISA_LIT>LibFunc :: Param ( & leads ) [ <NUM_LIT> ] , const ManglingRule & rule ) : Leads ( leads ) , Rule ( rule ) , Index ( <NUM_LIT> ) {


<s> return false ; } if ( eatTerm ( FuncName , <STR_LIT> ) ) F . Impl = std :: make_unique < <ISA_LIT>MangledLibFunc > ( ) ;

else F . Impl = std :: make_unique < <ISA_LIT>UnmangledLibFunc > ( ) ;


<s> F . Impl = std :: unique_ptr < <ISA_LIT>LibFuncImpl > ( ) ; return false ; } if ( eatTerm ( FuncName , <STR_LIT> ) ) F . Impl = std :: make_unique < <ISA_LIT>MangledLibFunc > ( ) ;


<s> return Impl -> parseFuncName ( MangledName ) ;


<s> res . VectorSize = parseVecSize ( param ) ; if ( res . VectorSize == <NUM_LIT> || ! eatTerm ( param , '_' ) ) return false ; } char const TC = param . front ( ) ; if ( isDigit ( TC ) ) { res . ArgType = StringSwitch < <ISA_LIT>LibFunc :: EType > ( eatLengthPrefixedName ( param ) ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG1DA ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG1DB ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG2DA ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG1D ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG2D ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG3D ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: DUMMY ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: DUMMY ) . Default ( <ISA_LIT>LibFunc :: DUMMY ) ; } else { drop_front ( param ) ; switch ( TC ) { case 'h' : res . ArgType = <ISA_LIT>LibFunc :: U8 ; break ; case 't' : res . ArgType = <ISA_LIT>LibFunc :: U16 ; break ; case 'j' : res . ArgType = <ISA_LIT>LibFunc :: U32 ; break ; case 'm' : res . ArgType = <ISA_LIT>LibFunc :: U64 ; break ; case 'c' : res . ArgType = <ISA_LIT>LibFunc :: I8 ; break ; case 's' : res . ArgType = <ISA_LIT>LibFunc :: I16 ; break ; case 'i' : res . ArgType = <ISA_LIT>LibFunc :: I32 ; break ; case 'l' : res . ArgType = <ISA_LIT>LibFunc :: I64 ; break ; case 'f' : res . ArgType = <ISA_LIT>LibFunc :: F32 ; break ; case 'd' : res . ArgType = <ISA_LIT>LibFunc :: F64 ; break ; case 'D' : if ( ! eatTerm ( param , 'h' ) ) return false ; res . ArgType = <ISA_LIT>LibFunc :: F16 ; break ; case 'S' : if ( ! eatTerm ( param , '_' ) ) { eatNumber ( param ) ; if ( ! eatTerm ( param , '_' ) ) return false ; } res . VectorSize = Prev . VectorSize ; res . ArgType = Prev . ArgType ; break ; default : ; } }

if ( res . ArgType == <NUM_LIT> ) return false ;


<s> static <ISA_LIT>LibFunc :: ENamePrefix parseNamePrefix ( StringRef & mangledName ) { std :: pair < StringRef , StringRef > const P = mangledName . split ( '_' ) ;

<ISA_LIT>LibFunc :: ENamePrefix Pfx = StringSwitch < <ISA_LIT>LibFunc :: ENamePrefix > ( P . first ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: NATIVE ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: HALF ) . Default ( <ISA_LIT>LibFunc :: NOPFX ) ;


<s> FuncId = static_cast < EFuncId > ( manglingRulesMap . lookup ( FullName ) ) ; return FuncId != EI_NONE ;


<s> switch ( Len ) { case <NUM_LIT> : case <NUM_LIT> : case <NUM_LIT> : case <NUM_LIT> : case <NUM_LIT> : return Len ; default : break ;


<s> void setFunctionType ( FunctionType * FT ) { FuncTy = FT ;


<s> void setId ( EFuncId Id ) { Impl -> setId ( Id ) ;


<s> void setName ( StringRef N ) { Impl -> setName ( N ) ;


<s> void setPrefix ( ENamePrefix PFX ) { Impl -> setPrefix ( PFX ) ;


<s> static ID toFuncId ( unsigned Index ) { assert ( Index < TableSize && <STR_LIT> ) ;

return static_cast < ID > ( Index + <NUM_LIT> + static_cast < unsigned > ( <ISA_LIT>LibFunc :: EI_LAST_MANGLED ) ) ;


<s> return static_cast < ID > ( Index + <NUM_LIT> + static_cast < unsigned > ( <ISA_LIT>LibFunc :: EI_LAST_MANGLED ) ) ;


<s> return static_cast < unsigned > ( Id ) - <NUM_LIT> - static_cast < unsigned > ( <ISA_LIT>LibFunc :: EI_LAST_MANGLED ) ;


assert ( static_cast < unsigned > ( Id ) > static_cast < unsigned > ( <ISA_LIT>LibFunc :: EI_LAST_MANGLED ) && <STR_LIT> ) ;


<s> bool trySubst ( Stream & os , const <ISA_LIT>LibFunc :: Param & p ) { int const subst = findSubst ( p ) ; if ( subst < <NUM_LIT> ) return false ;

if ( subst == <NUM_LIT> ) os << <STR_LIT> ;


<s> if ( subst < <NUM_LIT> ) return false ; if ( subst == <NUM_LIT> ) os << <STR_LIT> ; else os << 'S' << ( subst - <NUM_LIT> ) << '_' ; return true ;


<s> constexpr UnmangledFuncInfo ( const char * _Name , unsigned _NumArgs ) : Name ( _Name ) , NumArgs ( _NumArgs ) {


<s> const char * Pfx = <STR_LIT> ; switch ( FKind ) { case NATIVE : Pfx = <STR_LIT> ; break ; case HALF : Pfx = <STR_LIT> ; break ; default : break ; } if ( ! Name . empty ( ) ) {


<s> virtual ~ <ISA_LIT>LibFuncImpl ( ) {


<s> case <ISA_LIT>LibFunc :: IMG2D : case <ISA_LIT>LibFunc :: IMG3D : T = StructType :: create ( C , <STR_LIT> ) -> getPointerTo ( ) ; break ; case <ISA_LIT>LibFunc :: SAMPLER : T = StructType :: create ( C , <STR_LIT> ) -> getPointerTo ( ) ; break ; case <ISA_LIT>LibFunc :: EVENT : T = StructType :: create ( C , <STR_LIT> ) -> getPointerTo ( ) ; break ; default : llvm_unreachable ( <STR_LIT> ) ; return nullptr ; } if ( P . VectorSize > <NUM_LIT> ) T = FixedVectorType :: get ( T , P . VectorSize ) ;

if ( P . PtrKind != <ISA_LIT>LibFunc :: BYVALUE ) T = useAddrSpace ? T -> getPointerTo ( ( P . PtrKind & <ISA_LIT>LibFunc :: ADDR_SPACE ) - <NUM_LIT> ) : T -> getPointerTo ( ) ;


<s> C = M -> getOrInsertFunction ( FuncName , FuncTy ) ; } else { AttributeList Attr ; LLVMContext & Ctx = M -> getContext ( ) ; Attr . addAttribute ( Ctx , AttributeList :: FunctionIndex , Attribute :: ReadOnly ) ; Attr . addAttribute ( Ctx , AttributeList :: FunctionIndex , Attribute :: NoUnwind ) ; C = M -> getOrInsertFunction ( FuncName , FuncTy , Attr ) ; }


<s> } } Constant * C = nullptr ; if ( hasPtr ) { C = M -> getOrInsertFunction ( FuncName , FuncTy ) ; } else { AttributeList Attr ; LLVMContext & Ctx = M -> getContext ( ) ; Attr . addAttribute ( Ctx , AttributeList :: FunctionIndex , Attribute :: ReadOnly ) ; Attr . addAttribute ( Ctx , AttributeList :: FunctionIndex , Attribute :: NoUnwind ) ; C = M -> getOrInsertFunction ( FuncName , FuncTy , Attr ) ; }


<s> std :: string const FuncName = fInfo . mangle ( ) ; Function * F = dyn_cast_or_null < Function > ( M -> getValueSymbolTable ( ) . lookup ( FuncName ) ) ; if ( F && ! F -> isDeclaration ( ) && ! F -> isVarArg ( ) && F -> arg_size ( ) == fInfo . getNumArgs ( ) ) { return F ; } FunctionType * FuncTy = fInfo . getFunctionType ( * M ) ; bool hasPtr = false ; for ( FunctionType :: param_iterator PI = FuncTy -> param_begin ( ) , PE = FuncTy -> param_end ( ) ; PI != PE ; ++ PI ) { const Type * argTy = static_cast < const Type * > ( * PI ) ; if ( argTy -> isPointerTy ( ) ) { hasPtr = true ; break ; } } Constant * C = nullptr ; if ( hasPtr ) { C = M -> getOrInsertFunction ( FuncName , FuncTy ) ; } else { AttributeList Attr ; LLVMContext & Ctx = M -> getContext ( ) ; Attr = Attr . addAttribute ( Ctx , AttributeList :: FunctionIndex , Attribute :: ReadOnly ) ;

Attr = Attr . addAttribute ( Ctx , AttributeList :: FunctionIndex , Attribute :: NoUnwind ) ;


<s> auto Loc = Map . find ( Name ) ; if ( Loc != Map . end ( ) ) { Id = toFuncId ( Loc -> second ) ; return true ; } Id = <ISA_LIT>LibFunc :: EI_NONE ;


<s> for ( auto Rule : manglingRules ) insert ( { Rule . Name , Id ++ } ) ;


<s> ManglingRulesMap ( ) : StringMap < int > ( sizeof ( manglingRules ) / sizeof ( manglingRules [ <NUM_LIT> ] ) ) {


<s> for ( unsigned I = <NUM_LIT> ; I != TableSize ; ++ I ) ( * this ) [ Table [ I ] . Name ] = I ;


<s> FuncId = static_cast < EFuncId > ( manglingRulesMap . lookup ( FullName ) ) ;


<s> explicit <ISA_LIT>UnmangledLibFunc ( StringRef FName , FunctionType * FT ) { Name = std :: string ( FName ) ; FuncTy = FT ;


<s> P . VectorSize = <NUM_LIT> ; P . PtrKind = <ISA_LIT>LibFunc :: BYVALUE ; break ; case E_CONSTPTR_ANY : P . PtrKind |= <ISA_LIT>LibFunc :: CONST ; break ; case E_VLTLPTR_ANY : P . PtrKind |= <ISA_LIT>LibFunc :: VOLATILE ; break ; case E_SETBASE_I32 : P . ArgType = <ISA_LIT>LibFunc :: I32 ; break ; case E_SETBASE_U32 : P . ArgType = <ISA_LIT>LibFunc :: U32 ; break ; case E_MAKEBASE_UNS : P . ArgType &= ~ <ISA_LIT>LibFunc :: BASE_TYPE_MASK ; P . ArgType |= <ISA_LIT>LibFunc :: UINT ; break ; case E_IMAGECOORDS : switch ( P . ArgType ) { case <ISA_LIT>LibFunc :: IMG1DA : P . VectorSize = <NUM_LIT> ; break ; case <ISA_LIT>LibFunc :: IMG1DB : P . VectorSize = <NUM_LIT> ; break ; case <ISA_LIT>LibFunc :: IMG2DA : P . VectorSize = <NUM_LIT> ; break ; case <ISA_LIT>LibFunc :: IMG1D : P . VectorSize = <NUM_LIT> ; break ; case <ISA_LIT>LibFunc :: IMG2D : P . VectorSize = <NUM_LIT> ; break ; case <ISA_LIT>LibFunc :: IMG3D : P . VectorSize = <NUM_LIT> ; break ; } P . PtrKind = <ISA_LIT>LibFunc :: BYVALUE ; P . ArgType = <ISA_LIT>LibFunc :: I32 ; break ; case E_CONSTPTR_SWAPGL : switch ( P . PtrKind & <ISA_LIT>LibFunc :: ADDR_SPACE ) { case <ISA_LIT>LibFunc :: GLOBAL : P . PtrKind = <ISA_LIT>LibFunc :: LOCAL ; break ; case <ISA_LIT>LibFunc :: LOCAL : P . PtrKind = <ISA_LIT>LibFunc :: GLOBAL ; break ; } P . PtrKind |= <ISA_LIT>LibFunc :: CONST ; break ; default : llvm_unreachable ( <STR_LIT> ) ;


<s> } else { switch ( param . front ( ) ) { case '1' : res . PtrKind |= <ISA_LIT>LibFunc :: GLOBAL ; break ; case '2' : res . PtrKind |= <ISA_LIT>LibFunc :: READONLY ; break ; case '3' : res . PtrKind |= <ISA_LIT>LibFunc :: LOCAL ; break ; case '4' : res . PtrKind |= <ISA_LIT>LibFunc :: GENERIC ; break ; case '5' : res . PtrKind |= <ISA_LIT>LibFunc :: OTHER ; break ; default : return false ; } drop_front ( param , <NUM_LIT> ) ; } } else { res . PtrKind = <ISA_LIT>LibFunc :: BYVALUE ; } if ( eatTerm ( param , <STR_LIT> ) ) { res . VectorSize = parseVecSize ( param ) ; if ( res . VectorSize == <NUM_LIT> || ! eatTerm ( param , '_' ) ) return false ; } char const TC = param . front ( ) ; if ( :: isDigit ( TC ) ) { res . ArgType = StringSwitch < <ISA_LIT>LibFunc :: EType > ( eatLengthPrefixedName ( param ) ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG1DA ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG1DB ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG2DA ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG1D ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG2D ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG3D ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: DUMMY ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: DUMMY ) . Default ( <ISA_LIT>LibFunc :: DUMMY ) ; } else { drop_front ( param ) ; switch ( TC ) { case 'h' : res . ArgType = <ISA_LIT>LibFunc :: U8 ; break ; case 't' : res . ArgType = <ISA_LIT>LibFunc :: U16 ; break ; case 'j' : res . ArgType = <ISA_LIT>LibFunc :: U32 ; break ; case 'm' : res . ArgType = <ISA_LIT>LibFunc :: U64 ; break ; case 'c' : res . ArgType = <ISA_LIT>LibFunc :: I8 ; break ; case 's' : res . ArgType = <ISA_LIT>LibFunc :: I16 ; break ; case 'i' : res . ArgType = <ISA_LIT>LibFunc :: I32 ; break ; case 'l' : res . ArgType = <ISA_LIT>LibFunc :: I64 ; break ; case 'f' : res . ArgType = <ISA_LIT>LibFunc :: F32 ; break ;

case 'd' : res . ArgType = <ISA_LIT>LibFunc :: F64 ; break ;


<s> case '2' : res . PtrKind |= <ISA_LIT>LibFunc :: READONLY ; break ; case '3' : res . PtrKind |= <ISA_LIT>LibFunc :: LOCAL ; break ; case '4' : res . PtrKind |= <ISA_LIT>LibFunc :: GENERIC ; break ; case '5' : res . PtrKind |= <ISA_LIT>LibFunc :: OTHER ; break ; default : return false ; } drop_front ( param , <NUM_LIT> ) ; } } else { res . PtrKind = <ISA_LIT>LibFunc :: BYVALUE ; } if ( eatTerm ( param , <STR_LIT> ) ) { res . VectorSize = parseVecSize ( param ) ; if ( res . VectorSize == <NUM_LIT> || ! eatTerm ( param , '_' ) ) return false ; } char const TC = param . front ( ) ; if ( :: isDigit ( TC ) ) { res . ArgType = StringSwitch < <ISA_LIT>LibFunc :: EType > ( eatLengthPrefixedName ( param ) ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG1DA ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG1DB ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG2DA ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG1D ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG2D ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG3D ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: DUMMY ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: DUMMY ) . Default ( <ISA_LIT>LibFunc :: DUMMY ) ; } else { drop_front ( param ) ; switch ( TC ) { case 'h' : res . ArgType = <ISA_LIT>LibFunc :: U8 ; break ; case 't' : res . ArgType = <ISA_LIT>LibFunc :: U16 ; break ; case 'j' : res . ArgType = <ISA_LIT>LibFunc :: U32 ; break ; case 'm' : res . ArgType = <ISA_LIT>LibFunc :: U64 ; break ; case 'c' : res . ArgType = <ISA_LIT>LibFunc :: I8 ; break ; case 's' : res . ArgType = <ISA_LIT>LibFunc :: I16 ; break ; case 'i' : res . ArgType = <ISA_LIT>LibFunc :: I32 ; break ; case 'l' : res . ArgType = <ISA_LIT>LibFunc :: I64 ; break ; case 'f' : res . ArgType = <ISA_LIT>LibFunc :: F32 ; break ; case 'd' : res . ArgType = <ISA_LIT>LibFunc :: F64 ; break ; case 'D' : if ( ! eatTerm ( param , 'h' ) ) return false ; res . ArgType = <ISA_LIT>LibFunc :: F16 ; break ; case 'S' : if ( ! eatTerm ( param , '_' ) ) { eatNumber ( param ) ; if ( ! eatTerm ( param , '_' ) ) return false ; } res . VectorSize = Prev . VectorSize ; res . ArgType = Prev . ArgType ; break ; default : ; } } if ( res . ArgType == <NUM_LIT> ) return false ; Prev . VectorSize = res . VectorSize ; Prev . ArgType = res . ArgType ; return true ;


<s> case <ISA_LIT>LibFunc :: U16 : case <ISA_LIT>LibFunc :: I16 : T = Type :: getInt16Ty ( C ) ; break ; case <ISA_LIT>LibFunc :: U32 : case <ISA_LIT>LibFunc :: I32 : T = Type :: getInt32Ty ( C ) ; break ; case <ISA_LIT>LibFunc :: U64 : case <ISA_LIT>LibFunc :: I64 : T = Type :: getInt64Ty ( C ) ; break ; case <ISA_LIT>LibFunc :: F16 : T = Type :: getHalfTy ( C ) ; break ; case <ISA_LIT>LibFunc :: F32 : T = Type :: getFloatTy ( C ) ; break ; case <ISA_LIT>LibFunc :: F64 : T = Type :: getDoubleTy ( C ) ; break ; case <ISA_LIT>LibFunc :: IMG1DA : case <ISA_LIT>LibFunc :: IMG1DB : case <ISA_LIT>LibFunc :: IMG2DA : case <ISA_LIT>LibFunc :: IMG1D : case <ISA_LIT>LibFunc :: IMG2D : case <ISA_LIT>LibFunc :: IMG3D : T = StructType :: create ( C , <STR_LIT> ) -> getPointerTo ( ) ; break ; case <ISA_LIT>LibFunc :: SAMPLER : T = StructType :: create ( C , <STR_LIT> ) -> getPointerTo ( ) ; break ; case <ISA_LIT>LibFunc :: EVENT : T = StructType :: create ( C , <STR_LIT> ) -> getPointerTo ( ) ; break ; default : llvm_unreachable ( <STR_LIT> ) ; return nullptr ; } if ( P . VectorSize > <NUM_LIT> ) T = VectorType :: get ( T , P . VectorSize ) ; if ( P . PtrKind != <ISA_LIT>LibFunc :: BYVALUE ) T = useAddrSpace ? T -> getPointerTo ( ( P . PtrKind & <ISA_LIT>LibFunc :: ADDR_SPACE ) - <NUM_LIT> ) : T -> getPointerTo ( ) ;


<s> std :: string <ISA_LIT>MangledLibFunc :: getName ( ) const {


<s> raw_svector_ostream OS ( Buf ) ; writeName ( OS ) ; return OS . str ( ) ;


<s> break ; } } FunctionCallee C ; if ( hasPtr ) { C = M -> getOrInsertFunction ( FuncName , FuncTy ) ; } else { AttributeList Attr ; LLVMContext & Ctx = M -> getContext ( ) ; Attr = Attr . addAttribute ( Ctx , AttributeList :: FunctionIndex , Attribute :: ReadOnly ) ; Attr = Attr . addAttribute ( Ctx , AttributeList :: FunctionIndex , Attribute :: NoUnwind ) ; C = M -> getOrInsertFunction ( FuncName , FuncTy , Attr ) ;


<s> } FunctionCallee C ; if ( hasPtr ) { C = M -> getOrInsertFunction ( FuncName , FuncTy ) ; } else { AttributeList Attr ; LLVMContext & Ctx = M -> getContext ( ) ; Attr = Attr . addAttribute ( Ctx , AttributeList :: FunctionIndex , Attribute :: ReadOnly ) ; Attr = Attr . addAttribute ( Ctx , AttributeList :: FunctionIndex , Attribute :: NoUnwind ) ;

C = M -> getOrInsertFunction ( FuncName , FuncTy , Attr ) ;


<s> ItaniumMangler Mangler ( true ) ; ParamIterator I ( Leads , manglingRules [ FuncId ] ) ; Param P ; while ( ( P = I . getNextParam ( ) ) . ArgType != <NUM_LIT> ) Mangler ( S , P ) ;


<s> if ( eatTerm ( FuncName , <STR_LIT> ) ) F . Impl = make_unique < <ISA_LIT>MangledLibFunc > ( ) ; else F . Impl = make_unique < <ISA_LIT>UnmangledLibFunc > ( ) ;

if ( F . Impl -> parseFuncName ( FuncName ) ) return true ;


<s> if ( eatTerm ( param , 'K' ) ) res . PtrKind |= <ISA_LIT>LibFunc :: CONST ; if ( eatTerm ( param , 'V' ) ) res . PtrKind |= <ISA_LIT>LibFunc :: VOLATILE ; unsigned AS ; if ( ! eatTerm ( param , <STR_LIT> ) ) { AS = <NUM_LIT> ; } else { AS = param . front ( ) - '0' ; drop_front ( param , <NUM_LIT> ) ; } res . PtrKind |= <STR_LIT> ( AS ) ; } else { res . PtrKind = <ISA_LIT>LibFunc :: BYVALUE ; } if ( eatTerm ( param , <STR_LIT> ) ) { res . VectorSize = parseVecSize ( param ) ; if ( res . VectorSize == <NUM_LIT> || ! eatTerm ( param , '_' ) ) return false ; } char const TC = param . front ( ) ; if ( :: isDigit ( TC ) ) { res . ArgType = StringSwitch < <ISA_LIT>LibFunc :: EType > ( eatLengthPrefixedName ( param ) ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG1DA ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG1DB ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG2DA ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG1D ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG2D ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG3D ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: DUMMY ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: DUMMY ) . Default ( <ISA_LIT>LibFunc :: DUMMY ) ; } else { drop_front ( param ) ; switch ( TC ) { case 'h' : res . ArgType = <ISA_LIT>LibFunc :: U8 ; break ;

case 't' : res . ArgType = <ISA_LIT>LibFunc :: U16 ; break ;


<s> if ( eatTerm ( param , 'V' ) ) res . PtrKind |= <ISA_LIT>LibFunc :: VOLATILE ; unsigned AS ; if ( ! eatTerm ( param , <STR_LIT> ) ) { AS = <NUM_LIT> ; } else { AS = param . front ( ) - '0' ; drop_front ( param , <NUM_LIT> ) ; } res . PtrKind |= <STR_LIT> ( AS ) ; } else { res . PtrKind = <ISA_LIT>LibFunc :: BYVALUE ; } if ( eatTerm ( param , <STR_LIT> ) ) { res . VectorSize = parseVecSize ( param ) ; if ( res . VectorSize == <NUM_LIT> || ! eatTerm ( param , '_' ) ) return false ; } char const TC = param . front ( ) ; if ( :: isDigit ( TC ) ) { res . ArgType = StringSwitch < <ISA_LIT>LibFunc :: EType > ( eatLengthPrefixedName ( param ) ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG1DA ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG1DB ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG2DA ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG1D ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG2D ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: IMG3D ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: DUMMY ) . Case ( <STR_LIT> , <ISA_LIT>LibFunc :: DUMMY ) . Default ( <ISA_LIT>LibFunc :: DUMMY ) ; } else { drop_front ( param ) ; switch ( TC ) { case 'h' : res . ArgType = <ISA_LIT>LibFunc :: U8 ; break ; case 't' : res . ArgType = <ISA_LIT>LibFunc :: U16 ; break ; case 'j' : res . ArgType = <ISA_LIT>LibFunc :: U32 ; break ; case 'm' : res . ArgType = <ISA_LIT>LibFunc :: U64 ; break ; case 'c' : res . ArgType = <ISA_LIT>LibFunc :: I8 ; break ; case 's' : res . ArgType = <ISA_LIT>LibFunc :: I16 ; break ; case 'i' : res . ArgType = <ISA_LIT>LibFunc :: I32 ; break ; case 'l' : res . ArgType = <ISA_LIT>LibFunc :: I64 ; break ; case 'f' : res . ArgType = <ISA_LIT>LibFunc :: F32 ; break ; case 'd' : res . ArgType = <ISA_LIT>LibFunc :: F64 ; break ; case 'D' : if ( ! eatTerm ( param , 'h' ) ) return false ; res . ArgType = <ISA_LIT>LibFunc :: F16 ; break ; case 'S' : if ( ! eatTerm ( param , '_' ) ) { eatNumber ( param ) ; if ( ! eatTerm ( param , '_' ) ) return false ; } res . VectorSize = Prev . VectorSize ; res . ArgType = Prev . ArgType ; break ; default : ; } } if ( res . ArgType == <NUM_LIT> ) return false ; Prev . VectorSize = res . VectorSize ;


<s> UnmangledFuncInfo ( StringRef _Name , unsigned _NumArgs ) : Name ( _Name ) , NumArgs ( _NumArgs ) {


<s> Function * ParentFunc = Memcpy -> getParent ( ) -> getParent ( ) ; const TargetTransformInfo & TTI = getAnalysis < TargetTransformInfoWrapperPass > ( ) . getTTI ( * ParentFunc ) ; expandMemCpyAsLoop ( Memcpy , TTI ) ; Changed = true ; Memcpy -> eraseFromParent ( ) ; } break ; } case <STR_LIT> : { auto * Memmove = cast < MemMoveInst > ( Inst ) ; if ( shouldExpandOperationWithSize ( Memmove -> getLength ( ) ) ) { expandMemMoveAsLoop ( Memmove ) ; Changed = true ;


<s> AU . addRequired < TargetTransformInfoWrapperPass > ( ) ;


<s> bool Changed = false ; for ( auto * U : F . users ( ) ) { auto * CI = dyn_cast < CallInst > ( U ) ; if ( ! CI ) continue ; Changed |= <ISA_LIT>CommonSubtarget :: get ( TM , F ) . makeLIDRangeMetadata ( CI ) ;


<s> <ISA_LIT>LowerIntrinsics ( const TargetMachine * TM = nullptr ) : ModulePass ( ID ) , TM ( TM ) {


<s> bool Changed = false ; const <ISA_LIT>Subtarget & ST = TM -> getSubtarget < <ISA_LIT>Subtarget > ( F ) ; for ( auto * U : F . users ( ) ) { auto * CI = dyn_cast < CallInst > ( U ) ; if ( ! CI ) continue ; Changed |= ST . makeLIDRangeMetadata ( CI ) ;


<s> bool <ISA_LIT>LowerIntrinsics :: makeLIDRangeMetadata ( Function & F ) const { auto * TPC = getAnalysisIfAvailable < TargetPassConfig > ( ) ; if ( ! TPC ) return false ; const TargetMachine & TM = TPC -> getTM < TargetMachine > ( ) ; bool Changed = false ; for ( auto * U : F . users ( ) ) { auto * CI = dyn_cast < CallInst > ( U ) ; if ( ! CI ) continue ; Changed |= <ISA_LIT>Subtarget :: get ( TM , F ) . makeLIDRangeMetadata ( CI ) ; }


<s> auto * CI = dyn_cast < CallInst > ( U ) ; if ( ! CI ) continue ; Changed |= <ISA_LIT>Subtarget :: get ( TM , F ) . makeLIDRangeMetadata ( CI ) ; } return Changed ;


<s> if ( ! TPC ) return false ; const TargetMachine & TM = TPC -> getTM < TargetMachine > ( ) ; bool Changed = false ; for ( auto * U : F . users ( ) ) { auto * CI = dyn_cast < CallInst > ( U ) ; if ( ! CI ) continue ; Function * Caller = CI -> getParent ( ) -> getParent ( ) ; const <ISA_LIT>Subtarget & ST = <ISA_LIT>Subtarget :: get ( TM , * Caller ) ; Changed |= ST . makeLIDRangeMetadata ( CI ) ; }


<s> const TargetTransformInfo & TTI = getAnalysis < TargetTransformInfoWrapperPass > ( ) . getTTI ( * ParentFunc ) ; expandMemCpyAsLoop ( Memcpy , TTI ) ; Changed = true ; Memcpy -> eraseFromParent ( ) ; } break ; } case <STR_LIT> : { auto * Memmove = cast < MemMoveInst > ( Inst ) ; if ( shouldExpandOperationWithSize ( Memmove -> getLength ( ) ) ) { expandMemMoveAsLoop ( Memmove ) ; Changed = true ; Memmove -> eraseFromParent ( ) ; } break ; } case <STR_LIT> : { auto * Memset = cast < MemSetInst > ( Inst ) ; if ( shouldExpandOperationWithSize ( Memset -> getLength ( ) ) ) { expandMemSetAsLoop ( Memset ) ; Changed = true ; Memset -> eraseFromParent ( ) ; } break ; }


<s> for ( Function & F : M ) { if ( ! F . isDeclaration ( ) ) continue ; switch ( F . getIntrinsicID ( ) ) { case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : if ( expandMemIntrinsicUses ( F ) ) Changed = true ; break ; default :


<s> switch ( F . getIntrinsicID ( ) ) { case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : if ( expandMemIntrinsicUses ( F ) ) Changed = true ; break ; case <STR_LIT> : case <STR_LIT> :


<s> bool Changed = false ; for ( Function & F : M ) { if ( ! F . isDeclaration ( ) ) continue ; switch ( F . getIntrinsicID ( ) ) { case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : if ( expandMemIntrinsicUses ( F ) ) Changed = true ; break ; case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : Changed |= makeLIDRangeMetadata ( F ) ;


<s> <ISA_LIT>LowerIntrinsics ( ) : ModulePass ( ID ) {


<s> ModulePass * llvm :: create<ISA_LIT>LowerIntrinsicsPass ( ) {

return new <ISA_LIT>LowerIntrinsics ( ) ;


<s> auto * Memcpy = cast < MemCpyInst > ( Inst ) ; if ( shouldExpandOperationWithSize ( Memcpy -> getLength ( ) ) ) { expandMemCpyAsLoop ( Memcpy ) ; Changed = true ; Memcpy -> eraseFromParent ( ) ; } break ; } case <STR_LIT> : { auto * Memmove = cast < MemMoveInst > ( Inst ) ; if ( shouldExpandOperationWithSize ( Memmove -> getLength ( ) ) ) { expandMemMoveAsLoop ( Memmove ) ; Changed = true ; Memmove -> eraseFromParent ( ) ;


<s> StringRef getPassName ( ) const override { return <STR_LIT> ;


<s> bool Changed = false ; for ( auto * U : F . users ( ) ) { auto * CI = dyn_cast < CallInst > ( U ) ; if ( ! CI ) continue ; Changed |= ST . makeLIDRangeMetadata ( CI ) ; }


<s> bool Changed = false ; for ( Function & F : M ) { if ( ! F . isDeclaration ( ) ) continue ; switch ( F . getIntrinsicID ( ) ) { case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : if ( expandMemIntrinsicUses ( F ) ) Changed = true ; break ; case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : Changed |= makeLIDRangeMetadata ( F ) ; break ; default : break ; }


<s> AdjustedArgTy = V4Ty ; } ArgPtr = Builder . CreateBitCast ( ArgPtr , AdjustedArgTy -> getPointerTo ( AS ) , ArgPtr -> getName ( ) + <STR_LIT> ) ; LoadInst * Load = Builder . CreateAlignedLoad ( AdjustedArgTy , ArgPtr , AdjustedAlign . value ( ) ) ; Load -> setMetadata ( LLVMContext :: MD_invariant_load , MDNode :: get ( Ctx , { } ) ) ; MDBuilder MDB ( Ctx ) ; if ( isa < PointerType > ( ArgTy ) ) { if ( Arg . hasNonNullAttr ( ) ) Load -> setMetadata ( LLVMContext :: MD_nonnull , MDNode :: get ( Ctx , { } ) ) ; uint64_t DerefBytes = Arg . getDereferenceableBytes ( ) ; if ( DerefBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefBytes ) ) ) ) ; } uint64_t DerefOrNullBytes = Arg . getDereferenceableOrNullBytes ( ) ; if ( DerefOrNullBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable_or_null , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefOrNullBytes ) ) ) ) ; } unsigned ParamAlign = Arg . getParamAlignment ( ) ; if ( ParamAlign != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_align , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , ParamAlign ) ) ) ) ; } } if ( DoShiftOpt ) { Value * ExtractBits = OffsetDiff == <NUM_LIT> ? Load : Builder . CreateLShr ( Load , OffsetDiff * <NUM_LIT> ) ; IntegerType * ArgIntTy = Builder . getIntNTy ( Size ) ; Value * Trunc = Builder . CreateTrunc ( ExtractBits , ArgIntTy ) ; Value * NewVal = Builder . CreateBitCast ( Trunc , ArgTy , Arg . getName ( ) + <STR_LIT> ) ; Arg . replaceAllUsesWith ( NewVal ) ; } else if ( IsV3 ) { Value * Shuf = Builder . CreateShuffleVector ( Load , UndefValue :: get ( V4Ty ) , { <NUM_LIT> , <NUM_LIT> , <NUM_LIT> } , Arg . getName ( ) + <STR_LIT> ) ;


<s> if ( CC != CallingConv :: <ISA_LIT>_KERNEL || F . arg_empty ( ) ) return false ; auto & TPC = getAnalysis < TargetPassConfig > ( ) ; const TargetMachine & TM = TPC . getTM < TargetMachine > ( ) ; const GCNSubtarget & ST = TM . getSubtarget < GCNSubtarget > ( F ) ; LLVMContext & Ctx = F . getParent ( ) -> getContext ( ) ; const DataLayout & DL = F . getParent ( ) -> getDataLayout ( ) ; BasicBlock & EntryBlock = * F . begin ( ) ; IRBuilder < > Builder ( & * EntryBlock . begin ( ) ) ; const unsigned KernArgBaseAlign = <NUM_LIT> ; const uint64_t BaseOffset = ST . getExplicitKernelArgOffset ( F ) ; unsigned MaxAlign ; const uint64_t TotalKernArgSize = ST . getKernArgSegmentSize ( F , MaxAlign ) ; if ( TotalKernArgSize == <NUM_LIT> ) return false ; CallInst * KernArgSegment = Builder . CreateIntrinsic ( <STR_LIT> , nullptr , F . getName ( ) + <STR_LIT> ) ; KernArgSegment -> addAttribute ( AttributeList :: ReturnIndex , Attribute :: NonNull ) ; KernArgSegment -> addAttribute ( AttributeList :: ReturnIndex , Attribute :: getWithDereferenceableBytes ( Ctx , TotalKernArgSize ) ) ; unsigned AS = KernArgSegment -> getType ( ) -> getPointerAddressSpace ( ) ; uint64_t ExplicitArgOffset = <NUM_LIT> ; for ( Argument & Arg : F . args ( ) ) { Type * ArgTy = Arg . getType ( ) ; unsigned Align = DL . getABITypeAlignment ( ArgTy ) ; unsigned Size = DL . getTypeSizeInBits ( ArgTy ) ; unsigned AllocSize = DL . getTypeAllocSize ( ArgTy ) ; bool IsExtArg = AllocSize < <NUM_LIT> && ( Arg . hasZExtAttr ( ) || Arg . hasSExtAttr ( ) ) && ! ST . isAmdHsaOS ( ) ; if ( IsExtArg ) AllocSize = <NUM_LIT> ; uint64_t EltOffset = alignTo ( ExplicitArgOffset , Align ) + BaseOffset ; ExplicitArgOffset = alignTo ( ExplicitArgOffset , Align ) + AllocSize ; if ( Arg . use_empty ( ) ) continue ; if ( PointerType * PT = dyn_cast < PointerType > ( ArgTy ) ) { if ( PT -> getAddressSpace ( ) == <ISA_LIT>AS :: LOCAL_ADDRESS && ST . getGeneration ( ) == <ISA_LIT>Subtarget :: SOUTHERN_ISLANDS ) continue ; if ( Arg . hasNoAliasAttr ( ) ) continue ; } VectorType * VT = dyn_cast < VectorType > ( ArgTy ) ;

bool IsV3 = VT && VT -> getNumElements ( ) == <NUM_LIT> ;


<s> if ( PointerType * PT = dyn_cast < PointerType > ( ArgTy ) ) { if ( PT -> getAddressSpace ( ) == <ISA_LIT>AS :: LOCAL_ADDRESS && ST . getGeneration ( ) == <ISA_LIT>Subtarget :: SOUTHERN_ISLANDS ) continue ; if ( Arg . hasNoAliasAttr ( ) ) continue ; } VectorType * VT = dyn_cast < VectorType > ( ArgTy ) ; bool IsV3 = VT && VT -> getNumElements ( ) == <NUM_LIT> ; VectorType * V4Ty = nullptr ; int64_t AlignDownOffset = alignDown ( EltOffset , <NUM_LIT> ) ; int64_t OffsetDiff = EltOffset - AlignDownOffset ; unsigned AdjustedAlign = MinAlign ( KernArgBaseAlign , AlignDownOffset ) ; Value * ArgPtr ; if ( Size < <NUM_LIT> && ! ArgTy -> isAggregateType ( ) ) { ArgPtr = Builder . CreateConstInBoundsGEP1_64 ( KernArgSegment , AlignDownOffset , Arg . getName ( ) + <STR_LIT> ) ; ArgPtr = Builder . CreateBitCast ( ArgPtr , Builder . getInt32Ty ( ) -> getPointerTo ( AS ) , ArgPtr -> getName ( ) + <STR_LIT> ) ; } else { ArgPtr = Builder . CreateConstInBoundsGEP1_64 ( KernArgSegment , AlignDownOffset , Arg . getName ( ) + <STR_LIT> ) ; ArgPtr = Builder . CreateBitCast ( ArgPtr , ArgTy -> getPointerTo ( AS ) , ArgPtr -> getName ( ) + <STR_LIT> ) ; } assert ( ( ! IsExtArg || ! IsV3 ) && <STR_LIT> ) ; if ( IsV3 && Size >= <NUM_LIT> ) { V4Ty = VectorType :: get ( VT -> getVectorElementType ( ) , <NUM_LIT> ) ; ArgPtr = Builder . CreateBitCast ( ArgPtr , V4Ty -> getPointerTo ( AS ) ) ; } LoadInst * Load = Builder . CreateAlignedLoad ( ArgPtr , AdjustedAlign ) ; Load -> setMetadata ( LLVMContext :: MD_invariant_load , MDNode :: get ( Ctx , { } ) ) ; MDBuilder MDB ( Ctx ) ; if ( isa < PointerType > ( ArgTy ) ) { if ( Arg . hasNonNullAttr ( ) ) Load -> setMetadata ( LLVMContext :: MD_nonnull , MDNode :: get ( Ctx , { } ) ) ; uint64_t DerefBytes = Arg . getDereferenceableBytes ( ) ; if ( DerefBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefBytes ) ) ) ) ; } uint64_t DerefOrNullBytes = Arg . getDereferenceableOrNullBytes ( ) ; if ( DerefOrNullBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable_or_null , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefOrNullBytes ) ) ) ) ;


<s> int64_t OffsetDiff = EltOffset - AlignDownOffset ; unsigned AdjustedAlign = MinAlign ( KernArgBaseAlign , AlignDownOffset ) ; Value * ArgPtr ; if ( Size < <NUM_LIT> && ! ArgTy -> isAggregateType ( ) ) { ArgPtr = Builder . CreateConstInBoundsGEP1_64 ( KernArgSegment , AlignDownOffset , Arg . getName ( ) + <STR_LIT> ) ; ArgPtr = Builder . CreateBitCast ( ArgPtr , Builder . getInt32Ty ( ) -> getPointerTo ( AS ) , ArgPtr -> getName ( ) + <STR_LIT> ) ; } else { ArgPtr = Builder . CreateConstInBoundsGEP1_64 ( KernArgSegment , AlignDownOffset , Arg . getName ( ) + <STR_LIT> ) ; ArgPtr = Builder . CreateBitCast ( ArgPtr , ArgTy -> getPointerTo ( AS ) , ArgPtr -> getName ( ) + <STR_LIT> ) ; } assert ( ( ! IsExtArg || ! IsV3 ) && <STR_LIT> ) ; if ( IsV3 && Size >= <NUM_LIT> ) { V4Ty = VectorType :: get ( VT -> getVectorElementType ( ) , <NUM_LIT> ) ; ArgPtr = Builder . CreateBitCast ( ArgPtr , V4Ty -> getPointerTo ( AS ) ) ; } LoadInst * Load = Builder . CreateAlignedLoad ( ArgPtr , AdjustedAlign ) ; Load -> setMetadata ( LLVMContext :: MD_invariant_load , MDNode :: get ( Ctx , { } ) ) ; MDBuilder MDB ( Ctx ) ; if ( isa < PointerType > ( ArgTy ) ) { if ( Arg . hasNonNullAttr ( ) ) Load -> setMetadata ( LLVMContext :: MD_nonnull , MDNode :: get ( Ctx , { } ) ) ; uint64_t DerefBytes = Arg . getDereferenceableBytes ( ) ; if ( DerefBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefBytes ) ) ) ) ; } uint64_t DerefOrNullBytes = Arg . getDereferenceableOrNullBytes ( ) ; if ( DerefOrNullBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable_or_null , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefOrNullBytes ) ) ) ) ; } unsigned ParamAlign = Arg . getParamAlignment ( ) ; if ( ParamAlign != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_align , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , ParamAlign ) ) ) ) ; } } if ( Size < <NUM_LIT> && ! ArgTy -> isAggregateType ( ) ) { if ( IsExtArg && OffsetDiff == <NUM_LIT> ) { Type * I32Ty = Builder . getInt32Ty ( ) ; bool IsSext = Arg . hasSExtAttr ( ) ; Metadata * LowAndHigh [ ] = {

ConstantAsMetadata :: get ( ConstantInt :: get ( I32Ty , IsSext ? minIntN ( Size ) : <NUM_LIT> ) ) , ConstantAsMetadata :: get ( ConstantInt :: get ( I32Ty , IsSext ? maxIntN ( Size ) + <NUM_LIT> : maxUIntN ( Size ) + <NUM_LIT> ) ) } ;


<s> if ( Arg . hasNoAliasAttr ( ) ) continue ; } VectorType * VT = dyn_cast < VectorType > ( ArgTy ) ; bool IsV3 = VT && VT -> getNumElements ( ) == <NUM_LIT> ; bool DoShiftOpt = Size < <NUM_LIT> && ! ArgTy -> isAggregateType ( ) ; VectorType * V4Ty = nullptr ; int64_t AlignDownOffset = alignDown ( EltOffset , <NUM_LIT> ) ; int64_t OffsetDiff = EltOffset - AlignDownOffset ; Align AdjustedAlign = commonAlignment ( KernArgBaseAlign , DoShiftOpt ? AlignDownOffset : EltOffset ) ; Value * ArgPtr ; Type * AdjustedArgTy ; if ( DoShiftOpt ) { ArgPtr = Builder . CreateConstInBoundsGEP1_64 ( Builder . getInt8Ty ( ) , KernArgSegment , AlignDownOffset , Arg . getName ( ) + <STR_LIT> ) ; AdjustedArgTy = Builder . getInt32Ty ( ) ; } else { ArgPtr = Builder . CreateConstInBoundsGEP1_64 ( Builder . getInt8Ty ( ) , KernArgSegment , EltOffset , Arg . getName ( ) + <STR_LIT> ) ; AdjustedArgTy = ArgTy ; } if ( IsV3 && Size >= <NUM_LIT> ) { V4Ty = VectorType :: get ( VT -> getVectorElementType ( ) , <NUM_LIT> ) ; AdjustedArgTy = V4Ty ; } ArgPtr = Builder . CreateBitCast ( ArgPtr , AdjustedArgTy -> getPointerTo ( AS ) , ArgPtr -> getName ( ) + <STR_LIT> ) ; LoadInst * Load = Builder . CreateAlignedLoad ( AdjustedArgTy , ArgPtr , AdjustedAlign ) ; Load -> setMetadata ( LLVMContext :: MD_invariant_load , MDNode :: get ( Ctx , { } ) ) ; MDBuilder MDB ( Ctx ) ; if ( isa < PointerType > ( ArgTy ) ) { if ( Arg . hasNonNullAttr ( ) ) Load -> setMetadata ( LLVMContext :: MD_nonnull , MDNode :: get ( Ctx , { } ) ) ; uint64_t DerefBytes = Arg . getDereferenceableBytes ( ) ; if ( DerefBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefBytes ) ) ) ) ; } uint64_t DerefOrNullBytes = Arg . getDereferenceableOrNullBytes ( ) ;


<s> IRBuilder < > Builder ( & * getInsertPt ( EntryBlock ) ) ; const Align KernArgBaseAlign ( <NUM_LIT> ) ; const uint64_t BaseOffset = ST . getExplicitKernelArgOffset ( F ) ; Align MaxAlign ; const uint64_t TotalKernArgSize = ST . getKernArgSegmentSize ( F , MaxAlign ) ; if ( TotalKernArgSize == <NUM_LIT> ) return false ; CallInst * KernArgSegment = Builder . CreateIntrinsic ( <STR_LIT> , { } , { } , nullptr , F . getName ( ) + <STR_LIT> ) ; KernArgSegment -> addAttribute ( AttributeList :: ReturnIndex , Attribute :: NonNull ) ; KernArgSegment -> addAttribute ( AttributeList :: ReturnIndex , Attribute :: getWithDereferenceableBytes ( Ctx , TotalKernArgSize ) ) ; unsigned AS = KernArgSegment -> getType ( ) -> getPointerAddressSpace ( ) ; uint64_t ExplicitArgOffset = <NUM_LIT> ; for ( Argument & Arg : F . args ( ) ) { const bool IsByRef = Arg . hasByRefAttr ( ) ; Type * ArgTy = IsByRef ? Arg . getParamByRefType ( ) : Arg . getType ( ) ; MaybeAlign ABITypeAlign = IsByRef ? Arg . getParamAlign ( ) : None ; if ( ! ABITypeAlign ) ABITypeAlign = DL . getABITypeAlign ( ArgTy ) ; uint64_t Size = DL . getTypeSizeInBits ( ArgTy ) ; uint64_t AllocSize = DL . getTypeAllocSize ( ArgTy ) ; uint64_t EltOffset = alignTo ( ExplicitArgOffset , ABITypeAlign ) + BaseOffset ; ExplicitArgOffset = alignTo ( ExplicitArgOffset , ABITypeAlign ) + AllocSize ; if ( Arg . use_empty ( ) ) continue ; if ( IsByRef ) { Value * ArgOffsetPtr = Builder . CreateConstInBoundsGEP1_64 ( Builder . getInt8Ty ( ) , KernArgSegment , EltOffset , Arg . getName ( ) + <STR_LIT> ) ; Value * CastOffsetPtr = Builder . CreatePointerBitCastOrAddrSpaceCast ( ArgOffsetPtr , Arg . getType ( ) ) ; Arg . replaceAllUsesWith ( CastOffsetPtr ) ; continue ; } if ( PointerType * PT = dyn_cast < PointerType > ( ArgTy ) ) { if ( ( PT -> getAddressSpace ( ) == <ISA_LIT>AS :: LOCAL_ADDRESS || PT -> getAddressSpace ( ) == <ISA_LIT>AS :: REGION_ADDRESS ) && ! ST . hasUsableDSOffset ( ) ) continue ; if ( Arg . hasNoAliasAttr ( ) ) continue ; } auto * VT = dyn_cast < FixedVectorType > ( ArgTy ) ; bool IsV3 = VT && VT -> getNumElements ( ) == <NUM_LIT> ; bool DoShiftOpt = Size < <NUM_LIT> && ! ArgTy -> isAggregateType ( ) ; VectorType * V4Ty = nullptr ; int64_t AlignDownOffset = alignDown ( EltOffset , <NUM_LIT> ) ; int64_t OffsetDiff = EltOffset - AlignDownOffset ; Align AdjustedAlign = commonAlignment ( KernArgBaseAlign , DoShiftOpt ? AlignDownOffset : EltOffset ) ;


<s> AllocaInst * AI = dyn_cast < AllocaInst > ( & * InsPt ) ; if ( ! AI || ! AI -> isStaticAlloca ( ) ) break ; }


<s> MDBuilder MDB ( Ctx ) ; if ( isa < PointerType > ( ArgTy ) ) { if ( Arg . hasNonNullAttr ( ) ) Load -> setMetadata ( LLVMContext :: MD_nonnull , MDNode :: get ( Ctx , { } ) ) ; uint64_t DerefBytes = Arg . getDereferenceableBytes ( ) ; if ( DerefBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefBytes ) ) ) ) ; } uint64_t DerefOrNullBytes = Arg . getDereferenceableOrNullBytes ( ) ; if ( DerefOrNullBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable_or_null , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefOrNullBytes ) ) ) ) ; } unsigned ParamAlign = Arg . getParamAlignment ( ) ; if ( ParamAlign != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_align , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , ParamAlign ) ) ) ) ; } } if ( DoShiftOpt ) { Value * ExtractBits = OffsetDiff == <NUM_LIT> ? Load : Builder . CreateLShr ( Load , OffsetDiff * <NUM_LIT> ) ; IntegerType * ArgIntTy = Builder . getIntNTy ( Size ) ; Value * Trunc = Builder . CreateTrunc ( ExtractBits , ArgIntTy ) ; Value * NewVal = Builder . CreateBitCast ( Trunc , ArgTy , Arg . getName ( ) + <STR_LIT> ) ; Arg . replaceAllUsesWith ( NewVal ) ; } else if ( IsV3 ) { Value * Shuf = Builder . CreateShuffleVector ( Load , UndefValue :: get ( V4Ty ) , ArrayRef < int > { <NUM_LIT> , <NUM_LIT> , <NUM_LIT> } , Arg . getName ( ) + <STR_LIT> ) ; Arg . replaceAllUsesWith ( Shuf ) ; } else { Load -> setName ( Arg . getName ( ) + <STR_LIT> ) ; Arg . replaceAllUsesWith ( Load ) ; } } KernArgSegment -> addAttribute ( AttributeList :: ReturnIndex , Attribute :: getWithAlignment ( Ctx , std :: max ( KernArgBaseAlign , MaxAlign ) ) ) ; return true ;


<s> if ( Arg . hasNonNullAttr ( ) ) Load -> setMetadata ( LLVMContext :: MD_nonnull , MDNode :: get ( Ctx , { } ) ) ; uint64_t DerefBytes = Arg . getDereferenceableBytes ( ) ; if ( DerefBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefBytes ) ) ) ) ; } uint64_t DerefOrNullBytes = Arg . getDereferenceableOrNullBytes ( ) ; if ( DerefOrNullBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable_or_null , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefOrNullBytes ) ) ) ) ; } unsigned ParamAlign = Arg . getParamAlignment ( ) ; if ( ParamAlign != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_align , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , ParamAlign ) ) ) ) ; } } if ( DoShiftOpt ) { Value * ExtractBits = OffsetDiff == <NUM_LIT> ? Load : Builder . CreateLShr ( Load , OffsetDiff * <NUM_LIT> ) ; IntegerType * ArgIntTy = Builder . getIntNTy ( Size ) ; Value * Trunc = Builder . CreateTrunc ( ExtractBits , ArgIntTy ) ; Value * NewVal = Builder . CreateBitCast ( Trunc , ArgTy , Arg . getName ( ) + <STR_LIT> ) ; Arg . replaceAllUsesWith ( NewVal ) ; } else if ( IsV3 ) { Value * Shuf = Builder . CreateShuffleVector ( Load , UndefValue :: get ( V4Ty ) , ArrayRef < int > { <NUM_LIT> , <NUM_LIT> , <NUM_LIT> } , Arg . getName ( ) + <STR_LIT> ) ; Arg . replaceAllUsesWith ( Shuf ) ; } else { Load -> setName ( Arg . getName ( ) + <STR_LIT> ) ; Arg . replaceAllUsesWith ( Load ) ;


<s> } if ( IsV3 && Size >= <NUM_LIT> ) { V4Ty = VectorType :: get ( VT -> getVectorElementType ( ) , <NUM_LIT> ) ; AdjustedArgTy = V4Ty ; } ArgPtr = Builder . CreateBitCast ( ArgPtr , AdjustedArgTy -> getPointerTo ( AS ) , ArgPtr -> getName ( ) + <STR_LIT> ) ; LoadInst * Load = Builder . CreateAlignedLoad ( AdjustedArgTy , ArgPtr , AdjustedAlign ) ; Load -> setMetadata ( LLVMContext :: MD_invariant_load , MDNode :: get ( Ctx , { } ) ) ; MDBuilder MDB ( Ctx ) ; if ( isa < PointerType > ( ArgTy ) ) { if ( Arg . hasNonNullAttr ( ) ) Load -> setMetadata ( LLVMContext :: MD_nonnull , MDNode :: get ( Ctx , { } ) ) ; uint64_t DerefBytes = Arg . getDereferenceableBytes ( ) ; if ( DerefBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefBytes ) ) ) ) ; } uint64_t DerefOrNullBytes = Arg . getDereferenceableOrNullBytes ( ) ; if ( DerefOrNullBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable_or_null , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefOrNullBytes ) ) ) ) ; } unsigned ParamAlign = Arg . getParamAlignment ( ) ; if ( ParamAlign != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_align , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , ParamAlign ) ) ) ) ; } } if ( DoShiftOpt ) { Value * ExtractBits = OffsetDiff == <NUM_LIT> ? Load : Builder . CreateLShr ( Load , OffsetDiff * <NUM_LIT> ) ; IntegerType * ArgIntTy = Builder . getIntNTy ( Size ) ; Value * Trunc = Builder . CreateTrunc ( ExtractBits , ArgIntTy ) ;

Value * NewVal = Builder . CreateBitCast ( Trunc , ArgTy , Arg . getName ( ) + <STR_LIT> ) ;


<s> if ( IsV3 && Size >= <NUM_LIT> ) { V4Ty = VectorType :: get ( VT -> getVectorElementType ( ) , <NUM_LIT> ) ; ArgPtr = Builder . CreateBitCast ( ArgPtr , V4Ty -> getPointerTo ( AS ) ) ; } LoadInst * Load = Builder . CreateAlignedLoad ( ArgPtr , AdjustedAlign ) ; Load -> setMetadata ( LLVMContext :: MD_invariant_load , MDNode :: get ( Ctx , { } ) ) ; MDBuilder MDB ( Ctx ) ; if ( isa < PointerType > ( ArgTy ) ) { if ( Arg . hasNonNullAttr ( ) ) Load -> setMetadata ( LLVMContext :: MD_nonnull , MDNode :: get ( Ctx , { } ) ) ; uint64_t DerefBytes = Arg . getDereferenceableBytes ( ) ; if ( DerefBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefBytes ) ) ) ) ; } uint64_t DerefOrNullBytes = Arg . getDereferenceableOrNullBytes ( ) ; if ( DerefOrNullBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable_or_null , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefOrNullBytes ) ) ) ) ; } unsigned ParamAlign = Arg . getParamAlignment ( ) ; if ( ParamAlign != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_align , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , ParamAlign ) ) ) ) ; } } if ( Size < <NUM_LIT> && ! ArgTy -> isAggregateType ( ) ) { Value * ExtractBits = OffsetDiff == <NUM_LIT> ? Load : Builder . CreateLShr ( Load , OffsetDiff * <NUM_LIT> ) ; IntegerType * ArgIntTy = Builder . getIntNTy ( Size ) ; Value * Trunc = Builder . CreateTrunc ( ExtractBits , ArgIntTy ) ; Value * NewVal = Builder . CreateBitCast ( Trunc , ArgTy , Arg . getName ( ) + <STR_LIT> ) ; Arg . replaceAllUsesWith ( NewVal ) ; } else if ( IsV3 ) {

Value * Shuf = Builder . CreateShuffleVector ( Load , UndefValue :: get ( V4Ty ) , { <NUM_LIT> , <NUM_LIT> , <NUM_LIT> } , Arg . getName ( ) + <STR_LIT> ) ;


<s> } else { ArgPtr = Builder . CreateConstInBoundsGEP1_64 ( Builder . getInt8Ty ( ) , KernArgSegment , EltOffset , Arg . getName ( ) + <STR_LIT> ) ; AdjustedArgTy = ArgTy ; } if ( IsV3 && Size >= <NUM_LIT> ) { V4Ty = VectorType :: get ( VT -> getVectorElementType ( ) , <NUM_LIT> ) ; AdjustedArgTy = V4Ty ; } ArgPtr = Builder . CreateBitCast ( ArgPtr , AdjustedArgTy -> getPointerTo ( AS ) , ArgPtr -> getName ( ) + <STR_LIT> ) ; LoadInst * Load = Builder . CreateAlignedLoad ( AdjustedArgTy , ArgPtr , AdjustedAlign ) ; Load -> setMetadata ( LLVMContext :: MD_invariant_load , MDNode :: get ( Ctx , { } ) ) ; MDBuilder MDB ( Ctx ) ; if ( isa < PointerType > ( ArgTy ) ) { if ( Arg . hasNonNullAttr ( ) ) Load -> setMetadata ( LLVMContext :: MD_nonnull , MDNode :: get ( Ctx , { } ) ) ; uint64_t DerefBytes = Arg . getDereferenceableBytes ( ) ; if ( DerefBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefBytes ) ) ) ) ; } uint64_t DerefOrNullBytes = Arg . getDereferenceableOrNullBytes ( ) ; if ( DerefOrNullBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable_or_null , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefOrNullBytes ) ) ) ) ; } unsigned ParamAlign = Arg . getParamAlignment ( ) ; if ( ParamAlign != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_align , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , ParamAlign ) ) ) ) ; } } if ( DoShiftOpt ) { Value * ExtractBits = OffsetDiff == <NUM_LIT> ? Load : Builder . CreateLShr ( Load , OffsetDiff * <NUM_LIT> ) ; IntegerType * ArgIntTy = Builder . getIntNTy ( Size ) ; Value * Trunc = Builder . CreateTrunc ( ExtractBits , ArgIntTy ) ; Value * NewVal = Builder . CreateBitCast ( Trunc , ArgTy , Arg . getName ( ) + <STR_LIT> ) ; Arg . replaceAllUsesWith ( NewVal ) ;


<s> for ( Argument & Arg : F . args ( ) ) { Type * ArgTy = Arg . getType ( ) ; Align ABITypeAlign = DL . getABITypeAlign ( ArgTy ) ; unsigned Size = DL . getTypeSizeInBits ( ArgTy ) ; unsigned AllocSize = DL . getTypeAllocSize ( ArgTy ) ; uint64_t EltOffset = alignTo ( ExplicitArgOffset , ABITypeAlign ) + BaseOffset ; ExplicitArgOffset = alignTo ( ExplicitArgOffset , ABITypeAlign ) + AllocSize ; if ( Arg . use_empty ( ) ) continue ; if ( PointerType * PT = dyn_cast < PointerType > ( ArgTy ) ) { if ( ( PT -> getAddressSpace ( ) == <ISA_LIT>AS :: LOCAL_ADDRESS || PT -> getAddressSpace ( ) == <ISA_LIT>AS :: REGION_ADDRESS ) && ! ST . hasUsableDSOffset ( ) ) continue ; if ( Arg . hasNoAliasAttr ( ) ) continue ; } auto * VT = dyn_cast < FixedVectorType > ( ArgTy ) ; bool IsV3 = VT && VT -> getNumElements ( ) == <NUM_LIT> ; bool DoShiftOpt = Size < <NUM_LIT> && ! ArgTy -> isAggregateType ( ) ; VectorType * V4Ty = nullptr ; int64_t AlignDownOffset = alignDown ( EltOffset , <NUM_LIT> ) ; int64_t OffsetDiff = EltOffset - AlignDownOffset ; Align AdjustedAlign = commonAlignment ( KernArgBaseAlign , DoShiftOpt ? AlignDownOffset : EltOffset ) ; Value * ArgPtr ; Type * AdjustedArgTy ; if ( DoShiftOpt ) { ArgPtr = Builder . CreateConstInBoundsGEP1_64 ( Builder . getInt8Ty ( ) , KernArgSegment , AlignDownOffset , Arg . getName ( ) + <STR_LIT> ) ; AdjustedArgTy = Builder . getInt32Ty ( ) ; } else { ArgPtr = Builder . CreateConstInBoundsGEP1_64 ( Builder . getInt8Ty ( ) , KernArgSegment , EltOffset , Arg . getName ( ) + <STR_LIT> ) ; AdjustedArgTy = ArgTy ; } if ( IsV3 && Size >= <NUM_LIT> ) { V4Ty = FixedVectorType :: get ( VT -> getElementType ( ) , <NUM_LIT> ) ; AdjustedArgTy = V4Ty ; } ArgPtr = Builder . CreateBitCast ( ArgPtr , AdjustedArgTy -> getPointerTo ( AS ) , ArgPtr -> getName ( ) + <STR_LIT> ) ; LoadInst * Load = Builder . CreateAlignedLoad ( AdjustedArgTy , ArgPtr , AdjustedAlign ) ; Load -> setMetadata ( LLVMContext :: MD_invariant_load , MDNode :: get ( Ctx , { } ) ) ; MDBuilder MDB ( Ctx ) ; if ( isa < PointerType > ( ArgTy ) ) { if ( Arg . hasNonNullAttr ( ) ) Load -> setMetadata ( LLVMContext :: MD_nonnull , MDNode :: get ( Ctx , { } ) ) ; uint64_t DerefBytes = Arg . getDereferenceableBytes ( ) ; if ( DerefBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefBytes ) ) ) ) ; } uint64_t DerefOrNullBytes = Arg . getDereferenceableOrNullBytes ( ) ;


<s> <ISA_LIT>LowerKernelArguments ( ) : FunctionPass ( ID ) {


<s> FunctionPass * llvm :: create<ISA_LIT>LowerKernelArgumentsPass ( ) {

return new <ISA_LIT>LowerKernelArguments ( ) ;


<s> AU . addRequired < TargetPassConfig > ( ) ;


<s> Value * ArgPtr ; if ( DoShiftOpt ) { ArgPtr = Builder . CreateConstInBoundsGEP1_64 ( KernArgSegment , AlignDownOffset , Arg . getName ( ) + <STR_LIT> ) ; ArgPtr = Builder . CreateBitCast ( ArgPtr , Builder . getInt32Ty ( ) -> getPointerTo ( AS ) , ArgPtr -> getName ( ) + <STR_LIT> ) ; } else { ArgPtr = Builder . CreateConstInBoundsGEP1_64 ( KernArgSegment , EltOffset , Arg . getName ( ) + <STR_LIT> ) ; ArgPtr = Builder . CreateBitCast ( ArgPtr , ArgTy -> getPointerTo ( AS ) , ArgPtr -> getName ( ) + <STR_LIT> ) ; } if ( IsV3 && Size >= <NUM_LIT> ) { V4Ty = VectorType :: get ( VT -> getVectorElementType ( ) , <NUM_LIT> ) ; ArgPtr = Builder . CreateBitCast ( ArgPtr , V4Ty -> getPointerTo ( AS ) ) ; } LoadInst * Load = Builder . CreateAlignedLoad ( ArgPtr , AdjustedAlign ) ; Load -> setMetadata ( LLVMContext :: MD_invariant_load , MDNode :: get ( Ctx , { } ) ) ; MDBuilder MDB ( Ctx ) ; if ( isa < PointerType > ( ArgTy ) ) { if ( Arg . hasNonNullAttr ( ) ) Load -> setMetadata ( LLVMContext :: MD_nonnull , MDNode :: get ( Ctx , { } ) ) ; uint64_t DerefBytes = Arg . getDereferenceableBytes ( ) ; if ( DerefBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefBytes ) ) ) ) ; } uint64_t DerefOrNullBytes = Arg . getDereferenceableOrNullBytes ( ) ; if ( DerefOrNullBytes != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_dereferenceable_or_null , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , DerefOrNullBytes ) ) ) ) ; } unsigned ParamAlign = Arg . getParamAlignment ( ) ; if ( ParamAlign != <NUM_LIT> ) { Load -> setMetadata ( LLVMContext :: MD_align , MDNode :: get ( Ctx , MDB . createConstant ( ConstantInt :: get ( Builder . getInt64Ty ( ) , ParamAlign ) ) ) ) ; } } if ( DoShiftOpt ) { Value * ExtractBits = OffsetDiff == <NUM_LIT> ? Load : Builder . CreateLShr ( Load , OffsetDiff * <NUM_LIT> ) ; IntegerType * ArgIntTy = Builder . getIntNTy ( Size ) ; Value * Trunc = Builder . CreateTrunc ( ExtractBits , ArgIntTy ) ; Value * NewVal = Builder . CreateBitCast ( Trunc , ArgTy , Arg . getName ( ) + <STR_LIT> ) ; Arg . replaceAllUsesWith ( NewVal ) ; } else if ( IsV3 ) { Value * Shuf = Builder . CreateShuffleVector ( Load , UndefValue :: get ( V4Ty ) , { <NUM_LIT> , <NUM_LIT> , <NUM_LIT> } , Arg . getName ( ) + <STR_LIT> ) ;


<s> StringRef DispatchPtrName = <STR_LIT> ( <STR_LIT> ) ; Function * DispatchPtr = F . getParent ( ) -> getFunction ( DispatchPtrName ) ; if ( ! DispatchPtr ) return PreservedAnalyses :: all ( ) ; for ( Instruction & I : instructions ( F ) ) { if ( CallInst * CI = dyn_cast < CallInst > ( & I ) ) { if ( CI -> getCalledFunction ( ) == DispatchPtr ) processUse ( CI ) ; } } return PreservedAnalyses :: all ( ) ;


<s> Function * DispatchPtr = M . getFunction ( DispatchPtrName ) ; if ( ! DispatchPtr ) return false ; bool MadeChange = false ; SmallPtrSet < Instruction * , <NUM_LIT> > HandledUses ; for ( auto * U : DispatchPtr -> users ( ) ) {

CallInst * CI = cast < CallInst > ( U ) ;


<s> CallInst * CI = cast < CallInst > ( U ) ; if ( HandledUses . insert ( CI ) . second ) {

if ( processUse ( CI ) ) MadeChange = true ;


<s> case GRID_<ISA_LIT>ZE_Y : if ( LoadSize == <NUM_LIT> ) GridSizeY = Load ; break ; case GRID_<ISA_LIT>ZE_Z : if ( LoadSize == <NUM_LIT> ) GridSizeZ = Load ; break ; default : break ; } } bool MadeChange = false ; Value * WorkGroupSizes [ <NUM_LIT> ] = { WorkGroupSizeX , WorkGroupSizeY , WorkGroupSizeZ } ; Value * GridSizes [ <NUM_LIT> ] = { GridSizeX , GridSizeY , GridSizeZ } ; for ( int I = <NUM_LIT> ; HasUniformWorkGroupSize && I < <NUM_LIT> ; ++ I ) { Value * GroupSize = WorkGroupSizes [ I ] ; Value * GridSize = GridSizes [ I ] ; if ( ! GroupSize || ! GridSize ) continue ; using namespace llvm :: PatternMatch ; auto GroupIDIntrin = I == <NUM_LIT> ? m_Intrinsic < <STR_LIT> > ( ) : ( I == <NUM_LIT> ? m_Intrinsic < <STR_LIT> > ( ) : m_Intrinsic < <STR_LIT> > ( ) ) ; for ( User * U : GroupSize -> users ( ) ) { auto * ZextGroupSize = dyn_cast < ZExtInst > ( U ) ; if ( ! ZextGroupSize ) continue ; for ( User * UMin : ZextGroupSize -> users ( ) ) { if ( match ( UMin , m_UMin ( m_Sub ( m_Specific ( GridSize ) , m_Mul ( GroupIDIntrin , m_Specific ( ZextGroupSize ) ) ) , m_Specific ( ZextGroupSize ) ) ) ) { if ( HasReqdWorkGroupSize ) { ConstantInt * KnownSize = mdconst :: extract < ConstantInt > ( MD -> getOperand ( I ) ) ; UMin -> replaceAllUsesWith ( ConstantExpr :: getIntegerCast ( KnownSize , UMin -> getType ( ) , false ) ) ; } else { UMin -> replaceAllUsesWith ( ZextGroupSize ) ; } MadeChange = true ; } } } } if ( ! HasReqdWorkGroupSize ) return MadeChange ; for ( int I = <NUM_LIT> ; I < <NUM_LIT> ; ++ I ) { Value * GroupSize = WorkGroupSizes [ I ] ; if ( ! GroupSize ) continue ; ConstantInt * KnownSize = mdconst :: extract < ConstantInt > ( MD -> getOperand ( I ) ) ; GroupSize -> replaceAllUsesWith ( ConstantExpr :: getIntegerCast ( KnownSize , GroupSize -> getType ( ) , false ) ) ; MadeChange = true ; } return MadeChange ;


<s> const bool HasReqdWorkGroupSize = MD && MD -> getNumOperands ( ) == <NUM_LIT> ; const bool HasUniformWorkGroupSize = F -> getFnAttribute ( <STR_LIT> ) . getValueAsBool ( ) ; if ( ! HasReqdWorkGroupSize && ! HasUniformWorkGroupSize ) return false ; Value * WorkGroupSizeX = nullptr ; Value * WorkGroupSizeY = nullptr ; Value * WorkGroupSizeZ = nullptr ; Value * GridSizeX = nullptr ; Value * GridSizeY = nullptr ; Value * GridSizeZ = nullptr ; const DataLayout & DL = F -> getParent ( ) -> getDataLayout ( ) ; for ( User * U : CI -> users ( ) ) { if ( ! U -> hasOneUse ( ) ) continue ; int64_t Offset = <NUM_LIT> ; if ( GetPointerBaseWithConstantOffset ( U , Offset , DL ) != CI ) continue ; auto * BCI = dyn_cast < BitCastInst > ( * U -> user_begin ( ) ) ; if ( ! BCI || ! BCI -> hasOneUse ( ) ) continue ; auto * Load = dyn_cast < LoadInst > ( * BCI -> user_begin ( ) ) ; if ( ! Load || ! Load -> isSimple ( ) ) continue ; unsigned LoadSize = DL . getTypeStoreSize ( Load -> getType ( ) ) ; switch ( Offset ) { case WORKGROUP_<ISA_LIT>ZE_X : if ( LoadSize == <NUM_LIT> ) WorkGroupSizeX = Load ; break ; case WORKGROUP_<ISA_LIT>ZE_Y : if ( LoadSize == <NUM_LIT> ) WorkGroupSizeY = Load ; break ; case WORKGROUP_<ISA_LIT>ZE_Z : if ( LoadSize == <NUM_LIT> ) WorkGroupSizeZ = Load ; break ; case GRID_<ISA_LIT>ZE_X : if ( LoadSize == <NUM_LIT> ) GridSizeX = Load ; break ; case GRID_<ISA_LIT>ZE_Y :

if ( LoadSize == <NUM_LIT> ) GridSizeY = Load ;


<s> switch ( Offset ) { case WORKGROUP_<ISA_LIT>ZE_X : if ( LoadSize == <NUM_LIT> ) WorkGroupSizeX = Load ; break ; case WORKGROUP_<ISA_LIT>ZE_Y : if ( LoadSize == <NUM_LIT> ) WorkGroupSizeY = Load ; break ; case WORKGROUP_<ISA_LIT>ZE_Z : if ( LoadSize == <NUM_LIT> ) WorkGroupSizeZ = Load ; break ; case GRID_<ISA_LIT>ZE_X : if ( LoadSize == <NUM_LIT> ) GridSizeX = Load ; break ; case GRID_<ISA_LIT>ZE_Y : if ( LoadSize == <NUM_LIT> ) GridSizeY = Load ; break ; case GRID_<ISA_LIT>ZE_Z : if ( LoadSize == <NUM_LIT> ) GridSizeZ = Load ; break ; default : break ; } } bool MadeChange = false ; Value * WorkGroupSizes [ <NUM_LIT> ] = { WorkGroupSizeX , WorkGroupSizeY , WorkGroupSizeZ } ; Value * GridSizes [ <NUM_LIT> ] = { GridSizeX , GridSizeY , GridSizeZ } ; for ( int I = <NUM_LIT> ; HasUniformWorkGroupSize && I < <NUM_LIT> ; ++ I ) { Value * GroupSize = WorkGroupSizes [ I ] ; Value * GridSize = GridSizes [ I ] ; if ( ! GroupSize || ! GridSize ) continue ; for ( User * U : GroupSize -> users ( ) ) { auto * ZextGroupSize = dyn_cast < ZExtInst > ( U ) ; if ( ! ZextGroupSize ) continue ; for ( User * ZextUser : ZextGroupSize -> users ( ) ) { auto * <ISA_LIT> = dyn_cast < SelectInst > ( ZextUser ) ; if ( ! <ISA_LIT> ) continue ; using namespace llvm :: PatternMatch ; auto GroupIDIntrin = I == <NUM_LIT> ? m_Intrinsic < <STR_LIT> > ( ) : ( I == <NUM_LIT> ? m_Intrinsic < <STR_LIT> > ( ) : m_Intrinsic < <STR_LIT> > ( ) ) ; auto SubExpr = m_Sub ( m_Specific ( GridSize ) , m_Mul ( GroupIDIntrin , m_Specific ( ZextGroupSize ) ) ) ; ICmpInst :: Predicate Pred ; if ( match ( <ISA_LIT> , m_Select ( m_ICmp ( Pred , SubExpr , m_Specific ( ZextGroupSize ) ) , SubExpr , m_Specific ( ZextGroupSize ) ) ) && Pred == ICmpInst :: ICMP_ULT ) { if ( HasReqdWorkGroupSize ) { ConstantInt * KnownSize = mdconst :: extract < ConstantInt > ( MD -> getOperand ( I ) ) ; <ISA_LIT> -> replaceAllUsesWith ( ConstantExpr :: getIntegerCast ( KnownSize , <ISA_LIT> -> getType ( ) , false ) ) ; } else { <ISA_LIT> -> replaceAllUsesWith ( ZextGroupSize ) ; } MadeChange = true ; }