<s> size_t MaskLen = Mask . size ( ) ; int ElemSizeInBits = VT . getVectorElementType ( ) . getSizeInBits ( ) ; if ( ( Subtarget . useHVXSglOps ( ) && ( ElemSizeInBits * MaskLen ) == <NUM_LIT> * <NUM_LIT> ) || ( Subtarget . useHVXDblOps ( ) && ( ElemSizeInBits * MaskLen ) == <NUM_LIT> * <NUM_LIT> ) ) { StridedLoadKind Pattern = isStridedLoad ( Mask ) ; if ( Pattern == StridedLoadKind :: NoPattern ) return SDValue ( ) ; SDValue Vec0 = Op . getOperand ( <NUM_LIT> ) ; SDValue Vec1 = Op . getOperand ( <NUM_LIT> ) ; SDValue StridePattern = DAG . getConstant ( Pattern , dl , <STR_LIT> ) ; SDValue Ops [ ] = { Vec1 , Vec0 , StridePattern } ; return DAG . getNode ( <STR_LIT> , dl , VT , Ops ) ; } } return SDValue


<s> static SDValue CreateCopyOfByValArgument ( SDValue Src , SDValue Dst , SDValue Chain , <STR_LIT> Flags , SelectionDAG & DAG , SDLoc dl ) { SDValue SizeNode = DAG . getConstant ( Flags


<s> const char * <ISA_LIT>TargetLowering :: getTargetNodeName ( unsigned Opcode ) const { switch ( ( <STR_LIT> ) Opcode ) { case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> : return <STR_LIT> ; case <STR_LIT> :


<s> const SDLoc & dl ( Op ) ; unsigned Opc = Op . getOpcode ( ) ; if ( Opc == <STR_LIT> ) { SDValue P = Op . getOperand ( <NUM_LIT> ) ; switch ( P . getOpcode ( ) ) { case <STR_LIT> : return DCI . DAG . getConstant ( - <NUM_LIT> , dl , ty ( Op ) ) ; case <STR_LIT> : return getZero ( dl , ty ( Op ) , DCI . DAG ) ; default : break ; } } else if ( Opc == <STR_LIT> ) { SDValue Cond = Op . getOperand ( <NUM_LIT> ) ; if ( Cond -> getOpcode ( ) == <STR_LIT> ) { SDValue C0 = Cond . getOperand ( <NUM_LIT> ) , C1 = Cond . getOperand ( <NUM_LIT> ) ; if ( C1 -> getOpcode ( ) == <STR_LIT> ) { SDValue VSel = DCI . DAG . getNode ( <STR_LIT> , dl , ty ( Op ) , C0 , Op . getOperand ( <NUM_LIT> ) , Op . getOperand ( <NUM_LIT> ) ) ; return


<s> if ( ! Results . empty ( ) ) return ; } const SDLoc & dl ( N ) ; switch ( N -> getOpcode ( ) ) { case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : return ; case <STR_LIT> : if ( N -> getValueType ( <NUM_LIT> )


<s> bool <ISA_LIT>TargetLowering :: allowsHvxMemoryAccess ( MVT VecTy , MachineMemOperand :: Flags Flags , bool * Fast ) const { if ( VecTy . getSizeInBits ( ) > <NUM_LIT> * Subtarget . getVectorLength ( ) ) return false ; if ( ! Subtarget . isHVXVectorType ( VecTy , false ) ) return false ; if ( Fast ) *


<s> bool <ISA_LIT>TargetLowering :: allowsHvxMisalignedMemoryAccesses ( MVT VecTy , MachineMemOperand :: Flags Flags , bool * Fast ) const { if ( ! Subtarget . isHVXVectorType ( VecTy ) ) return false ; if ( Fast ) * Fast = true ; return


<s> } } else { for ( unsigned I = <NUM_LIT> ; I != VecLen ; I += <NUM_LIT> ) { unsigned B = <NUM_LIT> ; for ( ; B != <NUM_LIT> ; ++ B ) { if ( ! Values [ I + B ] . isUndef ( ) ) break ; } SDValue F = Values [ I + B ] ; AllT &= IsTrue ( F ) ; AllF &= IsFalse ( F ) ; SDValue Ext = ( B < <NUM_LIT> ) ? DAG . getZExtOrTrunc ( F , dl , <STR_LIT> ) : DAG . getUNDEF ( <STR_LIT> ) ; Bytes . push_back ( Ext ) ; for ( ; B != <NUM_LIT> ; ++ B ) assert ( Values [ I + B ] . isUndef ( ) || Values [ I + B ] == F ) ; } } if ( AllT ) return DAG . getNode ( <STR_LIT> , dl , VecTy ) ; if ( AllF ) return DAG . getNode ( <STR_LIT> , dl , VecTy ) ; MVT ByteTy = <STR_LIT> ( <STR_LIT> , HwLen ) ; SDValue ByteVec = buildHvxVectorReg ( Bytes , dl , ByteTy , DAG ) ; return DAG . getNode ( <STR_LIT> , dl , VecTy , ByteVec


<s> SDValue Ext = ! V . isUndef ( ) ? DAG . getZExtOrTrunc ( V , dl , <STR_LIT> ) : DAG . getUNDEF ( <STR_LIT> ) ; for ( unsigned B = <NUM_LIT> ; B != BitBytes ; ++ B ) Bytes . push_back ( Ext ) ; } } else { for ( unsigned I = <NUM_LIT> ; I != VecLen ; I += <NUM_LIT> ) { unsigned B = <NUM_LIT> ; for ( ; B != <NUM_LIT> ; ++ B ) { if ( ! Values [ I + B ] . isUndef ( ) ) break ; } SDValue F = Values [ I + B ] ; AllT &= IsTrue ( F ) ; AllF &= IsFalse ( F ) ; SDValue Ext = ( B < <NUM_LIT> ) ? DAG . getZExtOrTrunc ( F , dl , <STR_LIT> ) : DAG . getUNDEF ( <STR_LIT> ) ; Bytes . push_back ( Ext ) ; for ( ; B != <NUM_LIT> ; ++ B ) assert ( Values [ I + B ] . isUndef ( ) || Values [ I + B ] == F ) ; } } if ( AllT ) return DAG . getNode ( <STR_LIT> , dl , VecTy ) ; if ( AllF ) return DAG . getNode ( <STR_LIT> , dl , VecTy ) ; MVT ByteTy = <STR_LIT> ( <STR_LIT> ,


<s> SDValue CP = LowerConstantPool ( DAG . getConstantPool ( CV , ByteTy , Alignment ) , DAG ) ; SDValue Bytes = DAG . getLoad ( ByteTy , dl , DAG . getEntryNode ( ) , CP , MachinePointerInfo :: getConstantPool ( MF ) , Alignment ) ; SDValue Sel = DAG . getSelect ( dl , VecTy , VecQ , DAG . getBitcast ( VecTy , Bytes ) , getZero ( dl , VecTy , DAG ) ) ; SDValue All1 = DAG . getSplatBuildVector ( <STR_LIT> , dl , DAG . getConstant ( <NUM_LIT> , dl , <STR_LIT> ) ) ; SDValue Vrmpy = getInstr ( <STR_LIT> , dl , ByteTy , { Sel , All1 } , DAG ) ; SDValue Rot = getInstr ( <STR_LIT> , dl , ByteTy , { Vrmpy , Vrmpy , DAG . getTargetConstant ( <NUM_LIT> , dl , <STR_LIT> ) } , DAG ) ; SDValue Vor = DAG . getNode ( <STR_LIT> , dl , ByteTy , { Vrmpy , Rot } ) ; SmallVector < int ,


<s> auto Hi32 = [ & DAG , & dl ] ( SDValue P ) { return DAG . getTargetExtractSubreg ( <STR_LIT> , dl , <STR_LIT> , P ) ; } ; SDValue W0 = isUndef ( PredV ) ? DAG . getUNDEF ( <STR_LIT> ) : DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , PredV ) ; Words [ IdxW ] . push_back ( Hi32 ( W0 ) ) ; Words [ IdxW ] . push_back ( Lo32 ( W0 ) ) ; while ( Bytes < BitBytes ) { IdxW ^= <NUM_LIT> ; Words [ IdxW ] . clear ( ) ; if ( Bytes < <NUM_LIT> ) { for ( const SDValue & W : Words [ IdxW ^ <NUM_LIT> ] ) { SDValue T = expandPredicate ( W , dl , DAG ) ; Words [ IdxW ] . push_back ( Hi32 ( T ) ) ; Words [ IdxW ] . push_back ( Lo32 ( T ) ) ; } } else { for ( const SDValue & W : Words [ IdxW ^ <NUM_LIT> ] ) { Words [ IdxW ] . push_back ( W ) ; Words [ IdxW ] . push_back ( W ) ; } } Bytes *= <NUM_LIT> ; } assert ( Bytes == BitBytes ) ; SDValue Vec = ZeroFill ? getZero ( dl , ByteTy , DAG ) : DAG . getUNDEF ( ByteTy ) ; SDValue S4 = DAG . getConstant ( HwLen - <NUM_LIT> , dl , <STR_LIT> ) ; for ( const SDValue & W : Words [ IdxW ] ) { Vec = DAG . getNode ( <STR_LIT> , dl , ByteTy , Vec , S4 ) ; Vec = DAG . getNode ( <STR_LIT> , dl , ByteTy , Vec , W ) ; } return


<s> assert ( Subtarget . isHVXVectorType ( ResTy ) ) ; if ( ! ZeroExt ) return DAG . getNode ( <STR_LIT> , dl , ResTy , VecV ) ; assert ( ty ( VecV ) . getVectorNumElements ( ) == ResTy . getVectorNumElements ( ) ) ; SDValue True = DAG . getNode ( <STR_LIT> , dl , ResTy , DAG . getConstant (


<s> SDValue <ISA_LIT>TargetLowering :: extendHvxVectorPred ( SDValue VecV , const SDLoc & dl , MVT ResTy , bool ZeroExt , SelectionDAG & DAG ) const { assert ( Subtarget . isHVXVectorType ( ResTy ) ) ; if ( ! ZeroExt ) return DAG . getNode ( <STR_LIT> , dl , ResTy , VecV ) ; assert ( ty ( VecV ) . getVectorNumElements ( ) == ResTy . getVectorNumElements ( ) ) ; SDValue True = DAG . getNode ( <STR_LIT> , dl , ResTy , DAG . getConstant ( <NUM_LIT> , dl , <STR_LIT> ) ) ; SDValue False = getZero ( dl


<s> assert ( ResTy == <STR_LIT> ) ; unsigned HwLen = Subtarget . getVectorLength ( ) ; MVT ByteTy = <STR_LIT> ( <STR_LIT> , HwLen ) ; SDValue ByteVec = DAG . getNode ( <STR_LIT> , dl , ByteTy , VecV ) ; unsigned Scale = HwLen / ty ( VecV ) . getVectorNumElements


<s> MVT ByteTy = <STR_LIT> ( <STR_LIT> , HwLen ) ; SDValue ByteVec = DAG . getNode ( <STR_LIT> , dl , ByteTy , VecV ) ; unsigned Scale = HwLen / ty ( VecV ) . getVectorNumElements ( ) ; SDValue ScV = DAG . getConstant ( Scale , dl , <STR_LIT> ) ; IdxV = DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , IdxV , ScV ) ; SDValue ExtB = extractHvxElementReg ( ByteVec , IdxV , dl , <STR_LIT> , DAG ) ; SDValue Zero = DAG . getTargetConstant ( <NUM_LIT> , dl , <STR_LIT> ) ; return getInstr ( <STR_LIT> , dl , <STR_LIT> , {


<s> SDValue ByteIdx = convertToByteIndex ( IdxV , ElemTy , DAG ) ; SDValue ExWord = DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , { VecV , ByteIdx } ) ; if ( ElemTy == <STR_LIT> ) return ExWord ; SDValue SubIdx = getIndexInWord32 ( IdxV , ElemTy , DAG ) ; SDValue ExVec = DAG . getBitcast ( tyVector ( ty ( ExWord ) , ElemTy ) , ExWord ) ; return extractVector ( ExVec , SubIdx , dl ,


<s> for ( unsigned i = <NUM_LIT> ; i != HwLen / Rep ; ++ i ) { for ( unsigned j = <NUM_LIT> ; j != Rep ; ++ j ) Mask . push_back ( i + Offset ) ; } SDValue ShuffV = DAG . getVectorShuffle ( ByteTy , dl , ByteVec , Undef , Mask ) ; return DAG . getNode ( <STR_LIT> , dl , ResTy , ShuffV ) ; } unsigned Rep = <NUM_LIT> / ResLen ; for ( unsigned r = <NUM_LIT> ; r != HwLen / ResLen ; ++ r ) { for ( unsigned i = <NUM_LIT> ; i != ResLen ; ++ i ) { for ( unsigned j = <NUM_LIT> ; j != Rep ; ++ j ) Mask . push_back ( Offset + i * BitBytes ) ; } } SDValue Zero = getZero ( dl , <STR_LIT> , DAG ) ; SDValue ShuffV = DAG . getVectorShuffle ( ByteTy , dl , ByteVec , Undef , Mask ) ; SDValue W0 = DAG . getNode ( <STR_LIT> , dl , <STR_LIT>


<s> unsigned WordIdx = ( Idx * ElemWidth ) / <NUM_LIT> ; SDValue W0Idx = DAG . getConstant ( WordIdx , dl , <STR_LIT> ) ; SDValue W0 = extractHvxElementReg ( WordVec , W0Idx , dl , <STR_LIT> , DAG ) ; if ( ResTy . getSizeInBits ( ) == <NUM_LIT> ) return DAG . getBitcast ( ResTy , W0 ) ; SDValue W1Idx = DAG . getConstant ( WordIdx + <NUM_LIT> , dl , <STR_LIT> ) ; SDValue W1 = extractHvxElementReg ( WordVec , W1Idx , dl , <STR_LIT> , DAG ) ; SDValue WW = DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , { W1 , W0 } ) ; return DAG . getBitcast ( ResTy ,


<s> SmallVector < SDValue , <NUM_LIT> > IntOps ; IntOps . push_back ( DAG . getConstant ( IntId , dl , <STR_LIT> ) ) ; append_range ( IntOps , Ops


<s> if ( ElemTy == <STR_LIT> ) { for ( MVT T : Tys ) { assert ( T != <STR_LIT> ) ; auto A = getPreferredHvxVectorAction ( <STR_LIT> ( T , VecLen ) ) ; if ( A != ~ <NUM_LIT> ) return A ; } return ~ <NUM_LIT> ; } if ( llvm :: is_contained ( Tys


<s> unsigned HwLen = Subtarget . getVectorLength ( ) ; MVT ByteTy = <STR_LIT> ( <STR_LIT> , HwLen ) ; SDValue ByteVec = DAG . getNode ( <STR_LIT> , dl , ByteTy , VecV ) ; unsigned Scale = HwLen / ty ( VecV ) . getVectorNumElements ( ) ; SDValue ScV = DAG . getConstant ( Scale , dl , <STR_LIT> ) ; IdxV = DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , IdxV , ScV ) ; ValV = DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , ValV ) ; SDValue InsV = insertHvxElementReg ( ByteVec , IdxV , ValV , dl , DAG ) ; return DAG . getNode ( <STR_LIT> , dl , ty ( VecV )


<s> assert ( Subtarget . isHVXVectorType ( VecTy , true ) ) ; unsigned VecLen = VecTy . getVectorNumElements ( ) ; unsigned HwLen = Subtarget . getVectorLength ( ) ; assert ( HwLen % VecLen == <NUM_LIT> && <STR_LIT> ) ; unsigned Scale = VecLen / SubTy . getVectorNumElements ( ) ; unsigned BitBytes = HwLen / VecLen ; unsigned BlockLen = HwLen / Scale ; MVT ByteTy = <STR_LIT> ( <STR_LIT> , HwLen ) ; SDValue ByteVec = DAG . getNode ( <STR_LIT> , dl , ByteTy , VecV ) ; SDValue ByteSub = createHvxPrefixPred ( SubV , dl , BitBytes , false , DAG ) ; SDValue ByteIdx ; auto * IdxN = dyn_cast < ConstantSDNode > ( IdxV . getNode ( ) ) ; if ( ! IdxN || ! IdxN -> isZero ( ) ) { ByteIdx = DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , IdxV , DAG . getConstant ( BitBytes , dl , <STR_LIT> ) ) ; ByteVec = DAG . getNode ( <STR_LIT> , dl , ByteTy , ByteVec , ByteIdx ) ; } assert ( BlockLen < HwLen && <STR_LIT>


<s> V1 = DAG . getTargetExtractSubreg ( <STR_LIT> , dl , SingleTy , VecV ) ; SDValue HalfV = DAG . getConstant ( SingleTy . getVectorNumElements ( ) , dl , <STR_LIT> ) ; PickHi = DAG . getSetCC ( dl , <STR_LIT> , IdxV , HalfV , <STR_LIT> ) ; if ( isHvxSingleTy ( SubTy ) ) { if ( const auto * CN = dyn_cast < const ConstantSDNode > ( IdxV . getNode ( ) ) ) { unsigned Idx = CN -> getZExtValue ( ) ; assert ( Idx == <NUM_LIT> || Idx == VecTy . getVectorNumElements ( ) / <NUM_LIT> ) ; unsigned SubIdx = ( Idx == <NUM_LIT> ) ? <STR_LIT> : <STR_LIT> ; return DAG . getTargetInsertSubreg ( SubIdx , dl , VecTy , VecV , SubV ) ; } SDValue InLo = DAG . getNode ( <STR_LIT> , dl , VecTy , { SubV , V1 } ) ; SDValue InHi = DAG . getNode ( <STR_LIT> , dl , VecTy , { V0 , SubV } ) ; return DAG . getNode ( <STR_LIT> , dl , VecTy , PickHi , InHi , InLo ) ; } SDValue S = DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , IdxV , HalfV ) ; IdxV = DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , PickHi , S , IdxV ) ; SingleV = DAG . getNode ( <STR_LIT> , dl , SingleTy , PickHi , V1 , V0 ) ; } assert ( SubTy . getSizeInBits ( ) == <NUM_LIT> || SubTy . getSizeInBits ( ) == <NUM_LIT> ) ; auto * IdxN = dyn_cast < ConstantSDNode > ( IdxV . getNode ( ) ) ; if ( ! IdxN || ! IdxN -> isZero ( ) ) { IdxV = DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , IdxV , DAG . getConstant ( ElemWidth / <NUM_LIT> , dl , <STR_LIT> ) ) ; SingleV = DAG . getNode ( <STR_LIT> , dl , SingleTy , SingleV , IdxV ) ; } unsigned RolBase = HwLen ; if ( VecTy . getSizeInBits ( ) == <NUM_LIT> ) { SDValue V = DAG . getBitcast ( <STR_LIT> , SubV ) ; SingleV = DAG . getNode ( <STR_LIT> , dl , SingleTy , V ) ; } else { SDValue V = DAG . getBitcast ( <STR_LIT> , SubV ) ; SDValue R0 = DAG . getTargetExtractSubreg ( <STR_LIT> , dl , <STR_LIT> , V ) ; SDValue R1 = DAG . getTargetExtractSubreg ( <STR_LIT> , dl , <STR_LIT> , V ) ; SingleV = DAG . getNode ( <STR_LIT> , dl , SingleTy , SingleV , R0 ) ; SingleV = DAG . getNode ( <STR_LIT> , dl , SingleTy , SingleV , DAG . getConstant ( <NUM_LIT> , dl , <STR_LIT> ) ) ; SingleV = DAG . getNode ( <STR_LIT> , dl , SingleTy


<s> return Subtarget . isHVXVectorType ( Ty , true ) && Ty . getVectorElementType ( )


<s> auto IsHvxTy = [ this ] ( EVT Ty ) { return Ty . isSimple ( ) && Subtarget . isHVXVectorType ( Ty . getSimpleVT ( ) , true ) ; } ; auto IsHvxOp = [ this ] ( SDValue Op ) { return Op . getValueType ( ) . isSimple ( ) && Subtarget . isHVXVectorType ( ty ( Op ) , true ) ; } ; if ( llvm :: any_of ( N -> values ( ) , IsHvxTy ) || llvm :: any_of ( N -> ops ( ) , IsHvxOp ) ) return true ; auto IsWidenedToHvx = [ this , & DAG ] ( SDValue Op ) { if ( ! Op . getValueType ( ) . isSimple ( ) ) return false ; MVT ValTy = ty ( Op ) ; return ValTy . isVector ( ) && shouldWidenToHvx ( ValTy ,


<s> bool <ISA_LIT>TargetLowering :: isHvxPairTy ( MVT Ty ) const { return Subtarget . isHVXVectorType ( Ty ) && Ty . getSizeInBits ( ) == <NUM_LIT> *


<s> SDValue VQ = compressHvxPred ( Val , dl , WordTy , DAG ) ; unsigned BitWidth = ResTy . getSizeInBits ( ) ; if ( BitWidth < <NUM_LIT> ) { SDValue W0 = extractHvxElementReg ( VQ , DAG . getConstant ( <NUM_LIT> , dl , <STR_LIT> ) , dl , <STR_LIT> , DAG ) ; if ( BitWidth == <NUM_LIT> ) return W0 ; assert ( BitWidth < <NUM_LIT> ) ; return DAG . getZExtOrTrunc ( W0 , dl , ResTy ) ; } assert ( BitWidth == <NUM_LIT> || BitWidth == <NUM_LIT> ) ; SmallVector < SDValue , <NUM_LIT> > Words ; for ( unsigned i = <NUM_LIT> ; i != BitWidth / <NUM_LIT> ; ++ i ) { SDValue W = extractHvxElementReg ( VQ , DAG . getConstant ( i , dl , <STR_LIT> ) , dl , <STR_LIT> , DAG ) ; Words . push_back ( W ) ; } SmallVector < SDValue , <NUM_LIT> > Combines ; assert ( Words . size ( ) % <NUM_LIT> == <NUM_LIT> ) ; for ( unsigned i = <NUM_LIT> , e = Words . size ( ) ; i < e ; i += <NUM_LIT> ) { SDValue C = DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , { Words [ i + <NUM_LIT> ] , Words [ i ] } ) ; Combines . push_back ( C ) ; } if ( BitWidth == <NUM_LIT> ) return Combines [ <NUM_LIT> ] ; return DAG . getNode ( <STR_LIT> , dl , ResTy , Combines ) ; } if ( isHvxBoolTy ( ResTy ) && ValTy . isScalarInteger ( ) ) { unsigned BitWidth = ValTy . getSizeInBits ( ) ; unsigned HwLen = Subtarget . getVectorLength ( ) ; assert ( BitWidth == HwLen ) ; MVT ValAsVecTy = <STR_LIT> ( <STR_LIT> , BitWidth / <NUM_LIT> ) ; SDValue ValAsVec = DAG . getBitcast ( ValAsVecTy , Val ) ; SmallVector < SDValue , <NUM_LIT> > Bytes ; SmallVector < SDValue , <NUM_LIT> > Tmp ; for ( unsigned I = <NUM_LIT> ; I != HwLen / <NUM_LIT> ; ++ I ) { SDValue Idx = DAG . getConstant ( I , dl , <STR_LIT> ) ; SDValue Byte = DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , ValAsVec , Idx ) ; for ( unsigned J = <NUM_LIT> ; J != <NUM_LIT> ; ++ J ) { Bytes . push_back ( Byte )


<s> SDValue <ISA_LIT>TargetLowering :: LowerHvxBuildVector ( SDValue Op , SelectionDAG & DAG ) const { const SDLoc & dl ( Op ) ; MVT VecTy = ty ( Op ) ; unsigned Size = Op . getNumOperands ( ) ; SmallVector < SDValue , <NUM_LIT> > Ops ; for ( unsigned i = <NUM_LIT> ; i != Size ; ++ i ) Ops . push_back ( Op . getOperand ( i ) ) ; if ( VecTy . getSizeInBits ( ) == <NUM_LIT> * Subtarget . getVectorLength ( ) ) { ArrayRef < SDValue > A ( Ops ) ; MVT SingleTy = typeSplit ( VecTy ) . first ; SDValue V0 = buildHvxVectorReg ( A . take_front ( Size / <NUM_LIT> ) , dl , SingleTy , DAG ) ; SDValue V1 = buildHvxVectorReg ( A . drop_front ( Size / <NUM_LIT> ) , dl , SingleTy , DAG ) ; return DAG . getNode ( <STR_LIT> , dl , VecTy , V0 , V1 ) ; } if ( VecTy . getVectorElementType ( ) == <STR_LIT> ) return buildHvxVectorPred ( Ops , dl , VecTy , DAG ) ; if ( VecTy . getVectorElementType ( ) == <STR_LIT> ) { SmallVector < SDValue , <NUM_LIT> > NewOps ; for ( unsigned i = <NUM_LIT> ; i != Size ; i ++ ) NewOps . push_back ( DAG . getBitcast ( <STR_LIT> , Ops [ i ] ) ) ; SDValue T0 = DAG . getNode ( <STR_LIT> , dl , tyVector ( VecTy , <STR_LIT> ) , NewOps ) ; return DAG . getBitcast ( tyVector ( VecTy , <STR_LIT> ) , T0 ) ; } return buildHvxVectorReg ( Ops , dl , VecTy


<s> case <STR_LIT> : Elems [ i ] = V . getOperand ( <NUM_LIT> ) ; break ; default : llvm_unreachable ( <STR_LIT> ) ; } } } return DAG . getBuildVector ( VecTy , dl , Elems ) ; } assert ( VecTy . getVectorElementType ( ) == <STR_LIT> ) ; unsigned HwLen = Subtarget . getVectorLength ( ) ; assert ( isPowerOf2_32 ( NumOp ) && HwLen % NumOp == <NUM_LIT> ) ; SDValue Op0 = Op . getOperand ( <NUM_LIT> ) ; if ( Subtarget . isHVXVectorType ( ty ( Op0 ) , true ) ) { if ( NumOp == <NUM_LIT> ) return DAG . getNode ( <STR_LIT> , dl , VecTy , Op0 , Op . getOperand ( <NUM_LIT> ) ) ; ArrayRef < SDUse > U ( Op . getNode ( ) -> ops ( ) ) ; SmallVector < SDValue , <NUM_LIT> > SV ( U . begin ( ) , U . end ( ) ) ; ArrayRef < SDValue > Ops ( SV ) ; MVT HalfTy = typeSplit ( VecTy ) . first ; SDValue V0 = DAG . getNode ( <STR_LIT> , dl , HalfTy , Ops . take_front ( NumOp / <NUM_LIT> ) ) ; SDValue V1 = DAG . getNode ( <STR_LIT> , dl , HalfTy , Ops . take_back ( NumOp / <NUM_LIT> ) ) ; return DAG . getNode ( <STR_LIT> , dl , VecTy , V0 , V1 ) ; } unsigned BitBytes = HwLen / VecTy . getVectorNumElements ( ) ; SmallVector < SDValue , <NUM_LIT> > Prefixes ; for ( SDValue V : Op . getNode ( ) -> op_values ( ) ) { SDValue P = createHvxPrefixPred ( V , dl , BitBytes , true , DAG ) ; Prefixes . push_back ( P ) ; } unsigned InpLen = ty ( Op . getOperand ( <NUM_LIT> ) ) . getVectorNumElements ( ) ; MVT ByteTy = <STR_LIT> ( <STR_LIT> , HwLen ) ; SDValue S = DAG . getConstant ( InpLen * BitBytes


<s> SDValue InpV = Op . getOperand ( <NUM_LIT> ) ; assert ( ResTy == ty ( InpV ) ) ; MVT ElemTy = ty ( InpV ) . getVectorElementType ( ) ; unsigned ElemWidth = ElemTy . getSizeInBits ( ) ; SDValue Vec1 = DAG . getNode ( <STR_LIT> , dl , ResTy , DAG . getConstant ( <NUM_LIT> , dl , <STR_LIT> ) ) ; SDValue VecW = DAG . getNode ( <STR_LIT> , dl , ResTy , DAG . getConstant ( ElemWidth , dl , <STR_LIT> ) ) ; SDValue VecN1 = DAG . getNode ( <STR_LIT> , dl , ResTy , DAG . getConstant ( - <NUM_LIT> , dl , <STR_LIT> ) ) ; SDValue A = DAG . getNode ( <STR_LIT> , dl , ResTy , { DAG . getNode ( <STR_LIT> , dl , ResTy , { InpV , VecN1 } ) , DAG . getNode ( <STR_LIT> , dl , ResTy , { InpV , Vec1 } ) } ) ; return DAG . getNode ( <STR_LIT> , dl , ResTy , { VecW , DAG . getNode


<s> assert ( Op . getOpcode ( ) == <STR_LIT> ) ; return DAG . getNode ( <STR_LIT> , SDLoc ( Op ) , ty (

Op ) , Op . getOperand ( <NUM_LIT> ) ) ;


<s> SDValue <ISA_LIT>TargetLowering :: LowerHvxExtend ( SDValue Op , SelectionDAG & DAG ) const { assert ( Op . getOpcode


<s> if ( ElemTy == <STR_LIT> ) return extractHvxElementPred ( VecV , IdxV , dl , ty ( Op ) , DAG ) ; return extractHvxElementReg ( VecV , IdxV , dl , ty ( Op


<s> assert ( Idx % DstTy . getVectorNumElements ( ) == <NUM_LIT> ) ; ( void ) Idx ; const SDLoc & dl ( Op ) ; MVT ElemTy = SrcTy . getVectorElementType ( ) ; if ( ElemTy == <STR_LIT> ) return extractHvxSubvectorPred ( SrcV , IdxV


<s> assert ( Idx % DstTy . getVectorNumElements ( ) == <NUM_LIT> ) ; ( void ) Idx ; const SDLoc & dl ( Op ) ; MVT ElemTy = SrcTy . getVectorElementType ( ) ; if ( ElemTy == <STR_LIT> ) return extractHvxSubvectorPred ( SrcV , IdxV , dl , DstTy , DAG ) ; return extractHvxSubvectorReg ( SrcV , IdxV ,


<s> assert ( VecTy == <STR_LIT> && ArgTy == <STR_LIT> ) ; SDValue F16Vec = Op . getOperand ( <NUM_LIT> ) ; APFloat FloatVal = APFloat ( <NUM_LIT> ) ; bool Ignored ; FloatVal . convert ( APFloat :: IEEEhalf ( ) , APFloat :: rmNearestTiesToEven , & Ignored ) ; SDValue Fp16Ones = DAG . getConstantFP ( FloatVal , dl , ArgTy ) ; SDValue VmpyVec = getInstr ( <STR_LIT> , dl , VecTy , { F16Vec , Fp16Ones } , DAG ) ; MVT HalfTy = typeSplit ( VecTy ) . first ; VectorPair Pair = opSplit ( VmpyVec , dl , DAG ) ; SDValue LoVec = getInstr ( <STR_LIT> , dl , HalfTy , { Pair . first } , DAG ) ; SDValue HiVec = getInstr ( <STR_LIT> , dl , HalfTy , { Pair . second } , DAG ) ; SDValue ShuffVec = getInstr ( <STR_LIT> , dl , VecTy , { HiVec , LoVec , DAG . getConstant ( - <NUM_LIT> , dl , <STR_LIT> ) } , DAG ) ; return ShuffVec


<s> assert ( Subtarget . useHVXQFloatOps ( ) ) ; assert ( Op -> getOpcode ( ) == <STR_LIT> ) ; MVT VecTy = ty ( Op ) ; MVT ArgTy = ty ( Op . getOperand ( <NUM_LIT> ) ) ; const SDLoc & dl ( Op ) ; assert ( VecTy == <STR_LIT> && ArgTy == <STR_LIT> ) ; SDValue F16Vec = Op . getOperand ( <NUM_LIT> ) ; APFloat FloatVal = APFloat ( <NUM_LIT> ) ; bool Ignored ; FloatVal . convert ( APFloat :: IEEEhalf ( ) , APFloat :: rmNearestTiesToEven , & Ignored ) ; SDValue Fp16Ones = DAG . getConstantFP ( FloatVal , dl , ArgTy ) ; SDValue VmpyVec = getInstr ( <STR_LIT> , dl , VecTy , {


<s> SDValue ValV = Op . getOperand ( <NUM_LIT> ) ; SDValue IdxV = Op . getOperand ( <NUM_LIT> ) ; const SDLoc & dl ( Op ) ; MVT VecTy = ty ( VecV ) ; MVT ElemTy = VecTy . getVectorElementType ( ) ; if ( ElemTy == <STR_LIT> ) return insertHvxSubvectorPred ( VecV , ValV , IdxV , dl ,


<s> SDValue ValV = Op . getOperand ( <NUM_LIT> ) ; SDValue IdxV = Op . getOperand ( <NUM_LIT> ) ; const SDLoc & dl ( Op ) ; MVT VecTy = ty ( VecV ) ; MVT ElemTy = VecTy . getVectorElementType ( ) ; if ( ElemTy == <STR_LIT> ) return insertHvxSubvectorPred ( VecV , ValV , IdxV , dl , DAG ) ; return insertHvxSubvectorReg ( VecV , ValV , IdxV , dl , DAG


<s> SDValue Load = DAG . getLoad ( ValTy , dl , Chain , Base , MemOp ) ; SDValue Thru = cast < MaskedLoadSDNode > ( MaskN ) -> getPassThru ( ) ; if ( isUndef ( Thru ) ) return Load ; SDValue VSel = DAG . getNode ( <STR_LIT> , dl , ValTy , Mask , Load , Thru ) ; return DAG . getMergeValues ( { VSel , Load . getValue ( <NUM_LIT> ) } , dl ) ; } unsigned StoreOpc = <STR_LIT> ; SDValue Value = cast < MaskedStoreSDNode > ( MaskN ) -> getValue ( ) ; SDValue Offset0 = DAG . getTargetConstant ( <NUM_LIT> , dl , ty ( Base ) ) ; if ( MaskN -> getAlign ( ) . value ( ) % HwLen == <NUM_LIT> ) { SDValue Store = getInstr ( StoreOpc , dl , <STR_LIT> , { Mask , Base , Offset0 , Value , Chain } , DAG ) ; DAG . setNodeMemRefs ( cast < MachineSDNode > ( Store . getNode ( ) ) , { MemOp } ) ; return Store ; } auto StoreAlign = [ & ] ( SDValue V , SDValue A ) { SDValue Z = getZero ( dl , ty ( V ) , DAG ) ; SDValue LoV = getInstr ( <STR_LIT> , dl , ty ( V ) , { V , Z , A } , DAG ) ; SDValue HiV = getInstr ( <STR_LIT> , dl , ty ( V ) , { Z , V , A } , DAG ) ; return std :: make_pair ( LoV , HiV ) ; } ; MVT ByteTy = <STR_LIT> ( <STR_LIT> , HwLen ) ; MVT BoolTy = <STR_LIT> ( <STR_LIT> , HwLen ) ; SDValue MaskV = DAG . getNode ( <STR_LIT> , dl , ByteTy , Mask ) ; VectorPair Tmp = StoreAlign ( MaskV , Base ) ; VectorPair MaskU = { DAG . getNode ( <STR_LIT> , dl , BoolTy , Tmp . first ) , DAG . getNode ( <STR_LIT> , dl , BoolTy , Tmp . second ) } ; VectorPair ValueU = StoreAlign ( Value , Base ) ; SDValue Offset1 = DAG . getTargetConstant ( HwLen , dl , <STR_LIT> ) ; SDValue StoreLo = getInstr ( StoreOpc , dl , <STR_LIT> , { MaskU . first , Base , Offset0 , ValueU . first , Chain } , DAG ) ; SDValue StoreHi = getInstr ( StoreOpc , dl , <STR_LIT> , { MaskU . second , Base , Offset1 , ValueU .


<s> break ; case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : return SplitHvxPairOp ( Op , DAG ) ; } } switch ( Opc ) { default : break ; case <STR_LIT> : return LowerHvxBuildVector ( Op , DAG ) ; case <STR_LIT> : return LowerHvxSplatVector ( Op , DAG ) ; case <STR_LIT> : return LowerHvxConcatVectors ( Op , DAG ) ; case <STR_LIT> : return LowerHvxInsertSubvector ( Op , DAG ) ; case <STR_LIT> : return LowerHvxInsertElement ( Op , DAG ) ; case <STR_LIT> : return LowerHvxExtractSubvector ( Op , DAG ) ; case <STR_LIT> : return LowerHvxExtractElement ( Op , DAG ) ; case <STR_LIT> : return LowerHvxBitcast ( Op , DAG ) ; case <STR_LIT> : return LowerHvxAnyExt ( Op , DAG ) ; case <STR_LIT> : return LowerHvxSignExt ( Op , DAG ) ; case <STR_LIT> : return LowerHvxZeroExt ( Op , DAG ) ; case <STR_LIT> : return LowerHvxCttz ( Op , DAG ) ; case <STR_LIT> : return LowerHvxSelect ( Op , DAG ) ; case


<s> case <STR_LIT> : case <STR_LIT> : if ( shouldWidenToHvx ( ty ( Op . getOperand ( <NUM_LIT> ) ) , DAG ) ) { if ( SDValue T = WidenHvxExtend ( Op , DAG ) ) Results . push_back ( T ) ; } break ; case <STR_LIT> : if ( shouldWidenToHvx ( ty ( Op . getOperand ( <NUM_LIT> ) ) , DAG ) ) { if ( SDValue T = WidenHvxSetCC ( Op , DAG ) ) Results . push_back ( T ) ; } break ; case <STR_LIT> : if ( shouldWidenToHvx ( ty ( Op . getOperand ( <NUM_LIT> ) ) , DAG ) ) { if ( SDValue T = WidenHvxTruncate ( Op , DAG ) ) Results . push_back ( T ) ; } break ; case <STR_LIT> : { if ( shouldWidenToHvx ( ty ( cast < StoreSDNode > ( N ) -> getValue ( ) ) , DAG ) ) { SDValue Store = WidenHvxStore ( Op , DAG ) ; Results . push_back ( Store ) ; } break ; } case <STR_LIT> : if ( isHvxPairTy ( ty ( Op ) ) ) { SDValue S = SplitHvxMemOp ( Op , DAG ) ; assert ( S -> getOpcode ( ) == <STR_LIT> ) ; Results . push_back ( S . getOperand ( <NUM_LIT> ) ) ; Results . push_back ( S . getOperand ( <NUM_LIT> ) ) ; } break ; case


<s> unsigned ElemSize = HwLen / VecLen ; MVT VecTy = <STR_LIT> ( <STR_LIT> ( ElemSize * <NUM_LIT> ) , VecLen ) ; SDValue S = DAG . getNode ( <STR_LIT> , dl , VecTy , Op . getOperand ( <NUM_LIT> ) , DAG . getNode ( <STR_LIT> , dl , VecTy , Op . getOperand ( <NUM_LIT> ) ) , DAG . getNode ( <STR_LIT> , dl , VecTy , Op . getOperand ( <NUM_LIT> ) ) ) ; return DAG . getNode ( <STR_LIT> , dl , ResTy ,


<s> if ( SDValue S = getVectorShiftByInt ( Op , DAG ) ) return S ; return Op


<s> SDValue <ISA_LIT>TargetLowering :: LowerHvxSplatVector ( SDValue Op , SelectionDAG & DAG ) const { const SDLoc & dl ( Op ) ; MVT VecTy = ty ( Op ) ; MVT ArgTy = ty ( Op . getOperand ( <NUM_LIT> ) ) ; if ( ArgTy == <STR_LIT> ) { MVT SplatTy = <STR_LIT> ( <STR_LIT> , VecTy . getVectorNumElements ( ) ) ; SDValue ToInt16 = DAG . getBitcast ( <STR_LIT> , Op . getOperand ( <NUM_LIT> ) ) ; SDValue ToInt32 = DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , ToInt16 ) ; SDValue Splat = DAG . getNode ( <STR_LIT> , dl , SplatTy , ToInt32 ) ; return DAG . getBitcast ( VecTy , Splat ) ; } return SDValue (


<s> MVT ElemTy = ty ( InpV ) . getVectorElementType ( ) ; if ( ElemTy == <STR_LIT> && Subtarget . isHVXVectorType ( ResTy ) ) return extendHvxVectorPred ( InpV , SDLoc ( Op ) , ty ( Op ) , true , DAG ) ; return Op


<s> SDValue <ISA_LIT>TargetLowering :: opCastElem ( SDValue Vec , MVT ElemTy , SelectionDAG & DAG ) const { if ( ty ( Vec ) . getVectorElementType ( ) == ElemTy ) return Vec ; MVT CastTy = tyVector ( Vec . getValueType (


<s> SDValue <ISA_LIT>TargetLowering :: opJoin ( const VectorPair & Ops , const SDLoc & dl , SelectionDAG


<s> <ISA_LIT>TargetLowering :: VectorPair <ISA_LIT>TargetLowering :: opSplit ( SDValue Vec , const SDLoc & dl , SelectionDAG & DAG ) const { TypePair Tys = typeSplit ( ty ( Vec ) ) ; if ( Vec . getOpcode ( ) == <STR_LIT> ) return VectorPair ( Vec . getOperand ( <NUM_LIT> ) , Vec . getOperand (


<s> if ( SDValue T = WidenHvxExtend ( Op , DAG ) ) Results . push_back ( T ) ; } break ; case <STR_LIT> : if ( shouldWidenToHvx ( ty ( Op ) , DAG ) ) { if ( SDValue T = WidenHvxSetCC ( Op , DAG ) ) Results . push_back ( T ) ; } break ; case <STR_LIT> : if ( shouldWidenToHvx ( ty ( Op ) , DAG ) ) { if ( SDValue T = WidenHvxTruncate ( Op , DAG ) ) Results . push_back


<s> assert ( WideTy . isSimple ( ) ) ; return Subtarget . isHVXVectorType ( WideTy . getSimpleVT ( ) , true ) ; } return false


<s> if ( Op . getOpcode ( ) == <STR_LIT> ) { if ( const auto * N = dyn_cast < const VTSDNode > ( A . getNode ( ) ) ) P = SplitVTNode ( N ) ; } OpsL . push_back ( P . first ) ; OpsH . push_back ( P . second ) ; } MVT ResTy = ty ( Op ) ; MVT HalfTy = typeSplit ( ResTy ) . first ; SDValue L = DAG . getNode ( Op . getOpcode ( ) , dl , HalfTy , OpsL ) ; SDValue H = DAG . getNode ( Op . getOpcode ( ) , dl , HalfTy , OpsH ) ; SDValue S = DAG . getNode ( <STR_LIT> , dl , ResTy , L , H ) ; return S


<s> MVT NewElemTy = <STR_LIT> ( ElemTy . getSizeInBits ( ) * Factor ) ; return <STR_LIT> ( NewElemTy , VecTy . getVectorNumElements


<s> MVT HalfTy = <STR_LIT> ( VecTy . getVectorElementType ( ) , NumElem / <NUM_LIT> ) ; return { HalfTy , HalfTy


<s> MVT ElemTy = VecTy . getVectorElementType ( ) ; MVT NewElemTy = <STR_LIT> ( ElemTy . getSizeInBits ( ) / Factor ) ; return <STR_LIT> ( NewElemTy , VecTy . getVectorNumElements ( )


<s> const SDLoc & dl ( Op ) ; unsigned HwWidth = <NUM_LIT> * Subtarget . getVectorLength ( ) ; SDValue Op0 = Op . getOperand ( <NUM_LIT> ) ; MVT ResTy = ty ( Op ) ; MVT OpTy = ty ( Op0 ) ; if ( ! Subtarget . isHVXElementType ( OpTy ) || ! Subtarget . isHVXElementType ( ResTy ) ) return SDValue ( ) ; auto getFactor = [ HwWidth ] ( MVT Ty ) { unsigned Width = Ty . getSizeInBits ( ) ; return HwWidth > Width ? HwWidth / Width : <NUM_LIT> ; } ; auto getWideTy = [ getFactor ] ( MVT Ty ) { unsigned WideLen = Ty . getVectorNumElements ( ) * getFactor ( Ty ) ; return <STR_LIT> ( Ty . getVectorElementType ( ) , WideLen ) ; } ; unsigned Opcode = Op . getOpcode ( ) == <STR_LIT> ? <STR_LIT> : <STR_LIT> ; SDValue WideOp = appendUndef ( Op0 , getWideTy ( OpTy ) , DAG ) ; SDValue WideRes = DAG . getNode ( Opcode , dl , getWideTy ( ResTy ) , WideOp ) ; return WideRes


<s> SDValue <ISA_LIT>TargetLowering :: WidenHvxExtend ( SDValue Op , SelectionDAG & DAG ) const { const SDLoc & dl ( Op ) ; unsigned HwWidth = <NUM_LIT> * Subtarget . getVectorLength ( ) ; SDValue Op0 = Op . getOperand ( <NUM_LIT> ) ; MVT ResTy = ty ( Op ) ; MVT OpTy = ty ( Op0 ) ; if ( ! Subtarget . isHVXElementType ( OpTy ) || ! Subtarget . isHVXElementType ( ResTy ) ) return SDValue ( ) ; auto getFactor = [ HwWidth ] ( MVT Ty ) { unsigned Width = Ty . getSizeInBits ( ) ; return HwWidth > Width ? HwWidth / Width : <NUM_LIT> ; } ; auto getWideTy = [ getFactor ] ( MVT Ty ) { unsigned WideLen = Ty . getVectorNumElements ( ) * getFactor ( Ty ) ; return <STR_LIT> ( Ty . getVectorElementType ( ) , WideLen ) ; } ; unsigned Opcode = Op . getOpcode ( ) == <STR_LIT> ? <STR_LIT> : <STR_LIT> ; SDValue WideOp = appendUndef ( Op0 , getWideTy ( OpTy ) , DAG ) ; SDValue WideRes = DAG . getNode ( Opcode , dl , getWideTy ( ResTy ) ,


<s> const SDLoc & dl ( Op ) ; auto * LoadN = cast < LoadSDNode > ( Op . getNode ( ) ) ; assert ( LoadN -> isUnindexed ( ) && <STR_LIT> ) ; assert ( LoadN -> getMemoryVT ( ) . getVectorElementType ( ) != <STR_LIT> && <STR_LIT> ) ; SDValue Chain = LoadN -> getChain ( ) ; SDValue Base = LoadN -> getBasePtr ( ) ; SDValue Offset = DAG . getUNDEF ( <STR_LIT> ) ; MVT ResTy = ty ( Op ) ; unsigned HwLen = Subtarget . getVectorLength ( ) ; unsigned ResLen = ResTy . getStoreSize ( ) ; assert ( ResLen < HwLen && <STR_LIT> ) ; MVT BoolTy = <STR_LIT> ( <STR_LIT> , HwLen ) ; SDValue Mask = getInstr ( <STR_LIT> , dl , BoolTy , { DAG . getConstant ( ResLen , dl , <STR_LIT> ) } , DAG ) ; MVT LoadTy = <STR_LIT> ( <STR_LIT> , HwLen ) ; MachineFunction & MF = DAG . getMachineFunction ( ) ; auto * MemOp = MF . getMachineMemOperand ( LoadN -> getMemOperand ( ) , <NUM_LIT> , HwLen ) ; SDValue Load = DAG . getMaskedLoad ( LoadTy , dl , Chain , Base , Offset , Mask , DAG . getUNDEF ( LoadTy ) , LoadTy , MemOp , <STR_LIT> , <STR_LIT> , false ) ; SDValue Value = opCastElem ( Load , ResTy . getVectorElementType


<s> unsigned HwLen = Subtarget . getVectorLength ( ) ; unsigned WideOpLen = ( <NUM_LIT> * HwLen ) / ElemTy . getSizeInBits ( ) ; assert ( WideOpLen * ElemTy . getSizeInBits ( ) == <NUM_LIT> * HwLen ) ; MVT WideOpTy = <STR_LIT> ( ElemTy , WideOpLen ) ; if ( ! Subtarget . isHVXVectorType ( WideOpTy , true ) ) return SDValue ( ) ; SDValue WideOp0 = appendUndef ( Op0 , WideOpTy , DAG ) ; SDValue WideOp1 = appendUndef ( Op1 , WideOpTy , DAG ) ; EVT ResTy = getSetCCResultType ( DAG . getDataLayout ( ) , * DAG . getContext ( ) , WideOpTy ) ; SDValue SetCC = DAG . getNode ( <STR_LIT> , dl , ResTy , { WideOp0 , WideOp1 , Op . getOperand ( <NUM_LIT> ) } ) ; EVT RetTy = getTypeToTransformTo ( * DAG . getContext ( ) , ty ( Op ) ) ; return DAG . getNode ( <STR_LIT> , dl , RetTy , {

SetCC , getZero ( dl , <STR_LIT> , DAG ) } ) ;


<s> auto * StoreN = cast < StoreSDNode > ( Op . getNode ( ) ) ; assert ( StoreN -> isUnindexed ( ) && <STR_LIT> ) ; assert ( StoreN -> getMemoryVT ( ) . getVectorElementType ( ) != <STR_LIT> && <STR_LIT> ) ; SDValue Chain = StoreN -> getChain ( ) ; SDValue Base = StoreN -> getBasePtr ( ) ; SDValue Offset = DAG . getUNDEF ( <STR_LIT> ) ; SDValue Value = opCastElem ( StoreN -> getValue ( ) , <STR_LIT> , DAG ) ; MVT ValueTy = ty ( Value ) ; unsigned ValueLen = ValueTy .


<s> } ; auto getWideTy = [ getFactor ] ( MVT Ty ) { unsigned WideLen = Ty . getVectorNumElements ( ) * getFactor ( Ty ) ; return <STR_LIT> ( Ty . getVectorElementType ( ) , WideLen ) ; } ; if ( Subtarget . isHVXVectorType ( OpTy ) ) return DAG . getNode ( <STR_LIT> , dl , getWideTy ( ResTy ) , Op0 ) ; assert ( ! isTypeLegal ( OpTy ) && <STR_LIT> ) ; SDValue WideOp = appendUndef ( Op0 , getWideTy ( OpTy )


<s> const SDLoc & dl ( Op ) ; unsigned HwWidth = <NUM_LIT> * Subtarget . getVectorLength ( ) ; SDValue Op0 = Op . getOperand ( <NUM_LIT> ) ; MVT ResTy = ty ( Op ) ; MVT OpTy = ty ( Op0 ) ; if ( ! Subtarget . isHVXElementType ( OpTy ) || ! Subtarget . isHVXElementType ( ResTy ) ) return SDValue ( ) ; auto getFactor = [ HwWidth ] ( MVT Ty ) { unsigned Width = Ty . getSizeInBits ( ) ; assert ( HwWidth % Width == <NUM_LIT> ) ; return HwWidth / Width ; } ; auto getWideTy = [ getFactor ] ( MVT Ty ) { unsigned WideLen = Ty . getVectorNumElements ( ) * getFactor ( Ty ) ; return <STR_LIT> ( Ty . getVectorElementType ( ) , WideLen ) ; } ; if ( Subtarget . isHVXVectorType ( OpTy ) ) return DAG . getNode ( <STR_LIT> , dl , getWideTy ( ResTy ) , Op0 ) ; assert ( ! isTypeLegal ( OpTy


<s> MVT PartVT = <STR_LIT> ( VecTy . getVectorElementType ( ) , OpsPerWord ) ; for ( unsigned i = <NUM_LIT> ; i != VecLen ; i += OpsPerWord ) { SDValue W = buildVector32 ( Values . slice ( i , OpsPerWord ) , dl , PartVT , DAG ) ; Words . push_back ( DAG . getBitcast ( <STR_LIT> , W ) ) ; } } else { Words . assign ( Values . begin ( ) , Values . end ( ) ) ; } unsigned NumWords = Words . size ( ) ; bool IsSplat = true , IsUndef = true ; SDValue SplatV ; for ( unsigned i = <NUM_LIT> ; i != NumWords && IsSplat ; ++ i ) { if ( isUndef ( Words [ i ] ) ) continue ; IsUndef = false ; if ( ! SplatV . getNode ( ) ) SplatV = Words [ i ] ; else if ( SplatV != Words [ i ] ) IsSplat = false ; } if ( IsUndef ) return DAG . getUNDEF ( VecTy ) ; if ( IsSplat ) { assert ( SplatV . getNode ( ) ) ; auto * IdxN = dyn_cast < ConstantSDNode > ( SplatV . getNode ( ) ) ; if ( IdxN && IdxN -> isNullValue ( ) ) return getZero ( dl , VecTy , DAG ) ; return DAG . getNode ( <STR_LIT> , dl , VecTy , SplatV ) ; } SmallVector < ConstantInt * , <NUM_LIT> > Consts ( VecLen ) ; bool AllConst = getBuildVectorConstInts ( Values , VecTy , DAG , Consts ) ; if ( AllConst ) { ArrayRef < Constant * > Tmp ( ( Constant * * ) Consts . begin ( ) , ( Constant * * ) Consts . end ( ) ) ; Constant * CV = ConstantVector :: get ( Tmp ) ; unsigned Align = HwLen ; SDValue CP = LowerConstantPool ( DAG . getConstantPool ( CV , VecTy , Align ) , DAG ) ; return DAG . getLoad ( VecTy , dl , DAG . getEntryNode ( ) , CP , MachinePointerInfo :: getConstantPool ( MF ) , Align ) ; } assert ( <NUM_LIT> * Words . size ( ) == Subtarget . getVectorLength ( ) ) ; SDValue HalfV0 = getInstr ( <STR_LIT> , dl , VecTy , { } , DAG ) ; SDValue HalfV1 = getInstr ( <STR_LIT> , dl , VecTy , { } , DAG ) ; SDValue S = DAG . getConstant ( <NUM_LIT> , dl , <STR_LIT> ) ; for ( unsigned i = <NUM_LIT> ; i != NumWords / <NUM_LIT> ; ++ i


<s> if ( IsSplat ) { assert ( SplatV . getNode ( ) ) ; auto * IdxN = dyn_cast < ConstantSDNode > ( SplatV . getNode ( ) ) ; if ( IdxN && IdxN -> isNullValue ( ) ) return getZero ( dl , VecTy , DAG ) ; return DAG . getNode ( <STR_LIT> , dl , VecTy , SplatV ) ; } SmallVector < ConstantInt * , <NUM_LIT> > Consts ( VecLen ) ; bool AllConst = getBuildVectorConstInts ( Values , VecTy , DAG , Consts ) ; if ( AllConst ) { ArrayRef < Constant * > Tmp ( ( Constant * * ) Consts . begin ( ) , ( Constant * * ) Consts . end ( ) ) ; Constant * CV = ConstantVector :: get ( Tmp ) ; unsigned Align = HwLen ; SDValue CP = LowerConstantPool ( DAG . getConstantPool ( CV , VecTy , Align ) , DAG ) ; return DAG . getLoad ( VecTy , dl , DAG . getEntryNode ( ) , CP , MachinePointerInfo :: getConstantPool ( MF ) , Align ) ; } assert ( <NUM_LIT> * Words . size ( ) == Subtarget . getVectorLength ( ) ) ; SDValue HalfV0 = getInstr ( <STR_LIT> , dl , VecTy , { } , DAG ) ; SDValue HalfV1 = getInstr ( <STR_LIT> , dl , VecTy , { } , DAG ) ; SDValue S = DAG . getConstant ( <NUM_LIT> , dl , <STR_LIT> ) ; for ( unsigned i = <NUM_LIT> ; i != NumWords / <NUM_LIT> ; ++ i ) { SDValue N = DAG . getNode ( <STR_LIT> , dl , VecTy , { HalfV0 , Words [ i ]


<s> return DAG . getZeroExtendVectorInReg ( Op . getOperand ( <NUM_LIT> ) , SDLoc ( Op )


<s> if ( IsPairOp ) { switch ( Opc ) { default : break ; case <STR_LIT> : case <STR_LIT> : return SplitHvxMemOp ( Op , DAG ) ; case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : return SplitHvxPairOp ( Op , DAG ) ; } } switch ( Opc ) { default : break ; case <STR_LIT> : return LowerHvxBuildVector ( Op , DAG ) ; case <STR_LIT> : return LowerHvxConcatVectors ( Op , DAG ) ; case <STR_LIT> : return LowerHvxInsertSubvector ( Op , DAG ) ; case <STR_LIT> : return LowerHvxInsertElement ( Op , DAG ) ; case <STR_LIT> : return LowerHvxExtractSubvector ( Op , DAG ) ; case <STR_LIT> : return LowerHvxExtractElement ( Op , DAG ) ; case <STR_LIT> : return LowerHvxAnyExt ( Op , DAG ) ; case <STR_LIT> : return LowerHvxSignExt ( Op , DAG ) ; case <STR_LIT> : return LowerHvxZeroExt ( Op , DAG ) ; case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : return LowerHvxShift ( Op , DAG ) ; case <STR_LIT> : return LowerHvxMul ( Op , DAG ) ; case <STR_LIT> : case <STR_LIT> : return LowerHvxMulh ( Op , DAG ) ; case <STR_LIT> : return LowerHvxExtend ( Op , DAG ) ; case <STR_LIT> : case <STR_LIT> : return Op ; case <STR_LIT> : return SDValue ( ) ; } Op . dumpr ( & DAG ) ; llvm_unreachable ( <STR_LIT>


<s> } switch ( Opc ) { default : break ; case <STR_LIT> : return LowerHvxBuildVector ( Op , DAG ) ; case <STR_LIT> : return LowerHvxConcatVectors ( Op , DAG ) ; case <STR_LIT> : return LowerHvxInsertSubvector ( Op , DAG ) ; case <STR_LIT> : return LowerHvxInsertElement ( Op , DAG ) ; case <STR_LIT> : return LowerHvxExtractSubvector ( Op , DAG ) ; case <STR_LIT> : return LowerHvxExtractElement ( Op , DAG ) ; case <STR_LIT> : return LowerHvxAnyExt ( Op , DAG ) ; case <STR_LIT> : return LowerHvxSignExt ( Op , DAG ) ; case <STR_LIT> : return LowerHvxZeroExt ( Op , DAG ) ; case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : return LowerHvxShift


<s> SDValue BaseNoOff = ( BaseOpc != <STR_LIT> ) ? DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , BO . first ) : BO . first ; SDValue Base0 = DAG . getMemBasePlusOffset ( BaseNoOff , BO . second , dl ) ; SDValue Base1 = DAG . getMemBasePlusOffset ( BaseNoOff , BO . second + HwLen , dl ) ; MachineMemOperand * WideMMO = nullptr ; if ( MachineMemOperand * MMO = LN -> getMemOperand ( ) ) { MachineFunction & MF = DAG . getMachineFunction ( ) ; WideMMO = MF . getMachineMemOperand ( MMO -> getPointerInfo ( ) , MMO -> getFlags ( ) , <NUM_LIT> * HwLen , HwLen , MMO -> getAAInfo ( ) , MMO -> getRanges ( ) , MMO -> getSyncScopeID ( ) , MMO -> getOrdering ( ) , MMO -> getFailureOrdering ( ) ) ; } SDValue Load0 = DAG . getLoad ( VecTy , dl , Chain , Base0 , WideMMO ) ; SDValue Load1 = DAG . getLoad ( VecTy , dl , Chain , Base1 , WideMMO ) ; SDValue Aligned = getInstr ( <STR_LIT> , dl , VecTy , { Load1 , Load0

, BaseNoOff . getOperand ( <NUM_LIT> ) } , DAG ) ;


<s> if ( VecTy . getVectorElementType ( ) != <STR_LIT> ) { assert ( ( ElemSize == <NUM_LIT> || ElemSize == <NUM_LIT> ) && <STR_LIT> ) ; unsigned OpsPerWord = ( ElemSize == <NUM_LIT> ) ? <NUM_LIT> : <NUM_LIT> ; MVT PartVT = <STR_LIT> ( VecTy . getVectorElementType ( ) , OpsPerWord ) ; for ( unsigned i = <NUM_LIT> ; i != VecLen ; i += OpsPerWord ) { SDValue W = buildVector32 ( Values . slice ( i , OpsPerWord ) , dl , PartVT , DAG ) ; Words . push_back ( DAG . getBitcast ( <STR_LIT> , W ) ) ; } } else { Words . assign ( Values . begin ( ) , Values . end ( ) ) ; } unsigned NumWords = Words . size ( ) ; bool IsSplat = true , IsUndef = true ; SDValue SplatV ; for ( unsigned i = <NUM_LIT> ; i != NumWords && IsSplat ; ++ i ) { if ( isUndef ( Words [ i ] ) ) continue ; IsUndef = false ; if ( ! SplatV . getNode ( ) ) SplatV = Words [ i ] ; else if ( SplatV != Words [ i ] ) IsSplat = false ; } if ( IsUndef ) return DAG . getUNDEF ( VecTy ) ; if ( IsSplat ) { assert ( SplatV . getNode ( ) ) ; auto * IdxN = dyn_cast < ConstantSDNode > ( SplatV . getNode ( ) ) ; if ( IdxN && IdxN -> isNullValue ( ) ) return getZero ( dl , VecTy , DAG ) ; MVT WordTy = <STR_LIT> ( <STR_LIT> , HwLen / <NUM_LIT> ) ; SDValue S = DAG . getNode ( <STR_LIT> , dl , WordTy , SplatV ) ; return DAG . getBitcast ( VecTy , S ) ; } SmallVector < ConstantInt * , <NUM_LIT> > Consts ( VecLen ) ; bool AllConst = getBuildVectorConstInts ( Values , VecTy , DAG , Consts ) ; if ( AllConst ) { ArrayRef < Constant * > Tmp ( ( Constant * * ) Consts . begin ( ) , ( Constant * * ) Consts . end ( ) ) ; Constant * CV = ConstantVector :: get ( Tmp ) ; Align Alignment ( HwLen ) ; SDValue CP = LowerConstantPool ( DAG . getConstantPool ( CV , VecTy , Alignment ) , DAG ) ; return DAG . getLoad ( VecTy , dl , DAG . getEntryNode ( ) , CP , MachinePointerInfo :: getConstantPool ( MF ) , Alignment ) ; } auto IsBuildFromExtracts = [ this , & Values ] ( SDValue & SrcVec , SmallVectorImpl < int > & SrcIdx ) { SDValue Vec ; for ( SDValue V : Values ) { if ( isUndef ( V ) ) { SrcIdx . push_back ( - <NUM_LIT> ) ; continue ; } if ( V . getOpcode ( ) != <STR_LIT> ) return false ; SDValue T = V . getOperand (


<s> SDValue C0 = Cond . getOperand ( <NUM_LIT> ) , C1 = Cond . getOperand ( <NUM_LIT> ) ; if ( C1 -> getOpcode ( ) == <STR_LIT> ) return DAG . getNode ( <STR_LIT> , dl , ty ( Op ) , C0 , Ops [ <NUM_LIT> ] , Ops [ <NUM_LIT> ] ) ; } break ; } case <STR_LIT> : if ( Ops [ <NUM_LIT> ] . getOpcode ( ) == <STR_LIT> ) { if ( const auto * C = dyn_cast < ConstantSDNode > ( Ops [ <NUM_LIT> ] . getOperand ( <NUM_LIT> ) ) ) return C -> isNullValue ( ) ? DAG . getNode ( <STR_LIT> , dl , ty ( Op ) ) : DAG . getNode ( <STR_LIT> , dl , ty ( Op ) ) ; } break ; case <STR_LIT> : if ( Ops [ <NUM_LIT> ] . getOpcode ( ) == <STR_LIT> ) return DAG . getNode ( <STR_LIT> , dl , ty ( Op ) , DAG . getConstant ( - <NUM_LIT> , dl , <STR_LIT> ) ) ; if ( Ops [ <NUM_LIT> ] . getOpcode ( ) == <STR_LIT> ) return getZero ( dl , ty ( Op ) , DAG ) ; break ; case <STR_LIT> : if ( isUndef ( Ops [ <NUM_LIT> ] ) ) return Ops [ <NUM_LIT> ] ; ; break ; case <STR_LIT> : { if ( Ops [ <NUM_LIT> ] . getOpcode ( ) == <STR_LIT> ) { SDValue Vec = Ops [ <NUM_LIT> ] . getOperand ( <NUM_LIT> ) ; SDValue Rot0 = Ops [ <NUM_LIT> ] , Rot1 = Ops [ <NUM_LIT> ] . getOperand ( <NUM_LIT> ) ; SDValue Rot = DAG . getNode ( <STR_LIT> , dl ,


<s> SDValue Base1 = DAG . getMemBasePlusOffset ( Base0 , TypeSize :: Fixed ( HwLen ) , dl ) ; MachineMemOperand * MOp0 = nullptr , * MOp1 = nullptr ; if ( MachineMemOperand * MMO = MemN -> getMemOperand ( ) ) { MachineFunction & MF = DAG . getMachineFunction ( ) ; MOp0 = MF . getMachineMemOperand ( MMO , <NUM_LIT> , HwLen ) ; MOp1 = MF . getMachineMemOperand ( MMO , HwLen , HwLen ) ; } unsigned MemOpc = MemN -> getOpcode ( ) ; if ( MemOpc == <STR_LIT> ) { assert ( cast < LoadSDNode > ( Op ) -> isUnindexed ( ) ) ; SDValue Load0 = DAG . getLoad ( SingleTy , dl , Chain , Base0 , MOp0 ) ; SDValue Load1 = DAG . getLoad ( SingleTy , dl , Chain , Base1 , MOp1 ) ; return DAG . getMergeValues ( { DAG . getNode ( <STR_LIT> , dl , MemTy , Load0 , Load1 ) , DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , Load0 . getValue ( <NUM_LIT> ) , Load1 . getValue ( <NUM_LIT> ) ) } , dl ) ; } if ( MemOpc == <STR_LIT> ) { assert ( cast < StoreSDNode > ( Op ) -> isUnindexed ( ) ) ; VectorPair Vals = opSplit ( cast < StoreSDNode > ( Op ) -> getValue ( ) , dl , DAG ) ; SDValue Store0 = DAG . getStore ( Chain , dl , Vals . first , Base0 , MOp0 ) ; SDValue Store1 = DAG . getStore ( Chain , dl , Vals . second , Base1 , MOp1 ) ; return DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , Store0 , Store1 ) ; } assert ( MemOpc == <STR_LIT> || MemOpc == <STR_LIT> ) ; auto MaskN = cast < MaskedLoadStoreSDNode > ( Op ) ; assert ( MaskN -> isUnindexed ( ) ) ; VectorPair Masks = opSplit ( MaskN -> getMask ( ) , dl , DAG ) ; SDValue Offset = DAG . getUNDEF ( <STR_LIT> ) ; if ( MemOpc == <STR_LIT> ) { VectorPair Thru = opSplit ( cast < MaskedLoadSDNode > ( Op


<s> SDValue Bytes = DAG . getLoad ( ByteTy , dl , DAG . getEntryNode ( ) , CP , MachinePointerInfo :: getConstantPool ( MF ) , Align ) ; SDValue Sel = DAG . getSelect ( dl , VecTy , VecQ , DAG . getBitcast ( VecTy , Bytes ) , getZero ( dl , VecTy , DAG ) ) ; SDValue All1 = DAG . getSplatBuildVector ( <STR_LIT> , dl , DAG . getConstant ( <NUM_LIT> , dl , <STR_LIT> ) ) ; SDValue Vrmpy = getInstr ( <STR_LIT> , dl , ByteTy , { Sel , All1 } , DAG ) ; SDValue Rot = getInstr ( <STR_LIT> , dl , ByteTy , { Vrmpy , Vrmpy , DAG . getTargetConstant ( <NUM_LIT> , dl , <STR_LIT> ) } , DAG ) ; SDValue Vor = DAG . getNode ( <STR_LIT> , dl , ByteTy , { Vrmpy , Rot } ) ; SmallVector < int ,


<s> for ( unsigned j = <NUM_LIT> ; j != <NUM_LIT> ; ++ j ) Tmp . push_back ( ConstantInt :: get ( Int8Ty , <NUM_LIT> << j ) ) ; } Constant * CV = ConstantVector :: get ( Tmp ) ; unsigned Align = HwLen ; SDValue CP = LowerConstantPool ( DAG . getConstantPool ( CV , ByteTy , Align ) , DAG ) ; SDValue Bytes = DAG . getLoad ( ByteTy , dl , DAG . getEntryNode ( ) , CP , MachinePointerInfo :: getConstantPool ( MF ) , Align ) ; SDValue Sel = DAG . getSelect ( dl , VecTy , VecQ , DAG . getBitcast ( VecTy , Bytes ) , getZero ( dl , VecTy , DAG ) ) ; SDValue All1 = DAG . getSplatBuildVector ( <STR_LIT> , dl , DAG . getConstant ( <NUM_LIT> , dl , <STR_LIT> ) ) ; SDValue Vrmpy = getInstr ( <STR_LIT> , dl , ByteTy , { Sel , All1 } , DAG ) ; SDValue Rot = getInstr ( <STR_LIT> , dl , ByteTy , { Vrmpy , Vrmpy , DAG . getTargetConstant ( <NUM_LIT> , dl , <STR_LIT> ) } , DAG ) ; SDValue Vor = DAG . getNode ( <STR_LIT> , dl , ByteTy , { Vrmpy , Rot } ) ; SmallVector < int , <NUM_LIT> > Mask ; for ( unsigned i = <NUM_LIT> ; i != HwLen ; ++ i ) Mask . push_back ( ( <NUM_LIT> * i ) % HwLen + i / ( HwLen / <NUM_LIT> ) ) ; SDValue Collect = DAG . getVectorShuffle ( ByteTy , dl , Vor , DAG . getUNDEF ( ByteTy ) ,


<s> if ( VecTy . getSizeInBits ( ) > <NUM_LIT> * Subtarget . getVectorLength ( ) ) return false ; if ( ! Subtarget . isHVXVectorType ( VecTy , false ) ) return false ; if ( Fast ) * Fast =


<s> bool <ISA_LIT>TargetLowering :: allowsHvxMisalignedMemoryAccesses ( MVT VecTy , unsigned Align , MachineMemOperand :: Flags Flags , bool * Fast ) const { if ( ! Subtarget . isHVXVectorType ( VecTy ) ) return false ; if ( Fast ) * Fast


<s> bool <ISA_LIT>TargetLowering :: allowsHvxMisalignedMemoryAccesses ( MVT VecTy , unsigned Align , MachineMemOperand :: Flags Flags , bool * Fast ) const { if ( ! Subtarget . isHVXVectorType ( VecTy ) ) return false ; if ( Fast ) * Fast = true ; return true


<s> } unsigned NumWords = Words . size ( ) ; bool IsSplat = true , IsUndef = true ; SDValue SplatV ; for ( unsigned i = <NUM_LIT> ; i != NumWords && IsSplat ; ++ i ) { if ( isUndef ( Words [ i ] ) ) continue ; IsUndef = false ; if ( ! SplatV . getNode ( ) ) SplatV = Words [ i ] ; else if ( SplatV != Words [ i ] ) IsSplat = false ; } if ( IsUndef ) return DAG . getUNDEF ( VecTy ) ; if ( IsSplat ) { assert ( SplatV . getNode ( ) ) ; auto * IdxN = dyn_cast < ConstantSDNode > ( SplatV . getNode ( ) ) ; if ( IdxN && IdxN -> isNullValue ( ) ) return getZero ( dl , VecTy , DAG ) ; return DAG . getNode ( <STR_LIT> , dl , VecTy , SplatV ) ; } SmallVector < ConstantInt * , <NUM_LIT> > Consts ( VecLen ) ; bool AllConst = getBuildVectorConstInts ( Values , VecTy , DAG , Consts ) ; if ( AllConst ) { ArrayRef < Constant * > Tmp ( ( Constant * * ) Consts . begin ( ) , ( Constant * * ) Consts . end ( ) ) ; Constant * CV = ConstantVector :: get ( Tmp ) ; Align Alignment ( HwLen ) ; SDValue CP = LowerConstantPool ( DAG . getConstantPool ( CV , VecTy , Alignment ) , DAG ) ; return DAG . getLoad ( VecTy , dl , DAG . getEntryNode ( ) , CP , MachinePointerInfo :: getConstantPool ( MF ) , Alignment ) ; } auto IsBuildFromExtracts = [ this , & Values ] ( SDValue & SrcVec , SmallVectorImpl < int > & SrcIdx ) { SDValue Vec ; for ( SDValue V : Values ) { if ( isUndef ( V ) ) { SrcIdx . push_back ( - <NUM_LIT> ) ; continue ; } if ( V . getOpcode ( ) != <STR_LIT> ) return false ; SDValue T = V . getOperand


<s> return Ty . isSimple ( ) && Subtarget . isHVXVectorType ( Ty . getSimpleVT ( ) , true ) ; } ; auto IsHvxOp = [ this ] ( SDValue Op ) { return Subtarget . isHVXVectorType ( ty ( Op ) , true ) ; } ; return llvm :: any_of ( N -> values ( ) , IsHvxTy )

|| llvm :: any_of ( N -> ops ( ) , IsHvxOp ) ;


<s> return DAG . getZExtOrTrunc ( W0 , dl , ResTy ) ; } assert ( BitWidth == <NUM_LIT> || BitWidth == <NUM_LIT> ) ; SmallVector < SDValue , <NUM_LIT> > Words ; for ( unsigned i = <NUM_LIT> ; i != BitWidth / <NUM_LIT> ; ++ i ) { SDValue W = extractHvxElementReg ( VQ , DAG . getConstant ( i , dl , <STR_LIT> ) , dl , <STR_LIT> , DAG ) ; Words . push_back ( W ) ; } SmallVector < SDValue , <NUM_LIT> > Combines ; assert ( Words . size ( ) % <NUM_LIT> == <NUM_LIT> ) ; for ( unsigned i = <NUM_LIT> , e = Words . size ( ) ; i < e ; i += <NUM_LIT> ) { SDValue C = DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , { Words [ i + <NUM_LIT> ] , Words [ i ] } ) ; Combines . push_back ( C ) ; } if ( BitWidth == <NUM_LIT> )


<s> const SDLoc & dl ( N ) ; SDValue Op ( N , <NUM_LIT> ) ; unsigned Opc = Op . getOpcode ( ) ; if ( Opc == <STR_LIT> ) { SDValue Cond = Op . getOperand ( <NUM_LIT> ) ; if ( Cond -> getOpcode ( ) == <STR_LIT> ) { SDValue C0 = Cond . getOperand ( <NUM_LIT> ) , C1 = Cond . getOperand ( <NUM_LIT> ) ; if ( C1 -> getOpcode


<s> MVT VecTy ; int Length ; if ( Subtarget . useHVX64BOps ( ) ) { VecTy = <STR_LIT> ( <STR_LIT> , <NUM_LIT> ) ; Length = <NUM_LIT> ; } if ( Subtarget . useHVX128BOps ( ) ) { VecTy = <STR_LIT> ( <STR_LIT> , <NUM_LIT> ) ; Length = <NUM_LIT> ; } SDValue C8421 = DAG . getTargetConstant ( <NUM_LIT> , dl , <STR_LIT> ) ; SDValue InstrC8421 = getInstr ( <STR_LIT> , dl , <STR_LIT> , C8421 , DAG ) ; SDValue Vand = getInstr ( <STR_LIT> , dl , VecTy , { Op , InstrC8421 } , DAG ) ; SDValue C2 = DAG . getTargetConstant ( <NUM_LIT> , dl , <STR_LIT> ) ; SDValue Valign = getInstr ( <STR_LIT> , dl , VecTy , { Vand , Vand , C2 } , DAG ) ; SDValue Vor = getInstr ( <STR_LIT> , dl , VecTy , { Vand , Valign } , DAG ) ; SDValue C1 = DAG . getTargetConstant ( <NUM_LIT> , dl , <STR_LIT> ) ; SDValue Valign1 = getInstr ( <STR_LIT> , dl , VecTy , { Vor , Vor , C1 } , DAG ) ; SDValue Vor1 = getInstr ( <STR_LIT> , dl , VecTy , { Vor , Valign1 } , DAG ) ; SDValue Cff = DAG . getTargetConstant ( <NUM_LIT> , dl , <STR_LIT> ) ; SDValue InstrCff = getInstr ( <STR_LIT> , dl , <STR_LIT> , Cff , DAG ) ; SDValue Vsplat = getInstr ( <STR_LIT> , dl , VecTy , InstrCff , DAG ) ; SDValue Vand1 = getInstr ( <STR_LIT> , dl , VecTy , { Vor1 , Vsplat } , DAG ) ; SmallVector < SDValue , <NUM_LIT> > Elems ; for ( int i = <NUM_LIT> ; i < Length ; ++ i ) { Elems . push_back ( DAG . getConstant ( <NUM_LIT> , dl , <STR_LIT> ) ) ; Elems . push_back ( DAG . getConstant ( <NUM_LIT> , dl , <STR_LIT> ) ) ; Elems . push_back ( DAG . getConstant ( <NUM_LIT> , dl , <STR_LIT> ) ) ; Elems . push_back ( DAG . getConstant ( <NUM_LIT> , dl , <STR_LIT> ) ) ; Elems . push_back ( DAG . getConstant ( <NUM_LIT> ,


<s> SDValue <ISA_LIT>TargetLowering :: LowerHvxStore ( SDValue Op , SelectionDAG & DAG ) const { auto * N = Op . getNode ( ) ; const SDLoc & dl ( Op ) ; SDValue Q0 = N -> getOperand ( <NUM_LIT> ) ; EVT VTOp = Q0 . getNode ( ) -> getValueType ( <NUM_LIT> ) ; if ( Op . getOpcode ( ) != <STR_LIT> || VTOp != <STR_LIT> ) return Op ; SDValue Vor4 = HvxVecPredBitcastComputation ( Q0 , DAG ) ; SDValue C0 = DAG . getTargetConstant ( <NUM_LIT> , dl , <STR_LIT> ) ; SDValue InstrC0 = getInstr ( <STR_LIT> , dl , <STR_LIT> , C0 , DAG ) ; SDValue Vextract0 = getInstr ( <STR_LIT> , dl , <STR_LIT> , { Vor4 , InstrC0 } , DAG ) ; SDValue C32 = DAG . getTargetConstant ( <NUM_LIT> , dl , <STR_LIT> ) ; SDValue InstrC32 = getInstr ( <STR_LIT> , dl , <STR_LIT> , C32 , DAG ) ; SDValue Vextract1 = getInstr ( <STR_LIT> , dl , <STR_LIT> , { Vor4 , InstrC32 } , DAG ) ; SDValue Combine0 = getInstr ( <STR_LIT> , dl , <STR_LIT> , { Vextract1 , Vextract0 } , DAG ) ; SDValue C64 = DAG . getTargetConstant ( <NUM_LIT> , dl , <STR_LIT> ) ; SDValue InstrC64 = getInstr ( <STR_LIT> , dl , <STR_LIT> , C64 , DAG ) ; SDValue Vextract2 = getInstr ( <STR_LIT> , dl , <STR_LIT> , { Vor4 , InstrC64 } , DAG ) ; SDValue C96 = DAG . getTargetConstant ( <NUM_LIT> , dl , <STR_LIT> ) ; SDValue InstrC96 = getInstr ( <STR_LIT> , dl , <STR_LIT> , C96 , DAG ) ; SDValue Vextract3 = getInstr ( <STR_LIT> , dl , <STR_LIT> , { Vor4 , InstrC96 } , DAG ) ; SDValue Combine1 = getInstr ( <STR_LIT> , dl , <STR_LIT> , { Vextract3 , Vextract2 } , DAG ) ; StoreSDNode * ST = cast < StoreSDNode > ( Op . getNode ( ) ) ; SDValue C8 = DAG . getTargetConstant (


<s> const SDLoc & dl ( Op ) ; SmallVector < int , <NUM_LIT> > ShuffMask ; MVT ElemTy = ResTy . getVectorElementType ( ) ; unsigned VecLen = ResTy . getVectorNumElements ( ) ; SDValue Vs = Op . getOperand ( <NUM_LIT> ) ; SDValue Vt = Op . getOperand ( <NUM_LIT> ) ; switch ( ElemTy . SimpleTy ) { case <STR_LIT> : { MVT ExtTy = typeExtElem ( ResTy , <NUM_LIT> ) ; unsigned MpyOpc = ElemTy == <STR_LIT> ? <STR_LIT> : <STR_LIT> ; SDValue M = getInstr ( MpyOpc , dl , ExtTy , { Vs , Vt } , DAG ) ; for ( unsigned I = <NUM_LIT> ; I < VecLen ; I += <NUM_LIT> ) { ShuffMask . push_back ( I ) ; ShuffMask . push_back ( I + VecLen ) ; } VectorPair P = opSplit ( opCastElem ( M , ElemTy , DAG ) , dl , DAG ) ; SDValue BS = getByteShuffle ( dl , P . first , P . second , ShuffMask , DAG ) ; return DAG . getBitcast ( ResTy , BS ) ; } case <STR_LIT> : return getInstr ( <STR_LIT> , dl , ResTy , { Vs , Vt } , DAG ) ; case <STR_LIT> : { auto MulL_V60 = [ & ] ( SDValue Vs , SDValue Vt ) { SDValue S16 = DAG . getConstant ( <NUM_LIT> , dl , <STR_LIT> ) ; SDValue T0 = getInstr ( <STR_LIT> , dl , ResTy , { Vs , Vt } , DAG ) ; SDValue T1 = getInstr ( <STR_LIT> , dl , ResTy , { T0 , S16 } , DAG ) ; SDValue T2 = getInstr ( <STR_LIT> , dl , ResTy , { T1 , Vs , Vt } , DAG ) ; return T2 ; } ; auto MulL_V62 = [ & ] ( SDValue Vs , SDValue Vt


<s> SDValue <ISA_LIT>TargetLowering :: PerformHvxDAGCombine ( SDNode * N , DAGCombinerInfo & DCI ) const { if ( DCI . isBeforeLegalizeOps ( ) ) return SDValue ( ) ; const SDLoc & dl ( N ) ; SelectionDAG & DAG = DCI . DAG ; SDValue Op ( N , <NUM_LIT> ) ; unsigned Opc = Op . getOpcode ( ) ; switch ( Opc ) { case <STR_LIT> : { SDValue Cond = Op . getOperand ( <NUM_LIT> ) ; if ( Cond -> getOpcode ( ) == <STR_LIT> ) { SDValue C0 = Cond . getOperand ( <NUM_LIT> ) , C1 = Cond . getOperand ( <NUM_LIT> ) ; if ( C1 -> getOpcode ( ) == <STR_LIT> ) return DAG . getNode ( <STR_LIT> , dl , ty ( Op ) , C0 , Op . getOperand ( <NUM_LIT> ) , Op . getOperand ( <NUM_LIT> ) ) ; } break ; } case <STR_LIT> : if ( isUndef ( Op . getOperand ( <NUM_LIT> ) ) ) return Op . getOperand ( <NUM_LIT> ) ; break ; case <STR_LIT> : { SDValue Op0 = Op . getOperand


<s> unsigned HwLen = Subtarget . getVectorLength ( ) ; assert ( VecLen <= HwLen || VecLen == <NUM_LIT> * HwLen ) ; SmallVector < SDValue , <NUM_LIT> > Bytes ; if ( VecLen <= HwLen ) { assert ( HwLen % VecLen == <NUM_LIT> ) ; unsigned BitBytes = HwLen / VecLen ; for ( SDValue V : Values ) { SDValue Ext = ! V . isUndef ( ) ? DAG . getZExtOrTrunc ( V , dl , <STR_LIT> ) : DAG . getConstant ( <NUM_LIT> , dl , <STR_LIT> ) ; for ( unsigned B = <NUM_LIT> ; B != BitBytes ; ++ B ) Bytes . push_back ( Ext ) ; } } else { for ( unsigned I = <NUM_LIT> ; I != VecLen ; I += <NUM_LIT> ) { unsigned B = <NUM_LIT> ; for ( ; B != <NUM_LIT> ; ++ B ) { if ( ! Values [ I + B ] . isUndef ( ) ) break ; } SDValue F = Values [ I + B ] ; SDValue Ext = ( B < <NUM_LIT> ) ? DAG . getZExtOrTrunc ( F , dl , <STR_LIT> ) : DAG . getConstant ( <NUM_LIT> , dl , <STR_LIT> ) ; Bytes . push_back ( Ext ) ; for ( ; B != <NUM_LIT> ; ++ B ) assert ( Values [ I + B ] . isUndef ( ) || Values [ I + B ] == F ) ; } } MVT ByteTy = <STR_LIT> (


<s> unsigned Align = HwLen ; SDValue CP = LowerConstantPool ( DAG . getConstantPool ( CV , VecTy , Align ) , DAG ) ; return DAG . getLoad ( VecTy , dl , DAG . getEntryNode ( ) , CP , MachinePointerInfo :: getConstantPool ( MF ) , Align ) ; } unsigned ElemSize = ElemWidth / <NUM_LIT> ; assert ( ElemSize * VecLen == HwLen ) ; SmallVector < SDValue , <NUM_LIT> > Words ; if ( VecTy . getVectorElementType ( ) != <STR_LIT> ) { assert ( ( ElemSize == <NUM_LIT> || ElemSize == <NUM_LIT> ) && <STR_LIT> ) ; unsigned OpsPerWord = ( ElemSize == <NUM_LIT> ) ? <NUM_LIT> : <NUM_LIT> ; MVT PartVT = <STR_LIT> ( VecTy . getVectorElementType ( ) , OpsPerWord ) ; for ( unsigned i = <NUM_LIT> ; i != VecLen ; i += OpsPerWord ) { SDValue W = buildVector32 ( Values . slice ( i , OpsPerWord ) , dl , PartVT , DAG ) ; Words . push_back ( DAG . getBitcast ( <STR_LIT> , W ) ) ; } } else { Words . assign ( Values . begin ( ) , Values . end ( ) ) ; } assert ( <NUM_LIT> * Words . size ( ) == Subtarget . getVectorLength ( ) ) ; SDValue HalfV0 = getNode ( <STR_LIT> , dl , VecTy , { } , DAG ) ; SDValue HalfV1 = getNode ( <STR_LIT> , dl ,


<s> for ( unsigned i = <NUM_LIT> ; i != Size ; ++ i ) Ops . push_back ( Op . getOperand ( i ) ) ; if ( VecTy . getVectorElementType ( ) == <STR_LIT> ) return buildHvxVectorPred ( Ops , dl , VecTy , DAG ) ; if ( VecTy . getSizeInBits ( ) == <NUM_LIT> * Subtarget


<s> SDValue V0 = buildHvxVectorSingle ( A . take_front ( Size / <NUM_LIT> ) , dl , SingleTy , DAG ) ; SDValue V1 = buildHvxVectorSingle ( A . drop_front ( Size / <NUM_LIT> ) , dl , SingleTy , DAG ) ; return DAG . getNode ( <STR_LIT> , dl , VecTy , V0 , V1 ) ; } return buildHvxVectorSingle ( Ops , dl , VecTy ,


<s> SDValue ValV = Op . getOperand ( <NUM_LIT> ) ; SDValue IdxV = Op . getOperand ( <NUM_LIT> ) ; MVT ElemTy = ty ( VecV ) . getVectorElementType ( ) ; unsigned ElemWidth = ElemTy . getSizeInBits ( ) ; assert ( ElemWidth >= <NUM_LIT> && ElemWidth <= <NUM_LIT> ) ; ( void ) ElemWidth ; auto InsertWord = [ & DAG , & dl , this ] ( SDValue VecV , SDValue ValV , SDValue ByteIdxV ) { MVT VecTy = ty ( VecV ) ; unsigned HwLen = Subtarget . getVectorLength ( ) ; SDValue MaskV = DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , { ByteIdxV , DAG . getConstant ( - <NUM_LIT> , dl , <STR_LIT> ) } ) ; SDValue RotV = DAG . getNode ( <STR_LIT> , dl , VecTy , { VecV , MaskV } ) ; SDValue InsV = DAG . getNode ( <STR_LIT> , dl , VecTy , { RotV , ValV } ) ; SDValue SubV = DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , { DAG . getConstant ( HwLen / <NUM_LIT> , dl , <STR_LIT> ) , MaskV } ) ; SDValue TorV = DAG . getNode ( <STR_LIT> , dl , VecTy , { InsV , SubV } ) ; return TorV ; } ; SDValue ByteIdx = convertToByteIndex ( IdxV , ElemTy , DAG ) ; if ( ElemTy == <STR_LIT> ) return InsertWord ( VecV , ValV , ByteIdx ) ; SDValue WordIdx = DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , { ByteIdx , DAG . getConstant ( <NUM_LIT> , dl , <STR_LIT> ) } ) ; SDValue Ex0 = DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , { opCastElem ( VecV , <STR_LIT> , DAG ) , WordIdx } ) ; SDValue Ext = LowerHvxExtractElement ( Ex0 , DAG ) ; SDValue SubIdx = getIndexInWord32 ( IdxV , ElemTy , DAG ) ; MVT SubVecTy = tyVector ( ty ( Ext ) , ElemTy ) ; SDValue Ins = insertVector ( DAG . getBitcast ( SubVecTy , Ext ) , ValV , SubIdx ,


<s> SDValue ValV = Op . getOperand ( <NUM_LIT> ) ; SDValue IdxV = Op . getOperand ( <NUM_LIT> ) ; MVT ElemTy = ty ( VecV ) . getVectorElementType ( ) ; unsigned ElemWidth = ElemTy . getSizeInBits ( ) ; assert ( ElemWidth >= <NUM_LIT> && ElemWidth <= <NUM_LIT> ) ; ( void ) ElemWidth ; auto InsertWord = [ & DAG , & dl , this ] ( SDValue VecV , SDValue ValV , SDValue ByteIdxV ) { MVT VecTy = ty ( VecV ) ; unsigned HwLen = Subtarget . getVectorLength ( ) ; SDValue MaskV = DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , { ByteIdxV , DAG . getConstant ( - <NUM_LIT> , dl , <STR_LIT> ) } ) ; SDValue RotV = DAG . getNode ( <STR_LIT> , dl , VecTy , { VecV , MaskV } ) ; SDValue InsV = DAG . getNode ( <STR_LIT> , dl , VecTy , { RotV , ValV } ) ; SDValue SubV = DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , { DAG . getConstant ( HwLen / <NUM_LIT> , dl , <STR_LIT> ) , MaskV } ) ; SDValue TorV = DAG . getNode ( <STR_LIT> , dl , VecTy , { InsV , SubV } ) ; return TorV ; } ; SDValue ByteIdx = convertToByteIndex ( IdxV , ElemTy , DAG ) ; if ( ElemTy == <STR_LIT> ) return InsertWord ( VecV , ValV , ByteIdx ) ; SDValue WordIdx = DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , { ByteIdx , DAG . getConstant ( <NUM_LIT> , dl , <STR_LIT> ) } ) ; SDValue Ex0 = DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , { opCastElem ( VecV , <STR_LIT> , DAG ) , WordIdx } ) ; SDValue Ext = LowerHvxExtractElement ( Ex0 , DAG ) ; SDValue SubIdx = getIndexInWord32 ( IdxV , ElemTy ,


<s> SDValue IdxV = Op . getOperand ( <NUM_LIT> ) ; auto * IdxN = dyn_cast < ConstantSDNode > ( IdxV . getNode ( ) ) ; if ( ! IdxN ) return SDValue ( ) ; unsigned Idx = IdxN -> getZExtValue ( ) ; SDValue DstV = Op . getOperand ( <NUM_LIT> ) ; SDValue SrcV = Op . getOperand ( <NUM_LIT> ) ; MVT DstTy = ty ( DstV ) ; MVT SrcTy = ty ( SrcV


<s> MVT ElemTy = VecTy . getVectorElementType ( ) ; unsigned ElemWidth = ElemTy . getSizeInBits ( ) ; assert ( isPowerOf2_32 ( ElemWidth ) ) ; auto getIdx = [ ] ( unsigned Code ) { static const unsigned Idx [ ] = { <STR_LIT> , <STR_LIT> , <STR_LIT> } ; for ( unsigned I = <NUM_LIT> , E = array_lengthof ( Idx ) ; I != E ; ++ I ) if ( Code == Idx [ I ] ) return I ; llvm_unreachable ( <STR_LIT> ) ; } ; static unsigned OpcTable [ <NUM_LIT> ] [ <NUM_LIT> ] = { { <STR_LIT> , <STR_LIT> , <STR_LIT> } , { <STR_LIT> , <STR_LIT> , <STR_LIT> } , { <STR_LIT> , <STR_LIT> , <STR_LIT> } } ; unsigned CmpOpc = OpcTable [ Log2_32 ( ElemWidth ) - <NUM_LIT> ] [ getIdx


<s> MVT VecTy = ty ( Op . getOperand ( <NUM_LIT> ) ) ; assert ( VecTy == ty ( Op . getOperand ( <NUM_LIT> ) ) ) ; SDValue Cmp = Op . getOperand ( <NUM_LIT> ) ; <STR_LIT> CC = cast < CondCodeSDNode > ( Cmp ) -> get ( ) ; bool Negate = false , Swap = false ; switch ( CC ) { case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : CC = <STR_LIT> ( CC , true ) ; Negate = true ; break ; default : break ; } switch ( CC ) { case <STR_LIT> : case <STR_LIT> : CC = <STR_LIT> ( CC ) ; Swap = true ; break ; default : break ; } assert ( CC == <STR_LIT> || CC == <STR_LIT> || CC == <STR_LIT> ) ; MVT ElemTy = VecTy


<s> TypePair Tys = typeSplit ( ty ( Vec ) ) ; return DAG . SplitVector ( Vec , dl , Tys .


<s> <ISA_LIT>TargetLowering :: VectorPair <ISA_LIT>TargetLowering :: opSplit ( SDValue Vec , const SDLoc & dl , SelectionDAG & DAG ) const { TypePair Tys = typeSplit ( ty ( Vec ) ) ; return DAG . SplitVector ( Vec , dl ,


<s> SmallVector < SDValue , <NUM_LIT> > Ops ; for ( unsigned i = <NUM_LIT> ; i != Size ; ++ i ) Ops . push_back ( Op . getOperand ( i ) ) ; if ( VecTy . getVectorElementType ( ) == <STR_LIT> ) return buildHvxVectorPred ( Ops , dl , VecTy , DAG ) ; if ( VecTy . getVectorElementType ( ) == <STR_LIT> ) { SmallVector < SDValue , <NUM_LIT> > NewOps ; for ( unsigned i = <NUM_LIT> ; i != Size ; i ++ ) NewOps . push_back ( DAG . getBitcast ( <STR_LIT> , Ops [ i ] ) ) ; SDValue T0 = DAG . getNode ( <STR_LIT> , dl , tyVector ( VecTy , <STR_LIT> ) , NewOps ) ; return DAG . getBitcast ( tyVector ( VecTy , <STR_LIT> ) , T0 ) ; } if ( VecTy . getSizeInBits ( ) == <NUM_LIT> * Subtarget . getVectorLength ( ) ) { ArrayRef < SDValue > A ( Ops ) ; MVT SingleTy = typeSplit ( VecTy ) . first ; SDValue V0 = buildHvxVectorReg ( A . take_front ( Size / <NUM_LIT> ) , dl , SingleTy , DAG ) ; SDValue V1 = buildHvxVectorReg ( A . drop_front ( Size / <NUM_LIT> ) , dl , SingleTy , DAG ) ; return DAG . getNode ( <STR_LIT> , dl , VecTy , V0


<s> case <STR_LIT> : case <STR_LIT> : return SplitHvxPairOp ( Op , DAG ) ; } } switch ( Opc ) { default : break ; case <STR_LIT> : return LowerHvxBuildVector ( Op , DAG ) ; case <STR_LIT> : return LowerHvxConcatVectors ( Op , DAG ) ; case <STR_LIT> : return LowerHvxInsertSubvector ( Op , DAG ) ; case <STR_LIT> : return LowerHvxInsertElement ( Op , DAG ) ; case <STR_LIT> : return LowerHvxExtractSubvector ( Op , DAG ) ; case <STR_LIT> : return LowerHvxExtractElement ( Op , DAG ) ; case <STR_LIT> : return LowerHvxAnyExt ( Op , DAG ) ; case <STR_LIT> : return LowerHvxSignExt ( Op , DAG ) ; case <STR_LIT> : return LowerHvxZeroExt ( Op , DAG ) ; case <STR_LIT> : return LowerHvxCttz ( Op , DAG ) ; case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : return LowerHvxShift ( Op , DAG ) ; case <STR_LIT> : return LowerHvxMul ( Op , DAG ) ; case <STR_LIT>


<s> case <STR_LIT> : case <STR_LIT> : return SplitHvxMemOp ( Op , DAG ) ; case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : return SplitHvxPairOp ( Op , DAG ) ; } } switch ( Opc ) { default : break ; case <STR_LIT> : return LowerHvxBuildVector ( Op , DAG ) ; case <STR_LIT> : return LowerHvxConcatVectors ( Op , DAG ) ; case <STR_LIT> : return LowerHvxInsertSubvector ( Op , DAG ) ; case <STR_LIT> : return LowerHvxInsertElement ( Op , DAG ) ; case <STR_LIT> : return LowerHvxExtractSubvector ( Op , DAG ) ; case <STR_LIT> : return LowerHvxExtractElement ( Op , DAG ) ; case <STR_LIT> : return LowerHvxBitcast ( Op , DAG ) ; case <STR_LIT> : return LowerHvxAnyExt ( Op , DAG ) ; case <STR_LIT> : return LowerHvxSignExt ( Op , DAG ) ; case <STR_LIT> : return LowerHvxZeroExt ( Op , DAG ) ; case <STR_LIT> : return LowerHvxCttz ( Op , DAG ) ; case <STR_LIT> : return LowerHvxSelect ( Op , DAG ) ; case <STR_LIT> : case


<s> void <ISA_LIT>TargetLowering :: LowerHvxOperationWrapper ( SDNode * N , SmallVectorImpl < SDValue > & Results , SelectionDAG & DAG ) const { unsigned Opc = N -> getOpcode ( ) ; SDValue Op ( N , <NUM_LIT> ) ; switch ( Opc ) { case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : assert ( shouldWidenToHvx ( ty ( Op . getOperand ( <NUM_LIT> ) ) , DAG ) && <STR_LIT> ) ; if ( SDValue T = WidenHvxExtend ( Op , DAG ) ) Results . push_back ( T ) ; break ; case <STR_LIT> : assert ( shouldWidenToHvx ( ty ( Op . getOperand ( <NUM_LIT> ) ) , DAG ) && <STR_LIT> ) ; if ( SDValue T = WidenHvxTruncate ( Op , DAG ) ) Results . push_back ( T ) ; break ; case <STR_LIT> : { assert ( shouldWidenToHvx ( ty ( cast < StoreSDNode > ( N ) -> getValue ( ) ) , DAG ) && <STR_LIT> ) ; SDValue Store = WidenHvxStore ( SDValue ( N , <NUM_LIT> ) , DAG ) ; Results . push_back ( Store ) ; break ; } case <STR_LIT> : if ( isHvxPairTy ( ty ( Op ) ) ) { SDValue S = SplitHvxMemOp ( Op , DAG ) ; assert ( S -> getOpcode ( ) == <STR_LIT> ) ; Results . push_back ( S . getOperand ( <NUM_LIT> ) ) ; Results . push_back ( S . getOperand ( <NUM_LIT> ) ) ; } break ; case <STR_LIT> : if ( isHvxPairTy ( ty ( Op -> getOperand ( <NUM_LIT> ) ) ) ) { SDValue S = SplitHvxMemOp ( Op


<s> auto Action = getPreferredHvxVectorAction ( Ty ) ; if ( Action == TargetLoweringBase :: TypeWidenVector ) { EVT WideTy = getTypeToTransformTo ( * DAG . getContext ( ) , Ty ) ; assert ( WideTy .


<s> if ( Cond -> getOpcode ( ) == <STR_LIT> ) { SDValue C0 = Cond . getOperand ( <NUM_LIT> ) , C1 = Cond . getOperand ( <NUM_LIT> ) ; if ( C1 -> getOpcode ( ) == <STR_LIT> ) return DAG . getNode ( <STR_LIT> , dl , ty ( Op ) , C0 , Op . getOperand ( <NUM_LIT> ) , Op . getOperand ( <NUM_LIT> ) ) ; } break ; } case <STR_LIT> : if ( isUndef ( Op . getOperand ( <NUM_LIT> ) ) ) return Op . getOperand ( <NUM_LIT> ) ; break ; case <STR_LIT> : { SDValue Op0 = Op . getOperand ( <NUM_LIT> ) ; if ( Op0 . getOpcode (


<s> MVT ElemTy = VecTy . getVectorElementType ( ) ; unsigned VecLen = VecTy . getVectorNumElements ( ) ; unsigned HwLen = Subtarget . getVectorLength ( ) ; if ( ElemTy == <STR_LIT> && VecLen == <NUM_LIT> * HwLen ) return TargetLoweringBase :: TypeSplitVector ; if ( ElemTy == <STR_LIT> ) ElemTy = <STR_LIT> ; ArrayRef < MVT > Tys = Subtarget . getHVXElementTypes ( ) ; if ( llvm :: is_contained ( Tys , ElemTy ) ) { unsigned VecWidth = VecTy . getSizeInBits ( ) ; bool HaveThreshold = HvxWidenThreshold . getNumOccurrences


<s> if ( ! SplatV . getNode ( ) ) SplatV = Words [ i ] ; else if ( SplatV != Words [ i ] ) IsSplat = false ; } if ( IsSplat ) { assert ( SplatV . getNode ( ) ) ; return DAG . getNode ( <STR_LIT> , dl , VecTy , SplatV ) ; } assert ( <NUM_LIT> * Words . size ( ) == Subtarget . getVectorLength ( ) ) ; SDValue HalfV0 = getNode ( <STR_LIT> , dl , VecTy , { } , DAG ) ; SDValue HalfV1 = getNode ( <STR_LIT> , dl , VecTy , { } , DAG ) ; SDValue S = DAG . getConstant ( <NUM_LIT> , dl , <STR_LIT> ) ; for ( unsigned i = <NUM_LIT> ; i != NumWords / <NUM_LIT> ; ++ i ) { SDValue N = DAG . getNode ( <STR_LIT> , dl , VecTy , { HalfV0 , Words [ i ] } ) ; SDValue M = DAG . getNode ( <STR_LIT> , dl , VecTy , { HalfV1 , Words [ i + NumWords / <NUM_LIT> ] } ) ; HalfV0 = DAG . getNode ( <STR_LIT> , dl , VecTy , { N , S } ) ; HalfV1 = DAG . getNode ( <STR_LIT> , dl , VecTy , { M , S } ) ; } HalfV0 = DAG . getNode ( <STR_LIT> , dl , VecTy , { HalfV0 ,

DAG . getConstant ( HwLen / <NUM_LIT> , dl , <STR_LIT> ) } ) ;


<s> MVT PredTy = ty ( PredV ) ; unsigned HwLen = Subtarget . getVectorLength ( ) ; MVT ByteTy = <STR_LIT> ( <STR_LIT> , HwLen ) ; if ( Subtarget . isHVXVectorType ( PredTy , true ) ) { SDValue T = DAG . getNode ( <STR_LIT> , dl , ByteTy , PredV ) ; SmallVector < int , <NUM_LIT> > Mask ( HwLen ) ; unsigned Scale = HwLen / ( PredTy . getVectorNumElements ( ) * BitBytes ) ; unsigned BlockLen = PredTy . getVectorNumElements ( ) * BitBytes ; for ( unsigned i = <NUM_LIT> ; i != HwLen ; ++ i ) { unsigned Num = i % Scale ; unsigned Off = i / Scale ; Mask [ BlockLen * Num + Off ] = i ; } SDValue S = DAG . getVectorShuffle ( ByteTy , dl , T , DAG . getUNDEF ( ByteTy ) , Mask ) ; if ( ! ZeroFill ) return S ; MVT BoolTy = <STR_LIT> ( <STR_LIT> , HwLen ) ; SDValue Q = getNode ( <STR_LIT> , dl , BoolTy , { DAG . getConstant ( BlockLen , dl , <STR_LIT> ) } , DAG ) ; SDValue M = DAG . getNode ( <STR_LIT> , dl , ByteTy , Q ) ; return DAG . getNode ( <STR_LIT> , dl , ByteTy , S , M ) ; } assert ( PredTy == <STR_LIT> || PredTy == <STR_LIT> || PredTy == <STR_LIT> ) ; unsigned Bytes = <NUM_LIT> / PredTy . getVectorNumElements ( ) ; SmallVector < SDValue , <NUM_LIT> > Words [ <NUM_LIT> ] ; unsigned IdxW = <NUM_LIT> ; auto Lo32 = [ & DAG , & dl ] ( SDValue P ) { return DAG . getTargetExtractSubreg ( <STR_LIT> , dl , <STR_LIT> , P ) ; } ; auto Hi32 = [ & DAG , & dl ] ( SDValue P ) { return DAG . getTargetExtractSubreg ( <STR_LIT> , dl , <STR_LIT> , P ) ; } ; SDValue W0 = isUndef ( PredV ) ? DAG . getUNDEF ( <STR_LIT> ) : DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , PredV ) ; Words [ IdxW ] . push_back ( Hi32 ( W0 ) ) ; Words [ IdxW ] . push_back ( Lo32 ( W0 ) ) ; while ( Bytes < BitBytes ) { IdxW ^= <NUM_LIT> ; Words [ IdxW ] . clear ( ) ; if ( Bytes < <NUM_LIT> ) { for ( const SDValue & W : Words [ IdxW ^ <NUM_LIT> ] ) { SDValue T = expandPredicate ( W , dl , DAG ) ; Words [ IdxW ] . push_back ( Hi32 ( T ) ) ; Words [ IdxW ] . push_back ( Lo32 ( T ) ) ; } } else


<s> SDValue ByteVec = DAG . getNode ( <STR_LIT> , dl , ByteTy , VecV ) ; unsigned Scale = HwLen / ty ( VecV ) . getVectorNumElements ( ) ; SDValue ScV = DAG . getConstant ( Scale , dl , <STR_LIT> ) ; IdxV = DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , IdxV , ScV ) ; SDValue ExtB = extractHvxElementReg ( ByteVec , IdxV , dl , <STR_LIT> , DAG ) ; SDValue Zero = DAG . getTargetConstant ( <NUM_LIT> , dl , <STR_LIT> ) ; return getNode ( <STR_LIT> , dl , <STR_LIT> , { ExtB , Zero } ,


<s> if ( Idx * ElemWidth >= <NUM_LIT> * HwLen ) { SubIdx = <STR_LIT> ; Idx -= VecTy . getVectorNumElements ( ) / <NUM_LIT> ; } else { SubIdx = <STR_LIT> ; } VecTy = typeSplit ( VecTy ) . first ; VecV = DAG . getTargetExtractSubreg ( SubIdx , dl , VecTy , VecV ) ; if ( VecTy == ResTy ) return VecV ; } assert ( ResTy . getSizeInBits ( ) == <NUM_LIT> || ResTy . getSizeInBits ( ) == <NUM_LIT> ) ; MVT WordTy = tyVector ( VecTy , <STR_LIT> ) ; SDValue WordVec = DAG . getBitcast ( WordTy , VecV ) ; unsigned WordIdx = ( Idx * ElemWidth ) / <NUM_LIT> ; SDValue W0Idx = DAG . getConstant (


<s> unsigned BitBytes = HwLen / VecLen ; unsigned BlockLen = HwLen / Scale ; MVT ByteTy = <STR_LIT> ( <STR_LIT> , HwLen ) ; SDValue ByteVec = DAG . getNode ( <STR_LIT> , dl , ByteTy , VecV ) ; SDValue ByteSub = createHvxPrefixPred ( SubV , dl , BitBytes , false , DAG ) ; SDValue ByteIdx ; auto * IdxN = dyn_cast < ConstantSDNode > ( IdxV . getNode ( ) ) ; if ( ! IdxN || ! IdxN -> isNullValue ( ) ) { ByteIdx = DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , IdxV , DAG . getConstant ( BitBytes , dl , <STR_LIT> ) ) ; ByteVec = DAG . getNode ( <STR_LIT> , dl , ByteTy , ByteVec , ByteIdx ) ; } MVT BoolTy = <STR_LIT> ( <STR_LIT> , HwLen ) ; SDValue Q = getNode ( <STR_LIT> , dl , BoolTy , { DAG . getConstant ( BlockLen , dl , <STR_LIT> ) } , DAG ) ; ByteVec = getNode ( <STR_LIT> , dl , ByteTy , { Q , ByteSub , ByteVec } , DAG ) ; if ( ! IdxN || ! IdxN -> isNullValue ( ) ) { SDValue HwLenV = DAG . getConstant ( HwLen , dl , <STR_LIT> ) ; SDValue ByteXdi = DAG . getNode ( <STR_LIT> , dl ,


<s> unsigned NumOp = Op . getNumOperands ( ) ; assert ( isPowerOf2_32 ( NumOp ) && HwLen % NumOp == <NUM_LIT> ) ; ( void ) NumOp ; unsigned BitBytes = HwLen / VecTy . getVectorNumElements ( ) ; SmallVector < SDValue , <NUM_LIT> > Prefixes ; for ( SDValue V : Op . getNode ( ) -> op_values ( ) ) { SDValue P = createHvxPrefixPred ( V , dl , BitBytes , true , DAG ) ; Prefixes . push_back ( P ) ; } unsigned InpLen = ty ( Op . getOperand ( <NUM_LIT> ) ) . getVectorNumElements ( ) ; MVT ByteTy = <STR_LIT> ( <STR_LIT> , HwLen ) ; SDValue S = DAG . getConstant ( InpLen * BitBytes , dl


<s> ShuffMask . push_back ( I + VecLen + <NUM_LIT> ) ; } VectorPair P = opSplit ( opCastElem ( M , ElemTy , DAG ) , dl , DAG ) ; SDValue BS = getByteShuffle ( dl , P . first , P . second , ShuffMask , DAG ) ; return DAG . getBitcast ( ResTy , BS ) ; } assert ( ElemTy == <STR_LIT> ) ; SDValue S16 = DAG . getConstant ( <NUM_LIT> , dl , <STR_LIT> ) ; if ( IsSigned ) { SDValue T0 = getNode ( <STR_LIT> , dl , ResTy , { Vt , Vs } , DAG ) ; SDValue S0 = getNode ( <STR_LIT> , dl , ResTy , { Vs , S16 } , DAG ) ; SDValue T1 = getNode ( <STR_LIT> , dl , ResTy , { T0 , S0 , Vt } , DAG ) ; SDValue S2 = getNode ( <STR_LIT> , dl , ResTy , { T1 , S16 } , DAG ) ; SDValue T2 = getNode ( <STR_LIT> , dl , ResTy , { S0 , Vt } , DAG ) ; SDValue T3 = DAG . getNode ( <STR_LIT> , dl , ResTy , { S2 , T2 } ) ; return T3 ; } auto LoVec = [ & DAG , ResTy , dl ] ( SDValue Pair ) { return DAG . getTargetExtractSubreg ( <STR_LIT> , dl , ResTy , Pair ) ; } ; auto HiVec = [ & DAG , ResTy , dl ] ( SDValue Pair ) { return DAG . getTargetExtractSubreg ( <STR_LIT> , dl , ResTy , Pair ) ; } ; MVT PairTy = typeJoin ( { ResTy , ResTy } ) ; SDValue P = getNode ( <STR_LIT> , dl , ResTy , { DAG . getConstant ( <NUM_LIT> , dl , <STR_LIT> ) } , DAG ) ; SDValue T0 = getNode ( <STR_LIT> , dl , PairTy , { Vs , Vt } , DAG ) ; SDValue T1 = getNode ( <STR_LIT> , dl , ResTy , { LoVec ( T0 ) , S16 } , DAG ) ; SDValue D0 = getNode ( <STR_LIT> , dl , ResTy , { Vt , P } , DAG ) ; SDValue T2 = getNode ( <STR_LIT> , dl , PairTy ,


<s> bool AllT = true , AllF = true ; auto IsTrue = [ ] ( SDValue V ) { if ( const auto * N = dyn_cast < ConstantSDNode > ( V . getNode ( ) ) ) return ! N -> isNullValue ( ) ; return false ; } ; auto IsFalse = [ ] ( SDValue V ) { if ( const auto * N = dyn_cast < ConstantSDNode > ( V . getNode ( ) ) ) return N -> isNullValue ( ) ; return false ; } ; if ( VecLen <= HwLen ) { assert ( HwLen % VecLen == <NUM_LIT> ) ; unsigned BitBytes = HwLen / VecLen ; for ( SDValue V : Values ) { AllT &= IsTrue ( V ) ; AllF &= IsFalse ( V ) ; SDValue Ext = ! V . isUndef ( ) ? DAG . getZExtOrTrunc ( V , dl , <STR_LIT> ) : DAG . getUNDEF ( <STR_LIT> ) ; for ( unsigned B = <NUM_LIT> ; B != BitBytes ; ++ B ) Bytes . push_back ( Ext ) ; } } else { for ( unsigned I = <NUM_LIT> ; I != VecLen ; I += <NUM_LIT> ) { unsigned B = <NUM_LIT> ; for ( ; B != <NUM_LIT> ; ++ B ) { if ( ! Values [ I + B ] . isUndef ( ) ) break ; } SDValue F = Values [ I + B ] ; AllT &= IsTrue ( F ) ; AllF &= IsFalse


<s> MVT ElemTy = VecTy . getVectorElementType ( ) ; unsigned ElemWidth = ElemTy . getSizeInBits ( ) ; unsigned HwLen = Subtarget . getVectorLength ( ) ; unsigned ElemSize = ElemWidth / <NUM_LIT> ; assert ( ElemSize * VecLen == HwLen ) ; SmallVector < SDValue , <NUM_LIT> > Words ; if ( VecTy . getVectorElementType ( ) != <STR_LIT> ) { assert ( ( ElemSize == <NUM_LIT> || ElemSize == <NUM_LIT> ) && <STR_LIT> ) ; unsigned OpsPerWord = ( ElemSize == <NUM_LIT> ) ? <NUM_LIT> : <NUM_LIT> ; MVT PartVT = <STR_LIT> ( VecTy . getVectorElementType ( ) , OpsPerWord ) ; for ( unsigned i = <NUM_LIT> ; i != VecLen ; i += OpsPerWord ) { SDValue W = buildVector32 ( Values . slice ( i , OpsPerWord ) , dl , PartVT , DAG ) ; Words . push_back ( DAG . getBitcast ( <STR_LIT> , W ) ) ; } } else { Words . assign ( Values . begin ( ) , Values . end ( ) ) ; } unsigned NumWords = Words . size ( ) ; bool IsSplat = true , IsUndef = true ; SDValue SplatV ; for ( unsigned i = <NUM_LIT> ; i != NumWords && IsSplat ; ++ i ) { if ( isUndef ( Words [ i ] ) ) continue ; IsUndef = false ; if ( ! SplatV . getNode ( ) ) SplatV = Words [ i ] ; else if ( SplatV != Words [ i ] ) IsSplat = false ; } if ( IsUndef ) return DAG . getUNDEF ( VecTy ) ; if ( IsSplat ) { assert ( SplatV . getNode ( ) ) ; auto * IdxN = dyn_cast < ConstantSDNode > ( SplatV . getNode ( ) ) ; if ( IdxN && IdxN -> isNullValue ( ) ) return getZero ( dl , VecTy , DAG ) ; return DAG . getNode ( <STR_LIT> , dl , VecTy , SplatV ) ; } SmallVector < ConstantInt * , <NUM_LIT> > Consts ( VecLen ) ; bool AllConst = getBuildVectorConstInts ( Values , VecTy , DAG , Consts ) ; if ( AllConst ) { ArrayRef < Constant * > Tmp ( ( Constant * * ) Consts . begin ( ) , ( Constant * * ) Consts . end ( ) ) ; Constant * CV = ConstantVector :: get ( Tmp ) ; unsigned Align = HwLen ; SDValue CP = LowerConstantPool ( DAG . getConstantPool ( CV , VecTy , Align ) , DAG ) ; return DAG . getLoad ( VecTy , dl , DAG . getEntryNode ( ) , CP , MachinePointerInfo :: getConstantPool ( MF ) , Align ) ; } auto IsBuildFromExtracts = [ this , & Values ] ( SDValue & SrcVec , SmallVectorImpl < int > & SrcIdx ) { SDValue Vec ; for ( SDValue V :


<s> MVT PartVT = <STR_LIT> ( VecTy . getVectorElementType ( ) , OpsPerWord ) ; for ( unsigned i = <NUM_LIT> ; i != VecLen ; i += OpsPerWord ) { SDValue W = buildVector32 ( Values . slice ( i , OpsPerWord ) , dl , PartVT , DAG ) ; Words . push_back ( DAG . getBitcast ( <STR_LIT> , W ) ) ; } } else { Words . assign ( Values . begin ( ) , Values . end ( ) ) ; } unsigned NumWords = Words . size ( ) ; bool IsSplat = true , IsUndef = true ; SDValue SplatV ; for ( unsigned i = <NUM_LIT> ; i != NumWords && IsSplat ; ++ i ) { if ( isUndef ( Words [ i ] ) ) continue ; IsUndef = false ; if ( ! SplatV . getNode ( ) ) SplatV = Words [ i ] ; else if ( SplatV != Words [ i ] ) IsSplat = false ; } if ( IsUndef ) return DAG . getUNDEF ( VecTy ) ; if ( IsSplat ) { assert ( SplatV . getNode ( ) ) ; auto * IdxN = dyn_cast < ConstantSDNode > ( SplatV . getNode ( ) ) ; if ( IdxN && IdxN -> isNullValue ( ) ) return getZero ( dl , VecTy , DAG ) ; return DAG . getNode ( <STR_LIT> , dl , VecTy , SplatV ) ; } SmallVector < ConstantInt * , <NUM_LIT> > Consts ( VecLen ) ; bool AllConst = getBuildVectorConstInts ( Values , VecTy , DAG , Consts ) ; if ( AllConst ) { ArrayRef < Constant * > Tmp ( ( Constant * * ) Consts . begin ( ) , ( Constant * * ) Consts . end ( ) ) ; Constant * CV = ConstantVector :: get ( Tmp ) ; unsigned Align = HwLen ; SDValue CP = LowerConstantPool ( DAG . getConstantPool ( CV , VecTy , Align ) , DAG ) ; return DAG . getLoad ( VecTy , dl , DAG . getEntryNode ( ) , CP , MachinePointerInfo :: getConstantPool ( MF ) , Align ) ; } auto IsBuildFromExtracts = [ this , & Values ] ( SDValue & SrcVec , SmallVectorImpl < int > & SrcIdx ) { SDValue Vec ; for ( SDValue V : Values ) { if ( isUndef ( V ) ) { SrcIdx . push_back ( - <NUM_LIT> ) ; continue ; } if ( V . getOpcode ( ) != <STR_LIT> ) return false ; SDValue T = V . getOperand (


<s> unsigned IdxW = <NUM_LIT> ; auto Lo32 = [ & DAG , & dl ] ( SDValue P ) { return DAG . getTargetExtractSubreg ( <STR_LIT> , dl , <STR_LIT> , P ) ; } ; auto Hi32 = [ & DAG , & dl ] ( SDValue P ) { return DAG . getTargetExtractSubreg ( <STR_LIT> , dl , <STR_LIT> , P ) ; } ; SDValue W0 = isUndef ( PredV ) ? DAG . getUNDEF ( <STR_LIT> ) : DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , PredV ) ; Words [ IdxW ] . push_back ( Hi32 ( W0 ) ) ; Words [ IdxW ] . push_back ( Lo32 ( W0 ) ) ; while ( Bytes < BitBytes ) { IdxW ^= <NUM_LIT> ; Words [ IdxW ] . clear ( ) ; if ( Bytes < <NUM_LIT> ) { for ( const SDValue & W : Words [ IdxW ^ <NUM_LIT> ] ) { SDValue T = expandPredicate ( W , dl , DAG ) ; Words [ IdxW ] . push_back ( Hi32 ( T ) ) ; Words [ IdxW ] . push_back ( Lo32 ( T ) ) ; } } else { for ( const SDValue & W : Words


<s> return DAG . getTargetExtractSubreg ( <STR_LIT> , dl , <STR_LIT> , P ) ; } ; auto Hi32 = [ & DAG , & dl ] ( SDValue P ) { return DAG . getTargetExtractSubreg ( <STR_LIT> , dl , <STR_LIT> , P ) ; } ; SDValue W0 = isUndef ( PredV ) ? DAG . getUNDEF ( <STR_LIT> ) : DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , PredV ) ; Words [ IdxW ] . push_back ( Hi32 ( W0 ) ) ; Words [ IdxW ] . push_back ( Lo32 ( W0 ) ) ; while ( Bytes < BitBytes ) { IdxW ^= <NUM_LIT> ; Words [ IdxW ] . clear ( ) ; if ( Bytes < <NUM_LIT> ) { for ( const SDValue & W : Words [ IdxW ^ <NUM_LIT> ] ) { SDValue T = expandPredicate ( W , dl , DAG ) ; Words [ IdxW ] . push_back ( Hi32 ( T ) ) ; Words [ IdxW ] . push_back ( Lo32 ( T ) ) ; } } else { for ( const SDValue & W : Words [ IdxW ^ <NUM_LIT> ] ) { Words [ IdxW ] . push_back ( W ) ; Words [ IdxW ] . push_back ( W ) ; } } Bytes *= <NUM_LIT> ; } assert ( Bytes == BitBytes ) ; SDValue Vec = ZeroFill ? getZero ( dl , ByteTy , DAG ) : DAG . getUNDEF ( ByteTy ) ; SDValue S4 = DAG . getConstant ( HwLen - <NUM_LIT> , dl , <STR_LIT> ) ; for ( const SDValue & W : Words [ IdxW ] ) { Vec = DAG . getNode ( <STR_LIT> , dl , ByteTy , Vec , S4 ) ; Vec = DAG . getNode ( <STR_LIT> , dl , ByteTy , Vec , W ) ; } return Vec


<s> for ( const SDValue & Op : Ops ) IntOps . push_back ( Op ) ; return DAG . getNode ( <STR_LIT> , dl


<s> SmallVector < SDValue , <NUM_LIT> > IntOps ; IntOps . push_back ( DAG . getConstant ( IntId , dl , <STR_LIT> ) ) ; for ( const SDValue & Op : Ops ) IntOps . push_back ( Op ) ; return DAG . getNode ( <STR_LIT> , dl , ResTy


<s> setOperationAction ( Opc , FromTy , Promote ) ; AddPromotedToType ( Opc , FromTy , ToTy ) ; } ; setOperationAction ( <STR_LIT> , ByteV , Legal ) ; setOperationAction ( <STR_LIT> , ByteW , Legal ) ; for ( MVT T : LegalV ) { setIndexedLoadAction ( <STR_LIT> , T , Legal ) ; setIndexedStoreAction ( <STR_LIT> , T , Legal ) ; setOperationAction ( <STR_LIT> , T , Legal ) ; setOperationAction ( <STR_LIT> , T , Legal ) ; setOperationAction ( <STR_LIT> , T , Legal ) ; setOperationAction ( <STR_LIT> , T , Legal ) ; setOperationAction ( <STR_LIT> , T , Legal ) ; setOperationAction ( <STR_LIT> , T , Legal ) ; setOperationAction ( <STR_LIT> , T , Legal ) ; if ( T != ByteV ) { setOperationAction ( <STR_LIT> , T , Legal ) ; setOperationAction ( <STR_LIT> , T , Legal ) ; setOperationAction ( <STR_LIT> , T , Legal ) ; } setOperationAction ( <STR_LIT> , T , Custom ) ; setOperationAction ( <STR_LIT> , T , Custom ) ; setOperationAction ( <STR_LIT> , T , Custom ) ; setOperationAction ( <STR_LIT> , T , Custom ) ; setOperationAction ( <STR_LIT> , T , Custom ) ; setOperationAction ( <STR_LIT> , T , Custom ) ; setOperationAction ( <STR_LIT> , T , Custom ) ; setOperationAction ( <STR_LIT> , T , Custom ) ; setOperationAction ( <STR_LIT> , T , Custom ) ; setOperationAction ( <STR_LIT> , T , Custom ) ; setOperationAction ( <STR_LIT> , T , Custom ) ; setOperationAction ( <STR_LIT> , T , Custom ) ; setOperationAction ( <STR_LIT> , T , Custom ) ; setOperationAction ( <STR_LIT> , T , Custom ) ; if ( T != ByteV ) { setOperationAction ( <STR_LIT> , T , Custom ) ; setOperationAction ( <STR_LIT> , T , Custom ) ; setOperationAction ( <STR_LIT> , T , Custom ) ; setOperationAction ( <STR_LIT> , T , Custom ) ; setPromoteTo ( <STR_LIT> , T , ByteV ) ; } setCondCodeAction ( <STR_LIT> , T , Expand ) ; setCondCodeAction ( <STR_LIT> , T , Expand ) ; setCondCodeAction ( <STR_LIT> , T , Expand ) ; setCondCodeAction ( <STR_LIT> , T , Expand ) ; setCondCodeAction ( <STR_LIT> , T , Expand ) ; setCondCodeAction ( <STR_LIT> , T , Expand ) ; setCondCodeAction ( <STR_LIT> , T , Expand ) ; } for ( MVT T : LegalW ) { setOperationAction ( <STR_LIT> , T , Custom ) ; setOperationAction ( <STR_LIT> , T , Custom ) ; setOperationAction ( <STR_LIT> , T , Custom ) ; setOperationAction ( <STR_LIT> , T ,


<s> MVT ByteTy = <STR_LIT> ( <STR_LIT> , HwLen ) ; SDValue ByteVec = DAG . getNode ( <STR_LIT> , dl , ByteTy , VecV ) ; SDValue ByteSub = createHvxPrefixPred ( SubV , dl , BitBytes , false , DAG ) ; SDValue ByteIdx ; auto * IdxN = dyn_cast < ConstantSDNode > ( IdxV . getNode ( ) ) ; if ( ! IdxN || ! IdxN -> isNullValue ( ) ) { ByteIdx = DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , IdxV , DAG . getConstant ( BitBytes , dl , <STR_LIT> ) ) ; ByteVec = DAG . getNode ( <STR_LIT> , dl , ByteTy , ByteVec , ByteIdx ) ; } MVT BoolTy = <STR_LIT> ( <STR_LIT>


<s> MVT VecTy = ty ( VecV ) ; MVT SubTy = ty ( SubV ) ; assert ( Subtarget . isHVXVectorType ( VecTy , true ) ) ; unsigned VecLen = VecTy . getVectorNumElements ( ) ; unsigned HwLen = Subtarget . getVectorLength ( ) ; assert ( HwLen % VecLen == <NUM_LIT> && <STR_LIT> ) ; unsigned Scale = VecLen / SubTy . getVectorNumElements ( ) ; unsigned BitBytes = HwLen / VecLen ; unsigned BlockLen = HwLen / Scale ; MVT ByteTy = <STR_LIT> ( <STR_LIT> , HwLen ) ; SDValue ByteVec = DAG . getNode ( <STR_LIT> , dl , ByteTy , VecV ) ; SDValue ByteSub = createHvxPrefixPred ( SubV , dl , BitBytes , false , DAG ) ; SDValue ByteIdx ; auto * IdxN = dyn_cast < ConstantSDNode > ( IdxV . getNode ( ) ) ; if ( ! IdxN || ! IdxN -> isNullValue ( ) ) { ByteIdx = DAG . getNode ( <STR_LIT> , dl , <STR_LIT> , IdxV , DAG . getConstant ( BitBytes , dl , <STR_LIT>


<s> unsigned Size = Op . getNumOperands ( ) ; SmallVector < SDValue , <NUM_LIT> > Ops ; for ( unsigned i = <NUM_LIT> ; i != Size ; ++ i ) Ops . push_back ( Op . getOperand ( i ) ) ; if ( VecTy . getVectorElementType ( ) == <STR_LIT> ) return buildHvxVectorPred ( Ops , dl , VecTy , DAG ) ; if ( VecTy . getSizeInBits ( ) == <NUM_LIT> *


<s> unsigned Size = Op . getNumOperands ( ) ; SmallVector < SDValue , <NUM_LIT> > Ops ; for ( unsigned i = <NUM_LIT> ; i != Size ; ++ i ) Ops . push_back ( Op . getOperand ( i ) ) ; if ( VecTy . getVectorElementType ( ) == <STR_LIT> ) return buildHvxVectorPred ( Ops , dl , VecTy , DAG ) ; if ( VecTy . getSizeInBits ( ) == <NUM_LIT> * Subtarget . getVectorLength ( ) ) { ArrayRef < SDValue > A ( Ops ) ; MVT SingleTy = typeSplit ( VecTy ) . first ; SDValue V0 = buildHvxVectorReg ( A . take_front ( Size / <NUM_LIT> ) , dl , SingleTy , DAG ) ; SDValue V1 = buildHvxVectorReg ( A . drop_front ( Size / <NUM_LIT> ) , dl , SingleTy


<s> assert ( VecTy . getVectorElementType ( ) == <STR_LIT> ) ; unsigned HwLen = Subtarget . getVectorLength ( ) ; assert ( isPowerOf2_32 ( NumOp ) && HwLen % NumOp == <NUM_LIT> ) ; SDValue Op0 = Op . getOperand ( <NUM_LIT> ) ; if ( Subtarget . isHVXVectorType ( ty ( Op0 ) , true ) ) { if ( NumOp == <NUM_LIT> ) return DAG . getNode ( <STR_LIT> , dl , VecTy , Op0 , Op . getOperand ( <NUM_LIT> ) ) ; ArrayRef < SDUse > U ( Op . getNode ( ) -> ops ( ) ) ; SmallVector < SDValue , <NUM_LIT> > SV ( U . begin ( ) , U . end ( ) ) ; ArrayRef < SDValue > Ops ( SV ) ; MVT HalfTy = typeSplit ( VecTy ) . first ; SDValue V0 = DAG . getNode ( <STR_LIT> , dl , HalfTy , Ops . take_front ( NumOp / <NUM_LIT> ) ) ; SDValue V1 = DAG . getNode ( <STR_LIT> , dl , HalfTy , Ops . take_back ( NumOp / <NUM_LIT> ) ) ; return DAG . getNode ( <STR_LIT> , dl , VecTy , V0 , V1 ) ; } unsigned BitBytes = HwLen / VecTy . getVectorNumElements ( ) ; SmallVector < SDValue , <NUM_LIT> > Prefixes ; for ( SDValue V : Op . getNode ( ) -> op_values ( ) ) { SDValue P = createHvxPrefixPred ( V , dl , BitBytes , true , DAG ) ; Prefixes . push_back ( P ) ; } unsigned InpLen = ty ( Op . getOperand ( <NUM_LIT> ) ) . getVectorNumElements ( ) ; MVT ByteTy = <STR_LIT> ( <STR_LIT> , HwLen ) ; SDValue S = DAG . getConstant ( InpLen * BitBytes , dl , <STR_LIT> ) ; SDValue Res = getZero ( dl , ByteTy , DAG ) ; for ( unsigned i = <NUM_LIT> , e = Prefixes . size ( ) ; i != e ; ++ i ) { Res = DAG . getNode ( <STR_LIT> , dl , ByteTy , Res , S ) ; Res = DAG . getNode ( <STR_LIT> , dl , ByteTy , Res , Prefixes [ e - i - <NUM_LIT> ] ) ; } return DAG . getNode ( <STR_LIT> , dl , VecTy ,


<s> assert ( isPowerOf2_32 ( ElemWidth ) && ElemWidth <= <NUM_LIT> ) ; for ( unsigned i = <NUM_LIT> ; i != <NUM_LIT> / ElemWidth ; ++ i ) { Splat1 = ( Splat1 << ElemWidth ) | <NUM_LIT> ; SplatW = ( SplatW << ElemWidth ) | ElemWidth ; } SDValue Vec1 = DAG . getNode ( <STR_LIT> , dl , ResTy , DAG . getConstant ( uint32_t ( Splat1 ) , dl , <STR_LIT> ) ) ; SDValue VecW = DAG . getNode ( <STR_LIT> , dl , ResTy , DAG . getConstant ( uint32_t ( SplatW ) , dl , <STR_LIT> ) ) ; SDValue VecN1 = DAG . getNode ( <STR_LIT> , dl , ResTy , DAG . getConstant ( - <NUM_LIT> , dl , <STR_LIT> ) ) ; SDValue A = DAG . getNode ( <STR_LIT> , dl , ResTy , { DAG . getNode ( <STR_LIT> , dl , ResTy , { InpV , VecN1 } ) , DAG . getNode ( <STR_LIT> , dl , ResTy , { InpV , Vec1 } ) } ) ; return DAG . getNode ( <STR_LIT> , dl , ResTy , { VecW , DAG . getNode ( <STR_LIT> , dl , ResTy ,


<s> SDValue IdxV = Op . getOperand ( <NUM_LIT> ) ; MVT ElemTy = ty ( VecV ) . getVectorElementType ( ) ; if ( ElemTy == <STR_LIT> ) return insertHvxElementPred ( VecV , IdxV , ValV , dl , DAG ) ; return insertHvxElementReg ( VecV , IdxV , ValV


<s> const SDLoc & dl ( Op ) ; SDValue VecV = Op . getOperand ( <NUM_LIT> ) ; SDValue ValV = Op . getOperand ( <NUM_LIT> ) ; SDValue IdxV = Op .


<s> unsigned VecLen = ResTy . getVectorNumElements ( ) ; SDValue Vs = Op . getOperand ( <NUM_LIT> ) ; SDValue Vt = Op . getOperand ( <NUM_LIT> ) ; switch ( ElemTy . SimpleTy ) { case <STR_LIT> : { MVT ExtTy = typeExtElem ( ResTy , <NUM_LIT> ) ; unsigned MpyOpc = ElemTy == <STR_LIT> ? <STR_LIT> : <STR_LIT> ; SDValue M = getInstr ( MpyOpc , dl , ExtTy , { Vs , Vt } , DAG ) ; for ( unsigned I = <NUM_LIT> ; I < VecLen ; I += <NUM_LIT> ) { ShuffMask . push_back ( I ) ; ShuffMask . push_back ( I + VecLen ) ; } VectorPair P = opSplit ( opCastElem ( M , ElemTy , DAG ) , dl , DAG ) ; SDValue BS = getByteShuffle ( dl , P . first , P . second , ShuffMask , DAG ) ; return DAG . getBitcast ( ResTy , BS ) ; } case <STR_LIT> : return getInstr ( <STR_LIT> , dl , ResTy , { Vs , Vt } , DAG ) ; case <STR_LIT> : { SDValue S16 = DAG . getConstant ( <NUM_LIT> , dl , <STR_LIT> ) ; SDValue T0 = getInstr ( <STR_LIT> , dl , ResTy , { Vs


<s> } assert ( ElemTy == <STR_LIT> ) ; SDValue S16 = DAG . getConstant ( <NUM_LIT> , dl , <STR_LIT> ) ; if ( IsSigned ) { SDValue T0 = getInstr ( <STR_LIT> , dl , ResTy , { Vt , Vs } , DAG ) ; SDValue S0 = getInstr ( <STR_LIT> , dl , ResTy , { Vs , S16 } , DAG ) ; SDValue T1 = getInstr ( <STR_LIT> , dl , ResTy , { T0 , S0 , Vt } , DAG ) ; SDValue S2 = getInstr ( <STR_LIT> , dl , ResTy , { T1 , S16 } , DAG ) ; SDValue T2 = getInstr ( <STR_LIT> , dl , ResTy , { S0 , Vt } , DAG ) ; SDValue T3 = DAG . getNode ( <STR_LIT> , dl , ResTy , { S2 , T2 } ) ; return T3 ; } auto LoVec = [ & DAG , ResTy , dl ] ( SDValue Pair ) { return DAG . getTargetExtractSubreg ( <STR_LIT> , dl , ResTy , Pair ) ; } ; auto HiVec = [ & DAG , ResTy , dl ] ( SDValue Pair ) { return DAG . getTargetExtractSubreg ( <STR_LIT> , dl , ResTy , Pair ) ; } ; MVT PairTy = typeJoin ( { ResTy , ResTy } ) ; SDValue P = getInstr ( <STR_LIT> , dl , ResTy , { DAG . getConstant ( <NUM_LIT> , dl , <STR_LIT> ) } , DAG ) ; SDValue T0 = getInstr ( <STR_LIT> , dl , PairTy , { Vs , Vt } , DAG ) ; SDValue T1 = getInstr ( <STR_LIT> , dl , ResTy , { LoVec ( T0 ) , S16 } , DAG ) ; SDValue D0 = getInstr ( <STR_LIT> , dl , ResTy , { Vt , P } , DAG ) ; SDValue T2 = getInstr ( <STR_LIT> , dl , PairTy , { Vs , D0 } , DAG ) ; SDValue T3 = getInstr ( <STR_LIT> , dl , PairTy , { LoVec ( T2 ) , HiVec ( T2 ) } , DAG ) ; SDValue T4 = DAG . getNode ( <STR_LIT> , dl , ResTy , { T1 , LoVec ( T3 ) } ) ; SDValue T5 = getInstr ( <STR_LIT> , dl , ResTy , { T4 , S16 } , DAG ) ; SDValue T6 = DAG . getNode ( <STR_LIT> , dl , ResTy , { HiVec ( T0


<s> SDValue S2 = getInstr ( <STR_LIT> , dl , ResTy , { T1 , S16 } , DAG ) ; SDValue T2 = getInstr ( <STR_LIT> , dl , ResTy , { S0 , Vt } , DAG ) ; SDValue T3 = DAG . getNode ( <STR_LIT> , dl , ResTy , { S2 , T2 } ) ; return T3 ; } auto LoVec = [ & DAG , ResTy , dl ] ( SDValue Pair ) { return DAG . getTargetExtractSubreg ( <STR_LIT> , dl , ResTy , Pair ) ; } ; auto HiVec = [ & DAG , ResTy , dl ] ( SDValue Pair ) { return DAG . getTargetExtractSubreg ( <STR_LIT> , dl , ResTy , Pair ) ; } ; MVT PairTy = typeJoin ( { ResTy , ResTy } ) ; SDValue P = getInstr ( <STR_LIT> , dl , ResTy , { DAG . getConstant ( <NUM_LIT> , dl , <STR_LIT> ) } , DAG ) ; SDValue T0 = getInstr ( <STR_LIT> , dl , PairTy , { Vs , Vt } , DAG ) ; SDValue T1 = getInstr ( <STR_LIT> , dl , ResTy , { LoVec ( T0 ) , S16 } , DAG ) ; SDValue D0 = getInstr ( <STR_LIT> , dl , ResTy , { Vt , P } , DAG ) ; SDValue T2 = getInstr ( <STR_LIT> , dl , PairTy , { Vs , D0 } , DAG ) ; SDValue T3 = getInstr ( <STR_LIT> , dl , PairTy , { LoVec ( T2 ) , HiVec ( T2 ) } , DAG ) ; SDValue T4 = DAG . getNode ( <STR_LIT> , dl , ResTy , { T1 , LoVec ( T3 ) } ) ; SDValue T5 = getInstr ( <STR_LIT> , dl , ResTy , { T4 , S16 } , DAG ) ; SDValue T6 = DAG . getNode ( <STR_LIT> , dl , ResTy , { HiVec ( T0 ) , HiVec ( T3 ) } ) ; SDValue T7 = DAG . getNode ( <STR_LIT> , dl , ResTy


<s> unsigned Opc = Op . getOpcode ( ) ; bool IsPairOp = isHvxPairTy ( ty ( Op ) ) || llvm :: any_of ( Op . getNode ( ) -> ops ( ) , [ this ] ( SDValue V ) { return isHvxPairTy ( ty ( V ) ) ; } ) ; if ( IsPairOp ) { switch ( Opc ) { default : break ; case <STR_LIT> : case <STR_LIT> : return SplitHvxMemOp ( Op , DAG ) ; case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : return SplitHvxPairOp ( Op , DAG ) ; } } switch ( Opc ) { default : break ; case <STR_LIT> : return LowerHvxBuildVector ( Op , DAG ) ; case <STR_LIT> : return LowerHvxConcatVectors ( Op , DAG ) ; case <STR_LIT> : return LowerHvxInsertSubvector ( Op , DAG ) ; case <STR_LIT> : return LowerHvxInsertElement ( Op , DAG ) ; case <STR_LIT> : return LowerHvxExtractSubvector ( Op , DAG ) ; case <STR_LIT> : return LowerHvxExtractElement ( Op , DAG ) ; case <STR_LIT> : return LowerHvxAnyExt ( Op , DAG ) ; case <STR_LIT> : return LowerHvxSignExt ( Op , DAG ) ; case <STR_LIT> : return LowerHvxZeroExt ( Op , DAG ) ; case <STR_LIT> : return LowerHvxCttz ( Op , DAG ) ; case <STR_LIT> : case <STR_LIT> : case <STR_LIT> : return LowerHvxShift ( Op , DAG ) ; case <STR_LIT> : return LowerHvxMul ( Op , DAG ) ; case <STR_LIT> : case <STR_LIT> : return LowerHvxMulh ( Op , DAG ) ; case <STR_LIT> : return LowerHvxExtend ( Op , DAG ) ; case <STR_LIT> : case <STR_LIT> : return Op ; case <STR_LIT> : return SDValue ( ) ; } Op . dumpr ( & DAG ) ; llvm_unreachable ( <STR_LIT>