class LoopVectorizationCostModel

Declaration

class LoopVectorizationCostModel { /* full declaration omitted */ };

Description

LoopVectorizationCostModel - estimates the expected speedups due to vectorization. In many cases vectorization is not profitable. This can happen because of a number of reasons. In this class we mainly attempt to predict the expected speedup/slowdowns due to the supported instruction set. We use the TargetTransformInfo to query the different backends for the cost of different operations.

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:960

Member Variables

public llvm::Loop* TheLoop
The loop that we evaluate.
public llvm::PredicatedScalarEvolution& PSE
Predicated scalar evolution analysis.
public llvm::LoopInfo* LI
Loop Info analysis.
Vectorization legality.
public const llvm::TargetTransformInfo& TTI
Vector target information.
public const llvm::TargetLibraryInfo* TLI
Target Library Info.
public llvm::DemandedBits* DB
Demanded bits analysis.
public llvm::AssumptionCache* AC
Assumption cache.
public llvm::OptimizationRemarkEmitter* ORE
Interface to emit optimization remarks.
public const llvm::Function* TheFunction
public const llvm::LoopVectorizeHints* Hints
Loop Vectorize Hint.
public llvm::InterleavedAccessInfo& InterleaveInfo
The interleave access information contains groups of interleaved accesses with the same stride and close to each other.
public SmallPtrSet<const llvm::Value*, 16> ValuesToIgnore
Values to ignore in the cost model.
public SmallPtrSet<const llvm::Value*, 16> VecValuesToIgnore
Values to ignore in the cost model when VF > 1.

Method Overview

Methods

LoopVectorizationCostModel(
    llvm::ScalarEpilogueLowering SEL,
    llvm::Loop* L,
    llvm::PredicatedScalarEvolution& PSE,
    llvm::LoopInfo* LI,
    llvm::LoopVectorizationLegality* Legal,
    const llvm::TargetTransformInfo& TTI,
    const llvm::TargetLibraryInfo* TLI,
    llvm::DemandedBits* DB,
    llvm::AssumptionCache* AC,
    llvm::OptimizationRemarkEmitter* ORE,
    const llvm::Function* F,
    const llvm::LoopVectorizeHints* Hints,
    llvm::InterleavedAccessInfo& IAI)

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:962

Parameters

llvm::ScalarEpilogueLowering SEL
llvm::Loop* L
llvm::PredicatedScalarEvolution& PSE
llvm::LoopInfo* LI
llvm::LoopVectorizationLegality* Legal
const llvm::TargetTransformInfo& TTI
const llvm::TargetLibraryInfo* TLI
llvm::DemandedBits* DB
llvm::AssumptionCache* AC
llvm::OptimizationRemarkEmitter* ORE
const llvm::Function* F
const llvm::LoopVectorizeHints* Hints
llvm::InterleavedAccessInfo& IAI

bool blockNeedsPredication(llvm::BasicBlock* BB)

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1296

Parameters

llvm::BasicBlock* BB

SmallVector<llvm::LoopVectorizationCostModel::
                RegisterUsage,
            8>
calculateRegisterUsage(ArrayRef<unsigned int> VFs)

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1028

Parameters

ArrayRef<unsigned int> VFs

Returns

Returns information about the register usages of the loop for the given vectorization factors.

bool canTruncateToMinimalBitwidth(
    llvm::Instruction* I,
    unsigned int VF) const

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1090

Parameters

llvm::Instruction* I
unsigned int VF

Returns

True if instruction \p I can be truncated to a smaller bitwidth for vectorization factor \p VF.

void collectInstsToScalarize(unsigned int VF)

Description

Collects the instructions to scalarize for each predicated instruction in the loop.

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1187

Parameters

unsigned int VF

void collectUniformsAndScalars(unsigned int VF)

Description

Collect Uniform and Scalar values for the given \p VF. The sets depend on CM decision for Load/Store instructions that may be vectorized as interleave, gather-scatter or scalarized.

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1192

Parameters

unsigned int VF

void collectValuesToIgnore()

Description

Collect values we want to ignore in the cost model.

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1031

Optional<unsigned int> computeMaxVF()

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:977

Returns

An upper bound for the vectorization factor, or None if vectorization and interleaving should be avoided up front.

bool foldTailByMasking() const

Description

Returns true if all loop blocks should be masked to fold tail loop.

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1294

const InterleaveGroup<llvm::Instruction>*
getInterleavedAccessGroup(
    llvm::Instruction* Instr)

Description

Get the interleaved access group that \p Instr belongs to.

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1276

Parameters

llvm::Instruction* Instr

const MapVector<llvm::Instruction*, uint64_t>&
getMinimalBitwidths() const

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1036

Returns

The smallest bitwidth each instruction can be represented with. The vector equivalents of these instructions should be truncated to this type.

std::pair<unsigned int, unsigned int>
getSmallestAndWidestTypes()

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:998

Returns

The size (in bits) of the smallest and widest types in the code that needs to be vectorized. We ignore values that remain scalar such as 64 bit loop indices.

unsigned int getVectorCallCost(
    llvm::CallInst* CI,
    unsigned int VF,
    bool& NeedToScalarize)

Description

Estimate cost of a call instruction CI if it were vectorized with factor VF. Return the cost of the instruction, including scalarization overhead if it's needed. The flag NeedToScalarize shows if the call needs to be scalarized - i.e. either vector version isn't available, or is too expensive.

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1310

Parameters

llvm::CallInst* CI
unsigned int VF
bool& NeedToScalarize

unsigned int getVectorIntrinsicCost(
    llvm::CallInst* CI,
    unsigned int VF)

Description

Estimate cost of an intrinsic call instruction CI if it were vectorized with factor VF. Return the cost of the instruction, including scalarization overhead if it's needed.

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1303

Parameters

llvm::CallInst* CI
unsigned int VF

unsigned int getWideningCost(llvm::Instruction* I,
                             unsigned int VF)

Description

Return the vectorization cost for the given instruction \p I and vector width \p VF.

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1151

Parameters

llvm::Instruction* I
unsigned int VF

llvm::LoopVectorizationCostModel::InstWidening
getWideningDecision(llvm::Instruction* I,
                    unsigned int VF)

Description

Return the cost model decision for the given instruction \p I and vector width \p VF. Return CM_Unknown if this instruction did not pass through the cost modeling.

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1134

Parameters

llvm::Instruction* I
unsigned int VF

bool interleavedAccessCanBeWidened(
    llvm::Instruction* I,
    unsigned int VF = 1)

Description

Returns true if \p I is a memory instruction in an interleaved-group of memory accesses that can be vectorized with wide vector loads/stores and shuffles.

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1267

Parameters

llvm::Instruction* I
unsigned int VF = 1

bool isAccessInterleaved(llvm::Instruction* Instr)

Description

Check if \p Instr belongs to any interleaved access group.

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1270

Parameters

llvm::Instruction* Instr

bool isLegalGatherOrScatter(llvm::Value* V)

Description

Returns true if the target machine can represent \p V as a masked gather or scatter operation.

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1229

Parameters

llvm::Value* V

bool isLegalMaskedGather(
    llvm::Type* DataType,
    llvm::MaybeAlign Alignment)

Description

Returns true if the target machine supports masked gather operation for the given \p DataType.

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1223

Parameters

llvm::Type* DataType
llvm::MaybeAlign Alignment

bool isLegalMaskedLoad(llvm::Type* DataType,
                       llvm::Value* Ptr,
                       llvm::MaybeAlign Alignment)

Description

Returns true if the target machine supports masked load operation for the given \p DataType and kind of access to \p Ptr.

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1210

Parameters

llvm::Type* DataType
llvm::Value* Ptr
llvm::MaybeAlign Alignment

bool isLegalMaskedScatter(
    llvm::Type* DataType,
    llvm::MaybeAlign Alignment)

Description

Returns true if the target machine supports masked scatter operation for the given \p DataType.

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1217

Parameters

llvm::Type* DataType
llvm::MaybeAlign Alignment

bool isLegalMaskedStore(
    llvm::Type* DataType,
    llvm::Value* Ptr,
    llvm::MaybeAlign Alignment)

Description

Returns true if the target machine supports masked store operation for the given \p DataType and kind of access to \p Ptr.

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1203

Parameters

llvm::Type* DataType
llvm::Value* Ptr
llvm::MaybeAlign Alignment

bool isOptimizableIVTruncate(llvm::Instruction* I,
                             unsigned int VF)

Description

Return True if instruction \p I is an optimizable truncate whose operand is an induction variable. Such a truncate will be removed by adding a new induction variable with the destination type.

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1162

Parameters

llvm::Instruction* I
unsigned int VF

bool isPredicatedInst(llvm::Instruction* I)

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1250

Parameters

llvm::Instruction* I

bool isProfitableToScalarize(
    llvm::Instruction* I,
    unsigned int VF) const

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1042

Parameters

llvm::Instruction* I
unsigned int VF

Returns

True if it is more profitable to scalarize instruction \p I for vectorization factor \p VF.

bool isScalarAfterVectorization(
    llvm::Instruction* I,
    unsigned int VF) const

Description

Returns true if \p I is known to be scalar after vectorization.

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1073

Parameters

llvm::Instruction* I
unsigned int VF

bool isScalarEpilogueAllowed() const

Description

Returns true if a scalar epilogue is not allowed due to optsize or a loop hint annotation.

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1289

bool isScalarWithPredication(llvm::Instruction* I,
                             unsigned int VF = 1)

Description

Returns true if \p I is an instruction that will be scalarized with predication. Such instructions include conditional stores and instructions that may divide by zero. If a non-zero VF has been calculated, we check if I will be scalarized predication for that VF.

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1245

Parameters

llvm::Instruction* I
unsigned int VF = 1

bool isUniformAfterVectorization(
    llvm::Instruction* I,
    unsigned int VF) const

Description

Returns true if \p I is known to be uniform after vectorization.

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1057

Parameters

llvm::Instruction* I
unsigned int VF

bool memoryInstructionCanBeWidened(
    llvm::Instruction* I,
    unsigned int VF = 1)

Description

Returns true if \p I is a memory instruction with consecutive memory access that can be widened.

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1262

Parameters

llvm::Instruction* I
unsigned int VF = 1

bool requiresScalarEpilogue() const

Description

Returns true if an interleaved group requires a scalar iteration to handle accesses with gaps, and there is nothing preventing us from creating a scalar epilogue.

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1283

bool runtimeChecksRequired()

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:981

Returns

True if runtime checks are required for vectorization, and false otherwise.

unsigned int selectInterleaveCount(
    unsigned int VF,
    unsigned int LoopCost)

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1004

Parameters

unsigned int VF
unsigned int LoopCost

Returns

The desired interleave count. If interleave count has been specified by metadata it will be returned. Otherwise, the interleave count is computed and returned. VF and LoopCost are the selected vectorization factor and the cost of the selected VF.

void selectUserVectorizationFactor(
    unsigned int UserVF)

Description

Setup cost-based decisions for user vectorization factor.

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:990

Parameters

unsigned int UserVF

llvm::VectorizationFactor
selectVectorizationFactor(unsigned int MaxVF)

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:987

Parameters

unsigned int MaxVF

Returns

The most profitable vectorization factor and the cost of that VF. This method checks every power of two up to MaxVF. If UserVF is not ZERO then this vectorization factor will be selected if vectorization is possible.

void setCostBasedWideningDecision(unsigned int VF)

Description

Memory access instruction may be vectorized in more than one way. Form of instruction after vectorization depends on cost. This function takes cost-based decisions for Load/Store instructions and collects them in a map. This decisions map is used for building the lists of loop-uniform and loop-scalar instructions. The calculated cost is saved with widening decision in order to avoid redundant calculations.

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1013

Parameters

unsigned int VF

void setWideningDecision(
    const InterleaveGroup<llvm::Instruction>* Grp,
    unsigned int VF,
    llvm::LoopVectorizationCostModel::InstWidening
        W,
    unsigned int Cost)

Description

Save vectorization decision \p W and \p Cost taken by the cost model for interleaving group \p Grp and vector width \p VF.

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1116

Parameters

const InterleaveGroup<llvm::Instruction>* Grp
unsigned int VF
llvm::LoopVectorizationCostModel::InstWidening W
unsigned int Cost

void setWideningDecision(
    llvm::Instruction* I,
    unsigned int VF,
    llvm::LoopVectorizationCostModel::InstWidening
        W,
    unsigned int Cost)

Description

Save vectorization decision \p W and \p Cost taken by the cost model for instruction \p I and vector width \p VF.

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:1108

Parameters

llvm::Instruction* I
unsigned int VF
llvm::LoopVectorizationCostModel::InstWidening W
unsigned int Cost