1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363
|
/*========================== begin_copyright_notice ============================
Copyright (C) 2025 Intel Corporation
SPDX-License-Identifier: MIT
============================= end_copyright_notice ===========================*/
# Description : Refactor getPreviousDefRecursive to getPreviousDefIterative
diff --git a/llvm/include/llvm/Analysis/MemorySSAUpdater.h b/llvm/include/llvm/Analysis/MemorySSAUpdater.h
--- a/llvm/include/llvm/Analysis/MemorySSAUpdater.h
+++ b/llvm/include/llvm/Analysis/MemorySSAUpdater.h
@@ -251,10 +251,7 @@ private:
MemoryAccess *getPreviousDef(MemoryAccess *);
MemoryAccess *getPreviousDefInBlock(MemoryAccess *);
MemoryAccess *
- getPreviousDefFromEnd(BasicBlock *,
- DenseMap<BasicBlock *, TrackingVH<MemoryAccess>> &);
- MemoryAccess *
- getPreviousDefRecursive(BasicBlock *,
+ getPreviousDefIterative(BasicBlock *,
DenseMap<BasicBlock *, TrackingVH<MemoryAccess>> &);
MemoryAccess *recursePhi(MemoryAccess *Phi);
MemoryAccess *tryRemoveTrivialPhi(MemoryPhi *Phi);
diff --git a/llvm/lib/Analysis/MemorySSAUpdater.cpp b/llvm/lib/Analysis/MemorySSAUpdater.cpp
--- a/llvm/lib/Analysis/MemorySSAUpdater.cpp
+++ b/llvm/lib/Analysis/MemorySSAUpdater.cpp
@@ -20,6 +20,7 @@
#include "llvm/IR/Dominators.h"
#include "llvm/Support/Debug.h"
#include <algorithm>
+#include <stack>
#define DEBUG_TYPE "memoryssa"
using namespace llvm;
@@ -33,66 +34,42 @@ using namespace llvm;
// that there are two or more definitions needing to be merged.
// This still will leave non-minimal form in the case of irreducible control
// flow, where phi nodes may be in cycles with themselves, but unnecessary.
-MemoryAccess *MemorySSAUpdater::getPreviousDefRecursive(
- BasicBlock *BB,
+MemoryAccess *MemorySSAUpdater::getPreviousDefIterative(
+ BasicBlock *BBB,
DenseMap<BasicBlock *, TrackingVH<MemoryAccess>> &CachedPreviousDef) {
- // First, do a cache lookup. Without this cache, certain CFG structures
- // (like a series of if statements) take exponential time to visit.
- auto Cached = CachedPreviousDef.find(BB);
- if (Cached != CachedPreviousDef.end())
- return Cached->second;
-
- // If this method is called from an unreachable block, return LoE.
- if (!MSSA->DT->isReachableFromEntry(BB))
- return MSSA->getLiveOnEntryDef();
- if (BasicBlock *Pred = BB->getUniquePredecessor()) {
- VisitedBlocks.insert(BB);
- // Single predecessor case, just recurse, we can only have one definition.
- MemoryAccess *Result = getPreviousDefFromEnd(Pred, CachedPreviousDef);
- CachedPreviousDef.insert({BB, Result});
- return Result;
- }
+ // There're 5 cases, case 3 (easy) and case 5 (hard) has recursives.
+ // We need special states to handle their recursive returns
+ enum State {COMMON, CASE3, CASE5};
- if (VisitedBlocks.count(BB)) {
- // We hit our node again, meaning we had a cycle, we must insert a phi
- // node to break it so we have an operand. The only case this will
- // insert useless phis is if we have irreducible control flow.
- MemoryAccess *Result = MSSA->createMemoryPhi(BB);
- CachedPreviousDef.insert({BB, Result});
- return Result;
- }
+ // This is the common frame required for everything
+ struct Frame {
+ BasicBlock *bb;
+ MemoryAccess *rtn;
+ State st;
+ };
- if (VisitedBlocks.insert(BB).second) {
- // Mark us visited so we can detect a cycle
+ // This is the additional info only required by Case 5
+ struct FrameCase5 {
SmallVector<TrackingVH<MemoryAccess>, 8> PhiOps;
+ bool UniqueIncomingAccess;
+ MemoryAccess *SingleAccess;
+ pred_iterator PredIt;
+ };
- // Recurse to get the values in our predecessors for placement of a
- // potential phi node. This will insert phi nodes if we cycle in order to
- // break the cycle and have an operand.
- bool UniqueIncomingAccess = true;
- MemoryAccess *SingleAccess = nullptr;
- for (auto *Pred : predecessors(BB)) {
- if (MSSA->DT->isReachableFromEntry(Pred)) {
- auto *IncomingAccess = getPreviousDefFromEnd(Pred, CachedPreviousDef);
- if (!SingleAccess)
- SingleAccess = IncomingAccess;
- else if (IncomingAccess != SingleAccess)
- UniqueIncomingAccess = false;
- PhiOps.push_back(IncomingAccess);
- } else
- PhiOps.push_back(MSSA->getLiveOnEntryDef());
- }
-
+ auto Case5AfterLoop = [&](SmallVector<TrackingVH<MemoryAccess>, 8> & PhiOps,
+ bool & UniqueIncomingAccess, MemoryAccess *& SingleAccess,
+ BasicBlock * BB) -> MemoryAccess * {
// Now try to simplify the ops to avoid placing a phi.
// This may return null if we never created a phi yet, that's okay
MemoryPhi *Phi = dyn_cast_or_null<MemoryPhi>(MSSA->getMemoryAccess(BB));
// See if we can avoid the phi by simplifying it.
- auto *Result = tryRemoveTrivialPhi(Phi, PhiOps);
+ MemoryAccess *Result = tryRemoveTrivialPhi(Phi, PhiOps);
// If we couldn't simplify, we may have to create a phi
if (Result == Phi && UniqueIncomingAccess && SingleAccess) {
- // A concrete Phi only exists if we created an empty one to break a cycle.
+ // A concrete Phi only exists if we created an empty one to break a
+ // cycle.
if (Phi) {
assert(Phi->operands().empty() && "Expected empty Phi");
Phi->replaceAllUsesWith(SingleAccess);
@@ -104,12 +81,13 @@ MemoryAccess *MemorySSAUpdater::getPreviousDefRecursive(
Phi = MSSA->createMemoryPhi(BB);
// See if the existing phi operands match what we need.
- // Unlike normal SSA, we only allow one phi node per block, so we can't just
- // create a new one.
+ // Unlike normal SSA, we only allow one phi node per block, so we
+ // can't just create a new one.
if (Phi->getNumOperands() != 0) {
// FIXME: Figure out whether this is dead code and if so remove it.
if (!std::equal(Phi->op_begin(), Phi->op_end(), PhiOps.begin())) {
- // These will have been filled in by the recursive read we did above.
+ // These will have been filled in by the recursive read we did
+ // above.
llvm::copy(PhiOps, Phi->op_begin());
std::copy(pred_begin(BB), pred_end(BB), Phi->block_begin());
}
@@ -126,8 +104,170 @@ MemoryAccess *MemorySSAUpdater::getPreviousDefRecursive(
VisitedBlocks.erase(BB);
CachedPreviousDef.insert({BB, Result});
return Result;
+ };
+
+ // We may want to switch to vector to boot performance
+ std::stack<Frame> SF;
+ std::stack<FrameCase5> SF5;
+ // The return frame
+ SF.push({nullptr, nullptr, COMMON});
+ // The entry frame
+ SF.push({BBB, nullptr, COMMON});
+
+ while (SF.size() > 1) {
+
+ if (COMMON == SF.top().st) {
+ auto BB = SF.top().bb;
+ auto Cached = CachedPreviousDef.find(BB);
+ if (Cached != CachedPreviousDef.end()) {
+ SF.pop();
+ SF.top().rtn = Cached->second;
+ continue;
+ } else if (!MSSA->DT->isReachableFromEntry(BB)) {
+ SF.pop();
+ SF.top().rtn = MSSA->getLiveOnEntryDef();
+ continue;
+ } else if (BasicBlock *Pred = BB->getUniquePredecessor()) {
+ VisitedBlocks.insert(BB);
+ // Single predecessor case, just recurse, we can only have one
+ // definition.
+ MemoryAccess *prevDefFromEnd = nullptr;
+ auto *Defs = MSSA->getWritableBlockDefs(Pred);
+ if (Defs) {
+ CachedPreviousDef.insert({Pred, &*Defs->rbegin()});
+ prevDefFromEnd = &*Defs->rbegin();
+ } else {
+ SF.top().st = CASE3;
+ SF.push({Pred, nullptr, COMMON});
+ continue;
+ }
+ MemoryAccess *Result = prevDefFromEnd;
+ CachedPreviousDef.insert({BB, Result});
+ SF.pop();
+ SF.top().rtn = Result;
+ continue;
+ } else if (VisitedBlocks.count(BB)) {
+ // We hit our node again, meaning we had a cycle, we must insert a phi
+ // node to break it so we have an operand. The only case this will
+ // insert useless phis is if we have irreducible control flow.
+ MemoryAccess *Result = MSSA->createMemoryPhi(BB);
+ CachedPreviousDef.insert({BB, Result});
+ SF.pop();
+ SF.top().rtn = Result;
+ continue;
+ } else if (VisitedBlocks.insert(BB).second) {
+ // Mark us visited so we can detect a cycle
+ SmallVector<TrackingVH<MemoryAccess>, 8> PhiOps;
+
+ // Recurse to get the values in our predecessors for placement of a
+ // potential phi node. This will insert phi nodes if we cycle in order
+ // to break the cycle and have an operand.
+ bool UniqueIncomingAccess = true;
+ MemoryAccess *SingleAccess = nullptr;
+ bool halt = false;
+ for (auto PredIt = predecessors(BB).begin();
+ PredIt != predecessors(BB).end(); PredIt++) {
+ auto Pred = *PredIt;
+ if (MSSA->DT->isReachableFromEntry(Pred)) {
+ MemoryAccess *prevDefFromEnd = nullptr;
+ auto *Defs = MSSA->getWritableBlockDefs(Pred);
+ if (Defs) {
+ CachedPreviousDef.insert({Pred, &*Defs->rbegin()});
+ prevDefFromEnd = &*Defs->rbegin();
+ } else {
+ SF.top().st = CASE5;
+ SF.push({Pred, nullptr, COMMON});
+ SF5.push({
+ std::move(PhiOps), UniqueIncomingAccess, SingleAccess,
+ std::move(PredIt)
+ });
+ halt = true;
+ break;
+ }
+ auto *IncomingAccess = prevDefFromEnd;
+ if (!SingleAccess)
+ SingleAccess = IncomingAccess;
+ else if (IncomingAccess != SingleAccess)
+ UniqueIncomingAccess = false;
+ PhiOps.push_back(IncomingAccess);
+ } else
+ PhiOps.push_back(MSSA->getLiveOnEntryDef());
+ }
+ if (halt)
+ continue;
+
+ auto Result =
+ Case5AfterLoop(PhiOps, UniqueIncomingAccess, SingleAccess, BB);
+
+ // Set ourselves up for the next variable by resetting visited state.
+ VisitedBlocks.erase(BB);
+ CachedPreviousDef.insert({BB, Result});
+ SF.pop();
+ SF.top().rtn = Result;
+ continue;
+ }
+ llvm_unreachable("Should have hit one of the five cases above");
+ } else if (CASE3 == SF.top().st) {
+ auto Result = SF.top().rtn;
+ CachedPreviousDef.insert({SF.top().bb, Result});
+ SF.pop();
+ SF.top().rtn = Result;
+ continue;
+ } else { // CASE5
+ // recover header
+ auto &PhiOps = SF5.top().PhiOps;
+ auto &UniqueIncomingAccess = SF5.top().UniqueIncomingAccess;
+ auto &SingleAccess = SF5.top().SingleAccess;
+ auto &PredIt = SF5.top().PredIt;
+ auto IncomingAccess = SF.top().rtn;
+ auto BB = SF.top().bb;
+
+ // in-loop remaining code
+ if (!SingleAccess)
+ SingleAccess = IncomingAccess;
+ else if (IncomingAccess != SingleAccess)
+ UniqueIncomingAccess = false;
+ PhiOps.push_back(IncomingAccess);
+
+ // remaining loop
+ bool halt = false;
+ for (PredIt++; PredIt != predecessors(BB).end(); PredIt++) {
+ auto Pred = *PredIt;
+ if (MSSA->DT->isReachableFromEntry(Pred)) {
+ MemoryAccess *prevDefFromEnd = nullptr;
+ auto *Defs = MSSA->getWritableBlockDefs(Pred);
+ if (Defs) {
+ CachedPreviousDef.insert({Pred, &*Defs->rbegin()});
+ prevDefFromEnd = &*Defs->rbegin();
+ } else {
+ SF.push({Pred, nullptr, COMMON});
+ halt = true;
+ break;
+ }
+ auto *IncomingAccess = prevDefFromEnd;
+ if (!SingleAccess)
+ SingleAccess = IncomingAccess;
+ else if (IncomingAccess != SingleAccess)
+ UniqueIncomingAccess = false;
+ PhiOps.push_back(IncomingAccess);
+ } else
+ PhiOps.push_back(MSSA->getLiveOnEntryDef());
+ }
+ if (halt)
+ continue;
+ // after loop
+ auto Result =
+ Case5AfterLoop(PhiOps, UniqueIncomingAccess, SingleAccess, BB);
+ SF.pop();
+ SF.top().rtn = Result;
+ SF5.pop();
+ continue;
+ }
+
+ llvm_unreachable("Should have hit one of the three cases above");
}
- llvm_unreachable("Should have hit one of the three cases above");
+ assert(0 == SF5.size());
+ return SF.top().rtn;
}
// This starts at the memory access, and goes backwards in the block to find the
@@ -138,7 +278,7 @@ MemoryAccess *MemorySSAUpdater::getPreviousDef(MemoryAccess *MA) {
if (auto *LocalResult = getPreviousDefInBlock(MA))
return LocalResult;
DenseMap<BasicBlock *, TrackingVH<MemoryAccess>> CachedPreviousDef;
- return getPreviousDefRecursive(MA->getBlock(), CachedPreviousDef);
+ return getPreviousDefIterative(MA->getBlock(), CachedPreviousDef);
}
// This starts at the memory access, and goes backwards in the block to the find
@@ -168,19 +308,6 @@ MemoryAccess *MemorySSAUpdater::getPreviousDefInBlock(MemoryAccess *MA) {
return nullptr;
}
-// This starts at the end of block
-MemoryAccess *MemorySSAUpdater::getPreviousDefFromEnd(
- BasicBlock *BB,
- DenseMap<BasicBlock *, TrackingVH<MemoryAccess>> &CachedPreviousDef) {
- auto *Defs = MSSA->getWritableBlockDefs(BB);
-
- if (Defs) {
- CachedPreviousDef.insert({BB, &*Defs->rbegin()});
- return &*Defs->rbegin();
- }
-
- return getPreviousDefRecursive(BB, CachedPreviousDef);
-}
// Recurse over a set of phi uses to eliminate the trivial ones
MemoryAccess *MemorySSAUpdater::recursePhi(MemoryAccess *Phi) {
if (!Phi)
@@ -396,7 +523,17 @@ void MemorySSAUpdater::insertDef(MemoryDef *MD, bool RenameUses) {
auto *BBIDF = MPhi->getBlock();
for (auto *Pred : predecessors(BBIDF)) {
DenseMap<BasicBlock *, TrackingVH<MemoryAccess>> CachedPreviousDef;
- MPhi->addIncoming(getPreviousDefFromEnd(Pred, CachedPreviousDef), Pred);
+ // inline getPreviousDefFromEnd start
+ MemoryAccess *prevDefFromEnd = nullptr;
+ auto *Defs = MSSA->getWritableBlockDefs(Pred);
+ if (Defs) {
+ CachedPreviousDef.insert({Pred, &*Defs->rbegin()});
+ prevDefFromEnd = & * Defs->rbegin();
+ } else {
+ prevDefFromEnd = getPreviousDefIterative(Pred, CachedPreviousDef);
+ }
+ // inline getPreviousDefFromEnd end
+ MPhi->addIncoming(prevDefFromEnd, Pred);
}
}
|