79160973

Date: 2024-11-05 23:28:39
Score: 0.5
Natty:
Report link

So the problem is that you are running many instances of your Step Function in parallel, and since they each call the same Lambdas you are ending up with a lot of simultaneous executions and hit the limit.

To avoid this, you should get rid of Function0 and instead wrap your current State Machine in a Map State which will execute the steps for each entry in your data JSON; Within the Map State definition you can then set a maximum concurrency, which will allow you to make use of the efficiency of running in parallel without going over your limits. You can then call the updated State Machine with your full data list, and it will handle the concurrency internally.

I don't work with CloudFormation directly, so I don't know how to achieve this in your configuration, but here is how you would implement something like this in CDK;

const lambda_func[n] = LambdaInvoke(...)

// Start with your current step function body
const inner_steps = lambda_func1.next(lambda_func2)
                        .next(lambda_func3)
                        .next(lambda_func4)
                        .next(lambda_func5)

// Wrap the steps in a Map state, each of which takes one element of the `data` list as input
const wrapped_sfn = new MapStep(
    stack,
    "Run Lambdas for each data element",
    {
        maxConcurrency: 20,
        itemsPath: JsonPath.stringAt("$.data"),
        itemSelector: {  // Wraps each item in its own JSON with the key "data", as in function0
            "data": JsonPath.stringAt("$$.Map.Item.Value")
        },
        resultPath: "$.processedData",
    }
);
wrapped_sfn.itemProcessor(inner_steps);

Doing the equivalent of this in your CloudFormation should enable you to run all the processes as you expect while ensuring none of your Lambdas exceed their concurrency limits.

Reasons:
  • Blacklisted phrase (1): how to achieve
  • Long answer (-1):
  • Has code block (-0.5):
  • Low reputation (1):
Posted by: AngusB