Standard vs Express Step Functions Workflows: A Comprehensive Comparison

Q: What are the key differences between AWS Step Functions Standard and Express Workflows?

Standard and Express Workflows differ in execution semantics and pricing: Standard follows an exactly-once model and is priced per state transition, while Express follows an at-least-once model and is priced based on executions, duration, and memory usage.

Table of Contents

What are AWS Step Functions?

AWS Step Functions is a serverless workflow service that allows developers to design and manage complex workflows. A workflow, in this context, is a series of steps (or ‘states’), with each step representing a single unit of work. These steps are pieced together to form a state machine that directs the execution flow of the application. When developing these workflows, one critical decision you’ll face is choosing between Standard and Express workflows. This article will delve into the distinguishing features, benefits, and use cases of both, helping you make an informed choice.

Standard Workflows in AWS Step Functions

Standard Workflows are designed to handle long-running, durable, and auditable workflows. Their key features include:

Long Execution Time: These workflows can run up to one year, making them suitable for tasks that may take considerable time to complete.
Exactly-once Execution: Each task and state in a Standard Workflow is run only once, unless a Retry behavior is specified in the Amazon States Language (ASL).
Full Execution History: You can retrieve the full execution history for up to 90 days after the execution completes.
Billing: The pricing model for Standard Workflows is based on the number of state transitions.

Standard Workflows are especially suitable for orchestrating non-idempotent actions, such as starting an Amazon EMR cluster or processing payments. If you’re interested in learning more about Amazon EMR, check out our interview guide for it.

The following script creates a standard workflow using the AWS SDK for Python (Boto3). The workflow contains two states: a “HelloWorld” state and an “End” state.

import boto3

client = boto3.client('stepfunctions')

definition = {
    "Comment": "A Hello World example of the Amazon States Language using a Pass state",
    "StartAt": "HelloWorld",
    "States": {
      "HelloWorld": {
        "Type": "Pass",
        "Result": "Hello, World!",
        "End": True
      }
    }
}

response = client.create_state_machine(
    name='HelloWorldStateMachine',
    definition=str(definition),
    roleArn='arn:aws:iam::123456789012:role/service-role/MyRole'
)

print(response['stateMachineArn'])

Express Workflows in AWS Step Functions

Express Workflows, on the other hand, are optimized for high-volume, event-processing workloads. Their main features include:

Short Execution Time: These workflows can run up to five minutes, making them ideal for tasks that require quick processing.
At-least-once Execution: Express Workflows follow the at-least-once execution model, implying that a task could potentially run more than once.
Billing: Express Workflows are billed based on the number of executions, execution duration, and memory consumed during the execution.

Express Workflows are particularly beneficial for orchestrating idempotent actions, such as transforming input data or executing a PUT action in Amazon DynamoDB.

The following script creates an express workflow. The workflow definition is similar to the standard workflow, but it’s created using the create_state_machinetype port boto3

import boto3

client = boto3.client('stepfunctions')

definition = {
    "Comment": "A Hello World example of the Amazon States Language using a Pass state",
    "StartAt": "HelloWorld",
    "States": {
      "HelloWorld": {
        "Type": "Pass",
        "Result": "Hello, World!",
        "End": True
      }
    }
}

response = client.create_state_machine(
    name='HelloWorldExpressStateMachine',
    definition=str(definition),
    roleArn='arn:aws:iam::123456789012:role/service-role/MyRole',
    type='EXPRESS',  # Type set to EXPRESS for express workflows
    loggingConfiguration={
        'level': 'OFF',  # Logging can be set to ERROR, FATAL, or OFF
        'includeExecutionData': False,
        'destinations': [
            {
                'cloudWatchLogsLogGroup': {
                    'logGroupArn': 'arn:aws:logs:us-east-1:123456789012:log-group:my-log-group'
                }
            },
        ]
    }
)

print(response['stateMachineArn'])

(Replace `’arn:aws:iam::123456789012:role/service-role/MyRole’` and `’arn:aws:logs:us-east-1:123456789012:log-group:my-log-group’` with your own IAM Role ARN and CloudWatch Logs log group ARN, respectively…Also, the script assumes that you’ve configured your AWS credentials, either by setting the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN environment variables, or by using the AWS CLI or AWS SDKs to configure your credentials.)

Key Differences: Standard vs Express Workflows

Difference 1: Duration and Volume of Workloads

Standard workflows are designed for long-running processes and can run for up to a year, making them ideal for workflows that require durability and auditing. They are also suited for orchestrating non-idempotent actions, like starting an Amazon EMR cluster or processing payments.

Express workflows, on the other hand, are tailored for high-volume, short-duration (up to five minutes) event-processing workloads. They are ideal for scenarios like IoT data ingestion, streaming data processing and transformation, or mobile application backends.

Difference 2: Execution Semantics

Standard workflows follow an “exactly-once” execution model, which means that tasks and states in your workflow are never run more than once, unless you have specified Retry behavior. This ensures data consistency and prevents duplication of non-repeatable operations.

Express workflows, however, follow an “at-least-once” execution model, which means a task may be executed more than once. This model works best with idempotent operations, where repeating an operation does not change the outcome.

Difference 3: Service Integrations and Activities

Both Standard and Express workflows support a wide range of service integrations, enabling you to orchestrate complex workflows involving multiple AWS services. However, there are differences in the types of integrations they support.

Standard workflows support all service integration patterns, including Job-run (.sync) and Callback (.waitForTaskToken) patterns. They also support Step Functions activities, which are worker programs that interact with your workflow.

Express workflows support all service integrations but do not support Job-run and Callback service integration patterns. Additionally, Express workflows do not support Step Functions activities.

AWS Step Functions Pricing

Understanding the pricing model of AWS Step Functions is essential for managing costs effectively.

Related Reading: Step Functions Pricing and Cost Optimization

Standard Workflows Pricing

For Standard Workflows, you are charged based on the number of state transitions required to execute your workflow. AWS counts a state transition each time a step of your workflow is executed, including retries. The first 4,000 state transitions per month are free, andyou are charged for each state transition above this free tier. If you include retry error handling in any steps of your workflow, each retry will be charged as an additional state transition.

Standard Workflow pricing example: If you have a simple application workflow with four state transitions (Start, Upload RAW File, Delete RAW File, End) and execute this state machine 100,000 times in a month without any retries due to errors, you would have a total of 400,000 state transitions. After subtracting the free tier of 4,000 transitions, you would be billed for 396,000 state transitions. With a price per state transition of $0.000025 in the US East (N. Virginia) region, your monthly charges would be $9.90.

If your application workflow has more complex branching conditions, the number of state transitions will increase. For instance, a workflow with nine state transitions executed 100,000 times in a month will result in 900,000 state transitions. After subtracting the free tier, you would be billed for 896,000 state transitions, resulting in monthly charges of $22.40.

Express Workflows Pricing

Express Workflows have a different pricing model. You are charged based on the number of requests for your workflow and its duration. A request is counted each time a workflow starts executing, including tests from the console. The duration is calculated from the time your workflow begins executing until it completes or otherwise terminates, rounded up to the nearest 100ms. Memory consumption is also a factor in pricing, which is based on the size of a workflow definition, the use of map or parallel states, and the execution (payload) data size.

Express Workflow pricing example: If you have a workflow that executes 5,000 times in a month with an average execution time of 10 seconds and memory consumption of 64 MB, you would have a total duration of 50,000 seconds (or about 13.89 hours) and would be billed for 125,000 100ms chunks (since AWS rounds up to the nearest 100ms). The cost would be $208.34 in the US East (N. Virginia) region.

Choosing Between Standard and Express Workflows

Your choice between Standard and Express workflows will depend on your specific use case, the nature of your tasks, and the operational and cost constraints of your application.

If your workflow is long-running, requires auditing, or deals with non-idempotent tasks, Standard workflows would be a better choice because of their durability and “exactly-once” execution semantics. Standard workflows would also be the choice if your workflows involve complex service integration patterns like Job-run and Callback, or if they interact with Step Functions activities.

If your workflows are high-volume, short-duration tasks, especially those dealing with idempotent actions like data transformation and storage, Express workflows would be a more suitable choice. They are designed to handle high volumes of requests at a faster rate, making them ideal for event-driven or real-time use cases.

Finally, cost considerations should also factor into your decision. Standard workflows are charged per state transition, while Express workflows are charged per execution, duration, and memory used. Depending on the complexity and execution frequency of your workflows, one model may be more cost-effective than the other. You should carefully analyze your workflows and estimate their costs under both models to make an informed decision.

Summary: Standard vs Express Step Functions Workflows

Feature/Aspect	Standard Workflow	Express Workflow
Ideal for	Long-running (up to one year), durable, auditable workflows such as starting an Amazon EMR cluster or processing payments.	High-volume, event-processing workloads like IoT data ingestion, streaming data processing and transformation, mobile application backends.
Maximum duration	One year	Five minutes
Execution semantics	Exactly-once execution	At-least-once execution
Service integrations	Supports all service integrations and patterns	Supports all service integrations but does not support Job-run (`.sync`) or Callback (`.waitForTaskToken`) service integration patterns
Step Functions activities	Supports Step Functions activities	Does not support Step Functions activities
Billing	Billed according to the number of state transitions processed	Billed by the number of executions, the duration of execution, and the memory consumed while the execution ran
Execution history	Executions can be listed and described with Step Functions APIs. Executions can be visually debugged through the console.	Executions can be visually debugged through the console. They can also be inspected in CloudWatch Logs by enabling logging on your state machine.
Automatic start triggers	Can automatically start in response to events such as HTTP requests from Amazon API Gateway, IoT Rules, etc.	Can automatically start in response to events such as HTTP requests from Amazon API Gateway, IoT Rules, etc.
Price per state transition/execution	The price per state transition in US East (N. Virginia) is $0.000025 with a Free Tier providing 4,000 state transitions per month.	Express Workflows are charged based on the number of requests for the workflow and its duration, including the amount of memory used in the execution.

FAQ

Q1: What are the key differences between AWS Step Functions Standard and Express Workflows?

Standard and Express Workflows differ in execution semantics and pricing: Standard follows an exactly-once model and is priced per state transition, while Express follows an at-least-once model and is priced based on executions, duration, and memory usage.

Q2: When should I use AWS Step Functions Standard Workflows versus Express Workflows?

Use Standard Workflows for long-running, durable tasks, or when you need an exactly-once execution model. Express Workflows are ideal for high-volume, short-duration tasks, particularly when idempotency is guaranteed.

Q3: How are AWS Step Functions Standard Workflows and Express Workflows priced?

Standard Workflows are priced based on the number of state transitions. Express Workflows are priced based on the number of executions, their duration, and memory consumption.

Q4: Can I use the same code for AWS Step Functions Standard and Express Workflows?

The Amazon States Language (ASL) code for defining the workflows can be very similar. However, Express Workflows don’t support certain service integration patterns and Step Functions activities. So, the specific capabilities of your workflow may necessitate different code.

Q5: Are there any duration limits for AWS Step Functions Standard and Express Workflows?

Yes, Standard Workflows can run for up to one year, while Express Workflows have a maximum duration of five minutes.

Conclusion

In conclusion, AWS Step Functions are an excellent tool for orchestrating microservices and managing complex workflows in a serverless environment. The service offers robust, durable, and highly scalable workflows that can handle a wide range of tasks, from data transformation and processing to the coordination of distributed applications.

However, it’s important to understand the differences between Standard and Express Workflows and to consider their pricing models when designing your applications. By doing so, you can ensure that you are using the right tools for your needs and that you are managing your costs effectively.