Customer-defined DAGs on Temporal

NOTE: This is a summary of a talk given by the Retool team at Replay 2023.

Retool is a developer-friendly company that accelerates the creation of internal tools. Using drag-and-drop components and pre-built functionality, developers can build custom dashboards, admin panels, and data management interfaces without extensive coding, while still maintaining flexibility for complex logic through custom code.

One of the core capabilities of Retool allows devs to create custom workflows for their applications. They offer a visual interface to orchestrate complex customer-defined directed acyclic graphs (DAGs). This functionality can be particularly useful for developers who need to design workflows that can include arbitrary code, network requests, database queries, etc.

Challenges with the original architecture

Retool initially built workflows on a queue-based system with AWS Lambda for execution and a separate service for database queries. This architecture resulted in limitations including:

• All-or-nothing execution: If any step in the workflow failed, the entire workflow failed.
• Debugging difficulties: Due to the all-or-nothing nature, it was challenging to pinpoint failures.
• Limited execution time: Workflows were capped at a 15-minute execution window.
• Inability to pause workflows: The original architecture did not support pausing workflows.

Why Temporal

To address these limitations, Retool decided to adopt Temporal for the following reasons:

• Resilience: Temporal workflows could be retried upon failures.
• Visibility: Temporal provided better visibility into workflow execution.
• Long-running workflows: Temporal workflows could handle executions longer than 15 minutes.
• Pausing workflows: Temporal allowed pausing workflows.

Key considerations when migrating to Temporal

The buildout of new workflows on Temporal was relatively straightforward, however, the team did have some key lessons learned

• Identify trade-offs: Not all workflow blocks require the full durability features provided by Temporal. Retool extended their DSL to differentiate between retryable and non-retryable blocks to optimize costs.
• Data storage optimization: Initially, Retool stored all workflow payloads in S3, regardless of size. This proved to be inefficient for small payloads. They implemented a mechanism to selectively store payloads in S3 based on size considerations.
• Compression: Retool employed LZ4 compression to reduce payload sizes further.

Overall, the migration to Temporal allowed Retool to build complete and scalable workflow and addressed the limitations of their original architecture.

Replay 2023 Video

This summary was pulled from a talk that the Retool team provided at Replay 2023. You can check out the video below or see all the videos from the show on our Replay 2023 video page.