We released the ability to execute the workflow/node interactively. It allows seeing the output of a Node at design time. This means that if we are writing sql, scala, jython or using one of the 100+ processors, we can immediately confirm the output result.
It makes the whole process of building data pipelines a fun experience. Combined with schema propagation, it makes things very smooth.
As we see in the workflow below, each Node has a play button in it. It executes the workflow till that Node, and displays the result in a dialog.
There were a number of optimizations we did for an ideal user experience. The goal being that the results come back in 1-2 seconds.
- The workflow is executed on only 100 rows of each dataset.
- The workflow stops execution when the selected node has finished execution.
- Predictor nodes like Random Forest, Logistic Regression are not executed. These nodes can take a significant long time to execute and slow down the everyone's experience.
- Above all, the workflow is run locally in the Sparkflows server and not submitted to be run on the cluster.
Interactive Workflow Execution fills the user experience gap in Sparkflows in a big way and opens the door to focus on continued high value nodes to be added to Sparkflows.