Snowflake had been teasing about Dynamic Tables for months now. They hit private preview in November 2022 and the momentum built pretty heavily up to Summit that they might release to public preview while in Vegas.
Spoiler Alert: They did!
There’s been so much buzz about Dynamic Tables and I’ve actually had many conversations about them and what it actually means in terms of shifting workloads or development overhaul. Let’s take a look at exactly what this means and why it’s so important (and why everyone is drooling over them finally being released).
As I said in a previous post, if you come from the Oracle world, these are Materialized Views. Plain and simple, they take complicated logic that can be executed in one sql statement and freezes it in time so that when you query the table, the data returns lighting fast though with a lag. You sacrifice latency for performance (a typical trade off in all things data).
Snowflake describes Dynamic Tables as a shift from writing imperative data pipelines (with tasks and streams) to declarative pipelines (defining the end result of the data you want to see and the pipeline to get it there just automagically happens). Declarative pipelining is really important for a couple of reasons:
- You start developing at the end instead of the beginning meaning that you are working from the requested state instead of having to define all the steps along the way to get to that state.
- Snowflake figures out the most efficient way to compute the pipeline; this is HUGE. Snowflake has REALLY smart engineers who focus soley on these types of problems and they are letting us use their brains.
What makes Dynamic Tables in particular so amazing:
- The lag is a parameter on the Dynamic Table and thus if the business decides that the latency needs to be reduced (and the business will ALWAYS change its mind after giving requirements — don’t fight this, start embracing it), it’s a simple update. You do not have to go back and reconfigure multiple objects for a new latency or redevelop anything.
- The dynamic table only has to update changed data; it does not recalculate the entire pipeline at each refresh interval so it saves money compared to calling a complicated view each time which DOES recalculate everything (more of less — there is a caching element involved).
Want to start playing with Dynamic Tables? Here are some other great resources to help you out!
- Slowly Changing Dimensions with Dynamic Tables Simplify your SCD data pipelines
- Understanding Dynamic Tables Refresh in Snowflake
- Snowflake Dynamic Tables VS Streams & Tasks VS Materialized Views Confused about dynamic tables? How they are different from Streams and tasks or from Materialized views?