Data Feeds
What are data feeds?
Data feeds are ongoing streams of structured data sent to your Snowflake instance. Data feeds are attached to flows to capture detail about the flow, actions of that flow, and customers sent through the flow.
For example, you can use data feeds to understand which variant customers were assigned to within an experiment and measure effectiveness in your downstream reporting tools.
You'll see your Data Feed fields listed at the bottom of your flow creation screen, but they are controlled from Admin, Settings:
Enable Data Feeds
- From the left navigation, expand Admin Center then click Settings.
- Click Data Feed Configurations.
- Check the box to Enable Data Feed. (If this is disabled then any live flows with data feeds will not export or write data.)
- Snowflake is chosen by default (it's currently the sole option); optionally check the box to Hash identity using SHA-256 to anonymize and deduplicate data. Note, you can only choose to hash identity from this global level, not from the flow level.
Default fields
Under Fields, there are multiple default fields that can't be edited. Some commonly default fields are simon_id, flow_id, flow_name etc.; these fields will always be sent to your reporting destinations downstream.
Default data fields
| Field Name | Description |
|---|---|
| action | The channel action (destination) used in the flow |
| flow_id | ID for the flow. You'll also see this ID appended in the URL when you open a flow from your Simon Flows dashboard. |
| variant | If you performed an experiment, the name for the variant the contact fell into |
| simon_id | Simon's identifier for the contact |
| flow_name | The flow's name |
| operation | If the customer was added or removed from a list, the action that was taken (add or remove) |
| timestamp | The time the customer was synced |
| segment_name | The name for the segment used in the flow |
| experiment_id | ID for the experiment, if applicable. You'll see this ID appended to the URL when you open Experiment information from the Flow Details Page. This ID is useful is because you can use an experiment across many flows and any contacts that enter multiple flows will always get the same variant treatment (e.g. if you're testing email vs. SMS, a specific contact will always get assigned to email instead of randomly being assigned either across different messages). |
| identity_type | The identity type for the customer being synced (email, phone number, etc.) |
| identity_value | The identity value for the customer being synced |
| experiment name | Experiment's name, if applicable |
View Data Feeds data
Data Routing required
To use Data Feeds, Data Routing must be configured. Learn more here.
SD_DATA_ROUTING schema
The data routing product is used to route event data from Simon into your Snowflake instance. There is one “pipe” per data source (i.e. Simon Signal and Data Feeds).
During general Simon onboarding, we create one file format: NLJSON (new line delimited JSON).
When onboarding each data source, we create the following:
- One pipe per source:
_DATA_ROUTING_<INTEGRATION>_EVENTS_PIPE - One stage per source:
_DATA_ROUTING_<INTEGRATION>_EVENTS_STAGE - One table per source:
_DATA_ROUTING_<INTEGRATION>_EVENTS
For each event type in the data that gets routed to Snowflake, we create one view that’s filtered to the specific event type: _DATA_ROUTING_VIEW_<INTEGRATION>_<EVENT_TYPE>_EVENTS
Disable Data Feeds globally
If you have live flows with data feeds, you can disable data feeds globally. Those live flow syncs will no longer write/export data feed information. When you create a new flow with data feed off, then the new flow will not display data feed options. To do so:
- From the left navigation, expand Admin Center then click Settings.
- Click Data Feed Configurations.
- Uncheck the box to Enable Data Feed.
Updated 8 months ago