Data Ingestion Overview

Overview

To get the complete picture of your company's Revenue and processes—Clari goes beyond just data from Salesforce. We tap into other business apps, too. External Data Ingestion extracts the data living outside of Salesforce.

Use Cases

Sales Leaders

Sales leaders can spot and analyze upsell/cross-sell opportunities for their team's accounts—and tune their efforts to fill gaps and reach targets.

They can also see all accounts without activity over the last 30 days, where consumption is below an acceptable threshold over the last year, and a renewal is coming up in the next 90 days.

And also, they can get details on actual dollar amounts closed by team members using billings data and compare it against the amount committed in Forecasting.

Customer Success Managers

With external licensing data, Customer success managers can track how many sold licenses are not utilized in each account they own.

Account Managers

Account managers can ensure customers use their product/service to the amount the customer signed for.

Finance Managers

Finance leaders can reconcile what the sales team booked and what the customer actually consumed.


Executives

Business leaders can see closed-won deals that estimate the volume of bookings. Only after the customer starts using a product/service can we count that as attainment and retire quota.

How Data Ingestion Works

First, you’ll prep the data in whichever data source you plan to use. Then, a Clari admin can add that new data source to Clari. Once the data loads into Clari, your admin can go into Field Configuration and create a custom mode for the new data source. 

Before we start with our step-by-step guide, let’s cover some prereqs. 
 

General Prerequisites

Let's cover the prerequisites you need to get started with Data Ingestion.

Currently, Clari supports the ingestion of external data from these sources:

1. Snowflake

2. PostgreSQL (aka Postgres)

• Generic PostgreSQL
• Amazon Aurora PostgreSQL: Note: we do not support Amazon Aurora Serverless PostgreSQL
• Amazon RDS PostgreSQL
• Azure PostgreSQL
• Google Cloud PostgreSQL
• Heroku Postgres

3. SQL Server

• Generic SQL Server
• Amazon SQL Server RDS
• Azure SQL Database
• Azure SQL Managed Instance
• Google Cloud SQL for SQL Server

External Data Ingestion is only supported if your Clari instance meets the following criteria: 
 

• Your Clari instance (specifically, the module that you want to ingest external data to) must be running on RevDB (formerly Data Hub).
• You use the Role Hierarchy in Salesforce (not Enterprise Territory Management).
• You're using Field Configuration in your Clari instance to create your own modes.

You also want to be sure to: 
 

• Whitelist the IP addresses listed for Google Cloud Platform for the region you're based in. Here's the list of addresses you should whitelist. 
• Prepare a table in the source system with the data you want to ingest that follows Clari-specific schema requirements (explained below).

Clari Schema Requirements

Key considerations

• Records from the source system are not editable within Clari.
• Records imported do not contribute to the CRM Score currently.

To expose the ingested data in Clari’s Forecasting module

• Each row in Forecasting will map to a user within the Salesforce hierarchy. The user should be mapped as an “owner” to the underlying records in the source Postgres table. Note: Clari cannot support custom hierarchies through ingestion—if it is territory-based in Salesforce, it is the same in Clari. The same thing applies if the hierarchy is user-based. 
• Forecast fields are calculated based on the owner and the roll-up amount. Each user will map to the sum of all the records owned by themselves and all their reports.

To expose the ingested data in Accounts and Opportunities

• Each row in Accounts maps to an ingested record with a matching crm_account_id
• Each row in the Opportunity Grid maps to an ingested record with a matching crm_opportunity_id

Clari’s data model is built around CRM objects, so it requires some key identifiers to map the source data to those objects correctly.

The table below captures detailed schema requirements.

Note: Apart from [field_name], which is the custom field(s) you’d like to ingest, all other fields should be named exactly as denoted below. 

 

Field Name	Data Type	Description	Required
ID	int	Primary key of each record you’re trying to push into Clari. This value must be unique across the dataset.	Yes
crm_opportunity_id	string	Salesforce opportunity ID that the individual records stored on this table are attributed to.	At least ONE of the three ID fields must be specified Ideally, all three.
crm_account_id	string	Salesforce account ID that the individual records stored on this table are attributed to.	At least ONE of the three ID fields must be specified Ideally, all three.
crm_owner_id	string	Salesforce owner of the respective record.	At least ONE of the three ID fields must be specified Ideally, all three.
[field_name]	decimal	Numerical value (i.e., number of seats) that gets rolled up into the CRM records.	One (or more) [field_name] required
[field_name]	money	Numerical value (i.e., Forecast ARR) that gets rolled up into the CRM records.Must include the currency ISO code field if multi-currency is enabled.	One (or more) [field_name] required
created_on	date	Date the record was originally created.	Yes (Do NOT change when updating an existing record)
updated_on	date	Date the record was updated. Default should be created_on date.	Yes, if the same record will be updated over time. No, if there will be no record updates.
crm_currency_iso_code	string	The currency code for [field_name]	Only if multi-currency is enabled

 

How To Add a New Data Source

Now that you have the prereqs, it's time to add a data source. Follow these steps.

1. Go to Clari Studio. From the top-left menu click Integrations. Expand the Data Source section. 

2. Click Prepare for Revenue Sync and wait for the progress bar to finish. Then, click Proceed. 

3. Select + Click here to add sources to add a new data source. Select the source type and click Apply. 



4. Click Set Up in the new card, name the data source connection, and click Save.

5. Saving takes you to a new page. From there—click Continue. 



6. Now you're on the login page of the data source you added (Aurora PostgreSQL in this example). Enter your login credentials to complete authentication.

For help adding data in the required fields (other than credentials), check out this document.

7. Finally, click Save & Test.

Note: After you Save your data source, you'll redirect back to the Clari Integrations page. It takes time for your data to load into Clari—the timing will depend on the volume of data you add. 

Field Configuration Steps

Now, it's time to head back to Clari Studio and create a mode that uses your new data source. These steps happen in Field Configuration. 

Before you get started here, you might peek at our Field Configuration article or our Field Configuration course if you haven't used Field Configuration before—or, if you want a quick refresher. 😜 

1. Clari Admins can navigate to Field Configuration from Clari Studioin the top-right corner. From the top-left menu  select Field Configuration.

2. In the top-right corner, click Actions > Create Custom Mode. 

3. Select the mode properties in Initial Mode Setup and click Next.

Need a quick explanation of the properties below? Check out this guide. 




4. Now, choose your layouts and layout order. Layouts let you choose which modules you want the new data to display in—like Opportunities or Accounts (see the top of the screenshot below).

When the toggle beside a Field is green, that field is included in the module. So in the screenshot below, ID, Name, and Annual Revenue would show up in Opportunities unless we toggle them off. 



 

5. Preview: Review and confirm the mode details.

6. Because it's a new mode, you need to add fields. To do so, in the top right, click Actions > Add Fields. Choose your new data source and the other fields you want to use.

For more on adding fields in Field Configuration, check out this article. 

7. Once you're ready, Click Review Changes in the top right corner. Then click Publish Changes. 

Note: When you make multiple changes (like adding more than one field at a time), make all your edits/changes and THEN publish them all at once—instead of publishing after each individual edit/change.

And that's it! 🥳 You've got a new mode using your new data source. Remember, it takes time for this data to show up in any Clari modules. And just in case you're curious, here's what your ingested data might look like in Clari: