Using the Data Loader

Intro to Loader

The Loader in Tools under Data can be used to bring customer data in bulk into Tendo’s platform.

The Loader can bring in data for providers, locations, value sets, value mappings, patient information for guest user purposes such as outreach campaigns, and insurance plan information (for healthcare organizations that use Epic for their Electronic Health Record).

Once this data has been loaded into the Tendo Platform, it must be managed, as customers need to continue to provide updated provider and location data on an ongoing basis and insurance plan information on a regular basis (at least quarterly).

Customer Engagement therefore needs to:

• Import, export, and modify a large number of records.
• Upsert records using either a Tendo ID or an external ID to indicate whether a new record should be recreated or an existing record updated.

Loading Data

To upload data, you must have been granted Manage Objects permission.

Log in to Atrium and select Tools from the dropdown menu in the top left corner. Click on Data in the main left navigation bar.

All data must be loaded based on the Tendo object that it is related to, so you need to know what object the data you want to load or update is related to.

You can view a list of Objects by choosing the first tab at the top, Objects, and searching through the object list that appears to determine the object you need to load data for. For example, the Appointment object is shown below. On the detail view for the object to the right, you can see the description of the object, which can help you determine if this is the object you need.

You can click on the Fields tab to see all of the types of data that are included in this object:

Once you have determined the object for the data you want to load, click the Loader tab, then on the … dropdown menu in the top right corner, and choose Load Data.

On the Load Data modal that appears, click on the dropdown menu.

It includes the following choices:

• Create - This creates new records.  It will error on records with a matching identifier for existing records.
• Delete - This deletes records that match on the Identifying Value, either Record Id or Identifier.
• Transaction - This is used to load records into a transaction workflow.
• Update - This updates an existing record that matches on the Record Name or Record Id.
• Upsert - This updates existing records or creates a new record when an existing record is not found with a matching Identifier.

You can add notes about your record. Then click the Object dropdown menu, and input the object you want to load data for. For example, if you want to update the Appointment object records, choose it.

Once you input the name of the object, you will see a link called the [name of the object]-template.csv. Click on the download icon to the left of its label to download it.

Check your computer’s download file for the template, and open it. This template will give you the fields that you will need to include in the .csv document that you upload for this object.

This will help you to properly format the data from the customer organization so that it will load into Tendo’s system. The data must be in .csv format with the same fields as the template. Once you have added the data to the fields, you can load the data in your .csv file by dropping the file in the Load Data modal or browsing in your computer to find the file, and uploading it.

Depending on the Operation and Object you select, a Template Generation Option field will appear with options you can choose in a dropdown. One or more of these options will appear, depending on the Object selected:

• ID & Identifier
• Alternate Key

An Alternate Key is a set of one or more object fields that are unique to the object record. The main unique key on any object is the record ID (ID field). Each object can have up to 10 Alternate Key definitions. These keys are implemented in the database as unique keys and guarantee uniqueness of records within the object. Alternate keys are a data matching option for the Loader options CREATE, UPSERT, UPDATE, and DELETE.

Under Template Generation Options, if Alternate Key is selected, the generated .csv template will include columns for referencing fields by alternate key.

If the object referenced in the field has multiple alternate keys, columns will be generated for each one. Be sure to only provide the alternate key field values for one alternate key. If multiple keys are provided, the first alternate key values parsed from the csv will be used to establish the reference for that field.

When a combination of ID, Identifier, and Alternate Key are provided for a reference field, a note will display in the column: “Only provide ID, Identifier, or Alternate Key.”

Click Create, and the data will load.

Guidelines for Field Formatting for a Loader Template

If the data isn’t in the correct format in your template, it will throw an error. Here are some guidelines for inputting the data:

When the template asks for something like "Person.object", it doesn’t want an object reference or an identifier. It wants the name of the object being referenced. For Person.object, input Person.

Boolean fields

Don't post a file with null value for a boolean field. If you don’t want a value for a boolean, remove the column. Otherwise, it will throw an error.

Identifier Values for Objects

An identifier for an object must be present, as it is used as a unique key to match data in the object against. The identifier should include the system the data is coming from, the type of identifier from that system, and the identifier’s value:

• identifiers.system
• identifiers.type
• identifiers.value

If the loader file column references an object, such as “person.object” on the Provider file, the name of the object that is being referenced is input as the value. In this example, Person would be the value.

Since there can be references to abstract objects (such as Order ), the concrete object that is being referenced is specified because identifiers are unique per concrete type. For example, Appointment has a basedOn field that references Order . When loading appointments, there will be a basedOn.object column. The value should be one of the objects that is an Order , i.e. MedicationOrder , ServiceOrder , etc.

The system value is used to specify where the file came from. It usually is filled with static values such as epic or athena. The identifiers must be unique, so the system value is used to ensure that there are no conflicts when two Electronic Health Record systems have the same external ID for an object.

In this example for loading a custom object, there is a Patient object reference. There are four columns that make the reference - patientId__c.object, patientId__c.system, patientId__c.type, and patientId__c.value. The rows are tied together via the ID column. They also must be positioned one after another.

Id is Tendo’s system-generated record id, which is unique. When uploading data with the CREATE option, this column is ignored because the system will generate the value. It should be left blank, but if data is there, it will be ignored.

Dates

Dates must be loaded in this format:

2023-11-14T12:00:00Z

Don't load columns with null for Date fields. If you don’t want a value for a date, remove the column.

Notes

Don’t Include extra columns for notes in the loader file. 

Data Order

When uploading object data, consider the order in which the data should be created. For example, a provider load would contain both person and provider objects (among others depending on the data). The person object can be created first so that the person reference in the provider object can be filled out. If the person reference is filled out on the provider object before the person object is created, the relationship will be created but the person will not exist (resulting in a null value in GraphQL).

Null columns

Empty values will not cause an error, such as if a new person is loaded with no address information.

If a column value is empty in an update to existing data, the existing data will not be purged. To purge the data when an update is made, place null in the column.

It will not change the data if the column value is empty. To clear the data out, you can place null in the column.

Invalid Column Headers

The system will throw an error for any column in a CSV that it doesn't recognize. However, if a # is placed anywhere in the column name, the Loader will ignore the column instead.

Zip Packages are Not Allowed

You cannot ZIP a set of files and create a manifest file to load them, as Tendo doesn’t support ZIP data packages.

Multiple-value Fields

When setting up a loader file with records that have multiple child objects, multiple rows must be created for those multiple-value fields. Multiple-value fields must be separated by row, including array value sets. For example, if a provider has five specialties, you need five rows, one for each value in the multi-value field. Separate rows aren’t needed for sub elements.

You can consolidate multiple multi-value fields in the same set of rows. For arrays (multi-value fields), you need to load the value, not the text, with one row per value.

Multiple Qualifications

If a provider has multiple qualifications, including one or more separate objects (not array), they need to be loaded as separate csv files. If they are an array, detail object, value set or other field, then they can be loaded by creating a second row with the same ID. The second reference should be in the second row.

Reference Fields

Reference fields are dependent on the object being referenced, and depend on the data being provided (id, identifier, or alternate key). When multiples are provided, precedence will be ID, Alternate Key, then Identifier.

Exporting Data

To export an existing record for an object, select Export in the … dropdown menu in the upper right corner of the Loader list view. When you select Export, you will see options for exporting data. Choose the object you’d like to export in the dropdown menu. You can add a GraphQL Where filter to narrow down what you want to export if desired. Check the checkbox next to Export Attachments if you want to create a ZIP file that contains the attachments for the exported records. Click Create.

On the right in each row is a … dropdown menu with these options:

View log - allows you to view a log of the actions taken on this record.

Download - When the record has finished running, you can download it by clicking on the … dropdown menu on the right side of the record in the list view and choosing Download.

Navigate to your Downloads folder in your computer and click on the record. It will open in a CSV spreadsheet. You can view the fields and change or update the data. Then you can upload it again, choosing Update to update the existing record.

Terminate - This allows you to stop creation of a record mid-stream if you decide you don’t want to create it.

Converting Customer Data into Tendo Objects

Customer data is often delivered in a format that requires massaging or transformation before it can be added to a csv template and uploaded. For example, the SER and EMP exports for Epic require manual work to convert into an object template even though they contain information Tendo can use. This involves identifying the data Tendo needs from the export and copying it over to its respective column in the object template.

This manual process can be time consuming, so it may need to be automated with a script.