Load data

The load data phase manages the creation and updating of items in the external system.

Triggering event

Airdrop initiates data loading by starting the snap-in with a message containing an event of type START_LOADING_DATA.

Implementation

This phase is defined in load-data.ts.

Loading is performed by providing a list of item types to load (itemTypesToLoad), ordered in the sequence they should be loaded.

Each item type must provide create and update functions, which handle the denormalization of records to the schema of the external system and facilitate HTTP calls to the external system. Both loading functions must manage rate limiting for the external system and handle errors. The create and update functions should return an id of the record in the external system and optionally also modifiedDate. If a record cannot be created or updated, they indicate the rate-limiting offset or errors.

Same as with extraction, the SDK library exports the processTask function to structure the work within each phase and the onTimeout function to handle timeouts.

1
processTask<LoaderState>({
2
  task: async ({ adapter }) => {
3
    const { reports, processed_files } = await adapter.loadItemTypes({
4
      itemTypesToLoad: [
5
        {
6
          itemType: 'todos',
7
          create: createTodo,
8
          update: updateTodo,
9
        },
10
      ],
11
    });
12
13
    await adapter.emit(LoaderEventType.DataLoadingDone, {
14
      reports,
15
      processed_files,
16
    });
17
  },
18
  onTimeout: async ({ adapter }) => {
19
    await adapter.emit(LoaderEventType.DataLoadingProgress, {
20
      reports: adapter.reports,
21
      processed_files: adapter.processedFiles,
22
    });
23
  },
24
});