Load data

The load data phase manages the creation and updating of items in the external system.

Triggering event

Airdrop initiates data loading by sending a message with the event type START_LOADING_DATA to the snap-in.

If the maximum Airdrop snap-in runtime (13 minutes) has been reached, the snap-in must respond to Airdrop with a message with event type of DATA_LOADING_PROGRESS, together with an optional progress estimate.

In case of DATA_LOADING_PROGRESS, Airdrop starts the snap-in with a message with event type CONTINUE_LOADING_DATA.

Once the data extraction is done, the snap-in must respond to Airdrop with a message with event type DATA_LOADING_DONE.

Implementation

This phase is defined in load-data.ts.

Loading is performed by providing a list of item types to load (itemTypesToLoad), ordered in the sequence they should be loaded.

Each item type must provide create and update functions, which handle the denormalization of records to the schema of the external system and facilitate HTTP calls to the external system. Both loading functions must manage rate limiting for the external system and handle errors. The create and update functions should return an id of the record in the external system and optionally also modifiedDate. If a record cannot be created or updated, they indicate the rate-limiting offset or errors.

1
processTask<LoaderState>({
2
  task: async ({ adapter }) => {
3
    const { reports, processed_files } = await adapter.loadItemTypes({
4
      itemTypesToLoad: [
5
        {
6
          itemType: 'todos',
7
          create: createTodo,
8
          update: updateTodo,
9
        },
10
      ],
11
    });
12
13
    await adapter.emit(LoaderEventType.DataLoadingDone, {
14
      reports,
15
      processed_files,
16
    });
17
  },
18
  onTimeout: async ({ adapter }) => {
19
    await adapter.emit(LoaderEventType.DataLoadingProgress, {
20
      reports: adapter.reports,
21
      processed_files: adapter.processedFiles,
22
    });
23
  },
24
});