Attachments extraction

For the attachment extraction phase of the import process, the extractor has to upload each attachment to DevRev’s S3 using the S3Interact API.

Triggering event

Airdrop initiates the attachment extraction by starting the snap-in with a message with an event of type EXTRACTION_ATTACHMENTS_START. This is done after the data extraction, transformation and loading into DevRev are completed.

During the attachment extraction phase, the snap-in extracts attachments from the external system and uploads them as artifacts to DevRev.

The snap-in must respond to Airdrop with a message with an event of type EXTRACTION_ATTACHMENTS_PROGRESS together with an optional progress estimate and relevant artifacts when it extracts some data and the maximum snap-in run time (12 minutes) has been reached.

The snap-in must respond to Airdrop with a message with an event of type EXTRACTION_ATTACHMENTS_DELAY and specify a back-off time when the extraction has been rate-limited by the external system and back-off is required.

In both cases, Airdrop starts the snap-in with a message with an event of type EXTRACTION_ATTACHMENTS_CONTINUE. The restart is immediate in case of EXTRACTION_ATTACHMENTS_PROGRESS, or delayed in case of EXTRACTION_ATTACHMENTS_DELAY.

Once the attachment extraction phase is done, the snap-in must respond to Airdrop with a message with an event of type EXTRACTION_ATTACHMENTS_DONE.

If attachment extraction fails the snap-in must respond to Airdrop with a message with an event of type EXTRACTION_ATTACHMENTS_ERROR.

Snap-in response

After uploading an attachment or a batch of attachments, the extractor also has to prepare and upload a file specifying the extracted and uploaded attachments.

It should contain the DevRev IDs of the extracted and uploaded attachments, along with the parent domain object ID from the external system and the actor ID from the external system.

The uploaded artifact is structured like a normal artifact containing extracted data in JSON Lines (JSONL) format and requires specifying ssor_attachment as the item type.

Examples

Here is an example of an SSOR attachment file:

lines
1{
2  "id": {
3    "devrev": "don:core:dvrv-us-1:devo/1:artifact/1", // DON of the artifact, that S3interact returned
4    "external": "111" // ID of the artifact in the external service
5  },
6  "parent_id": {
7    "external": "1111" // ID of the parent object in the external service
8  },
9  "actor_id": {
10    "external": "11111" // ID of the actor that uploaded/modified the artifact in the external service
11  }
12}
13{
14  "id": {
15    "devrev": "don:core:dvrv-us-1:devo/1:artifact/2",
16    "external": "222"
17  },
18  "parent_id": {
19    "external": "2222"
20  },
21  "actor_id": {
22    "external": "22222"
23  }
24}