Metadata extraction

Declare the extracted record types

Record types are the types of records that has a well-defined schema you extract from or load to the external system, a domain object in the external system.

If the snap-in is extracting issues and comments, a good starting point to declare record types in external_domain_metadata.json would be:

1
{
2
  "record_types": {
3
    "issues": {},
4
    "comments": {}
5
  }
6
}

Although the declaration of record types is arbitrary, they must match the item_type field in the artifacts you will upload.

Declare the custom record types

If the external system supports custom types, or custom variants of some base record type, and you want to airdrop those too, you have to declare them in the metadata at runtime. That is, the extractor will use APIs of the external system to dynamically discover what custom record types exist.

The output of this process might look like this:

1
{
2
"record_types": {
3
    "issues_stock_epic": {},
4
    "issues_custom2321": {},
5
    "issues_custom2322": {},
6
    "comments": {}
7
  }
8
}

Provide human-readable names to external record types

Define human-readable names for the record types defined in your metadata file.

1
  {
2
  "record_types":{
3
    "issues_stock_epic":{
4
      "name":"Epic"
5
    },
6
    "issues_custom2321":{
7
      "name":"Incident report"
8
    },
9
    "issues_custom2322":{
10
      "name":"Problem"
11
    },
12
    "comments":{
13
      "name":"Comment"
14
    }
15
  }
16
}

Categorize external record types

The metadata allows each external record type to be annotated with one category. The category key can be an arbitrary string, but it must match the categories declared under record_type_categories.

Categories of external record types simplify mappings so that a mapping can be applied to a whole category of record types. Categories also provide a way how custom record types can be mapped.

If the external system allows records to change the record type within the category, while still preserving identity, this can be defined by the are_transitions_possible field in the record_type_categories section. For example, if an issue that can be moved to become a problem in the external system.

1
{
2
  "record_types":{
3
    "issues_stock_epic":{
4
      "name":"Epic",
5
      "category":"issue"
6
    },
7
    "issues_custom2321":{
8
      "name":"Incident report",
9
      "category":"issue"
10
    },
11
    "issues_custom2322":{
12
      "name":"Problem",
13
      "category":"issue"
14
    },
15
    "comments":{
16
      "name":"Comment"
17
    }
18
  },
19
  "record_type_categories":{
20
    "issue":{
21
      "are_transitions_possible":true
22
    }
23
  }
24
}

Declare fields for each record type

Fields’ keys must match what is actually found in the extracted data in the artifacts.

The supported types are:

• bool
• int
• float
• text
• rich_text: Formatted text with mentions and images.
• reference: IDs referring to another record. References have to declare what they can refer to, which can be one or more record types (#record:) or categories (#category:).
• enum: A string from a predefined set of values with the optional human-readable names for each value.
• date
• timestamp
• struct

If the external system supports custom fields, the set of custom fields in each record type you wish to extract must be declared too.

Enum fields’ set of possible values can often be customizable. A good practice is to retrieve the set of possible values for all enum fields from the external system’s APIs in each sync run.

ID (primary key) of the record, created_date, and modified_date must not be declared.

Example:

1
{
2
  "record_types": {
3
    "issues_stock_epic": {
4
      "name": "Epic",
5
      "category": "issue",
6
      "fields": {
7
        "actual_close_date": {
8
          "name": "Closed at",
9
          "type": "timestamp"
10
        },
11
        "owner": {
12
          "is_required": true,
13
          "type": "reference",
14
          "reference": {
15
            "refers_to": {
16
              "#record:user": {}
17
            }
18
          }
19
        },
20
        "creator": {
21
          "is_required": true,
22
          "type": "reference",
23
          "reference": {
24
            "refers_to": {
25
              "#record:user": {}
26
            }
27
          }
28
        },
29
        "priority": {
30
          "name": "Priority",
31
          "is_required": true,
32
          "type": "enum",
33
          "enum": {
34
            "values": [
35
              {
36
                "key": "P-0",
37
                "name": "Super important"
38
              },
39
              {
40
                "key": "P-1"
41
              },
42
              {
43
                "key": "P-2",
44
                "is_deprecated": true
45
              }
46
            ]
47
          }
48
        },
49
        "target_close_date": {
50
          "type": "date"
51
        },
52
        "headline": {
53
          "name": "Headline",
54
          "is_required": true,
55
          "type": "text"
56
        }
57
      }
58
    }
59
  }
60
}

Declare arrays

If the field is array in the extracted data, it is still typed with the one of the supported types. Lists must be marked as a collection.

1
{
2
  "name": "Assignees",
3
  "is_required": true,
4
  "type": "reference",
5
  "reference": {
6
    "refers_to": {
7
      "#category:agents": {}
8
    }
9
  },
10
  "collection": {
11
    "max_length": 5
12
  }
13
}

Handle special references

• Some references have role of parent or child. This means that the child record doesn’t make sense without its parent, for example a comment attached to a ticket. Assigning a role helps Airdrop correctly handle such fields in case the end-user decides to filter some of the parent records out.

• Sometimes the external system uses references besides the primary key of records, for example when referring to a case by serial number, or to a user by their email. To correctly resolve such references, they must be marked with ‘by_field’, which must be a field existing in that record type, marked ‘is_identifier’. For example:

1
{
2
"record_types": {
3
  "users": {
4
    "fields": {
5
      "email": {
6
        "type": "text",
7
        "is_identifier":true
8
      }
9
    }
10
  },
11
  "comments": {
12
    "fields": {
13
      "user_email": {
14
        "type": "reference",
15
        "reference": {
16
          "refers_to": {
17
            "#record:users": {
18
              "by_field": "email"
19
            }
20
          }
21
        }
22
      }
23
    }
24
  }
25
}
26
}