Create Web Crawler Job (Beta) — DevRev

Creates a web crawler job whose objective is to crawl the provided URLs/sitemaps and generate corresponding webpages as artifacts.

Request

This endpoint expects an object.

applies_to_partslist of stringsRequired

The parts to which created webpage/articles during this crawler job will be linked to.

accept_regexeslist of stringsOptional

The list of regexes a URL must satisfy to be crawled.

descriptionstringOptionalformat: "text"

The description of the job.

domain_nameslist of stringsOptional

The list of allowed domain names to crawl.

frequencyintegerOptional

Number of days between re-sync job runs. If 0, the job will run only once.

max_depthintegerOptional

The maximum depth to crawl.

notify_on_completebooleanOptional

Whether to notify the user when the job is complete. Default is true.

reject_regexeslist of stringsOptional

The list of regexes which if satisfied by a URL results in rejection of the URL. If a URL matches both accept and reject regexes, it is rejected.

sitemap_index_urlslist of stringsOptional

The list of sitemap index URLs to crawl.

sitemap_urlslist of stringsOptional

The list of sitemap URLs to crawl.

urlslist of stringsOptional

The list of URLs to crawl.

accept_regexstringOptionalformat: "text"Deprecated

The regex a URL must satisfy to be crawled.

reject_regexstringOptionalformat: "text"Deprecated

The regex which if satisfied by a URL results in rejection of the URL. If a URL matches both accept and reject regexes, it is rejected.

Response

The response to create a web crawler job.

web_crawler_jobobject

1 curl -X POST https://api.devrev.ai/web-crawler-jobs.create \ 2 -H "Authorization: Bearer <token>" \ 3 -H "Content-Type: application/json" \ 4 -d '{ 5 "applies_to_parts": [ 6 "applies_to_parts", 7 "applies_to_parts" 8 ] 9 }'

1 { 2 "web_crawler_job": { 3 "id": "id", 4 "accept_regexs": [ 5 "accept_regexs", 6 "accept_regexs" 7 ], 8 "created_by": { 9 "type": "dev_user", 10 "id": "id", 11 "display_id": "display_id", 12 "display_name": "display_name", 13 "display_picture": { 14 "id": "id", 15 "display_id": "display_id", 16 "file": {} 17 }, 18 "email": "email", 19 "full_name": "full_name", 20 "state": "active" 21 }, 22 "created_date": "created_date", 23 "description": "description", 24 "display_id": "display_id", 25 "domain_names": [ 26 "domain_names", 27 "domain_names" 28 ], 29 "frequency": 1, 30 "max_depth": 1, 31 "modified_by": { 32 "type": "dev_user", 33 "id": "id", 34 "display_id": "display_id", 35 "display_name": "display_name", 36 "display_picture": { 37 "id": "id", 38 "display_id": "display_id", 39 "file": {} 40 }, 41 "email": "email", 42 "full_name": "full_name", 43 "state": "active" 44 }, 45 "modified_date": "modified_date", 46 "no_parent": true, 47 "notify_on_complete": true, 48 "num_bytes": 1, 49 "num_timeout_urls": 1, 50 "num_urls_scraped": 1, 51 "reject_regexs": [ 52 "reject_regexs", 53 "reject_regexs" 54 ], 55 "sitemap_index_urls": [ 56 "sitemap_index_urls", 57 "sitemap_index_urls" 58 ], 59 "sitemap_urls": [ 60 "sitemap_urls", 61 "sitemap_urls" 62 ], 63 "state": "aborted", 64 "urls": [ 65 "urls", 66 "urls" 67 ] 68 } 69 }

1	curl -X POST https://api.devrev.ai/web-crawler-jobs.create \
2	-H "Authorization: Bearer <token>" \
3	-H "Content-Type: application/json" \
4	-d '{
5	"applies_to_parts": [
6	"applies_to_parts",
7	"applies_to_parts"
8	]
9	}'

1	{
2	"web_crawler_job": {
3	"id": "id",
4	"accept_regexs": [
5	"accept_regexs",
6	"accept_regexs"
7	],
8	"created_by": {
9	"type": "dev_user",
10	"id": "id",
11	"display_id": "display_id",
12	"display_name": "display_name",
13	"display_picture": {
14	"id": "id",
15	"display_id": "display_id",
16	"file": {}
17	},
18	"email": "email",
19	"full_name": "full_name",
20	"state": "active"
21	},
22	"created_date": "created_date",
23	"description": "description",
24	"display_id": "display_id",
25	"domain_names": [
26	"domain_names",
27	"domain_names"
28	],
29	"frequency": 1,
30	"max_depth": 1,
31	"modified_by": {
32	"type": "dev_user",
33	"id": "id",
34	"display_id": "display_id",
35	"display_name": "display_name",
36	"display_picture": {
37	"id": "id",
38	"display_id": "display_id",
39	"file": {}
40	},
41	"email": "email",
42	"full_name": "full_name",
43	"state": "active"
44	},
45	"modified_date": "modified_date",
46	"no_parent": true,
47	"notify_on_complete": true,
48	"num_bytes": 1,
49	"num_timeout_urls": 1,
50	"num_urls_scraped": 1,
51	"reject_regexs": [
52	"reject_regexs",
53	"reject_regexs"
54	],
55	"sitemap_index_urls": [
56	"sitemap_index_urls",
57	"sitemap_index_urls"
58	],
59	"sitemap_urls": [
60	"sitemap_urls",
61	"sitemap_urls"
62	],
63	"state": "aborted",
64	"urls": [
65	"urls",
66	"urls"
67	]
68	}
69	}