Create Web Crawler Job (Beta) — DevRev

Creates a web crawler job whose objective is to crawl the provided URLs/sitemaps and generate corresponding webpages as artifacts.

Request

This endpoint expects an object.

applies_to_partslist of stringsRequired

The parts to which created webpage/articles during this crawler job will be linked to.

accept_regexeslist of stringsOptional

The list of regexes a URL must satisfy to be crawled.

descriptionstringOptionalformat: "text"

The description of the job.

domain_nameslist of stringsOptional

The list of allowed domain names to crawl.

frequencyintegerOptional

Number of days between re-sync job runs. If 0, the job will run only once.

max_depthintegerOptional

The maximum depth to crawl.

notify_on_completebooleanOptional

Whether to notify the user when the job is complete. Default is true.

reject_regexeslist of stringsOptional

The list of regexes which if satisfied by a URL results in rejection of the URL. If a URL matches both accept and reject regexes, it is rejected.

sitemap_index_urlslist of stringsOptional

The list of sitemap index URLs to crawl.

sitemap_urlslist of stringsOptional

The list of sitemap URLs to crawl.

urlslist of stringsOptional

The list of URLs to crawl.

user_agentstringOptionalformat: "text"<=1024 characters

User agent to use for crawling websites in this job.

accept_regexstringOptionalformat: "text"Deprecated

The regex a URL must satisfy to be crawled.

reject_regexstringOptionalformat: "text"Deprecated

The regex which if satisfied by a URL results in rejection of the URL. If a URL matches both accept and reject regexes, it is rejected.

Response

The response to create a web crawler job.

web_crawler_jobobject

Errors

1 curl -X POST https://api.devrev.ai/web-crawler-jobs.create \ 2 -H "Authorization: Bearer <token>" \ 3 -H "Content-Type: application/json" \ 4 -d '{ 5 "applies_to_parts": [ 6 "foo" 7 ] 8 }'

1 { 2 "web_crawler_job": { 3 "id": "foo", 4 "accept_regexs": [ 5 "foo" 6 ], 7 "created_by": { 8 "display_id": "foo", 9 "id": "foo", 10 "display_name": "foo", 11 "display_picture": { 12 "display_id": "foo", 13 "id": "foo", 14 "file": { 15 "type": "foo", 16 "name": "foo", 17 "size": 42 18 } 19 }, 20 "email": "foo", 21 "full_name": "foo", 22 "state": "active" 23 }, 24 "created_date": "2023-01-01T12:00:00.000Z", 25 "description": "foo", 26 "display_id": "foo", 27 "domain_names": [ 28 "foo" 29 ], 30 "frequency": 42, 31 "max_depth": 42, 32 "modified_by": { 33 "display_id": "foo", 34 "id": "foo", 35 "display_name": "foo", 36 "display_picture": { 37 "display_id": "foo", 38 "id": "foo", 39 "file": { 40 "type": "foo", 41 "name": "foo", 42 "size": 42 43 } 44 }, 45 "email": "foo", 46 "full_name": "foo", 47 "state": "active" 48 }, 49 "modified_date": "2023-01-01T12:00:00.000Z", 50 "no_parent": true, 51 "notify_on_complete": true, 52 "num_bytes": 42, 53 "num_timeout_urls": 42, 54 "num_urls_scraped": 42, 55 "reject_regexs": [ 56 "foo" 57 ], 58 "sitemap_index_urls": [ 59 "foo" 60 ], 61 "sitemap_urls": [ 62 "foo" 63 ], 64 "state": "aborted", 65 "urls": [ 66 "foo" 67 ], 68 "user_agent": "foo" 69 } 70 }

1	curl -X POST https://api.devrev.ai/web-crawler-jobs.create \
2	-H "Authorization: Bearer <token>" \
3	-H "Content-Type: application/json" \
4	-d '{
5	"applies_to_parts": [
6	"foo"
7	]
8	}'

1	{
2	"web_crawler_job": {
3	"id": "foo",
4	"accept_regexs": [
5	"foo"
6	],
7	"created_by": {
8	"display_id": "foo",
9	"id": "foo",
10	"display_name": "foo",
11	"display_picture": {
12	"display_id": "foo",
13	"id": "foo",
14	"file": {
15	"type": "foo",
16	"name": "foo",
17	"size": 42
18	}
19	},
20	"email": "foo",
21	"full_name": "foo",
22	"state": "active"
23	},
24	"created_date": "2023-01-01T12:00:00.000Z",
25	"description": "foo",
26	"display_id": "foo",
27	"domain_names": [
28	"foo"
29	],
30	"frequency": 42,
31	"max_depth": 42,
32	"modified_by": {
33	"display_id": "foo",
34	"id": "foo",
35	"display_name": "foo",
36	"display_picture": {
37	"display_id": "foo",
38	"id": "foo",
39	"file": {
40	"type": "foo",
41	"name": "foo",
42	"size": 42
43	}
44	},
45	"email": "foo",
46	"full_name": "foo",
47	"state": "active"
48	},
49	"modified_date": "2023-01-01T12:00:00.000Z",
50	"no_parent": true,
51	"notify_on_complete": true,
52	"num_bytes": 42,
53	"num_timeout_urls": 42,
54	"num_urls_scraped": 42,
55	"reject_regexs": [
56	"foo"
57	],
58	"sitemap_index_urls": [
59	"foo"
60	],
61	"sitemap_urls": [
62	"foo"
63	],
64	"state": "aborted",
65	"urls": [
66	"foo"
67	],
68	"user_agent": "foo"
69	}
70	}