List Web Crawler Jobs (POST)

POST

Lists web crawler jobs.

Request

This endpoint expects an object.

created_bylist of stringsOptional

Filters for web crawler jobs created by any of the provided users.

created_dateobjectOptional

Provides ways to specify date ranges on objects.

cursorstringOptionalformat: "text"

The cursor to resume iteration from. If not provided, then iteration starts from the beginning.

limitintegerOptional

The maximum number of web crawler jobs to return. The default is ‘50’.

modeenumOptional

Allowed values: afterbefore

The iteration mode to use. If “after”, then entries after the provided cursor will be returned, or if no cursor is provided, then from the beginning. If “before”, then entries before the provided cursor will be returned, or if no cursor is provided, then from the end. Entries will always be returned in the specified sort-by order.

modified_dateobjectOptional

Provides ways to specify date ranges on objects.

Response

The response to list web crawler jobs.

web_crawler_jobslist of objects

The list of jobs.

next_cursorstringOptionalformat: "text"

The cursor used to iterate subsequent results in accordance to the sort order. If not set, then no later elements exist.

prev_cursorstringOptionalformat: "text"

The cursor used to iterate preceding results in accordance to the sort order. If not set, then no prior elements exist.

Errors

POST

1 curl -X POST https://api.devrev.ai/web-crawler-jobs.list \
2      -H "Authorization: Bearer <token>" \
3      -H "Content-Type: application/json" \
4      -d '{}'

200webCrawlerJobListWebCrawlerJobsPostExample

1 {
2   "web_crawler_jobs": [
3     {
4       "id": "foo",
5       "accept_regexs": [
6         "foo"
7       ],
8       "created_by": {
9         "display_id": "foo",
10         "id": "foo",
11         "display_name": "foo",
12         "display_picture": {
13           "display_id": "foo",
14           "id": "foo",
15           "file": {
16             "type": "foo",
17             "name": "foo",
18             "size": 42
19           }
20         },
21         "email": "foo",
22         "full_name": "foo",
23         "state": "active"
24       },
25       "created_date": "2023-01-01T12:00:00.000Z",
26       "description": "foo",
27       "display_id": "foo",
28       "domain_names": [
29         "foo"
30       ],
31       "frequency": 42,
32       "max_depth": 42,
33       "modified_by": {
34         "display_id": "foo",
35         "id": "foo",
36         "display_name": "foo",
37         "display_picture": {
38           "display_id": "foo",
39           "id": "foo",
40           "file": {
41             "type": "foo",
42             "name": "foo",
43             "size": 42
44           }
45         },
46         "email": "foo",
47         "full_name": "foo",
48         "state": "active"
49       },
50       "modified_date": "2023-01-01T12:00:00.000Z",
51       "no_parent": true,
52       "notify_on_complete": true,
53       "num_bytes": 42,
54       "num_timeout_urls": 42,
55       "num_urls_scraped": 42,
56       "reject_regexs": [
57         "foo"
58       ],
59       "sitemap_index_urls": [
60         "foo"
61       ],
62       "sitemap_urls": [
63         "foo"
64       ],
65       "state": "aborted",
66       "urls": [
67         "foo"
68       ],
69       "user_agent": "foo"
70     }
71   ],
72   "next_cursor": "foo",
73   "prev_cursor": "foo"
74 }

Create Webhook

Up Next

Built with

1	curl -X POST https://api.devrev.ai/web-crawler-jobs.list \
2	-H "Authorization: Bearer <token>" \
3	-H "Content-Type: application/json" \
4	-d '{}'

1	{
2	"web_crawler_jobs": [
3	{
4	"id": "foo",
5	"accept_regexs": [
6	"foo"
7	],
8	"created_by": {
9	"display_id": "foo",
10	"id": "foo",
11	"display_name": "foo",
12	"display_picture": {
13	"display_id": "foo",
14	"id": "foo",
15	"file": {
16	"type": "foo",
17	"name": "foo",
18	"size": 42
19	}
20	},
21	"email": "foo",
22	"full_name": "foo",
23	"state": "active"
24	},
25	"created_date": "2023-01-01T12:00:00.000Z",
26	"description": "foo",
27	"display_id": "foo",
28	"domain_names": [
29	"foo"
30	],
31	"frequency": 42,
32	"max_depth": 42,
33	"modified_by": {
34	"display_id": "foo",
35	"id": "foo",
36	"display_name": "foo",
37	"display_picture": {
38	"display_id": "foo",
39	"id": "foo",
40	"file": {
41	"type": "foo",
42	"name": "foo",
43	"size": 42
44	}
45	},
46	"email": "foo",
47	"full_name": "foo",
48	"state": "active"
49	},
50	"modified_date": "2023-01-01T12:00:00.000Z",
51	"no_parent": true,
52	"notify_on_complete": true,
53	"num_bytes": 42,
54	"num_timeout_urls": 42,
55	"num_urls_scraped": 42,
56	"reject_regexs": [
57	"foo"
58	],
59	"sitemap_index_urls": [
60	"foo"
61	],
62	"sitemap_urls": [
63	"foo"
64	],
65	"state": "aborted",
66	"urls": [
67	"foo"
68	],
69	"user_agent": "foo"
70	}
71	],
72	"next_cursor": "foo",
73	"prev_cursor": "foo"
74	}