List Web Crawler Jobs

GET

Lists web crawler jobs.

Query parameters

created_bylist of stringsOptional

Filters for web crawler jobs created by any of the provided users.

cursorstringOptionalformat: "text"

The cursor to resume iteration from. If not provided, then iteration starts from the beginning.

limitintegerOptional

The maximum number of web crawler jobs to return. The default is ‘50’.

modeenumOptional

The iteration mode to use, otherwise if not set, then “after” is used.

Allowed values: afterbefore

Response

The response to list web crawler jobs.

web_crawler_jobslist of objects

The list of jobs.

next_cursorstringOptionalformat: "text"

The cursor used to iterate subsequent results in accordance to the sort order. If not set, then no later elements exist.

prev_cursorstringOptionalformat: "text"

The cursor used to iterate preceding results in accordance to the sort order. If not set, then no prior elements exist.

Errors

GET

1 curl https://api.devrev.ai/web-crawler-jobs.list \
2      -H "Authorization: Bearer <token>"

200listWebCrawlerJobsExample

1 {
2   "web_crawler_jobs": [
3     {
4       "id": "foo",
5       "accept_regexs": [
6         "foo"
7       ],
8       "created_by": {
9         "display_id": "foo",
10         "id": "foo",
11         "display_name": "foo",
12         "display_picture": {
13           "display_id": "foo",
14           "id": "foo",
15           "file": {
16             "type": "foo",
17             "name": "foo",
18             "size": 42
19           }
20         },
21         "email": "foo",
22         "full_name": "foo",
23         "state": "active"
24       },
25       "created_date": "2023-01-01T12:00:00.000Z",
26       "description": "foo",
27       "display_id": "foo",
28       "domain_names": [
29         "foo"
30       ],
31       "frequency": 42,
32       "max_depth": 42,
33       "modified_by": {
34         "display_id": "foo",
35         "id": "foo",
36         "display_name": "foo",
37         "display_picture": {
38           "display_id": "foo",
39           "id": "foo",
40           "file": {
41             "type": "foo",
42             "name": "foo",
43             "size": 42
44           }
45         },
46         "email": "foo",
47         "full_name": "foo",
48         "state": "active"
49       },
50       "modified_date": "2023-01-01T12:00:00.000Z",
51       "no_parent": true,
52       "notify_on_complete": true,
53       "num_bytes": 42,
54       "num_timeout_urls": 42,
55       "num_urls_scraped": 42,
56       "reject_regexs": [
57         "foo"
58       ],
59       "sitemap_index_urls": [
60         "foo"
61       ],
62       "sitemap_urls": [
63         "foo"
64       ],
65       "state": "aborted",
66       "urls": [
67         "foo"
68       ],
69       "user_agent": "foo"
70     }
71   ],
72   "next_cursor": "foo",
73   "prev_cursor": "foo"
74 }

List Web Crawler Jobs (POST)

Up Next

Built with

1	curl https://api.devrev.ai/web-crawler-jobs.list \
2	-H "Authorization: Bearer <token>"

1	{
2	"web_crawler_jobs": [
3	{
4	"id": "foo",
5	"accept_regexs": [
6	"foo"
7	],
8	"created_by": {
9	"display_id": "foo",
10	"id": "foo",
11	"display_name": "foo",
12	"display_picture": {
13	"display_id": "foo",
14	"id": "foo",
15	"file": {
16	"type": "foo",
17	"name": "foo",
18	"size": 42
19	}
20	},
21	"email": "foo",
22	"full_name": "foo",
23	"state": "active"
24	},
25	"created_date": "2023-01-01T12:00:00.000Z",
26	"description": "foo",
27	"display_id": "foo",
28	"domain_names": [
29	"foo"
30	],
31	"frequency": 42,
32	"max_depth": 42,
33	"modified_by": {
34	"display_id": "foo",
35	"id": "foo",
36	"display_name": "foo",
37	"display_picture": {
38	"display_id": "foo",
39	"id": "foo",
40	"file": {
41	"type": "foo",
42	"name": "foo",
43	"size": 42
44	}
45	},
46	"email": "foo",
47	"full_name": "foo",
48	"state": "active"
49	},
50	"modified_date": "2023-01-01T12:00:00.000Z",
51	"no_parent": true,
52	"notify_on_complete": true,
53	"num_bytes": 42,
54	"num_timeout_urls": 42,
55	"num_urls_scraped": 42,
56	"reject_regexs": [
57	"foo"
58	],
59	"sitemap_index_urls": [
60	"foo"
61	],
62	"sitemap_urls": [
63	"foo"
64	],
65	"state": "aborted",
66	"urls": [
67	"foo"
68	],
69	"user_agent": "foo"
70	}
71	],
72	"next_cursor": "foo",
73	"prev_cursor": "foo"
74	}