List Web Crawler Jobs

1 curl https://api.devrev.ai/web-crawler-jobs.list \ 2 -H "Authorization: Bearer <token>"

1 { 2 "web_crawler_jobs": [ 3 { 4 "id": "foo", 5 "accept_regexs": [ 6 "foo" 7 ], 8 "created_by": { 9 "display_id": "foo", 10 "id": "foo", 11 "display_name": "foo", 12 "display_picture": { 13 "display_id": "foo", 14 "id": "foo", 15 "file": { 16 "type": "foo", 17 "name": "foo", 18 "size": 42 19 } 20 }, 21 "email": "foo", 22 "full_name": "foo", 23 "state": "active" 24 }, 25 "created_date": "2023-01-01T12:00:00.000Z", 26 "description": "foo", 27 "display_id": "foo", 28 "domain_names": [ 29 "foo" 30 ], 31 "frequency": 42, 32 "max_depth": 42, 33 "modified_by": { 34 "display_id": "foo", 35 "id": "foo", 36 "display_name": "foo", 37 "display_picture": { 38 "display_id": "foo", 39 "id": "foo", 40 "file": { 41 "type": "foo", 42 "name": "foo", 43 "size": 42 44 } 45 }, 46 "email": "foo", 47 "full_name": "foo", 48 "state": "active" 49 }, 50 "modified_date": "2023-01-01T12:00:00.000Z", 51 "no_parent": true, 52 "notify_on_complete": true, 53 "num_bytes": 42, 54 "num_timeout_urls": 42, 55 "num_urls_scraped": 42, 56 "reject_regexs": [ 57 "foo" 58 ], 59 "sitemap_index_urls": [ 60 "foo" 61 ], 62 "sitemap_urls": [ 63 "foo" 64 ], 65 "state": "aborted", 66 "urls": [ 67 "foo" 68 ], 69 "user_agent": "foo" 70 } 71 ], 72 "next_cursor": "foo", 73 "prev_cursor": "foo" 74 }

Lists web crawler jobs.

Query parameters

created_bylist of stringsOptional

Filters for web crawler jobs created by any of the provided users.

cursorstringOptionalformat: "text"

The cursor to resume iteration from. If not provided, then iteration starts from the beginning.

limitintegerOptional

The maximum number of web crawler jobs to return. The default is '50'.

modeenumOptional

The iteration mode to use, otherwise if not set, then "after" is used.

Allowed values:

Response

The response to list web crawler jobs.

web_crawler_jobslist of objects

The list of jobs.

next_cursorstring or nullformat: "text"

The cursor used to iterate subsequent results in accordance to the sort order. If not set, then no later elements exist.

prev_cursorstring or nullformat: "text"

The cursor used to iterate preceding results in accordance to the sort order. If not set, then no prior elements exist.

Headers

Query parameters

Response

Errors