List Web Crawler Jobs (POST) | DevRev

1 curl -X POST https://api.devrev.ai/web-crawler-jobs.list \ 2 -H "Authorization: Bearer <token>" \ 3 -H "Content-Type: application/json" \ 4 -d '{}'

1 { 2 "web_crawler_jobs": [ 3 { 4 "id": "foo", 5 "accept_regexs": [ 6 "foo" 7 ], 8 "created_by": { 9 "display_id": "foo", 10 "id": "foo", 11 "display_name": "foo", 12 "display_picture": { 13 "display_id": "foo", 14 "id": "foo", 15 "file": { 16 "type": "foo", 17 "name": "foo", 18 "size": 42 19 } 20 }, 21 "email": "foo", 22 "full_name": "foo", 23 "state": "active" 24 }, 25 "created_date": "2023-01-01T12:00:00.000Z", 26 "description": "foo", 27 "display_id": "foo", 28 "domain_names": [ 29 "foo" 30 ], 31 "frequency": 42, 32 "max_depth": 42, 33 "modified_by": { 34 "display_id": "foo", 35 "id": "foo", 36 "display_name": "foo", 37 "display_picture": { 38 "display_id": "foo", 39 "id": "foo", 40 "file": { 41 "type": "foo", 42 "name": "foo", 43 "size": 42 44 } 45 }, 46 "email": "foo", 47 "full_name": "foo", 48 "state": "active" 49 }, 50 "modified_date": "2023-01-01T12:00:00.000Z", 51 "no_parent": true, 52 "notify_on_complete": true, 53 "num_bytes": 42, 54 "num_timeout_urls": 42, 55 "num_urls_scraped": 42, 56 "reject_regexs": [ 57 "foo" 58 ], 59 "sitemap_index_urls": [ 60 "foo" 61 ], 62 "sitemap_urls": [ 63 "foo" 64 ], 65 "state": "aborted", 66 "urls": [ 67 "foo" 68 ], 69 "user_agent": "foo" 70 } 71 ], 72 "next_cursor": "foo", 73 "prev_cursor": "foo" 74 }

Lists web crawler jobs.

Headers

AuthorizationstringRequired

Bearer authentication of the form Bearer <token>, where token is your auth token.

Request

This endpoint expects an object.

created_bylist of stringsOptional

Filters for web crawler jobs created by any of the provided users.

created_dateobjectOptional

Provides ways to specify date ranges on objects.

cursorstringOptionalformat: "text"

The cursor to resume iteration from. If not provided, then iteration starts from the beginning.

limitintegerOptional

The maximum number of web crawler jobs to return. The default is '50'.

modeenumOptional

The iteration mode to use. If “after”, then entries after the provided cursor will be returned, or if no cursor is provided, then from the beginning. If “before”, then entries before the provided cursor will be returned, or if no cursor is provided, then from the end. Entries will always be returned in the specified sort-by order.

Allowed values:

modified_dateobjectOptional

Provides ways to specify date ranges on objects.

Response

The response to list web crawler jobs.

web_crawler_jobslist of objects

The list of jobs.

next_cursorstring or nullformat: "text"

The cursor used to iterate subsequent results in accordance to the sort order. If not set, then no later elements exist.

prev_cursorstring or nullformat: "text"

The cursor used to iterate preceding results in accordance to the sort order. If not set, then no prior elements exist.

Headers

Request

Response

Errors