Pagination

👍

This is important!

To get the best value out of your data feeds do pay close attention to the paging and sorting sections following. See the Pagination tutorial for more information. We suggest that you first get your filtered searches worked out using the playground, and then use them in a script.

With the technicalities of pagination out of sight, you can concentrate on what is important - your data feeds!

How does pagination work?

• Each request can return up to 100 matching entities for your query.
• Pagination is managed through three fields included in every response:
  • totalResults - The total number of results matching your search query.
  • moreResultsAvailable - An integer indicating the number of results currently available for retrieval beyond the current response.
  • next - More results may still be available.
• Pagination Behavior:
  • When newAvailableResults > 0, additional results can be retrieved immediately by calling the URL in next.
  • When newAvailableResults = 0, all currently available results have been consumed. The same next URL remains valid and may be used at a later time (e.g., daily) to check if new results have been added since the last retrieval.
  • When next = null, the query has reached a boundary (such as a time limit defined by crawled_to or breach_to parameters). In this case, no further continuation is possible, and a new query with adjusted parameters must be issued to retrieve additional data.