Pagination and sorting

Webz.io’s open web API uses pagination and sorting to help users manage large volumes of data by controlling the order and number of results returned. Here’s how each of these features works:

Pagination
When a query matches a large number of posts, each API request can return up to 100 posts. To retrieve all the data matching your query, you need to use pagination by continuously calling the "next" URL provided in each response. This "next" URL ensures that you can access all matching posts in sequence.

• The "next" URL uses different pagination parameters depending on the sorting criteria.
• If you sort by crawl date (default setting), the "next" URL includes the "ts" (timestamp) parameter, which fetches posts crawled after this timestamp, effectively streaming the data in chronological order.
• If you sort by any other available field, the "next" URL includes the "from" parameter, allowing you to retrieve posts based on your selected sorting criterion.

Sorting Options

You can customize the order in which posts appear by selecting from various sorting options. Here are the main sorting fields you can choose:

1. Crawl Date (default): Orders posts based on when they were crawled.
2. Relevancy: Orders posts based on their relevance to your query.
3. Social Metrics: Allows sorting by engagement metrics such as Facebook likes, shares, comments.
4. Engagement Metrics: Sorts posts by replies_count (number of replies) or participants_count (number of unique participants in discussions).
5. Performance Score: Sorts posts by their virality score (from 0 to 10).
6. Publication Dates: Sorts by published (content publish date) or thread.published (date of the originating thread).
7. Domain Rank: Orders posts based on the popularity rank of the source domain.
8. Thread Position: ord_in_thread sorts posts based on their position within a discussion thread.
9. Rating: Orders posts by rating, if available.

Sort Order

For any of the numeric sorting values, you can set the order parameter to define the direction of the sorting:

• asc (default): Results are sorted from the oldest to the newest, or from the lowest to the highest value of the chosen sort field.
• desc: Results are sorted from the newest to the oldest, or from the highest to the lowest value.

When using &order=desc, results are retrieved from the present (most recent) backward, through the ts or from timestamp you’ve specified. With &order=asc, the results are retrieved starting from the specified timestamp up to the present, making the most recent results appear last in the result set.

Example Use Cases

1. Streaming Data Chronologically: To monitor data in real-time, set sort by crawl date with default ascending order to consume data as it’s crawled.
2. Finding the Most Viral Content: Sort by performance_score or social.facebook.likes in descending order to view the most popular content first.
3. Researching Content by Publish Date: Sort by published in ascending order to retrieve articles starting from the earliest publish date up to the present.

By leveraging pagination and sorting, Webz.io’s News API makes it easy to control the flow of large data sets, providing a flexible way to access information in the most relevant order for your needs.