Step 1: Add New Search Method

Overview

In order to initiate collection of product documents, the process begins with our Search Methods endpoint. Collection of product documents from search methods takes up to 48 hours to complete.

To begin, define what is the eCommerce domain to be crawled and identify the best method to get your product data. We have 3 types of parsers or web scrapers. They are categorized according to the type of result they return.

1. Search - these scrapers are designed to help you extract search results data from the different categories sources contain.
   1. The output is up to 100 product documents per page from the url added.
   2. An example of a category can be: Headphones, Wearable Technology, etc.
2. Keyword - these scrapers are designed to help you extract results through a keyword search through the main search bar of a domain.
   1. The output is up to 100 product documents per page from the url added.
3. Product - these scrapers are designed to help you extract details for a single product by using its url.
   1. The output is the corresponding product doc to the product url added.

URL Structure

• Use the following url to call on the “Add New Search Method” endpoint.
  • [https://api.webz.io/addNewSearch?token=[MY_TOKEN]

Example

• As a user, I want to find products with keyword “samsung tv” from amazon.com
• User inputs the following query into Add New Search Method endpoint:
  • https://api.webz.io/addNewSearch?token=XXXXXXXXX&method=KEYWORD&domain=amazon.com&value=samsung%20tv

Input:

Output:

If you’re interested in learning more about our current HTTP status codes, see the corresponding section.

---

Step 2: Get Products Data

Overview

After a search method has been successfully added, use the “Get Product Data” endpoint in order to extract the products collected given the corresponding method. Here you will find product documents (structured and sorted data by products crawled). It’s important to remember that data may only be visible in the endpoint in up to 48 hours since the method was added. Updated product information data will only be available after product activation.

URL Structure:

• Use the following url to access the “Get Products Data” endpoint.
  • [https://api.webz.io/getProducts?token=[MY_TOKEN]

Example

• As a user I want to retrieve product documents from all search methods collecting data from amazon.com
• User inputs the following query into the Get product Data endpoint:
  • https://api.webz.io/getProducts?token=XXXXXXXXXXX&q=domain:amazon.com

Input & Output

Product Object:

---

Step 3: Update Products (Review Activation)

Overview

This endpoint gives you the ability to manage which product documents (a specific uuid) you are interested in retrieving continuous new data on and extract reviews on. By setting the reviews_retrieval field to "true" (review collection activation), we will begin collecting all available historical reviews on the corresponding product or entity and then continue to collect reviews on an ongoing basis to bring the delta - or until classified otherwise. Additionally, activated products will also be re-crawled in parallel to update the product information fields which can be found in the getProducts endpoint. If historical reviews collection is not of your scope, please set the historical_collection field to "false" and by doing so, the product activated will only have new reviews collected. It’s important to remember that review data may take up to 48 hours to be collected.

URL Structure:

• Use the following url to access the “Update Products” endpoint.
  • [https://api.webz.io/updateProduct?token=[MY_TOKEN]

Update/Activation URL Example:

• https://api.webz.io/updateProduct?token=XXXXXXXXXX&uuid=XXXXXXXXXXX&reviews_retrieval=true

Usage Example:

• As a user, after receiving product data from a search method collection, I want to start collecting review data for the following product uuid:
  • “033cd2dc1eadfff83fcc242dcd8b3031496cac7c”
• User uses the following endpoint to activate review retrieval on. the product uuid
  • https://api.webz.io/updateProduct?token=XXXXXXXXXXX&uuid=033cd2dc1eadfff83fcc242dcd8b3031496cac7c&reviews_retrieval=true

Input:

Output:

If you’re interested in learning more about our current HTTP status codes, see the corresponding section.

---

Step 4: Get Product Review Data

Overview

This endpoint gives you the ability to extract reviews on activated products (review_retrieval field set to “true”). Once initial historical data has been collected, we will continue to crawl activated products to deliver new reviews (the delta from what was collected, if any) in up to every 48 hours. Users will receive data in descending order.

URL Structure:

• Use the following url to access the “Get Product Review Data” endpoint.
  • [https://api.webz.io/getReviews?token=[MY_TOKEN]

Example:

• As a user, I want to see the review data for a specific product uuid.
• User Inputs the following query on the Get Product Review Data endpoint:
  • https://api.webz.io/getReviews?token=XXXXXXXXXXXX&q=product_uuid:XXXXXXXXXXXXXXXXX

Input & Output

Review Object, Core Fields:

Review Object, Optional Fields:

---

Get Search Methods

External endpoint - not part of main flow

Overview

This endpoint gives you the ability to manage, access, and sort through existing search methods added. Accessing your active search methods can give you an understanding of what type of data you extracted in the past and can give you ideas into what you may want to extract in the future.

URL Structure:

• Use the following url to access the “Get Search Methods” endpoint.
  • [https://api.webz.io/getSearches?token=[MY_TOKEN]

Example:

• As a user I want to see all my search methods collecting data from amazon.com
• User inputs the following query into Get Search Methods endpoint
  • https://api.webz.io/getSearches?token=XXXXXXXX&q=domain:amazon.com

Input & Output:

Restriction

• Currently there is only an ability to query by “domain” field in this endpoint (however, you can still use all the general get parameters).

---

Get Status

External endpoint - not part of main flow

Overview

This endpoint gives API subscription customers the ability to check and understand how many credits are available to them for products and reviews collection.

To learn more about our pricing plans and subscription models please contact [email protected]

URL Structure:

Use the following url to access the “Get Status” endpoint:

• [https://api.webz.io/getStatus?token=[MY_TOKEN]

Usage Example:

• As a user I want to check how many products and reviews credits I have available in order to plan out my next requests.
• User inputs the following query into Get Status endpoint:
  • https://api.webz.io/getStatus?token=XXXXXXXX

Input

Output

---

Delete Products

External endpoint - not part of main flow

Overview

This endpoint gives API subscription customers the ability to delete products collected - helping the user manage their products' credit capacity in accordance with the package purchased. Please note that when a product is deleted, the corresponding reviews associated with the product will be deleted as well, but the credits used for those reviews will not be recycled.

To learn more about our pricing plans and subscription models please contact [email protected]

URL Structure

Use the following url to access the “Delete Products” endpoint:

• [https://api.webz.io/deleteProducts?token=[MY_TOKEN]

Usage Example:

• As a user I no longer need data corresponding to a specific product and want to delete it from my data collected.
• User inputs the following query into Delete Products endpoint:
  • https://api.webz.io/deleteProducts?token=XXXXXXXX&uuid=XXXXXXXXXXXXXX

Input

Output