Automatic request parameters

To send a Scrapy request through Zyte API letting Zyte API request parameters be automatically chosen based on the parameters of that Scrapy request, set the zyte_api_automap key in Request.meta to True.

For example:

import scrapy


class SampleQuotesSpider(scrapy.Spider):
    name = "sample_quotes"

    def start_requests(self):
        yield scrapy.Request(
            url="https://quotes.toscrape.com/",
            meta={
                "zyte_api_automap": True,
            },
        )

    def parse(self, response):
        print(response.text)
        # "<html>…</html>"

In transparent mode, zyte_api_automap is True by default.

See Request mapping to learn how exactly request parameters are mapped when using automatic request parameters.

Changing parameters

You may set zyte_api_automap in Request.meta to a dict of Zyte API parameters to add, modify, or remove (by setting to False) automatic request parameters. This also works in transparent mode.

Enabling browserHtml, screenshot, or an automatic extraction property, unsets httpResponseBody and httpResponseHeaders, and makes Request.headers become requestHeaders instead of customHttpRequestHeaders. For example, the following Scrapy request:

Request(
    url="https://quotes.toscrape.com",
    headers={"Referer": "https://example.com/"},
    meta={"zyte_api_automap": {"browserHtml": True}},
)

Results in a request to the Zyte API data extraction endpoint with the following parameters:

{
    "browserHtml": true,
    "experimental": {
        "responseCookies": true
    },
    "requestHeaders": {"referer": "https://example.com/"},
    "url": "https://quotes.toscrape.com"
}