The Referer header

By default, Scrapy automatically sets a Referer header on every request yielded from a callback (see the RefererMiddleware).

However, when using transparent mode or automatic request parameters, this behavior is disabled by default for Zyte API requests, and when using manual request parameters, all request headers are always ignored for Zyte API requests.

Why is it disabled by default?

A misuse of the Referer header can increase the risk of bans.

By not setting the header, your Zyte API requests let Zyte API choose which value to use, if any, to minimize bans.

If you do set the header, while Zyte API might still ignore your value to avoid bans, it may also keep your value regardless of its impact on bans.

How to override?

To set the header anyway when using transparent mode or automatic request parameters, do any of the following:

When using manual request parameters, you always need to set the header through the customHttpRequestHeaders or requestHeaders field through the ZYTE_API_DEFAULT_PARAMS setting or the zyte_api request metadata key.