Reading configuration variables in a Python project I am new to python, in .NET we use web.config to store environment specific variables... I want to achieve the same for my python program (scraper)
I have created .env file at the root of the project and added the following variable to it:
CHROME_WEB_DRIVER=C:/webdrivers/chromedriver.exe 

Note that I have added .env to gitignore as this file changes in different environments.
And I am using decouple package to use this setting from .env file in the constructor of the class:
import scrapy
from decouple import config
from python_json_config import ConfigBuilder
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

class MySpider(scrapy.Spider):
    name = 'my_spider'

    def __init__(self):
        # .env variables
        self.CHROME_WEB_DRIVER = config('CHROME_WEB_DRIVER')

        self.job_highlight_dict = {}
        self.wanted_organisations_dict = {}
        self.output_path = &quot;&quot;
        self.start_urls = []
        chrome_options = Options()
        chrome_options.add_argument('--headless')
        chrome_options.add_argument('--no-sandbox')
        chrome_options.add_argument('--disable-dev-shm-usage')
        self.driver = webdriver.Chrome(self.CHROME_WEB_DRIVER, chrome_options=chrome_options)

        builder = ConfigBuilder()
        config_values = builder.parse_config('config/config.json')
        self.start_urls = config_values.urls
        for o in config_values.organisations:
            self.wanted_organisations_dict[o[&quot;brandName&quot;]] = o[&quot;organisationName&quot;]
        self.output_path = config_values.output

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):

        # parse response


Note that I also have a config.json file, but this file contains configuration variables which are consistent across all environments.
This is how the config.json file looks like:
{
    &quot;urls&quot;: [
        &quot;https://www.site-to-scrape-1.com&quot;,
        &quot;https://www.site-to-scrape-2.com&quot;
    ],
    &quot;organisations&quot;: [
        { &quot;brandName&quot; : &quot;brand-1&quot;, &quot;organisationName&quot; : &quot;organisation-1&quot; },
        { &quot;brandName&quot; : &quot;brand-2&quot;, &quot;organisationName&quot; : &quot;organisation-2&quot; }
    ],
    &quot;output&quot;: &quot;../path/to/s3/bucket&quot;
}

I have not included all of spider class, but it is the main class of this project which crawls the given urls and then parse them. I am not sure if the constructor of the spider is the best place to read config.json and .env files?
Add a declarative format for the config file, possibly. If you encounter errors, the output will look messy today (KeyError is never fun for an end user)
I would say init is a good time, but unclear if it's the right place. It's looking messy. I would either (1) split out parsing the config file into a helper method or (2) consolodate some lines. (1) is better if you expect to add more config.
I would consider making the chrome driver at the start of 'parse', as this is more of a &quot;can fail&quot; type operation--it will make writing tests easier.
Consolidating:

'builder' is used only once, don't declare it
don't assign to self.wanted_organizations_dict, or self.output_path, and self.start_urls twice. just initialize them once, correctly, in one line each.
self.job_highlight_dict is not initialized. i don't know what this is, but are you sure you want to declare it here?
self.CHROME_WEB_DRIVER does not appear to be used outside this, change it to a local variable or inline it, unless you move driver init to parse time
declare self.wanted_organisations_dict in a single-line dict comprehension, not 3 lines.
Initialize chrome_options in a single line, not 4.