How to build a webscraper for indeed.com
Scraping a website is very particular to the case. To try and make this answer useful to other people I will go through some basic concepts as to how this can be done successfully. Below I will use your URL as an example to put everything together.

First we need a means to access the HTML content of the website. For static websites we can use BeautifulSoup. This is a very powerful package which allows you to navigate an HTML. To get the HTML we will use the requests packages. 

from bs4 import BeautifulSoup
import requests

url = 'https://www.indeed.com/jobs?q=data+scientist+$20,000&amp;l=New+York&amp;start=10'
response = requests.get(url)
soup = BeautifulSoup(response.content, 'lxml')


BeautifulSoup requires a parser, I have had a lot of luck using lxml, however html.parser is also very popular. The parser is what is used to access the HTML tags and identify its inner elements. 

Most websites are quite hard to scrape because they will reuse the same name for multiple element tages. This makes it harder to get the elements and extract their values. This is either due to the developers laziness or their expertise to avoid people abusing their resources by running scrapers on their data. To circumvent this you can use this simple BeautifulSoup trick. You can chain your .find calls to hone into the part of the HTML you want to scrape. 

Using the above example URL we can identify where in the HTML we can find the results on the page for each job posting.

len(soup.find_all('div', {'class': 'row'}))



  16


Now we will want to collect data from each of these HTML elements. The data we will want to collect is: job title, company name, job description, city, location, and salary. 

Each record of data will be collected into a dictionary with all the desired data as listed above. These dictionaries will be collected into a list which will be used to create the pandas DataFrame.

To find the individual data points you can use your browser and inspect elements to find out the name of the element parameters in order to use BeautifulSoup to get the information.

data = []
for i in soup.find_all('div', {'class': 'row'}):
    job_title = i.find('a', {'data-tn-element': 'jobTitle'})['title']
    company_name = i.find('span', {'class': 'company'}).text.strip()    
    job_summary = ''.join([j.text.strip() for j in i.find_all('span', 
                                                              {'class': 'summary'})])
    location = i.find('span', {'class': 'location'})
    if location is not None:
        location = location.text.strip()

    salary_range = i.find('span', {'class': 'no-wrap'})
    if salary_range is not None:
        salary_range = salary_range.text.strip()


    datum = {'job_title': job_title,
             'company_name': company_name,
             'job_summary': job_summary,
             'location': location,
             'salary_range': salary_range}

    data.append(datum)

df = pd.DataFrame(data)
df.head()