title: How to scrape an article which requires login to view full content using python?
body: <p>I am trying to scrape an article from The Wall Street Journal and it requires log-in to view the whole content. So, I have written a code like the below using Python Requests:</p>
  <p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
  <div class="snippet-code">
  <pre class="snippet-code-js lang-js prettyprint-override"><code>import requests
  from bs4 import BeautifulSoup
  import re
  import base64
  import json
  
  username= &lt;username&gt;
  password= &lt;password&gt;
  base_url= "https://accounts.wsj.com"
  
  session = requests.Session()
  r = session.get("{}/login".format(base_url))
  soup = BeautifulSoup(r.text, "html.parser")
  
  credentials_search = re.search("Base64\.decode\('(.*)'", r.text, re.IGNORECASE)
  base64_decoded = base64.b64decode(credentials_search.group(1))
  credentials = json.loads(base64_decoded)
  connection = &lt;connection_name&gt;
  
  r = session.post(
      'https://sso.accounts.dowjones.com/usernamepassword/login',
      data = {
          "username": username,
          "password": password,
          "connection": connection,
          "client_id": credentials["clientID"],
          "state": credentials["internalOptions"]["state"],
          "nonce": credentials["internalOptions"]["nonce"],
          "scope": credentials["internalOptions"]["scope"],
          "tenant": "sso",
          "response_type": "code",
          "protocol": "oauth2",
          "redirect_uri": "https://accounts.wsj.com/auth/sso/login"
      })
  soup = BeautifulSoup(r.text, "html.parser")
  
  login_result = dict([ 
      (t.get("name"), t.get("value")) 
      for t in soup.find_all('input') 
      if t.get("name") is not None
  ])
  
  r = session.post(
      'https://sso.accounts.dowjones.com/login/callback',
      data = login_result,
      headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Safari/537.36"},
      )
  
  # article get request
  r = session.get(
    "https://www.wsj.com/articles/singapore-prime-minister-lee-rejects-claims-he-misused-state-powers-in-family-feud-1499094761",
    headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Safari/537.36"}
    )
  
  print(r.text)</code></pre>
  </div>
  </div>
  </p>
  <p>am able to login through the request but still I am not getting full article to scrape. Can anyone help me with this? Thanks in advance :-)</p>