Title: Who took the python requests shib_idp_session cookies from the cookie jar?
Tags: python;cookies;python-requests;single-sign-on;saml
Question: I am trying to use python (3.9.1) and requests (2.25.1) to log in via Shibboleth and Duo two factor authentication (2FA). I have all of the appropriate credentials to do so and furthermore, routinely log in with a browser. Ultimately I am trying to automate some tasks that will happen after a successful login. Indeed, I was successfully able to log in via Shibboleth only and I posted an answer to an old question upon solving it. Now however, my institution is requiring Duo 2FA and I must rework this authentication part of my code. I almost have it working but am failing at nearly the last step. I have been using Chrome Developer Tools to track the flow and am using that as a guide for what requests to make and what information to pass in each request. In this post, I have replaced goodies I do not wish to share with &quot;XXXXXXXXXXX&quot;.
The flow of requests using a session object is as follows. Note that I show the request response redirection history in italics and the status code in bold at the end of each request description. No italicized code means no redirection. All my code is below this text description.
Request 1 - Request the ultimate final URL where I wish to grab data and begin my automation. 302, 302, 200
Request 2 - The session handles a redirection to an IDP SAML based url and is expecting my username and password. 302, 200
Request 3 - After successfully passing credentials, the session is passed off to duo. Duo checks my IDP authentication. 302, 200
Request 4 - After successful IDP authentication, duo prompts my device for 2FA and I do receive this prompt. 200
Request 5 - Ask the duo api for status. Duo answers with 'OK' and that a login request was 'pushed'. 200
Request 6 - Ask the duo api again for status. Duo answers with 'Success' and provides the txid needed for last step of duo 2FA. 200
Request 7 - Post the txid to the duo api. 200
Request 8 - This url was given as the response to Request 2 and passed through duo as the parent. A successful request here will give me the RelayState and the SAMLResponse. I am successful at obtaining these two variables and yes, the SAMLResponse sometimes approaches 50,000 characters. 200
Request 9 - Post the RelayState and SAMLResponse to Shibboleth. 500
Request 10 - Finally, I would expect to arrive at my ultimate final destination URL but so far, I can't get past request 9.
The salient parts of the response text from request 9 are:
&quot;opensaml75b5:f2c1:f5d7:b954:8dc3:cee9:baf:a8c0talProfileException&quot;
&quot;Status: urn:oasis:names:tc:SAML:2.0:status:Requester&quot;
&quot;Sub-Status: urn:oasis:names:tc:SAML:2.0:status:AuthnFailed&quot;
The requests session cookie jar seems to successfully grab all cookies that I see when digging through Chrome DevTools for all requests above except for request 8. In Chrome DevTools, this Response Header includes five instances of &quot;Set-Cookie&quot; and I only see one of them in the cookie jar. The one cookie I do observe is the &quot;BIGipServer~prt_shib~pl_idpXXXXXXX&quot; which was placed in the cookie jar after request #1. The missing cookies that I do not observe in the response to request 8 but I do observe in Chrome DevTools are &quot;shib_idp_session_ss&quot;, &quot;XXXXX-rememberme-XXXXXXXXX&quot;, &quot;XXXX-optin-XXXXXXXXXX&quot;, and &quot;shib_idp_session&quot;. I focus on these cookies because: (1) they are a difference between Chrome DevTools and my requests session and (2) in searching for the response text to the failed request 9 I see various posts around the web regarding cookie issues. Because I am restarting my session from scratch each time, I should not have any stale cookies or other active logins. Regarding the active logins, I can be logged in with firefox, chrome, and safari all at once with the same credentials.
Am I correct in focusing on these shib_session cookies in an effort to solve my problem? Should I be looking higher up in the request flow (e.g. prior to duo) for reasons I am not passed these cookies? If I am successfully receiving the RelayState and SAMLResponse, why am I also not receiving the shib_session cookies?
Lastly, I know http requests can be sensitive to headers. I am naively using the status-code as my indicator of getting the headers correct. Perhaps I am missing a header specification early on that will then prompt the IDP to give me the shib cookies?
Here is my code:
import re
import requests
## start requests session
s = requests.Session()
## Request 1
url1 = '[  ultimate target URL  ]' # This is the ultimate target URL
r1 = s.get(url1)
## Request 2
url2 = r1.url # redirection url from r1 is next up and used as url2
cred = {'j_username': 'XXXXXXXXXX', 'j_password': 'XXXXXXXXXX', '_eventId_proceed' : 'Sign in'}
r2 = s.post(url2, data = cred)
## Request 3
ss3 = re.search('data-host=&quot;',r2.text)
ss4 = re.search('&quot;\n                data-sig-request=&quot;',r2.text)
data_sig_request = r2.text[ss4.span(0)[1]:ss5.span(0)[0]]
ss6 = re.search('data-post-action=&quot;',r2.text)
ss7 = re.search('&quot;\n                frameborder=',r2.text)
data_post_action = r2.text[ss6.span(0)[1]:ss7.span(0)[0]]
prnt = '[  idp url  ]' + data_post_action
prnt = prnt.replace('/','%2F').replace(':','%3A').replace(';','%3B').replace('=','%3D').replace('?','%3F')
url3 = 'https://api-XXXXXXXXXX.duosecurity.com/frame/web/v1/auth?tx=' + data_sig_request + '&amp;parent=' + prnt + '&amp;v=2.6'
headers3 = {'Referer': '[  idp url  ]'}
s.headers.update(headers3)
r3 = s.post(url3)
## Request 4 - duo prompt
url4 = 'https://api-XXXXXXXXXX.duosecurity.com/frame/prompt'
ss8 = re.search('value=&quot;',r3.text)
ss9 = re.search('&quot;&gt;\n<input type=&quot;hidden&quot; name=&quot;url&quot; value=&quot;',r3.text)
sid = r3.text[ss8.span(0)[1]:ss9.span(0)[0]]
sid = sid.replace('&amp;#x3d;','=').replace('&amp;#x7c;','|')
ss10 = re.search('&quot;ukey&quot; value=&quot;',r3.text)
ss11 = re.search('&quot;&gt;\n\n<input type=&quot;hidden&quot; name=&quot;out_of_date&quot;',r3.text)
formdata4 = {'sid':sid, 'device':'phone1', 'factor':'Duo Push'}
r4 = s.post(url4,formdata4)
## Request 5 - Duo Status
url5 = 'https://api-XXXXXXXXXX.duosecurity.com/frame/status'
sid5 = sid.replace('=','%3D').replace('|','%7C')
ss12 = re.search('&quot;txid&quot;: &quot;',r4.text)
ss13 = re.search('&quot;}}',r4.text)
txid = r4.text[ss12.span(0)[1]:ss13.span(0)[0]]
formdata5 = {'sid':sid, 'txid':txid}
r5 = s.post(url5,formdata5)
## Request 6 - second status call to duo
url6 = url5
formdata6 = formdata5
r6 = s.post(url6,formdata6)
## Request 7 - Duo - evaluate push
url7 = 'https://api-XXXXXXXXXX.duosecurity.com/frame/status/' + txid
formdata7 = {'sid':sid}
r7 = s.post(url7,formdata7)
## Request 8
url8 = r2.url # r2.url is the redirection url from url2
ss18 = re.search('&quot;\n                data-post-action',r2.text)
sig_response = r2.text[ss4.span(0)[1]:ss18.span(0)[0]]
sig_response = sig_response.replace('TX|','AUTH|')
formdata8 = {'_eventId':'proceed', 'sig_response':sig_response}
r8 = s.post(url8,formdata8)
## Request 9 - should redirect to ultimate url (url1)
url9 = 'https://XXXXXXXXXXXX/Shibboleth.sso/SAML2/POST'
ss19 = re.search('name=&quot;RelayState&quot; value=&quot;',r8.text)
ss20 = re.search('&quot;/&gt;                \n                                \n                <input type=&quot;hidden&quot; name=&quot;SAMLResponse&quot; value=&quot;',r8.text)
RelayState = r8.text[ss19.span(0)[1]:ss20.span(0)[0]]
RelayState = RelayState.replace('&amp;#x3a;',':')
ss21 = re.search('&quot;/&gt;                \n            </div&gt;\n            <noscript&gt;\n                <div&gt;\n                    <input type=&quot;submit&quot; value=&quot;Continue&quot;/&gt;',r8.text)
SAMLResponse = r8.text[ss20.span(0)[1]:ss21.span(0)[0]]
formdata9 = {'RelayState':RelayState, 'SAMLResponse':SAMLResponse}
r9 = s.post(url9,formdata9,allow_redirects=True)
## Request 10 - this would be the ultimate final destination URL and is url1 from above.

Here is the accepted answer: Finally I solved this. The problem was in Request 8. Before I knew the problem was in Request 8, I decoded my SAMLResponse using https://www.samltool.com/decode.php only to find that I had a failed authentication. The problem with Request 8 was the sig_response. It only included half of the required information. A proper sig_response should have first, the DUO response cookie and second, the last half of the data-sig-request from Request 2.
With a properly assembled sig_response, the idp returns the missing shib_idp_session cookies, but also, an acceptable SAMLResponse key. Note also, that the python requests default headers were used in this successful script.
Here is my code, slightly refactored from the question and with the error fixed:
import re
import requests
import time
## start HTTP request session
s = requests.Session()
## Request 1
url1 = '[  ultimate target URL  ]'
r1 = s.get(url1)
## Request 2
url2 = r1.url
cred = {'j_username': 'XXXXXXXXXX', 'j_password': 'XXXXXXXXXX', '_eventId_proceed' : 'Sign in'}
r2 = s.post(url2, data = cred)
## Request 3
ss4 = re.search('&quot;\n                data-sig-request=&quot;',r2.text)
ss5 = re.search(':APP',r2.text)
data_sig_request = r2.text[ss4.span(0)[1]:ss5.span(0)[0]]
ss6 = re.search('data-post-action=&quot;',r2.text)
ss7 = re.search('&quot;\n                frameborder=',r2.text)
data_post_action = r2.text[ss6.span(0)[1]:ss7.span(0)[0]]
prnt = '[  idp url  ]' + data_post_action
prnt = prnt.replace('/','%2F').replace(':','%3A').replace(';','%3B').replace('=','%3D').replace('?','%3F')
url3 = 'https://api-XXXXXXXXXX.duosecurity.com/frame/web/v1/auth?tx=' + data_sig_request + '&amp;parent=' + prnt + '&amp;v=2.6'
r3 = s.get(url3)
## Request 4
url4 = url3
formdata4 = {'tx': data_sig_request, 'parent': url2.replace('=e1s1','=e1s2')}
r4 = s.post(url4, data=formdata4)
## Request 5
url5 = 'https://api-XXXXXXXXXX.duosecurity.com/frame/prompt'
ss8 = re.search('value=&quot;',r4.text)
ss9 = re.search('&quot;&gt;\n<input type=&quot;hidden&quot; name=&quot;url&quot; value=&quot;',r4.text)
sid = r4.text[ss8.span(0)[1]:ss9.span(0)[0]]
sid = sid.replace('&amp;#x3d;','=').replace('&amp;#x7c;','|')
ss10 = re.search('&quot;ukey&quot; value=&quot;',r4.text)
ss11 = re.search('&quot;&gt;\n\n<input type=&quot;hidden&quot; name=&quot;out_of_date&quot;',r4.text)
formdata5 = {'sid':sid, 'device':'phone1', 'factor':'Duo Push', 'days_to_block': 'None'}
r5 = s.post(url5, data = formdata5)
## Request 6
url6 = 'https://api-XXXXXXXXXX.duosecurity.com/frame/status'
ss12 = re.search('&quot;txid&quot;: &quot;',r5.text)
ss13 = re.search('&quot;}}',r5.text)
txid = r5.text[ss12.span(0)[1]:ss13.span(0)[0]]
formdata6 = {'sid':sid, 'txid':txid}
r6 = s.post(url6, data = formdata6)
## Request 7
url7 = url6
formdata7 = formdata6
r7 = s.post(url7, data = formdata7)
## Request 8
url8 = 'https://api-XXXXXXXXXX.duosecurity.com/frame/status' + txid
formdata8 = {'sid':sid}
r8 = s.post(url8, data = formdata8)
## Request 9
url9 = r2.url
ss14 = re.search('APP',r2.text)
ss15 = re.search('&quot;\n                data-post-action=&quot;',r2.text)
r8td = ast.literal_eval(r8.text)
data_sig_request_APP = r2.text[ss14.span(0)[1]+1:ss15.span(0)[0]]
sig_response = r8td['response']['cookie'] + ':APP|' + data_sig_request_APP
formdata9 = {'_eventId':'proceed',
'sig_response':sig_response}
r9 = s.post(url9, data = formdata9)
## Request 10
url10 = '[  ultimate target URL  ]/Shibboleth.sso/SAML2/POST'
ss19 = re.search('name=&quot;RelayState&quot; value=&quot;',r9.text)
ss20 = re.search('&quot;/&gt;                \n                                \n                <input type=&quot;hidden&quot; name=&quot;SAMLResponse&quot; value=&quot;',r9.text)
RelayState = r9.text[ss19.span(0)[1]:ss20.span(0)[0]]
RelayState = RelayState.replace('&amp;#x3a;',':')
ss21 = re.search('&quot;/&gt;                \n            </div&gt;\n            <noscript&gt;\n                <div&gt;\n                    <input type=&quot;submit&quot; value=&quot;Continue&quot;/&gt;',r9.text)
SAMLResponse = r9.text[ss20.span(0)[1]:ss21.span(0)[0]]
formdata10 = {'RelayState':RelayState, 'SAMLResponse':SAMLResponse}
r10 = s.post(url10, data = formdata10)```


{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 1, 'CommentCount': 0, 'ContentLicense': 'CC BY-SA 4.0', 'CreationDate': '2021:00:08/06/21', 'FavoriteCount': None, 'PostTypeId': 1, 'Tags': '<python><cookies><python-requests><single-sign-on><saml>', 'Title': 'Who took the python requests shib_idp_session cookies from the cookie jar?', 'ViewCount': 292, 'id': 68688351, 'user_AboutMe': '<p>I repair, clean, code, teach, write, tinker, question, bike, and try.</p>\n', 'user_AccountId': 20804475.0, 'user_DisplayName': 'ajschauer', 'user_Reputation': 106.0, 'user_Views': 13.0, 'user_id': 15279693.0}}