S3_USE_HTTPS 0 is being ignored - TF insists on HTTPS
Hi @edi-bice, 

Setting `os.environ['S3_USE_HTTPS'] = '0'` in TensorFlow can cause an error because TensorFlow by default uses HTTPS to communicate with S3. If you set S3_USE_HTTPS to '0', you are telling TensorFlow to use plain HTTP to communicate with S3, which can cause a security issue because the traffic is not encrypted and can be intercepted and modified by attackers.

Additionally, some S3 endpoints may not even allow non-HTTPS connections, so setting S3_USE_HTTPS to '0' could result in connection failures or other errors. Therefore, it is generally not recommended to set S3_USE_HTTPS to '0' in TensorFlow. Thank you! @synandi agree completely but should have made it clearer that this is TF running in TFX/KFP pipelines on Kubeflow accessing its cluster-local Minio storage via S3 protocol etc. Minio is bundled with Kubeflow and at least the gcp-blueprint distribution seems to have it configured without SSL. On a different Kubeflow 1.6 deployment on GCP, it seems to be making request against http endpoint though regardless of the value in os.environ['S3_USE_HTTPS'] (whether 0 or 1) and regardless of os.environ['S3_ENDPOINT'] being '10.12.2.51:9000' or 'http://10.12.2.51:9000' as is reported by Minio pod logs.

2023-03-15 12:26:40.083090: I tensorflow/c/logging.cc:34] Making request to http://10.12.2.51:9000/edi_bice/models/clientXYZ/model_arch/poisson_model/model_layers.json

`2023-03-15 12:26:30.063771: I tensorflow/c/logging.cc:34] Checking HOME for the home directory.
2023-03-15 12:26:30.063836: I tensorflow/c/logging.cc:34] Environment value for variable HOME is /home/jovyan
2023-03-15 12:26:30.063851: I tensorflow/c/logging.cc:34] Home directory is missing the final / appending one to normalize
2023-03-15 12:26:30.063857: I tensorflow/c/logging.cc:34] Final Home Directory is /home/jovyan/
2023-03-15 12:26:30.063867: I tensorflow/c/logging.cc:34] Initializing config loader against fileName /home/jovyan/.aws/credentials and using profilePrefix = 0
2023-03-15 12:26:30.063874: I tensorflow/c/logging.cc:34] Checking HOME for the home directory.
2023-03-15 12:26:30.063879: I tensorflow/c/logging.cc:34] Environment value for variable HOME is /home/jovyan
2023-03-15 12:26:30.063885: I tensorflow/c/logging.cc:34] Home directory is missing the final / appending one to normalize
2023-03-15 12:26:30.063890: I tensorflow/c/logging.cc:34] Final Home Directory is /home/jovyan/
2023-03-15 12:26:30.063896: I tensorflow/c/logging.cc:34] Initializing config loader against fileName /home/jovyan/.aws/config and using profilePrefix = 1
2023-03-15 12:26:30.063902: I tensorflow/c/logging.cc:34] Checking HOME for the home directory.
2023-03-15 12:26:30.063907: I tensorflow/c/logging.cc:34] Environment value for variable HOME is /home/jovyan
2023-03-15 12:26:30.063913: I tensorflow/c/logging.cc:34] Home directory is missing the final / appending one to normalize
2023-03-15 12:26:30.063918: I tensorflow/c/logging.cc:34] Final Home Directory is /home/jovyan/
2023-03-15 12:26:30.063948: I tensorflow/c/logging.cc:34] Unable to open config file /home/jovyan/.aws/credentials for reading.
2023-03-15 12:26:30.063955: I tensorflow/c/logging.cc:34] Failed to reload configuration.
2023-03-15 12:26:30.063961: I tensorflow/c/logging.cc:34] Checking HOME for the home directory.
2023-03-15 12:26:30.063970: I tensorflow/c/logging.cc:34] Environment value for variable HOME is /home/jovyan
2023-03-15 12:26:30.063979: I tensorflow/c/logging.cc:34] Home directory is missing the final / appending one to normalize
2023-03-15 12:26:30.063988: I tensorflow/c/logging.cc:34] Final Home Directory is /home/jovyan/
2023-03-15 12:26:30.064001: I tensorflow/c/logging.cc:34] Unable to open config file /home/jovyan/.aws/config for reading.
2023-03-15 12:26:30.064010: I tensorflow/c/logging.cc:34] Failed to reload configuration.
2023-03-15 12:26:30.064042: I tensorflow/c/logging.cc:34] Initializing Curl library with version: 7.85.0, ssl version: BoringSSL
2023-03-15 12:26:30.064059: I tensorflow/c/logging.cc:34] ClientConfiguration will use SDK Auto Resolved profile: [default] if not specified by users.
2023-03-15 12:26:30.064080: W tensorflow/c/logging.cc:37] Retry Strategy will use the default max attempts.
2023-03-15 12:26:30.064096: I tensorflow/c/logging.cc:34] Creating AWSHttpResourceClient with max connections 2 and scheme http
2023-03-15 12:26:30.064108: I tensorflow/c/logging.cc:34] Initializing CurlHandleContainer with size 2
2023-03-15 12:26:30.064128: I tensorflow/c/logging.cc:34] ClientConfiguration will use SDK Auto Resolved profile: [default] if not specified by users.
2023-03-15 12:26:30.064139: W tensorflow/c/logging.cc:37] Retry Strategy will use the default max attempts.
2023-03-15 12:26:30.064164: I tensorflow/c/logging.cc:34] Checking HOME for the home directory.
2023-03-15 12:26:30.064176: I tensorflow/c/logging.cc:34] Environment value for variable HOME is /home/jovyan
2023-03-15 12:26:30.064187: I tensorflow/c/logging.cc:34] Home directory is missing the final / appending one to normalize
2023-03-15 12:26:30.064196: I tensorflow/c/logging.cc:34] Final Home Directory is /home/jovyan/
2023-03-15 12:26:30.064207: I tensorflow/c/logging.cc:34] Initializing config loader against fileName /home/jovyan/.aws/credentials and using profilePrefix = 0
2023-03-15 12:26:30.064218: I tensorflow/c/logging.cc:34] Checking HOME for the home directory.
2023-03-15 12:26:30.064227: I tensorflow/c/logging.cc:34] Environment value for variable HOME is /home/jovyan
2023-03-15 12:26:30.064237: I tensorflow/c/logging.cc:34] Home directory is missing the final / appending one to normalize
2023-03-15 12:26:30.064250: I tensorflow/c/logging.cc:34] Final Home Directory is /home/jovyan/
2023-03-15 12:26:30.064261: I tensorflow/c/logging.cc:34] Checking HOME for the home directory.
2023-03-15 12:26:30.064271: I tensorflow/c/logging.cc:34] Environment value for variable HOME is /home/jovyan
2023-03-15 12:26:30.064280: I tensorflow/c/logging.cc:34] Home directory is missing the final / appending one to normalize
2023-03-15 12:26:30.064288: I tensorflow/c/logging.cc:34] Final Home Directory is /home/jovyan/
2023-03-15 12:26:30.064301: I tensorflow/c/logging.cc:34] Setting provider to read credentials from /home/jovyan/.aws/credentials for credentials file and /home/jovyan/.aws/config for the config file , for use with profile default
2023-03-15 12:26:30.064314: I tensorflow/c/logging.cc:34] Setting process credentials provider to read config from default
2023-03-15 12:26:30.064332: W tensorflow/c/logging.cc:37] Token file must be specified to use STS AssumeRole web identity creds provider.
2023-03-15 12:26:30.064344: I tensorflow/c/logging.cc:34] Setting sso credentials provider to read config from default
2023-03-15 12:26:30.064357: I tensorflow/c/logging.cc:34] The environment variable value AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is 
2023-03-15 12:26:30.064366: I tensorflow/c/logging.cc:34] The environment variable value AWS_CONTAINER_CREDENTIALS_FULL_URI is 
2023-03-15 12:26:30.064375: I tensorflow/c/logging.cc:34] The environment variable value AWS_EC2_METADATA_DISABLED is 
2023-03-15 12:26:30.064386: I tensorflow/c/logging.cc:34] ClientConfiguration will use SDK Auto Resolved profile: [default] if not specified by users.
2023-03-15 12:26:30.064397: W tensorflow/c/logging.cc:37] Retry Strategy will use the default max attempts.
2023-03-15 12:26:30.064409: I tensorflow/c/logging.cc:34] Creating AWSHttpResourceClient with max connections 2 and scheme http
2023-03-15 12:26:30.064419: I tensorflow/c/logging.cc:34] Initializing CurlHandleContainer with size 2
2023-03-15 12:26:30.064430: I tensorflow/c/logging.cc:34] Creating Instance with default EC2MetadataClient and refresh rate 300000
2023-03-15 12:26:30.064448: I tensorflow/c/logging.cc:34] Added EC2 metadata service credentials provider to the provider chain.
2023-03-15 12:26:30.064477: I tensorflow/c/logging.cc:34] Found credential in environment with access key id minio
2023-03-15 12:26:30.064491: I tensorflow/c/logging.cc:34] Found secret key
2023-03-15 12:26:30.064547: I tensorflow/c/logging.cc:34] Initializing CurlHandleContainer with size 25
2023-03-15 12:26:30.064729: I tensorflow/c/logging.cc:34] No content body, content-length headers
2023-03-15 12:26:30.064795: I tensorflow/c/logging.cc:34] Found credential in environment with access key id minio
2023-03-15 12:26:30.064811: I tensorflow/c/logging.cc:34] Found secret key
2023-03-15 12:26:30.064824: I tensorflow/c/logging.cc:34] Using cached empty string sha256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 because payload is empty.
2023-03-15 12:26:30.064870: I tensorflow/c/logging.cc:34] Canonical Header String: amz-sdk-invocation-id:CD996767-93DA-4B24-AC07-CF3DBD6AC7A3
amz-sdk-request:attempt=1
content-type:application/xml
host:10.12.2.51:9000
x-amz-api-version:2006-03-01
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20230315T122630Z

2023-03-15 12:26:30.064883: I tensorflow/c/logging.cc:34] Signed Headers value:amz-sdk-invocation-id;amz-sdk-request;content-type;host;x-amz-api-version;x-amz-content-sha256;x-amz-date
2023-03-15 12:26:30.064913: I tensorflow/c/logging.cc:34] Canonical Request String: HEAD
/edi_bice/models/clientXYZ/model_arch/poisson_model/model_layers.json

amz-sdk-invocation-id:CD996767-93DA-4B24-AC07-CF3DBD6AC7A3
amz-sdk-request:attempt=1
content-type:application/xml
host:10.12.2.51:9000
x-amz-api-version:2006-03-01
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20230315T122630Z

amz-sdk-invocation-id;amz-sdk-request;content-type;host;x-amz-api-version;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2023-03-15 12:26:30.064938: I tensorflow/c/logging.cc:34] Final String to sign: AWS4-HMAC-SHA256
20230315T122630Z
20230315/us-east-1/s3/aws4_request
e3cae3b1a464d54b492c8a1e21c4a99785ea29c76e307538f7af5db540f48360
2023-03-15 12:26:30.064952: I tensorflow/c/logging.cc:34] Final computed signing hash: 89accf1cc9a4a9570c50324fa82195ca53aaae9764b2c70bdbafec1fe976609f
2023-03-15 12:26:30.064965: I tensorflow/c/logging.cc:34] Signing request with: AWS4-HMAC-SHA256 Credential=minio/20230315/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-type;host;x-amz-api-version;x-amz-content-sha256;x-amz-date, Signature=89accf1cc9a4a9570c50324fa82195ca53aaae9764b2c70bdbafec1fe976609f
2023-03-15 12:26:30.064977: I tensorflow/c/logging.cc:34] Request Successfully signed
2023-03-15 12:26:30.064997: I tensorflow/c/logging.cc:34] Making request to http://10.12.2.51:9000/edi_bice/models/clientXYZ/model_arch/poisson_model/model_layers.json
2023-03-15 12:26:30.065011: I tensorflow/c/logging.cc:34] Including headers:
2023-03-15 12:26:30.065021: I tensorflow/c/logging.cc:34] amz-sdk-invocation-id: CD996767-93DA-4B24-AC07-CF3DBD6AC7A3
2023-03-15 12:26:30.065032: I tensorflow/c/logging.cc:34] amz-sdk-request: attempt=1
2023-03-15 12:26:30.065042: I tensorflow/c/logging.cc:34] authorization: AWS4-HMAC-SHA256 Credential=minio/20230315/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-type;host;x-amz-api-version;x-amz-content-sha256;x-amz-date, Signature=89accf1cc9a4a9570c50324fa82195ca53aaae9764b2c70bdbafec1fe976609f
2023-03-15 12:26:30.065052: I tensorflow/c/logging.cc:34] content-type: application/xml
2023-03-15 12:26:30.065063: I tensorflow/c/logging.cc:34] host: 10.12.2.51:9000
2023-03-15 12:26:30.065073: I tensorflow/c/logging.cc:34] user-agent: aws-sdk-cpp/1.8.186 Linux/5.15.0-1024-gke x86_64 GCC/7.3.1
2023-03-15 12:26:30.065082: I tensorflow/c/logging.cc:34] x-amz-api-version: 2006-03-01
2023-03-15 12:26:30.065091: I tensorflow/c/logging.cc:34] x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2023-03-15 12:26:30.065100: I tensorflow/c/logging.cc:34] x-amz-date: 20230315T122630Z
2023-03-15 12:26:30.065111: I tensorflow/c/logging.cc:34] Attempting to acquire curl connection.
2023-03-15 12:26:30.065119: I tensorflow/c/logging.cc:34] No current connections available in pool. Attempting to create new connections.
2023-03-15 12:26:30.065128: I tensorflow/c/logging.cc:34] attempting to grow pool size by 2
2023-03-15 12:26:30.065153: I tensorflow/c/logging.cc:34] Pool grown by 2
2023-03-15 12:26:30.065166: I tensorflow/c/logging.cc:34] Connection has been released. Continuing.
2023-03-15 12:26:30.065177: I tensorflow/c/logging.cc:34] Returning connection handle 0x557f0495ca40
2023-03-15 12:26:30.065186: I tensorflow/c/logging.cc:34] Obtained connection handle 0x557f0495ca40
2023-03-15 12:26:40.081483: I tensorflow/c/logging.cc:34] HTTP/1.1 503 Service Unavailable

2023-03-15 12:26:40.081573: I tensorflow/c/logging.cc:34] content-length: 91

2023-03-15 12:26:40.081592: I tensorflow/c/logging.cc:34] content-type: text/plain

2023-03-15 12:26:40.081603: I tensorflow/c/logging.cc:34] date: Wed, 15 Mar 2023 12:26:39 GMT

2023-03-15 12:26:40.081612: I tensorflow/c/logging.cc:34] server: envoy

2023-03-15 12:26:40.081620: I tensorflow/c/logging.cc:34] 

2023-03-15 12:26:40.081647: I tensorflow/c/logging.cc:34] Returned http response code 503
2023-03-15 12:26:40.081655: I tensorflow/c/logging.cc:34] Returned content type text/plain
2023-03-15 12:26:40.081661: I tensorflow/c/logging.cc:34] Releasing curl handle 0x557f0495ca40
2023-03-15 12:26:40.081680: I tensorflow/c/logging.cc:34] Releasing curl handle 0x557f0495ca40
2023-03-15 12:26:40.081689: I tensorflow/c/logging.cc:34] Notified waiting threads.
2023-03-15 12:26:40.081713: I tensorflow/c/logging.cc:34] Request returned error. Attempting to generate appropriate error codes from response
2023-03-15 12:26:40.081737: E tensorflow/c/logging.cc:40] HTTP response code: 503
Resolved remote host IP address: 
Request ID: 
Exception name: 
Error message: No response body.
4 response headers:
content-length : 91
content-type : text/plain
date : Wed, 15 Mar 2023 12:26:39 GMT
server : envoy
2023-03-15 12:26:40.081762: W tensorflow/c/logging.cc:37] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2023-03-15 12:26:40.081771: I tensorflow/c/logging.cc:34] Server time is Wed, 15 Mar 2023 12:26:39 GMT, while client time is Wed, 15 Mar 2023 12:26:40 GMT
2023-03-15 12:26:40.081778: W tensorflow/c/logging.cc:37] Request failed, now waiting 0 ms before attempting again.` Same code (and library versions) seems to work on Kubeflow 1.5 deployed on GCP via gcp-blueprints

`2023-03-14 21:59:13.723573: I tensorflow/c/logging.cc:34] No content body, content-length headers
2023-03-14 21:59:13.723658: I tensorflow/c/logging.cc:34] Found credential in environment with access key id minio
2023-03-14 21:59:13.723680: I tensorflow/c/logging.cc:34] Found secret key
2023-03-14 21:59:13.723696: I tensorflow/c/logging.cc:34] Using cached empty string sha256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 because payload is empty.
2023-03-14 21:59:13.723766: I tensorflow/c/logging.cc:34] Canonical Header String: amz-sdk-invocation-id:5D5AC5F0-5091-4804-A1BE-EE45C33DAC84
amz-sdk-request:attempt=1
content-type:application/xml
host:10.12.2.51:9000
x-amz-api-version:2006-03-01
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20230314T215913Z

2023-03-14 21:59:13.723792: I tensorflow/c/logging.cc:34] Signed Headers value:amz-sdk-invocation-id;amz-sdk-request;content-type;host;x-amz-api-version;x-amz-content-sha256;x-amz-date
2023-03-14 21:59:13.723830: I tensorflow/c/logging.cc:34] Canonical Request String: HEAD
/edi_bice/models/clientXYZ/model_arch/poisson_model/model_layers.json

amz-sdk-invocation-id:5D5AC5F0-5091-4804-A1BE-EE45C33DAC84
amz-sdk-request:attempt=1
content-type:application/xml
host:10.12.2.51:9000
x-amz-api-version:2006-03-01
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20230314T215913Z

amz-sdk-invocation-id;amz-sdk-request;content-type;host;x-amz-api-version;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2023-03-14 21:59:13.723874: I tensorflow/c/logging.cc:34] Final String to sign: AWS4-HMAC-SHA256
20230314T215913Z
20230314/us-east-1/s3/aws4_request
1238bfb28a22deb82193ab1d9129bc2f72feaf1e511c934ff9912ff3f5ff759a
2023-03-14 21:59:13.723911: I tensorflow/c/logging.cc:34] Final computed signing hash: 9aff3e1cfabe7f6bdabb78953a14b1d3da73a4ea72035a8da890706f7f115c4d
2023-03-14 21:59:13.723931: I tensorflow/c/logging.cc:34] Signing request with: AWS4-HMAC-SHA256 Credential=minio/20230314/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-type;host;x-amz-api-version;x-amz-content-sha256;x-amz-date, Signature=9aff3e1cfabe7f6bdabb78953a14b1d3da73a4ea72035a8da890706f7f115c4d
2023-03-14 21:59:13.723956: I tensorflow/c/logging.cc:34] Request Successfully signed
2023-03-14 21:59:13.723986: I tensorflow/c/logging.cc:34] Making request to http://10.12.2.51:9000/edi_bice/models/clientXYZ/model_arch/poisson_model/model_layers.json
2023-03-14 21:59:13.724013: I tensorflow/c/logging.cc:34] Including headers:
2023-03-14 21:59:13.724031: I tensorflow/c/logging.cc:34] amz-sdk-invocation-id: 5D5AC5F0-5091-4804-A1BE-EE45C33DAC84
2023-03-14 21:59:13.724054: I tensorflow/c/logging.cc:34] amz-sdk-request: attempt=1
2023-03-14 21:59:13.724069: I tensorflow/c/logging.cc:34] authorization: AWS4-HMAC-SHA256 Credential=minio/20230314/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-type;host;x-amz-api-version;x-amz-content-sha256;x-amz-date, Signature=9aff3e1cfabe7f6bdabb78953a14b1d3da73a4ea72035a8da890706f7f115c4d
2023-03-14 21:59:13.724085: I tensorflow/c/logging.cc:34] content-type: application/xml
2023-03-14 21:59:13.724108: I tensorflow/c/logging.cc:34] host: 10.12.2.51:9000
2023-03-14 21:59:13.724124: I tensorflow/c/logging.cc:34] user-agent: aws-sdk-cpp/1.8.186 Linux/5.4.217+ x86_64 GCC/7.3.1
2023-03-14 21:59:13.724137: I tensorflow/c/logging.cc:34] x-amz-api-version: 2006-03-01
2023-03-14 21:59:13.724161: I tensorflow/c/logging.cc:34] x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2023-03-14 21:59:13.724177: I tensorflow/c/logging.cc:34] x-amz-date: 20230314T215913Z
2023-03-14 21:59:13.724200: I tensorflow/c/logging.cc:34] Attempting to acquire curl connection.
2023-03-14 21:59:13.724216: I tensorflow/c/logging.cc:34] Connection has been released. Continuing.
2023-03-14 21:59:13.724241: I tensorflow/c/logging.cc:34] Returning connection handle 0x55d7fe080310
2023-03-14 21:59:13.724542: I tensorflow/c/logging.cc:34] Obtained connection handle 0x55d7fe080310
2023-03-14 21:59:13.798382: I tensorflow/c/logging.cc:34] HTTP/1.1 200 OK

2023-03-14 21:59:13.798456: I tensorflow/c/logging.cc:34] accept-ranges: bytes

2023-03-14 21:59:13.798481: I tensorflow/c/logging.cc:34] content-length: 717

2023-03-14 21:59:13.798505: I tensorflow/c/logging.cc:34] content-security-policy: block-all-mixed-content

2023-03-14 21:59:13.798525: I tensorflow/c/logging.cc:34] content-type: application/json

2023-03-14 21:59:13.798546: I tensorflow/c/logging.cc:34] etag: "a88e35987284c3a006ee88c43f902a7a"

2023-03-14 21:59:13.798635: I tensorflow/c/logging.cc:34] last-modified: Thu, 04 Feb 2021 19:21:01 GMT

2023-03-14 21:59:13.798665: I tensorflow/c/logging.cc:34] server: istio-envoy

2023-03-14 21:59:13.798684: I tensorflow/c/logging.cc:34] vary: Origin

2023-03-14 21:59:13.798703: I tensorflow/c/logging.cc:34] x-amz-request-id: 174C680A1BEFE503

2023-03-14 21:59:13.798722: I tensorflow/c/logging.cc:34] x-xss-protection: 1; mode=block

2023-03-14 21:59:13.798743: I tensorflow/c/logging.cc:34] date: Tue, 14 Mar 2023 21:59:13 GMT

2023-03-14 21:59:13.798764: I tensorflow/c/logging.cc:34] x-envoy-upstream-service-time: 68

2023-03-14 21:59:13.798789: I tensorflow/c/logging.cc:34] x-envoy-decorator-operation: minio-service.kubeflow.svc.cluster.local:9000/*

2023-03-14 21:59:13.798849: I tensorflow/c/logging.cc:34] 

2023-03-14 21:59:13.798892: I tensorflow/c/logging.cc:34] Returned http response code 200
2023-03-14 21:59:13.798919: I tensorflow/c/logging.cc:34] Returned content type application/json
2023-03-14 21:59:13.798939: I tensorflow/c/logging.cc:34] Releasing curl handle 0x55d7fe080310
2023-03-14 21:59:13.798964: I tensorflow/c/logging.cc:34] Releasing curl handle 0x55d7fe080310
2023-03-14 21:59:13.799030: I tensorflow/c/logging.cc:34] Notified waiting threads.
2023-03-14 21:59:13.799069: I tensorflow/c/logging.cc:34] Request returned successful response.
2023-03-14 21:59:13.799099: I tensorflow/c/logging.cc:34] Request successful returning.
2023-03-14 21:59:13.799232: I tensorflow/c/logging.cc:34] No content body, content-length headers
2023-03-14 21:59:13.799291: I tensorflow/c/logging.cc:34] Found credential in environment with access key id minio
2023-03-14 21:59:13.799315: I tensorflow/c/logging.cc:34] Found secret key
2023-03-14 21:59:13.799331: I tensorflow/c/logging.cc:34] Using cached empty string sha256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 because payload is empty.
2023-03-14 21:59:13.799428: I tensorflow/c/logging.cc:34] Canonical Header String: amz-sdk-invocation-id:6E5C4870-932F-495A-8CA3-BA80DEC0DB7A
amz-sdk-request:attempt=1
content-type:application/xml
host:10.12.2.51:9000
x-amz-api-version:2006-03-01
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20230314T215913Z

2023-03-14 21:59:13.799458: I tensorflow/c/logging.cc:34] Signed Headers value:amz-sdk-invocation-id;amz-sdk-request;content-type;host;x-amz-api-version;x-amz-content-sha256;x-amz-date
2023-03-14 21:59:13.799510: I tensorflow/c/logging.cc:34] Canonical Request String: GET
/edi_bice
list-type=2&max-keys=1&prefix=models0.000000clientXYZ0.000000model_arch0.000000poisson_model0.000000model_layers.json0.000000
amz-sdk-invocation-id:6E5C4870-932F-495A-8CA3-BA80DEC0DB7A
amz-sdk-request:attempt=1
content-type:application/xml
host:10.12.2.51:9000
x-amz-api-version:2006-03-01
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20230314T215913Z

amz-sdk-invocation-id;amz-sdk-request;content-type;host;x-amz-api-version;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2023-03-14 21:59:13.799561: I tensorflow/c/logging.cc:34] Final String to sign: AWS4-HMAC-SHA256
20230314T215913Z
20230314/us-east-1/s3/aws4_request
47c1d82682b93d2b38c74ff49c8d39ea99c2a055e73ae8713f12cfe941631e4b
2023-03-14 21:59:13.799581: I tensorflow/c/logging.cc:34] Final computed signing hash: 0e2b24c44f0bb058d251d2bddaab0f002dcaf531a9ca48bbafe2ea1d6a45ec1b
2023-03-14 21:59:13.799607: I tensorflow/c/logging.cc:34] Signing request with: AWS4-HMAC-SHA256 Credential=minio/20230314/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-type;host;x-amz-api-version;x-amz-content-sha256;x-amz-date, Signature=0e2b24c44f0bb058d251d2bddaab0f002dcaf531a9ca48bbafe2ea1d6a45ec1b
2023-03-14 21:59:13.799634: I tensorflow/c/logging.cc:34] Request Successfully signed
2023-03-14 21:59:13.799666: I tensorflow/c/logging.cc:34] Making request to http://10.12.2.51:9000/edi_bice?list-type=2&max-keys=1&prefix=models0.000000clientXYZ0.000000model_arch0.000000poisson_model0.000000model_layers.json0.000000
2023-03-14 21:59:13.799743: I tensorflow/c/logging.cc:34] Including headers:
2023-03-14 21:59:13.799805: I tensorflow/c/logging.cc:34] amz-sdk-invocation-id: 6E5C4870-932F-495A-8CA3-BA80DEC0DB7A
2023-03-14 21:59:13.799846: I tensorflow/c/logging.cc:34] amz-sdk-request: attempt=1
2023-03-14 21:59:13.799896: I tensorflow/c/logging.cc:34] authorization: AWS4-HMAC-SHA256 Credential=minio/20230314/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-type;host;x-amz-api-version;x-amz-content-sha256;x-amz-date, Signature=0e2b24c44f0bb058d251d2bddaab0f002dcaf531a9ca48bbafe2ea1d6a45ec1b
2023-03-14 21:59:13.800022: I tensorflow/c/logging.cc:34] content-type: application/xml
2023-03-14 21:59:13.800108: I tensorflow/c/logging.cc:34] host: 10.12.2.51:9000
2023-03-14 21:59:13.800226: I tensorflow/c/logging.cc:34] user-agent: aws-sdk-cpp/1.8.186 Linux/5.4.217+ x86_64 GCC/7.3.1
2023-03-14 21:59:13.800283: I tensorflow/c/logging.cc:34] x-amz-api-version: 2006-03-01
2023-03-14 21:59:13.800301: I tensorflow/c/logging.cc:34] x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2023-03-14 21:59:13.800315: I tensorflow/c/logging.cc:34] x-amz-date: 20230314T215913Z
2023-03-14 21:59:13.800344: I tensorflow/c/logging.cc:34] Attempting to acquire curl connection.
2023-03-14 21:59:13.800640: I tensorflow/c/logging.cc:34] Connection has been released. Continuing.
2023-03-14 21:59:13.800677: I tensorflow/c/logging.cc:34] Returning connection handle 0x55d7fe080310
2023-03-14 21:59:13.800793: I tensorflow/c/logging.cc:34] Obtained connection handle 0x55d7fe080310
2023-03-14 21:59:13.878099: I tensorflow/c/logging.cc:34] HTTP/1.1 200 OK

2023-03-14 21:59:13.878238: I tensorflow/c/logging.cc:34] accept-ranges: bytes

2023-03-14 21:59:13.878309: I tensorflow/c/logging.cc:34] content-length: 319

2023-03-14 21:59:13.878331: I tensorflow/c/logging.cc:34] content-security-policy: block-all-mixed-content

2023-03-14 21:59:13.878356: I tensorflow/c/logging.cc:34] content-type: application/xml

2023-03-14 21:59:13.878371: I tensorflow/c/logging.cc:34] server: istio-envoy

2023-03-14 21:59:13.878396: I tensorflow/c/logging.cc:34] vary: Origin

2023-03-14 21:59:13.878411: I tensorflow/c/logging.cc:34] x-amz-request-id: 174C680A2038EBCD

2023-03-14 21:59:13.878427: I tensorflow/c/logging.cc:34] x-xss-protection: 1; mode=block

2023-03-14 21:59:13.878451: I tensorflow/c/logging.cc:34] date: Tue, 14 Mar 2023 21:59:13 GMT

2023-03-14 21:59:13.878470: I tensorflow/c/logging.cc:34] x-envoy-upstream-service-time: 75

2023-03-14 21:59:13.878495: I tensorflow/c/logging.cc:34] x-envoy-decorator-operation: minio-service.kubeflow.svc.cluster.local:9000/*

2023-03-14 21:59:13.878510: I tensorflow/c/logging.cc:34] 

2023-03-14 21:59:13.878530: I tensorflow/c/logging.cc:34] 319 bytes written to response.
2023-03-14 21:59:13.878573: I tensorflow/c/logging.cc:34] Returned http response code 200
2023-03-14 21:59:13.878594: I tensorflow/c/logging.cc:34] Returned content type application/xml
2023-03-14 21:59:13.878608: I tensorflow/c/logging.cc:34] Response content-length header: 319
2023-03-14 21:59:13.878621: I tensorflow/c/logging.cc:34] Response body length: 319
2023-03-14 21:59:13.878644: I tensorflow/c/logging.cc:34] Releasing curl handle 0x55d7fe080310
2023-03-14 21:59:13.878665: I tensorflow/c/logging.cc:34] Releasing curl handle 0x55d7fe080310
2023-03-14 21:59:13.878687: I tensorflow/c/logging.cc:34] Notified waiting threads.
2023-03-14 21:59:13.878711: I tensorflow/c/logging.cc:34] Request returned successful response.
2023-03-14 21:59:13.878731: I tensorflow/c/logging.cc:34] Request successful returning.` Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59990">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59990">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

No

### Source

binary

### Tensorflow Version

2.11.0

### Custom Code

No

### OS Platform and Distribution

cos_containerd

### Mobile device

_No response_

### Python version

3.9.2

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current Behaviour?

```shell
Running the "Standalone code to reproduce the issue" on Kubeflow 1.6 trying to use Minio storage via S3 protocol produces the logs attached with the following excerpt showing it is ignoring the S3_USE_HTTPS 0 environment variable: 

2023-03-14 18:07:48.690914: I tensorflow/c/logging.cc:34] Making request to https://10.244.1.171:9000/mlpipeline/tfx-template/data/taxi/data.csv
...
Error message: curlCode: 35, SSL connect error

Note, the following code using boto3 works just fine:

import boto3
session = boto3.session.Session()
s3client = session.client(service_name='s3',
                          aws_access_key_id='minio', 
                          aws_secret_access_key='Wi939w9', 
                          endpoint_url='http://10.244.1.171:9000', 
                          use_ssl='0', 
                          verify='0', 
                          region_name='us-east-1')

{'ResponseMetadata': {'RequestId': '174C5DC82C4E602D',
  'HostId': '',
  'HTTPStatusCode': 200,
  'HTTPHeaders': {'accept-ranges': 'bytes',
   'content-length': '667',
   'content-security-policy': 'block-all-mixed-content',
   'content-type': 'application/xml',
   'server': 'envoy',
   'vary': 'Origin',
   'x-amz-request-id': '174C5DC82C4E602D',
   'x-xss-protection': '1; mode=block',
   'date': 'Tue, 14 Mar 2023 18:51:15 GMT',
   'x-envoy-upstream-service-time': '1'},
  'RetryAttempts': 0},
 'IsTruncated': False,
 'Marker': '',
 'Contents': [{'Key': 'tfx-template/data/taxi/data.csv',
   'LastModified': datetime.datetime(2023, 2, 27, 23, 47, 55, 766000, tzinfo=tzlocal()),
   'ETag': '"43550bbc6946bcabc55328e0adb129b3-1"',
   'Size': 1922812,
   'StorageClass': 'STANDARD',
   'Owner': {'DisplayName': '',
    'ID': '02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4'}}],
 'Name': 'mlpipeline',
 'Prefix': 'tfx-template/data/taxi/data.csv',
 'Delimiter': '',
 'MaxKeys': 1000,
 'EncodingType': 'url'}
```


### Standalone code to reproduce the issue

```shell
import tensorflow as tf
import tensorflow_io as tfio

import os
os.environ['AWS_REGION'] = 'us-east-1'
os.environ['S3_ENDPOINT'] = 'http://10.244.1.171:9000'
os.environ["AWS_ACCESS_KEY_ID"] = 'minio'
os.environ["AWS_SECRET_ACCESS_KEY"] = 'Wi939w9'
os.environ['S3_USE_HTTPS'] = '0'
os.environ['S3_VERIFY_SSL'] = '0'
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "0"
os.environ["TF_CPP_MIN_VLOG_LEVEL"] = "5"
os.environ["AWS_LOG_LEVEL"] = "trace"

tf.io.gfile.glob('s3://mlpipeline/tfx-template/data/taxi/data.csv')
# exists produces similar logs
#tf.io.gfile.exists('s3://mlpipeline/tfx-template/data/taxi/data.csv')
```


### Relevant log output

```shell
2023-03-14 18:07:48.690794: I tensorflow/c/logging.cc:34] No content body, content-length headers
2023-03-14 18:07:48.690823: I tensorflow/c/logging.cc:34] Found credential in environment with access key id minio
2023-03-14 18:07:48.690830: I tensorflow/c/logging.cc:34] Found secret key
2023-03-14 18:07:48.690836: I tensorflow/c/logging.cc:34] Note: Http payloads are not being signed. signPayloads=0 http scheme=https
2023-03-14 18:07:48.690857: I tensorflow/c/logging.cc:34] Canonical Header String: amz-sdk-invocation-id:26CFFBE4-9F97-48D7-BC0A-FFA9613B9DC0
amz-sdk-request:attempt=1
content-type:application/xml
host:10.244.1.171:9000
x-amz-api-version:2006-03-01
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20230314T180748Z

2023-03-14 18:07:48.690861: I tensorflow/c/logging.cc:34] Signed Headers value:amz-sdk-invocation-id;amz-sdk-request;content-type;host;x-amz-api-version;x-amz-content-sha256;x-amz-date
2023-03-14 18:07:48.690872: I tensorflow/c/logging.cc:34] Canonical Request String: HEAD
/mlpipeline/tfx-template/data/taxi/data.csv

amz-sdk-invocation-id:26CFFBE4-9F97-48D7-BC0A-FFA9613B9DC0
amz-sdk-request:attempt=1
content-type:application/xml
host:10.244.1.171:9000
x-amz-api-version:2006-03-01
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20230314T180748Z

amz-sdk-invocation-id;amz-sdk-request;content-type;host;x-amz-api-version;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD
2023-03-14 18:07:48.690886: I tensorflow/c/logging.cc:34] Final String to sign: AWS4-HMAC-SHA256
20230314T180748Z
20230314/us-east-1/s3/aws4_request
04da96fd45298dbfd632d7a49f6b1d7de1b62583c81bb099e9b860b2b86628a3
2023-03-14 18:07:48.690891: I tensorflow/c/logging.cc:34] Final computed signing hash: 23e330b317746f6b48081889f4b7bb2756aa9eefe49c098b8c82ca81da3a6c39
2023-03-14 18:07:48.690899: I tensorflow/c/logging.cc:34] Signing request with: AWS4-HMAC-SHA256 Credential=minio/20230314/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-type;host;x-amz-api-version;x-amz-content-sha256;x-amz-date, Signature=23e330b317746f6b48081889f4b7bb2756aa9eefe49c098b8c82ca81da3a6c39
2023-03-14 18:07:48.690905: I tensorflow/c/logging.cc:34] Request Successfully signed
2023-03-14 18:07:48.690914: I tensorflow/c/logging.cc:34] Making request to https://10.244.1.171:9000/mlpipeline/tfx-template/data/taxi/data.csv
2023-03-14 18:07:48.690919: I tensorflow/c/logging.cc:34] Including headers:
2023-03-14 18:07:48.690922: I tensorflow/c/logging.cc:34] amz-sdk-invocation-id: 26CFFBE4-9F97-48D7-BC0A-FFA9613B9DC0
2023-03-14 18:07:48.690926: I tensorflow/c/logging.cc:34] amz-sdk-request: attempt=1
2023-03-14 18:07:48.690929: I tensorflow/c/logging.cc:34] authorization: AWS4-HMAC-SHA256 Credential=minio/20230314/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-type;host;x-amz-api-version;x-amz-content-sha256;x-amz-date, Signature=23e330b317746f6b48081889f4b7bb2756aa9eefe49c098b8c82ca81da3a6c39
2023-03-14 18:07:48.690932: I tensorflow/c/logging.cc:34] content-type: application/xml
2023-03-14 18:07:48.690935: I tensorflow/c/logging.cc:34] host: 10.244.1.171:9000
2023-03-14 18:07:48.690939: I tensorflow/c/logging.cc:34] user-agent: aws-sdk-cpp/1.8.186 Linux/5.15.0-67-generic x86_64 GCC/7.3.1
2023-03-14 18:07:48.690945: I tensorflow/c/logging.cc:34] x-amz-api-version: 2006-03-01
2023-03-14 18:07:48.690949: I tensorflow/c/logging.cc:34] x-amz-content-sha256: UNSIGNED-PAYLOAD
2023-03-14 18:07:48.690954: I tensorflow/c/logging.cc:34] x-amz-date: 20230314T180748Z
2023-03-14 18:07:48.690960: I tensorflow/c/logging.cc:34] Attempting to acquire curl connection.
2023-03-14 18:07:48.690966: I tensorflow/c/logging.cc:34] Connection has been released. Continuing.
2023-03-14 18:07:48.690971: I tensorflow/c/logging.cc:34] Returning connection handle 0x56396c2af710
2023-03-14 18:07:48.690976: I tensorflow/c/logging.cc:34] Obtained connection handle 0x56396c2af710
2023-03-14 18:07:48.695426: E tensorflow/c/logging.cc:40] Curl returned error code 35 - SSL connect error
2023-03-14 18:07:48.695462: I tensorflow/c/logging.cc:34] Destroy curl handle: 0x56396c2af710
2023-03-14 18:07:48.695483: I tensorflow/c/logging.cc:34] Created replacement handle and released to pool: 0x56396c2af710
2023-03-14 18:07:48.695495: I tensorflow/c/logging.cc:34] Request returned error. Attempting to generate appropriate error codes from response
2023-03-14 18:07:48.695504: E tensorflow/c/logging.cc:40] HTTP response code: -1
Resolved remote host IP address: 
Request ID: 
Exception name: 
Error message: curlCode: 35, SSL connect error
0 response headers:
2023-03-14 18:07:48.695516: W tensorflow/c/logging.cc:37] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2023-03-14 18:07:48.695521: I tensorflow/c/logging.cc:34] Date header was not found in the response, can't attempt to detect clock skew
2023-03-14 18:07:48.695527: W tensorflow/c/logging.cc:37] Request failed, now waiting 0 ms before attempting again.
2023-03-14 18:07:48.695595: I tensorflow/c/logging.cc:34] No content body, content-length headers
2023-03-14 18:07:48.695603: I tensorflow/c/logging.cc:34] Found credential in environment with access key id minio
2023-03-14 18:07:48.695609: I tensorflow/c/logging.cc:34] Found secret key
2023-03-14 18:07:48.695617: I tensorflow/c/logging.cc:34] Note: Http payloads are not being signed. signPayloads=0 http scheme=https
2023-03-14 18:07:48.695637: I tensorflow/c/logging.cc:34] Canonical Header String: amz-sdk-invocation-id:26CFFBE4-9F97-48D7-BC0A-FFA9613B9DC0
```
</details>