HF space auth to read dataset to ducdkdb (using secrets)
Hi,I am trying to use the duckdb feature describedherewhich allows you to read a table from a private dataset.I use the following code:with duckdb.connect() as con:
                 con.execute(f"""CREATE SECRET REDACTED_TOKEN (TYPE HUGGINGFACE, TOKEN '{REDACTED_TOKEN}');""")
                con.execute("""CREATE TABLE IF NOT EXISTS documents (...)""")
                df = con.execute(f"""SELECT * FROM '{dataset_uri}';""").df()
               con.sql("INSERT INTO documents SELECT * FROM df")I have tested it and it runs on a docker image with the exact same code,  variables and requirements. However, when running it in the space it throws a HTTP 401 error:HTTPException: HTTP Error: HTTP GET error on ‘https://huggingface.co/datasets/user/dataset/data/*.parquet’ (HTTP 401)Could you please help me understand the issue or If I have missed a similar forum post could you point me to it?Thanks!
2024-11-12T01:32:25Z
The contents of thedataset_uri variableare probably wrong. I think you were trying to use awildcardspecification, but the program is trying to accessthe filehttps://huggingface.co/datasets/user/dataset/data/*.parquetas if it actually exists.It’s difficult to specify wildcards for HF datasets, but you can use the following function to get a list of the dataset files.huggingface.coHfApi ClientWe’re on a journey to advance and democratize artificial intelligence through open source and open science.
2024-11-12T09:43:28Z
So a bit more info I should have probably provided in the initial issue. The dataset is of form:hf://datasets/⟨my_username⟩/⟨my_dataset⟩/⟨path_to_file⟩I have tried pointing to a single file like so:SELECT *
FROM 'hf://datasets/datasets-examples/doc-formats-parquet-1/data/train-00000-of-00001.parquet';or to the whole set of parquet files of the dataset like so:SELECT count(*) AS count
FROM 'hf://datasets/cais/mmlu/astronomy/*.parquet';Indeed I have tried andusing both a local setup and a docker image I can read from the datasetusing the code provided in the initial post.Therefore, my suspicion is that it could either be an issue with the spaces’ privileges or variable handling.To be more precise, variables are handled with the python-dotenv library, using load_dotenv(). dataset_uri is then initialized with:load_dotenv()
os.getenv("dataset_uri")
2024-11-12T12:54:35Z
Just to confirm, if you are using a private or gated repo, you will need your read token.The following settings seem to be required for this library.Some HF libraries may implicitly use the HF_TOKEN environment variable, but third-party libraries usually require it to be explicitly specified.DuckDB – 29 May 24Access 150k+ Datasets from Hugging Face with DuckDBDuckDB can now read data from Hugging Face via the hf:// prefix.
2024-11-12T14:42:31Z
Indeed I am using the steps provided in the documentation you sent the link to (it was in my initial post and all the examples were extracted from there).I am using a private dataset repo; REDACTED_TOKEN (with read permissions) is a space secret and then loaded with python-dotenv:load_dotenv()
REDACTED_TOKEN = os.getenv("HF_TOKEN")
with duckdb.connect() as con:
                 con.execute(f"""CREATE SECRET REDACTED_TOKEN (TYPE HUGGINGFACE, TOKEN '{REDACTED_TOKEN}');""")Again, using this approachworks locally and with a docker image. So my suspicion is that there might be some privilege issue in spaces that blocks the request. Any ideas?
2024-11-12T14:51:24Z
Spaces has a lot of restrictions that we users don’t really understand. If it’s a configurable item, you can generally set it from the following.huggingface.coSpaces Configuration ReferenceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.