Argilla: What's the simplest way to use Suggestions when I log() new records from Python dicts?
Hey there! I’m a beginner, using Argilla for labeling. I’m deployed in a HF Space for my startup. I’m using the Python API, and amlog()ing dicts rather than creatingrg.Recordobjects.I’m a bit confused about how to useSuggestions to have both AI-generated responses to myQuestions, as well as human-labeled responses.My schema looks like this:settings = rg.Settings(
    guidelines="""Given the Attribute we're trying to evaluate, and with the given examples \
    and the Behavior Interview Question in mind, score the candidate's response on a scale of 0 to 10 and \
    identify quotes from their response which provide evidence to back up your score.""",
    fields=[
        rg.TextField(
            name="candidate",
            title="Candidate",
            description="The candidate we are trying to judge.",
            use_markdown=False,
        ),
        rg.TextField(
            name="interview_id",
            title="Interview ID",
            description="The internal ID for the interview.",
            use_markdown=False,
        ),
        rg.TextField(
            name="attribute",
            title="Attribute",
            description="The attribute we are trying to judge in the candidate's response to the Behavioral Interview Question.",
            use_markdown=True,
        ),
        rg.TextField(
            name="attribute_definition",
            title="Definition",
            description="Definition of this Attribute.",
            use_markdown=True,
        ),
        rg.TextField(
            name="examples",
            title="Examples",
            description="Example rating and quotes for this Attribute.",
            use_markdown=True,
        ),
        rg.TextField(
            name="biq",
            title="BIQ",
            description="The Behavioral Interview Question we have asked in order to judge the candidate's fit with the given Attribute.",
            use_markdown=True,
        ),
        rg.TextField(
            name="response",
            title="Candidate Response",
            description="The Candidate's response to the Behavioral Interview Question.",
            use_markdown=True,
        ),
    ],
    questions=[
        rg.RatingQuestion(
            name="rating",
            title="numeric rating",
            description="What is the candidate's score for this attribute, from 0-10, using the examples as guidance?",
            values=list(range(11)),  # 0..10
        ),
        rg.SpanQuestion(
            name="quotes",
            title="Quotes",
            description="Candidate quotes that provide evidence for the score.",
            field="response",
            allow_overlapping=True,
            labels=["evidence"],
        ),
        rg.TextQuestion(
            name="reasoning",
            title="Reasoning",
            description="LLM's explanation of its rating.",
        )
    ],
)I’m using a Pydantic model for a bit of runtime type checking, but then convert the data into a simpledictbefore logging it:record = ArgillaValuesModel(candidate = interview_info["candidateName"],
                                    interview_id = interview_config.interview_id(),
                                    attribute = attr_name,
                                    attribute_definition = attr.description,
                                    examples = "<br>".join([f"Text: {example.text}\nScore: {example.score}" for example in attr.examples]),
                                    biq = q_text,
                                    response = response,
                                    quotes = quote_ranges,
                                    rating = score,
                                    reasoning = reasoning,
                                   )
        pprint.pp(record.dict())
        argilla_records.append(record.dict())I’m then callinglog()thusly:dataset.records.log(argilla_records)Ok, so… in this case the things I’m inserting for theQuestionfields are from the LLM, so they should beSuggestions, right? What’s the simplest way to tweak this?The example atAdd, update, and delete records - Argilla Docsdoesn’t exactly match up to my situation; it’s geared about adding suggestions for a label. In my case, I have three questions:rating,quotes, andreasoning…I want to keep both the AI-generated answersandthe human-generated labels in my Argilla dataset.What’s the easiest way to add this?
2024-10-23T22:21:30Z
Following the example here:rg.Suggestion - Argilla Docswhich looks like this:dataset.records.log(
    [
        {
            "prompt": "Hello World, how are you?",
            "label": "negative",  # this will be used as a suggestion
            "score": 0.9,  # this will be used as the suggestion score
            "model": "model_name",  # this will be used as the suggestion agent
        },
    ],
    mapping={
        "score": "label.suggestion.score",
        "model": "label.suggestion.agent",
    },  # `label` is the question name in the dataset settings
)I added amappingdict to my call tolog():dataset.records.log(
    argilla_records,
    mapping={
        "model": "rating.suggestion.agent",
        "model": "quotes.suggestion.agent",
        "model": "reasoning.suggestion.agent",        
    }
)Note that my threeQuestionfields for which I have my model making predictions arerating, which is numeric;quotes, which is a list of spans, andreasining, which is an explanation of the reasoning behind the numericrating.If I add thismappingdict, thereasoningfield disappears from the web UI, and the other twoQuestions appear unchanged…There must be something obvious I’m missing!
2024-10-24T17:44:07Z
Ok, I switched to using rg.Record instead of the dict api, and now it’s working:record = rg.Record(
            fields = {
                "candidate": interview_info["candidateName"],
                "interview_id": interview_config.interview_id(),
                "attribute": attr_name,
                "attribute_definition": attr.description,
                "examples": "<br>".join([f"Text: {example.text}\nScore: {example.score}" for example in attr.examples]),
                "biq": q_text,
                "response": response,
            },
            suggestions = [
                rg.Suggestion(value=quote_ranges, question_name="quotes", type="model"),
                rg.Suggestion(value=score, question_name="rating", type="model"),
                rg.Suggestion(value=reasoning, question_name="reasoning", type="model"),
            ]
        )Yay.IMO, ifSuggestions are supposed to work with thedictform oflog()the docs / examples need to be better documented.