Abliterated concern
Hi!
I know that mistral usually doesn't censor their models and with just regular prompting it will pretty much do anything you tell it to. Considering this and the fact that abliteration is usually dumbing down the model, does it make sense to use abliterated as part of the merge ?
Hello, when i tested pure mistral version, i encountered some refusals and moralizing, so i decided to use abliterated model. I didn't knew about dumbing thing, so if this merge will be not "smart" enough on testing, i will redo it with pure mistral. Thank you for this information.
Let's see how it behaves. Hopefully the model's smart will not be affected. With this in mind, it would be amazing if positivity bias is tones down. Not sure how bias pure mistral is, as I have yet to test it. But in my experience most of them tend to be too eager to jump on the "let's all hold hands and skip" bandwagon.
I don't know about the base Mistral model being smart... I guess when it comes to "smart" we all imagine different things. I don't mean to treat the company behind Mistral models harshly, but this latest model simply cannot answer something the oldest, much smaller Mistral v1 answered fairly well (fairly well for its small size that is). It feels like that particular knowledge is simply missing here, because no matter what settings I tried, never did it even get close to an accurate answer. Mistral v1 was so much better at that particular topic. I mean, how bad it really is if a much smaller and older model from the same company gives more accurate answer? To me it just feels like every new Mistral model is somehow less smart than the last one or at least less smart than it could have been, given the usual curve in technological advancement.
It's unfair to judge any model's intelligence by the base model, especially when it comes to RP contexts—Especially considering that 24B isn't using synthetic logs (nor did it benchmax, which has a side effect of trading intelligence for stale souless RP). I'm very excited to see where these models and merges go!
I disagree. Intelligence of every model starts with the base model first and foremost no matter the use (such as RP in this case). Finetuning or merging is just adding to that base model. If the base model doesn't contain enough data about certain topics and finetunes and or models merged into that base model don't contain enough data about those topics either then that model will just not be good enough for those topics, so no matter how many finetunes or merges the model went through, your roleplay about those topics will not be good enough.
Base models tend to perform quite poorly at various benchmarks. Further training enhances knowledge and intelligence. This isn't even a controversial fact, it's a well known fact. This is reproducible and shown for years on every benchmark you can imagine.
Simply look at any benchmark that measures knowledge. Now look at the base model. Now look at a finetune on that base model. Number go up? Proof that it's possible. Number only goes down? Then I'll concede that you're right for that specific benchmark and that specific base model.
[edit: Nevermind, even that wouldn't be enough. It could still be the case that a future finetune could improve this number, but that it simply hasn't been done yet. After all, the overwhelming evidence shows that this is possible on virtually any model.]
Fair point, however I never argued that to begin with. I was just pointing out that if there's no knowledge on certain topic in the base model, nor is it added by merges and / or finetunes, then the model will perform poorly on that particular topic. Also it is quite disturbing to see that a very old small model is better at certain topic than a much newer and more than 3 times bigger model from the same company. It feels just wrong like the knowledge of that particular topic has been cut off and downgraded and now it's limited compared to that much smaller older model.