Oh wow thanks for this. I have no clue what is the reason, but despite the error I was still able to use and even quant this model?! Testing the fixed version against this one gave me very different outputs too. If anyone understands this I would love to hear about it?

@Tarek07 hmm what quant tool did you use?

I tried to use following tools against this one, and all failed due to the shape error that I mentioned:

• lm-evaluation-harness
• vllm
• sglang
• AutoAWQ
• transformer (using AutoModelForCausalLM)

Following is lm-evaluation-harness result that I ran against V3.3 and V5 (fixed one), and despite you mentioned about the output being different, evaluation result seems promising?

V3.3

V5

@Tarek07 If you can kindly share what tools did you use to load and run this model (and version), I am happy to try it out on my end and dive deeper!

@y-ryan So for the quant I used 'koboldcpp_tools_19dec' First the 'convert_hf_to_gguf.py' script and then the 'quantize_gguf.exe' to Q8. (this is what I run most models with locally.)

Then as for deployment I used Friendli and the HF Inference Endpoints on Google cloud, both of which worked and gave me good outputs. (I deployed your fixed version in the same way to test some test prompts I use.)

I can certainly see the .safetensor files show: model.layers.0.input_layernorm.weight [1, 8 192]

When it should be: model.layers.0.input_layernorm.weight [8 192]

I just have no idea why it works and works well (from the test prompts at least)?

The GGUF Q8 quant also seemed to work well (only did one RP to test.)

**On some further testing with my GGUF qaunt there are glaring logic issues. It is truly broken. **

@Tarek07 checking out the old version of convert_hf_to_gguf.py from koboldcpp, I found following:

    def prepare_tensors(self):
        ...
        for name, data_torch in chain(self.generate_extra_tensors(), self.get_tensors()):
            ...
            for new_name, data_torch in (self.modify_tensors(data_torch, name, bid)):
                data = data_torch.squeeze().numpy()
                ...

Above code effectively converts [1, 8192] to [8192] by using torch.squeeze().
This code has been then updated (by this commit) not to use torch.squeeze() no more:

-                data = data_torch.squeeze().numpy()
+                # TODO: why do we squeeze here?
+                # data = data_torch.squeeze().numpy()
+                data = data_torch.numpy()

If my assumption is correct, running the latest code will break.

I used torch.Tensor.view() to reshape from [1, 8192] to [8192]. While I believe torch.Tensor.view() and torch.squeeze() will basically have same effect in this case, I will try to reshape again using torch.squeeze() and re-upload the model to see if it makes any difference.

@Tarek07 checking out the old version of convert_hf_to_gguf.py from koboldcpp, I found following:

    def prepare_tensors(self):
        ...
        for name, data_torch in chain(self.generate_extra_tensors(), self.get_tensors()):
            ...
            for new_name, data_torch in (self.modify_tensors(data_torch, name, bid)):
                data = data_torch.squeeze().numpy()
                ...

Above code effectively converts [1, 8192] to [8192] by using torch.squeeze().
This code has been then updated (by this commit) not to use torch.squeeze() no more:

-                data = data_torch.squeeze().numpy()
+                # TODO: why do we squeeze here?
+                # data = data_torch.squeeze().numpy()
+                data = data_torch.numpy()

If my assumption is correct, running the latest code will break.

I used torch.Tensor.view() to reshape from [1, 8192] to [8192]. While I believe torch.Tensor.view() and [torch.squeeze()] will basically have same effect in this case, I will try to reshape again using [torch.squeeze()] and re-upload the model to see if it makes any difference.

Aha! Good catch.

It is broken. On the gguf quant I tried another more complex character card and it had glaring logic issues which other Progenitor models didnt have. So I am convinced somethign broke. Its weird because when I deployed the models on the endpoints the outputs were stellar? Unless the endpoints also fix the error somehow? Regardless I will redo this model just in case. I really appreciate letting me know about the issue I would never have seen it!

@Tarek07
I ran following code:

import os
import torch
from safetensors.torch import load_file, save_file

MODEL_DIR = "/root/.cache/huggingface/hub/models--Tarek07--Progenitor-V5-Final-LLaMa-70B/snapshots/2364d825a8ea4414d3a661b9cf2fa0f29af71756"

def check_shard(shard_path):
    data = load_file(shard_path)

    for key, tensor in data.items():
        if list(tensor.shape) == [1, 8192]:
            print(f"  Checking {key} in {os.path.basename(shard_path)} with size {tensor.shape}")
            viewed_tensor = tensor.view(8192)
            squeezed_tensor = tensor.squeeze()
            print(f"    Equality: {torch.equal(viewed_tensor, squeezed_tensor)}")

def main():
    # Look for .safetensors files in MODEL_DIR
    for filename in sorted(os.listdir(MODEL_DIR)):
        if filename.endswith(".safetensors"):
            shard_path = os.path.join(MODEL_DIR, filename)

            print(f"Processing: {shard_path}")
            check_shard(shard_path)

if __name__ == "__main__":
    main()

and confirmed that tensor.view() and tensor.squeeze() indeed returned identical Tensors. Thus I'm not reuploading the tensor.squeeze() version to my repo.

Glad my debugging was helpful, and thanks again for the great work! Please feel free to let me know at any time if you need other help. I will be around to test the redo-ed version out when it's ready.
I will update the readme on my repo to let others know that you will be redoing and we all are looking forward for it!

@y-ryan Much appreciated!