Deploying multiple pre-trained model (tar.gz files) on Sagemaker in a single endpoint
I also have been looking for answers regarding this before, and after several days of trying with my friend, we manage to do it. I attach some code snippet that we use, you may modify it according to your use case
image = '763104351884.dkr.ecr.us-east-1.amazonaws.com/tensorflow-inference:2.2.0-cpu'
container = { 
    'Image': image,
    'ModelDataUrl': model_data_location,
    'Mode': 'MultiModel'
}

sagemaker_client = boto3.client('sagemaker')

# Create Model
response = sagemaker_client.create_model(
              ModelName = model_name,
              ExecutionRoleArn = role,
              Containers = [container])

# Create Endpoint Configuration
response = sagemaker_client.create_endpoint_config(
    EndpointConfigName = endpoint_configuration_name,
    ProductionVariants=[{
        'InstanceType': 'ml.t2.medium',
        'InitialInstanceCount': 1,
        'InitialVariantWeight': 1,
        'ModelName': model_name,
        'VariantName': 'AllTraffic'}])

# Create Endpoint
response = sagemaker_client.create_endpoint(
              EndpointName = endpoint_name,
              EndpointConfigName = endpoint_configuration_name)

# Invoke Endpoint
sagemaker_runtime_client = boto3.client('sagemaker-runtime')

content_type = &quot;application/json&quot; # The MIME type of the input data in the request body.
accept = &quot;application/json&quot; # The desired MIME type of the inference in the response.
payload = json.dumps({&quot;instances&quot;: [1.0, 2.0, 5.0]}) # Payload for inference.
target_model = 'model1.tar.gz'


response = sagemaker_runtime_client.invoke_endpoint(
    EndpointName=endpoint_name, 
    ContentType=content_type,
    Accept=accept,
    Body=payload,
    TargetModel=target_model,
)

response

also, make sure your model tar.gz files have this structure
└── model1.tar.gz
     └── &lt;version number&gt;
         ├── saved_model.pb
         └── variables
            └── ...

more info regarding this