text: confounder gadgets, queries originally routed to GPT are still routed to GPT and no queries are ever routed to Claude. For 
  queries originally routed to Llama, some gadgets result in all of them being rerouted to GPT, and some have no impact. 
  Specifically, 4 out of the 10 gadgets we optimized using RSW caused all queries to be rerouted to GPT, 2/10 using RM F , 
  and 3/10 using RLLM . None of the gadgets optimized using RCLS had any impact on routing. In terms of costs, having 
  all queries being rerouted to GPT results with an average cost of $0.25, a greater than 8× increase over the cost of the 
  original queries. Given the lack of documentation of the routing algorithm being used, we are unsure what explains the 
  variability across gadgets. 
  
  Martian. This router is supposed to let the user provide a list of models and to specify the maximum amount the user is 
  willing to pay for a query or for 1M tokens. Unfortunately, as of November 14, 2024, the router appears to ignore the list 
  models provided by the user, and forwards the input to the same LLM regardless of it. We tested this in settings including 
  one, two, or multiple models. While responses do not specify which LLM was used, they were identical across settings, 
  so we excluded Martian from our evaluation. We notified Martian about the seemingly buggy behavior. 