text: Perplexity-based filtering. As explained in Section 6, perplexity is a measure of how “natural” the text looks. Perplexity- 
  based filtering has been suggested in many contexts as a defense against adversarial text inputs [16, 36]. This defense 
  computes the perplexity of multiple “trusted” texts, then compares it with the perplexity of the suspicious text. If the latter 
  is significantly higher, or above some predefined threshold, the text is considered adversarial. Specifically, we assume the 
  defender has access to a set of unmodified queries. The defender computes their perplexity values and uses these values 
  to establish a threshold. Given a new query, the defender checks if its perplexity exceeds the threshold. If so, the query 
  is flagged as adversarial. The defender can then decide how to handle such queries. Options include rejecting them or 
  routing them all to the weak model. Computing the perplexity of a query can be cheap to do, e.g., using GPT-2 as we do 
  in this work; this makes it viable for use as a defense that doesn’t undermine the benefits of routing. 
  
  To evaluate the effectiveness of such a defense against our attack, we compare the perplexity values of original and 
  confounded queries. Figure 5 presents histograms of perplexity values for both the original evaluated GSM8K queries and 
  their corresponding confounded versions, generated using one of the confounder gadgets, sampled uniformly at random. 
  Additionally, the figure displays the ROC curve for the defense that detects confounded queries by checking if their 
  perplexity exceeds a threshold. As can be seen, the confounded queries exhibit significantly higher perplexity values, 
  making them readily distinguishable from the original queries. For instance, in the case of the RSW router, setting the 
  threshold value at 55 yields a false-positive rate of 3% and a true-positive rate of 97%. Results are similar for other gadgets 
  and benchmarks and were omitted due to space constraints. 