title: Big Bird Pegasus Summarization output is repeating itself
body: <p>I am trying to use Big Bird Pegasus to summarize various long texts. The output is repeating the same concept in each sentence.</p>
  <p>Here is my code using a news article I copied from NPR. The text is longer than the 4096 token limit, so it takes the first few thousand words from my input.</p>
  <pre><code>from transformers import BigBirdPegasusForConditionalGeneration, AutoTokenizer, AutoModelForSeq2SeqLM
  
  model = BigBirdPegasusForConditionalGeneration.from_pretrained(&quot;google/bigbird-pegasus-large-arxiv&quot;)
  tokenizer = AutoTokenizer.from_pretrained(&quot;google/bigbird-pegasus-large-arxiv&quot;)
  
  src_text = '''
  On a sleepy cul-de-sac amid the bucolic vineyards and grassy hills of California's Sonoma Valley, a $4 million house has become the epicenter of a summer-long spat between angry neighbors and a new venture capital-backed startup buying up homes around the nation. The company is called Pacaso. It says it's the fastest company in American history to achieve the &quot;unicorn&quot; status of a billion-dollar valuation — but its quarrels in wine country, one of the first regions where it's begun operations, foreshadow business troubles ahead.
  Brad Day and his wife, Holly Kulak, were first introduced to Pacaso in May after a romantic sunset dinner in their yard. &quot;And we just saw this drone, coming up and over our backyard,&quot; Day says. &quot;And we're like, what is that?&quot;
  Pacaso denies directing or paying a drone operator to film the neighborhood. But its website does have drone photos of the house in question, located at 1405 Old Winery Court. It says it bought the photos after the fact.
  Nonetheless, after the drone incident, Day and Kulak got suspicious about what was going on in their neighborhood. About a week later, their neighbors told them they were moving and selling their house to a limited liability corporation, or LLC. But they were super vague about it.
  Day and Kulak began speaking with other residents on their cul-de-sac. One of them, Nancy Gardner, had learned from a friend in nearby Napa Valley about a new company called Pacaso that was buying houses in the area. The company was co-founded by a Napa resident, and it converts houses into LLCs. Pacaso then sells shares of these corporate houses to multiple investors. Gardner Googled Pacaso, and, sure enough, the house on their cul-de-sac was on its website. The company had named the house &quot;Chardonnay&quot; and was now selling investors the chance to buy a one-eighth share of it for $606,000.
  Pacaso was founded in October 2020 by Austin Allison and Spencer Rascoff, two former executives at Zillow. The company is based in San Francisco, and as is typical of tech startups in the Silicon Valley area, its founders tell a lofty story about their business that's about more than just making money. The company says the motivation for the venture began when Allison and his wife, both based in Napa, bought a second home in Lake Tahoe. The night after they closed on the house, Allison says in a promotional video, he and his wife sat around a fire &quot;thinking how appreciative we were to be second homeowners. And, from that moment, I've always been inspired about making the dream of second home ownership possible for more people.&quot;
  To make second home ownership possible for more people — and, of course, make money — Pacaso uses a &quot;fractional home ownership&quot; model. They buy a house, lightly refurbish it, furnish it and then create an LLC for it. They then divvy up ownership of this corporatized house into eight fractions and sell those shares on their website.
  If you buy a share in a house, you're able to stay in it 44 nights per year in increments that can't exceed 14 consecutive days per visit. You can also &quot;gift&quot; these stays to friends or family. Pacaso offers an app to handle the logistics of booking stays. It oversees management, maintenance and cleaning of the property. In exchange for all this, it charges 12% of the home's purchase price upfront and monthly fees going forward. If you buy a share in a house, you have to hold on to it for a year. After that, you can sell it and profit from any appreciation in the home's value (or be on the hook for any depreciation).
  When Day, Kulak and their neighbors learned about Pacaso's business model, they were appalled. They saw the venture capital-backed company as invading their community and converting their neighbor's house into a revolving carousel of vacationers. They imagined endless parties, noise and cars overflowing their cul-de-sac. They worried those staying at &quot;Chardonnay&quot; would drive too fast and fail to heed local concerns about wildfires and droughts. But, most of all, they feared the Pacaso house and more like it would destroy their sense of community and turn their neighborhood into an &quot;adult Disneyland.&quot;
  The county, Day says, had designated their neighborhood an &quot;exclusion zone,&quot; which bans Airbnb-style, short-term rentals to preserve the &quot;residential character&quot; of communities. But Pacaso argues that its clients are not short-term renters. They are co-owners of an LLC. This also means they don't have to pay the typical taxes on short-term rentals. Likewise, in the nearby town of St. Helena, Pacaso was trying to circumnavigate a city ban against timeshares with the same argument. Day says he and his neighbors saw Pacaso's newfangled business model as nothing more than a &quot;glorified timeshare&quot; with a legal strategy aimed at &quot;skirting regulations that are designed to keep communities intact.&quot;
  The cul-de-sac sprang into action. It formed an organization called Sonomans Together Opposing Pacaso, which, not coincidentally, has the acronym STOP. It contacted the county Board of Supervisors. It created an anti-Pacaso website and circulated an online petition. It flooded the local newspaper with op-eds and letters to the editor. It lobbied local real estate agents not to work with Pacaso. &quot;It feels like we're waging a war by land, air and sea,&quot; Day says.
  Protest signs festoon the neighborhood's lawns, fences and cars. They say things such as &quot;Stop Pacaso&quot; and &quot;Not here, Pacaso!&quot; Day's favorite sign reads, &quot;The Pacaso house is the big one on the right with no soul.&quot;
  The signs, of course, make the prospect of buying a share in the Pacaso house awkward, to say the least. Alfred Miller, however, bought a share in &quot;Chardonnay&quot; before ever seeing it in person. Miller is a risk management consultant based in Los Angeles. He believes in Pacaso's business model. And he likes wine and Sonoma's climate. As he researched &quot;Chardonnay&quot; online, he liked the modern architecture and pool, and he decided he'd buy a one-eighth share of the house. It wasn't until a couple weeks after he made the purchase that he first drove up to Sonoma and witnessed the spectacle around his new investment.
  &quot;So, imagine me as a new owner driving up, and I get to the corner of Old Winery Court,&quot; Miller says. &quot;There's a full-on, professionally printed sign that says 'No Pacaso.' '' Miller then turned right onto Old Winery Court &quot;and the more I drive into the neighborhood, the more signs I see. Brad Day has three vehicles in front of his house, and each vehicle has an anti-Pacaso sign on it. I pull into the driveway — there are two signs on each side of the property. I mean, it was not what I would call very welcoming.&quot;
  As it did on Old Winery Court, controversy erupted in Napa after the company bought a home worth $1.13 million. That's about 35% higher than Napa's median home price. Pacaso insists it only buys luxury and ultra-luxury houses, and it therefore isn't competing with local middle-class families in the housing market. But this home, located two blocks from a high school, didn't quite fit its talking points. Some Napans were pissed. Pacaso says the house was the victim of trespassing and &quot;illegal signage.&quot; Pacaso even claims it had to file a police report after a local wrote to the company and said, &quot;I will burn down any home you buy in Napa. This is no joke.&quot;
  Pacaso's CEO, who lives in Napa, saw firsthand how angry Napans were, and the company responded. In June, Pacaso agreed to sell the Napa home in a traditional manner &quot;to a whole home buyer&quot; rather than convert it into a corporation and sell it to multiple people. The company also pledged to beef up its &quot;Owner Code Of Conduct&quot; to include &quot;decibel limits on all home sound systems,&quot; create a &quot;local liaison&quot; dedicated to assisting neighbors, not buy any homes in the area for under $2 million, and, for each house sold in Napa and Sonoma counties, donate $20,000 to a local nonprofit dedicated to affordable housing.
  But while it has been trying to placate local communities with business reforms, Pacaso has waged a court battle with the town of St. Helena over whether its homes should be classified as timeshares. Pacaso is dead set against that classification. One reason might be that timeshares have a bad rap: While they're a popular way to go on vacations, their costs and associated fees tend to make them money losers rather than a profitable investment.
  Potentially even more damaging to Pacaso's ambitions, however: Timeshares are banned in many vacation communities around the nation. Hence, Pacaso has strong reasons to insist its homes are not timeshares.
  &quot;Unlike a timeshare model, the co-owners that Pacaso serves collectively own real estate, not time,&quot; says Ellen Haberle, director of community and government relations for Pacaso.
  St. Helena disagrees, declaring Pacaso homes are not allowed in the town because of a city ordinance against timesharing. &quot;Simply calling them co-ownership arrangements does not change that fact,&quot; City Attorney Ethan Walsh said. In response to the ban, Pacaso sued the town in federal court. The lawsuit is still pending.
  Pacaso says it plans to expand across North America and Europe. Given the company's billion-dollar valuation, investors seem to believe that many people will be attracted to its model of fractional second home ownership. But local residents will likely continue to fight the unicorn stampeding into their towns.
  '''
  
  device = 'cuda' if torch.cuda.is_available() else 'cpu'
  batch = tokenizer(src_text, truncation=True, padding='longest', return_tensors=&quot;pt&quot;).to(device)
  translated = model.generate(**batch)
  tgt_text = tokenizer.batch_decode(translated, skip_special_tokens=True)
  </code></pre>
  <p>Here is the output. It doesn't mention anything but the topic of fractional ownership. I repeated for another input text and got similar results - each sentence was a slight variation of each other.</p>
  <pre><code>tgt_text
  ['the notion of a fractional ownership in a real property was introduced in the 19th century.&lt;n&gt; fractional ownership in a real property was defined to be the fraction of the value of the property minus the cost of its construction.&lt;n&gt; the fractional ownership of a real property was defined to be the fraction of the value of the property minus the cost of its construction.&lt;n&gt; the fractional ownership of a home is defined to be the fraction of the value of the home minus the cost of its construction. &lt;n&gt; the notion of a fractional ownership in a real property was introduced in the 19th century.&lt;n&gt; the fractional ownership of a home is the fraction of the value of the home minus the cost of its construction.&lt;n&gt; the fractional ownership of a real property was defined to be the fraction of the value of the home minus the cost of its construction.&lt;n&gt; the fractional ownership of a home is the fraction of the value of the home minus the cost of its construction.&lt;n&gt; the fractional ownership of a real property was defined to be the fraction of the value of the home minus the cost of its construction.&lt;n&gt; the fractional ownership of a home was defined to be the fraction of the value of the home minus the cost of its construction.&lt;n&gt; the']
  </code></pre>