Sherwood took over in February with Villa in the bottom three and led the club to Premier League safety and 17th.
"We gave the fans nothing to cheer about today but I promise them it will get better," Sherwood told BBC Sport.
"We've stayed in the division but we have a losing mentality. We don't want to be scraping relegation next season."
Villa were outclassed by Arsenal, whose victory at Wembley saw them become the most successful club in FA Cup history with a 12th title.
Sherwood acknowledged his side "came up short" against the Gunners, adding that his players "need to learn how to be winners".
Media playback is not supported on this device
"Of course we believed we could win this but I can't make any excuses," the 46-year-old added. "It's a learning curve and we need to build a new team.
"Some of the boys have already been told they won't be here next year or they can find other clubs. I want to be honest with players and don't want to be stringing experienced players along.
"But it's not just new additions to the squad but also getting some of the players we do have to play in a different way, to change the mentality of a few of them."
Villa captain Fabian Delph agreed Arsenal had been worthy winners and also stressed the importance of Villa finding a "winning mentality" next season.
"We were up against it, we tried to dig deep but their quality shone through and we struggled," the 25-year-old said.
"The manager said that we had it in us to cause an upset but we couldn't find the performance. But a few weeks ago we would have snapped your hand off to stay in the league and get to this final. Now it's important that we get a winning mentality.
"We have to forget about today, some players will come through the door to strengthen us and I am sure we will get better and better."
Aston Villa manager Tim Sherwood promised to rid the club of its "losing mentality" after suffering a 4-0 defeat by Arsenal in the FA Cup final.