Making an alphabatical list grouped by letters
Well, you have a good learning exercise project. here is some notes on the artiest code you've provided :

You're using string query, which opens SQL Injections. To overcome that, you'll need first to mark your query as const or readonly and end the query with a semicolon (which you did), and ensure that the query itself not having any SQL bugs that might be used in inappropriate way. However, if you want the proper way to manager your database, you can use Object-relational Mapping AKA ORM such as Entity Framework.
Store images in a psychical disk and store their path (or filename) into the database. Storing the file bytes into the database would impact the system performance as you'll need to read and render the images every time you select them. While if you store the files physically, it would be faster to render. So, create a folder inside the project folder, store the images in that folder (in any structure you want) then store the path of each image in the database. Then, in your application, you just get the path, adjust the path to be viewable to the user. For example, if the stored path images/artists/someartist.png then you just add the current web url to it like https://www.somewebsite.com/images/artists/someartist.png.
Always use PNG images for web, as png images are more balanced between the quality and size, and would be better option than other types for the web.

In your code, sorting operations should not be done on the presentation layer, in fact, it would be more beneficial if it's moved to early application stages like sorting artists in your case, your application has a core feature to filter artists names in alphabetical order. This means, this functionality needs to be applied to the data before it's even inserted to the database, because it's used as a core feature and not an addon functionality.
There are different ways to do it though, one way is to add a property stores the first name letter like this :
public class LibraryArtistViewModel
{
    public char FirstLetter =&gt; FirstName?.Length &gt; 0 ? FirstName.ToUpper()[0] : char.MinValue;

    public string FirstName { get; set; }

    public string LastName { get; set; }

    public string FullName { get; set; }

    public string PrimarySlug { get; set; }

    public bool HasImage { get; set; }

    public int NumberOfLyrics { get; set; }
        
}

Then, you would do this in GetAllArtistsAsync():
  while (await reader.ReadAsync())

      {
        var artist = new LibraryArtistViewModel
        {
            FirstName = reader[0]?.ToString(),
            LastName = reader[1]?.ToString(),
            FullName = textInfo.ToTitleCase($&quot;{reader[0]?.ToString()} {reader[1]?.ToString()}&quot;),
            PrimarySlug = reader[2]?.ToString(),
            HasImage = reader[3] != System.DBNull.Value;
            NumberOfLyrics = int.TryParse(reader[4]?.ToString(), out int number) ? number : 0;  
        };
        
        artists.Add(artist);
      }
    
    var dictionary = artists.GroupBy(x=&gt; x.FirstLetter).ToDictionary(x =&gt; x.Key, x =&gt; x.ToList()).OrderBy(x=&gt; x.Key);

Though this would give you what you want, it would take out some tiny performance. As I stated before, since your artists are supposed to be sorted, you need to sort it from the time you insert the data, either from the database which would be faster, or on CreateNewArtistAsync action. So, in your Get you won't need to re-sort on every request.