Merge & remove duplicates
The query for populating the users_club_meta table looks good to me except for two issues:


I am not entirely convinced about HAVING COUNT(*) &gt; 1. If you want to create entries for all combinations of (user_id, name, function, description), remove that condition.

If you want to omit the groups consisting only of empty school_grade values, you can use this condition instead:

HAVING COUNT(school_grade) &gt; 0


That is assuming that by "empty" you mean NULL. If you mean to exclude empty strings ('') as well, you could modify the above condition like this:

HAVING COUNT(NULLIF(school_grade, '')) &gt; 0

You are selecting id in that query but the column is not in GROUP BY. It is true that MySQL allows you to do that but the manual also discourages that you do that in cases where the non-aggregated non-GROUP BY column has multiple values per group.

If you want predictable results, use an aggregate function, for instance MIN():

INSERT INTO
  user_clubs_meta (user_clubs_id, meta_key, meta_value) 
SELECT
  MIN(id),
  'school_grades',
  CONCAT('[', GROUP_CONCAT(CONCAT('"', school_grade, '"')), ']')
FROM
  user_clubs 
GROUP BY
  user_id, name, function, description 
HAVING
  COUNT(school_grade) > 0
;


For the DELETE query, take the final draft of the INSERT statement's SELECT query keeping only MIN(id) in the SELECT clause and removing the HAVING clause:

SELECT
  MIN(id)
FROM
  user_clubs 
GROUP BY
  user_id, name, function, description
;


That gives you the first ID of every unique combination of (user_id, name, function, description). That will be the rows you want to keep. All the others you want to remove. This is an anti-join, and there are various ways to implement it. In this case you can go with:


NOT IN:

DELETE FROM
  user_clubs
WHERE
  id NOT IN
  (
    SELECT MIN(id)
    FROM user_clubs 
    GROUP BY user_id, name, function, description
  )
;

LEFT JOIN + WHERE IS NULL:

DELETE
  uc
FROM
  user_clubs AS uc
  LEFT JOIN
  (
    SELECT MIN(id) AS min_id
    FROM user_clubs 
    GROUP BY user_id, name, function, description
  ) AS filter
  ON uc.id = filter.min_id
WHERE
  filter.min_id IS NULL
;



The left join method is known to work fast in MySQL but you should test for yourself in your environment to choose what works better for you.