In this paper the efficiency of using VNS (Variable Neighborhood Search) algorithm for forming four member heterogeneous groups within CSCL (Computer supporting collaborative learning) is analyzed. A mathematical model, based on Kagan's instructions, was created and then the VNS algorithm, the metaheuristic for solving the mathematical optimization problems, was applied to the model. The proposed VNS method is tested on a set of problem instances and results are compared with the optimal results obtained by CPLEX solver applied to the proposed formulation. VNS method showed better performance in terms of execution time and being able to solve large problem instances. The CSCL was applied to three groups of the first year college students, each consisting of 172 students. These three groups were divided into smaller ones of four students: by using VNS algorithm in 2015 (group E), by using Kagan's instructions in 2014 (group K), and randomly in 2013 (group R). The students were tested before and after CSCL of calculus contents. The statistical analysis shows that the students divided by VNS algorithm had significantly better results than the students divided randomly. But the students divided by VNS algorithm were as successful as the students divided without computer. This means that the students' learning achievement in calculus contents is better when they are divided by VNS than randomly, but is as successful as the cooperative learning in heterogeneous groups when VNS was not applied. (C) 2017 Elsevier Ltd. All rights reserved.