title: collect a sequence of paged results ( 776 pages ) from REST API ( JSON response) and insert final result into DB SQL or create csv file
body: <p>I am consuming a Rest API which is exposing 6212514 rows from SQL Server Database. The response of REST API is JSON which represents the database rows.</p>
  <p>To call this API REST and get the hole data , I am using in each iteration a pagination through parameters <strong>offset</strong> and <strong>limit</strong> :
  exemple:</p>
  <ul>
  <li>call 1 :</li>
  </ul>
  <p><code>http://localhost:8080/myapi/dwcopa/getSlice?**limit**=8000&amp;**offset**=1&amp;buCd=XXX</code></p>
  <ul>
  <li>call 2 :</li>
  </ul>
  <p><code>http://localhost:8080/myapi/dwcopa/getSlice?**limit**=8000&amp;**offset**=2&amp;buCd=XXX</code></p>
  <p>.
  .
  .</p>
  <ul>
  <li>call 760 :</li>
  </ul>
  <p><code>http://localhost:8080/myapi/dwcopa/getSlice?**limit**=8000&amp;**offset**=760&amp;buCd=XXX</code></p>
  <p><em>limit</em> is the number of returned objects by each call to the API REST and <em>offset</em> is the page number</p>
  <p>Example : limit = 8000 and offset 1: Call will return the first 8000 json objects of the page number 1</p>
  <p>To get 6212514 rows i need to do 776 paginations  / call to the client API REST (= 6212514 /8000)  to get all results which I need to store either  in database SQL server table or make a csv file with these 6212514 rows.</p>
  <p>Actually I am using this code in a Spring Boot application (version 2.5.1)  (standalone Spring Boot not REST API)  ( Java 11 / Spring webflux / Webclient ) to consume the client RESt API and fetch all data by pagination :</p>
  <p>In a service JAVA class :</p>
  <p>My Spring <strong>Webclient</strong> to consume the REST API :</p>
  <pre><code>public DwCopaServiceImpl() {
          this.webClient = WebClient.builder()
                  .codecs(codecs -&gt;codecs.defaultCodecs().maxInMemorySize(memSize) )
                  .baseUrl(API_BASE_URL)
                  .defaultHeader(HttpHeaders.CONTENT_TYPE, API_MIME_TYPE)
                  .build();
      }
  </code></pre>
  <p>I am using <strong>Reactive Mono</strong> to non blocking the call API and iterating all pagination util the response.isLast() is false ( this field indicates that this page of REST API is the last )</p>
  <p><strong>DwCopaEntity</strong> is an entity which presents a row ( JSON ) and contains many String fields  :</p>
  <pre><code>public Mono&lt;List&lt;DwCopaEntity&gt;&gt; getItems() {
          String url = &quot;/dwcopa/getSlice?limit=&quot;+limit+&quot;&amp;offset=&quot;+offset+&quot;&amp;buCd=&quot;+BUCD;
  
          return fetchItems(url).expand(response -&gt; {
              if (**response.isLast()** ) {
                  return Mono.empty();
              }
              offset += 1 ;
              
              return fetchItems(&quot;/dwcopa/getSlice?limit=&quot;+limit+&quot;&amp;offset=&quot;+offset+&quot;&amp;buCd=&quot;+BUCD);
          }).flatMap(response -&gt; Flux.fromIterable(response.getContent())).collectList();
      }
  
  private Mono&lt;ResponseApiNeo&gt; fetchItems(String url) {
  
      System.out.println(url);
      return webClient.get().uri(url).retrieve().bodyToMono(ResponseApiNeo.class);
  }
  </code></pre>
  <p>After that to get all results :</p>
  <pre><code>writeStreamToFile ( myService.getItems().block().parallelStream().map(data -&gt; data.toString()), &quot;C:\\Users\\myfolder\\Documents\\optfile.txt&quot;) ;
  </code></pre>
  <p>I am new to <em>webflux</em> and <em>Webclient</em>. This solution takes 25 min to get all Api REST pages and creates a file txt which is not validated by the client:</p>
  <p>Is there a better solution multi threading to:</p>
  <ul>
  <li>paginate a hole rest API REST ( 760 pages):  (large data response : JSON 6 GB in total )</li>
  <li>Insert all response into SQL Database or create a csv file ?</li>
  <li>Performance is crucial ( max 5 min to get all data)</li>
  <li>Should I correct my code to get faster / add Threads ?</li>
  </ul>
  <p>Many Thanks for your helps .</p>