title: For Loop For Set Weeks from Function in R
body: <p>I am using Databricks to retrieve data from Snowflake. Currently the function being used in R works as is. However, my dataset is very large (100B rows) and would like to call my dataset in the form of a for loop to lower the memory. How would I retrieve the dataset like below in terms of a for loop?</p>
  <pre><code>## Call Libraries to retrieve dataset from Snowflake
  library(rJava)
  library(RJDBC)
  library(DBI)
  
  ## Connect to Snowflake
  
  jdbcDriver &lt;- JDBC(driverClass=&quot;com.snowflake.client.jdbc.SnowflakeDriver&quot;,
                    classPath=&quot;dbfs:/FileStore/jars/ced63da8_4613_4687_9051_1ea61dbf3b76- 
  snowflake_jdbc_3_12_12-30836.jar&quot;)
  
  jdbcConnection &lt;- dbConnect(jdbcDriver,
                             &quot;jdbc:snowflake://xxx.us-east-1.snowflakecomputing.com/&quot;,
                             &quot;xxxx&quot;,
                             dbutils.secrets.get(scope=&quot;xxx&quot;, key=&quot;xxx&quot;))
  </code></pre>
  <p>Below is where I would like the for loop to be implemented to retrieve weekly data instead of whole dataset and append results to one another. The goal is to create a new table in Snowflake from these results but curious how to create the for loop first.</p>
  <pre><code>## Run SQL Query to retrieve dataset from Snowflake in the form of an R function
  sq &lt;- function(query){
   dbGetQuery(jdbcConnection, knitr::knit_expand(text=query))
  }
  
  
  show1 &lt;- function(a) 
    sq(
      paste(
  &quot;  
  select
  data_source,
  media_id, 
  sum(activity_count)
  from&quot;, a, &quot;
  where START_DATE_EST between '2019-11-01' and '2021-11-14' &quot;
  )
  
  )
  activity_by_month&lt;-show1(&quot;SCHEMA.TABLE_NAME&quot;)
  </code></pre>
  <p>If I were to run this it would take days and it will timeout due to the amount of data. I want to run this function in the format of a for loop where I collect the data by week and append the results together. Anyone ever do this before?</p>
  <p>The ideal approach would be to run this like this:</p>
  <pre><code>start_date_vector &lt;- c('2019-11-01', '2019-11-09', etc.)
  end_date_vector &lt;- c('2019-11-08', '2019-11-16', etc.)
  
  for(i in start_date_vector) {
    for(j in end_date_vector) {
      show1}}
  
  rbind(show1)
  </code></pre>
  <p>The SQL query above would look something like this:</p>
  <p><code>where START_DATE_EST between&quot;, start_date_vector,&quot; and&quot;,end_date_vector &quot;</code></p>
  <p>And then write a SQL query that will union the results together and save a table.</p>