title: SAS Proc Freq with PySpark (Frequency, percent, cumulative frequency, and cumulative percent)
body: <p>I'm looking for a way to reproduce the SAS Proc Freq code in PySpark. I found <a href="https://medium.com/eduonline24/can-we-get-sas-proc-freq-with-python-c6af752f387a" rel="nofollow noreferrer">this code that does exactly what I need</a>. However, it is given in Pandas.
  I want to make sure it does use the best what Spark can offer, as the code will run with massive datasets. <a href="http://tech.magnetic.com/2016/05/computing-distributed-groupwise-cumulative-sums-in-pyspark.html" rel="nofollow noreferrer">In this other post</a> (<a href="https://stackoverflow.com/a/42907258/7224879">which was also adapted for this StackOverflow answer</a>), I also found instructions to compute distributed groupwise cumulative sums in PySpark, but not sure how to adapt it to my end.</p>
  <p>Here's an input and output example (my original dataset will have a couple of billions rows):</p>
  <p><strong>Input dataset:</strong></p>
  <pre><code>        state
  0       Delaware
  1       Delaware
  2       Delaware
  3       Indiana
  4       Indiana
  ...     ...
  1020    West Virginia
  1021    West Virginia
  1022    West Virginia
  1023    West Virginia
  1024    West Virginia
  
  1025 rows × 1 columns
  </code></pre>
  <p><strong>Expected output:</strong></p>
  <pre><code>    state           Frequency   Percent Cumulative Frequency    Cumulative Percent
  0   Vermont         246         24.00   246                     24.00
  1   New Hampshire   237         23.12   483                     47.12
  2   Missouri        115         11.22   598                     58.34
  3   North Carolina  100         9.76    698                     68.10
  4   Indiana         92          8.98    790                     77.07
  5   Montana         56          5.46    846                     82.54
  6   West Virginia   55          5.37    901                     87.90
  7   North Dakota    53          5.17    954                     93.07
  8   Washington      39          3.80    993                     96.88
  9   Utah            29          2.83    1022                    99.71
  10  Delaware        3           0.29    1025                    100.00
  </code></pre>