Adventures in R: A Function for Basic Data Screening in R

Note: I’m using my dissertation as an excuse to learn R. I’m going to put out some of my code for others to peruse/use/improve/correct. Hopefully we can all learn a little bit from it. If you do use it or change it, please let me know in the comments what you did so I can learn too!

Most statistics that you are going to run rely on a number of assumptions about your data. One of the most frequent (pun intended) assumptions is that your data is normally distributed.  The function I’ve provided below will output into a text file (and the R terminal): basic summary statistics, Skewness, Kurtosis, and the Shapiro-Wilks Test for Normality. The function will also output a histogram, density graph, box plot, and Q-Q plot so that you can really a really good idea of what the ‘shape’ of your data look like. Just like the reliability function, you will need to load the ‘psych’ package to use this.

datascreening <- function (x) {

pdf(paste(deparse(substitute(x)), ".pdf"))
sink("screeningcalcs.txt", append=TRUE, split=TRUE)

split.screen(figs=c(2,3))

screen(1)
hist(x, xlab = deparse(substitute(x)))
screen(2)
plot(density(x))
screen(3)
boxplot(x, xlab = deparse(substitute(x)))
screen(4)
qqnorm(x)

summary <- summary(x)
skew <- skew(x)
kurtosis <- kurtosi(x)
shapiro <-shapiro.test(x)

cat(paste("############################################################", "\n"))
cat(paste("SCREENING RESULTS FOR VARIABLE", deparse(substitute(x)), "\n", "\n"))
print(summary)
cat(paste("\n", "\n", "Skewness:", skew))
cat(paste("\n", "\n", "Kurtosis:", kurtosis))
cat(paste("\n", "\n", "Shapiro-Wilks test for normality (want >0.05):", shapiro\$p.value,"\n", "\n", "\n", "\n"))

close.screen(all=TRUE)
dev.off()
sink()

}
{"ft_lang_label":"__label__en","ft_lang_prob":0.78516996,"math_prob":0.9634191,"size":1707,"snap":"2019-43-2019-47","text_gpt3_token_len":454,"char_repetition_ratio":0.14973576,"word_repetition_ratio":0.0,"special_character_ratio":0.30931458,"punctuation_ratio":0.15680474,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.99463755,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2019-10-18T03:32:52Z\",\"WARC-Record-ID\":\"<urn:uuid:14f13723-e222-4c86-b356-d2ac0b4ccb16>\",\"Content-Length\":\"51239\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:a2cebd78-d4a4-4738-a1c6-3e3e53214520>\",\"WARC-Concurrent-To\":\"<urn:uuid:a51a589c-69c8-49e7-9625-3b63a77eb9f1>\",\"WARC-IP-Address\":\"108.178.51.244\",\"WARC-Target-URI\":\"http://georgebeckham.com/2015/basic-data-screening-in-r/\",\"WARC-Payload-Digest\":\"sha1:JESGLVW6Q6X2H7VMYUM44ICB7BFP3GUH\",\"WARC-Block-Digest\":\"sha1:KWD3F7S7USUOKVPQ336DVWI7XSKXVZPO\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2019/CC-MAIN-2019-43/CC-MAIN-2019-43_segments_1570986677884.28_warc_CC-MAIN-20191018032611-20191018060111-00404.warc.gz\"}"}