<story><title>Modern Data Practice and the SQL Tradition</title><url>https://tselai.com/modern-data-practice-and-the-sql-tradition.html</url></story><parent_chain><item><author>danso</author><text>I&amp;#x27;ve personally grown to love SQL and I think it is by far the clearest (if verbose) way to describe data transformation work. I learned traditional programming (e.g. languages like Python) long before I stumbled on SQL, but I&amp;#x27;m not sure I could&amp;#x27;ve understood Pandas&amp;#x2F;R as well without having learned SQL, particularly the concepts of joins.&lt;p&gt;That said, my affection for SQL correlates with my overall growth as a software engineer, particularly grokking the concept of piping data between scripts: e.g. writing a SQL script to extract, filter, and transform data, and then a separate Python&amp;#x2F;R script that creates a visualization. I think SQL seems so unattractive to data scientists (i.e. people who may not be  particularly focused on software development) because it seems so alien and separate from the frameworks (R&amp;#x2F;Pandas) they use to do the actual analysis and visualization. And so it&amp;#x27;s understandable to just learn how to do it all in R&amp;#x2F;Pandas, rather than a separate language and paradigm for data wrangling.</text></item></parent_chain><comment><author>mumblemumble</author><text>I think it might go deeper than that. I wear a lot of hats - part software engineer, part data engineer, part data scientist - and I find that the only perspective from which RDBMSes feel like the clear front-runner is when I&amp;#x27;m wearing the software engineer hat.&lt;p&gt;When I&amp;#x27;m wearing the data engineer hat, I start to get interested in some of the options that aren&amp;#x27;t even databases at all, such as storing the data in ORC files. Because they sometimes offer big performance advantages. The data intake pipelines I&amp;#x27;m working on are batch-oriented, so they benefit little from how RDBMSes are optimized for random access to records, and maybe also ranges of records, but only at the cost of write performance if you&amp;#x27;re looking to filter on a natural key instead of a synthetic key. . . it gets thorny. ORC is less-than-stellar at selective ad-hoc queries, but I&amp;#x27;m not doing selective ad-hoc queries against the internals of my data intake pipeline, and I will fight to the death to prevent the BI team from doing so.&lt;p&gt;When I&amp;#x27;m wearing my data scientist hat, I&amp;#x27;ve got no idea what my next query on the data is, so there&amp;#x27;s no indexing strategy that could possibly meet my needs. That kills one of the RDBMS&amp;#x27;s biggest value propositions. So let&amp;#x27;s just wing it with map-reduce and get on with life. And, TBH, SQL is just too declarative for the way my brain works when I&amp;#x27;m in analyst mode - the semi-imperative workflow I get out of something like Spark SQL or R or Pandas feels more natural.</text></comment>
<story><title>Modern Data Practice and the SQL Tradition</title><url>https://tselai.com/modern-data-practice-and-the-sql-tradition.html</url></story><parent_chain><item><author>danso</author><text>I&amp;#x27;ve personally grown to love SQL and I think it is by far the clearest (if verbose) way to describe data transformation work. I learned traditional programming (e.g. languages like Python) long before I stumbled on SQL, but I&amp;#x27;m not sure I could&amp;#x27;ve understood Pandas&amp;#x2F;R as well without having learned SQL, particularly the concepts of joins.&lt;p&gt;That said, my affection for SQL correlates with my overall growth as a software engineer, particularly grokking the concept of piping data between scripts: e.g. writing a SQL script to extract, filter, and transform data, and then a separate Python&amp;#x2F;R script that creates a visualization. I think SQL seems so unattractive to data scientists (i.e. people who may not be  particularly focused on software development) because it seems so alien and separate from the frameworks (R&amp;#x2F;Pandas) they use to do the actual analysis and visualization. And so it&amp;#x27;s understandable to just learn how to do it all in R&amp;#x2F;Pandas, rather than a separate language and paradigm for data wrangling.</text></item></parent_chain><comment><author>alexhutcheson</author><text>I wish SQL had syntax more similar to dplyr[1] for complicated queries. I dislike having to do a nested sub-query every time I re-group the data - the query ends up being a deeply indented mess that needs to be read from the inside-out. Using &amp;quot;WITH&amp;quot; clauses helps a lot, but it&amp;#x27;s still not as easy to read as the more imperative dplyr style.&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;dplyr.tidyverse.org&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;dplyr.tidyverse.org&amp;#x2F;&lt;/a&gt;</text></comment>