<https://ru.coursera.org/lecture/open-source-tools-for-data-science/data-refinery-tRvop#lecture>
<http://schema.org/transcript>
"Hi, I'm Sonali Surange Dev. Data scientists often end up spending a lot of time doing mundane tasks like cleansing, shaping and preparing data. Typically these tasks are roadblocks for starting the more enjoyable part of analyzing the data sets or building and training machine learning models. This is because data sets typically are not in a format that can be readily used. They first need to be cleansed, refined before they are useable by a data scientist. IBM Data Refinery addresses this issue and simplifies the task of refining data and its workflows. It provides a self-service data preparation environment where you can quickly analyze, cleanse and prepare data sets. Data refinery is available with Watson Studio on public cloud, private cloud and desktop. In the rest of the video we will walk through a scenario and see Data Refinery in action. In this scenario we will use Data Refinery to find the best deals using data about discounts offered over time. We will then automate the analysis to run on a regular schedule. Before the Data Scientist starts, she looks at the data distribution and notices that the inSale column is missing data. She visualizes the offer column and notices that it contains valuable information about discounts. Many fields contain the percent of information, some contain references to previous price indicating a new reduced price being available. She decides to derive sale from offer. She uses a conditional decrease operation to derive if the product is on sale. Next she uses a filter operation to eliminate deals that are not on sale She then wants to pick up the bargains. She uses the replace substring operation and provides a pattern that extracts the discounts from the offer. After converting the discount values to a decimal she can visually see the discounts that were available. She needs to find the months that offered the best deals. She visualizes the dateUpdated and notices that the date field has a variety of formats, some with dashes some with slashes and some with months as text. She hopes that Data Refinery can normalize the data and extract a month. She uses the convert column operation to convert to date and selects ymd. Next she extracts month and creates a derived column called discountMonth. The data now represents all brands and products providing sales and the month the offer was available. The data scientist is only interested in her preferred brands. Over time she has built a list of preferred brands and has imported the data in her project. Data Refinery provides relational transformations such as left, inner right, full, semi and anti-join. To ensure that the data only contains her preferred brand she uses a semi-join operation which narrows the brands to match her preferences. She then selects the keys for the join and the resulting fields. The visual results now confirms that the brands match the preferences. To find the best possible deals she needs to perform some aggregations. Several features determine a good deal. She is interested in the best offer and duration when the discounts are active. Aggregating the sale data will help understand the deals. She groups the columns by brand and discountMonth and calculates the maximum discount. Finally she sorts the result in descending order Data refinery is now displaying the best deals by brand preferences and the duration which the offer is available. The last step is to execute the analysis on the full dataset. She starts the full analysis, which she can monitor for the completion status. It's time to automate the analysis which runs on a regular basis. The data in the database can grow over time. She uses a personalized runtime to match the larger data volumes and sets a schedule for automation. The hourly schedule reads from updated data from the database and writes to the target table. Data Refinery has helped her uncover deals in the raw data through a small set of operations and transformations with the bulk of the work done for her. Thank you for watching"
<https://ru.coursera.org/lecture/open-source-tools-for-data-science/data-refinery-tRvop>