 and natural language pro- cessing (NLP) to extract structured features that can be further processed for insights and actions. 4. Variability: Raw data and derived data often have variability and changes over time. This can be a challenge for big data applications, which need to adapt based on the chang- ing data. For instance, a "recommender engine" for movies needs to adapt and update its recommendations as a user's tastes change or new movies come out. In addition, some actions triggered from analytics on the data may utilize previously computed batch processed data, as well as real-time data. Thus, it is important that any big data archi- tecture facilitate both batch and real-time processing. One such architectural approach to accomplish this is referred to as the "lambda architecture" as will be described in Section 8.2.3. 5. Validity/veracity: Most have heard the phrase "garbage in equals garbage out" and manag- ing the quality or validity/veracity of the data is one of the largest challenges in big data. For instance, how much of the data has "missing values"? Is the data in the same format? (e.g., Do some set of phone numbers have the country code, area code, phone number, while others only have area code, phone number) Does the data have expected format? (e.g., Does a phone number have at least 10 numbers? Do names have both first name and last name?) Any big data processing system needs to have processes and checks in place to check, resolve if possible, or at minimum flag any data quality issues. Data quality issues are dealt with in a subsequent section. 6. Visibility/visualization: Understanding some of the basic attributes and summaries of a big data set is typically very hard, and having the ability to visualize the data via charts, graphs, heat maps, etc. is often very useful. Nevertheless, visualizing and manipulating large sets of data is a challenge for which new tools such as AT&T's nano-cubes [Att13] were invented. This tool allows visualizing and summarizing billions of data points in a co