text:
Tagged: Exadata
Exadata Smart Flash Logging Explained
I've seen some posts on the blogosphere where people attempt to explain (or should I say guess) how Exadata Smart Flash Logging works and most of them are wrong. Hopefully this post will help clear up some the misconceptions out there.
The following is an excerpt from the paper entitled "Exadata Smart Flash Cache Features and the Oracle Exadata Database Machine" that goes into technical detail on the Exadata Smart Flash Logging feature.
Smart Flash Logging works as follows. When receiving a redo log write request, Exadata will do
parallel writes to the on-disk redo logs as well as a small amount of space reserved in the flash
hardware. When either of these writes has successfully completed the database will be
immediately notified of completion. If the disk drives hosting the logs experience slow response
times, then the Exadata Smart Flash Cache will provide a faster log write response time.
Conversely, if the Exadata Smart Flash Cache is temporarily experiencing slow response times
(e.g., due to wear leveling algorithms), then the disk drive will provide a faster response time.
Given the speed advantage the Exadata flash hardware has over disk drives, log writes should be
written to Exadata Smart Flash Cache, almost all of the time, resulting in very fast redo write
performance. This algorithm will significantly smooth out redo write response times and provide
overall better database performance.
The Exadata Smart Flash Cache is not used as a permanent store for redo data – it is just a
temporary store for the purpose of providing fast redo write response time. The Exadata Smart
Flash Cache is a cache for storing redo data until this data is safely written to disk. The Exadata
Storage Server comes with a substantial amount of flash storage. A small amount is allocated for
database logging and the remainder will be used for caching user data. The best practices and
configuration of redo log sizing, duplexing and mirroring do not change when using Exadata
Smart Flash Logging. Smart Flash Logging handles all crash and recovery scenarios without
requiring any additional or special administrator intervention beyond what would normally be
needed for recovery of the database from redo logs. From an end user perspective, the system
behaves in a completely transparent manner and the user need not be aware that flash is being
used as a temporary store for redo. The only behavioral difference will be consistently low
latencies for redo log writes.
By default, 512 MB of the Exadata flash is allocated to Smart Flash Logging. Relative to the 384
GB of flash in each Exadata cell this is an insignificant investment for a huge performance
benefit. This default allocation will be sufficient for most situations. Statistics are maintained to
indicate the number and frequency of redo writes serviced by flash and those that could not be
serviced, due to, for example, insufficient flash space being allocated for Smart Flash Logging.
For a database with a high redo generation rate, or when many databases are consolidated on to
one Exadata Database Machine, the size of the flash allocated to Smart Flash Logging may need
to be enlarged. In addition, for consolidated deployments, the Exadata I/O Resource Manager
(IORM) has been enhanced to enable or disable Smart Flash Logging for the different databases
running on the Database Machine, reserving flash for the most performance critical databases.
Troubleshooting Oracle Exadata
The crew at My Oracle Support (MOS) [@myoraclesupport] have an excellent starting point for troubleshooting Oracle Exadata. I'd recommend to add this one to your MOS bookmarks.
Oracle Database Machine and Exadata Storage Server Information Center [ID 1306791.1]
Debunking More Netezza FUD About Exadata
A reader recently left a comment for which my reply was longer than I'd like to leave for a comment so I'm answering it in detail with this blog post.
Nice article. I am just reading the Netezza paper.
You don't appear to have debunked the following statement.
"Exadata is unable to process this three table join in its MPP tier and instead must inefficiently move all the data required by the calculation across the network to Oracle RAC."
Not many queries exist where data is only required from two tables. Are Oracle suggesting we need to change the way data is structured to enable best use of Exadata – increasing TCO significantly?
Thanks & Nice post.
There is a reason that I did not debunk that statement – it did not exist in the original version of Netezza's paper. It seems they have taken the shopping basket example that I debunked in my previous post and replaced it with this one. Nonetheless lets take a look at Netezza's claim:
Exadata's storage tier provides Bloom filters to implement simple joins between one large and one smaller table, anything more complex cannot be processed in MPP. Analytical queries commonly require joins more complex than those supported by Exadata. Consider the straightforward case of an international retailer needing insight to the dollar value of sales made in stores located in the UK. This simple SQL query requires a join across three tables – sales, currency and stores.select sum(sales_value * exchange_rate) us_dollar_sales from sales, currency, stores where sales.day = currency.day and stores.country = 'UK' and currency.country = 'USA'
Exadata is unable to process this three table join in its MPP tier and instead must inefficiently move all the data required by the calculation across the network to Oracle RAC.
Before I comment, did you spot the error with the SQL query? Hint: Count the number of tables and joins.
Now that we can clearly see that Netezza marketing can not write good SQL because this query contains a cross product as there is no JOIN between sales and stores thus the value returned from this query is not "the [US] dollar value of sales made in stores located in the UK", it's some other rubbish number.
Netezza is trying to lead you to believe that sending data to the database nodes (running Oracle RAC) is a bad thing, which is most certainly is not. Let's remember what Exadata is – Smart Storage. Exadata itself is not an MPP database, so of course it needs to send some data back to the Oracle database nodes where the Oracle database kernel can use Parallel Execution to easily parallelize the execution of this query in an MPP fashion efficiently leveraging all the CPUs and memory of the database cluster.
The reality here is that both Netezza and Oracle will do the JOIN in their respective databases, however, Oracle can push a Bloom filter into Exadata for the STORES.COUNTRY predicate so that the only data that is returned to the Oracle database are rows matching that criteria.
Let's assume for a moment that the query is correctly written with two joins and the table definitions look like such (at least the columns we're interested in):
create table sales ( store_id number, day date, sales_value number ); create table currency ( day date, country varchar2(3), exchange_rate number ); create table stores ( store_id number, country varchar2(3) ); select sum(sales.sales_value * currency.exchange_rate) us_dollar_sales from sales, currency, stores where sales.day = currency.day and sales.store_id = stores.store_id and stores.country = 'UK' and currency.country = 'USA'
For discussion's sake, let's assume the following:
- There is 1 year (365 days) in the SALES table of billions of rows
- There are 5000 stores in the UK (seems like a realistic number to me)
There is no magic in those numbers, it's just something to add context to the discussion, so don't think I picked them for some special reason. Could be more, could be less, but it really doesn't matter.
So if we think about the the cardinality for the tables:
- STORES has a cardinality of 5000 rows
- CURRENCY has a cardinality of 365 rows (1 year)
The table JOIN order should be STORES -> SALES -> CURRENCY.
With Exadata what will happen is such:
- Get STORE_IDs from STORE where COUNTRY = 'UK'
- Build a Bloom Filter of these 5000 STORE_IDs and push them into Exadata
- Scan SALES and apply the Bloom Filter in storage, retuning only rows for UK STORE_IDs and project only the necessary columns
- JOIN that result to CURRENCY
- Compute the SUM aggregate
All of these operations are performed in parallel using Oracle's Parallel Execution.
Netezza suggests that Exadata can use Bloom filters for only two table joins (1 big, 1 small) and that analytical queries are more complex than that so Exadata can not use a Bloom filter and provide an example to suggest such. The reality is not only is their example incorrectly written SQL, it also works great with Exadata Bloom filters and it is more than 2 tables! In addition, it is a great demonstration of efficient and smart data movement as Exadata can smartly filter using Bloom filters and needs to only project a very few columns, thus likely creating a big savings versus sending all the columns/rows from the storage. Thus Exadata Bloom filters can work with complex analytical queries of more than two tables and efficiently send data across the network to the Oracle RAC cluster where Parallel Execution will work on the JOINs and aggregation in an MPP manor.
Now to specifically answer your question: No, Oracle is not suggesting you need to change your data/queries to support two table joins, Exadata will likely work fine with what you have today. And to let you and everyone else in on a little secret: Exadata actually supports applying multiple Bloom filters to a table scan (we call this a Bloom filter list denoted by the Predicate Information of a query plan by SYS_OP_BLOOM_FILTER_LIST), so you can have multiple JOIN filters being applied in the Exadata storage, so in reality Bloom filters are not even limited to just 2 table JOINs.
Oh well, so much for Netezza competitive marketing. Just goes to show that Netezza has a very poor understanding how Exadata really works (yet again).
Making the Most of Oracle Exadata – A Technical Review
Over the past few weeks several people have asked me about an Exadata article entitled "Making the Most of Oracle Exadata" by Marc Fielding of Pythian. Overall it's an informative article and touches on many of the key points of Exadata, however, even though I read (skimmed is a much better word) and briefly commented on the article back in August, after further review I found some technical inaccuracies with this article so I wanted to take the time to clarify this information for the Exadata community.
Exadata Smart Scans
Marc writes:
Smart scans: Smart scans are Exadata's headline feature. They provide three main benefits: reduced data transfer volumes from storage servers to databases, CPU savings on database servers as workload is transferred to storage servers, and improved buffer cache efficiency thanks to column projection. Smart scans use helper processes that function much like parallel query processes but run directly on the storage servers. Operations off-loadable through smart scans include the following:
- Predicate filtering – processing WHERE clause comparisons to literals, including logical operators and most SQL functions.
- Column projection – by looking at a query's SELECT clause, storage servers return only the columns requested, which is a big win for wide tables.
- Joins – storage servers can improve join performance by using Bloom filters to recognize rows matching join criteria during the table scan phase, avoiding most of the I/O and temporary space overhead involved in the join processing.
- Data mining model scoring – for users of Oracle Data Mining, scoring functions like PREDICT() can be evaluated on storage servers.
I personally would not choose a specific number of benefits from Exadata Smart Scan, simply stated, the design goal behind Smart Scan is to reduce the amount of data that is sent from the storage nodes (or storage arrays) to the database nodes (why move data that is not needed?). Smart Scan does this in two ways: it applies the appropriate column projection and row restriction rules to the data as it streams off of disk. However, projection is not limited to just columns in the SELECT clause, as Marc mentions, it also includes columns in the WHERE clause as well. Obviously JOIN columns need to be projected to perform the JOIN in the database nodes. The one area that Smart Scan does not help with at all is improved buffer cache efficiency. The reason for this is quite simple: Smart Scan returns data in blocks that were created on-the-fly just for that given query — it contains only the needed columns (projections) and has rows filtered out from the predicates (restrictions). Those blocks could not be reused unless someone ran the exact same query (think of those blocks as custom built just for that query). The other thing is that Smart Scans use direct path reads (cell smart table scan) and these reads are done into the PGA space, not the shared SGA space (buffer cache).
As most know, Exadata can easily push down simple predicates filters (WHERE c1 = 'FOO') that can be applied as restrictions with Smart Scan. In addition, Bloom Filters can be applied as restrictions for simple JOINs, like those commonly found in Star Schemas (Dimensional Data Models). These operations can be observed in the query execution plan by the JOIN FILTER CREATE and JOIN FILTER USE row sources. What is very cool is that Bloom Filters can also pass their list of values to Storage Indexes to aid in further I/O reductions if there is natural clustering on those columns or it eliminates significant amounts of data (as in a highly selective set of values). Even if there isn't significant data elimination via Storage Indexes, a Smart Scan Bloom Filter can be applied post scan to prevent unneeded data being sent to the database servers.
Exadata Storage Indexes
Marc writes:
Storage indexes: Storage indexes reduce disk I/O volumes by tracking high and low values in memory for each 1-megabyte storage region. They can be used to give partition pruning benefits without requiring the partition key in the WHERE clause, as long as one of these columns is correlated with the partition key. For example, if a table has order_date and processed_date columns, is partitioned on order_date, and if orders are processed within 5 days of receipt, the storage server can track which processed_date values are included in each order partition, giving partition pruning for queries referring to either order_date or processed_date. Other data sets that are physically ordered on disk, such as incrementing keys, can also benefit.
In Marc's example he states there is correlation between the two columns PROCESSED_DATE and ORDER_DATE where PROCESSED_DATE = ORDER_DATE + [0..5 days]. That's fine and all, but to claim partition pruning takes place when specifying ORDER_DATE (the partition key column) or PROCESSED_DATE (non partition key column) in the WHERE clause because the Storage Index can be used for PROCESSED_DATE is inaccurate. The reality is, partition pruning can only take place when the partition key, ORDER_DATE, is specified, regardless if a Storage Index is used for PROCESSED_DATE.
Partition Pruning and Storage Indexes are completely independent of each other and Storage Indexes know absolutely nothing about partitions, even if the partition key column and another column have some type of correlation, as in Marc's example. The Storage Index simply will track which Storage Regions do or do not have rows that match the predicate filters and eliminate reading the unneeded Storage Regions.
Exadata Hybrid Columnar Compression
Marc writes:
Columnar compression: Hybrid columnar compression (HCC) introduces a new physical storage concept, the compression unit. By grouping many rows together in a compression unit, and by storing only unique values within each column, HCC provides storage savings in the range of 80 90% based on the compression level selected. Since data from full table scans remains compressed through I/O and buffer cache layers, disk savings translate to reduced I/O and buffer cache work as well. HCC does, however, introduce CPU and data modification overhead that will be discussed in the next section.
The Compression Unit (CU) for Exadata Hybrid Columnar Compression (EHCC) is actually a logical construct, not a physical storage concept. The compression gains from EHCC come from column-major organization of the rows contained in the CU and the encoding and transformations (compression) that can be done because of that organization (like values are more common within the same column across rows, vs different columns in the same row). To say EHCC only stores unique values within each column is inaccurate, however, the encoding and transformation algorithms use various techniques that yield very good compression by attempting to represent the column values with as few bytes as possible.
Data from EHCC full table scans only remains fully compressed if the table scan is not a Smart Scan, in which case the compressed CUs are passed directly up to the buffer cache and the decompression will then be done by the database servers. However, if the EHCC full table scan is a Smart Scan, then only the columns and rows being returned to the database nodes are decompressed by the Exadata servers, however, predicate evaluations can be performed directly on the EHCC compressed data.
Read more: Exadata Hybrid Columnar Compression Technical White Paper
Marc also writes:
Use columnar compression judiciously: Hybrid columnar compression (HCC) in Exadata has the dual advantages of reducing storage usage and reducing I/O for large reads by storing data more densely. However, HCC works only when data is inserted using bulk operations. If non-compatible operations like single-row inserts or updates are attempted, Exadata reverts transparently to the less restrictive OLTP compression method, losing the compression benefits of HCC. When performing data modifications such as updates or deletes, the entire compression unit must be uncompressed and written in OLTP-compressed form, involving an additional disk I/O penalty as well.
EHCC does require bulk direct path load operations to work. This is because the compression algorithms that are used for EHCC need sets of rows as input, not single rows. What is incorrect with Marc's comments is that when a row in a CU is modified (UPDATE or DELETE), the entire CU is not uncompressed and changed to non-EHCC compression, only the rows that are UPDATED are migrated to non-EHCC compression. For DELETEs no row migrations take place at all. This is easily demonstrated by tracking ROWIDs as in the example at the bottom of this post.
Exadata Smart Flash Cache
Marc writes:
Flash cache: Exadata s flash cache supplements the database servers buffer caches by providing a large cache of 384 GB per storage server and up to 5 TB in a full Oracle Exadata Database Machine, considerably larger than the capacity of memory caches. Unlike generic caches in traditional SAN storage, the flash cache understands database-level operations, preventing large non-repeated operations such as backups and large table scans from polluting the cache. Since flash storage is nonvolatile, it can cache synchronous writes, providing performance benefits to commit-intensive applications.
While flash (SSD) storage is indeed non-volatile, the Exadata Smart Flash Cache is volatile – it loses all of its contents if the Exadata server is power cycled. Also, since the Exadata Smart Flash is currently a write-through cache, it offers no direct performance advantages to commit-intensive applications, however, it does offer indirect performance advantages by servicing read requests that would otherwise be serviced by the HDDs, thus allowing the HDDs to service more write operations.
Read more: Exadata Smart Flash Cache Technical White Paper
EHCC UPDATE and DELETE Experiment
-- -- EHCC UPDATE example - only modified rows migrate -- SQL> create table order_items1 2 compress for query high 3 as 4 select rownum as rnum, x.* 5 from order_items x 6 where rownum <= 10000; Table created. SQL> create table order_items2 2 as 3 select rowid as rid, x.* 4 from order_items1 x; Table created. SQL> update order_items1 2 set quantity=10000 3 where rnum in (1,100,1000,10000); 4 rows updated. SQL> commit; Commit complete. SQL> select b.rnum, b.rid before_rowid, a.rowid after_rowid 2 from order_items1 a, order_items2 b 3 where a.rnum(+) = b.rnum 4 and (a.rowid != b.rid 5 or a.rowid is null) 6 order by b.rnum 7 ; RNUM BEFORE_ROWID AFTER_ROWID --------------- ------------------ ------------------ 1 AAAWSGAAAAAO1aTAAA AAAWSGAAAAAO1aeAAA 100 AAAWSGAAAAAO1aTABj AAAWSGAAAAAO1aeAAB 1000 AAAWSGAAAAAO1aTAPn AAAWSGAAAAAO1aeAAC 10000 AAAWSGAAAAAO1aXBEv AAAWSGAAAAAO1aeAAD -- -- EHCC DELETE example - no rows migrate -- SQL> create table order_items1 2 compress for query high 3 as 4 select rownum as rnum, x.* 5 from order_items x 6 where rownum <= 10000; Table created. SQL> create table order_items2 2 as 3 select rowid as rid, x.* 4 from order_items1 x; Table created. SQL> delete from order_items1 2 where rnum in (1,100,1000,10000); 4 rows deleted. SQL> commit; Commit complete. SQL> select b.rnum, b.rid before_rowid, a.rowid after_rowid 2 from order_items1 a, order_items2 b 3 where a.rnum(+) = b.rnum 4 and (a.rowid != b.rid 5 or a.rowid is null) 6 order by b.rnum 7 ; RNUM BEFORE_ROWID AFTER_ROWID --------------- ------------------ ------------------ 1 AAAWSIAAAAAO1aTAAA 100 AAAWSIAAAAAO1aTABj 1000 AAAWSIAAAAAO1aTAPn 10000 AAAWSIAAAAAO1aXBEv
Oracle Exadata Database Machine Offerings: X2-2 and X2-8
For those who followed or attended Oracle OpenWorld last week you may have seen the introduction of the new hardware for the Oracle Exadata Database Machine. Here's a high level summary of what was introduced:
- Updated Exadata Storage Server nodes (based on the Sun Fire X4270 M2)
- Updated 2 socket 12 core database nodes for the X2-2 (based on the Sun Fire X4170 M2)
- New offering of 8 socket 64 core database nodes using the Intel 7500 Series (Nehalem-EX) processors for the X2-8 (based on the Sun Fire X4800)
The major updates in the X2-2 compared to V2 database nodes are:
- CPUs updated from quad-core Intel 5500 Series (Nehalem-EP) processors to six-core Intel 5600 Series (Westmere-EP)
- Network updated from 1 GbE to 10 GbE
- RAM updated from 72 GB to 96 GB
The updates to the Exadata Storage Servers (which are identical for both the X2-2 and X2-8 configurations) are:
- CPUs updated to the six-core Intel 5600 Series (Westmere-EP) processors
- 600 GB 15k RPM SAS offering now known as HP (High Performance)
- 2 TB 7.2k RPM SAS offering now known as HC (High Capacity) [previously the 2 TB drives were 7.2k RPM SATA]
One of the big advantages of the CPU updates to the Intel 5600 Series (Westmere-EP) processors is that the Oracle Database Transparent Data Encryption can leverage the Intel Advanced Encryption Standard New Instructions (Intel AES-NI) found in the Intel Integrated Performance Primitives (Intel IPP). This "in silicon" functionality results in a 10x increase in encryption and an 8x increase in decryption using 256 bit keys per the Oracle Press release.
The differences (as I quickly see) between the X2-2 and the X2-8 offerings are:
- X2-8 only comes in full racks (of 2 database nodes)
- X2-8 has 2 TB of RAM per rack (compared to 768 GB for the X2-2)
- X2-8 has 16s/128c/256t per rack vs. 16s/96c/192t for the X2-2 (s=sockets, c=cores, t=threads)
One of the other Exadata related announcements was that Solaris x86 will be an option for the database OS in addition to Linux.
In summary, the Oracle Exadata Database Machine is riding the wave of Intel processors and is leveraging the Intel IPP functionality and will likely do so for the foreseeable future.
If you want all hardware details, check out the product data sheets:
- Oracle Exadata Database Machine X2-8
- Oracle Exadata Database Machine X2-2
- Oracle Exadata Storage Servers
I just noticed that Alex Gorbachev has a nice table format of the hardware components for your viewing pleasure as well.
Oracle Exadata and Netezza TwinFin Compared – An Engineer's Analysis
There seems to be little debate that Oracle's launch of the Oracle Exadata Storage Server and the Sun Oracle Database Machine has created buzz in the database marketplace. Apparently there is so much buzz and excitement around these products that two competing vendors, Teradata and Netezza, have both authored publications that contain a significant amount of discussion about the Oracle Database with Real Application Clusters (RAC) and Oracle Exadata. Both of these vendor papers are well structured but make no mistake, these are marketing publications written with the intent to be critical of Exadata and discuss how their product is potentially better. Hence, both of these papers are obviously biased to support their purpose.
My intent with this blog post is simply to discuss some of the claims, analyze them for factual accuracy, and briefly comment on them. After all, Netezza clearly states in their publication:
The information shared in this paper is made available in the spirit of openness. Any inaccuracies result from our mistakes, not an intent to mislead.
In the interest of full disclosure, my employer is Oracle Corporation, however, this is a personal blog and what I write here are my own ideas and words (see disclaimer on the right column). For those of you who don't know, I'm a database performance engineer with the Real-World Performance Group which is part of Server Technologies. I've been working with Exadata since before it was launched publicly and have worked on dozens of data warehouse proofs-of-concept (PoCs) running on the Exadata powered Sun Oracle Database Machine. My thoughts and comments are presented purely from an engineer's standpoint.
The following writings are the basis of my discussion:
- Teradata: Exadata – the Sequel: Exadata V2 is Still Oracle
- Daniel Abadi: Defending Oracle Exadata
- Netezza: Oracle Exadata and Netezza TwinFin™ Compared
If you have not read Daniel Abadi's blog post I strongly suggest you do before proceeding further. I think it is very well written and is presented from a vendor neutral point of view so there is no marketing gobbledygook to sort through. Several of the points in the Teradata writing which he discusses are also presented (or similarly presented) in the Netezza eBook, so you can relate his responses to those arguments as well. Since I feel Daniel Abadi did an excellent job pointing out the major flaws with the Teradata paper, I'm going to limit my discuss to the Netezza eBook.
Understanding Exadata Smart Scan
As a prerequisite for the discussion of the Netezza and Teradata papers, it's imperative that we take a minute to understand the basics of Exadata Smart Scan. The Smart Scan optimizations include the following:
- Data Elimination via Storage Indexes
- Restriction/Row Filtering/Predicate Filtering
- Projection/Column Filtering
- Join Processing/Filtering via Bloom Filters and Bloom Pruning
The premise of these optimizations is reduce query processing times in the following ways:
- I/O Elimination – don't read data off storage that is not needed
- Payload Reduction – don't send data to the Oracle Database Servers that is not needed
OK. Now that you have a basic understanding, let's dive into the claims…
Netezza's Claims
Let's discuss a few of Netezza claims against Exadata:
Claim: Exadata Smart Scan does not work with index-organized tables or clustered tables.
While this is a true statement, its intent is clearly to mislead you. Both of these structures are designed for OLTP workloads, not data warehousing. In fact, if one were to actually read the Oracle Database 11.2 documentation for index-organized tables you would find the following (source):
Index-organized tables are ideal for OLTP applications, which require fast primary key access
If one were to research table clusters you would find the Oracle Database 11.2 documentation offers the following guidelines (source):
Typically, clustering tables is not appropriate in the following situations:
- The tables are frequently updated.
- The tables frequently require a full table scan.
- The tables require truncating.
As anyone can see from the Oracle Database 11.2 Documentation, neither of these structures are appropriate for data warehousing.
Apparently this was not what Netezza really wanted you to know so they uncovered a note on IOTs from almost a decade ago, dating back to 2001 – Oracle 9i time frame, that while it clearly states:
[an IOT] enables extremely fast access to table data for primary key based [OLTP] queries
it also suggests that an IOT may be used as a fact table. Clearly this information is quite old and outdated and should probably be removed. What was a recommendation for Oracle Database 9i Release 1 in 2001 is not necessarily a recommendation for Oracle Database 11g Release 2 in 2010. Technology changes so using the most recent recommendations as a basis for discussion is appropriate, not some old, outdated stuff from nearly 10 years ago. Besides, the Oracle Database Machine runs version 11g Release 2, not 9i Release 1.
Bottom line: I'd say this "limitation" has an impact on a nice round number of Exadata data warehouse customers – exactly zero (zero literally being a round number). IOTs and clustered tables are both structures optimized for fast primary key access, like the type of access in OLTP workloads, not data warehousing. The argument that Smart Scan does not work for these structures is really no argument at all.
Claim: Exadata Smart Scan does not work with the TIMESTAMP datatype.
Phil Francisco seems to have left out some very important context in making this accusation, because this is not at all what the cited blog post by Christian Antognini discusses. This post clearly states the discussion is about:
What happens [with Smart Scan] when predicates contain functions or expressions?
Nowhere at all does that post make an isolated reference that Smart Scan does not work with the TIMESTAMP datatype. What this blog post does state is this:
when a TIMESTAMP datatype is involved [with datetime functions], offloading almost never happens
While the Netezza paper references what the blog post author has written, some very important context has been omitted. In doing so, Netezza has taken a specific reference and turned it into a misleading generalization.
The reality is that Smart Scan does indeed work for the TIMESTAMP datatype and here is a basic example to demonstrate such:
SQL> describe t
Name Null? Type
-------------- -------- ------------------
SQL> SELECT * FROM t WHERE t = to_timestamp('01-01-2010','DD-MM-YYYY');
Execution Plan
Plan hash value: 1601196873
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
Predicate Information (identified by operation id):
1 - storage("T"=TIMESTAMP' 2010-01-01 00:00:00.000000000')
filter("T"=TIMESTAMP' 2010-01-01 00:00:00.000000000')
You can see that the Smart Scan offload is taking place by the presence of the storage clause (highlighted) in the Predicate Information section above. What Christian Antognini did observe is bug 9682721 and the bugfix resolves the datetime function offload issues for all but a couple scenarios (which he blogs about here) and those operations can (and usually are) expressed differently. For example, an expression using ADD_MONTHS() can easily be expressed using BETWEEN.
Bottom line: Exadata Smart Scan does work with the TIMESTAMP datatype.
Claim: When transactions (insert, update, delete) are operating against the data warehouse concurrent with query activity, smart scans are disabled. Dirty buffers turn off smart scan.
Yet again, Netezza presents only a half-truth. While it is true that an active transaction disables Smart Scan, they fail to further clarify that Smart Scan is only disabled for those blocks that contain an active transaction – the rest of the blocks are able to be Smart Scanned. The amount of data that is impacted by insert, update, delete will generally be a very small fraction of the total data in a data warehouse. Also, data that is inserted via direct path operations is not subject to MVCC (the method Oracle uses for read consistency) as the blocks that are used are new blocks so no read consistent view is needed.
Bottom line: While this claim is partially true, it clearly attempts to overstate the impact of this scenario in a very negative way. Not having Smart Scan for small number of blocks will have a negligible impact on performance.
Also see Daniel Abadi: Exadata does NOT Support Active Data Warehousing
Claim: Using [a shared-disk] architecture for a data warehouse platform raises concern that contention for the shared resource imposes limits on the amount of data the database can process and the number of queries it can run concurrently.
It is unclear what resource Netezza is referring to here, it simply states "the shared resource". You know the one? Yes, that one… Perhaps they mean the disks themselves, but that is completely unknown. Anyway…
Exadata uses at least a 4 MB Automatic Storage Management (ASM) allocation unit (AU) [more on ASM basics]. This means that there is at least 4 MB of contiguous physical data laid out on the HDD which translates into 4 MB of contiguous data streamed off of disk for full table scans before the head needs to perform a seek. With such large I/O requests the HDDs are able to spend nearly all the time transferring data, and very little time finding it and that is what matters most. Clearly if Exadata is able to stream data off of disk at 125 MB/s per disk (near physics speed for this type of workload) then any alleged "contention" is really not an issue. In many multi-user data warehouse workloads for PoCs, I've observed that each Exadata Storage Server is able to perform very close or at the data sheet physical HDD I/O rate of 1500 MB/s per server.
Bottom line: The scalability differences between shared-nothing and shared-disk are very much exaggerated. By doing large sequential I/Os the disk spends its time returning data, not finding it. Simply put – there really is no "contention".
Also see Daniel Abadi: 1) Exadata does NOT Enable High Concurrency & 2) Exadata is NOT Intelligent Storage; Exadata is NOT Shared-Nothing
Claim: Analytical queries, such as "find all shopping baskets sold last month in Washington State, Oregon and California containing product X with product Y and with a total value more than $35" must retrieve much larger data sets, all of which must be moved from storage to database.
I find it so ironic that Netezza mentions this type of query as nearly an identical (but more complex) one was used by my group at Oracle OpenWorld 2009 in our The Terabyte Hour with the Real-World Performance Group session. The exact analytical query we ran live for the audience to demonstrate the features of Oracle Exadata and the Oracle Database Machine was, "What were the most popular items in the baskets of shoppers who visited stores in California in the first week of May and did not buy bananas?"
Let's translate the Netezza analytical question into some tables and SQL to see what the general shape of this query may look like:
select count(*) -- using count(*) for simplicity of the example from ( select td.transaction_id, sum(td.sales_dollar_amt) total_sales_amt, sum(case when p.product_description in ('Brand #42 beer') then 1 else 0 end) count_productX, sum(case when p.product_description in ('Brand #42 frozen pizza') then 1 else 0 end) count_productY from transaction_detail td join d_store s on (td.store_key = s.store_key) join d_product p on (td.product_key = p.product_key) where s.store_state in ('CA','OR','WA') and td.transaction_time >= timestamp '2010-07-01 00:00:00' and td.transaction_time < timestamp '2010-08-01 00:00:00' group by td.transaction_id ) x where total_sales_amt > 35 and count_productX > 0 and count_productY > 0
To me, this isn't a particularly complex analytical question/query. As written, it's just a 3 table join (could be 4 if I added a D_DATE I suppose), but it doesn't require anything fancy – just a simple GROUP BY with a CASE in the SELECT to count how many times Product X and Product Y appear in a given basket.
Netezza claims that analytical queries like this one must move all the data from storage to the database, but that simply is not true. Here is why:
- Simple range partitioning on the event timestamp (a very common data warehousing practice for those databases that support partitioning), or even Exadata Storage Indexes, will eliminate any I/O for data other than the one month window that is required for this query.
- A bloom filter can be created and pushed into Exadata to be used as a storage filter for the list of STORE_KEY values that represent the three state store restriction.
Applying both of #1 and #2, the only data that is returned to the database for the fact table are rows for stores in Washington State, Oregon and California for last month. Clearly this is only a subset of the data for the entire fact table.
This is just one example, but there are obviously different representations of the same data and query that could be used. I chose what I thought was the most raw, unprocessed, uncooked form simply because Netezza seems to boast about brute force type of operations. Even then, considering a worst case scenario, Exadata does not have to move all the data back to the database. Other data/table designs that I've seen from customers in the retail business would allow even less data to be returned.
Bottom line: There are numerous ways that Exadata can restrict the data that is set to the database servers and it's likely that any query with any predicate restrictions can do so. Certainly it is possible even with the analytic question that Netezza mentions.
Claim: To evenly distribute data across Exadata's grid of storage servers requires administrators trained and experienced in designing, managing and maintaining complex partitions, files, tablespaces, indices, tables and block/extent sizes.
Interestingly enough, the author of the Teradata paper seems to have a better grasp than Netezza on how data distribution and ASM work describing it on page 9:
Distribution of data on Exadata storage is managed by Oracle's Automatic Storage Manager (ASM). By default, ASM stripes each Oracle data partition across all available disks on every Exadata cell.
So if by default ASM evenly stripes data across all available disks on Exadata Storage Server (and it does, in a round robin manner) what exactly is so difficult here? What training and experience is really required for something that does data distribution automatically? I can only assert that Phil Francisco has not even read the Teradata paper (but it would seem he has since he even mentions it on his blog), let alone Introduction to Oracle Automatic Storage Management. It's claims like this that really make me question how genuine his "no intent to mislead" statement really is.
Bottom line: Administrators need not worry about data distribution with Exadata and ASM – it is done automatically and evenly for you.
I'm always extremely reluctant to believe much of what vendors say about other vendors, especially when they preface their publication with something like: "One caveat: Netezza has no direct access to an Exadata machine", and "Any inaccuracies result from our mistakes, not an intent to mislead" yet they still feel qualified enough to write about said technology and claim it as fact. I also find it interesting that both Teradata and Netezza have published anti-Exadata papers, but neither Netezza nor Teradata have published anti-vendor papers about each other (that I know of). Perhaps Exadata is much more of a competitor than either of them let on. They do protest too much, methinks.
The list of claims I've discussed certainly is not an exhaustive list by any means but I think it is fairly representative of the quality found in Netezza's paper. While sometimes the facts are correct, the arguments are overstated and misleading. Other times, the facts are simply wrong. Netezza clearly attempts to create the illusion of problems simply where they do not exist.
Hopefully this blog post has left you a more knowledgeable person when it comes to Oracle and Exadata. I've provided fact and example wherever possible and kept assertions to a minimum.
I'd like to end with a quote from Daniel Abadi's response to the Teradata paper which I find more than applicable to the Netezza paper as well:
Many of the claims and inferences made in the paper about Exadata are overstated, and the reader needs to be careful not to be mislead into believing in the existence problems that don't actually present themselves on realistic data sets and workloads.
Courteous and professional comments are welcome. Anonymous comments are discouraged. Snark and flame will end up in the recycle bin. Comment accordingly.
The Core Performance Fundamentals Of Oracle Data Warehousing – Set Processing vs Row Processing
[back to Introduction]
In over six years of doing data warehouse POCs and benchmarks for clients there is one area that I frequently see as problematic: "batch jobs". Most of the time these "batch jobs" take the form of some PL/SQL procedures and packages that generally perform some data load, transformation, processing or something similar. The reason these are so problematic is that developers have hard-coded "slow" into them. I'm generally certain these developers didn't know they had done this when they coded their PL/SQL, but none the less it happened.
So How Did "Slow" Get Hard-Coded Into My PL/SQL?
Generally "slow" gets hard-coded into PL/SQL because the PL/SQL developer(s) took the business requirements and did a "literal translation" of each rule/requirement one at a time instead of looking at the "before picture" and the "after picture" and determining the most efficient way to make those data changes. Many times this can surface as cursor based row-by-row processing, but it also can appear as PL/SQL just running a series of often poorly thought out SQL commands.
Hard-Coded Slow Case Study
The following is based on a true story. Only the
facts names have been changed to protect the innocent.
Here is a pseudo code snippet based on a portion of some data processing I saw in a POC:
{truncate all intermediate tables} insert into temp1 select * from t1 where create_date = yesterday; insert into temp1 select * from t2 where create_date = yesterday; insert into temp1 select * from t3 where create_date = yesterday; insert into temp2 select * from temp1 where {some conditions}; insert into target_table select * from temp2; for each of 20 columns loop update target_table t set t.column_name = (select column_name from t4 where t.id=t4.id ) where i.column_name is null; end loop update target_table t set {list of 50 columns} = select {50 columns} from t5 where t.id=t5.id;
I'm going to stop there as any more of this will likely make you cry more than you already should be.
I almost hesitate to ask the question, but isn't it quite obvious what is broken about this processing? Here's the major inefficiencies as I see them:
- What is the point of inserting all the data into temp1, only then to filter some of it out when temp2 is populated. If you haven't heard the phrase "filter early" you have some homework to do.
- Why publish into the target_table and then perform 20 single column updates, followed by a single 50 column update? Better question yet: Why perform any bulk updates at all? Bulk updates (and deletes) are simply evil – avoid them at all costs.
So, as with many clients that come in and do an Exadata Database Machine POC, they really weren't motivated to make any changes to their existing code, they just wanted to see how much performance the Exadata platform would give them. Much to their happiness, this reduced their processing time from over 2.5 days (weekend job that started Friday PM but didn't finish by Monday AM) down to 10 hours, a savings of over 2 days (24 hours). Now, it could fail and they would have time to re-run it before the business opened on Monday morning. Heck, I guess if I got back 24 hours out of 38 I'd be excited too, if I were not a database performance engineer who knew there was even more performance left on the table, waiting to be exploited.
Feeling unsatisfied, I took it upon myself to demonstrate the significant payback that re-engineering can yield on the Exadata platform and I coded up an entirely new set-based data flow in just a handful of SQL statements (no PL/SQL). The result: processing an entire week's worth of data (several 100s of millions of rows) now took just 12 minutes. That's right — 7 days worth of events scrubbed, transformed, enriched and published in just 12 minutes.
When I gently broke the news to this client that it was possible to load the week's events in just 12 minutes they were quite excited (to say the least). In fact, one person even said (a bit out of turn), "well, that would mean that a single day's events could be loaded in just a couple minutes and that would give a new level of freshness to the data which would allow the business to make faster, better decisions due to the timeliness of the data." My response: "BINGO!" This client now had the epiphany of what is now possible with Exadata where previously it was impossible.
It's Not a Need, It's a Want
I'm not going to give away my database engineer hat for a product marketing hat just yet (or probably ever), but this is the reality that exists. IT shops started with small data sets and use small data set programming logic on their data, and that worked for some time. The reason: because inefficient processing on a small data set is only a little inefficient, but the same processing logic on a big data set is very inefficient. This is why I have said before: In oder to fully exploit the Oracle Exadata platform (or any current day platform) some re-engineering may be required. Do not be mistaken — I am not saying you need to re-engineer your applications for Exadata. I am saying you will want to re-engineer your applications for Exadata as those applications simply were not designed to leverage the massively parallel processing that Exadata allows one to do. It's time to base design decisions based on today's technology, not what was available when your application was designed. Fast forward to now.


text:
The Internet has quickly evolved during the past 20 years from being a luxury to a necessity, with everything from banking to babysitter selection now done with a few clicks. But all that easy access to virtually limitless information could be in for a seismic change as the Federal Communications Commission debates how much it wants to wade into the regulatory waters.
At issue is the doctrine of net neutrality. Net neutrality essentially ensures that any given Internet service provider, or ISP, cannot block or interfere in any way with the information that is transmitted over its network. Many ISPs, including Verizon, Comcast and AT&T, would like to see net neutrality modified. In fact, these providers, along with the National Cable and Telecommunications Association, have spent in excess of $64 million in 2012 lobbying against it, according to the Sunlight Foundation, which monitors the work of lobbying organizations.
Although it appears unlikely that the FCC will allow ISPs to interfere with the flow of content, federal regulators might allow them to sell faster processing capability to content companies that are willing to pay for it. For example, if the rules were to change, an ISP could begin charging companies that require faster processing, such as movie-streaming services. Although consumers would not necessarily be charged any more by ISPs, the companies that provide content, such as Netflix or Amazon, would undoubtedly look for ways to recoup their new expenses, with consumers likely paying more for content, if not for delivery.
There are also Internet purists who believe that a two-tiered system of online speed fundamentally violates one of the basic principles of the Internet: equality for all. Another argument holds that ISPs could make life more difficult for actual or potential competitors.
Although most of the major ISPs have vowed that they will never block any content, some net neutrality advocates remain skeptical.
Even members of the FCC appear divided on the issue. Of the five commissioners on the panel, the three Democrats approved a plan opening the discussion to public commentary. The remaining commissioners, both Republicans, dissented, saying that the FCC should stay out of this discussion and leave the matter up to Congress to regulate.
Both approaches face potential pitfalls. The majority's gambit would likely make the process more visible, but that does not necessarily mean any fairer. The Republican strategy seems equally fraught with problems; once legislation moves behind closed doors, it could turn into a lobbyist's field day. Members of Congress may have varying degrees of knowledge about how the Internet works, but there are few senators or representatives who fail to grasp the power of cash.
The Internet has well served the business world and individuals during the past few decades. Yes, it has its dark corners and less savory aspects, but it has proven itself as an invaluable tool for progress as well. Any move toward a system where wealthy companies essentially have easier access and control of information seems inherently unfair.
In truth, the system has always been unbalanced. Companies with the wherewithal can ensure that their names appear at the top of any related search through Google, just as certain companies can create comprehensive online marketing campaigns using a combination of email, social media and viral videos.
And yet, many of these developments and innovations came about naturally, as free market capitalism continued to expand and exploit existing freedom on the Internet. In the case of net neutrality, the FCC appears poised to aid and abet the fortunes of some of the largest communication companies in the world, essentially endorsing the creation of first- and second-class accommodations when it comes to Internet traffic.
Rather than fomenting a divided Internet, the FCC should instead be standing by consumers, ensuring that in at least one way, the Internet remains neutral.
Location, ST | website.com

subset: commoncrawl


text:
The Grapes Inn3
Scaling Dam, Easington, Scaling Dam, TS13 4TP
Price FromView Rooms
- Very Good
- 11 guest reviews
"I found the Grapes Inn to be comfortable and the owners friendly. The Grapes inn is in a lovely peaceful location with a lovely view. I woul..."
"For our stay perfect location with friendly staff food good portions well presented..."
"Mark was a lovely guy and made sure we had everything we needed. The food was fantastic and they have bonus points from me for the fact that..."
"Friendly clean and comfortable, lovely breakfast. Mark the landlord was helpful and friendly , nice place for touring...."
"Very friendly and helpful staff. Rooms were clean and simple. We had a family room which is spacious enough for 4. We went to a wedding up t..."
About the hotel
A comfortable and welcoming traditional country Inn set in the North Yorkshire Moors. With our location being just 10 miles from the historic fishing port of Whitby we are well positioned to explore the moors and 'Heartbeat' country as well as the Cleveland Way and coast.
Room Availability
Showing Wednesday 2 December 2015 to Thursday 3 December 2015Change Search
Facilities Show/hide
- Parking
- Car parking available onsite
- Dining
- Restaurant , Bar , Breakfast to go
- In room facilities
- Tea/Coffee making facilities in each room
- Family friendly
- Children's Facilities - Outdoor
- Leisure
- Horse Riding
Can't find the facilities you're looking for?
- Call us on 0333 300 1007 and we'll help find the perfect hotel for you
- Call
Useful Info Show/hide
- From 17:00 hours
- Until 10:30 hours
- The price shown is based on the occupancy and dates you have searched for. Prices and availability are subject to change based on the arrival date, number of nights and occupancy searched for. All prices are inclusive of all local taxes.
- Pets are not allowed
Customer Reviews
Our reviews are from genuine LateRooms.com customers. We verify that the customer has stayed before leaving a review.
Very Good
82% would recommend it to a friend
Customer service
Food quality
Room quality
Lovely place
I found the Grapes Inn to be comfortable and the owners friendly.
The Grapes inn is in a lovely peaceful location with a lovely view. I would stay here again.
Read More Less
For our stay perfect location with friendly staff food good portions well presented


text:
Live updates
- ITV Report
Norfolk welcomes Queen's Dragoon Guards
Medic wins Military Cross for bravery
A Norfolk-based serviceman as been awarded the Military Cross for his role in saving the lives of four wounded soldiers,
Lance Corporal Stephen Shaw, based with the Royal Medical Corps in Swanton Morely was commended for his actions after his patrol came under fire in Helmand Province, last July.
They were attacked by a sniper, four soldiers were seriously wounded. L/Cpl Shaw, rescued one of his colleagues and gave life-saving medical treatment to the casualties.
Jonathan Wills spoke to him today.
- ITV Report
Light Dragoons return home from Afghanistan
Norfolk-based soldier killed in Afghanistan named
A Norfolk-based soldier killed in Afghanistan by a roadside bomb has been named as Sergeant Lee Paul Davidson of The Light Dragoons.
The 32-year-old from Doncaster, South Yorkshire, died on Sunday after his vehicle was hit while patrolling with the Afghan Uniformed Police in the Nahr-e Saraj district.
The Light Dragoons are based at Swanton Morley in Norfolk.
Sgt Davidson leaves a wife, who is expected a baby daughter, and two sons.


text:
By On the occasion of our 20th anniversary
Why Do Mexis Like Wrought-Iron Fences?
By Gustavo Arellano
By R. Scott Moxley
By Alfonso Delgado
Women For: OC are Stitching and Bitching
By Courtney Hamilton
Santa Ana River Ready for Its Renaissance
By Joel Beers
By Peter Maguire
Can Tech Ruin the Kim Pham Murder Trial?
By Charles Lam
BUH-BYE! The county Board of Supervisors on June 27 fired county executive officer Jan Mittermeier, then cut her a $170,000 severance check on her way out the door —which sources tell us did not hit her on the ass. We suspect the 25-year county employee will land in the private sector, where shadowy, inflexible, autocratic management styles are all the rage these days. In other departures, controversial trustee Steven J. Frogue resigned from the South Orange County Community College District board effective June 30. Frogue sparked still-smoldering bitterness when he tried to hold a John F. Kennedy assassination seminar in 1997 that featured two speakers who wrote for an anti-Semitic newspaper and theorized that Israeli intelligence offed JFK. His stepping down fills us with perverse sadness. We looked forward to hearing "Springtime for Hitler" blaring over South County loudspeakers this coming election season. Auf wieder sehen, Steve.CONTINUED...
Sponsor Content


text:
2009 Resolutions
Mitt Romney wants to stop sleeping in his suit. Martha Stewart will try to save publishing. And Wolfgang Puck wants no minimum wage for waiters.
Many people at year's end reread that most influential man, Charles Dickens. Not only "A Christmas Carol," but also his sweeping books of the human spirit like "Great Expectations" and, especially, "Bleak House." Dickens's damp, soot-fogged London has the feel of America in deep winter, but he inflects that cold city with his bright, generous humanity, a reassuring combination that encourages our annual desire to greet January renewed. Dickens's best characters suggest that the story of life is in the striving, the ongoing effort to begin again and become, even for a moment, our very best selves. So that some day, after we are gone, someone may say, as Jo the homeless crossing sweep does of the dead opium addict Nemo, "He wos wery good to me, he wos." Nemo means "nobody" in Latin and that tragic man exists to tell us we can't do it alone.
The first of the year is that rare, earnest moment when we are all thinking this way together, resolving to be benevolent, compassionate, motivated, adventuresome. For the New Year, The Wall Street Journal asked some influential people three questions: What professional project do you plan to complete in 2009? What personal resolution do you finally hope to keep next year? And what problem should your industry or professional community tackle more effectively?
Join a Discussion
What are your New Year's resolutions? Share your thoughts in Journal Community.
Making Resolutions Rewarding
Psychologist John Norcross gives advice on setting reasonable goals -- and sticking to them. Read the Q&A with Mr. Norcross.
Think of our respondents for a moment as a cheerful cast of Dickensians, men and women who exist here to help us better know ourselves.
Philippe Petit, 59
The wirewalker in the documentary "Man on Wire," New York City
PROFESSIONAL: Street juggling, high-wire walking, giving lectures and workshops, a new book.
PERSONAL: My personal resolutions do not get triggered by a date printed on the calendar. In the middle of the year, in the middle of the night, a new dream, a new idée fixe takes hold of me.
INDUSTRY: Industry...professional community? I live in the clouds, among birds. My problems are the treacherous wind currents, and ah yes, on the ground, the men and women who have been trained to say no to miracles. So let's include in next year's school curriculum classes in daydreaming, in following your intuition, in romantic improvisations.
Justin Tuck, 25
New York Giants defensive end, New Jersey
PROFESSIONAL: I would like to further my RUSH for Literacy program, which assists children in New York and Alabama; repeat as Super Bowl champs; play in the Pro Bowl and be a team captain for next season.
PERSONAL: I need to get more sleep. My wife would agree that lack of sleep can leave me cranky and unproductive.
INDUSTRY: I'm most concerned about the lack of holding penalties called against opposing offensive linemen as they go up against a certain New York Giant wearing No. 91.
Mitt Romney, 61
Former governor of Massachusetts and 2008 Republican presidential candidate, Belmont, Mass.
PROFESSIONAL: I want to help restore balance in Washington.... Our democracy needs two strong parties if we are to deal effectively with the challenges our nation faces.
PERSONAL: Stop wearing a suit and tie to bed.
INDUSTRY: Government needs to take immediate action to boost the economy, without sacrificing our long-term growth. This means lowering taxes for individuals and businesses and investment in infrastructure, military modernization and energy research.
Frank Wilczek, 57
Nobel Prize-winning particle physicist and author, Cambridge, Mass.
PROFESSIONAL: I'm writing a physics murder mystery. The idea is that two men and two women from Harvard and MIT collaborate and discover dark matter. It's clear that they should win a Nobel Prize, but according to the rules of the prize, only three people at most can share.
PERSONAL: I'm looking forward to getting better acquainted with my Wii. I've been mucking around with Wii Sports. I like the boxing and bowling and tennis.
INDUSTRY: We have the LHC [Large Hadron Collider, a subatomic particle collider in Switzerland], which will explore the frontier of high-energy physics. There's a whole well-known agenda featuring unification, supersymmetry, looking for the Higgs particle, perhaps shedding light on the question of dark matter. The most important work in the area is yet to be done.
Adele, 20
Singer, London
PROFESSIONAL: My main priority is the second record. I'm really looking forward to getting right back into the thick of recording again.
PERSONAL: Keep enjoying everything and avoid crumbling when stuff goes pear-shaped. Get a better balance between my career and personal life.
INDUSTRY: Stop treating female singers' music as a gender and more like a genre.
Rich Ross, 47
President of Disney Channels Worldwide, Los Angeles
PROFESSIONAL: A big project is to give Kevin, Joe and Nick Jonas (also known as the Jonas Brothers) and Demi Lovato a great launch of their Disney Channel series, "Jonas" and "Sonny With a Chance" respectively.
PERSONAL: In 2009, I hope to get to the point where I can do a headstand in yoga without having to use the wall to hold me up. (And yes, that's also metaphorical for many other unspecified resolutions!)
INDUSTRY: To maximize full potential, the kids' media business has the challenge to reach boys more effectively. According to the boys and parents we've talked to, there is a shortage of programming that connects with them, challenges them and inspires them on their journey.
Wolfgang Puck, 59
Chef/restaurateur, Los Angeles
PROFESSIONAL: We'll open our newest restaurant, called Five-Sixty, which is at the Reunion Tower in Dallas, and Union Station, where we can do catering for up to 1,000 guests.
PERSONAL: Hopefully, next year I will spend more weekends with my family instead of working. It seems like every year I make the same resolution but since I'm turning 60 next year, I think that will be a good place to start.
INDUSTRY: The industry should lobby to establish a federal law that allows restaurants nationwide to exempt waiters who earn tips from the minimum wage. Waiters would not lose money because they could work overtime, and we could use the savings to provide health care or raise the wages of dishwashers.
George Fellows, 66
CEO of Callaway Golf Co., Rancho Santa Fe, Calif.
PROFESSIONAL: We've embarked on an effort to make the company a global one. Over 50% of our revenues are outside of the U.S., but we haven't begun to tap the global markets.
PERSONAL: I'm going to try to break [a golf score of] 90. It's reachable. I keep thinking I'm going to make it, but I never do. I haven't had enough time to play.
INDUSTRY: One of the sport's major initiatives is to try to get golf back in the Olympics. The last time it was in the Games was 1904. The earliest we could do it would be 2016, but it could be a transformational thing.
Martha Stewart, 67
Magazine publisher, television personality, entrepreneur, New York City
PROFESSIONAL: I have been working on a new entertaining book/lifestyle journal and hope to complete it in 2009 for publication in 2010. I have an encyclopedia of crafts being published in March and a new cookbook being published in the latter part of the year.
PERSONAL: I rarely make New Year's resolutions because I believe in constant evolution and change. Resolutions and contracts are very similar -- they need constant tweaking and editing to work really effectively.
INDUSTRY: The Internet has taken many readers from newspapers and magazines. We as publishers have to individually determine how our publications can make the leap from print to digital so that we can retain both the computer user and the more traditional reader.
Vikram Chandra, 47
Author, Mumbai and Berkeley, Calif.
PROFESSIONAL: I just started a new novel a couple of months ago, and in a magical, perfect world I'd finish it in 2009. But my last novel came in at 900 pages, so I'll settle for slow, steady progress.
PERSONAL: I'm the father of a 7-month-old baby, so I think it's time for me to get done with my driving lessons and face the terrors of the DMV.
INDUSTRY: I'd love the publishing industry the world over to accept fully and without further complaint that electronic publishing is here to stay, and to provide innovative, sophisticated and, above all, low-priced competition for the Kindle and Sony Reader.
Dan Barber, 39
Chef, farming advocate and owner of Blue Hill Farm, New York City
PROFESSIONAL: My brother David and I have stepped up Blue Hill Farm into a full-time dairy operation, and we're in the process of doubling the herd with the hope that we'll have grass-fed cheese by summer of 2009.
PERSONAL: I'd like to sit more often. And write a book. And also eat less sugar late into the night. (They are all related.)
INDUSTRY: We've all seen the success of the local foods movement.... It's time for chefs to think not just about how to feed our customers sustainably, but how to feed the nation sustainably. For that we need to address institutional changes, like food banks and school lunch programs.
Nico Muhly, 27
Composer, New York City
PROFESSIONAL: In 2009, I am going to finally finish a long series of short works for solo viola and tape [recorded elements]. I'm into this because it's weird and specialized, like taking a year of your life and learning how to make blood sausage.
PERSONAL: I am going to wake up much, much earlier.
INDUSTRY: It would be great to see cities take more pride in their orchestras and small ensembles.
Richard Meier, 74
Architect, New York City
PROFESSIONAL: A 15-story apartment building with 102 condos on Prospect Park in Brooklyn.
PERSONAL: To lose weight by walking to work from now on.
INDUSTRY: The creation of sustainable urban open public space. The state and federal governments should take an active role in protecting the open spaces we have now.
Donna Brazile, 49
Democratic strategist, author and professor, Washington, D.C.
PROFESSIONAL: My major goal, outside of teaching weekly at Georgetown University, is to complete my second book, "Post Racial Politics in the Age of Obama: Reconciliation or Recrimination."
PERSONAL: Cook more, but eat less of what I stir up for others.
INDUSTRY: It's time someone limits some pundits' ability to bloviate on issues they know nothing about. For example, a political strategist is not someone who just voted in the last election or wrote a BlogSpot. It's someone who worked on a major campaign and played a decision-making role.
Alain de Botton, 39
Author, London
PROFESSIONAL: In June, I'll be publishing a new book, "The Pleasures and Sorrows of Work," an essay on the highs and lows of the workplace, through the eyes of 10 different companies.
PERSONAL: This coming summer I very much hope to take my family on a driving holiday from Seattle to San Francisco.
INDUSTRY: Academics and university professors in the humanities should remember to produce work that can be understood by, and have an effect on, people outside of a narrow elite circle. High culture shouldn't cease to engage with mass culture but, in a fracturing world, should help to foster widespread compassion, wisdom and consolation.
Philippe Starck, 59
Designer, southwest France
PROFESSIONAL: Produce a series of products using a new concept called Democratic Ecology: affordable, easy-to-use, easy-to-find and easy-to-install products that produce energy with wind, sun and hydrogen. They include a miniature roof-top windmill that will produce up to 80% of a home's energy and is made of transparent plastic (so it isn't ugly).
PERSONAL: To continue to be happy that the U.S. has a new president.
INDUSTRY: Design must come back to its assets of using vision, high technology, engineering and politics to help people have the right evolution.
Mike Huckabee, 53
Former governor of Arkansas and 2008 Republican presidential candidate, North Little Rock, Ark.
PROFESSIONAL: The launch of my new radio commentary on ABC Radio, and I hope to work on another book.
PERSONAL: To train for another marathon.
INDUSTRY: Create more civil discussion on TV talk shows.
Lt. Gen. Jack Rives, 56
Air Force Judge Advocate General, Alexandria, Va.
PROFESSIONAL: Our legal school is on its way to becoming a top-tier focal point for JAG Corps education, research, thinking and outreach. Next year I hope to wrap all of its potential into a manpower, curriculum and building plan that sets them up for the next decade.
PERSONAL: Keep my level of computer savvy close to that of my 4- and 8-year-old grandsons. I suspect that 2009 will be the last year this will be a realistic goal.
INDUSTRY: The way we handle detainee treatment issues. I'm confident that the various agencies, military and civilian, will work together on an approach that satisfies the need for security consistent with the rule of law.
Katy Perry, 24
Singer, Los Angeles
PROFESSIONAL: I would like to sell out every date on "Hello Katy," my first world tour.
PERSONAL: Stop reading my own press.
INDUSTRY: Find a way to allow artists to find a middle ground with the paparazzi, allowing people to have a public life and a private life.
Marcel Dzama, 34
Artist, Brooklyn, N.Y.
PROFESSIONAL: I have recently made sketches for a piece in Mexico that will be permanently installed in a botanical garden in Culiacán. It will be a snowman kept alive by solar panels.
PERSONAL: Read more and plant a garden in my backyard.
INDUSTRY: I think the art world should make sure that charities are not being left behind with the economic downturn. I am worried that many are in danger, with less charitable contributions.
Janette Sadik Khan, 48
New York City commissioner of transportation, New York City
PROFESSIONAL: New bus-dedicated lanes for Madison Avenue and Fifth Avenue. The plan is ... to give them signal priority -- which means traffic lights recognize approaching buses and either turn or stay green so that the buses remain on schedule.
PERSONAL: To get a better understanding of the seventh grader's mindset. [Her son Max is 12 years old.] Everyone has a bad seventh grade.
INDUSTRY: Detroit should start making buses. There's a huge back order for buses -- every single transit agency in the country is ordering them right now and most are getting them from overseas. General Motors used to make them, so why can't they again?
Chris Brown, 19
Singer, Tappahannock, Va.
PROFESSIONAL: I plan to complete my new album, "Graffiti." I do my first European arena tour in January. Since I've also started songwriting, I plan to put together my writing team.
PERSONAL: I hope to achieve all aspects of success in a lot of different areas.
INDUSTRY: The community needs to tackle the paparazzi issue and also the issue of leaks of unreleased music.
Jerry Buss, 74
Owner of the Los Angeles Lakers, Playa del Ray, Calif.
PROFESSIONAL: In my second career -- playing professional poker -- I hope to win the World Series of Poker next year.
PERSONAL: I hope to remain calm, at all times, during close games when the referees miss calls.
INDUSTRY: The NBA Cares program is a good one and I think it should be expanded to accomplish more that we already have.
Bruce Stewart, 68
Head of Sidwell Friends School, Washington, D.C.
PROFESSIONAL: A major review of the curriculum with an eye toward what is the knowledge most worth having for the 21st century.
PERSONAL: I very much hope to stay on the cutting edge.... You've got to be alert ... and be adjusting if you're going to educate people in the best and most effective way. I just want to be darn sure I don't become complacent.
INDUSTRY: I'm concerned about maintaining accessibility to the professional class or middle class. There's a danger of independent schools becoming barbell-shaped -- very affluent people on one end and very impoverished people on the other end and a very thin line in the middle.
Mary Boone, 57
Art dealer, New York City
PROFESSIONAL: To continue with Young at Art, the art initiative with Publicolor that provides financial aid to art programs in public schools and enhances the curriculum.
PERSONAL: To laugh more.
INDUSTRY: To remember, especially in this difficult economy, that art can be restorative. Art gives our eyes and mind a chance to rest, to muse, to think.
Shashi Tharoor, 52
Former under-secretary-general of the United Nations, author, Trivandrum, India
PROFESSIONAL: I have been immersing myself more and more into Indian public life. With the terror attacks having brought the country to a crossroads and national elections due in the spring, I will have to decide whether and how far to involve myself in my country's political future.
PERSONAL: To take a real stab at beginning work on a novel -- my last was in 2001. Though there have been three nonfiction books since, I haven't been able to find the time or the space inside my head to create my own fictional world.
INDUSTRY: We need in 2009 to tackle the task of reinterpreting America to the world: post-Bush, post-9/11, and post the historic election that has galvanized Washington-watchers on every continent.
Albert Awad, 100
Owner of American Wire & Cable, Manila, the Phillippines
PROFESSIONAL: Expand the capacity within my own factory to produce more wire and cable to meet the growing demand for both industrial and housing needs.
PERSONAL: To live to be 105 and complete all my dreams, like seeing all of my children and grandchildren getting married, having more grandchildren and having some great-grandchildren, too.
INDUSTRY: The need for the electric industry to develop ways to service the people in the 7,000 islands of the Philippines with electricity 24 hours a day.
Mariner Kemper, 36
Chairman and CEO of UMB Financial, Kansas City, Mo.
PROFESSIONAL: I feel a personal responsibility to beat a loud drum about the good guys -- the banks and companies that operated with sound principles and followed the rules and haven't tried to fix their mistakes on the backs of taxpayers.
PERSONAL: A wise man, actually a Muppet, once said "It's not easy being green." I am resolved to make my home and company continuously more sustainable.
INDUSTRY: Our industry should focus on educating Americans to depend less on debt and to save more. It's a sad state to know that for every dollar of economic output, $1.01 is paying off debt.
Barry Bergdoll, 53
Chief curator of architecture & design at MoMA, New York City
PROFESSIONAL: A very ambitious Bauhaus exhibition I'm working on with a colleague. The challenges: To tackle such a complex topic in the space of galleries and give a new vision on a classic subject. We'd open in November.
PERSONAL: Go to yoga more regularly. Achieve a more healthy balance between mind and body. I go about once a week now.
INDUSTRY: An enormous cohort of architectural talent that's been formed in the last boom is about to be unemployed.... I'm just hoping the silver lining in all of this is that President-elect Obama's call to think about infrastructure means he's able to push through something on a large scale that could provide immediate opportunities for the talent that's suddenly untapped. It could be transformative for the profession and set themes for the next decades.
Simon Cooper, 63
CEO of Ritz-Carlton, McLean, Va.
PROFESSIONAL: To launch the first Ritz-Carlton "Reserve" in Krabi, Thailand.
PERSONAL: Do more listening! Age and experience tend to close the mind and the ear.
INDUSTRY: Remind everyone that a global economy requires global travel to both revive growth and sponsor broader understanding.
Duncan Sheik, 39
Songwriter and composer, New York City
PROFESSIONAL: An adaptation of Hans Christian Anderson's "The Nightingale," a collaboration with Steven Sater and James Lapine.
PERSONAL: To create a recording studio/rehearsal space close to New York City, where my coterie of musician friends and collaborators can work on their projects irrespective of financial considerations.
INDUSTRY: The music industry needs to return to a practice of strict quality control and artist development. Particularly, to supporting artists that create unique and progressive work.
Roy R. Neuberger, 105
Art collector and founder of the Neuberger Museum of Art, New York City
I hope the [museum] encourages the creation of art...and that PROFESSIONAL: we recognize creative art.
PERSONAL: Art has survived many generations. Art will persevere.... There are better things than art, but not many!
INDUSTRY: I'm looking forward to the new year. I'm not looking back any more.
Peter Singer, 62
Philosopher and author, New York City
PROFESSIONAL: My book, "The Life You Can Save: Acting Now to End World Poverty," will be published in March. It represents the culmination of more than 30 years of thinking about this issue, so I'm hoping it will stir some debate and some action too.
PERSONAL: When surfing, not to pull off those scary big waves.
INDUSTRY: Climate change. That goes for philosophers, as it goes for every industry and professional community.
Steven Holl, 61
Architect, New York City
PROFESSIONAL: A main project I am completing in 2009 is the printing of my book "URBANISMS: Working with Doubt," a book I have worked on for over 10 years.
PERSONAL: I never make New Year's resolutions. I prefer monthly experiments
INDUSTRY: With a promising new phase in USA leadership, thinking creatively about the problems of new infrastructure and how all things are integrally related—architecture/urbanism/landscape—should be a focus…as well as new types of public works.
Kip Tindell, 55
Chairman and CEO of the Container Store, Coppell, Texas
PROFESSIONAL: A new garage organizing system will be introduced in January. The goal is to get more men to buy. Our store is 85% female. Men now mainly take up aisle space and limit the amount of time their wives spend in the store.
PERSONAL: To evolve into a foodie. I was recently named to Whole Foods board of directors. I am into food and wine now but I want to learn even more about eating healthy. I am not going to become a vegan or anything. But when you turn 50 you start thinking about things like that. I eat like a Frenchman. (I am 55 now.)
INDUSTRY: Retail companies need to pay more attention to putting employees first instead of the shareholders, especially in this time when so many companies are having layoffs.If you put the employee first you'll have happy customers and if you have happy customers then shareholder will do well.
Neal Katyal, 38
Georgetown University professor of law, Washington, D.C.
PROFESSIONAL: To return to a winning record in the Supreme Court.
PERSONAL: To conquer mogul runs to the beat of a metronome.
INDUSTRY: To remind our citizens that the government can be a force for upholding the rule of law rather than evading it.
Jonathan Tisch, 55
CEO of Loews Hotels, New York City
PROFESSIONAL: We plan to continue to provide a supremely comfortable and welcoming experience to our guests and give them great value at the same time.
PERSONAL: My primary hope is to complete my third book, which focuses on civic engagement. While it has been in the planning stages for some time, the economic crisis along with the changes in our nation's capital make its premise more prescient.
INDUSTRY: The travel and tourism industry needs to continue to work with our elected officials in Washington and press for the passage of The Travel Promotion Act. It will help our country welcome more international visitors at a time when their dollars are sorely needed.
Alex Bogusky, 45
Founder of the Crispin Porter Bogusky advertising agency, Boulder, Colo.
PROFESSIONAL: We're going to begin pumping marketing into consumer's brain stems via ultrasonicwaves—to avoid the media costs. That, and a CP+B factory in Europe.
PERSONAL: Levitation. Just an inch or two.
INDUSTRY: To begin building institutions that people can trust.
Julia Fischer, 25
Violinist, Munich
PROFESSIONAL: I'm touring with Academy of St. Martin in the Fields in the U.S. this February. In May I have my debut with the Los Angeles Philharmonic, and in June I'm going to perform with the San Francisco Symphony Orchestra with Yefim Bronfman.
PERSONAL: I want to have time for myself. I want to have the chance to read books and learn foreign languages. Russian is at the top of my list.
INDUSTRY: The main issue classical musicians have to look at is, how do we get young people to play and spend more time with music? I always try to spend time in schools to talk to them about music and to get young people to come to one of the concerts or rehearsals.
Peter Sagal, 43
Host of NPR's weekly news quiz, "Wait, Wait...Don't Tell Me!", Oak Park, Ill.
PROFESSIONAL: In 2009, my professional goal is to have a candidate declare his or her intention to run for president on my show. Not only will it get press, but since they'll be declaring three or more years before the election, they will instantly become the frontrunner.
PERSONAL: I'd like to continue to resist the urge to declare that I'm running for president on my show, despite the advantages mentioned above. I have really no facility or desire to enter into politics, which is something I've been explaining to the voices in my head for years.
INDUSTRY: Public radio needs to cope with the problem that we're making less and less money as more and more people listen to us. My solution is to re-adapt the technology once successfully used on black and white TVs in bus station waiting rooms—coin operated receivers. Want more Car Talk? That'll be another fifty cents, please.
Gary Bettman, 56
Commissioner of the National Hockey League, Saddle River, N.J.
PROFESSIONAL: Doing everything possible to provide the most value to our fans (including our season ticket-holders, of course), sponsors and other business partners in these challenging economics times
PERSONAL: Stick to my diet this time!
INDUSTRY: Greater efficiencies on the cost side of the business.
Christopher Wheeldon, 35
Choreographer and founder of the ballet company Morphoses, New York City
PROFESSIONAL: I plan to complete choreography for a new production of Bizet's Carmen at The Metropolitan Opera which opens on New Years Eve 2009. So it'll be by the hairs on my chinny chin chin!
PERSONAL: I hope to keep my resolution of writing personal thank-you letters to as many people who donate money to my ballet company as possible.
INDUSTRY: The performing arts need to find creative ways to encourage people to come to the theater, even during difficult times. If it means we have to go easy on production values and create the magic in more frugal ways then so be it!
Michael Kaiser, 55
President of the John F. Kennedy Center for the Performing Arts, Washington, D.C.
PROFESSIONAL: In late February and early March 2009, the Kennedy Center will launch Arabesque: Arts of the Arab World, that will bring over 700 artists to Washington from the 22 countries in the League of Arab States. The three-week festival will include performances in dance, theater and music as well as exhibitions of photos, sculpture, mosaics, and fashion.
PERSONAL: I hope to keep my resolution to run over 2,000 miles next year.
INDUSTRY: The recession has reduced the available pool of contributed funds and earned revenue severely. All of us are looking for ways to cut costs to address these revenue shortfalls.
Paul Hawken, 62
Author, environmentalist, entrepreneur, Mill Valley, Calif.
PROFESSIONAL: I intend to fully launch Biomimicry Ventures Group, a company I formed with my partner, Janine Benyus, which will commercialize biologically inspired and mimetic technologies that address water capture, water purification, agriculture, thin film solar cell assembly, robust bio-polymers, and the ability to recover metals in waste streams.
PERSONAL: To spend a lot more time in nature with Mariel, my partner, and do a two-week silent retreat at Integral Meditation Society in Barre, Mass.
INDUSTRY: The key problem is energy and climate. The state-of-the-shelf technologies necessary to reduce energy use by 80%, improve our standard of living, and substitute carbon-based energy with renewables are in place. We need to create the financing mechanism— "green bonds"—that will invest $6-8 trillion dollars into the economy over the next ten years to completely retrofit America.
C.D. "Dan" Mote, 71
President of the University of Maryland, College Park
PROFESSIONAL: The top-goal for 2009 is to break ground on a $900 million, 38-acre town center project (East Campus) to revitalize the community next to the university. The project includes a hotel, student and market rate housing, retail, restaurants and entertainment. It is the largest university town center project in the country and requires extraordinary commitments by the university, community and the developers, especially at this time.
PERSONAL: Remain happy, healthy and employed.
INDUSTRY: The declining participation of young people in higher education is a time-bomb for our nation.
Danny Meyer, 50
Restaurateur, New York City
PROFESSIONAL: Our Hudson Yards Sports & Entertainment division was selected by the New York Mets to bring a groundbreaking approach to food and hospitality to their new Citi Field. When the Mets take the field for opening day in April, 2009, we'll be there.
PERSONAL: Giving up my near-addiction to playing Brick Breaker on my Blackberry.
INDUSTRY: I really hope restaurants will choose to hire more and more employees not just for how well they know how to cook and serve, but foremost for their "hospitality quotient"—the critical set of emotional skills that lead to making customers feel warmly welcomed.
Annabelle Selldorf
Architect, New York City
PROFESSIONAL: We are finishing a beautiful house in the mountains of western Colorado.
PERSONAL: Too many; drink more water in the course of the day….
INDUSTRY: Find a modern architectural expression for sustainable work.
Mark Yudof, 64
President of the University of California, Oakland, Calif.
PROFESSIONAL: I plan to work with colleagues in the other systems of California education to significantly increase community college transfers to four-year universities. Community colleges are diverse and relatively inexpensive, and students who transfer from them go on to perform very well in our institutions.
PERSONAL: I am still in search of the perfect pancake in California. I have been devoting significant attention to this objective, but so far, I have only found an 8 on a scale of 10! But on the hamburger side, I am happy to report I have found a 10.
INDUSTRY: We need to impress on the American people that our great country has been underinvesting in human capital for decades —and that the nation is at great peril in terms of global economic competitiveness and the well-being of its people.
Grant Achatz, 34
Chef/owner of Alinea, Chicago
PROFESSIONAL: We're in the development stage of a new restaurant here in Chicago. We don't have a name yet. The theme will be a more accessible level of creative cuisine.
PERSONAL: I made an agreement to my girlfriend that I would diligently start a yoga program. Cooking at this level is a young man's game, you have to be physically fit to do this.
INDUSTRY: Sustainability and the environment. We produce so much waste, whether it be actual physical waste of food, cardboard, paper products, and then all the resources—natural gas, electricity, water. That's one we're really going to focus on at the restaurant.
Brian Sharples, 48
CEO of HomeAway, Austin, Texas
PROFESSIONAL: If you take all our traffic it is the eighth largest online travel destination in the world, but it's still a conglomeration of sites. We want to achieve the full technical integration of all of our companies.
PERSONAL: To turn off my Blackberry and to stop thinking about work 24 hours a day.
INDUSTRY: Making it as easy to find and book vacation properties as easy as it is to do in hotels.
Contributing Reporters
Reed Albergotti, Alexandra Alter, Jess Bravin, Jamin Brophy-Warren, Amy Chozick, Juliet Chung, Kelly Crow, Darren Everson, Ellen Gamerman, Kevin Helliker, Candace Jackson, John Jurgensen, Hannah Karp, Nancy Keates, Christina S.N. Lewis, Sara Lin, Katy McLaughlin, Lauren Mechling


text:
by Tadas Vilkeliskis, August 14, 2013
This is my first post on machine learning, and hopefully not the last one. The main goal of these posts is to serve as a quick reference for simple machine learning problems and their solutions, meanwhile allowing me to get a better understanding of the field itself. That said, don't take anything for granted.
Linear regression is one of the algorithms used to predict scalar values given some inputs. Linear regression can be modelled like this:
It can also be written in a vector form as:
We first need to find the model parameters \(\beta\) before any predictions, \(y\), can be made for our inputs \(x\).
Given a set of inputs and outputs we would have to solve a set equations for \(\beta\). The vector form can be solved using linear algebra; the solution takes the form of:
This particular solution will find betas using the method of ordinary least squares. If linear algebra is not an option, another way to find betas is to minimize the difference between actual values and predicted values by using some function optimization method, e.g. gradient descent.
We are going to use a minimal version of the Iris data set in the code examples.
In the examples below I am going to implement a simple linear regression model that takes only a single input. This way we can much more easily visualize our data and model; however, the code can be easily extended to support more than one input.
In this example we are going to model the sepal length using sepal width only for setosa flower class.
import csv
def read_iris_data(filename):
- `filename`: name of the file that contains Iris data set.
sepalWidths = []
sepalLengths = []
with open(filename) as fd:
reader = csv.DictReader(fd)
for row in reader:
# Widths will be our inputs, include intercept
sepalWidths.append([1.0, float(row['sepalWidth'])])
return sepalWidths, sepalLengths
One thing to note here is that we are adding intercept to our inputs (ones). It's not very important in this particular case since our inputs will never evaluate to 0. In other cases it may be useful.
import numpy as np
def fit(inputs, outputs):
Solve a set of linear regression equations for betas using linear algebra.
- `inputs`: a list of inputs.
- `outputs`: a list of outputs for given inputs.
X = np.mat(inputs)
Y = np.mat(outputs).T
x_trans_x = X.T * X
if np.linalg.det(x_trans_x) == 0:
raise Exception('Cannot inverse singular matrix')
return (x_trans_x.I * (X.T * Y)).A.flatten()
The fit() function expects a list of tuples as inputs, and a list of output values. The model parameters will be returned as a list. If we are going to run this function on our data set we will get a these beta coefficients [2.63900125 0.69048972]. Thus, our regression function looks as follows:
scikit-learn package comes with a linear model which can be used to solve linear regression problems.
import numpy as np
from sklearn import linear_model
X = np.array(inputs)
Y = np.array(outputs)
# Don't include intercept since we already added it while loading
# data.
model = linear_model.LinearRegression(fit_intercept=False)
model.fit(X, Y)
The model parameters are stored in model.coef_ and for our dataset they are array([ 2.63900125, 0.69048972]). Predictions can be done by calling model.predict method.
For simple cases that have only a single input we can plot the regression line and the data points together to see how well our model fits the data.
The red line represents our original model, which I think looks fairly well. However, if we look at our dataset the first data point looks like an outlier. We can try to build a model using a data set without the outlier. The green regression line represents the model where the outlier was removed from the training set.
By looking at the graphs it's diffucult to tell which model is a better one. We can take a look at the coefficient of determination (\(R^2\)) for each of them. scikit-learn model provides score method which can be used to obtain it. For the first model we have \(R^2\) of 0.551375580392 and for the second one—0.547248091457. The closer \(R^2\) to 1 the better our model is.


text:
POSTED: 10:25 a.m. HST, Jun 6, 2013
LAST UPDATED: 10:30 a.m. HST, Jun 6, 2013
The University of Hawaii Board of Regents voted unanimously this morning to form a seven-member selection committee to identify candidates to replace UH President M.R.C. Greenwood, who is retiring Aug. 31.
The committee will be responsible for coming up with a detailed plan for the search and weighing the merits of hiring a search firm to help with the process, Regent Chairman Eric Martinson said.
Regent John Holzman was named chairman of the search committee. Other members include Jan Sullivan, Saedene Ota, Chuck Gee, Barry Mizuno, James Lee and Jeffrey Acido.
The full board is continuing to discuss the framework for the search this morning.
Earlier, the board also voted unanimously to authorize an internal task group to come up with two or three potential candidates to serve as interim president.
Holzman, who also chairs the task group, said it would be practical to look at candidates within the UH system to "provide for a smooth, orderly transition" and "keep the wheels spinning." He suggested the interim appointment be for an unspecified term, noting that "the person may be there for a while."
Recommended names are expected to be voted on at the board's July 18 meeting.