Ways to Tweak Slow SQL Queries - <AUTHOR> http://www.helenanderson.co.nz/sql-query-tweaks/ <comment><USER2> Heed the caution about CTEs being optimization fences. I had a case just this past week, where one of our engineers needed help with a query that used a LEFT JOIN against a CTE which generated text-search vectors for some subset of rows, in some large table. The query was taking 9-10 minutes, cache-warmed.

Used that way — an outer join against a CTE — the planner was forced to generate a plan that produced a row for every candidate row in the table being used as input for the text-search, whether or not it would be needed, visiting nearly all of its pages.

Just by moving the CTE inline (that is: "LEFT JOIN ( #{ subquery } ) AS blah"), the query completed in 17ms.

This way, the planner could apply conditions from the rest of the query to the subplan from which text-search vector rows were being produced, such that it only pages containing rows it already knew it would care about would even be retrieved.

The rest of this article is pretty on-point, too.

Source: This stuff is my day job.

EDIT: As a counter-point, because they are incredibly useful, I've also had countless cases where rewriting a query to _use_ CTEs was the several orders of magnitude win. This also wasn't the only possible fix; the qualifying conditions in the CTE could have been improved, eliminating the extra work where it would have occurred.

Like all things computers, the real answer is, "it depends..."

<comment><USER3> I think I read somewhere that they are fixing some of optimization issues for CTEs in the next release of Postgres. Does anyone know whether that's true or not?

<comment><USER2> Are you thinking of the "[NOT] MATERIALIZED" clause?

Yes, that's coming.

[https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit...](https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=608b167f9f9c4553c35bb1ec0eab9ddae643989b)

<comment><USER3> Thanks for the info! I am a big fan of the readability of CTEs. Though I guess in production code we use SqlAlchemy, so it would just be a matter of swapping `.cte()` for `.subquery()` probably...

</comment></comment></comment></comment><comment><USER4> The best strategy isn't mentioned: learning how to read EXPLAIN output and acting on it.

> Using wildcards at the start and end of a LIKE will slow queries down

That's only true if you don't have the right indexes in place. A wildcard at the end can use a regular (btree) index. Even a LIKE query with wildcards in other places can be fast with a trigram index.

[https://niallburkley.com/blog/index-columns-for-like-in- post...](https://niallburkley.com/blog/index-columns-for-like-in-postgres/)

<comment><USER5> > A wildcard at the end can use a regular (btree) index.

depending on the locale your db is in, it might be a little less "regular". recently fell in that trap.

[http://blog.cleverelephant.ca/2016/08/pgsql-text-pattern- ops...](http://blog.cleverelephant.ca/2016/08/pgsql-text-pattern-ops.html)

</comment></comment><comment><USER6> "Pull out the names of only the columns you need instead of using SELECT * to speed things up further."

This is not performance-related in most cases. Unless the bottleneck is in the amount of data being transferred to a client over a network connection and there is a large amount of columns, or you have an index which matches the limited column list query exactly, there would be no performance difference in SELECT * vs SELECT <column list>.

In columnar dbs it does matter because the less columns you select the less data gets accessed on disk. However, this does not hold true for row stores because data is stored in such a way that the whole row gets accessed no matter how many columns get specified in the query.

There are many other good reasons why SELECT * is acceptable only in development queries, but performance is not one of them.

<comment><USER7> In terms of performance, the primary rationale for only retrieving columns you need would be to allow the database engine to select a proper index to use, specifically, one that covers all the columns you are making use of. This can actually have a massive impact on the execution plan, far more than simply fetching say 2x more data, as you point out.

In fact, I'd go so far as to say that the practice of selecting all columns makes it practically impossible to make use of covering indexes for high impact queries.

<comment><USER8> It's not about choosing the right index. The point here is that the database can only read the index (without going into the table) if all the columns you need and use in where clause are in the index. Usually index segments are smaller, so it could be faster. I am talking from the Oracle perspective, but I believe PostgreSQL would work in a similar way.

<comment><USER7> Yes, of course, indexes work like that with where clauses. However here I was referring to covering indexes, where the columns in your _select_ are also present (after the ones in the where clause / join / predicate). This prevents having to do extra seeks into the clustered index to fetch that data, and can make very significant differences, especially if row counts are high.

Clearly, this type of index is both expensive to maintain, and the benefits are lessened if you include every column in a table in the covering index. So, to actually have real benefits, you need to have carefully crafted queries which only select what they need, and these being matched to carefully crafted covering indexes. Basically, you cannot have these - which are one of the better weapons you have for performance - if you always select everything.

<comment><USER8> I am talking about INDEX FAST FULL SCAN (Oracle) for example. If you do 'select distinct last_name from users;' and you have an index on users (last_name) you do not need to read the table at all. It is enough to read the index.

Otherwise it does not matter how many columns you select, because you will have to read all the columns for every row anyway.

<comment><USER9> > Otherwise it does not matter how many columns you select, because you will > have to read all the columns for every row anyway.

It does matter. An index can actually contain a cache of values for some columns. This is done with INCLUDE statement.

         CREATE INDEX idx
             ON book ( author_id )
             INCLUDE ( book_title )

This select will use only index:

           select author_id, book_title
           from books
           where author_id = 123

This select will use index, and then have to follow and fetch data out of rows:

           select *
           from books
           where author_id = 123

[https://use-the-index-luke.com/blog/2019-04/include-columns-...](https://use- the-index-luke.com/blog/2019-04/include-columns-in-btree-indexes)

</comment></comment></comment></comment></comment></comment><comment><USER10> Good set of tips, however "Not NOT IN" is not true in general, as a single counterexample:

        EXPLAIN (costs off)
        SELECT relname, relpages
          FROM pg_class
         WHERE relname IN ('pg_class');

                             QUERY PLAN
      </comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment>---
       Index Scan using pg_class_relname_nsp_index on pg_class
         Index Cond: (relname = 'pg_class'::name)
      (2 rows)

In general, I'd like to echo other's comments, learning how to use EXPLAIN and how to interpret query plans is the single most important tool, as it allows you to verify your hypothesis instead of relying on rules-of-thumb.

<comment><USER11> I actually had situation where the using IN resulted in a query that run for 35s and after replacing with `ANY(VALUES(...` it took ~600ms.

In my case `column IN ('value1', 'value2')` was converted to an array like this: `((column)::text ANY ('{value1,value2}'::text[]))`, an index on the column was completely ignored and PG performed a sequential scan.

After changing it to `column = ANY (VALUES ('value1'),('value2'))` it created a table in memory and used it when performing an index scan.

I have feeling this is maybe some kind of a bug in query planner? This is on PG 9.6.8.

<comment><USER12> I think I'd need more information to diagnose; were there any NULL values in the list? How many values?

<comment><USER11> The column can have nulls but after a quick check there's only a single record with a null value (probably added as a test, no idea).

In the bad plan a sequential scan was performed and then it removed over 4 million rows (Rows Removed by Filter: 4562849)

Edit: here is the query planner's output: [https://explain.depesz.com/s/eVXI](https://explain.depesz.com/s/eVXI)

After rewriting the query (change IN into ANY(VALUES( and adding an index to the "seven" table): [https://explain.depesz.com/s/b4kU](https://explain.depesz.com/s/b4kU)

<comment><USER12> I think adding the index to the "seven" table is probably the more meaningful change.

You said the index was ignored, but based on that plan output the planner made the right choice since the filter on the "sierra_zulu" table (only one there with an ANY so I assume that's the one) matches over 90% of the rows. An index scan on that would be far worse.

Adding an index on a different table allowed the driving side of the join to be different, and, with already prefiltered data allowed the other index to be useful as well (fewer index lookups). I'd bet with that index addition your IN clause works just fine.

<comment><USER11> If you look at sierra_zulu (line #3 on both plans) it took almost 22 seconds initially and after query rewrite (and no index change) took 713ms.

So that change of query alone reduced query time from 35s to ~5s, after that the seven table become the bottleneck and adding an index there reduced the whole query down to 750ms.

I should have saved the intermediate plan to illustrate it, but it essentially looked same as the second plan, except there was a sequence scan on seven table.

</comment></comment></comment></comment></comment></comment><comment><USER13> Some of this is probably SQL server specific, but we were having some scaling issues on legacy code earlier this year and I got a lot of bang for my buck checking for these things before diving in to specific issues:

\- code with a high row count table variable when a temp table would give more accurate execution plans due to better cardinality estimates.

\- ultra-complex join criteria with lots of OR logic that performed better as a UNION

\- lazy function calling, where a developer used a function that did more than they needed and could be replaced with a simple join to a table

\- looped calls to insert procs that could be done in bulk

There was another entire category of problem that I guess would be described as "I hate sets" and was mostly attributable to one former employee. These were recognizable immediately upon opening the code and were a nightmare.

<comment><USER14> >There was another entire category of problem that I guess would be described as "I hate sets" and was mostly attributable to one former employee. These were recognizable immediately upon opening the code and were a nightmare.

Care to elaboarate? I need a pick-me-up today.

<comment><USER13> It was a lot of stuff like

\- Get a list of x in a table var

\- while loop through x to build another list of y

\- while loop through y and update z one row at a time

All of that rather than updating with a join.

Some of his other patterns were reusing variables for different things in a proc, using inefficient functions in a way that they executed once per row before the result set was really reduced much, nesting those functions, and using loops any chance he got.

You'd open up these slow, 500-700 line monster procs and have to figure out what they were doing and refactor them, but it was nearly illegible and there were no tests for them. Really a great reminder of what happens when code reviews aren't done.

<comment><USER14> That's way worse than I thought, thanks for sharing.

</comment></comment></comment></comment><comment><USER11> About indexes, they improve query performance, but they also decrease write performance (and of course take space) that's why it's good idea to remove indices that aren't used or rarely used. Especially indices that are placed on columns that have random data and/or are frequently updated (prevents HOT updates) can hurt write performance.

<comment><USER15> One very powerful "trick" is to use partial indexes where useful. For example, a pretty common pattern is to have tables where most rows aren't actively used most of the time, eg when you have a _status_ column, and most rows are in the final status which you usually don't touch, so most where conditions include statuses which are not the final one. If you create a partial index WHERE status IN ('s1', 's2' ...), the index becomes a lot smaller and a lot faster.

<comment><USER16> I never thought about that, but that's a neat trick.

</comment></comment></comment><comment><USER17> The first principle for SQL optimization should be to reduce the number of queries. If you have a query inside a loop, the performances may be catastrophic. Using conditionals in queries was sometimes the only solution to avoid loops. The performance gain was enormous.

</comment><comment><USER18> Lightly put, the article leaves much to be desired, no mentioned of: execution plan, collecting statistics, partitioning, cost of indexing (e.g. when should you disable an index and rebuild it?), materialized views, etc.

<comment><USER19> Surprised they didn't mention citext columns

</comment></comment><comment><USER20> > If you are calling multiple views, or worse, views on views on views you are > asking the query engine to run multiple queries to return your columns.

I thought databases flatten all view layers into a single SQL statement before executing. Aka no performance impact except for a couple milliseconds to flatten out the views.

<comment><USER21> Yes, this was my thought as well. The views being flattened is certainly what I've found it's always looked like was happening, based on the EXPLAIN output.

</comment></comment><comment><USER22> Number one has got to be look at the execution plan. If it's complicated, try to simplify it, and that's where all these particular things should be checked.

<comment><USER18> Agreed, related point, collecting statistics.

</comment></comment><comment><USER23> _Try to avoid the use of 'IN' or 'NOT IN'. By doing this you are performing a full table scan as the query engine looks through every row to check if the condition is met._

It's not clear to me why IN () can't do an index scan or hash join?

<comment><USER15> IN () definitely does use indexes in most cases on Postgres. The article is wrong there.

<comment><USER24> Yes in general but I was hit with case when Postgres started to switch to FTS depending on argument count inside `IN ()` while in reality using index was faster for each and every case. I had to split it in multiple queries with 100 args each, even if few month earlier it didn't have any issue with thousands.

<comment><USER12> I'm guessing the index in question was a partial index defined with a "WHERE column NOT NULL" restriction?

This was a limit of the optimizer previously (IN clauses are broken down into AND/OR groups to prove inferences but only if <= 100 items).

But Postgres 12 includes a patch I wrote so that the optimizer can prove the NOT NULL inference directly from an array operator of any size.

<comment><USER11> In my case the index is not a partial index, but it is also on PG 9.6.8, in another response[1] I provided more details.

Unfortunately we are stuck with 9.6.x since before my time my company decided to use AWS Aurora, currently there's no easy path to do major version upgrade from 9.6.x and anyway 10.x is the most recent available version :( but any information why is this happening would be appreciated.

[1] [https://news.ycombinator.com/item?id=20863418](https://news.ycombinator.com/item?id=20863418)

</comment></comment></comment></comment></comment><comment><USER25> This article has sage advice, but SQL optimization goes far beyond. Optimization is a knee-jerk reaction for me whenever I do database development and involves every level (e.g. should I use IsNull or Coalesce? Try both, look at execution plans, etc.)

</comment><comment><USER26> These articles are fine, but it is very rare to see people using a profiler to discover what needs to be optimized.

</comment><comment><USER27> It should be somewhere reflected that this article applies in part only to PostgreSQL (some of the tips are more generic thou)

<comment><USER28> Immediately following the list TOC:

> _"This is written with PostgreSQL in mind but if you're using something > else, these may still help, give them a try._</comment></comment>


GCC and C vs C++ Speed, Measured - <AUTHOR> http://rusty.ozlabs.org/?p=330 <comment><USER2> Just to be clear, because this confused me at first:

\- GCC recently switched from building their own code as C, to building as C++. At the moment they only use the subset of C++ which is C, but this affects compiler behaviour.

\- People complained that C++ programs couldn't be compiled as efficiently as C programs, and this would make things slower.

\- This post shows that compiling GCC as C, versus C++, does not result in an appreciable performance change. He benchmarks this by building the kernel using both versions of GCC.

<comment><USER3> Most of C++ devs think that C is part of C++. Even in this document it is stated something like: C subset of C++ vs C. Why? Because _current_ C is not subset of C++, C++ is compatible with C89 or C90.

This can look like nitpicking, but it's really annoying to again and again hear such things from mouth of C++ gurus.

Now give it to me ;-)

<comment><USER4> <https://github.com/libav/c99-to-c89/>

<comment><USER3> And this is relevant here because of?

<comment><USER4> ...you're twisting my arm now, you know? Why a plain C99-to-C98 convertor it's relevant? The author explain it better in one of his blog posts: [https://blogs.gnome.org/rbultje/2012/09/27/microsoft- visual-...](https://blogs.gnome.org/rbultje/2012/09/27/microsoft-visual- studio-support-in-ffmpeg-and-libav/)

Quote:

"Reality is, many of these C language features are cosmetic extensions introduced in C99 that are trivially emulated using classic C89 syntax."

<comment><USER3> You just demonstrated another difference in C vs C++ mindset.

C doesn't need so many changes like C++, that is the fact and _advantage_. C++ introduces huge bunch of new stuff, you know add, add, add, then fix, fix, fix. At the end you get overengineered buerocratic 'give me my 5%' language.

So, in C99 some stuff are very useful like variable length arrays, restrict keywords, variadic macros... these are not cosmetic. And btw, _even_ _for_ that you don't need C99 to c89 converter, if you want c89 compatibility use macros. But, hey, macros are ugly, right? They are ugly but they don't make my eyes bleed as some stuff in C++.

So yes, i'm twisting your arm, sorry about that, but it needs to be twisted ;-)

P.S. Your compiler is your choice, just don't force others to use it.

</comment></comment></comment></comment></comment></comment><comment><USER5> > Using stats –trim-outliers, which throws away best and worse

people make this mistake over and over again. there's no point in taking anything but the best time when talking about performance in a preemptive multiprocessing environment. the best time isn't an outlier, every time other than it is!

<comment><USER6> The best time is irrelevant in practice if it only happens very infrequently, under a very specific set of circumstances.

Benchmarks are only useful if they're indicative of what the likely outcome in a real-world scenario will be.

What you propose is much akin to those completely unrealistic "The Computer Language Benchmarks Game"-style benchmarks that show JavaScript (or Java, or PHP, or whichever slower language you prefer to use instead) as being as fast as C or C++. Perhaps that's true, but only when you discard reality almost completely. JavaScript only ends up performing decently because the benchmark is trivial, and the implementation has been highly tuned, often into a form that we'd never actually see in any real-world product. And predictably, JavaScript's performance ends up being much worse than C's or C++'s when it comes to real software, contrary to what the flawed benchmark suggests.

<comment><USER7> > The best time is irrelevant in practice if it only happens very > infrequently, under a very specific set of circumstances.

This isn't a practical benchmark, though. This is an experimental performance comparison. So to ensure that your results aren't influenced by other variables, it's a good idea to attempt to produce the ideal environment for best performance across both tests. So binding cores(to eliminate as much variance from Linux's thread scheduler) and running nothing else on the machine are actually good decisions to make here.

Comparisons to the Computer Language Benchmarks Game is somewhat irrelevant here, because the comparison being run is actually extremely simple: Using a GCC executable compiled as C versus a GCC executable compiled as C++. The traditional failings of the game aren't here(implementation issues, language- specific tuning issues, etc.) because the code that GCC is compiled from is identical, the compiler options should be identical(except for C vs C++ compile setting), and the test setup should be run on the same machine, with the same compiler options and tuning choices.

</comment></comment></comment><comment><USER8> FWIW, RustyRussell is this Rusty: <http://en.wikipedia.org/wiki/Rusty_Russell>

Nice to see him here - "ciao" from Italy:-)

<comment><USER9> Yes. We can blame the ipchains/iptables syntax on him ;)

Also, he used to poke fun at C++ (many older C devs did that) but more and more of them are coming around to C++. The last automobile systems programmers I worked with were switching to C++ as well. In ten years, C++ will be the new C.

<comment><USER10> I'm not coming around to C++, I assure you.

But I like measurements, even if they don't confirm my baises.

</comment></comment></comment><comment><USER11> The results are what I expected and not that interesting. Compiling C as C++ usually yields the exact same native code as compiling it as C, except for some very minor differences.

What would be interesting is how long it now takes to compile GCC itself. For code that does not use C++ features, I also expect a very minor difference.

<comment><USER12> Compilation of PETSc, which is written in the intersection of C and C++, takes twice as long using a C++ compiler as using a C compiler. I'd be interested in the time difference for compiling GCC.

</comment></comment><comment><USER13> Interesting aside: There are some things that are faster in C++ than C, for example the standard library sort functions are massively faster because the compiler can in-line the comparison.

<comment><USER12> The performance is the same if you put the qsort implementation in a header.

<http://www.agottem.com/blog/inlining_qsort_sort>

<comment><USER14> or use link-time compile generation (slow, and it might not work for huge projects like gcc/webkit, but if split into shared modules it can)

</comment></comment></comment><comment><USER15> That is one hell of a misnamed article.

<comment><USER16> Why? It is measuring the difference between a compiled-as-C GCC and a compiled-as-C++ GCC.

<comment><USER15> Difference in compilation times.

<comment><USER10> Yeah, people seem confused by this. You can build the last release of gcc either with a C++ compiler or with a C compiler. I did both, and benchmarked the results.

Hope that helps! Rusty.

<comment><USER15> Sorry, I missed the sentence: "With this in mind, and Ian Taylor's bold assertion that "The C subset of C++ is as efficient as C", I wanted to test what had changed with some actual measurements."

</comment></comment></comment></comment></comment><comment><USER17> I don't think having a single statistics point is enough to make a judgement here. There are many different programs of which compilers are a very small and relatively rarely used subset.

</comment><comment><USER18> So 0.3% is a measurable difference and 0.1% is noise. That doesn't sound overly pragmatic to me. Benchmarking is an art, and from his setup 0.3% seems like noise as well.

</comment><comment><USER19> Aren't the files already cached in memory by the time the C++ compilation happens? Wouldn't flushing memory provide a better measure?

</comment><comment><USER20> I always ask myself, why are there people who put statistics without charts on the internet.

<comment><USER10> Because there are only two stats of note in the post, and they are 0.3% and 0.1%. That kind of difference doesn't make for useful graphs, does it?

<comment><USER21> They did 40 runs of compilations (10 each, twice). At the very least, a scatter or bar graph would be helpful, as it would let you see the data distribution (those numbers are good as uninterpretable on their own).

Saying that, 40 isn't very many times to draw statistical conclusions so you'd probably need more data.

<comment><USER22> I think a histogram or boxplot might be more appropriate to show the distribution. A scatter plot isn't appropriate for data whose independent variable is categorical (unless you jitter), and a bar plot (unless a histogram) would just show the means, wouldn't it?

</comment></comment></comment></comment><comment><USER23> So python is the fastest?

<comment><USER24> We don't know, because GCC does not execute as a Python program.

This _wasn't_ testing the speed of compiling a program as C++ versus compiling it as C; it was testing the speed of compiling the Linux kernel using GCC- compiled-as-C++ as the compiler versus using GCC-compiled-as-C as the compiler.

(And it concluded that the difference was insignificant anyway!)

<comment><USER23> Yeah my bad I thought we were measuring the speed of compilation in C Vs. C++. (looks like it wasn't just me)

</comment></comment></comment><comment><USER12> What is the time difference to compile GCC itself?

</comment><comment><USER25> Linus will now use LCC

</comment><comment><USER26> People were complaining that the binaries would be slower? All the complaints I heard were "now it will take 10 times longer to compile gcc", not "now gcc will run slower".</comment>


Ask HN: How important is SEO to your business and what tools do you use? - <AUTHOR>

And what is your most important tool? Website or desktop tool? Consultant? And do you pay for these tools/services or do it yourself? <comment><USER2> It really depends on the type of business you are opperating and the background you bring. Companies that operate content businesses or busineses that rely on advertising as their primary revenue model rely on SEO more than most others.

Regardless of the value you see in SEO, it drives a number of critical skills. SEO relys on data, keyword data, traffic data, bailout data, conversion data etc. and so learning about what makes SEO successful means you must have quality analytics in place and learn how to use them and that will make all of the operating parts of your business better and or more efficient.

Again depending on your needs and the size of your business SEO is best managed internally, rely on an experienced and reputable consultant if you need help.

Sites like SEOmoz and SEObook offer excellent information and tool sets to get you started for a minimal monthly fee.

</comment><comment> onreact-com I'm doing SEO myself as a consultant so it's very important for my business ;-) and I have to disappoint you:

1\. SEO is not about "a tool", it relies on a plethora of factors and tasks many of which user solost has already outlined in his comment

2\. There is no single tool that will allow you to do SEO work, you need a few tools, I'd say more or less 10 depending on what you actually do. Three weeks ago I compiled a list of the most important ones: [http://www.seoptimise.com/blog/2009/07/the-30-free-seo- tools...](http://www.seoptimise.com/blog/2009/07/the-30-free-seo-tools-you- must-know.html)

Nowadays SEO is more about "killer" content creation, social media and viral marketing than anything else. The basic on site SEO work and link building is trivial and not enough.

<comment><AUTHOR> thanks for the comments and that link.

definitely not disappointed - i wasn't implying it was a tool as much as wanting to know whether or not people would say 'you must have the seobook toolbar' or 'mozrank changed my perspective on valuing links', etc...</comment></comment>


I'm becoming scared of Facebook - <AUTHOR> https://www.reddit.com/r/technology/comments/5k4dn8/im_becoming_scared_of_facebook/? <comment><USER2> I can't believe people legitimately think Facebook is spying on them through microphones. There's so much that's more likely to be happening I can't fathom why they just default to assuming that and it only serves to make the conversation seem crazy to anyone with a hint of rationality.

Here's a more likely scenario that is scary in its own right: I have a private conversation with John Doe. John Google's it and ends up on a page with the Facebook tracking pixel. Google knows John and I know each other, knows John and I were in the same place and knows John googled something not long after or while we were in the same place. Facebook doesn't know we talked about it but it can make a guess. Worst case is they show me an ad without much context like dozens of other ads I ignore every single day.

That kind of profiling is what we should be talking about because it doesn't take much to figure out who's talking to who about what when you have information about their every move and page load.

<comment><USER3> I can _definitely_ believe that people think Facebook is spying on them through their microphone, because it's the most obvious answer. Most people don't know anything about how tracking works and how Facebook tailor ads to you, but everyone knows how a microphone works.

I realize that this was posted on the technology subreddit, but that subreddit reaches an extremely wide audience, most of which certainly doesn't know all the ins-and-outs of the tracking/advertising business.

<comment><USER4> Is it the most obvious answer? 20 years ago, I drank a glass of milk and watched the X Files, and then there was an advertisement for milk. Is the most obvious answer that my TV was watching me?

<comment><USER3> Well, maybe 'obvious' is the wrong term, but there's something different between drinking a glass of milk and seeing a milk ad, and pointing out an Arcadia in a parking lot and telling your friend you liked the car, and then just later that night getting an ad for "2017 GMC Arcadia - Low Financing".

The precision is what makes it seem weird, and without an understanding of how a company would target ads to you, the idea that you're being literally listened to by the app, seems possible.

<comment><USER5> Coincidences like that happen all the time. Attributing them to spying is a ticket to a level of paranoia that some people have a very hard time finding their way back from. It's healthier to deter them from such lines of thought when there is an absence of evidence that this is something that actually occurs.

<comment><USER3> Coincidences happen all the time, but targeted ads like those still differ from general audience ads. Just because instead of audio it is spying using metadata and keywords doesn't make seem any less invasive.

> Attributing them to spying is a ticket to a level of paranoia that some > people have a very hard time finding their way back from.

I'll state again that I don't believe FB is actually listening in on phone conversations, but to the majority of their users, they may as well be doing just that. It is all just spying, only using different sources.

<comment><USER5> I know what you were saying. I just chimed in with my opinion. It's unexpected to have the reply to your comment not be in opposition to yours. The expectation is that I wanted to undermine your opinion.

</comment></comment></comment></comment></comment></comment></comment><comment><USER6> This is the feature that is where it all started[1]

> If you don't turn it on, we won't use your microphone to try and match TV or > music when you write a status update. If you do choose to turn it on and > later decide it's not for you, you can easily turn it off at any time.

> Fact: Nope, no matter how interesting your conversation, this feature does > not store sound or recordings. Facebook isn't listening to or storing your > conversations.

> Here's how it works: if you choose to turn the feature on, when you write a > status update, the app converts any sound into an audio fingerprint on your > phone. This fingerprint is sent to our servers to try and match it against > our database of audio and TV fingerprints. By design, we do not store > fingerprints from your device for any amount of time. And in any event, the > fingerprints can't be reversed into the original audio because they don't > contain enough information.

The app does fingerprint the audio and send it to their server to match a movie or audio. But they say they don't save it.

[1]: [http://newsroom.fb.com/news/2014/05/a-new-optional-way-to- sh...](http://newsroom.fb.com/news/2014/05/a-new-optional-way-to-share-and- discover-music-tv-and-movies/)

<comment><USER6> Quick note, now that I reread what I quoted, I noticed how they say they do not store sound or recordings. But what if it is converted to text?

Notice below they say that the fingerprint cannot be reversed into the original audio because it doesn't contain enough information.

Text cannot be converted to the ORIGINAL audio but it can be converted to audio.

This may be just me being paranoid, but I do know that legalese is designed exactly to say something without saying everything.

Anyway.

</comment></comment><comment><USER7> > _Facebook is advertising products to me based on conversations I 've had in > bars ..._

Is there any evidence that Facebook surreptitiously listens to conversations? I could see mining the ambient content of any Facebook video/VOIP sessions, but beyond that I can't see it being true.

Facebook location services can be genuinely creepy though. Earlier this year I enabled location services on the app (while keeping location history disabled), and it would suggest anyone in my phone's proximity using Facebook. It even suggested the attendants at my local gas station, and nurses at my doctor's office. Neighbors too, but only the ones in the same building (~75ft). I suspect this was due to my friends list numbering in the single digits at the time, since I'd otherwise neglected Facebook since its inception.

Then, I took a brief vacation and added a bunch of new people. The creepy location telepathy suddenly vanished. Ever since, all it's been suggesting is acquaintances of said people, despite the fact I live thousands of miles away with location services enabled 24/7\. So, I guess Facebook can be as dumb as it is smart sometimes.

The notion of a private investigator (or psychotic stalker) buying a cheap smartphone, installing FB on it with a clean account, enabling location services, muting it, attaching an external battery and dumping it in a bush somewhere unfortunately seems feasible. It strikes me as a good experiment for a security researcher to conduct.

<comment><USER8> He must be spending time in awfully boring bars if Facebook can do speech-to- text on sound captured while his phone is in his pocket.

<comment><USER9> Voice is pretty good target for being separated from everything else. My "ok google" recognizes my voice pretty good even if loud music is playing, for example.

</comment></comment></comment><comment><USER10> > Yes. Facebook uses a trick to keep running in the background on your phone. > At least on iOS, most apps aren't able to run background processes > constantly. One of the exceptions is apps that play music, so Facebook plays > a silent audio clip in the background so it can stay running.

Interesting, according to this [1] article, Facebook claimed that running silent audio on a phone was a bug. That's one heck of an interesting bug.

[1] [https://techcrunch.com/2015/10/22/facebook-says-it-fixed- a-b...](https://techcrunch.com/2015/10/22/facebook-says-it-fixed-a-bug-that- caused-silent-audio-to-vampire-your-iphone-battery/)

<comment><USER11> I don't think it's that interesting. My guess is it was a glitch with the logic that autoplays videos silently while you scroll through the feed.

<comment><USER12> A glitch? I'm pretty sure they use location changes to notify the app just like Dropbox does for Auto Photo backup, but either way it's no glitch, audio doesn't even play unless you tap the video unless I'm mistaken.

<comment><USER11> Sure. But the audio has to be ready to play from one of any number of videos immediately that are already playing silently. My guess at architecture is that Facebook grabs audio device priority ahead of time and starts playing silent audio so that they can use their own logic to splice in the audio stream of any tapped video immediately when tapped, rather than first needing to acquire audio device priority.

Twitter had something similar for a while; if you were listening to music, then if you scrolled through your timeline and came across an autoplay video (which would play silently at first), your music would stop. They were preemptively grabbing exclusive audio mode in anticipation of needing it for the video, and then playing silence through it until you tapped on the video itself.

</comment></comment></comment></comment><comment><USER13> Hmmm. As crazy as it sounds, I've been freaked out by this recently too.

I recently had to get glasses due to degraded eyesight. I never posted anything about it on social media because quite frankly i was embarrassed by it. Never did any searches etc. Just went to my optometrist, did the test and got my glasses.

Whenever friends see me working, or out and about (where i usually have my phone with me), they usually comment "oh, you have glasses now!" Or something equivalent. I usually respond with a light hearted "oh yeah, i am going blind these days" or something along those lines.

Well dang - my Facebook feed is full of ads for optometrists and lasik surgery etc. these days!

Let me reiterate, i have never mentioned my affliction on any social media channels, nor done a search for anything related to my failing vision. The coincidence is spooky.

<comment><USER14> Microphone could be one possiility, what about additional knowledge from GPS (knowing you're at a location people have checked in on Facebook as being at the optometrist/glasses store, as well as being in locations where people post about such things without checking in.

With as many different users doing as many different actions as Facebook can see, it wouldnt take much to piece it together.

</comment></comment><comment><USER15> I'm having a hard time believing anything substantial from the linked thread. It seems like an obvious example of confirmation bias and frequency illusion. If anyone has some data or a "smoking gun" so to speak, I'm all ears. As it stand now though, I don't really care for another iteration of all the people who don't use facebook gloating about it and intentionally failing to understand the value it provides for its users.

</comment><comment><USER16> Are there any organizations that try to peddle these concepts to the masses?

Just basic things like "there's no such thing as privacy when you're on the internet", and "Facebook owns all your private chats and exploits them to coerce you to click ads and use more Facebook".

<comment> amiga-workbench This all used to be common sense and was drilled into people when the web was new, along with "when online, never share your real name or location with anybody, always use a pseudonym."

Roll on 2016 and people are pouring every personal detail into the web, what the fuck happened?

<comment><USER17> "social networks" happened

<comment><USER12> Yep. People think they're sharing with their "friends" because you get to choose by accepting people and don't seem to fully understand they're handing it over to a megacorp and their data hoarding buddies.

</comment></comment></comment></comment><comment><USER18> I'm more worried about Amazon Echo, which really is listening all the time. "The more you use Dot, the more it adapts to your speech patterns, vocabulary, and personal preferences. And because Echo Dot is always connected, updates are delivered automatically." "Always getting smarter and adding new features".

Some toys listen, too. Toys-R-U pulled "My Friend Cayla" from sale due to complaints from a watchdog organization.[1] "Most devices aren't recording continuously, and won't start recording unless activated. You know, a lot of the stuff in the future might get a little more fuzzy."

[1] [http://www.cbsnews.com/news/smart-devices-amazon-echo- google...](http://www.cbsnews.com/news/smart-devices-amazon-echo-google-home- my-friend-cayla-security-privacy/)

<comment><USER19> This is the scariest of all.

 __* Anything you say in your home is the property of Google /Amazon __*

You have sex with your wife? The sounds you make are their property.

Say something politically incorrect? Amazon/FB might not use it against you, but they might have the data stored and others can.

I really can't believe allow this stuff in their homes.

FB/G etc. are too big to do directly nefarious things like 'use the mic on your smartphone' without your permission, it would blow up on them ... but to do it _with_ your permission? It's crazy.

<comment><USER20> Keep an eye on Ars Technica in the next week or two, and also take a look at this article: [http://arstechnica.com/tech-policy/2016/12/police-ask- alexa-...](http://arstechnica.com/tech-policy/2016/12/police-ask-alexa-did- you-witness-a-murder/)

TL;DR: The Echo uploads audio for cloud-based recognition when it detects voices, but it's not clear if or how long Amazon keeps that. The Echo also has its internal audio buffer which is used for that detection, and in this case the police were looking for any audio that might have been retained in that buffer in the time period before the device was disconnected during the investigation.

</comment></comment></comment><comment><USER21> So I have noticed FB recommend people that all attended the same party that I have no previous relationship to.

It was more people than I'd assume would look me up or be able to spell my name, so quite weird. I didn't assume a mic situation, I assumed the app can read SSIDs or something similar. I don't allow FB to use location services, so either they buy my location from Waze and link it back to my FB ID somehow or the SSID thing. I could see it being technically feasible to sample audio on multiple devices to see who's in the same location, something like Shazam and less for actual content analysis.

Nonetheless it is scary. Imagine you were gay in 95% of the countries in the world. Or Muslim in the US. Or Palestinian in Israel. Or Uyghur in China. Just being in the same vicinity as others in your community establishes a network ready to catalog and analyze.

It's more than FB though.

<comment><USER22> > So I have noticed FB recommend people that all attended the same party that > I have no previous relationship to.

You probably knew some people at the party though, and some of _them_ had location services on. It's really not hard to guess that you were at a party.

<comment><USER21> Interesting, you think there is something at FB to detect an event like that?

It's true we all knew our host but it was one of those _worlds collide_ parties where he had invited friends, neighbors, old friends from high school, some work friends, etc.

</comment></comment></comment><comment><USER23> There is a name for this cognitive bias - 'Baader-Meinhof Phenomenon' or 'Frequency illusion'.

"The illusion in which a word, a name, or other thing that has recently come to one's attention suddenly seems to appear with improbable frequency shortly afterwards" [1]

[1] [https://en.wikipedia.org/wiki/List_of_cognitive_biases#Frequ...](https://en.wikipedia.org/wiki/List_of_cognitive_biases#Frequency_illusion)

<comment><USER24> There is a name for this cognitive bias - 'Baader-Meinhof Phenomenon' or 'Frequency illusion'.

"The illusion in which a word, a name, or other thing that has recently come to one's attention suddenly seems to appear with improbable frequency shortly afterwards" [1]

[1] [https://en.m.wikipedia.org/wiki/List_of_cognitive_biases#Fre...](https://en.m.wikipedia.org/wiki/List_of_cognitive_biases#Frequency_illusion)

<comment><USER23> Please do not plagiarise people's comments.

</comment></comment></comment><comment><USER25> Click the arrow in the top right corner of the ad and click "why am I seeing this ad". Usually the targeting explains it pretty well.

</comment><comment><USER22> I will pay $1,000 to anyone who is able to provide conclusive evidence of Facebook targeting advertising based on listening to all conversations.

It's an easily testable hypothesis which is blatantly false. If Facebook had an always-on phone recording, people's data limits would be swiftly used up. There are tons of alternative explanations.

<comment><USER26> >If Facebook had an always-on phone recording, people's data limits would be swiftly used up.

Not if they were converted to text with the phone before they were sent.

<comment><USER22> Even text would involve a vastly higher upload amount than Facebook actually uses at-rest.

<comment><USER26> Text is pretty small. It's even a multitude smaller when compressed. It could easily be added to normal send packets without anyone noticing. Furthermore, when things are encrypted, it distorts the actual size of the message. I mean I came up with a way to do it unnoticed in a few minutes, imagine what a top team could come up with in a few months.

I obviously don't know if it's happening or not (deleted my account years ago) but it isn't hard.

<comment><USER22> Perhaps, but I've monitored the volume of data uploads by the Facebook app and they're pretty minimal.

There's also the fact that Facebook has gone on the record specifically disavowing this conspiracy. If it were true, they wouldn't so unequivocally deny it. [0] There's far more harm for them deceiving customers than there is in them admitting to a creepy feature.

Additionally, everyone who has promulgated this conspiracy has been fairly suspect. None of them make any effort to analyze the alternative reasons.

But, like I said, I have $1,000 for whoever can prove this is happening. It really shouldn't be hard. If this conspiracy works the way people claim it does, you should be able to mutter about any random product to your phone and have Facebook start showing you ads for it.

[0] [http://www.theverge.com/2016/6/3/11854860/facebook- smartphon...](http://www.theverge.com/2016/6/3/11854860/facebook-smartphone- listening-eavesdrop-microphone-denial)

<comment><USER26> >If it were true, they wouldn't so unequivocally deny it.

That's a strange conclusion to come to. I think it was Shakespeare addressed that very thing in Hamlet.

>There's far more harm for them deceiving customers than there is in them admitting to a creepy feature.

If they've already deceived, then they certainly wouldn't admit it. If they stopped doing it, they could deny it was currently happening and not be lying.

>I have $1,000 for whoever can prove this is happening.

That number pretty much ensures anyone who has the ability to find out wouldn't bother.

Personally, I think they might have toyed with it but abandoned it. The trick isn't collecting the data without getting caught, that's simple (as earlier described). The trick is taking action on the data and not getting noticed.

If you use the conversation data as a weight to only be added to non- conversation data, you could do it completely unnoticed. In other words, gather conversation data but only apply it to data collected in other, less controversial means. So +1 for posting about Colgate, +1 for talking about it only if there is already +1 from the posting.

<comment><USER22> > That's a strange conclusion to come to.

Why? I posit that a good portion of the customer fallout from this functionality has _already_ happened since so many people believe the conspiracy.

By lying about it, they add a whole layer of difficulty. They'd also open themselves up to lawsuits: companies can't just record your audio and then lie about recording it.

> That number pretty much ensures anyone who has the ability to find out > wouldn't bother.

Why? It shouldn't take any sophisticated technical analysis at all.

Ideally we'd have actual network activity proving it, but I'd settle for someone talking to their phone about a product which I choose for them (ie. not something they'd already have on their consciousness) and having ads show up for it. If this feature works how people claim it does, you should be able to get ads on demand for anything you talk about.

<comment><USER26> >Why? I posit that a good portion of the customer fallout from this functionality has already happened since so many people believe the conspiracy.

The quote is "The lady doth protest too much, methinks." I'm not sure if you are familiar with it but it means that when people vehemently protest something it usually means they did it.

I actually read the link you supplied and I don't think they are _currently_ doing it based on what they said, but they might have done it in the past and stopped once people started noticing. They could be sending it to a third party, which would make their denial true. They might not be doing it at all. Who knows?

>companies can't just record your audio and then lie about recording it.

Depends on what people agreed to in the terms and also what wording they used to deny it. I mean they denied they gave NSA direct access to their servers, but NSA had direct access to their servers.

>Why? It shouldn't take any sophisticated technical analysis at all.

It can be easily obfuscated using compression, encryption and when they sent the data. I touched on this in another post on this thread.

They could gather the data and only apply it as a weight to an already collected data point that isn't related to the microphone. I posted this elsewhere also.

Aren't conspiracy theories fun?

<comment><USER22> > The quote is "The lady doth protest too much, methinks."

I'm quite familiar with the quote. Except that in practice, covering up a negative action is often exactly what lands people into hot water (see Watergate, Clinton, etc.).

Facebook strenuously denies this conspiracy. [0] There _are_ in fact laws governing consent for recording.

Of course, it's impossible to prove that Facebook has _never_ listened in on people's conversations, but it's a very suspect claim to say they are today.

> I mean they denied they gave NSA direct access to their servers, but NSA had > direct access to their servers.

They were legally obligated to deny that, so the exact opposite of the situation today (you're legally obligated to disclose if you're recording someone).

> They could gather the data and only apply it as a weight to an already > collected data point that isn't related to the microphone.

Perhaps, but that's an entirely different claim than what the conspiracy theorists are claiming.

[0] [http://newsroom.fb.com/news/h/facebook-does-not-use-your- pho...](http://newsroom.fb.com/news/h/facebook-does-not-use-your-phones- microphone-for-ads-or-news-feed-stories/)

<comment><USER26> >covering up a negative action is often exactly what lands people into hot water (see Watergate, Clinton, etc.).

You are mixing up testifying under oath versus a PR release. Companies lie to you all day long in PR releases. If FB were to go under oath and state that, it would be more believable, but not completely, see James Clapper.

I'm just pointing out how it's technically and orginizationally feasible to do this undetected. The only thing stopping FB from doing this is ethics and possibly backlash.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER27> Why don't people just use the mobile site in their browser? The passwords should be saved in your phone's keychain, and phone screen sizes are plenty big enough to use efficiently these days.

</comment><comment><USER28> Lots of people here saying Facebook probably isn't spying on your conversations are missing the point. I suspect many people finding excuses to exonerate a service they're addicted to. The fact of the matter is that you _should_ be scared of Facebook, because Facebook is fucking scary.

Facebook is a horrifying company. It's the largest and most personal and invasive tracking engine on the net, more so even than Google. They know _everything_ about you. Facebook is tracking and making available to advertisers everything from your employer to your political affiliations to your ethnicity to your charitable donations to your credit [1]. Facebook is (illegally) now combining your WhatsApp information with your Facebook profile [2]. Facebook once collected the unsubmitted status updates you typed and decided against submitting [3]. They track when you are asleep [4]. You already know that Facebook has trained a neural network to identify your face. Facebook engages in censorship on a regular and systematic basis, against journalists and political activists, and regular users too. Even when they aren't censoring explicitly they're building a filter bubble around you that is polarizing our population to extremes.

What happens when they get hacked, badly? Even big companies get hacked and have their databases dumped. Are you really comfortable with everything they know about you being made public? What happens if government actors get access to this data? What makes you think they don't already have it? I saw a stiff give a talk at HOPE who mentioned that they have an FBI agent staffed full- time at Facebook HQ.

You _should_ be scared of Facebook. [https://www.facebook.com/help/224562897555674](https://www.facebook.com/help/224562897555674)

[1] [https://www.washingtonpost.com/news/the- intersect/wp/2016/08...](https://www.washingtonpost.com/news/the- intersect/wp/2016/08/19/98-personal-data-points-that-facebook-uses-to-target- ads-to-you/)

[2] [https://www.theguardian.com/technology/2016/aug/25/whatsapp-...](https://www.theguardian.com/technology/2016/aug/25/whatsapp- backlash-facebook-data-privacy-users)

[3] [http://arstechnica.com/business/2013/12/facebook-collects- co...](http://arstechnica.com/business/2013/12/facebook-collects-conducts- research-on-status-updates-you-never-post/)

[4] [http://arstechnica.com/business/2013/12/facebook-collects- co...](http://arstechnica.com/business/2013/12/facebook-collects-conducts- research-on-status-updates-you-never-post/)

</comment><comment><USER29> Delete FB. At least mobile. Your brain will thank you.

</comment><comment><USER30> I deleted my Facebook account years ago. Don't miss it one bit.

<comment><USER31> I swear this exact text shows up in every conversation about Facebook wether it's related to the story at hand or not. It's almost like a meme at this point. Look we get it OP, but stating that by itself doesn't drive the conversation forward in a meaningful way.

<comment><USER32> Is it even possible to delete a FB account? I know someone who deletes their FB all the time. Every time they log in there it is again.

Unless, if you don't log in after X months/centuries it's completely removed, but I don't see why FB would do that. Apparently, according to what I've read here, even people without FB accounts have FB accounts (shadow profiles).

<comment><USER33> They're deactivating their account:

[https://www.facebook.com/help/214376678584711](https://www.facebook.com/help/214376678584711)

this is vastly differently then deleting the account:

[https://www.facebook.com/help/224562897555674](https://www.facebook.com/help/224562897555674)

The delete can take up to 90 days to remove all traces, but it is gone and will not allow logins when the process starts.

</comment></comment></comment></comment><comment><USER34> This literally just happened to me. I was having a conversation with my girlfriend and then opened facebook and saw a status about the exact thing we were talking about.

Is it possible to know if this is happening on an android phone? I'd love to eliminate the possibility it is happening...

<comment><USER35> Just a status? Any reason to think it was anything other than a coincidence?

<comment><USER34> It was really, really uncannily related to what we'd been talking about and a friend that I don't see updates from that often.

<comment><USER33> Do you think your Friend was coerced into making the status update? Or do you think the status update was faked by facebook?</comment></comment></comment></comment>


Throwing away working code - <AUTHOR> http://startuplessonslearned.blogspot.com/2009/02/throwing-away-working-code.html

<comment><USER2> It definitely takes some clear thinking and maturity to decide to throw away a significant chunk if it turns out to be inappropriate, I've learned this the hard way (waited too long to do it).

</comment><comment><USER3> large chunks of code always suck huge!!</comment>


The DAO crowdsale reaches $30M with 18 days to go - <AUTHOR> https://daohub.org/index.html <comment><AUTHOR> The DAO is a decentralized "venture" fund based on the Ethereum blockchain. It is meant to invest in projects and collect rewards for investors.

The amount raised 9 days into the fundraiser already makes it the #2 largest crowdfunded project. [https://en.wikipedia.org/wiki/List_of_highest_funded_crowdfu...](https://en.wikipedia.org/wiki/List_of_highest_funded_crowdfunding_projects)

More on DAOs: [https://www.ethereum.org/dao](https://www.ethereum.org/dao)

</comment><comment><AUTHOR> Post never took off, but 4 days later it's at $76M. 14 days to go still.</comment>


Designing a RISC-V CPU in VHDL: Arty S7 RPU SoC - <AUTHOR> http://labs.domipheus.com/blog/designing-a-risc-v-cpu-in-vhdl-part-16-arty-s7-rpu-soc-block-rams-720p-hdmi/ <comment><USER2> It's a bit disappointing that, at a time all monitors have enough memory and brains to display images out of their own memories (to say nothing about doing complex image operations on incoming video in real time), they still need perfectly timed pixels.

<comment><USER3> I'd rather they be dumber. There's too much input latency in modern hardware as result of all the smart layers.

<comment><USER4> There was a post here recently that measured hardware latency of modern computers and 80s computers and found that, in a modern computer, in the time it takes for a keypress to leave the keyboard's circuits, an 80s computer has already drawn the letter on the screen.

A USB keyboard, incidentally, has more transistors than an Apple II.

<comment><USER5> I still have no idea what was so wrong about the RS-232 as the keyboard and mouse interface that it had to be replaced first with PS/2 and then USB.

<comment><USER6> IIRC, there were quite a few economic/technical reasons why PS/2 has gone largely extinct. RS232 and PS/2 are hardware interrupt driven instead of using polling like USB so they have always had a relatively high opportunity cost. Nowadays you'll only see PS/2 ports on workstation motherboards with higher end chipsets (which have more lanes on memory/PCIe buses, more hardware interrupts available before they need to be coalesced into software interrupts, etc).

Also, PS/2 wasn't (generally) hot-swappable back then. That annoyance and some extreme pricing competition in the 2000s drove everyone towards USB.

</comment></comment></comment></comment></comment><comment><USER7> Brownie points for choosing VHDL. :)

Nice article.

<comment><USER8> Though generally, mimicking imperative languages (C and Ada for Verilog and VHDL respectively) is just such a bad fit for the massively parallel, circuit combining nature of FPGAs.

For my private projects (admittedly the only setting I currently use FPGAs for), I've since discovered clash, which is essentially Haskell for FPGAs, and am impressed how much more productive I am, how much abstraction a more fitting paradigm opens up, and how much more terse and readable the code is as a result.

I came about searching for an alternative when I got fed up with reimplement the same AXI protocol schemes over and over again, mostly through slightly altered cut and paste, and have instead slowly been building my own library of commonly used patterns and AXI protocol abstractions.

However, the one thing I'm not so sure about is whether I would be able to effectively use clash if I didn't have any experience in using Haskell for software before, or whether I would be constantly banging my head against it just to finally give up instead. I suspect that the price of clash's power of abstraction is a very steep learning curve.

<comment><USER9> You should checkout Chisel.

[https://chisel.eecs.berkeley.edu/](https://chisel.eecs.berkeley.edu/)

Its a HDL in Scala. It used extensively by Berkeley for things like the Open- Source RISC-V Rocket and Boom SoC chips. There are lots of projects based on it and you have some of the best open-source cores implemented in it.

See:

\- [https://github.com/freechipsproject/rocket- chip](https://github.com/freechipsproject/rocket-chip)

\- [https://github.com/ucb-bar/riscv-boom](https://github.com/ucb-bar/riscv- boom)

<comment><USER10> And I'm going to throw out there that, unlike most of the "software language used as HDL" crap you see, Chisel is fundamentally different and better for it.

It's not really compiling Scala to a netlist; it's more that you're writing a Scala program who's output is a generated netlist. It's pretty fantastic to use.

<comment><USER9> Its actually a frontend that compiles to FIRRTL and the idea is basically that FIRRTL is like LLVM bytecode. All the optimization run on FIRRTL.

The idea is that eventually different people will use different methods to produce the FIRRTL.

<comment><USER10> Sure, but FIRRTL is a netlist format.

</comment></comment></comment></comment></comment></comment><comment><USER11> If you want an open source DDR3 controller to go with your open source SoC I recommend LiteDRAM. Its getting pretty polished.

[https://github.com/enjoy-digital/litedram](https://github.com/enjoy- digital/litedram)

</comment><comment><USER2> Someone really needs to publish a nicer font in an easy-to-integrate way for these projects.

This MDA/EGA/VGA thing hurts my eyes.

<comment><USER12> The article looks to be using screenshots from the FPGA IDE (Xilinx Vivado in this case), which is probably why they look so bad. The later snippets are using actual text and look a lot better, at least on my screen. FPGA design is really really complicated and most of the development environments for them look like a hot mess because they have to cover so much functionality. Vivado is actually one of the sleeker tools available.</comment></comment>


Jasmine - A better YouTube Solution for iOS 6 - <AUTHOR> http://itunes.apple.com/in/app/id554937050?mt=8

<comment><USER2> From the creator of "Alien blue" (a reddit browsing client).

[http://itunes.apple.com/us/app/alien-blue-reddit- client/id37...](http://itunes.apple.com/us/app/alien-blue-reddit- client/id370144106?mt=8)</comment>


How to retire at 30 on $1 million (2010) - <AUTHOR> http://ryanwaggoner.com/2010/08/how-to-retire-at-30-on-1-million/

<comment><USER2> Running an apartment building with 151 rooms is a business, not retirement. You have control because you have responsibility. You employ people to manage this, but they need to make a profit.

The article makes some funny arguments about adding value. You can raise occupancy from 80% to 95%, you can raise rents, you can cut expenses, you can hire new management. There is some previous owner who is selling this apartment building, why is the occupancy only 80% and not 95% now? How do you expect to raise rents, cut expenses and still have people move in?

And even if you can do that, and hire a new management team, can you really consider this retirement? Instead that's running a successful business.

Instead of "How to retire at 30 with $1 million" it could be called "How to purchase a business that may generate a profit by putting 1.3 million of 4.3 million dollars into the down payment."

</comment><comment><USER3> This guy is bit thick. Buying a whole index is like buying a piece of economy, which may sound a bit more plausible for people who don't get it, like this one. Also shares have a built-in hedge against inflation (not totally true, but "true enough").</comment>


The Ballad of Buster Scruggs: What Is Its Source Material? - <AUTHOR> https://lithub.com/the-ballad-of-buster-scruggs-what-exactly-is-its-source-material/ <comment><USER2> Don't waste your time on this article (sibling comments are correct in their criticism). Instead read the Slate piece that it links to:

[https://slate.com/culture/2018/11/ballad-buster-scruggs- coen...](https://slate.com/culture/2018/11/ballad-buster-scruggs-coen- brothers-steward-edward-white-girl-gal-rattled-full-text.html)

<comment><USER3> Thanks. The piece includes the short story, which is good.

</comment></comment><comment><USER4> Such a click-bate article. What sort of a lazy half ass article is this? Proclaiming halfway through a piece about the source material that you didn't read the source material because you consider that action "undergrad-lit- class-y" not in anyway leveling with your readers or being refreshingly honest which I expect the author assumes. It's just downright lazy and would in an actual undergrad lit class just get you flunked out right.

I stopped reading at that point. If you are going to write some selfindulgent price about what's you assume the source material might have been, have the common decency to post that up front so people can avoid wasting their time on your prose.

<comment><USER5> I suppose the irony criticizing an article because the author admits to not reading all of their source material, then admitting you didn't read the whole article is lost on you.

</comment></comment><comment><USER6> The author of this piece doesn't know the answer, having apparently decided that skipping reading the source material somehow makes for a truer article. Shame.

</comment><comment><USER7> this article's reputability is immediately damaged by the statement that _intolerable cruelty_ is a bad film. just because it is a romantic comedy, a romcom, people immediately dismiss its brilliance. _intolerable cruelty_ is hilarious and very much in the center of what the coen brothers do best. there is some fantastic dialogue and the on screen compatibility between the actors is fantastic.

"freddie...it's a negotiation."

reading further, i don't think this person knows anything about the coens. _o brother, where art thou_ is not really an adaptation of as much as it is inspired by homer. _no country for old men_ is almost a direct copy of cormac mccarthy's book. irreverent is a strange way to describe that film's handling of the book's story. and in talking about "true adaptations", the author fails to mention _true grit_ , which is even more of a direct translation to film of its source material.

<comment><USER2> Yeah I don't get hating on Intolerable Cruelty. The only Coen brothers film I truly disliked was Ladykillers. I didn't really care for Burn After Reading either, but I feel like I probably just need to give it another chance.

My favorite segment of Buster Scrugs was The Gal Who Got Rattled, followed by Near Algodones. I found Meal Ticket too long and too dark. Didn't care for The Mortal Remains all that much either.

Poor gal, she shouldn't ought to have done it. Oh my.

</comment></comment><comment><USER8> A tangential thought:

Most films I watch are entertaining. I have a good time and then it's over and maybe I'll think about the movie again when there's an Internet meme from it.

But there's films (and some games), Ballad included, where regardless of how much I (dis)liked them critically, they kind of stick with me. They randomly creep into my brain for days, weeks, months later. I've come to crave this kind of experience.

I wonder what this is. If it has a name. If others experience the same thing with the same films or if it's very audience-specific.

<comment><USER9> IME, it's possible to figure out with a little meditation. It's often that a scene communicates or typifies a word or concept exceptionally well. They often remind me to periodically question my "fixations" and whether they are healthy, productive, and compassionate - generally a moral gut check.

</comment></comment><comment><USER10> I can't take this person seriously. They are conflating a historical piece with something that endorses its mild at worst content.

Apparently trying to tell a story within the context of it's time is "misogynistic" and "racist." I fail to see how either could be the case here. Did the coens somehow fail because they had an intentionally weak female character who is being attacked by native Americans? Did no similar events ever occur in history?

The native Americans have a clear motive for attacking the wagon train as during the period of the film the American army was committing genocide against them.

Additionally, the travelers have reason to feel negatively about them as they are literally trying to murder them.

Is every movie now supposed to have strong, wooden characters who faces no challenges nor tragedies? Is that realistic? Additionally, is a film set in a time period inherently offensive because we disagree with the values of that period?

Can we not tell stories anymore?

<comment><USER11> When the article says:-

      adapted from a Stewart Edward White story [...]
      This one is a dreadfully misogynistic short story, full
      of equally dreadful stereotypes about indigenous people

That's actually the article's author criticising of the _original story_ which was published in 1901, rather than the _film adaption_

In other words, the article author is saying "The directors could have hewed more closely to the source material, but I can see why they made the decision they made"</comment></comment>


How do I find an entry-level software position? - <AUTHOR> Recent CS grad here.
Is it me or have entry level positions almost completely dried up in the software industry? Almost everything that I find requires 2+ years of experience. I apply anyway, but needless to say I'm not having any luck finding a job.
Am I approaching the job hunt the wrong way or something? I'm beginning to worry that I'll never find any sort of desirable employment simply because there are hardly any reasonable positions for me to pursue.
P.S. I'm looking for jobs in and around Chicago (this may be contributing to my lack of luck finding a job, but I don't think that's the case). <comment><USER2> I just hired 3 PAID interns that can turn full time, and they have 0 experience in the industry. These positions are out there, go knock on as many doors as you can. I've also hired many entry level developers.

What do I look for? I look for "can you code" as an entry level developer. I don't expect you to understand design patterns and all that crap. Can you hack your way around and get the damn code running? Great, you are better than 50% of the candidates. Have you taught yourself any new tech recently? Great! Do you know more than one language? Great! Have you finished any online course? coursera, udacity, udemy, whatever, just something or a book? Awesome. Can you show me some of your shitty code for a silly side project? Awesome. Are you passionate and willing to learn? Great! Do you know other things that you need not know such as Unix, DB, RESTful API, git, etc?

You already met everything I need in tech. The only other thing is to at least pretend to be a decent person during the interview be nice, polite, respectful, punctual, clean.

Go knock on doors.

<comment><USER3> Not to be that guy, but emphasizing that it's paid? Is that really necessary? What are the chances that it wouldn't be...

<comment><USER4> Well there are lots of unpaid internships out there, and the OP is looking for a paying job, so I think it was worth mentioning that the internships were paid

<comment><USER3> I guess it depends where you live. I have never heard of or known anyone doing an unpaid internship in real life. I have only heard of it in whispers on the internet.

</comment></comment></comment></comment><comment><USER5> If you're applying for jobs requiring 2 years experience and you don't have that on your CV then probably you don't make it past HR. The hiring manager never sees your CV.

Also, most jobs ask for a particular programaming languages and usually an architecture. i.e. Ruby on Rails.

If your CV isn't targetting the skills asked for in the job you won't get past HR.

CV --> HR --> Hiring Manager --> Interview --> Hiring Manager --> 2nd Interview --> Offer

You might be stuck at the 1st stage.

When I was starting out I got on a free course that helped with my CV and got me an unpaid intern poisition. I worked for free for around 9 months, then I progressed rapidly after that.

</comment><comment><USER6> Unfortunately it's largely a numbers game.

If possible, be willing to relocate for work. Most entry level seekers are applying to everything they can, all over the place.

Work any and all connections you have. Anything you can do to get past the HR filter will help 10x your chances.

If you are pinched for money, you can always look for temp to hire work. It's unstable and the pay is usually crap, but it will get you in the door at one or multiple places and will get you to broaden your network.

Keep at it. It's discouraging and terrible, and the process is largely broken, but its entirely possible.

</comment><comment><USER7> You can get some working experience doing freelance gigs. You could probably find better ones than upwork.com, but otherwise, do some jobs there, do bigger and bigger ones, until you've got a couple of 1-month long gigs and some good reviews. Then go after the 1-2 year work experience ones. The lower end you go, the less onerous the requirements are, and it's possible you can build up your work experience that way.

</comment><comment><USER8> I think that they're just not as advertised, but they're there. I'd consider applying for any non-senior position in your situation (and in fact I did when out of university).

Are you sure your CV is on point? Without much working experience you get to use up the space to show that you're actually "passionate" about your craft by stating anything remotely interesting you've done.

</comment><comment><USER9> Keep learning while you are looking.

I would recommend learning Rails or Laravel developing web applications. Those jobs are pretty interesting and they are out there.

Usually when you find jobs it's through someone you know or meet so go to meet ups, contribute to an open source community. Pick a side project to build so you have some samples of your work.

Landing your first job takes work, keep at it.

Good luck.

</comment><comment><USER10> Highly recommend Glassdoor. Also, I work with people who attended coding bootcamps and then used the connections and new skills (front-end development) to join an established startup.

</comment><comment><USER11> In the meantime, find projects on GitHub that are written in languages you know and fix bugs.

That may help give you cred, where you have a dearth of it, otherwise.</comment>


Why should you use Evernote? Because it will change your life - <AUTHOR> http://www.andrewgertig.com/2013/05/why-should-you-use-evernote-because-it-will-change-your-life

<comment><USER2> I've started using Evernote over a year ago after a colleague suggested it. I was looking for a simple way to manage to do lists and other arbritary information (especially from what I receive in e-mails). I use the non- synchronizing private notebooks for work related stuff - I don't use Evernote for personal reasons.

So far it has served the purposes for what I needed, I have only one gripe with it at the moment - and that is the copy/pasting of HTML like content that often ends up mangled (which is sort of expected from HTML based WYSIWYG like editing).

<comment><AUTHOR> I agree that the formatting of pasted HTML is not ideal, it's difficult to accomplish anywhere though because you are not actually copying the CSS styles with it unless they are inlined.

</comment></comment><comment><USER3> not to be a debbie downer, but though I read your article, I dunno that the reason you gave (being searchable) was really worth using Evernote. A text file on your desktop could do just as well, if that were all Evernote was good for, right?

I signed up for an Evernote account years ago, but instantly got turned off by the fact that my personal notes would not be private any more. Completely unrelated to the break-in they suffered, just the fact that now, my notes, that I carefullly and meticulously take, would be on the internet somewhere. That actually didn't do it for me.

<comment><AUTHOR> Evernote's searchability is key because it applies to everything you put in it. It uses OCR to make images with text in them searchable, it does the same with PDFs etc. So it really is not the same as a text file.

Why do you think your personal notes are not private in Evernote? Wouldn't that be the same thing as not using Gmail because your email is on the internet somewhere? I see your point but it's hard to get away from using the cloud to store personal data.

<comment><USER3> So in addition to agreeing with jackrabbit --

Other non-connected applications can use OCR. Maybe you can make a case for Evernote's specialty recognition of cursive or handwriting, but, in the end, it boils down to text. they sure aren't doing picture-searching

It's not that personal notes are not private, it's more that anything that is put on the internet's privacy is questionable. Whether it SHOULD be that way or not is up for debate, but if you upload something somewhere, unless that somewhere is a black hole, someone might be able to retrieve it, through legitimate or illegitimate means.

The thing is, Gmail is something I can't do myself. It facilitates exchange at a level reasonable for use (like, I can't duplicate the USPS, and I can't duplicate Gmail) -- of course, I could just send email from my own IP, but they offer something special, verification of my identity, and make it easier to send messages... Where is that special functionality from Evernote?

As far as using the cloud to store personal data -- It might be hard, but it's easier to avoid actively pursuing new ways to put personal data on the internet</comment></comment></comment>


How Git shines above Subversion at Merging - <AUTHOR> http://tuxychandru.blogspot.com/2010/01/how-git-shines-above-subversion-at.html <comment><USER2> I think this post is missing the point entirely.

The strength of git has nothing to do with "magically resolving all possible corner cases". That was never the intended purpose of git.

To quote Linus himself:

> The important part of a merge is not how it handles conflicts (which need to > be verified by a human anyway if they are at all interesting), but that it > should meld the history together right so that you have a new solid base for > future merges.

> In other words, the important part is the _trivial_ part: the naming of the > parents, and keeping track of their relationship. Not the clashes.

[http://www.wincent.com/a/about/wincent/weblog/archives/2007/...](http://www.wincent.com/a/about/wincent/weblog/archives/2007/07/a_look_back_bra.php)

In other words, there _are_ corner cases where git might fail to resolve conflicts automatically.

If git does manage to resolve some obscure corner case on its own, it would be mostly accidental; it's not why git is better than svn.

git would still be an awesome tool even if it didn't handle all the obscure corner cases.

<comment><USER3> The post talks about merging working correctly with renamed files. That's not a "corner case"; that's exactly what Linus is referring to in your quote.

The author could've given an example like this where he got a conflict and that would still be an infinitely better outcome than what Subversion's doing. You'd actually know that there are changes that you need to reconcile, instead of it silently succeeding.

He's not missing the point.

</comment></comment><comment><USER4> So the one "small" hole in the merging infrastructure in subversion is what makes git "shine" above subversion?

This issue is well documented and accepted as a deficiency - [http://svnbook.red- bean.com/en/1.5/svn.branchmerge.advanced....](http://svnbook.red- bean.com/en/1.5/svn.branchmerge.advanced.html#svn.branchmerge.advanced.moves)

Personally I have found subversions merging support to be good enough for the past few years and when you are introducing a modern VCS into a shop which has used VSS, CVS or PVCS for the past age it is a much easier transition for the users than git would be and provides much better merging support that VSS or PVCS users ever had!

<comment><USER5> "Don't rename files on a branch" isn't a small hole.

<comment><USER6> Really it's "don't rename files if you use branches at all" (because renaming a file in trunk will cause merges of outstanding branches to fail)

<comment><USER7> I wouldn't say "fail". You'll just get a tree conflict where SVN complains that it can't merge changes because a file no longer exists. In TortoiseSVN such a conflict is trivial to resolve, because the tree conflict dialog gives the option to apply the changes to the renamed file instead: [http://tortoisesvn.net/docs/release/TortoiseSVN_en/tsvn- dug-...](http://tortoisesvn.net/docs/release/TortoiseSVN_en/tsvn-dug- conflicts.html#tsvn-dug-conflicts-tree-1)

<comment><USER8> The article claimed that "svn merge --reintegrate" did not complain about a conflict, just quietly gave the wrong result. I'm the designated svnmerge.py entrail reader at work, and I've seen that make even more severe mistakes.

(They keep saying our git repo will be ready Real Soon Now[tm]. So looking forward to that.)

<comment><USER7> The article demonstrated a silent bug in a specific use case: merging a rename for a file which was modified locally.

I was responding to the generalization "don't rename files if you use branches at all". Things are not that bad. As I explained above, merging a change to a file which was renamed locally works fine: it triggers a tree conflict which is easy to resolve.

svnmerge.py was deprecated by the introduction of merge tracking in subversion 1.5, so I'm not sure why you are still dealing with that. You can migrate with svnmerge-migrate-history.py

</comment></comment></comment></comment></comment></comment><comment><USER9> That's the thing about _distributed_ version control systems -- merging is such a central operation that is _has to_ work well.

</comment><comment><USER10> Welcome to the not evenly distributed future.</comment>


Show HN:  - <AUTHOR> https://ki.tc/otf <comment><AUTHOR> /otf is part of my CLI for the web project; dubbed "Keep It Tidy Charles(ki.tc)". As the name suggests; it's an Ephemeral File Sharing(EFS)-service that was designed initially as an api. The system has random readable urls/path through what3words's api; rationale: easier to read > easier to memorize > easier to verbally share. The process is actually fascinating as it's generating a random longitude and latitude for any given upload, passes it to what3words api, and then, the 3 words are passed and stored. When the api fails, it falls back to 5 random characters. * There are two links for any given file: [1] The download link [2] The download page link. The download page link was created to circumvent file deletion due to services that request header info[many social media sites and IM services do this]. * File retention is 27 days. * Max upload size is 400mb. * It does lack aesthetically in some areas; something weird happens when you maximise and then minimize the video player for instance.

* Latest Changelog

Changelog ki.tc/otf Changed supports ssl with auto redirect [let's encrypt] 27 days TTL; unaccessed files will be terminated Upload size increased to 400mb [3.4.0] - 2018-10-21 Added Streaming videos; [videojs] - file will be deleted upon download request.</comment>