Don't use Scatterplots - <AUTHOR> http://www.chrisstucchio.com/blog/2012/dont_use_scatterplots.html <comment><USER2> Great points, but in the author's 2D histograms the use of 'color' to show point density is a little misleading. Color is not a scalar, it's a vector, and simply varying hue doesn't map out well to the human perception of lightness or intensity. It also doesn't allow for meaningful interpolation. If you take a look at <http://vis4.net/blog/posts/avoid-equidistant-hsv-colors/> you can see a strategy for building perception-aware color palettes/gradients, which would be much more useful in this case.

(actually it looks like this wasn't the author's decision, but something built into the graphing library)

<comment><USER3> Interesting article you linked to and something to consider certainly. That being said, I somewhat disagree with you about the histograms being misleading. His color scheme looks to be based much the same as thermal imaging. Which I find is quite clear about what are the 'hot' zones (or in this case high point density) and the 'cold' zones (or low point density).

<comment><USER4> I think zacharyvoase's point was that it might either emphasize or deemphasize how _much_ hotter or colder something was. Because the eye doesn't see blue->yellow->red hue changes consistently, what looks like a "big" change on the graph might actually be fairly small.

</comment></comment></comment><comment><USER5> Don't use Density Plots ..

... without seriously considering the benefits of a scatterplot.

Here is some data I plotted recently: <http://i.imgur.com/lvraM.png>

Yes, you need to be careful about overlap (transparency can help), but without a scatterplot, I would not see the sharp edges, or have my attention drawn to the outliers.

Density plots imply a model: By creating bins, square or hex, you are adding a layer of interpretation on top of the 2D data, which can be bad. Also the bins of this article have sharp edges (high frequency content) and add artificial structure. I think smooth density plots, not covered by this article, are superior.

Best of both worlds? --> <http://www.survey- design.com.au/graphs/density_plot1.png>

</comment><comment><USER6> Having spent quite a bit of time looking at geospatial data in density plots, I'd warn that visualizing overlapping point data as a density plot is not a panacea for this problem. As it turns out, the distribution of colors along the density vector is also critically important.

e.g. Support you have lots of data where there are overlaps that aggregate to a value of 2, then a few that overlap to 5, and one or two that overlap to a value of 9.

Depending on how you distribute the colors you may either:

1\. Only really see the colors at the low end and the high end of the spectrum, thus missing the 5 values (or they may be so near the color for 2 that you can't perceive them).

2\. Clearly delinieate between the 2s the 5s and the 9s, but it's not clear that the 5s are more than magnitude than the 2s, while the 9s are not as big of a multiplier from the 5s.

3\. Some other distribution that shows a different story on the magnitude of the density, but can be interpreted wildly differently.

Different color gradients can also be perceived by our visual systems differently, blue-to-red doesn't always cut it.

<comment><USER7> Putting the data in log-scale before coloring should improve your ability to visualize magnitude differences. (issue 2 that you raised)

<comment><USER6> Another great one is Jenks. It doesn't do much to help with comparing magnitude between colored classification levels, but it can help visualize the fact the categories exist at all.

[http://en.wikipedia.org/wiki/Jenks_Natural_Breaks_Optimizati...](http://en.wikipedia.org/wiki/Jenks_Natural_Breaks_Optimization)

</comment></comment></comment><comment><USER8> One of his complaints is about truncation and therefore overlapping points. It's very common when you have multiple points occupying the same space to add a 3rd dimension of color ala the hexplot or size of point... or just add "jitter", which is easy in R or SAS.

Though, as [http://www.statisticalanalysisconsulting.com/scatterplots- de...](http://www.statisticalanalysisconsulting.com/scatterplots-dealing-with- overplotting/) points out, past 10,000 points, jitter isn't enough and alternative plots are probably best.

</comment><comment><USER9> (Going off-tangent here...sorry, hacker news makes me think like this). Chris, what do you think about setting up a stats-plotting-as-a-service kind of thing? People would give you data in some acceptable format and you chart it up and send it back? I am thinking of this more as a consumer service. At work, every once in a while, I get data back from testing and then I am scrambling to use gnuplot to chart data because using charts allows me to quickly visualize patterns and communicate the result of the data to others. But, I am not an expert at this and am continually refreshing my rusty knowledge and have always thought it would be great to send data out to someone who is comfortable doing this and get it back charted from them.

Note that I am thinking small (every once in a while; small amounts of data). Something like a subscription service for a small monthly fee. Or the ability to do one time payment for one-off jobs. Things I would look for in such a service are (1) willingness to clean data a little bit (most times I get excel spreadsheets, but I always need to do some tweaking to how the data is presented) (2) support to do a NDA -- this is mostly so the company I am with doesn't get annoyed that I am sending sensitive data to a random outside service.

My needs might be small, but to me it seems like this might be a valuable service for organizations that continually handle large amounts of data.

Just a thought...

<comment><USER10> We've got a service like this. We're currently working on a new version. See <http://prettygraph.com>

</comment></comment><comment><USER11> The guy who made the original scatter plot is even a math teacher... yikes.

The post author provides some nice python code to create the plots, though the choice of hex bins when the original data was rounded to the nearest integer is a bit strange. Square bins set to some integer size would be better I believe.

If you're using Matlab, I've recently found cloudPlot ([http://www.mathworks.com/matlabcentral/fileexchange/23238-cl...](http://www.mathworks.com/matlabcentral/fileexchange/23238-cloudplot)) to work rather nicely for "dense scatterplots".

<comment><USER12> See also [http://www.mathworks.com/matlabcentral/fileexchange/19280-bi...](http://www.mathworks.com/matlabcentral/fileexchange/19280-bivariant- kernel-density-estimation-v2-0/content/html/gkde2test.html) amongst other density estimators, for a nice smooth plot of the estimated joint density.

(Downside being there's still a somewhat-arbitrary choice to be made for the smoothing bandwidth, and it's not quite as visually obvious as the choice of number of bins in a histogram plot.)

<comment><USER13> Somewhat arbitrary, but not completely. There's some heavy math that suggests sqrt(n) scaling in the 1-d case.

</comment></comment></comment><comment><USER14> I've never seen a data visualization tool that is universally applicable, so a simple edict like "don't use scatterplots" is a bit too simple. This hexagonal plot looks cool for the problem under question but there are obvious cases where it would be unnecessarily complicated and less informative that a scatter plot. There's a reason why all of the various plot types were invented.

<comment><AUTHOR> Under what circumstance do you believe a scatterplot is superior to a density plot?

<comment><USER15> If your data can be displayed without points overlapping, a scatterplot can display all the information, while a density plot will always display only a summary of the data. The larger your grid size, the greater the loss of information.

<comment><USER16> And, by adding noise ("jitter" or "dither") to each point, you can still use a plain scatterplot even for many kinds of overlapping data.

It's simple to do and mimics reversing the effect of truncation of the data (at least for continuous quantities). Just use uniformly distributed values that are as wide as one bin width.

For most purposes, I prefer adding dither, and then using transparency, to moving to a density plot, for exactly the reason you mention -- the density plot introduces another parameter, the smoothing method, which puts another layer between you and the data.

</comment></comment></comment></comment><comment><USER17> Or use scatterplots with points that have opacity < 100%. C'mon there is still a bunch of cases where a scatterplot is much much clearer than that kind of 2d histogram, for example when there is a non linear relation with very high correlation.

<comment><AUTHOR> Here is a hexbin of a highly correlated nonlinear relationship (y=x^2 + gaussian(0,0.1)), together with a scatterplot with opacity:

opacity=35%: <http://i.imgur.com/VB1E0.png>

opacity=20%, and 2x bigger bins for the hexbin: <http://i.imgur.com/yDR3j.png>

Which image do you think better displays the image?

Just to make it interesting, I set it up so that the x values are not uniformly distributed. That's very easy to see in the hexbin, but still hard in the scatterplot.

The hexbin will always have a higher visual resolution than the scatterplot because the hexbin uses multiple colors to differentiate different densities. The scatterplot uses only one color.

<comment><USER18> Just FYI, the colormap you picked is terrible. There's many experiments to back this claim:

[http://gvi.seas.harvard.edu/paper/evaluation-artery- visualiz...](http://gvi.seas.harvard.edu/paper/evaluation-artery- visualizations-heart-disease-diagnosis)

[http://www.jwave.vt.edu/~rkriz/Projects/create_color_table/c...](http://www.jwave.vt.edu/~rkriz/Projects/create_color_table/color_07.pdf)

Colormap design is arguably as hard as visualization design. My favorite go-to place for them is <http://colorbrewer2.org>, but if you need to know only one thing about them, it's that varying hue continuously does not work nearly as well as you might think it does.

In addition, the fundamental reason scatterplots are bad, even with opacity, is essentially that opacity gives rise to an exponential relationship between overplotting and transparency.

There exists an alternative solution, which is to use additive blending and an adaptive linear colorscale, from zero to maximum-overdraw. Unfortunately, at present there exists no data visualization toolkits which support this.

<comment><USER17> >In addition, the fundamental reason scatterplots are bad, even with opacity, is essentially that opacity gives rise to an exponential relationship between overplotting and transparency.

can you elaborate on this, I don't see why it would not be linear.

>There exists an alternative solution, which is to use additive blending and an adaptive linear colorscale, from zero to maximum-overdraw. Unfortunately, at present there exists no data visualization toolkits which support this.

I think this _might_ be done in Mathematica, since Graphics objects can be manipulated symbolically, but I might be wrong.

<comment><USER18> It boils down to the (very reasonable) way alpha blending works. Alpha was originally designed to always lie between zero and one, which for compositing makes sense. For scatterplot colormapping, not so much:

If you create a plot with opacity alpha, and which puts N points on top of each other, the remaining 'transparency', that is, the resulting opacity is

1 - (1 - alpha)^N

This is an exponential, which has the unfortunate feature that it's flat for most of the regime, and then spikes in a relatively short scale. The spike is where we get color differentiation (different opacities get different colors). That's bad: color differentiation should be uniform across the scale.

I'm pretty certain Mathematica doesn't do this right either, because it's a pixel-based technique that requires frame buffer manipulation. Instead of rendering with the usual blending operation, you do everything with additive blending, compute the maximum overdraw, and then color-scale linearly.

</comment></comment></comment></comment></comment><comment><USER19> Choosing the right tool for data visualization is important and scatterplots aren't good for everything, but they have their place and can be really useful. For instance this scatterplot from carsabi.com shows a lot of very useful information: <http://cl.ly/GvnM>

Scatterplots can be a lot more useful when you do things like display the points in different colors or highlight a specific point of interest as in the Carsabi scatterplots.

Do you really think scatterplots should never be used? I think a headline like that takes away from your credibility. The problem isn't with scatterplots, it's choosing the right tool for the job.

<comment><AUTHOR> "Never" is a strong word, but I wanted a succinct title.

I think the default should be a density plot. It's only in special cases that a scatterplot would be appropriate. For example, that Carsabi plot actually works well, due to the fact that the reader is interested in finding a specific data point rather than understanding the global behavior.

<comment><USER15> I think the default should be the method that displays the most information. Why hide information if you don't have to? In the case of one dimensional data, a dotplot shows the reader everything. Using a boxplot reduces information content, mean-plus-errorbars reduces this further. The mean plus errorbars imposes a probability distribution, which may be wrong, it doesn't reveal a hidden truth.

The same holds in two dimensions. Show me all the data, and include a regression line or a spline to highlight a trend. Only start hiding information when the scatterplot becomes misleading. That is, when overplotting prevents me from accurately assessing the actual distribution of the points.

Jumping immediately to a density plot also restricts me to your interpretation. The original data is lost. With a scatterplot, the raw data can be recovered from the plot, so i can do my own analysis should i be interested. This is common in meta-analyses that extract data from multiple published papers. If those original papers had used density plots instead of scatterplots, reanalysis will require direct access to the underlying data. Once the original author dies, or loses the data, all further use of the data is lost.

<comment><USER20> The original data would be well represented in 100x100 matrix. since the data (grades 0-100) is already discrete. Basically the first picture in the article with a alpha setting that matches 1 (1=opaque) when multiplied with the maximum number of entries per field. e.g Max entries = 5 => alpha = 1/5 = 0.2. Alternatively aggregating for 10x10 20x20 25x25 50x50 would work to if the data is too sparse. There is in need for Hex binning in this case!

Best practice: [http://www.nytimes.com/interactive/2012/05/09/us/politics/sa...](http://www.nytimes.com/interactive/2012/05/09/us/politics/same- sex-marriage.html)

<comment><USER18> Alpha doesn't work additively for pretty much anydrawing packages:

<http://en.wikipedia.org/wiki/Alpha_compositing>

When overplotting, the usual compositing operator gives a final alpha of

1 - (1-alpha)^N

So your alpha = 1/5 overdrawn 5 times would give a final opacity of ~0.673. By its very nature, there is no alpha < 1 which when composited together a finite number of times gives alpha = 1.

<comment><USER20> I was aware of that this approach towards alpha was oversimplistic to begin with, should have pointed that out. Thanks for posting the correct formula.

</comment></comment></comment></comment></comment></comment><comment><USER20> Oh boy, this is apparently the scattershot approach to data visualization.

Always explore you data first and than use the visualization that best transports your intended message. (Yes there should be one, why else are you making a graphic?)

<comment><USER21> Alot of us make visualizations during data exploration, where we don't have a message.

</comment></comment><comment><USER22> What's really weird is how different 2008 and 2009 are for "same teacher, different subjects".

In 2008, it looks like a blob (no real trend). In 2009, some teachers obviously learn how to do well in VAM (excelling at both math and english "teaching"), and others don't. I guess that some teachers learnt what it takes to get good VAM scores.

<comment><USER23> As a (UK, math(s)) teacher I have to warn you that gaming the scores will inevitably happen when you have scores and those scores may impact on career.

<http://www.ofsted.gov.uk/resources/mathematics-made-measure>

Long, UK specific, but possibly of interest to those with school age children in educational systems using these metrics.

PS: most of the data I plot on scatter plots has low density and high variability so little stacking.

</comment></comment><comment><USER24> Other ways to handle this include transparency, smaller points, and jittered data.

I used to like the concept of density plots, but more and more I feel that they can be misleading. Plot your actual data if you can; use summarizing models as a fallback and a second step.

</comment><comment><USER25> One could hybridise the two - a scatter plot whose points change colour as a function of proximity (the equivalent of a density plot with 1px wide bins and the background colour for zero).

This would preserve the outlier flagging of a scatter plot while alleviating its risk of obfuscation via density. Plus, one need not worry about picking an appropriate bin width any longer.

<comment><USER26> You still have to pick a proximity radius for coloring. That's really just the bin width (or bandwidth of a density plot) in disguise.

<comment><USER25> True, but this is primarily to address the decreased ability of a density map to indicate outliers. One could also have the proximity be a power function over the entire field (one could do this with density, too, but it makes more sense with a point).

</comment></comment></comment><comment><USER27> Sometimes I've added random jitter to scatterplots to get something that represents density better when data is truncated.

</comment><comment><USER28> Would the stuff you guys are talking about in this thread be statistical analysis techniques or something more along the lines of data visualization? I ask because I'm fascinated by it and would love to find where I can learn more. I'm just not sure where to even start.

</comment><comment><USER29> Wen plotting dense, largely unconstrained data (i.e. not on a 0-100 scale), I've found a hybrid scatter/density plot to be nice. Basically, for each point you do a kernel density estimation, then color the points according to that. For the sparse data, the scatter plot seems just as useful, but at the dense parts, you seamlessly switch to color as the useful metric. Something like this: [http://www.mathworks.com/matlabcentral/fileexchange/8577-sca...](http://www.mathworks.com/matlabcentral/fileexchange/8577-scatplot)

</comment><comment><USER30> The point about the scatterplots is valid, but if he thinks a correlation of .26 (same teacher, subject and year, but different grade classroom) is sufficient to justify VAM, which arguably has no validity let alone reliability, then it's impossible to take him seriously.

</comment><comment><USER31> Just add a jitter.</comment>


Uber Elevate [pdf] - <AUTHOR> https://www.uber.com/elevate.pdf <comment><USER2> This is a cool idea but I would really like my car at the destination so I don't have to rely on uber or buses and light rail. Surely there could be an efficient way to load you and your car onto a train?</comment>


Mastering Linear Algebra in 10 Days: Astounding Experiments in Ultra-Learning - <AUTHOR> http://calnewport.com/blog/2012/10/26/mastering-linear-algebra-in-10-days-astounding-experiments-in-ultra-learning/ <comment><USER3> Cal Newport has the funniest definition of "mastering" and strangest definition of "world's most efficient studiers" (another blog post of his from a couple years ago)

<http://news.ycombinator.com/item?id=2658927>

I have ever seen. The shtick is getting old. Gee-whiz posts about a dilettante ramping up to a beginner's knowledge of a subject with little time and effort have nothing to do with the really challenging learning tasks in this world.

I'll be impressed when I see a headline like "Middle East diplomatic issues resolved by undergraduate who completed one course in international relations" or something like that. Show me someone who has solved a genuinely hard problem before proclaiming a new breakthrough in learning. For a refreshing change of pace from the usual blog post on quick-and-dirty learning, see Peter Norvig's "Teach Yourself Programming in Ten Years"

<http://norvig.com/21-days.html>

or Terence Tao's "Does one have to be a genius to do maths?"

[http://terrytao.wordpress.com/career-advice/does-one-have- to...](http://terrytao.wordpress.com/career-advice/does-one-have-to-be-a- genius-to-do-maths/)

for descriptions of the process of real learning of genuinely challenging subjects.

<comment><USER4> I have several concerns about the original article, but if the point of this guest post on Newport's blog is "look, I was able to grind through the MIT requirements in a year with reasonable success" it is not particularly damning that all the person achieved is a beginner's knowledge in several courses. Hopefully readers realize that this is not necessarily such a wonderful achievement while simultaneously respecting the effort made.

My bigger concern is that the overt focus on "time-compressed" learning is that it will devolve to pure hucksterism with no attempt to move beyond anecdote to real research and thought as to how teaching (either by self- directed methods or to students in a traditional way) can be improved. People seem so interested in short-cuts. I personally want to an "efficient studier," but what I mostly mean is that I do not want to waste time using bad learning methods or materials.

We do know that improvements in the learning process are out there - <USER2> books, spaced repetition for recall, better guidance and opportunity for early learners than they get in schools in subject-specific areas, etc.

These types of posts along with my reading of immersive learning experiences (like your own math programs, dual language programs, "programming school" opportunities) has me thinking about the innate value that immersion offers to the learner.

Would linear algebra learners be better off to spend 4 weeks focused on only that subject with a professor/mentor, a textbook, and deliberate practice than a single semester of normal lecture instruction?

</comment></comment><comment><USER5> There was effectively nothing about linear algebra on that page. After some link followings, it appears that <http://www.scotthyoung.com/mit/1806-exam.pdf> is the exam which the student was happy with (having done worse on the first version). The final score appears to be 66 out of 100.

Based on that test, I think the title is link-bait as it isn't "mastering linear algebra" but "passing an introductory algebra course."

<comment><USER6> I am quite confused looking at that final; it seems incredibly basic for even an introductory course at MIT (that _might_ have been a midterm when I was the graduate instructor for linear algebra at Berkeley).

<comment><USER7> 18.06 is the very basic, more applied version of linear algebra at MIT. About half of the students that take it aren't in the math department. The more theoretical option is 18.700. And then there's 18.701, Algebra 1, taught by Artin using his book. The latter two are almost completely theoretical/proof based.

<comment><USER6> To be clear: I mean that it seems too basic even for a final exam of an introductory applied linear algebra course for non-majors.

</comment></comment></comment></comment><comment><USER8> This is written from a student perspective, where "mastering" means passing the exam. I'll stick with Norvig's 10 000 hours.

<comment><USER9> Agreed. I had an interesting experience recently when I decided to "relearn" math. I had taken calculus courses in high school and university, and aced all the exams. But when I eventually came back to calculus out of personal interest, I realized that I didn't know what a derivative was! I didn't know what a limit was! How did I pass those classes? And if someone who got the highest marks didn't learn anything, what about the people who were actually struggling? How many people in that class actually learned anything?

And I could say the same for my science classes, foreign language classes, etc. I don't think you really learn anything unless you really want to understand the material and you work hard to do so. And if that were the case, hour for hour you'll get what you put into it. You wouldn't be studying to an exam. You wouldn't be satisfied with 60%+ on that exam. You wouldn't restrict yourself to a specific curriculum. You wouldn't put time limits on your learning. Instead you would learn the thing that you decided you wanted or needed to learn, and however long it took for you to really understand it, that's how much time you would spend.

<comment><USER10> The real test is how well you were able to pick up the concepts again. Not using it for, how ever many, years it's natural to forget. However if you can quickly reacquire the knowledge then you did learn it atrophied so to speak.

<comment><USER9> But I really think that I never learned those concepts in the first place. All I learned were rules to solve the problems given to me. For example I actually did remember the "chain rule". But those rules had no meaning to me, just moving numbers around. Which means I would never be able apply it to anything. For example, calculus clearly has applications all across basic physics. But I never really made that connection. All this makes me think that exams are much, MUCH less effective at measuring comprehension than we think.

</comment></comment></comment></comment><comment> Su-Shee I suggest reading very carefully.

I absolutely believe what he writes, because he's quite precise about his experiment and how he did it and this really works for a couple of reasons:

* This guy isn't 20 anymore. He has actually explored and learned and trained "productivity and focus" which he blogs and writes books about - so he doesn't start like a 18 year old directly from school, unexperienced maybe in this level of focus and discipline.

* He was pragmatic in his goals - very much so. He didn't write "becoming the world's foremost expert in linear algebra" but "passing an exam". And so he did. He also didn't write "passing everything with a top grade" but "just pass, if better - wonderful".

* He actually did his math on "hours to put in" - a semester doesn't take full 6 months, you usally don't attend lectures/lab every day 3 hours a day but 1-2 times a week, 2 (university) hours plus preparation. If you carefully add this up, you actually get a surprisingly low count of actual course/lesson hours.

* Taking in a course in a focused manner is actually quite efficient and helps you (at least it does for me) follow the material without interruptions. You also can repeat as often as you like (he mentions a fast forward and replay button in his TEDx talk) - which btw. makes part of the success of e.g. Khan university material.

* He also put some effort and training into the right way of learning and _that_ pays off massively in terms of speed.

Also, one of the points he is actually making is part of what most of you critizise: Going through the list of MIT requirements is something different compared to "becoming an expert in X" - don't mix that up.

</comment><comment><USER11> Learning for exams and learning for yourself are obviously different kinds of activities, even if the level of depth and rigour are similar.

For maths-heavy subjects, I'm not really inclined to believe that traditional exams are the best way to assess a student's knowledge and understanding of the material (especially with regard to rote memorisation). Exams in such subjects haven't changed fundamentally in many many decades, even though we now have lots and lots of new things we could do with them.

For instance: do more with computers - like getting the students to solve real-world, many-tentacled, hairy problems by numerical methods, rather than giving them some carefully pruned equation that just happens to have nice analytical solutions. Or introduce more computer-assisted mathematical modelling (e.g. use classical mechanics, to start with). Or on the pure front, teach students to write or at least understand some interesting automated theorem prover.

Stuff like that.

I suspect that traditional exams have survived simply because they serve their purpose: a percentage of exam-takers fail the exam (which allows the exam- setters to claim that their standards of assessment are rigorous), and a fair percentage will pass the exam, some with flying colours. Whether or not the actual learning goal was achieved has not been determined, since the exam is deemed to be the only instrument that can measure that.

<comment><USER12> Yeah, I find the best learning happens when you put that knowledge immediately to work in a real world problem, and preferably one that's meaningful to you, as opposed to most term projects. And that's where learning for yourself makes so much more sense; if you're learning for yourself then you probably already have a real world application for it which got you into the goal of learning it in the first place.

</comment></comment><comment><USER13> My best bursts of rapid learning are almost always project/puzzle driven. I didn't, for instance, set out to master FFT directly, but it seemed like something that could improve my abysmal performance on a Project Euler that I was working on, so I looked into it. I question (open-mindedly, not snarkily) the efficiency of ploughing through a course or series of courses. On the one hand there's the possibility of cross-pollination that having all sorts of cool bits of knowledge and techniques fraternizing in one's head for as long as possible promotes. On the other hand, there's the sense that the most efficient learning sequence is the one that matches the actual sequence of problems as they present themselves. Just-in-time learning of helicoptering, but if and only if you find yourself in a rooftop gunfight, as it were. Of course, then the issue becomes predicting forthcoming problems with enough lead time to learn the solution.

</comment><comment><USER14> I also am inclined to believe much of this. I actually did the same thing while at MIT for chemical engineering. Took all the required freshman through senior level classes at the same time, each semester, and finished all the requirements for a chemical engineering degree in a year. I loved MIT for this reason - they had no rules/regulations on the number of classes you could take in any semester, and they didn't enforce prerequisites/corequisites. Very different than other institutions I've trained at. I was still able to participate in extracurricular activities and develop relationships with lots of people.

It's true that I didn't attend a lot of classes (since they all overlapped anyways), and had 2-3exams virtually every week. The only issue I see is that there is only so much you can do online. I also did the same thing with Chemistry and Biology, which had lots of laboratory classes, and I don't see how one could gain the practical experience of putting knowledge to work in those fields without a wet lab class. EECS however is amenable to this (for the most part - likely hard for an optics laboratory), and most of my EECS labs were really done in Athena clusters instead of a distinct laboratory.

</comment><comment><USER15> People are kind of picking apart his use of the word "mastering," but I'd say that the crux of the article is spot on when it comes to learning techniques.

As an aside, I've never heard it called the "Feynman Techniques." However, one of my favorite things in the world is the so called "Feynman's Algorithm": (1) Write down the problem. (2) Think very hard. (3) Write down the answer. I just found to hilarious, but I digress.

There are two points of his with which I agree 100%.

Firstly, the process of writing a short summary paragraph of what you just read after reading a chapter or big section of a technical book. There is actually a fantastic book -- maybe one of my favorites of all time -- called, somewhat strangely, How to Read a Book. It's all about _very_ active reading over passive, almost to the point of having a "conversation" with the text you're reading.

Ever since reading that book, I've gotten into the habit of writing a summary of each thing that I read. It really forces you to confront whether or not you "got" the point of what the book is saying. I usually find that there are quite a few bits that I either missed, or didn't quite understand, at which point I go through and search for the pieces I'm missing.

Secondly, looking at all of the low level pieces to understand the whole. This is something Salman Khan, of the Khan Academy talks about in (I believe it was) his TED presentation. Quite often, I find that there is some early concept that I glossed over which is slowing my understanding of the current material significantly. For me, doing this makes me being 'honest' with myself over the state of my current understanding -- which was kind of hard at first when I took this new approach to learning. So much of my 'ego' seems to be unfortunately wrapped up in 'what I know,' and thus I convince myself incorrectly that I do understand something, even when I don't, just because it's something that I "should" already know. Admitting to myself that I didn't understand, for instance, some basic math concept that I should have learned in high school was somewhat difficult -- as odd as that may sound. I suppose I have a fragile ego! But sometimes, getting a good grasp on my modern course work, meant stopping what I was doing, and going back a couple of levels and starting at the beginning.

The question of "What do I need to know in order to understand this" is, I find, an extraordinarily powerful one.

</comment><comment><USER16> Although this is an accomplishment and there is some practical advice of value in the post, the "rules" he posts include correcting his own papers and tests and a minimum 50% passing grade.

<http://www.scotthyoung.com/blog/mit-challenge/>

Would be more compelling if he was not selling books. Nothing wrong with making a profit but I'm just saying...

</comment><comment><USER17> Ignoring the semantic controversy on 'mastery' and 'expertise', here's my story.

I dropped out of a CS program after first year. I was the classic case of a student who had always been told he was brilliant, so I never worked very hard. In high school, I coasted along simply on a fantastic memory, often 'studying' for the final exams that determine graduation the night before. I never learned how to learn.

Going to college was like being thrown into a bath of cold water. I had never been particularly conscientious, so being in an environment where I was now responsible for my learning was new to me. I skipped lectures, forgot homework that was due, turned in coursework late; the usual suspects. On raw talent though, I qualified for 2nd year, only failing Pre-Calculus. (I skipped the classes and tried to learn math from 1st principles. Ugh...)

I got a summer job at a small telecom startup. By time 2nd year rolled around, my student loan was denied, so I dropped out. I'd always hated school, so I didn't care. I never applied for leave of absence, nothing. I just didn't show up in September. That was 2006.

I was 20 then. I'm 26 now. I've had a lot of time (6 years!) to reflect on why I did so poorly despite being talented (not being conceited; my lecturers in 1st year said as much). There are quite a few reasons; but the major one is that _I didn't know how to learn._ So if something didn't immediately click, I'd give up in frustration, and decry the teacher as an idiot who couldn't teach (oftentimes true; but irrelevant). I didn't know there was another way.

Being around HN and places like LessWrong which exposes you to so many thought-leaders brought about some interesting side-effects, which culminated earlier this year. Upon reading an article on LW entitled "Humans are not automatically strategic", which was a reply to a Sebastian Marshall article "A failure to evaluate return on time fallacy", I had an epiphany that being _systematic_ about things was the route to accomplishing great things. "Rationalists should win", the LW meme goes, and it's correct. I came to realize that _for every goal, there exists an efficient path to achieve it._ My task was to find that path, and execute ruthlessly upon it.

Since then I've made leaps and bounds in my personal development. I still slack off sometimes, but I won't fall into my old perfectionist way of thinking that I'm a failure. It's better to be 80% there than 0%.

I made the decision a few weeks ago to get my CS degree, albeit at a different, larger university. Since then, I've been devouring articles like this one. I recently bought two of Cal's books and wanna sometimes slap myself when I realize that if I had had this knowledge and the discipline to implement it 6 years ago, my life would be so much better. But c'est la vie. These articles on meta-learning are priceless.

So if you're in school now, or are going soon, pay attention to articles like these, Here are a few gems I've dug up recently: <http://news.ycombinator.com/item?id=3427762>

<http://news.ycombinator.com/item?id=818157>

[http://www.quora.com/The-College-and-University- Experience/H...](http://www.quora.com/The-College-and-University- Experience/How-do-some-people-get-near-4-0-GPAs-in-college)

[http://www.quora.com/Harvard-College/What-are-the-best- Harva...](http://www.quora.com/Harvard-College/What-are-the-best-Harvard- College-study-hacks)

<http://www.quora.com/How-do-top-students-study>

Thanks to knowledge like this from Cal Newport and others, I'm going back to college full-time as someone with an above-average cognitive toolset, and a myriad of experiences that will suit me. I'm _much_ more sociable, have a great eye for design having moonlighted as a freelancer some years back, and will now know how to engage my lecturers on an adult level rather than the kid I was 6 years ago. I'm going for a 4.3 GPA. I'm tempted to say wish me luck, but with tools like these, I'll _make_ my own luck.

This rationalist will win.

PS If y'all have more articles like this, let me know. If you wanna chat privately, email's in profile.

EDIT: formatting; clarity

<comment><USER18> > I didn't know how to learn.

I'm not trolling here, but don't you think it's more accurate to say you didn't know how to work?

<comment><USER19> I disagree. College is the first time many students have to learn how to teach themselves.

I did quite well in a small town high school with minimal effort. Not perfect, but I graduated 11 out of 399. I can probably count the number of times I did _any_ work or studying at home on my fingers. During the day I listened in class, focused on school, and gave forth effort. It's pretty easy when classes move at a snail's pace lest any student get left behind.

When I went to college my first physics class kicked me in the face so damn hard and I didn't even see it coming. Lectures moved at blistering pace and entire chapters were covered in just one or two class periods. I remember going into the first big test, thinking I did well, and getting a grade so low the teacher pulled me aside and asked if I should drop it. Holy crap that was embarrassing.

After that I realized I needed to learn how to learn. I had to learn how to take the book, break everything down, ask questions when necessary, and master the material without guidance. It was a slow process but my scores got better with every single test to the point I tied for the highest score on the final.

You could argue that I had to learn how to work but I don't think that's accurate. I knew how to put in work rate. I was an Eagle scout and played on the high school soccer team. Both required tons of work. Learning how to learn certainly requires work and effort, but I think it's a skill the same as any other.

<comment><USER20> I can't speak more to the 'learn how to learn' statement.

I have a very similar story (Eagle scout/football and wrestling team) and had good marks in high school. I arrived at college and was destroyed by Calc 2 and having to learn at that pace when in high school I never studied at all.

After a rough freshman year I knew I had to change how I work and learn. My dad handed me 'Moon-walking with Einstein' and I found calnewport.com (along with other study blogs) and developed a study plan that I still use (I'm currently a Junior).

</comment></comment></comment></comment><comment><USER21> This person, Scott Young, did not "master" linear algebra. If anything, he mastered the _curriculum_. There's a difference, and in a year he won't remember a word of it.

He is, however, a master of self-marketing:

 _"To find out more about this, join Scott's newsletter and you'll get a free copy of his rapid learning ebook (and a set of detailed case studies of how other learners have used these techniques)."_

</comment><comment><USER22> I'm inclined to actually believing a lot of this.

When I got into university I found every course very easy, didn't attend any lectures, got all my workshops to run on the same day to reduce my face time and maxed out my free time to do whatever I wanted (work/friends/extra/etc). I'm a STEM major at a top 30 world ranked engineering school with good grades.

I've often asked if I could max out my classes and finish a degree within a year and a half - but I've never been allowed to skip more than a few subjects (tests/bugging the heads of departments).

University shouldn't be time capped or subject load restricted - people should be allowed to do as many as they wish - or you'll find more and more moving towards MOOCs instead.

<comment><USER23> (I'm a professor in a University, mostly for first year students.)

The problem with allowing the students to take all the courses they want is that some of them can't correctly measure how much work they can handle. So you will get horror stories like: "A student get enrolled in 3 times the expected courses, has too many homework, too many midterms the same week, and finally don't approve any of them."

Now we have a lost of students that come with questions like "I didn't approve Calculus I last semester. Can I enroll in Algebra I, Calculus I and Calculus II, so I don't lose a semester?" And it is hard to tell them, but most of the times the best thing is to take only "Algebra I" and approve that than taking the three classes and approving none of them.

I also understand that there are special cases, but should be evaluated case by case.

<comment><USER22> Agreed. I'd like to note I literally have zero problems with either system really.

I'm rather enjoying university right now and the lifestyle I have and am perfectly fine taking the scenic route. I learn plenty on my own and basically consider it a semi retirement stage :-)

<comment><USER24> I'm glad you feel that way. Those of us with jobs and families snicker at college students who claim to be "so busy". Life is good at this time; do your studies and use those massive amounts of free time to socialize and create new things.

<comment><USER25> > do your studies and use those massive amounts of free time to socialize and > create new things.

I never had massive amounts of free time at college. Maintaining a 3.9 in Computer Science without having the same amount of raw talent (and previous experience) as your peers is really hard, so I had to work my ass off. My schedule was close to 16 hours per day, 7 days per week (including classes). It was "Hell Week" for the ~4 years I was there. I worried my family and friends. It was unhealthy.

Trying to keep up with, frankly, __smarter__ (and, most likely, more intelligent) peers was a real challenge. I tried "Studying Smarter, not Harder", but lacked the skills from K-12 education to do so. Attending classes on "How to Study" and the like really didn't help and, no matter what I tried, I always fell back into investing more time as a solution.

I lacked the bio-hardware (read: brains) to go into Computer Science. I did it anyway. Don't forget about people like me!

> Life is good at this time

No, it wasn't. Not for me.

> Those of us with jobs and families snicker at college students who claim to > be "so busy".

I don't. I have more free time now that I'm married and working (a stable and well-paying job, thankfully).

</comment></comment></comment></comment></comment><comment><USER26> I'll leave the discussion of how embellished this post/blog/exercise of MIT Comp Sci in 1 year is to other comments. But! The explanation of Fourier transforms from Scott's notes (<http://www.scotthyoung.com/mit/fourier.pdf>) is one of the must succinct ones I've read. I've always understood what the transform does, but the nitty gritty on how the equation works was awesome

<comment><USER27> Yes, it was interesting to see it labelled out like that. I liked to think of the transform as a cross-correlation. You take a wave of a particular frequency (f) and cross-correlate it with your signal s(t) across all time. The fourier coefficients are the results of the cross-correlation, telling you how much of each frequency you have in your signal.

</comment></comment><comment><USER28> > However, eventually you'll reach a stopping point where you can't explain. > That's the precise gap in your understanding that you need to fill.

This _is_ a useful technique, giving motivation and focus. Though imperfect: it can't detect incorrect understandings that seem consistent. But to be fair, that's a tricky case.

</comment><comment><USER29> One of the achievements I am most proud of was doing a full year's university course in computer science in 6 weeks and passing. It was pure cramming though and very hard work. I got into a routine of full-on study from 9am-1am with short breaks every hour or so. 16 hours a day for 6 weeks.

Not something I would ever want to repeat and was first year level courses. Basically I was doing a correspondence 3 year degree while working full time. I got heavily involved in my work and decided that I wouldn't continue studying. Then with about 4 weeks to go to the 2 week final exams period I thought, what the heck let's give it a shot...

Amazing what focus and hard work can achieve!

</comment><comment><USER30> > he completed all 33 courses (...) in less than one year.

> That works out to around 1 course every 1.5 weeks

WTF? What kind of university imposes that you take only one course at any given time? It's not just linkbait, it starts from a wrong assumption. When you take many related courses simultaneously, you see the pieces meshing together and that helps learning. That's different from taking them in a serial manner.

<comment><USER31> "you see the pieces meshing together"

...or not. Depends on many things, among which the teachers are very important. If your lecturer thinks that his class is the most important and the rest of your courses are trash then you won't see "pieces meshing together". Been there, done that.

</comment></comment><comment><USER32> It's funny because the students at the competitive schools work their ass off for the entire semester/ quarter to learn this material. If you use the weakest possible definition for "learn," then you can claim you have learned anything you want. But that doesn't mean your skill will be comparable to someone who spent 3-5 months practicing non-stop.

</comment><comment><USER33> ...this guy really knows how tomuch puts the "bait" in "link bait" ...nothing about linear algebra in the article but the perfect title to hook the bank of HN fish ...congrats to the OP for pulling this one off :)

</comment><comment><USER34> I would imagine that the vast majority of students at Universities around the world who take Linear Algebra "master" it in 10 days. That is, the ten days before the exam, having spent most of the term drinking, socialising, falling asleep in lectures (or just staying in bed and skipping the lecture bit). I certainly know _I_ "mastered" elementary linear algebra in about 10 days.

</comment><comment><USER35> Not seeing this as link bait at all - more method for than what was being learned. Great read that gives a lot of interesting insights and methods - definitely not for everyone. A lot of the same information and ideas have been discussed on Study Hacks though it is great to see the provided examples.

</comment><comment><USER36> He may have accomplished something impressive, but I had trouble appreciating it because the article seemed so pretentious, and I found that distracting.

<comment><USER37> It is a marketing scam for a self-help eBook.

</comment></comment><comment><USER38> Dazzling title, even no Linear Algebra at all, but I like the systematic introduction about "Feynman Technique"

</comment><comment><USER39> I thought Linear Algebra was the easiest math-class?

</comment><comment><USER40> Master it in 10 days? Forget most in 10 more.</comment>


Google Search: **/* - <AUTHOR> https://encrypted.google.com/search?hl=en&q=**%2F* <comment><USER2> Doesn't seem limited to " * * / * ", but rather any search within asterixis that contain non-letter characters. E.g., "* '* ", "* @* ", ...

(Also, I'm not sure how to escape asterixis in HN comments other than adding a space after them.)

</comment><comment><AUTHOR> Thought this was a little odd. For me it just shows listings for local businesses. When I'm on a VPN, though, the search does not match any documents.</comment>


Acquiring administrative access to Azure's RedHat Update infrastructure - <AUTHOR> http://ianduffy.ie/blog/2016/11/26/azure-bug-bounty-pwning-red-hat-enterprise-linux/ <comment><USER2> > _Given no gpgcheck is enabled, with full administrative access to the Red > Hat Enterprise Linux Appliance REST API one could have uploaded packages > that would be acquired by client virtual machines on their next yum update._

And given how hard it is to detect backdoor software, this is a HUGE security blunder. This could have literally installed a rootkit on every rhel instance on Azure.

<comment><AUTHOR> That is correct..... I've another post I'll do in a few days that details how you can become storage account admin from being root on an instance.

</comment></comment><comment><AUTHOR> Follow up post for anyone that is interested... You've got root, can you get more access to the users azure account? The answer is yes. [http://ianduffy.ie/blog/2016/11/27/azure-bug-bounty-root- to-...](http://ianduffy.ie/blog/2016/11/27/azure-bug-bounty-root-to-storage- account-administrator/)

</comment><comment><USER3> I'm curious to know if a bounty was paid for this and how much.

<comment><AUTHOR> The bounty was paid out, leaving the amount as undisclosed but it was under 3500 USD.

<comment><USER4> That seems super low

<comment><USER5> Here we go.

</comment></comment></comment></comment><comment><USER6> There was a thread yesterday where lots of people were complaining about HSMs ([https://news.ycombinator.com/item?id=13031155](https://news.ycombinator.com/item?id=13031155)). I think this is an example where it would have helped to secure the private key in an HSM instead of the server itself.

Now the author states the keys have been rotated but now the next hacker know where to look.

<comment><AUTHOR> I'm not fully confident that they have actually been rotated....

<comment> Anthony-G Hi Ian. That was a very well-written account of a very serious vulnerability. I just thought I'd let you know that there's a typo in the closing sentence (being discussed here), "they claim to of rotated all secrets". That should be "have" rather than "of". It kind of threw me a little as I read it. Le gach dea ghuí.

<comment><AUTHOR> Perfect! Thank you Anthony, colloquialism slipping into my English.

</comment></comment></comment></comment><comment><USER7> So, it seems that the attack vector was that Microsoft was running RHUI Log Collector open to the public internet for some reason.

Considering that's from Redhat, and not Microsoft, I do wonder if this is a non sensible default setup issue and there may be many enterprises running this out in the open.

<comment><AUTHOR> That is very interesting @joneholland. Can you tell me more about the RHUI log collector? I thought it was something hand rolled.

</comment></comment><comment><USER8> Am I misreading this or does this really allow arbitrary packages to masquerade as legitimate packages?!

<comment><AUTHOR> That would be correct since GPG checking is disabled. Would just be a case of bumping the version number and releasing a package under the same name.

</comment></comment><comment><USER9> It sounds like they fixed the RHEL Update infrastructure, but they didn't fix this:

"Additionally, if you duplicated a Red Hat Enterprise Linux virtual hard disk and created a new instance from it all billing association seemed to be lost but repository access was still available"

<comment><AUTHOR> Debatable @shshhdhs, I can confirm they locked down access but I'm not confident the certificates were actually rotated. It would have meant they would have needed to push new SSL client certs out to every customer Red Hat Enterprise Linux Virtual Machine.

I'm confident that duplicating the virtual disk, certificates or installing the documented RPMs will result in repository access without being billed accordingly. It is considered fraud and I would imagine if one took large advantage of one would be disciplined accordingly.

</comment></comment><comment><USER10> Another reason to be away from MS cloud. Good work man.</comment>


Ask HN: Which websites keep status of new, upcoming or proposed TLDs? - <AUTHOR> Introduction of new TLDs give smaller startups a unique opportunity to grab a good domain name for their business.<p>I was wondering if there was any places on the web where you could view the status for new domains being introduced or domains in the process thereof? <comment><USER2> [https://gtldresult.icann.org/applicationstatus/viewstatus](https://gtldresult.icann.org/applicationstatus/viewstatus)

[https://newgtlds.icann.org/en/program-status/sunrise- claims-...](https://newgtlds.icann.org/en/program-status/sunrise-claims- periods)

<comment><AUTHOR> Great, thank you!</comment></comment>


The Pavlovian Response To Seeing Birthday Announcements On Facebook - <AUTHOR> http://www.techdirt.com/articles/20110826/01140715696/pavlovian-response-to-seeing-birthday-announcements-facebook.shtml <comment><USER2> I set my birthday to April 1, 1911 on March 31st, making April Fools Day my 100th birthday. However, when Facebook hid my age and just said "Today is your birthday" I was disappointed and changed it back.</comment>


Is anyone at the Global Future 2045 conference in NYC? - <AUTHOR> http://gf2045.com/

<comment><AUTHOR> And what do people think about this thing anyway?

Will we be able to evolve beyond our bodies by 2045?

[http://2045.com/](http://2045.com/)</comment>


Easter egg on German Apple website: Search your ToS, get insulting AIFF file - <AUTHOR>

If you search on the German Apple website for 'AGB', the German abbreviation for Allgemeine Geschäftsbedingungen or General Terms of Service, you get an AIFF file as your result.
The AIFF file contains a few seconds of sound in German saying 'Dummer Sack', literally 'stupid bag', a not to plesant term to call other people in German.
Step-by-step with actual search result:
1. http://www.apple.com/de/ 2. http://www.apple.com/de/search/?q=AGB&section=global&geo=de
FORMgⁿAIFFCOMM3á FORMgⁿAIFFCOMM3á@. ¼DINST< O0@╒SSNDgH■╓ à≡φ└δ«εL≤╩⌠1±ê≡╩⌠C∙f²≡{ueù ... http://www.apple.com/de/pro/profiles/otto/images/dummasackloop.aiff <comment><USER2> Ok, I'm flagging this. This is nonsense trying to muck up a controversy over nothing.

Common sense and a few seconds of research indicates this audio file of a music clip is from a musician profile on Apple's Professional Profiles website:

<http://www.apple.com/de/pro/profiles/otto/>

How did I figure out this URL? By clipping a couple levels of the AIFF URL:

[http://www.apple.com/de/pro/profiles/otto/images/dummasacklo...](http://www.apple.com/de/pro/profiles/otto/images/dummasackloop.aiff)

<comment><AUTHOR> Well, neither <http://www.apple.com/de/pro/profiles/otto/> nor the AIFF file exist anymore … maybe you need to be a native speaker of German to see the issue here although it's certainly not a controversy but mainly a funny occurrence.</comment></comment>


Insects Use Plant Like a "Telephone" - <AUTHOR> http://www.textually.org/textually/archives/2008/04/019899.htm <comment><USER2> This is not communication. The subterranean insects probably know nothing at all about the above ground insects feeding on the leaves.

Thinking about it evolutionarily in two ways 1) plants into which the subterranean insects unknowingly inject their distasteful chemicals live longer because above ground insects do not eat them to death, thus the subterranean insects live longer and propagate more.

2) in A Walk In the Woods, I read about trees that when chewed upon by an outside force, will produce a chemical that will drive the predator away. They can even send signals to other trees. This same chemical may be the one that fends off the above ground predators, but the subterranean ones are immune.

Anyway, there are many explanations for this phenomenon that do not include the conscious "use" of the plant to communicate with other animals of other species.

The article is misleading at best. using a telephone is a process which implies that the caller believes to some degree that someone else will answer the phone and a conversation can be had.

There's nothing of that sort going on here...

<comment><USER3> Well said.

I think they are trying to use the word "telephone" to mean any sort of communication or signaling between two entities, and that's just not a correct use of the word.

</comment></comment><comment><USER4> This is something that is going to happen very quickly:

You have a forest full of plants. These plants have insects living in their roots which are feeding on them. At some point in time, insects arrive and settle on the leaves of the plants. They start feeding, and the plants die on or two weeks later. So these insects move to other plants nearby - soon, all the plants in the field are dead, and then all the insects in the field are dead.

But let's say there is a single insect underground that feeds in such a way that the chemical composition of the plant is mildly changed from the standard plant. The above ground insects will avoid this plant, because it smells a bit different, and so this plant will be one of the last to die. As a result, it will be the dominant plan in the next generation of plants. In the next generation, the plant that emphasizes this trait will continue to win, and in a few generations, the evolution will occur.

This ties in with my theory that evolution can only occur under very high pressure, and it happens very suddenly. This slow evolution theory that is common now is wrong, in my opinion. I believe that mutation is common, but singificant changes happen only under immense pressure.

That's why humans are no longer evolving. That's why women still have to suffer at childbirth - midwives have removed the pressure.</comment>


Show HN: Texttop – Interactive desktop rendered in TTY with ASCII - <AUTHOR> https://github.com/tombh/texttop <comment><USER2> May be image quality could be improved by using whole unicode set, rather than just plain ASCII. E.g. braille-fonts, like in blessed-contrib[1].

[1] [https://github.com/yaronn/blessed- contrib](https://github.com/yaronn/blessed-contrib)

<comment><USER3> Aw, this is great but node.js is putting me off it :/ I was hoping I wouldn't have to install something so heavy for such a small utility...

<comment><USER4> so try [https://github.com/yaronn/wopr](https://github.com/yaronn/wopr)

<comment><USER3> That's interesting, but not very well-suited to real-time dashboards, is it? Seems to be for one-offs.

<comment><USER4> curling the markup is not for realtime. sometimes you can get away with curling in a loop with some sleep, and depending on the latency / scale you will get the same effect. however blessed-contrib can also be used as a server side component where clients curl for its endpoint (which is different than curling a static markup). that way it can return a live stream of data. you would use the same approach as [https://github.com/yaronn/wopr/blob/master/server/server.js#...](https://github.com/yaronn/wopr/blob/master/server/server.js#L27) where your present method implementation will run in some loop / timeout and keep sending data (by calling screen.render).

</comment></comment></comment></comment></comment><comment><USER5> This is so cool, very nice work! :D

I've been playing with writing little custom SSH server stuff recently as well—try this for example:

ssh hello.dn.ht

One thing I've been testing to get some extra vertical resolution on graphical stuff is the unicode half block character ([https://en.wikipedia.org/wiki/Block_Elements](https://en.wikipedia.org/wiki/Block_Elements)) with a separate foreground and background color. Works ok, but background color can bleed through depending on the font you use.

<comment><AUTHOR> Haha Wasn't expecting that, very cool! How did you do that? Have you some code somewhere? Thanks for the praise :) A big part of pushing forward with this was seeing the quality of @jart's hiptext[1] renderings, I find it creates much more aesthetically pleasing ASCII images than libcaca and libaa. It only uses those block elements, like your webcam video.

[1] [https://github.com/jart/hiptext](https://github.com/jart/hiptext)

<comment><USER5> Ok yeah wow, hiptext's output looks pretty fantastic. :)

I'm using my own text webcam thing: [https://github.com/dhotson/txtcam](https://github.com/dhotson/txtcam) .. not too fancy, just using greyscale with some shading characters to do some basic "dithering".

Also, if you're on Mac OSX and using iTerm2 version 3, you might be interested in its 24bit colour support. It's not that widely supported yet in other terminals unfortunately though: [https://gist.github.com/XVilka/8346728](https://gist.github.com/XVilka/8346728)

<comment><AUTHOR> Oh, that textcam code isn't too complicated and you get a really good effect from it.

I never even knew that there was true colour support in terminals, let alone how many terminals support it! Hmm, so I'd need to figure out if SSH sends the terminal's ability to render true colour. And then also update hiptext. It would make a really big difference to the "readability" of the texttop output.

</comment></comment></comment></comment><comment><USER6> I implemented this neat trick that gives 4x spatial and 2x color resolution using Unicode Block characters

[http://www.blitzcode.net/rust.shtml#term_gfx](http://www.blitzcode.net/rust.shtml#term_gfx)

[https://github.com/blitzcode/term- gfx/blob/master/src/main.r...](https://github.com/blitzcode/term- gfx/blob/master/src/main.rs)

I think it could work great for your software.

<comment><AUTHOR> Is that similar to this do you think?[https://github.com/jart/hiptext/blob/d50329e3b975ffac0209c0a...](https://github.com/jart/hiptext/blob/d50329e3b975ffac0209c0ade91120cec499918f/hiptext.cc#L134) Where `kUpperHalfBlock = L'\u2580'`

Because Hiptext is already supporting it, but when I enable the flag to support it, it crashes in Alpine, but not in my local env :/

<comment><AUTHOR> OMG, no, it's not the same, you go even further with 4 colours per char!! [https://github.com/blitzcode/term- gfx/blob/master/src/main.r...](https://github.com/blitzcode/term- gfx/blob/master/src/main.rs#L59)

<comment><USER6> You still only get 2 colors (bg & fg), but you get to chose for each of the 4 sub-pixel, it's a bit like DXTC texture compression

</comment></comment></comment></comment><comment><AUTHOR> Forgot to put the word "interactive" in the title. Edit: didn't know you could edit the title after submission.

</comment><comment><USER7> It's probably beside the point, you could double the number of pixels and foremost make them square if you used fore- and background color for U+2584 or go even further with U+2596 to U+259F.

<comment><AUTHOR> Hiptext supports this: [https://github.com/jart/hiptext/blob/d50329e3b975ffac0209c0a...](https://github.com/jart/hiptext/blob/d50329e3b975ffac0209c0ade91120cec499918f/hiptext.cc#L134) But it's crashing when I run in Alpine :/ I have hope because it works fine in my local Arch/ZSH env :)

</comment></comment><comment><USER8> Cool idea but I haven't been able to get it working. Docker is proving to be a real drag to install. Furthermore in the GIF on the README the text is illegible.

<comment><AUTHOR> The text only becomes legible at very high zoom levels. I needed to strike a balance when recording that GIF, so I didn't zoom in all the way.

</comment></comment><comment><USER9> NX / X2Go

<comment><AUTHOR> Had never heard of them, thanks. Do you have experience with them over connections that drop, eg; travelling on a train, or connections as low as 3kbps?

<comment><USER9> They work at least decently over connections that drop - I can't remember if they autoconnect or keep things alive like mosh or if it works more like screen. Low bandwidth is surprisingly good - not sure if I've gone as low as 3Kbps but I do remember working out of a hotel room in China to a Linux machine in Iceland over NX. The regular web was hardly usable but NX was fine!, even good enough to browse the web over.

<comment><AUTHOR> Wow, okay, that's good to know, another way around the Great Firewall. I will have to give NX a spin.</comment></comment></comment></comment>


How I Remember My YubiKey - <AUTHOR> https://blog.kamens.us/2019/06/05/how-i-remember-my-yubikey-take-three/ <comment><USER2> And to think I spent all that time attaching my YubiKey to my house keys, when I could have done this instead! How misplaced my efforts were!

<comment><USER3> I have left my yubikey attached to my house keys attached to my computer at work and gone home more than once.

I now perform an abbreviated version of the macarena before leaving the office.</comment></comment>


The Embedded Rust Book - <AUTHOR> https://rust-embedded.github.io/book/intro/index.html <comment><USER2> 20 years ago, I did embedded computing with systems that would scale to about 360 processors (PowerPC "G4" MPC7410 @ 500 MHz) and about 70 gigabytes of RAM.

Yes, it really was embedded. It fit in 9U (15.75 inches) and was rugged enough to fly in a military aircraft. The OS would let users turn off interrupts in order to squeeze out every bit of performance.

We could also run the OS on the SHARC DSP, which is a word-addressed Harvard- architecture chip without an MMU. Running on that would allow 3x the CPU density. One system got up over 1000 cores.

IO would typically come in via DMA, over a 32-bit link running at 40 MHz. There could be many of these.

The end result was generally something like a radar. The end-user sees a radar, buys a radar, and uses a radar. They don't see a computer. The computer is just a component embedded in the radar.

<comment><USER3> That's an interesting story but you have to admit high bandwidth real time signal processing is a pretty specialist embedded workload. I guess these days there'd be FPGAs and/or existing IC packages for the job.

<comment><USER2> There were several companies competing in this space. The main three were CSPI, Mercury Computing (now Mercury Systems), and SKY Computing. Matrox briefly tried to join the party.

Even back then, FPGAs and custom ASICs would be part of the compute fabric. I left that out. For example, there was an FFT accelerator chip. These days the companies add in a mix of FPGAs and GPUs, but the CPUs are still there. It's all the same stuff today, but faster.

To the end user, it was never a computer. It was a radar, a video processor, an MRI scanner, a sonar, an ultrasound, a chip wafer inspection device, a laser with real-time mirror warping, or some other tool.

<comment><USER3> You have a good point that people who set out to implement high end military/medical/high end industrial metrology hardware greenfield projects today may see value in using Rust and won't have a problem with more expensive hardware. However, I do think that's a vanishingly small percentage of embedded development.

</comment></comment></comment></comment><comment><USER3> I would absolutely _love_ to throw my lot down behind something other than C/C++ for embedded development, but after investigating Lua and Rust I could only come to the conclusion that they're currently inappropriate for resource constrained commercial development... particularly if you use a broad range of sensors, have higher-level protocol communication requirements, etc. Re- implementing this stuff isn't just a minor drag, it's commercial insanity.

1\. Example hardware in this book costs around 98RMB and has a huge form factor. AVR / ATmega328P based platforms like Arduino Nano are currently ~10RMB, which can be beefed up to full ethernet connectivity with prebuilt hardware modules for another 20RMB. So in short it is possible to procure over three(!!!) embedded, ethernet capable AVR platforms for the same price as this hardware, no soldering required. Why not use something cheaper and more broadly available? It would help more people get on board (no pun intended). Oh wait, there's no AVR support. I see...

2\. The nominal target hardware in the book seems overpowered/overpriced/out of date. For example, the target used in the early book is LM3S6965, which is NRND (Not Recommended for New Designs). Probably as a result of NRND status (I guess people buying up stock to keep old designs in manufacturing) it costs over 125RMB for the chip alone. The manufacturer's recommendation for replacement is TM4C12x, the cheapest development board for which is MINI-M4FORTIVA @ 22RMB (which features JTAG). ICs in this series begin at 22RMB or so, still many times AVR chip price, and with IMHO far too much hardware for novices (64KB flash, 12KB RAM, 12x12bit ADC channels, 49+ GPIO channels...). A direct quote from later in the book is _In this example, let 's assume we have an Texas Instruments TM4C123 - a middling 80MHz Cortex-M4 with 256 KiB of Flash._ That's not middling, that's ridiculously overpowered for almost anything embedded.

3\. See example code at [https://rust- embedded.github.io/book/start/registers.html](https://rust- embedded.github.io/book/start/registers.html) and compare C and Rust _Volatile Access_ examples at [https://rust- embedded.github.io/book/c-tips/index.html](https://rust- embedded.github.io/book/c-tips/index.html) and see which syntax you prefer...

<comment><USER4> Is there seriously still a commercial reason to use the ATmega328 (an 8-bit microcontroller with 2000 bytes of RAM) for new projects? I've been working on 3D printing firmware, and it's so easy and silly to reach the limit of this CPU. Been shaving bytes of RAM by reducing queue sizes, running into performance limits even with a few 32bit additions in the stepper loop. That was 6 years ago.

The only semi-valid reason to use this chip IMO is because people want to keep using existing firmware as-is.

<comment><USER5> Cost and temperature. Albeit not the ATmega 328 MCUs in the even more resource constrained ATTINY series go for under $.30 in quantity and run in 125C environments [https://octopart.com/search?q=avr%208%20bit&sort=median_pric...](https://octopart.com/search?q=avr%208%20bit&sort=median_price_1000&sort- dir=asc) . It's going in new products.

<comment><USER3> Only $0.13 for 2000+ ATTINY25-20SU according to [http://www.hqchip.com/search/ATTINY.html](http://www.hqchip.com/search/ATTINY.html)

<comment><USER6> A rather small team of 10 people (with only 4-5 devs) will cost 10K per week at some generic corporation. Let's say your project takes 12 months to make on a beefy processor ($2 per unit) and 15 months on a cheap processor ($1 per unit). That extra dev time to deal with those issues directly costs 120K.

That automatically means you have to be shipping at least 120K units just to break even with those costs. But you are also getting to market 3 months later which itself has huge business costs which means you probably need to be shipping 200K or more units. That's a pretty big number for a lot of embedded device designers.

If you double your chip constraint from $0.13 to $0.26, then that same team must now ship a million devices to break even.

</comment></comment></comment></comment></comment><comment><USER7> Is there a good STM32 USB library for Rust? I've seen some preliminary work on some stuff in the past but I'm not sure where it's at. If it's not particularly developed, has anyone had success calling into the ST provided libraries from rust to get USB up and running?

Rust for embedded stuff seems like an absolute dream. I'd love to make use of all the excellent work that's gone into it. Most of my work just needs USB.

<comment><USER8> There is this one, [https://github.com/mvirkkunen/usb- device/blob/master/README....](https://github.com/mvirkkunen/usb- device/blob/master/README.md) which apparently works well enough to power a keyboard: [https://github.com/TeXitoi/keyberon](https://github.com/TeXitoi/keyberon)

</comment></comment><comment><USER9> This book focuses on the ARM Cortex-M archtectured microcontrollers..

Is there a comprehensive list which other microcontroller is supported by Rust?

I really like the direction that Rust is heading in regards to embedded but I wish that when folks mentioned embedded, they should also indicate which microcontrollers are relevant. There are so many varieties around.

<comment><USER10> [https://forge.rust-lang.org/platform-support.html](https://forge.rust- lang.org/platform-support.html) is mostly accurate.

<comment><USER11> Wow: tier 1 contains no ARM or other CPU arch. Tier 1 is only 32bit i686 and 64bit x86_64.

Tier 2 platforms can be thought of as "guaranteed to build". Automated tests are not run so it's not guaranteed to produce a working build.

<comment><USER10> We've been thinking about re-doing the tier system, as it misses some important points. For example, Firefox has ARM as a tier 1 platform, so if we find bugs, they tend to get fixed up pretty quick. We're not sure the current way of defining stuff really maps to the reality.

</comment></comment></comment></comment><comment><USER12> Thanks much to japaric (Jorge Aparicio IIRC), he's contributed heavily to projects which support embedded use cases.

</comment><comment><USER13> Okay, can we _please_ get a primer that an intern can follow that starts from an empty Windows 10 install, installs everything including VSCode, flashes the boards with a Segger J-Link, blinks the LED, and single steps the debugger?

I'm serious. I will _go buy any board_ in order to check that tutorial out.

I'll go further. I would buy the Rust folks a couple boards and Seggers to get that.

<comment><USER14> Hey, my company has this. We do consulting and training for embedded rust, and our material is open source.

Check out [https://github.com/ferrous-systems/embedded- trainings/blob/m...](https://github.com/ferrous-systems/embedded- trainings/blob/master/INSTALL.md)

<comment><USER15> Thats ace!

BTW It is really cool that your company does this. It has a cracking name too!

</comment></comment></comment><comment><USER16> This is great, but unfortunately embedded doesnt always = ARM. Does anyone know if it would be possible to target mips32 since LLVM supports it? Is there any effort in rust for other arches? The mplabx + xc32 +harmony toolchain has got pretty darn awful so that would be amazing for the MIPS community

</comment><comment><USER17> Maybe using an ARM dev board boosts acceptance of this book among its target audience, at the moment. But, for open source hardware goals, a RISC-V board, such as the HiFive1, would be good cross-promotion/support.

<comment><USER17> Imagine Rust being considered _the_ way to develop for embedded RISC-V, from almost the start.

Similarly, as long as Rust is already shaking up C/C++ a bit, they could nudge some hobby/education and professional development towards RISC-V. (Especially on embedded, right now, as I've heard the open ISA is attractive to some projects.)

<comment><USER18> I feel the same way about Ada and RISC-V, to be honest.

[https://blog.adacore.com/ada-on-the-first-risc-v- microcontro...](https://blog.adacore.com/ada-on-the-first-risc-v- microcontroller)

[https://hackaday.com/2018/10/08/programming-a-risc-v- softcor...](https://hackaday.com/2018/10/08/programming-a-risc-v-softcore- with-ada/)

[https://riscv.org/2019/01/adacore-joins-the-risc-v- foundatio...](https://riscv.org/2019/01/adacore-joins-the-risc-v-foundation- to-provide-c-and-ada-compilation-support/)

</comment></comment></comment><comment><USER19> So, Nordic Semiconductor, when do we get Rust support in your SDKs? Please?

<comment><USER20> At least the soon to be released ESP32 successor seems to get this, by virtue of adopting a RISC-V core. Kind of a big deal really.

<comment><USER21> [citation needed]

<comment><USER20> My info is mainly from: [https://www.esp32.com/viewtopic.php?f=2&t=9768](https://www.esp32.com/viewtopic.php?f=2&t=9768)

I was incorrect, citation: d) it is a Tensilica / RISC V are in the pipelines.

So not "big deal" RISC-V yet, probably later.

</comment></comment></comment></comment><comment><USER22> Would love to learn Rust and currently debating on a few home applications.

Any ideas from the HN crowd? I'm currently looking at a Pi-based weather station where I would use Rust as the processing server for all the data each day. Nothing fancy, but perhaps a fun way to learn.

</comment><comment><USER23> I know some embedded devs who say Rust is still too slow compared to C.

What do they mean?

<comment><USER24> That they are happy with C and are sick of others trying to convince them to use something else. Only halfway joking... But if you really want to know what those people mean, then you should ask _them_.</comment></comment>


Russia Says Small Nuclear Reactor Blew Up in Deadly Accident - <AUTHOR> https://www.bloomberg.com/news/articles/2019-08-12/russian-says-small-nuclear-reactor-blew-up-in-deadly-accident <comment><USER2> Cruise missiles are generally short range. The main advantage of a nuclear cruise missile is its added long range and maintained ability to stay mostly undetectable and unstoppable to missile defense since they can travel at low altitude.

The other advantage is political. These missiles take a long a time to reach destination, and the time can be extended by a prolonged voyage (circle the globe a few times). The only people who can disarm the warhead are the enemies. You're not going to blow up your enemies while the missile is in the air. It's a great apolitical tool. Launch a missile and everyone starts talking and giving in to your demands.

Better article: [https://www.wsj.com/articles/russian-nuclear-agency- confirms...](https://www.wsj.com/articles/russian-nuclear-agency-confirms- deaths-during-missile-test-11565446468)

</comment><comment> orbital-decay What a clickbaity headline and a sloppy reporting job, didn't expect that from Bloomberg. The title says reactor but the content implies RTG, which is hazardous but many times less radioactive, and is completely different from a nuclear reactor. Also a bunch of unrelated facts, they even managed to mention Kursk disaster somehow. (I guess that's inevitable when there's little to no info available but you still need to write something)

<comment><USER3> Lets not forget that Bloomberg was also the publication that ran with and defended that whole 'Supermicro is totally getting spy hardware added to their motherboards by the Chinese government even though no one can seem to locate it' story. Something's... possibly askew with them of late.

</comment></comment><comment><USER4> Sounds similar to Project Pluto: [https://en.wikipedia.org/wiki/Project_Pluto](https://en.wikipedia.org/wiki/Project_Pluto)

</comment><comment><USER5> >iodine, which is believed to help prevent the thyroid gland from absorbing radiation.

"Believed to"? I thought that was accepted as fact?

<comment> your-nanny Is radioactive iodine the only radioactive isotope we'd expect to be exposed to.at lethal doses?

<comment><USER6> All kinds of radioactive isotopes might enter or attach to your body in many ways as consequence of a nuclear accident. Any radioactive particle that gets close to or into your body will cause some damage depending on the type (alpha, beta, gamma), strength of radiation and duration of exposure.

If you are not exposed to an immediately lethal dose, the damage in the body is a statistical thing. The longer and the more intense the radioactive particles remain on or in your body, the more likely is the chance for a serious health issue. The strategy is to remove radioactive particles as fast as possible to minimize the probability of damage.

Radioactive particles on your skin can be washed away. (This is not so fun as the every-day shower).

Radioactive particles that you inhaled are a really bad thing, I do not know how to deal with radioactive particles in the lungs (except wearing filter masks in time).

Particles that entered the body with food will be excreted within a couple of hours.

But: your body has an organ - the thyroid gland - which collects iodine from your food and stores it long time. A radioactive form of iodine - iodine 131 - is a common byproduct of nuclear contamination. If this gets stored in your body, it remains long enough to dramatically raise the chance of cancer. Therefore you fill your thyroid gland with non-radioactive iodine, hopefully BEFORE your body gets contaminated with radioactive iodine.

So the special thing with iodine 131 is, that it is common, that the body does not make a difference to nonradioactive iodine and there is an organ that stores it.

By taking iodine pills, you only improve probabilities on that one factor! There are plenty other ways to take damage from radioactive accidents.

<comment><USER7> Has this actually been tried? I know there's no way to do a proper controlled test, at least in humans ("OK,you take this pill, and you take this placebo. Now, here, breathe this."), but is there any naturally-occurring data that might imply that this actually works? Say, distributing pills in a place with a natural radioactive iodine source, or counting up thyroid cancers after Chernobyl and comparing people-who-took-pills to people who didn't?

I assume there is at least a rat study.

</comment></comment></comment></comment><comment><USER8> > The blast occurred Aug. 8 during a test of a missile that used "isotope > power sources" on an offshore platform in the Arkhangelsk region, close to > the Arctic Circle, Russia's state nuclear company Rosatom said over the > weekend. The Defense Ministry initially reported two were killed in the > accident, which it said involved testing of a liquid-fueled missile engine. > The ministry didn't mention the nuclear element.

Maybe a stupid question but why would they be anywhere near the missile during the actual testing?

Or was this some kind of premature explosion while configuring or fueling it?

<comment><USER9> > Maybe a stupid question but why would they be anywhere near the missile > during the actual testing?

They are Russians [https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3604355/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3604355/)

<comment><USER10> Very poor H&S regime unfortunately - its a bit grim but in this case you hope that the fatalities died quickly.

</comment></comment></comment><comment> your-nanny This weapon, and Project Pluto, makes me so angry. It's very existence is a crime against humanity.

</comment><comment><USER11> I don't really see what Russia or the US has to gain from designing new kinds of nuclear weapons. They already have the capability to wipe out whatever they're aiming at.

<comment><USER12> This is about nuclear reactor and not nuclear warhead. Nuclear reactors are used to produce electricity and/or heat and to breed radioactive elements.

<comment><USER13> If you read the news, it is much more akin to a nuclear engine, not a reactor.

Either powered by actual expensive artificial isotopes with enough activity to power an engine, OR they actually broke the ban and had it running on pu239 (which would also make its fuel reserve a nice yield booster)

<comment><USER12> Nuclear engine IS a nuclear reactor.

[https://en.wikipedia.org/wiki/Nuclear_reactor](https://en.wikipedia.org/wiki/Nuclear_reactor)

"A nuclear reactor, formerly known as an atomic pile, is a device used to initiate and control a self-sustained nuclear chain reaction."

Nuclear engine is a device used to initate and control a self-sustained nuclear chain reaction.

This is in contrast to nuclear bomb which initiates but does not control the chain reaction. Once the reaction is started nothing else can be done with it (ie. there is no control involved other than setting up initial parameters).

</comment></comment></comment></comment><comment><USER14> Makes one ponder whether russians don't know what they're doing or are others simply better at hiding failures. Scared to think what else is going on that, us, regular people are not aware of. I feel like one of these days some government is going to announce "You didn't ask for it, but we poisoned the environment, ozone layer is gone and water is not fit for drinking, BUT! We have an answer to century long dick measurement contest!"

 _puts on tin foil hat_

<comment><USER15> Rest assured, it's hard to hide nuclear accidents. There are so many sensors around the world which are owned by various governments and private entities. Any significant radiation leak will be detected and announced.

If you're worried about water contamination, you should focus your concerns on conventional chemicals. Oil drilling, coal mining, and manufacturing waste are the ones causing decades worth of damage.

<comment><USER16> That's reassuring if true, do you have evidence to back it up?

<comment><USER17> If you are asking for open data, then there's radioactive@home (based on BOINC).

But in this case none of the detectors is sensitive enough or sufficiently near the White Sea.

[http://radioactiveathome.org/map/](http://radioactiveathome.org/map/)

</comment></comment></comment></comment><comment><USER18> wow, it killed the > "elite of the Russian Federal Nuclear Center,"

>It caused a brief spike in radiation in the nearby port city of Severodvinsk, according to a statement on the local administration's website that was later removed. The Russian military said radiation levels were normal but disclosed few details about the incident.

News of the explosion set off in nearby cities and towns a run on iodine, which is believed to help prevent the thyroid gland from absorbing radiation. Norway said it had stepped up radiation monitoring after the incident but hadn't detected anything abnormal.

is it really worth this large amount of risk if even the top experts in the field can get it so very wrong

</comment><comment><USER19> Just a small one

</comment><comment><USER20> Sounds like a RTG to me, though the only reason I can think of to put one on the missile itself is if you want to park it in orbit as some sort of first strike weapon

<comment><USER21> It's a nuclear propelled cruise missile ([https://en.wikipedia.org/wiki/9M730_Burevestnik](https://en.wikipedia.org/wiki/9M730_Burevestnik)), conceptually similar to SLAM ([https://en.wikipedia.org/wiki/Supersonic_Low_Altitude_Missil...](https://en.wikipedia.org/wiki/Supersonic_Low_Altitude_Missile)).

An RTG makes little sense for earth orbit, because there's plentiful sunlight, and solar panels can be used. RTGs are only really suited to missions far away from the sun.

</comment></comment><comment><USER22> "The war machine springs to life, opens up one eager eye, focusing it on the sky, where 99 red balloons go by."

Never did I think my favorite 80s song would become relevant in my life.

</comment><comment><USER18> nuclear is dangerous. there are no headlines about solar or wind farms causing deadly accidents like this EDIT: that can impact several cities in a region

<comment><USER21> Solar and wind make for poor cruise missile propulsion systems.

<comment><USER18> sure, but if even the top russian experts die in a test then surely it is not suitable for normal power generation

<comment><USER21> A cruise missile nuclear jet engine is indeed not suitable for normal power generation.</comment></comment></comment></comment>


Reverse-engineering FPGAs [video] - <AUTHOR> https://media.ccc.de/v/34c3-9237-reverse_engineering_fpgas <comment><USER2> It seems every few years someone does this publicly, and then either gets DMCA'd or forgotten. Nonetheless, the _extremely_ regular structure of FPGAs, along with the availability of official documentation (which basically explains everything but stops short of mentioning which bits control what) mean that it's not all that difficult of a problem to "connect the dots".

As I've mentioned before, the bitstream formats are one of the worst-kept secrets in the industry. The only reason they aren't revealed more often is for legal, not technical reasons. I'd almost bet that a considerable number of those who work with FPGAs have figured out at least part of the structure and kept it to themselves.

<comment><USER3> That makes me wonder whether it would be more DMCA-safe / lawsuit-safe to not reveal the format itself but rather the exact process to discover it (avoiding endless hours of dead-end experiments because of a wrong assumption about the format)...

<comment><USER4> I'm replying here to a different comment of yours, which has been flagged for some reason.

> _Do you have any good source of what these problems are?_

A quick list:

* Bad verification support. Serious verification takes at least as much effort as the implementation. Improving that is a hard problem, and you won't know if you've improved it unless you have experience with real-world designs.

* Bad support for sequential code. Contrary to what is taught at college, there is _a lot_ of sequentialism in non-trivial designs (except if you design a CPU, but hardly anybody does that). Basically every protocol is sequential. Even most of the smaller external devices, like ADCs or DACs, typically have a sequential interface. VHDL and Verilog require you to explicitly code a state machine, which is needlessly verbose and error-prone.

* Lack of transparency. If the route tool can't meet timing, you have to be able to map the tool's report of the critical path to your high-level design, or you'll have a big problem at the most inconvenient time.

* Lack of a convenient VHDL/Verilog interface. If you can't easily instantiate VHDL/Verilog IP cores, your language will be an academic exercise, at most.

</comment></comment></comment><comment><USER5> Reverse engineering FPGAs is like stealing a train. It might be fun, but the overall benefit is rather limited.

Vidado isn't most comfortable tool, it does the job and handles all the primitives (memory controllers, clocking, transceivers, etc) very well. I wish somebody could design comfortable IP version control system.

<comment><USER6> The fact that digital logic design is constrained to one of two ancient languages by the proprietary toolsets is a serious problem.

There are ways round this by compiling to Verilog (eg Chisel) but it's an uncomfortable halfway house.

<comment><USER4> > _The fact that digital logic design is constrained to one of two ancient > languages by the proprietary toolsets is a serious problem._

Not really. You can compile any high-level language to a netlist and use that as input for place&route. Even if the proprietary toolchain didn't support EDIF netlists, you could still compile to a purely structural VHDL/Verilog file with the instantiated FPGA logic primitives (LUTs, nets, etc.), which is basically

The main (but not the only) reason why VHDL and Verilog have not been replaced yet, is that nearly all supposedly "better" alternatives fail to address the actual pain points in FPGA design. The most likely cause for this is that the overlap between professional FPGA designers and people capable and willing of designing and implementing a language is very, very small. The result is that you get a lot of languages designed by people who have never programmed larger FPGAs in a professional setting and consequently don't have a good understanding of the problems involved.

<comment><USER3> Do you have any good source of what these problems are? The reason I am asking is because I too am designing my home-grown HDL at the moment because I grew tired of both Verilog and VHDL. Since I never programmed FPGAs professionally -- the whole thing is a pet project only -- I'm likely to repeat those mistakes because I don't even know about them.

Note that I don't expect problems in the basics of language design and building tools, but rather specific mistakes in the context of FPGAs.

<comment><USER7> I'm curious to know more about your HDL, as I'm also working on an HDL of my own.

<comment><USER3> The code is at [https://github.com/MartinGeisse/mahdl- plugin](https://github.com/MartinGeisse/mahdl-plugin)

Feel free to open an issue on GitHub to start a discussion. (I'm hesitant to post my private email address here -- old habits.)

</comment></comment></comment></comment></comment></comment><comment><USER8> What tool is he using to show the circuit layout?

</comment><comment><USER9> Really quite impressive work.

</comment><comment><USER10> He kind of looks like Christopher Walken.</comment>