Our superficial scholars - <AUTHOR> http://www.washingtonpost.com/wp-dyn/content/article/2011/01/21/AR2011012104554_pf.html

<comment><USER2> >>>This narrowing has resulted in a curiously unprepared and superficial pre- professionalism.

I don't take these kinds of essays seriously anymore.

Rhodes scholars are chosen primarily because of their accomplishments, as a result of being widely successful in a field. What the author criticizes - single-mindedness - is what catapulted these bright stars in the first place, because people, like the author, in positions of authority and influence said, "Brilliant. She is a Physics Olympiad winner or he is a nationally recognized pianist. Let us reward them."

The same authors and leaders that lament the direction the nation's youths are taken have their hand in creating the very system they criticize. Why not look inward for the changes they can make rather than putting the microscope on other people? In the case of the Rhodes committee, they shouldn't even look at grades anymore. There are other ways to judge creativity and scholarship. Heavy grade focus already pushes students at top colleges into a narrow-minded pursuit of "excellence." And, yet Rhodes and other prestigious institutions reaffirm that idea. This is just an example - a rough one at that - but I hope it communicates the point. Don't pay lip service if you aren't willing to be part of the solution (and I mean taking action).

</comment><comment><USER3> Sounds more like a problem with their selection process than anything else. Rhodes scholars are by definition not representative of students. It may well be the case that people who want to be Rhodes scholars are no longer the people the committee wants to be Rhode scholars.

<comment><USER4> Good point. I applied in Canada about a decade ago. I do remember they mentioned at that time that they had toned down the "excellence in athletics" category. They now merely required participation and demonstration of sportsman spirit. This struck me as quite a reasonable decision on the part of the selection committee.

On a separate note, the Rhodes selection process is extremely competitive. While I didn't get in, I felt that just the process of applying forced me to self-reflect on my capabilities and accomplishments. I ended up making some positive changes in my life as a direct result of what I learned during the application process - sounds like the YC application eh? Heh Heh!

<comment><USER5> I totally agree and think that even if these awards aren't going to the "right" people it is still a worthwhile process for all of those involved. I was nominated by my school for the Goldwater scholarship and also applied to the NSF Graduate Fellowship and found both to be incredibly worthwhile. It wasn't until I wrote (and rewrote) my Goldwater essays that I had a good grasp of what I really wanted to do with life. Sure, at that point I had been doing 'research' and other creative activities, but I had never forced myself to really self-reflect and figure out how my interests come together.

</comment></comment></comment><comment><USER6> This is just a right-wing attack on education, disguised as genuine critique. Heather Wilson is a right-wing politician who is making false complaints. She's running for the Senate in 2012.

Lets take this line, "An outstanding biochemistry major wants to be a doctor and supports the president's health-care bill but doesn't really know why."

Really? I find that amazingly hard to believe. What I suspect is far closer to the truth is that this person supports the health-care bill for a reason that isn't consistent with why Heather wants to repeal it. Here's a quote from one of her letters requesting donations for her Senate run:

"Repeal and replace Obamacare with market-based reforms that are much more effective at controlling the skyrocketing cost of healthcare without compromising choice and quality;"

While its important to judge writing on its merits, outside of appeals to authority. It is important to know the history and motivation of the author. Especially when the piece really has no data, and is just subjective anecdotal conjecture.

</comment><comment><USER7> I think that the problem is students are being taught "how to think", not "what to think".

Back in the bad old days, students crammed lots of facts in into their heads. That gave them an understanding of how the world worked, how it has worked in the past, and a solid platform for thinking about how it might change.

Now, students learn "how to think". Of course, nobody can teach that. So instead, they learn how to churn out 3 point essays, pretend to examine both sides of the argument, then to declare that, on the balance, the side which they gave the most air time to has more points in favor of it.

<comment><USER8> I think students should be tought "how to think" and not "what to think". Teaching what to think is, to some extent, brainwashing. The problem, as you pointed out, is that institutions try to teach "how to think" but fail at it.

<comment><USER9> IMO, one cannot think without a base of knowledge and facts. To attempt to think without knowledge leads to erroneous conclusions base on what feels good. The profound level of ignorance that I see sometimes in the so-called educated young people of today is bad. Worse, they often don't even realize they're ignorant. And worst of all, some even revel in it and proclaim the knowledge is irrelevant.

</comment></comment></comment><comment><USER10> If this is a real problem, I would suggest reintroducing the study of philosophy right across the board. It deals precisely with all the "meta- questions" that fall outside the scope of established disciplines.

<comment><USER11> Some countries (such as Norway) do precisely this: <http://en.wikipedia.org/wiki/Examen_philosophicum>

</comment></comment><comment><USER12> As for the article itself I have similar thoughts to archgoon that it's more likely that their sample has changed rather than the actual population of students.

I find it funny how whenever some problem with education is brought up there's so many people suggesting simple solutions like changes to the curriculum across the board.

Each and every curriculum/educational philosophy WILL have it's problems and WILL tend to produce a certain kind of a person. Nobody denies that there's so many other countries that have better results in certain fields on standardized tests as compared to the US. But not only are those tests poor in measuring proficiency in a single subject[1], they are a complete joke if you want to use that to reflect on the education as a whole.

Unified curriculum is the problem itself. Although it's not very obvious as of right now I think we'll see a HUGE decentralization in how the education works. I won't be surprised if in 20-30 years the whole primary-middle-high school system will get redefined.

You can also rant about the universities in the US, but they are still the best in the world. There are no more than 5-10 universities in other countries that could compare to the top US institutions.

Generally speaking I think it's a nonsense to compare average quality institutions at all. It's a contest of who is least bad. Compare the best. It's not hard for Europeans to have slightly better public education if private is very limited in it's operations.

[1]The problem with international standardized tests is that the way subjects are thought differs between each country. And it happens that US schools usually have a very different approach to teaching subjects like math as compared to the rest of the world. I can easily see how those tests are skewed against the US.

<comment><USER13> _You can also rant about the universities in the US, but they are still the best in the world._

I think you would have to elaborate on that quite a bit to make it somewhat true. Lots of foreign students, including some of the brightest, get their higher-education degrees in the US, mainly at the graduate level. That happens because a good chunk of the brightest professionals are professors at these institutions, and the research resources are plentiful. Clearly the research experience for top-level students is outstanding, _both_ at the graduate and undergraduate levels. In this pretty narrow sense US universities are the best in the world.

But… West Civ I? English Comp? College Algebra, including topics such as completing the square? Calc I derivative shortcuts? For the love of Pete, compare this kind of curriculum with what is taught in Europe, Asia and (gasp) Latin America. You will find that your average college graduate knows much less than elsewhere. Granted, the best may be future Nobel laureates, but that doesn't hide the fact that undergraduate college education in the US has become a years-long remedial program for a disastrous K12 plan.

<comment><USER14> That's a stretch. Of the 40-50 kids I knew in college, not a single one took College Alegbra. Not to mention that my freshman English Comp classes was one of the most intense, interesting, useful classes I took in college. If some are guilty of comparing the tippy top of the US college system to the rest of the world's average, you are certainly guilty of comparing the bottom of the bottom to the world average.

<comment><USER13> _Of the 40-50 kids I knew in college, not a single one took College Alegbra._

But this kind of "College Algebra" is, understandably, considered college- level material. That pretty much happens only in the US (considering countries with educational systems...), and choices such as this influence the overall academic level.

 _my freshman English Comp classes was one of the most intense, interesting, useful classes I took in college_

That's great, and I happened to enjoy a lot my electives such as Theater 101 as a math undergrad (even if I was critical of the curriculum)... but the main point is: Should English composition courses be given at the college level? What, then, do you do during those twelve (OK... last three of those twelve?) previous years? I was puzzled when a student who clearly didn't need English composition defended this setup, arguing that "a college student should know how to write". Well, "a high-school graduate should know how to write", was my response then and now, and if you press me elementary-school students should be well on their way to accomplishing this.

I'm not saying this has an easy solution. Some countries "fix" their K12 woes by resetting the academic level at the college admission tests. As a result, a good chunk of the population has to find a way to raise their level, which is not ideal either. But what the US opts to do lowers the undergraduate level, no doubt. I wonder if this will eventually trickle down to Ph.D. programs.

Edit: All this leaves out grade inflation and related issues... It is downright _hard_ to fail most courses of some majors. This is bad enough.

</comment></comment></comment></comment><comment><USER15> Mmmm... pretty sure Adam Smith was on the money when he talked about division of labour and increasing specialisation, in 1776.

</comment><comment><USER16> This is exactly the same thing the Mike Rowe article discussed, but on the other end of the spectrum. Rowe talked about how there is a lack of vocational training, while Wilson is arguing there is a lack of philosophical training. The problem stems from teaching everyone the same curriculum in US high schools. It's time the US realizes that not everyone should be learning the same things after their years of elementary/primary school.

</comment><comment><USER17> >For most of the past 20 years I have served on selection committees for the Rhodes Scholarship.

...

>The writer represented New Mexico in the U.S. House from 1997 to 2008.

seems to me that the author wasn't just an innocent bystander, instead she was too active and powerful participant in shaping the society and its education to blame the end result on anybody else without acknowledging her share of responsibility.

<comment><USER18> It's a bit of a stretch to say that a congresswoman is responsible for the narrowing of university curricula, as if they're central planners.

<comment><USER17> universities respond to society demands. She has been definitely an important player in shaping the society to demand (incl. reward higher) the highly specialized professionals with impaired critical thinking/initiative/leadership abilities. Cushy corporate or academia job vs. "not a team player" - who is to blame for such a choice posed before the youngsters? Universities?

Even at the micro scale - her "blame the others" leadership approach clearly shows a leader who isn't tolerable to critical thinker and leader types even in her own staff.

</comment></comment></comment><comment><USER19> The kind of people who are inclined to think across disciplines are the same kind of people that are inclined to look at education an think, "this is bullshit." They may not know why they think this, because it takes some real experience and perspective to understand why, but they can smell the stench from miles away. Training a mind to see patterns across disparate fields also gives it an intuition for seeing what is fundamental, central and essential to a thing. This intuition leads to the realization that grades, awards, scholarships and the like are but the mere artifice of achievement.

The kind of person that deserves a Rhodes scholarship according to the author's definition of what one should be is the kind of person who would be unlikely to play the game necessary to qualify for one.

</comment><comment><USER20> Perhaps only tangentially relevant, but as they snarkily say here in Oxford: When you graduate as a Rhodes scholar you know your best days are behind you.

</comment><comment><USER21> "We interview the best graduates of U.S. universities..."

The best at _what?_

</comment><comment><USER22> Philosophy, rhetoric, and history.

By "philosophy" I mean a broad meta study of the field: the philosophy of science, epistemology, the philosophy of religion, the philosophy of language, etc. By "rhetoric" I mean both the construction and execution of persuasive argument in printed and verbal form. Everybody should have a few folks from history that they are intimately familiar with and feel that they faced similar problem in life.

Philosophy can give you a broad view of what is knowable, what is not, and how folks move from not being able to to know, to being able to know, to actually knowing -- a key component of both the advancement of science and the development of an individual. Topics like stoicism and existentialism, if presented appropriately, can provide a backstop of beliefs for any student, regardless of their religious or moral convictions. Rhetoric helps you spot when folks are trying to manipulate you, and it challenges you form coherent arguments yourself about things outside of one small area. History gives you context: every generation feels as if it is here at some kind of special and unique spot. Knowing that humans 500 years ago were the same model as the ones you meet on the street helps put your opinions and interests in context.

Unfortunately, if I hadn't defined what I mean by those three subjects my recommendation would be pointless. You can go to any college today and take courses under those headings and not receive those benefits -- perhaps this is because those who design the courses view them as introductions to a deep dive on a topic and not practical, broad, benefits-based interventions with young minds. I don't know. I understand that my advice is easier said than done: how would you introduce history to a large group of students in such a way that each student found their own historical figures that they would benefit from absorbing? How do you take topics that the students might find popular and feel-good and give them the ability to tear apart the fluff around them and find the good stuff underneath? Having a professor stand up and say something like "this is my opinion on X. I may be wrong, but we'll never know unless I give you the skills to identify and refute faulty argument" just isn't happening. Instead, classrooms are hierarchical affairs.

Instead of this discovery process, students get pre-canned little happy meals. Instead of taking charge of forming their own identity across multiple disciplines that they can describe and defend, we get these examples of kids simply being out of their depth once you stray just a little bit from their established path. An education that is both broad and self-determined is not an easy problem to solve. The system is not built for it.

<comment><USER23> I studied in India. I didn't have any teachers for 4 years while i was passing school in the 80's. We had temps who would come and leave in a week since we stayed way out in a remote place while my dad worked on some government projects. Till this day, I haven't forgotten how I studied, the discovery process was the only thing that held my interest, I got a couple of US university level books and solved through them. The discovery process made an autodidact for life, but the rewards went to the people with happy meals anyway. However, till this day, I am able to reconfigure my skills to meet the need of the hour. I am a quick hack away from happiness.

<comment><USER24> Interesting . did it have the same impact to the rest of your class ?

<comment><USER23> Many who could afford moving to a city did so. In the end the ones that were left were 5 girls and 4 guys. The girls got married and settled down a few years later. Thinking about it now, all of the city guys are now settled in the US.

Strangely, given the parent comment, I did study the history of science later and have a research degree to show for it. :)

</comment></comment></comment></comment><comment> dorian-graph Would a returning of a type of 'old' classical education specifically in primary and high school help?

<comment><USER25> Where would you get the teachers? The smart women who made the old system run these days work in commerce in industry.

<comment><USER26> Excellent point; I hadn't thought of that, and don't recall hearing it discussed in any of the "educational quality" debates.

<comment><USER25> Teacher quality has been mentioned, often in the context of the difficulty of firing bad teachers in many U.S. school systems.</comment></comment></comment></comment>


After 5 days, my OS doesn't crash when I press a key - <AUTHOR> http://jvns.ca/blog/2013/12/04/day-37-how-a-keyboard-works/ <comment><USER3> I've never done OS programming, but something about this just tickles me. I think it's the way she cheerfully acknowledges screwing up constantly but still gamely keeps tackling the problem. To admit your mistakes without shame, and try again without bitterness--to lose your ego, in other words--is something I certainly aspire to, as a programmer and a human being.

<comment><USER4> In a context where everything "low-level" is handed to us without a "how-to", looking under the hood to build your own thing is always a pleasure, be it a new programming language or a whole OS.

To attain this state of zen-itude, just look up something that you'd like to do where documentation is hard to come by ("how to write your own shell from scratch" is a funny one if you can want to learn a LOT about UNIX) and start your own project.

Related: とあるOS is a nice hobby OS too: [https://github.com/klange/toaruos](https://github.com/klange/toaruos)

</comment></comment><comment><USER5> Another good read and really gives the insight on how much time and effort it takes to make anything happen on bare metal.

@jvns: __SPOILERS AHEAD, DO NOT READ FURTHER IF YOU WANT TO GET TO THE HEART OF THIS PROBLEM YOURSELF __

As for your problem with the keyboard interrupt firing only once, I think (after a quick glance over the code) that you do not signal "end of interrupt" (EOI) message to the 8259 PIC chip after you have received an interrupt. I faintly recall having this same issue in my ordeals with the PIC chip, if you don't signal EOI, the PIC won't fire the next keyboard interrupt.

Although I think I was working with 8253/8254 PIT (Programmable Interval Timer) chip at the time but I think the same applies to the keyboard interrupt too, it's being handled by the PIC too.

Here's the relevant part from my project: [https://github.com/rikusalminen/danjeros/blob/master/src/arc...](https://github.com/rikusalminen/danjeros/blob/master/src/arch/x86_64/interrupt.c#L68) [https://github.com/rikusalminen/danjeros/blob/master/src/arc...](https://github.com/rikusalminen/danjeros/blob/master/src/arch/x86/pic.c#L41)

I'm sorry to spoil your "fun" debugging it.

Dealing with any of this is a fun exercise in being close to the metal but unfortunately has no application in modern computers because these days the built-in IO-APIC/LAPIC has replaced the legacy PIC chip.

<comment><USER6> Spoilers for a bug report... that's a new one! Very thoughtful. :)

</comment></comment><comment><USER7> Nothing quite like trying to write a complex system from scratch to realise just how many giant shoulders you stand on every day.

<comment><USER8> I love this article: [https://plus.google.com/112218872649456413744/posts/dfydM2Cn...](https://plus.google.com/112218872649456413744/posts/dfydM2Cnepe)

<comment><USER9> Now (re-)submitted as an item in its own right:

[https://news.ycombinator.com/item?id=6853813](https://news.ycombinator.com/item?id=6853813)

<comment><USER8> It has been submitted several times before if I'm not mistaken, in fact I know it from HN.

<comment><USER9> Indeed - I also knew it from HN. But

* it's not been submitted for a long time,

* discussion there is closed,

* it's _really_ good,

* it's technical, and

* it's relevant.

I commented about why I've resubmitted it here:

[https://news.ycombinator.com/item?id=6853824](https://news.ycombinator.com/item?id=6853824)

</comment></comment></comment></comment></comment><comment><USER10> When I was in undergrad I wanted to take Operating Systems 2 but it was only offered once/year and didn't match up with my free time. So I did an independent study where my plan was to write a basic ethernet driver for Linux.

I have never been as humbled by any sort of programming in my life. This was back in 2004 so the resources available online weren't nearly as good. I basically figured everything out by reading Intel's x86 documentation from start to finish. It took me a solid 3 weeks just to figure out how to start talking to the ethernet card. I spent days reading the same few pages in the manual over and over trying to understand how to get everything to fit together in assembly & C.

After 9 weeks I was able to address & initialize the ethernet card after the OS booted. I never was able to send or receive any data after a solid 50 hours of work.

Oh yeah, and I got a solid A for my work.

<comment><USER11> With regards to kernel programming, I'm reminded of the old quote from Star Trek, "It's life, Jim, but not as we know it."

</comment></comment><comment><USER12> Working at the layer below is a very useful exercise. As part of my undergrad course in EE in the late 1980's we had to design and simulate a 4 bit micro- processor using logic components (essentially NAND gates). I got completely immersed in that project. Not only did it force me to understand a complete, albeit simple processor but actually building it gave me a lot more confidence in writing code. Things have moved on a lot since then and there are many layers of abstraction meaning that most developers don't need to understand the hardware, but having a decent 'programmers model' for the OS can only be a good thing.

<comment><USER13> I built a 4 bit microcontroller the same way around 2000. We flashed the gates to an Altera EPROM and other than external memory chips, the state machine, ALU, and other sub systems were assembled using logic gates we burned. We supported 16 instructions and could access 16 words of memory. That was enough to code a "password" based security system that would either grant access by raising an external line or it would output an alarm state if the code was invalid. As I recall it could only handle about 25Hz max. Still one of the most influential projects I worked on in college.

</comment></comment><comment><USER14> I enjoyed this. It brought back memories of bringing up BSD2.9 on the PDP 11/55t we had in the lab. That particular PDP 11 was pretty rare (it had some memory split features to improve performance for computational tasks) and poking around in the kernel to get to the point where init was started. Nothing quite like that thrill, a mixture of power, and 'oh crap this is going to be a lot of work'.

<comment><USER15> What she's doing is surprisingly a lot harder. Getting a modern IA-32(e) architecture into the 64 bit mode we all know and love is INSANE.

<comment><USER14> I'm not going to argue on 'harder' or 'not as hard' as such things are often personal measures that are difficult to quantify. I will however point out the meta fallacy of even thinking about these things in terms of 'not hard.'

The truth is, computers have become exceptionally complex. That is one of the reasons I've been building a medium complexity standalone system (ARM Cortex M based) to give folks something that is somewhere between 8 bit Arduino type experiences and 64 bit IA-64 or even ARM Cortex A9 level complexity. I realized when I started finding ways to teach my kids about computation that I was very lucky to have things like PDP-11's, VAXen, 68000's, and DEC-10s to play with which did not present this huge wall of complexity that needed to be scaled to get to the fundamentals. My target is a self hosted 'DOS' style Monitor/OS for the Cortex M4 series. Complex enough to host its own development environment and tools, but simple enough that you can keep it all in your head at the same time.

<comment><USER15> I was actually just pointing out the irony that 30 years of processor innovation have lead to making the task of booting even more complicated than it used to be ;)

(Note: I understand why, that doesn't make it less ironic.)

</comment></comment></comment></comment><comment><USER16> For some reason HN ate my previous comment

Doing OS things is hard. In x86 it's _extra hard_

x86 is one kludge on top of another. Like interrupt chaining

These issues remind me of making a Sound Blaster card work on DOS (I was making a program to play a simple file in Pascal/C)

It's very complicated, and several trials to make it work (and no Stackoverflow, Wikis, etc)

<comment><USER17> That's very true from both compiler and OS point of view. Intel machine code is so kludgy: variable-length codes, subtle differences in prefixes in different modes, exceptions in register handling all over the place, plenty of addressing modes for memory access, special machine codes for a qualified register (`mov %ax` code differs from `mov %cx/%dx/%bx/...`), more lengthy encoding for more often uses (like `mov (%esp)` requires SIB byte whereas it's not true for all other registers) and so on.

From OS perspective, it's also overcomplicated: switching from real to protected mode is an exercise in doing every action in a long chain properly and in the right order or the whole undertaking fails. Remapping IRQs into interrupts that do not mess with the CPU internal exceptions, weird legacy memory layout with BIOS, a slew of CPU modes (real/protected/PAE/long/etc), crazy format of entries in Interrupt Descriptor Table (where four bits of a pointer may be stored separately from the main part), where bits are often used differently for different contexts; redundant built-in features like poor multitasking support (Linux and other sane kernels try to avoid it as much as possible, but it's still is not possible to go into kernel mode from userspace without filling kernel stack pointer into a TSS entry), memory segments (in protected mode, where a pointer can address the whole virtual memory unlike real mode), four privilege rings (whereas all modern OSes use two: 0 for kernel space and 3 for userspace).

All that does not feel like it should be and often it only gets worse.

<comment><USER18> Going from Motorola 68k to x86 with few preconceptions in the nineties was quite a shock after slowly having been convinced that the grass actually was greener.

On the x86 side I was surprised to find an unnecessarily cryptic assembly language, little-endian words and a byzantine memory model for starters.

Hacks upon layers of hacks and not the intuitive and aesthetically pleasing solutions seems to win out in most cases.

<comment><USER19> It's a wonder any of this stuff works at all. I find myself saying this almost every time I learn something new about how computer systems are designed, built and used.

</comment></comment></comment></comment><comment><USER20> > _I'm seriously amazed that operating systems exist and are available for > free._

Made me smile :)

</comment><comment><USER21> She better not keep this up, first it's just tinkering around and then boom... it's a slippery slope to creating a whole kernel.

</comment><comment><USER22> I really like the step by step list - nothing's been removed and I can see exactly how frustrating/enlightening the process was!

<comment><USER23> Agreed- the frustration was more than enough!

</comment></comment><comment><USER24> >Press keys. Nothing happens. Hours pass. Realize interrupts are turned off and I need to turn them on.

>THE OS IS STILL CRASHING WHEN I PRESS A KEY. This continues for 2 days. Remember that now that I have remapped interrupt 1 to interrupt 33 and I need to update my IDT.

I have issues like this a lot (i.e. forgetting I've tweaked something and wondering what is causing weird effects). Any advice on how to avoid it?

<comment><USER25> Use a rubberducking text file.

Every time you think, write down a line or two delineating your current reasoning and your debugging info. When you make changes, highlight them to mark them as pending ("need to update IDT"). You can also categorize, tag lines/pieces, and so on.

When you're stuck just backtrack this file, whether it is a hour or two days from now, and you'll avoid all of this.

</comment></comment><comment><USER26> I'm guessing that a lot of the difficulty comes from the fact he's creating an OS for a PC, which is a pretty quirky and inconsistent architecture.

<comment><USER27> The blog author is not a 'he' but a 'she.' Her name is mentioned through the article.

And _I 'm_ guessing that the difficulty comes from the fact that she's building a freaking operating system from scratch... ;)

<comment><USER28> Is the gender of said author supposed to be curious or worthy of attention? I'm not intending to be impolite, but this is the third time in several days that HN posters have brusquely corrected a gender pronoun in relation to a submission of nondescript importance.

<comment><USER29> It's quite disrespectful to call someone a 'he' when they are obviously a 'she.' This is only vaguely related to the recent discussion about 'they' for people of indeterminate gender.

It's also quite jarring to read; I was confused if the OP was speaking of someone else. People often point out grammar or factual mistakes, I see your parent as doing no different.

<comment><USER28> I was under the impression that banal grammar corrections are frowned upon here on HN, because they do nothing to further the topic or encourage interesting discussion. It's one of the reasons why Reddit is so tedious.

<comment><USER30> They point is not so much a correction of grammar, but a correction of the wrong implicit assumption that a person who writes about osdev has to be male. Its no drama if it happens once, but as said in this thread it happened several times in the last few days. And if it happens that often it effectively makes women in this field invisible, especially as role models for the next generation(s).

EDIT: And it goes a bit further. The sole reason why we are having this discussion is because the author explicitly used a female name and her writings still got perceived as those of a male author. If there no name or just a generic nick, most people in here - myself probably included - <USER2> have assumed its a male by default.

</comment></comment></comment></comment></comment></comment><comment><USER31> Reading this gives me a yearning in my belly to drop everything and hack away at a toy OS for a while

<comment><USER32> I did this last week in the three days before Thanksgiving. I got stuck at the bootloader. Oops? :)

</comment></comment><comment><USER33> This brings back memories...

I remember fresh out of high school following a few tutorials and writing a basic kernel & code for my old Pentium MMX and reading through the intel developer manuals for the 386 processor , and all the complexity involved that was inherited from the 8086 & 286 processors. Was a fantastic learning exercise... though I would suggest against using Pascal as the implementation language.

</comment><comment><USER34> I think, after I clear out my current project log, OS hacking is the next thing on my plate. I'll probably start with drivers, but I eventually want to get into learning the core concepts. OS kernels and compiler design are kind of the last "things" that I've not yet done with programming, and probably couldn't just jump into it feet first and be productive. I've done AI, graphics, real-time embedded programming, high-performance algorithm stuffs, large scale data crunching, etc. etc., and they all pretty much come down to the same stuff: know your math and don't waste cycles. 'Spose that's kernels and compilers, too.

</comment><comment><USER35> Would echoing the chracter on screen require a seperate video driver to mark the pixels that make up the character ? Considering how hard is to make the keyboard driver, wouldn't the video driver be many folds harder?

<comment><USER36> Text mode is super easy - you just set bytes in the correct memory location (0xb8000000 if I remember correctly).

Writing an actual graphics driver is tricky - although there are some ways to get decent frame buffer modes (like VESA Bios Extentions), to do things really well requires thousands of lines of code before you can even get a pixel on the screen, and is different for every card.

It may be a bit easier now - you could look at the kernel mode setting code in the Linux kernel that exists now for Intel, nVidia (through Norveau) and AMD cards that didn't exist a few years ago when I was dabbling around in OS dev. Still, it would be a massively difficult task. You'd probably want to use the full Mesa stack and open source drivers that use it if possible, but writing the runtime to support it would be a big task.

<comment><USER37> 0xB8000, to be precise. This is 16-bit 8086 legacy, where addresses were 20-bit and segmented.

The most fun mode was mode 0x13 from INT 0x10, with its fantastic 320x200 resolution and 256 colors. That one had the buffer at 0xA0000.

<comment><USER38> Oh mode 0x13 and warm memories. Super easy to get working, as well, just an interrupt and start writing pixels to memory. Would probably be one of the first thing I'd do if I were to write a kernel.

</comment></comment></comment></comment><comment><USER39> Cool, sounds like so much fun. I remember writing OS/2 drivers and getting interrupt handling wrong would cause strange ways of destabilizing the system. Once you get it sort of going you're afraid every time you press a key or do anything with the system because you know your brand new code is down there in the bowels waiting to cause a triple fault.

</comment><comment><USER40> How do OS developers typically debug this sort of thing? Do they attach external hardware to step through, or is it serial logging?

<comment><USER17> From my small OSdev experience: write printf before anything else (it's not that hard, just write bytes into *0xB8000). This can be done even before getting any interrupts working, fortunately (without interrupts any input is almost impossible, I don't want to talk about polling).

</comment></comment><comment><USER41> Back in the day when I was playing around with osdev, I ran my code in the Bochs emulator. WAY easier than running on actual hardware, especially starting the emulator was 10x faster then booting on an actual PC.

I do remember the excitement of getting interrupts to work - good times. Getting task switching to work was magic.

<comment><USER42> She mentions in the article that she's using Qemu.

<comment><USER41> Ah - I missed that.

True emulators like Bochs are still potentially a win if they provide fine- grained debugging features. Bochs was nice because you could single-step through assembly and view contents of registers/memory quite easily. You could also attach gdb to it (although this was somewhat flaky.)

</comment></comment></comment><comment><USER43> Congrats! For GNU/Hurd it took years to achieve it.

</comment><comment><USER44> and with two keys pressed at the same time?

</comment><comment><USER45> To be honest, my first thought on seeing the headline was a particularly bad linux horror story...

</comment><comment><USER46> I really wanted to not sound like an asshole and start critizicing this because I like low-level programming articles but this article is confunsing:

1) You don't need an OS to press keys and make them come out in the screen. An OS is kind of a library that an application uses to do things.

2) I believe what the article refers as "OS" is a small process manager. I don't know if it's preemptive, no mention of timer of any kind.

3) As far as I can see, this is not an OS but a ring-0 application that reads the keyboard using IRQs. IMHO an Operative System should at least supply filesystem/memory manager or process-manager services.

Hacker school alumni, those articles are awesome but please get your concepts right and at least learn the correct terminology or you will only confuse people. But I suspect those articles are vague on purpose to incite controversy.

<comment><USER5> You can call it a ring-0 application if wish to argue about semantics or the definition of an OS but you really do end up sounding like an asshole.

Every OS projects starts out as a "ring 0 application" running on bare metal. An application which does nothing except handle interrupts and spew out numbers on the screen to tell that it is working. And most OS projects never really evolve a lot past that, because they are primarily intended as educational tools for the author and nothing else.

If you had a point, you could have made it in a way that makes you seem like a decent person, not an angry antisocial geek.

<comment><USER47> Also, aortega wrote "Operative System" instead of "operating system", so because of that incorrect terminology, we can ignore everything in the post. That's how this works, right?

</comment></comment></comment><comment><USER48>
      >  This continues for 2 days

This thing made me scoff at the article the most. I've written a unix-like OS for ARM 2 years ago as a final project for an OS class I took on campus. There were 20 other people in the class, and all of us forgot to update the IDT at some point. It was a common mistake, so for this article to emphasize it like it's some bug that takes 2 days is just lame.

Imagine if my blog post would talk about my experiences with building an HTML page, and how it took me 2 days to change a <p> element to a <div>. Let's get serious here.

How long will it take the author to write a circular buffer to read and write characters to the UART? A month? That says nothing about the difficulty of the actual task.

You know, when it comes to writing operating systems, I like to think of this quote: "it's hard, it's harder than it looks, but it ain't THAT hard" (I first heard it when the commentators were laughing at Chris Andersen was struggling to dunk in an NBA Dunk Contest).

If you're interested in writing an OS, make sure your primary source isn't some blog post. My textbook was Tannenbaum's Operating System book, and it's a very solid book that I can recommend. Tannenbaum knows operating systems. He's not going to tell you how hard it is. He's going explain all the details to you and, when you understand them, you'll think how easy kernels really are.

Good luck!

<comment><USER49> > like it's some bug that takes 2 days is just lame.

Unless she was spending full time on it, it was not some bug that takes 2 days.

At the same time: If you've never come across some trivial, stupid little bug that has left you stumped for far longer than it should have, you're either superhuman (or have selective memory), or you're a total beginner.

Your dismissive tone is really offputting.

<comment><USER50> When I did Operating Systems in University (in a group), I remember distinctly only 3 bugs that we had difficulty debugging:

* re-using a loop variable from the outer loop in an inner loop (damn you `int i`!)

* putting a char[256] on the stack when the stack size was only 256 bytes and getting stack corruption (this one wasn't me, thankfully)

* signed vs unsigned code being passed between user mode and kernel mode causing problems (no idea why it was causing issues; i just made it all unsigned and the issue went away)

In hindsight, -Wall would have probably caught everything and saved a couple hours of "wtf" at 2-3 in the morning.

Often the simplest things are the hardest to discover.</comment></comment></comment>


For Women in Tech - <AUTHOR> http://siderly.com <comment><AUTHOR> Creator here, happy to discuss any development ideas!</comment>


F.A.A. Approves iPads in Cockpits, But... Not for Passengers - <AUTHOR> http://bits.blogs.nytimes.com/2011/12/14/f-a-a-approves-ipads-in-cockpits-but-not-for-passengers/

<comment><USER2> Common misconceptions: the ban on using electronic devices during take-off and landing is nothing to do with interference with aircraft systems, and everything to do with facilitating prompt passenger evacuation in event of an emergency. Books/magazines are allowed only because most people will abandon them willingly; for some reason a large minority are unwilling to simply ditch their SHINY!! if they have to leave the plane, which would result in faffing around with the potential to cost time and lives.

(It's part of the broader issue of how we behave in crises; some people freeze, some people continue to behave as if nothing untoward is happening -- only a minority behave appropriately, without training. And air passengers are in general anything _but_ trained.)

Even the NYTimes falls for this canard regularly ...</comment>


On the Apple Back Door Rumors … Remember Lavabit - <AUTHOR> https://www.emptywheel.net/2015/08/13/on-the-apple-back-door-rumors-remember-lavabit/#comments <comment><USER2> No one seems to agree on whether iMessage is either hackable or whether Apple can hand over unencrypted chat content when compelled by law enforcement. Which side is right? Are the claims theoretical or have they been tried out in practice? You don't hear of cases of people's private iMessage chats being revealed or used in a court case by way of Apple providing them.

<comment><USER3> As Matthew Green pointed out a few years ago ( [http://blog.cryptographyengineering.com/2013/06/can-apple- re...](http://blog.cryptographyengineering.com/2013/06/can-apple-read-your- imessages.html) ), if you can recover your iMessages after losing your device and resetting your password, then that demonstrates that Apple does have the technical capability to read your iMessages without your consent. And you can, so they do.</comment></comment>


World's roundest object & the kilogram - <AUTHOR> http://www.youtube.com/watch?v=ZMByI4s-D-Y

<comment><USER2> Interesting, but what's with all the weird flirting?</comment>


Think Bayes - Bayesian Statistics Made Simple - <AUTHOR> http://www.greenteapress.com/thinkbayes/ <comment><USER2> My problem with books like this is that they have almost no connection to why Bayesian statistics is successful: Bayesian statistics provides a unified recipe to tackle _complex_ data analysis problems. Arguably the only known unified recipe.

The Bayesian book I want should emphasize how Bayes is a recipe for studying complex problems and teach a broad range of model ingredients. Learning Bayesian statistics is about becoming fluent in describing scientific problems in probabilistic language. This requires knowing how to express and compose traditional models and build new ones based on first principles.

An unfortunate reality is that you still need to know computational methods too, but that should change soon enough.

<comment><USER3> Yes, that's exactly what the objective of this book is! I am not using computation out of necessity, but rather because I think it provides leverage for understanding the concepts, and learning to (as you say) compose traditional models and build new ones.

As the book comes along, I am finding that many ideas that are hard to explain and understand mathematically can be very easy to express computationally, especially using discrete approximations to continuous distributions.

For example, I just posted a section on ABC

[http://www.greenteapress.com/thinkbayes/html/thinkbayes008.h...](http://www.greenteapress.com/thinkbayes/html/thinkbayes008.html#toc44)

that (I think) really demonstrates the strength of this approach.

Of course, my premise only applies for people who are as comfortable with programming as with math, or more so.

<comment><USER2> I'd recommend using as many real examples as possible. Things like forecasting, product recommendations, topic modeling, etc. While you can conceptually explain how Bayesian statistics is a unified recipe, it's incredibly hard to have this sink in with toy problems. This is especially true since many people using traditional tools are actually using advanced methods to solve real problems, so when they start reading about urns or doors it all comes across as rather academic. That's sad because the benefit of Bayesian coherency is mostly that it leads to a highly productive mode of practical data analysis.

Definitely shoot me an email at firstname.lastname@example.com if you're interested in the computational side of this area. At Sense (<http://www.senseplatform.com>), we're working on making applied Bayesian analysis as amazing as it should be.

</comment></comment></comment><comment><USER4> So, I'm going to counter here and say I don't find this to be a good intro. I started reading and had not heard of the "Girl named Florida" problem and then went to the linked to blog post [http://allendowney.blogspot.com/2011/11/girl- named-florida-s...](http://allendowney.blogspot.com/2011/11/girl-named- florida-solutions.html)

The way he explains it I found to be confusing and counter-intuitive. I've taken basic stats in college, and learned some of the associated problems, though not this one, and learned the material though not in this particular way. I have to agree whole-heartedly with the commenter on that post "JeffJo" who stipulates why it's an ineffectual way to present the material. Furthermore, I found the author's dismissal of the valid criticism to be enough to not want me to read further.

<comment><USER3> I am coming around to the conclusion that this example is more trouble than it's worth. I think it's kind of fun, but it does seem to annoy people.

This kind of feedback is exactly why I like to post drafts early. Expect this example to magically disappear very soon :)

<comment><USER4> Wow, great to see a reply from you, and thanks for taking feedback :)

<comment><USER3> And... it's gone!

I re-read the chapter and decided that example was doing nothing except confusing half the audience and antagonizing the other half.

</comment></comment></comment></comment><comment><USER5> I'm really interested in knowing the prereqs I should have before picking up a book like this. Coming from a weak math background I find these books highly appealing but mildly intimidating. Also, could someone advise me on the preferred order of tackling the following Books?

1\. Think Bayes

2\. Think Stats

3\. Programming Collective Intelligence by T.Segaran

<comment><USER6> 3, 1, 2.

3: Get (back) into the swing of thinking about mathematics and algorithms.

1: Bayesian statistics is a principled, coherent, consistent, intuitive, complete framework for reasoning about uncertainty. A good foundation.

2\. Traditional statistics is more random and ad-hoc, but can be more practical than Bayesian methods. (Bayesian models are well-motivated, but it can be impractical to compute exact answers and you'll have to switch to approximation techniques, some of which are simple/universal/slow, and others get fairly complex.)

</comment></comment><comment><USER7> So this is all very well and good, I've had about 5 intros to Bayesian Statistics. But those are a fair bit away from actually applying that knowledge in practice in software.

Let's say we have N different kinds of events with unknown probabilities and unknown dependence or independence between them. The naive approach to gathering data on the probability of event n occurring following an occurrence of event m would require O(N²) in space. Let's say N ~ 10⁹~10¹⁰. Storing that much data as a raw matrix isn't practical in most cases, so we have to find a more efficient data structure - in terms of both space and the operations we need to perform. (and taking into account characteristics of the storage medium, i.e. memory or disk or a combination) What happens if the probabilistic properties of the system change over time?

Are there any introductory books or other resources on modeling this kind of problem? Clearly this has been tackled before, but I'm having a hard time making the leap from theory to practice - and I don't mean import the data into R or SPSS or whatever and let that grind out a solution, but coming up with approximations when you have runtime and space constraints that make that approach impractical.

<comment><USER8> I think the general approach is dimensionality reduction: start measuring, and round down to 0 for the low-correlation pairs of events.

Do you actually have a stream of more than N^2 observations to process? If not, then most of your correlations are in fact 0, and sparse-matrix techniques apply.

</comment></comment><comment><USER9> I'm a big fan of the author's other books, Think Python and Think Complexity (haven't had the time for Think Stats), I found them more understandable than most other books that purport to teach people of the same skill level.

I'm hoping this will be as good, but all the negative comments here leave me skeptical. Perhaps this is the crowd that would enjoy K&R C more than Think Python. The former is more of a reference to me than an introductory tome. Perhaps everyone here is just better at math than I am.

<comment><USER3> Ignore the haters -- Think Bayes is going to be awesome!

Just kidding (mostly), but your point is correct: there is no book that is right for all audiences. But if you can program, and the mathematical approach to this material doesn't do it for you, this book might.

</comment></comment><comment><AUTHOR> FB announcement where I found this - <https://www.facebook.com/thinkstats/posts/325778617519425>

</comment><comment><USER10> Link to _Think Stats_ , the author's previous book: <http://thinkstats.com/>

</comment><comment><USER11> Bayesian is cool because you can make arbitrarily complex models, and when you have the parameters estimated it is really easy to calculate all the cool things you want to.

Bayesian is not cool because estimating the parameters takes bloody ages on a supercomputer, unless you spend ages being really careful to specify your model.

Frequentist statistics is cool because it is a massive big bag of tricks to estimate all sorts of stuff, and pretty much all of the tricks are already in R.

Frequentist statistics is not so cool because calculating all the specific things you want to can be a pain in the ass.

Once either Quantum computers kick in or a better algorithm than MCMC for Bayesian is created, Bayesian will win.

There are some philosophical arguments about the objectivity of the prior in Bayesian statistics, but these wash out in a decision theoretic framework because of the subjectivity of the utility function at the other end of the process.

Also, less than 5% of people reporting p-values really know what a p-value is.

<comment><USER12> Definitely agree. Do check out Stan though. HMC is pretty fast compared to BUGS/JAGS.

</comment></comment><comment><USER13> I found Udacity's class (CS373, ST101) and the 2011 ai class also explained bayesian very well by Sebastian Thrun.

</comment><comment><USER14> > _This HTML version of is provided for convenience, but it is not the best > format for the book. In particular, some of the symbols are not rendered > correctly._

I would actually recommend the opposite - they have ASCII versions of the symbols that e.g. Chrome might not render correctly, and all I checked looked fine. The PDF meanwhile copies this text from the (not linked) link to the "girl named Florida" article:

      ❤tt♣✿✴✴❛❧❧❡♥❞♦✇♥❡②✳❜❧♦❣s♣♦t✳❝♦♠✴✷✵✶✶✴✶✶✴❣✐r❧✲♥❛♠❡❞✲❢❧♦r✐❞❛✲s♦❧✉t✐♦♥s✳❤t♠❧

And the sections are linked in the HTML version, where they are not in the PDF, which seems like a simple oversight (that infects the vast majority of PDFs, sadly).

</comment><comment><USER15> Has anyone used the Think Stats book? Is it a good intro to stats?

<comment><USER16> It's really, really good, especially if you're interested in the why and not just the "give me the damn test I need to run in SPSS and what number to look at". Plus, because you spend a lot of time coding, it's more fun and less dry than most stats books.

<comment><USER17> Think Java is also excellent. I think it may have been the first (non R) programming book I read, and it helped me get more into programming which wasn't just for stats.

I have recommended think stats to many people, and it appears to be somewhat of a success. And its free documentation, which is wonderful.

</comment></comment></comment><comment><USER18> I think you and Prof Scott Page could make a great team combining your think series with his model thinking class content.

<comment><USER18> U Mich. Prof Scott E. Page <http://masi.cscs.lsa.umich.edu/~spage/>

</comment></comment><comment><USER19> thank you ! should help in coursera's pgm course somewhat i guess</comment>


The Spy Cables: A glimpse into the world of espionage - <AUTHOR> http://www.aljazeera.com/news/2015/02/spy-cables-world-espionage-snowden-guardian-mi6-cia-ssa-mossad-iran-southafrica-leak-150218100147229.html

<comment><USER2> I find it more and more concerning that newspapers who receive these kinds of leaks decide to withhold most of them or only publish them in troves over a long period. The effort required for publishing them is indubitably huge (blacking out names etc.), yet when publication is dependent on the economical impact (i.e. sales numbers of a print magazine, ad impressions, ...), the leak itself is simply being exploited.

</comment><comment><USER3> Maximizing ad profits under the guise of nobility and transparency. Yay capitalism.

</comment><comment><USER4> Where are the cables? Just a teaser?

<comment><USER5> Hehe yeah. "We're going to release some things", as far as I can tell.

I'm not sure the point, lead with something interesting.

* Not to say I don't think it's worthwhile. Just a bit dry atm.

<comment><USER5> Honestly, I'll all for it. But the article had nothing of substance. After a deluge of NSA/GCHQ/ASD/Canadian/NZ links lately, this is like saying, "They're spying on us". Sure, and I despise that, but what's the point of posting this until there is something of substance posted? Whet our tastes?</comment></comment></comment>


Information is in fact "the negative of thermodynamic entropy." - <AUTHOR> http://nautil.us/blog/yes-your-brain-does-process-information <comment><USER2> Entropy is not the negative of information, it is the expected quantity of information:

H(X) = E[I(X)] = E[-log(P(X))]

The rest seemed just as confused.</comment>


Why Netflix walked away from personalization - <AUTHOR> http://www.thoughtgadgets.com/why-netflix-walked-away-from-personalization/

<comment><USER2> This article:

[http://www.theatlantic.com/technology/archive/2014/01/how- ne...](http://www.theatlantic.com/technology/archive/2014/01/how-netflix- reverse-engineered-hollywood/282679/)

Makes the case that they are personalizing as much as ever, they just aren't using fractional stars to do it.</comment>


UNCHR Earth Time Visualization of Refugee Movement (Only on 06/20/2018) - <AUTHOR> https://www.bing.com/ <comment><AUTHOR> I have done some looking around and the source is listed as [http://maps.unhcr.org](http://maps.unhcr.org), but I can't find this visualization anywhere else.

Here is a still of it:

[https://binged.it/2yrmNPE](https://binged.it/2yrmNPE)

If anyone can find another source so the mods can update the link, that would be awesome.</comment>


7 Startups Proving Berlin Is A Startup Badass - <AUTHOR> http://tech.li/2011/12/7-startups-proving-berlin-is-a-startup-empire-in-the-making/

<comment><USER2> What's funny is how SoundCloud shows up in articles about London's startup scene as well. [http://www.guardian.co.uk/business/2011/nov/27/tech-city- dig...](http://www.guardian.co.uk/business/2011/nov/27/tech-city-digital- startups-shoreditch)

For the record all their jobs are in Berlin so score 1 for the Germans and -1 for british journalism :-) <http://soundcloud.com/jobs>

<comment><USER3> Soundcloud has offices in London. They just opened up a new office in SF recently as well. But you're right in that their main HQ is definitely in Berlin, although were originally founded in Sweden!

Amazing company!

</comment></comment><comment><USER4> Sorry to say this, but it sounds pitiful when the final line of your number 1 startup is "even attracting attention and investment from Hollywood bigwigs such as Madonna's manager Guy Oseary and Ashton Kutcher."

That said, I'll be in Berlin this weekend to check out the scene.

</comment><comment><USER5> 6wunderkinder.de is definitly missing.

Also there is a huge split in between the copy-cat businesses usually operated by the samwer brothers and the next generation startups like soundcloud, 6wunderkinder, wooga and others.

<comment><USER6> I was thinking the exact same thing while reading this. Wunderkinder is missing and the majority of these companies are copy-cat businesses. Granted, Wunderlist is a to do app which there isn't a shortage of, but they've managed to keep their product free while maintaining cloud sync, great UI, and relevance among its productivity community. Meanwhile they opted to include a general "rebate networks" and deal network companies. This is my long-winded way of saying this list is mediocre and Wunderkinder deserves to be on a list like this.

<comment><USER7> Just checking out Wunderkinder for the first time. Thanks for passing it along. I've never heard of most of these companies before, but I'm glad to see something other than SanFran startups.

</comment></comment></comment><comment><USER8> I'll be working out of Berlin for a month or two early next year. Looking forward to it.</comment>


Hacking suspect could kill himself if extradited to US, court told - <AUTHOR> https://www.theguardian.com/uk-news/2016/jun/28/lauri-love-hacking-trial-extradition-united-states-nasa <comment><USER2> Sounds like a really weak excuse to weasel out of it.

"Don't let me answer for capital crimes, I might kill myself!"</comment>


Show HN: Phishar – Security Expert in Your Pocket - <AUTHOR> https://www.phishar.com/ <comment><USER2> How does it cope with non-ASCII characters and various fonts? Suppose you had "ɡoogle.com" instead of "gooble.com" in that demo video (the first "g" is not a regular "g", but " U+0261 LATIN SMALL LETTER SCRIPT G")… Browsers tend to show IDN names in the "xn--…" punycode format, but not always.

[https://en.wikipedia.org/wiki/IDN_homograph_attack](https://en.wikipedia.org/wiki/IDN_homograph_attack)

[https://unicode.org/cldr/utility/confusables.jsp?a=google&r=...](https://unicode.org/cldr/utility/confusables.jsp?a=google&r=None)

[https://ma.ttias.be/show-idn-punycode-firefox-avoid- phishing...](https://ma.ttias.be/show-idn-punycode-firefox-avoid-phishing- urls/)

</comment><comment><USER3> Thanks for posting!

While this might be common knowledge on this forum (is it?!), many users are unaware that even having 2FA does not protect them against credential theft via phishing. Such attacks are growing rapidly given recent exploit kits for OTP-based 2FA that are publicly available on Github ([https://github.com/drk1wi/Modlishka](https://github.com/drk1wi/Modlishka)).

PhishAR prevents such attacks by requiring that the visited domain and SSL are first checked by the user's app and only then revealing the OTP.

Any feedback would be very helpful. We are aiming at enterprise users where getting everyone to use FIDO U2F might be harder to achieve, but 2FA is mandatory (e.g. due to GDPR).

What do you think? Let us know if you would be happy testing the app once we get it a bit more polished.</comment>


Analyzing Your Browser History Using Python and Pandas - <AUTHOR> https://applecrazy.github.io/blog/posts/analyzing-browser-hist-using-python/ <comment><USER2> It seems a bit odd to export the data in the command line to then work with it in pandas, given that it's a stock sqlite3 database. I'd just use sqlite3.connect and pd.read_sql_query directly. (I know, the db will be locked when Chrome is running, but you can always shutil.copy it to a temp location.)

<comment><AUTHOR> Hmm something to keep in mind for next week's post.

<comment><USER3> Had the same idea with issuing the sqlite query directly from python. Unfortunately the browser has to be closed in that case, so I just copied the sqlite "History" file to temp location: [https://github.com/davidgengenbach/chrome_history](https://github.com/davidgengenbach/chrome_history)

</comment></comment></comment><comment><USER4> These two loops:

      raw_data = [line.split('|', 1) for line in [x.strip() for x in content]]

can be simplified to a single loop:

      raw_data = [line.strip().split('|', 1) for line in content]

Using str.replace here is also non-idiomatic:

      plt.title('Top $n Sites Visited'.replace('$n', str(topN)))

How about using str.format instead:

      plt.title('Top {n} Sites Visited'.format(n=topN))

<comment><USER5> You could even use the new format strings from Python3.6.

      f"Top {topN} Sites Visited"

</comment></comment><comment><USER6> Neat. As a Firefox user, it appears that this does the trick:

        sqlite3 ~/.mozilla/firefox/$YOUR_PROFILE_ID/places.sqlite "SELECT datetime(visit_date/1000000,'unixepoch'),url FROM moz_historyvisits JOIN moz_places ON place_id=moz_places.id ORDER BY visit_date DESC" > hist.txt

(Looks like the timestamp is in UTC here.)

\----

Does the original post's code perhaps count two visits to the same URL as one? Something like

        SELECT datetime(visit_time/1000000-11644473600,'unixepoch'),urls.url
        FROM visits JOIN urls ON visits.url=urls.id
        ORDER BY visit_time DESC;

might be worth a try.

<comment><USER7> Now if only Buku [0] used this to poll and automatically synchronize my bookmarks...

[0]: [https://github.com/jarun/Buku](https://github.com/jarun/Buku)

</comment></comment><comment><USER8> You can skip the entire SQLite output to file by reading directly from the database: [http://pandas.pydata.org/pandas- docs/version/0.20/generated/...](http://pandas.pydata.org/pandas- docs/version/0.20/generated/pandas.read_sql.html)

</comment><comment><USER9> Just tried this, but am getting an error with the datetime. How could I fix this?

      Traceback (most recent call last):
        File "hist.py", line 14, in <module>
          data.datetime = pd.to_datetime(data.datetime)
        File "/usr/local/lib/python2.7/site-packages/pandas/core/tools/datetimes.py", line 373, in to_datetime
          values = _convert_listlike(arg._values, True, format)
        File "/usr/local/lib/python2.7/site-packages/pandas/core/tools/datetimes.py", line 306, in _convert_listlike
          raise e
      pandas._libs.tslib.OutOfBoundsDatetime: Out of bounds
      nanosecond timestamp: 1601-01-01 00:00:00

The time on my Mac is correct, and datetime gets the correct time?

<comment><AUTHOR> Aah I got this strange issue too. Search for that 1601-01-01 time stamp in the exported text file and change it to something else that isn't before the Unix epoch

<comment><USER9> That sorted it, cheers!

How long is the Chrome database kept for? I use Chrome Canary and the standard Chrome, but only had a few hundred results in each, a lot less than I was expecting. I can't remember the last time I cleared my history either.

Cool graph though anyway!

<comment><AUTHOR> I'm not sure how long the history is stored, but I had over 2k entries for six months of history.

And, by the way, I'm not done with this series. Next week I'll try to predict browsing patterns using this dataset.

</comment></comment></comment></comment><comment><USER10> Awesome first content post, man! I like yours better than mine: [https://andythemoron.com/blog/2017-03-12/Scraping-For-Fun- An...](https://andythemoron.com/blog/2017-03-12/Scraping-For-Fun-And-Profit)

One note you may want to add (or step you may want to tweak) is that the initial sqlite3 command won't work if chrome is currently open. I just copied the History folder to somewhere else and did the same thing, as per the superuser thread you helpfully linked to.

</comment><comment><USER11> There's a video (now pay-walled) from Cameron Davidson Pilon of Bayesian Methods for Hackers fame, where he takes this approach to build a basic Markov Chain model to predict patterns in his browsing.

[https://dataorigami.net/collections/all/products/create- mark...](https://dataorigami.net/collections/all/products/create-markov- chains-using-your-chrome-browsing-history)

</comment><comment><USER12> Great post for a quick read! You could maybe have taken your google searches from within the URL and Stack Overflow question titles and done word frequency analysis and maybe learned what languages you code in most often/what type of problems you have most as well. Good intro for a quick read though.

</comment><comment><USER13> I don't think that the pulled data is *extremely messy(. You can also read `hist.txt` with:

      pd.read_csv('hist.txt', sep='|', header=None)

<comment><AUTHOR> Not quite. I tried that but some of the entries had vertical bars in URLs (Amazon urls), making Pandas complain and refuse to import the data.

<comment><USER13> Ahh I see. Then you should maybe us the Amazon example in your post to clarify why you are using this approach. Nice post by the way.

<comment><AUTHOR> Thanks for the feedback. Will edit my post soon.

</comment></comment></comment></comment><comment><USER14> I have a far more mundane use case - you have friends over, play songs off of youtube all night, after the fact, would like to publish a playlist on youtube, but it's not simple using browser history. I always figured I'd have to write a plugin and find clusters of short youtube sessions and go from there, but as a backend guy, I might play around with this stack instead.

</comment><comment><USER15> > Since Pandas probably doesn't do this out of the box

It does, use df[column_name].str.split(expand=True), where df is your DataFrame.

<comment><AUTHOR> I had pipe symbols in URLs. Will it split only on the first pipe symbol?

Edit: nevermind, it does do that ([https://pandas.pydata.org/pandas- docs/stable/generated/panda...](https://pandas.pydata.org/pandas- docs/stable/generated/pandas.Series.str.split.html))

<comment><USER16> Why do you mind? You throw away the rest of the URL in the next step anyway. :-)

</comment></comment></comment><comment><USER17> I'm learning Python currently and this is the sorta stuff that's neat to see!

I haven't had any reason to use pandas yet so it'll be a good excuse.

It's a project that's not very long, requires data everyone has and gives a cool insight at the end.

Nice!

<comment><USER18> Also learning Python and intending to learn about pandas and other data tools with this: [https://github.com/jakevdp/PythonDataScienceHandbook](https://github.com/jakevdp/PythonDataScienceHandbook)

I haven't made it very far yet, but so far it seems worth recommending.

</comment></comment><comment><USER19> I think the location on Linux (Ubuntu 16.04) is:

        ~/.config/google-chrome/Default

or:

        ~/.config/chromium/Default

<comment><AUTHOR> Thanks. Will fix soon.

</comment></comment><comment><USER20> This type of history stats is buildin out of the box in Vivaldi browser.

</comment><comment><USER21> What about the cmd+shift+N stuff ;)

<comment><AUTHOR> I'm not sure, there might a way to obtain the incognito cache and determine browser history from that, if that's what you're after.

</comment></comment><comment><USER3> Thanks a lot for this code! If someone is interested and does not want to copy the code by themself, I copied the code into a jupyter notebook: [https://github.com/davidgengenbach/chrome_history](https://github.com/davidgengenbach/chrome_history)

</comment><comment><USER22> Vivaldi does this for me in the history page itself. Pretty weird to use Python just to see a distribution of what sites you visited.

Iirc stock chrome also has a way to view this.

<comment><AUTHOR> Author of the post here.

The main purpose of the post was to demonstrate collection and cleaning of data and give an overview of it through a basic visualization.

In the future, I'd like to show how this data is a gold mine of information, using it to predict browsing trends, create a profile of interests, and more.

Mainly to show why ad tracking/selling of user browser history is bad, but also to teach some data science techniques along the way.

<comment><USER23> OT: which Hugo theme are you using? Custom?

<comment><AUTHOR> I'm using the Minos Hugo theme.</comment></comment></comment></comment>


Quora Chrome Extension - <AUTHOR> https://chrome.google.com/webstore/detail/official-quora-chrome-ext/blgpiknoghkmnhcneegdaknfckbkphhd

<comment><USER2> It's an optional toolbar that shows a number of new items. Nobody is required to install it and the description and screenshots are clear. I don't see how that is spam.

</comment><comment><USER3> Please don't editorialize the titles of stories you post to HN. This one was egregious: "Quora becoming desperate, makes browser extension to spam notifications".

Totally not ok.</comment>


Why Are Criminals Stealing Tide Detergent and Using It for Money? - <AUTHOR> http://www.theatlantic.com/business/archive/2012/03/why-are-criminals-stealing-tide-detergent-and-using-it-for-money/254631/ <comment><USER2> That's the best thing to steal? But when you think about it, it's tough to come up with something smaller or lighter that's still 20 bucks. Apart from razor blades, but those tend to be behind plastic these days.</comment>