GitHub releases an ImageNet for code and a CodeSearchNet challenge - <AUTHOR> https://github.blog/2019-09-26-introducing-the-codesearchnet-challenge/ <comment><USER3> Hello folks, this is Hamel from GitHub -- I'm one of the Machine Learning Engineers who worked on this project. The reason we are excited to host this data is that we believe the community will be able to innovate and advance the state of the art much faster if it is provided in a tractable format for machine learning researchers. This data is already public, however there is specialized knowledge required to acquire, parse, dedupe, and clean code from many programming languages at a massive scale on GitHub. We strived to reduce these barriers to encourage greater involvement.

While we present the task of information retrieval as one possible use case of this dataset, we know there could be other practical applications of this data (i.e. code summarization). While we went through great lengths to pre-process the data for the community, the data is still messy and often you will find that there might not be high-quality comments that are aligned with a code snippet we parsed. However, we believe this is part of the excitement of the dataset it poses challenges that machine learning practitioners will have to address.

Code is very different than natural language with regard to its structure and syntactic rules and may benefit from different approaches relative to natural language processing. Our baseline models and benchmarks mostly treat code as natural language, however we are aware that there could be an opportunity to innovate on this front. If anyone creates any interesting projects from this dataset, please do get in touch. Happy to answer any questions!

<comment><USER4> I know this sounds silly because code on github is visible to everybody anyway and that's a good thing, but I would appreciate a way to opt out my own code from such automated data gathering programs for machine learning purposes.

Something like a robots.txt for github projects. Not that anybody would really care, only to make my intent clear that I don't support this sort of mass data gathering nonsense.

<comment><USER5> You opt out by using a restrictive license or making your repo private.

<comment><USER4> Using a restrictive license seems a bit extreme when the whole point of a public github repo is to make the code available for other programmers.

I consider this mass data gathering use for machine learning an edge case which should not harm the 'legitimate' users of the repository.

<comment><USER6> Why do you see this effort as not "legitimate"?

</comment></comment></comment></comment></comment><comment><AUTHOR> Shawn from Weights & Biases here. We've been working with Github and Microsoft Research on this for just about a year now and we're super excited to launch it today.

We've seen huge advances in human language modeling and translation due to the success of deep learning. Often new directions start with a really motivated team producing a new kind of dataset. Who better to do that for code as language than Github!

This started as a grassroots effort inside of Github, and went through many iterations. When it was presented to Github's CEO six months ago, he correctly pointed out that we needed to go back and include Github's most popular language (javascript). As the project went on many smart people chipped in, and we produced something that we think is truly useful.

Check out the paper here: [https://arxiv.org/abs/1909.09436](https://arxiv.org/abs/1909.09436)

We overcame plenty of challenges to pull this off. For example: how do you clean this data? how do you label it? We've got folks from Github, Microsoft Research and Weights & Biases here to answer any and all questions you might have. Can't wait to see where this goes!

<comment><USER7> > We've seen huge advances in human language modeling and translation due to > the success of deep learning.

I wonder if we'll eventually see a system where instead of writing code you describe in natural language what you want the program to do and then ML is applied in order to generate the code for that.

I mean, a lot of people have been interested in the past in making human programming languages, and had varying degrees of success.

Personally I love writing code but, it could be, couldn't it?

Write some unit tests, a human description of what it does and based on the source code and description of existing software the system would basically "debug the program into existence" for you.

That'd be kind of freaky, kind of cool and a little bit scary.

<comment><USER8> Find a human who wants a computer program written (A). Find a human who can program computers (B). Have A describe what they want and have B code it without asking questions to further clarify the issues. What do you expect the result to be like? For me, my experience tells me that it will be a total failure.

The problem with programming is _not_ the encoding of the requirements in programming language for the most part. The problem is that the specifier (A in this example) usually does not have a full grasp of what they actually want. In fact, they usually don't have any idea at all. "Give me a e-shopping system to sell comic books" is the level of detail they can understand.

The closer A can come to expressing the requirements they need, the closer they are to actually being B in reality. B's real skill is not in knowing the syntax and grammar of the computer language, it's in knowing that in order to make a system that will satisfy A we need to do X, Y, and Z to the tiniest detail.

When we get into trouble with our software is when we write code that is dramatically more complex than the problem we are trying to represent. This doesn't happen so much because we don't know how to program. This happens because we are slowly extending the code base over time with imperfect knowledge about what we are ultimately building at any given time. We also have to trade-off the benefit for getting something done with discovering generalities that allow us to simplify the expression of code that we already have.

I don't think we will ever replace "programmers" with AI -- at least not until the AI can be trained to ask the important questions about what the system _really_ needs to be (and for that we need turing-test passing level AI). I think it's much more likely that we will build more and better tools that help programmers visualise and plan the programming situation. I think we _will_ have automatic code generation because we already have it: Look at "derive" in Haskell and Rust, for example. But I think _that 's_ the level of automatic code generation we're going to want for at leas the next 20 years or so.

Interestingly for testing, I think we'll actually go the opposite direction: We will spend more time thinking about requirements and the computer will help us by writing tests that challenge our assumptions: I've broken this function, are you sure you got it right? Again, we already have these kinds of systems and I think that _this_ is the most appropriate direction to invest in research.

<comment><USER9> The only way I think something like what you described in your first paragraph would work is if you had an AI system B that could present questions and prototypes back to requirement-setter A for feedback. Of course, that'd be a very difficult problem, even if you limit it to a constrained domain.

</comment></comment></comment></comment><comment><USER10> Code challenges like this are interesting - thank you for putting this together!

I do have a question about the set-up, if that's alright. Netflix and others have found that shared tasks can lead to great models, but not necessarily ones that are suited for use in a production environment. Have you put much thought into how best to set up a challenge such at this to make the obvious "ensemble everything" solution be less worthwhile?

Similarly, have you put much thought into how to encourage the sharing of information between participants?

Thanks again.

<comment><USER11> Excellent questions, thank you!

1\. We could log additional information about the model, such as inference time, number of parameters, memory usage, etc. and have the primary metric be overall efficiency (best NDCG with fewest parameters/fastest runtime/etc).

2\. We're experimenting with different kinds of benchmarks, and I am most excited about explicitly collaborative ones. In these there is no contest/prize (hence no incentive to cheat/withhold information); only the shared goal of improving the model and our collective understanding of the problem. I hope we can incentivize information sharing by tracking and acknowledging individual contributions to the eventual best model in the benchmark. We could approximate individual contribution by seeing which scripts, code segments, workflows, architectural changes, writeups, or discussion comments other participants rate as the most helpful or choose to include in their experiments most often as the benchmark evolves. Of course this could only be an estimate--as Shawn says above, any idea could have "actually happened in a hallway conversation". Still, this is much easier to achieve in a logging/visualization platform like W&B than in the current paradigm of "read research papers, clone relevant repos, spend weeks trying to synthesize/reproduce their results, run your own experiments, write them up in a research paper, hope it gets accepted to a conference before other people publish the same idea, try to integrate your changes/publish your own repo, repeat"\--and for hundreds of practitioners, ranging from brand new students to PhDs, working on related problems. This cycle is especially challenging for folks who are new to, working outside of, or trying to collaborate across the relatively few established/well-funded academic/industrial teams.

Collaborative benchmarks can be especially impactful for social good projects, where the primary incentive is to figure out and broadly implement the best solution ASAP (e.g. climate change!), not to make money or argue over the credit attribution. So, my long-term goal is for as much sharing of information and collaboration from as many folks as possible--the more inclusive and transparent the field of deep learning research becomes, the safer and better its outcomes. Very open to ideas on how to help make this happen.

~Stacey, deep learning engineer at W&B

</comment></comment><comment><USER12> Great effort in putting together this large corpus! While reading through your paper, I noticed the difficulties you faced in correctly annotating the code for their quality, correctness and hiring annotators for different languages. I can imagine how herculean this task could be.

I was wondering if you thought to include stackoverflow questions and answers, which have been vetted by thousands of programmers over a long period of time. Stackoverflow might even want to participate in this effort to provide a clean ground truth for this great project.

<comment><USER13> [I'm one of the Microsoft Research people who worked on this]

We did consider adding StackOverflow questions. Some of our queries in the CodeSearchNet challenge do actually come from StackOverflow (via StaQC [1]). It's certainly interesting to see how all other SO data can be useful for this task. Thanks for the suggestion!

The reason we didn't try this at this point:

Many people in research have tried working with SO data. In my experience I have observed an interesting problem with the data: it's deduplicated! This is great for users but bad for machine learning, since the data looks "sparse" (roughly, each concept appears once). Sparsity is an obstacle, since it's hard for most existing machine learning methods to generalize from sparse data. In contrast, in natural language there are (e.g.) multiple articles describing the same event more or less.

[1] [https://ml4code.github.io/publications/yao2018staqc/](https://ml4code.github.io/publications/yao2018staqc/)

</comment></comment><comment><USER14> Code reuse search engines is something that Haskell is excellent for. With Hoogle you can search for a generic type signature and probably find the method you want.

Want to flatten an array of Maybes? Just search for [Maybe a] -> [a] and you'll find catMaybes and takeWhileJust.

This of course only works for Haskell because so much of a function is determined by it's types. Doing this sort of thing in say JavaScript would be an absolute nightmare. Still, it's an interesting train of thought.

</comment><comment><USER15> I've been experimenting with my own custom markup for annotating code blocks. I want to document code use cases and automatically generate an index of independent sandbox examples. This approach doesn't require AI but rather manual effort remembering to annotate novel uses in code and a standard convention for doing so. One problem this solved is that I've written so much code that I lose track of where I experienced certain patterns and this would help me track that better. This also helps to facilitate knowledge sharing across teams with access to the source.

Has anyone worked on something such as this and can comment or share ideas?

<comment><USER16> sort of ...

I am playing with `todoinator` - it is a way of finding all the "todos" I leave scattered around my code, but also gives me ways to rank my code - <USER2> not quite where you are but I think the principle of having almost everythin derived from code is the guiding light here.

[#] [https://github.com/mikadosoftware/todoinator/blob/master/tod...](https://github.com/mikadosoftware/todoinator/blob/master/todoinator/todoinator.py)

</comment></comment><comment><USER8> I haven't looked at this in detail, but where I could see this technology going is by doing a kind of lint. We've got hand built linters for most languages and some of them are really good (Clippy in Rust is amazingly good -- it practically writes my code sometimes). However, a tool that analysed my code, picked out things that were idiomatically strange and then suggested example code that might be better _would_ be quite useful, I think.

<comment><USER17> I'd like to type my code in one language and have it translated into another.

Someday I'd like to learn Rust, for example.

If I could leverage the languages that I already know, I could more quickly build something more useful.

<comment><USER18> A seq2seq model trained on something like roseta code might be a viable way to do that...

</comment></comment></comment><comment><USER19> A phrase is never enough to describe what the task at hand is. It may work for simpler use cases like the results on stack overflow. Otherwise I do not see it doing better than a google search which leads me to stack overflow.

<comment><USER20> You're right. We'll need bigger datasets and more complex models to achieve more useful results, but machine learning models can give predictions in real time so you could have this running as you type in your IDE, instead of visiting a website, also with a model fine tuned to your code base. "simpler use cases like the results on stack overflow" could explain at least 10% of the work I do, so even just that tool would be very useful for mediocre programmers like myself.

Behind all the hype, predictive text is something machine learning models are beginning to do very well. G-mail has rolled out a lot of similar features from advancements in deep learning models.

</comment></comment><comment><USER21> > _Searching for code to reuse, call into, or to see how others handle a > problem is one of the most common tasks in a software developer's day_

I don't disagree at all that this is how we code these days... but I distinctly remember a time when this wasn't so. We had to do everything ourselves. We engineered our solutions based on various requirements and constraints, and most importantly, we had to figure it out ourselves. The only external help we had was with the APIs we used... and they had to be _studied_ to be understood.

Even in recent times, the most fun I've had _programming_ has been when it's all come from my wee little head, rather than trawling for solutions and shoehorning/ reworking something similar.

<comment><USER22> Only mildly related, but I had a recent experience that made me fond of Java's way of doing things that your comment reminded me of.

I had to use a tool for work (name withhold to protect the guilty) that has awful documentation. The tool allows you to write snippets of your own code, but provides no IDE and no documentation (AFAIK) of anything but the most trivial aspects of the API.

I started using Python, but with no debugger and no interactive shell there was no way I was going to guess the names of the functions I needed. Lucky for me, someone uploaded the Javadoc of an older version of the API, and that was the missing piece of my puzzle: having the function names, the return types, and Java's stack traces, I now had all I needed.

Back to the topic: like you, I sometimes wonder if there's a downside to not having to scroll through hundreds of manual pages anymore. But until someone shows some kind of evidence of something being lost, I won't worry too much.

That said, I definitely wish more companies would make their documentation available offline, if only as a static version of the online version. For those of us who regularly program in trains and planes, offline docs are a lifesaver.

</comment></comment><comment><USER23> A little off-topic.

> Our fully preprocessed CodeSearchNet Corpus is available for download on > Amazon S3

I am surprised that Github went with S3 for this download. Isn't there a Azure equivalent of S3 for large object storage ? This just shows the dominance of AWS.

<comment><USER24> It's a good thing that MS don't force every team to always choose MS product, especially for a trivial thing like a download storage. Maybe the team was more familiar with AWS, who knows. I'm just glad that they can make this decision.

</comment></comment><comment><USER25> Looks like they're using tokens as input as if it's natural language. You have an AST, use it. I think the limitation is the lack of graph and tree based neural networks, but I don't think you need a neural network to search code. You already have the AST. This could be solved with a traditional analytic approach, but it's probably a lot of hard work.

</comment><comment><USER26> Something like this might be useful for making better "documentation" for things like API's and man-pages.

Documentation tends to be either simplistic "hello-world" examples or everything-but-the-kitchen-sink dumps that take forever to consume. Neither of these is helpful to a practitioner that just wants to get through a basic task without starting a mini-research project, or getting burnt to a crisp on stackoverflow.

So basically, I am thinking this corpus could be used to filter for specific examples of usage within particular contexts and problem domains? Or maybe not?

</comment><comment><USER27> Since the CodeSearchNet Corpus contains metadata such as owner/repository, it would be nice to create a search tool for the data set itself. That way you could check if, by chance, some of your open source code is part of the corpus.

The data set is apparently ~20GB [0], so a cheap VPS instance might do the job of hosting the data in a searchable format.

[0] [https://github.com/github/CodeSearchNet#downloading-data- fro...](https://github.com/github/CodeSearchNet#downloading-data-from-s3)

</comment><comment><USER28> I think this is fantastic and I'm excited to see what people come up with here.

It's very disappointing to see a lot of the negative comments here, almost completely around licensing, despite the licensing being well explained.

You'd think this place was full of lazy bureaucrats.

</comment><comment><USER29> I'm not trying to be a jerk here, but screw it.

I'm sure whatever it is you've done, I'll probably use at Github at some point. However, from my 10 second scan of the page, I know absolutely NOTHING about what this is or what it can do for me. Is it just me or do I _constantly_ see example of developers making the worst marketers in the world? Where are the code examples or videos showing whatever it is this is and how it will help me. There is nothing on that page but a HUGE image and a bunch of text I can't understand without 4 PHDs and a half a tab of Adderall.

It's so frustrating see stories on Hacker News that are just piss poorly explained. You've most likely worked hard at this for a long time, why is it you can't take 5 minutes to explain it in lay man's terms for all to understand?

END RANT

<comment><USER30> This is a release of a dataset, to help people train machine learning models. That's why the blog post says "We're also releasing a large dataset to help data scientists build models for this task."

We also provide a way to evaluate how well your machine learning model works. That's why the blog post says "We're announcing the CodeSearchNet Challenge evaluation environment."

The released data and code (and hence, the announcement) are meant for data scientists and ML researchers who want to work on this problem, and the rest of the world does not need to care. There are no products or applications of this work at this time. If the terms in the blog post don't mean anything to you, then you are not in the target audience for this announcement, and that's OK.

</comment></comment><comment><USER31> I remember a search engine for developers called Symbolhound. Wasn't fruitful for me bit I'll never very forget it.

</comment><comment><USER32> Very cool! I wish I had more time!</comment>


Peter Molyneux: My Next Game a 'Significant Scientific Achievement' - <AUTHOR> http://blog.wired.com/games/2008/05/molyneux-my-nex.html <comment><USER2> You have to admire someone trying to do better than just "a better EverQuest/Ultima Online" (which is basically what World of Warcraft is) or a better Quake (Half-Life, Unreal Tournament) or whatever, but it can also really backfire on you.

The micromanagement in Black and White was terrible. "Villagers need food!"

</comment><comment><USER3> ""70 percent of people will be good," he says. "20 percent will dabble with evil, then be good. Only 10 percent will choose to be evil all the way through. What's fascinating is that this is very regionally dependent. It's different from Germany to the U.K. to Asia.""

So, what is the regional dependence?

</comment><comment><USER4> _"The way games are made now is fundamentally flawed," he responds. "If I was a betting man, I'd imagine that in the future, this business of getting more than a hundred people together for three, four years will look really odd. It's so incredibly expensive. I predict that we'll see a core of deeply talented people working on games beforehand, then a big team comes together for a brief period of time."

So, I say -- game design is moving toward something sort of like a movie, with years of preproduction by a small team, then a big production with set designers and key grips that only lasts a couple of months?

"Yes, exactly."_

If you can solve this problem, you will transform the face of the game industry and usher in a new era of creativity in the face of the current method of massive teams and budgets. Is it a good idea for a startup? Beats me. It's a large, open question. I don't have an answer as to how to approach a solution. Yet.

<comment><USER5> Agreed. I'd personally love to design a game world and story, but am not willing to devote 4 years of my life to just making something rudimentary.

<comment><USER6> There's a host of interesting startup ideas in this field, most related to 3D graphics. We need higher levels of abstraction for creating textures, architecture/models, decoration etc. The tools we have today are awfully rudimentary, and there's research indicating that much better solutions are possible.

See for instance [http://www.vision.ee.ethz.ch/~pmueller/wiki/CityEngine/Docum...](http://www.vision.ee.ethz.ch/~pmueller/wiki/CityEngine/Documents) \- a higher-level language for describing architecture. In the marketplace there are things like <http://en.wikipedia.org/wiki/SpeedTree>, which is already in use in a great number of games.

It is obvious that we could create a huge amount of tools to this purpose, and also that there is a huge market for software that makes game development easier. What's the sense in creating the 5000th 3D model of a human? Humans don't change that much. Not even fictional bipeds do. Have you ever heard a game developer or pundit complain about 'spiraling costs' for game development? It's because human developers have more and more work to do to make the games keep track with developments in hardware. It should be obvious to anybody that this development isn't sustainable; there's no way every room of a game designed by humans can have even as much detail as the room I'm sitting in right now. But most of this architecture is predictable, and could probably be generated by the right program.

Consider a lower bound of game development costs for this 'generation' of consoles at 3 million dollars. If even five percent of that cost could be eliminated by better development tools, that would mean savings (and potential for profit!) of 150,000 dollars. But the potential savings from automating menial modeling tasks are a _lot_ higher.

Seriously. We need software to make stuff like this easier. There's a major can of worms in implications of sorting this out - a lower barrier to entry, more creative freedom, better-looking games and so on and so on...

</comment></comment></comment><comment><USER7> This morality thing was already done nicely in the early Ultimas (I think starting with IV, or V). I remember the one dungeon where you are attacked by a monster called "little children". Before you had been trained to never fight children because it immediately makes you lose your virtues.

I don't think Molyneux approach sounds very scientific, though. What is scientific about giving game figures an artificial idea of morals? Scientific for me would be to not make those inbuilt, but let them evolve and see why certain morals make for more successful societies than others.

Sorry to rant about this, but it is one of my pet peeves with games (like civilization). They claim to be simulations, but they are not - they are just projections of the game designers limited perspective and morals. Of course the same could be said for many books and movies, too. I don't like it there, either, but somehow I feel that with games it is another dimension. The pseudo-simulation somehow makes the opinions of the game creators seem more legitimate.

<comment><USER8> "What is scientific about giving game figures an artificial idea of morals?"

I got the impression that the "next" game he was talking about, the scientific achievement that would be on the cover of Wired, would be the one _after_ Fable 2.

<comment><USER7> True, but he didn't mention anything new, so I assume he is just taking AI to the next level or whatever...

</comment></comment></comment><comment><USER9> I once emailed Steve Jackson (UK, Games Workshop, worked on Black & White) about a signed copy of Warlock of Firetop Mountain (paperback first edition). It had a 'dedicated to' the best guest house in that region of Scotland, and he actually remembered it. It's a pretty cute treasure.

</comment><comment><USER10> Unfortunately I can't read it at work, but it does interest me. So what's the proposal about?

</comment><comment><USER11> anyway i dig Molyneux, he boasts a lot but it comes from loving games.

he has done some remarkable games and always try to be inventive.

a true visionary and one of the great game designers.

<comment><USER4> He really does love his work. I admire his enthusiasm and dedication to ideals. It's infectious and he tends to bring out the best in otherwise jaded gamers. His Achilles' heel is that nobody can completely live up to those ideals. His audience feels disappointed when they see the flaws in the final product.

We need someone like him to keep pushing us forward. Bright futures don't _come_ , they are _made_.</comment></comment>


Pure JavaScript Version of Closure Compiler - <AUTHOR> https://github.com/google/closure-compiler-js <comment><USER2> I assume this will be nice for stuff like Atom and VS Code, as you could embed the compiler directly into the editor and not have to run it externally.</comment>


Mozilla warns decryption laws will break open source - <AUTHOR> https://www.itnews.com.au/news/mozilla-warns-decryption-laws-will-break-open-source-513898 <comment><USER2> In which way could Mozilla be coerced in building a backdoor into its software by the Australian government, in a way not being seen by the public? I can't imagine that an Australian law would either lead to shutting down open source projects nor lead to make them (or parts of it) closed source. Would Signal do this? I don't think so, they'd loose all trust immediately.

The only problem I see is that open source projects might not accept contributions from Australian residents or citizens.

<comment><USER3> Presumably in a way similar to Mozilla implementing closed source DRM in Firefox?

<comment><USER2> I can switch that off (afaik). And the question remains: how can Mozilla compelled to do this?</comment></comment></comment>


Defense subcontractor posted a job listing for XKEYSCORE 2 weeks ago - <AUTHOR> http://www.washingtonpost.com/blogs/worldviews/wp/2013/08/01/is-xkeyscore-still-active-defense-contractor-posted-a-job-listing-for-it-2-weeks-ago/ <comment><USER4> Also interesting in that the job description 'leaks' another name for a system, SKIDROWE. Some quick searches only turn up the same or similar positions for open for XKEYSCORE.

Cryptome is already on it: [http://cryptome.org/2013/08/nsa-xkeyscore- saic.htm](http://cryptome.org/2013/08/nsa-xkeyscore-saic.htm)

<comment><USER5> It's pretty easy to collect large lists of these types of code names from places like linkedin. Search for code name, find new code names, repeat.

[https://www.google.com/search?q=site%3Alinkedin.com%2Fpub%2F...](https://www.google.com/search?q=site%3Alinkedin.com%2Fpub%2F+XKEYSCORE)

<comment><USER6> Ah, I like the guy who lists his databases knowledge to include Excel & XKeyScore. Either XKeyScore is so slick that it is indistinguishable from Excel or this particular person doesn't know what a database is. Either way, that cannot be good.

<comment><USER7> At the end of the day, it's evidence of the NSA having low-skills computer users rifling through your calls and internet. Make of that what you will.

</comment></comment></comment></comment><comment><USER8> The first thing I did yesterday after seeing Snowden's leaked Powerpoint was search for mentions of XKeyscore in the past and I came across these same job postings (and copied them down since I doubted they would last).

I started compiling a database of the different programs, what's known about them, what you can do to stay off their radar, etc. Sound interesting to anyone?

Programs/tools I came across include: AGILITY, ANCHORY/MAUI, AUTOSOURCE, CONTRAOCTAVE, WISE, INFOSHARE, TREASUREMAP, TUNINGFORK, SCORPIOFORE, TAPERLAY, MAINWAY, PINWALE, Tripwire Analytic Capability, Combating Terrorism Knowledge Base (CTKB), etc. Quite a few and some of those names are Hollywood quality.

Tools that HN readers would know about that were mentioned: ArcGIS, Wireshark, IDA Pro, OLLY Dbg, Snort, Analyst Notebook.

<comment> fsck--off This article [1] also mentions finding lists of program names from LinkedIn profiles, especially this one [2].

[2] mentions:

      ANCHORY, AMHS, NUCLEON, TRAFFICTHIEF, ARCMAP, SIGNAV,
      COASTLINE, DISHFIRE, FASTSCOPE, OCTAVE/CONTRAOCTAVE,
      PINWALE, UTT, WEBCANDID, MICHIGAN, PLUS, ASSOCIATION,
      MAINWAY, FASCIA, OCTSKYWARD, INTELINK, METRICS, BANYAN,
      MARINA

The names of the programs aren't classified; if they were they would not show up on LinkedIn. What the programs actually do _is_ classified information.

[1] [http://front.kinja.com/job-networking-site-linkedin- filled-w...](http://front.kinja.com/job-networking-site-linkedin-filled-with- secret-nsa-pro-514057863)

[2] [http://www.linkedin.com/pub/jason- miller/39/741/a49](http://www.linkedin.com/pub/jason-miller/39/741/a49)

<comment><USER8> Names are the first step though. You can collect a decent amount from them, like when things started, if they are currently in use, which companies deal with which programs, etc.

<comment><USER9> Codenames can also be useful when social engineering. "Oh, it's ok, I know all about FOXYROT, what part did you work on?"

</comment></comment></comment></comment><comment><USER10> So - do you guys apply to these jobs? We need more Snowdens!

By the way - have you guys noticed the text RMS adds to his emails recently:

      [ To any NSA and FBI agents reading my email: please consider
      [ whether defending the US Constitution against all enemies,
      [ foreign or domestic, requires you to follow Snowden's example.

</comment><comment><USER11> I've been thinking about making a switch over to working for a defense contractor over start-up.

The pay seems to be better in lower cost of living area's (Washington DC metro vs. SF); job security better in BigCo like Mitre, SAIC and Booze and less competition from H1B visa holders that cannot obtain security clearance easily.

Get to work on cool technology with big scaling and parallel processing over web CRUD frameworks. Anyone has experience making the switch?

<comment><USER12> Why would you want to work for a defense contractor after seeing all the awful things they do on behalf of the U.S. government? Is a little extra cash in your pocket and a little more job security worth your selling your soul to the devil?

<comment><USER13> God damn. Not all defense contractors work on the NSA spying technologies. Personally, I think it'd be cool to work on anti-missile guidance systems or fighter jet avionics.

<comment><USER12> He specifically pointed out Booze Allen which is known for their NSA spying. It's not like you can walk into the organization and say, "Hey, I'd like a job. You guys don't spy on Americans, right?".

<comment><USER13> Not the comment I replied to. He just lumped all defense contractors into soulless assholes basically. And that's just intellectually dishonest.

</comment></comment></comment></comment></comment><comment><USER14> And we learn the importance of commas

Defense subcontractor posted a job listing for XKEYSCORE, 2 weeks ago

Defense subcontractor posted a job listing for XKEYSCORE 2, weeks ago

</comment><comment><USER15> Is it legal to deny people employment based on their having worked on projects like this?

<comment><USER13> Meet Bob. Bob got a job working on a random system through an NSA subcontractor. After several weeks on the job, and having gone through rigorous training, Bob was starting to feel uneasy with the work requirements and amount of details he knew. Bob had a TopSecret clearance however, and couldn't tell anyone - not even his wife. After a few more weeks, he decided to try his luck at another employer. Bob was damn good too, one of the best in his field - he was guaranteed a job anywhere for competitive pay.

Except word got out about Bob's employer - and specifically the subsystems Bob was responsible for implementing. After the negative press, the NSA didn't renew Bob's company's contract which forced said company to let Bob go. That's okay, Bob thought, he had enough experience in the field, right? Everywhere Bob looked was disgusted at his previous job and the moral choices his superiors' superiors made. But they took it out on Bob. Bob was never offered a job anywhere for close to the amount of money and psychological income he had at his former job.

Must suck to be Bob.

<comment><USER16> >After several weeks on the job...Bob had a TopSecret clearance.

Top Secret clearance usually takes a few months. I'm comfortable with a world in which someone who worked for an NSA subcontractor for several months being stuck with menial employment for the rest of their days. Sometimes it pays to have ethics.

<comment><USER17> My brother got a TS. It took him about a year to get the thing. He had no clue what the TS part of the company was like before 'stepping through the rabbit- hole.' He worked at the job just fine for a few years. One day, he said he knew he had to look for other work. It took him a few months to find a way to leave. Eventually, he just quit without any job at all. He has a job now, not nearly as glamorous but he is employed.

Just because they work for a contractor for the NSA, or they have a TS or S or Q clearance, doesn't mean they are not very smart and hard working. Hell, you know they went through the ringer to prove they are trustworthy and loyal. Because they wanted to help out all of us here in the US and then they felt they could no longer, that does NOT mean they should be punished. What did they themselves do wrong?

Besides, this is ONE program the NSA runs. Mostly likely the majority of the TS cleared people out there have nothing to do with all this stuff. Those people do make a lot of sacrifices for their job and for the US. A lot are multiply divorced because of the stress, the secrecy, and the unpredictability of the job.

"Hey Sugar, gotta go. I can't tell you where, for how long, or if I will be safe, or talk about a really traumatizing experience afterwards so I'll drink hard liquor quietly on the back porch a lot. Pick the kids up at basketball, will yah?"

<comment><USER18> > "Hey Sugar, gotta go. I can't tell you where, for how long, or if I will be > safe, or talk about a really traumatizing experience afterwards so I'll > drink hard liquor quietly on the back porch a lot. Pick the kids up at > basketball, will yah?"

In addition to the potential ethical issues, this is the second reason why I made the decision not to get involved with classified work (the majority of IT jobs in my hometown are at defence contractors). I simply couldn't handle _not_ being able to talk to my family and friends about my work - <USER2> if there were things that were causing me stress on the job.

</comment></comment></comment></comment></comment><comment><USER19> Am I the only thinking that working on a project like this would be awesome ? From an engineering point of view, they offer an incredible technical challenge.

<comment><USER20> There's a whole world of problems like that though, that don't involve questionable ethical practices (depending on your own personal beliefs).

Math is the language of reality, it's applicable to anything so I don't know why getting involved with these sorts of projects over just about anything else is so alluring from a technical standpoint.

</comment></comment><comment><USER21> Previous discussion: [https://news.ycombinator.com/item?id=6138205](https://news.ycombinator.com/item?id=6138205)

</comment><comment><USER22> When I saw the ads in my state for Booz-Allen-Hamilton jobs requiring clearance a few months ago I knew that it was spook work, but thought nothing of it at the time... (I refuse to relocate, it's telecommute or I go freelance) I wonder if Ed Snowden responded to those same advertisements?

</comment><comment><USER23> A script combing linkedin for skill solicitations that regex match ALLCAPS could probably uncover all NSA job postings. For supposed mastery of encryption/deception these fucks sure seem hamfisted in looking for employees.

</comment><comment><USER24> XKEYSCORE is just a Java front-end GUI, not servers/backend databases.

</comment><comment><USER25> That's one hell of an opportunity!

Anybody with a conscience in here? Please apply and report back!

Let's penetrate the Borg.

<comment><USER26> I'd apply just for an opportunity to be another leaker, but there's no f%!#ng way in Hades that the feds would ever hire me. I've been WAY too outspoken in my radical anti-government / libertarian / anarcho-capitalist viewpoints and have said way too many things that would - I'm pretty sure - <USER3> disqualify me.

Them: "Have you ever advocated for the overthrow of the US government?"

Me: "Well... ah, I mean, errm... aah... define 'overthrow', please?"

Them: "GTFO out here."

<comment><USER17> My brother has a TS. He said they mostly hire Mormons. They have a clean living background and can speak the languages due to their missions. Also he said they asked a lot about being a member of the Communist Party.

<comment><USER27> Really? The Community Party? Seems somewhat quaint. I would assume now they ask you stuff like, "What do you think about the war in Afghanistan?" "Have you ever traveled to the PRC?" etc.

</comment></comment></comment></comment><comment><USER28> SAIC is also running domestic surveillance, currently having a facility in Oakland built[1]. The facility is currently being funded by the DHS, being built while no privacy controls or data retention policies are known.

This is an ongoing trend of the names most know as being involved in national level surveillance actively involved on the state and local level too. Another case is Booz Allen Hamilton's processing of digital forensics for local law enforcement via federal funds[2].

Yea, no chance for conflict of interests when firms granted immunity, having access to the world's data, are involved in 'petty' things like domestic surveillance/forensics to assist in prosecutions of the wire-tapped non- immunes.

This is an absolutely horrific trend that must be halted, as in halting literal construction.

[1] [http://oaklandwiki.org/Domain_Awareness_Center](http://oaklandwiki.org/Domain_Awareness_Center) [2] [https://news.ycombinator.com/item?id=6014168](https://news.ycombinator.com/item?id=6014168)</comment>


Mercurial server in Rust, specifically designed to support large monorepos - <AUTHOR> https://github.com/facebookexperimental/mononoke <comment><USER2> Interesting to see it's GPL, unlike most other Facebook and Rust code (BSD or Apache).

Is Mononoke meant to be reminiscent of Monotone?

<comment><USER3> Maybe because Mercurial is GPL? I saw that too, and wondered as you did.

[https://www.mercurial- scm.org/repo/hg/file/tip/COPYING](https://www.mercurial- scm.org/repo/hg/file/tip/COPYING)</comment></comment>


Linux: Limit IPv6 connectivity to specific programs - <AUTHOR> http://fds-team.de/cms/articles/2013-06/linux-limit-ipv6-connectivity-to-specific-programs.html <comment><USER2> > "The main problem is to secure an IPv6 network which is much more > complicated than using a typical IPv4 network consisting of a router and > several devices behind it."

Does anyone know why this is the case? I'm not a network security expert but to me I don't see how IPv4/v6 makes a different in terms of security. I'd assume that each computer on the network could most likely be assigned a public IPv6 address rather than using NAT in which case how is configuring your perimeter firewall to drop incoming connections by default any different from not having any port forwarding setup by default? Even your average domestic router has some sort of basic firewall built in.

<comment><USER3> There is nothing special about firewalling off IPv6. NAT is not a security feature. The problem is that most consumer "routers" that people use nowadays are really: a router, a switch, a wireless access point, a firewall, and who knows what else. Here are some sample rules for firewalling off IPv4 (typed from memory, so use with caution):

      iptables -A INPUT -i lo -j ACCEPT
      iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
      iptables -A INPUT -s 192.168.1.0/24 -j ACCEPT
      iptables -A INPUT -p icmp -j ACCEPT
      iptables -A INPUT -j DROP

      iptables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
      iptables -A FORWARD -s 192.168.1.0/24 -j ACCEPT
      iptables -A FORWARD -p icmp -j ACCEPT
      iptables -A FORWARD -j DROP

Here are the IPv6 rules:

      ip6tables -A INPUT -i lo -j ACCEPT
      ip6tables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
      ip6tables -A INPUT -s 2001:xx:xx:xx::/64 -j ACCEPT
      ip6tables -A INPUT -p ipv6-icmp -j ACCEPT
      ip6tables -A INPUT -j DROP

      ip6tables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
      ip6tables -A FORWARD -s 2001:xx:xx:xx::/64 -j ACCEPT
      ip6tables -A FORWARD -p ipv6-icmp -j ACCEPT
      ip6tables -A FORWARD -j DROP

Does that look like it would be hard to do? Your router should come with these rules already. If it does not, ditch it and buy one that is supported by OpenWRT, where IPv6 support is not a second class citizen.

Edit: Naturally, IPv4 rules would have to be more complicated since you'll want to have your NAT setup in there. In this way, configuring IPv6 is actually easier :). Also, a real router would have rules set up for throttling certain types of traffic (e.g.: you don't want more than, say, 1000 ICMP messages per second). However, all those steps are identical for IPv6.

<comment><USER4> > NAT is not a security feature.

Why?

<comment><USER5> Because it was never intended as such and does not necessarily need to add any. The fact that many NAT implementations do add some security (by dropping inbound connections by default) is a side effect. I've seen NAT implementations that get it precisely wrong (consumer routers that set up .2 as the default DMZ), but that's still entirely valid.

<comment><USER4> Whether or not NAT was designed with security in mind doesn't matter.

Using NAT increases security simply by having deny by default.

<comment><USER6> But that is not a feature of NAT, that is a feature of a firewall (for example, it is possible to route incoming packets via the WAN as well as masquerade outgoing ones from the LAN - most people wouldn't even know their pants are down). It is a coincidence that home routers sometimes provide both leading people to conflate their firewall with their NAT system - but if a firewall is what is wanted (and is arguably the only valuable component), NAT is not the thing to ask for. Conflating NAT with firewalls also promotes the idea that NAT has a place in any network with abundant addresses. IMHO, it does not.

<comment><USER7> NAT will always have a place, because not everyone wants to expose a uniquely routable address for every device they own (probably based on a device's MAC address) to the world.

<comment><USER6> Can you give an example that isn't based on security (since the NAT isn't providing that)? I can only think of short-term (undesirable) transition or load-balancing mechanisms - all of which have better long-term solutions (anycast, multicast, and dynamic service discovery, etc)...

<comment><USER7> What's with the widespread fixation against attributing any positive security attributes to Network Address Translation? It's like the old warning about NAT not being enough to protect a private network got cargo culted into a universal anti-NAT maxim.

What would you call it when a router at the edge of a private network presents a single IP to the world no matter how many devices are behind it, for privacy reasons?

<comment><USER6> _" What's with the widespread fixation against attributing any positive security attributes to Network Address Translation"_

It probably has something to do with NAT being a huge obstacle later, when needs change. For example, while it might make sense at one point to have multiple hosts appear as one, it is at the expense of direct addressability, and the workaround - having a unique address+port combination - makes less sense than having a unique directly-routable address.

 _" What would you call it when a router at the edge of a private network presents a single IP to the world no matter how many devices are behind it, for privacy reasons?"_

The wrong tool for the job. :)

The privacy is gained not by translation but by blocking direct connections (which is a feature of a firewall - not of NAT). While there is some value in NAT's ability to falsify the origin of data - ie, to take credit for non-local flows - but if those applications ever grow they will be fighting to escape the single address of NAT and are thus only suitable in the short term. Ideally, NAT would go unused because every element is uniquely addressable and fully independent... Most people would not consider someone else who continually takes credit for their or someone else's work to be a feature, and so it is with NAT.

<comment><USER7> I'm thinking more in terms of consumer networks. I really don't want my ISP to know exactly which devices I'm using or how many, nor do I want to have every web site be able to track them by globally routable IPv6 address everywhere I go. Simply blocking incoming connections doesn't solve that.

Taking credit for someone else's work is not a useful analogy for NAT, nor are the corresponding moral implications relevant.

<comment><USER6> _" nor do I want to have every web site be able to track them by globally routable IPv6 address everywhere I go."_

So, you would rather use one address for everything, making it easy as pie to track you? You can pretty much pick IPv6 addresses at random (under your router prefix), and you have (many many many) more addresses than the whole IPv4 address space to choose from (it's a 128 bit address space and providers typically give a /48, /56, or /64 prefix at worst... that's 128-64=64 bits... that's 2^63.9999... more addresses than IPv4). In short, you don't really "map" the IPv6 space the same way you do the IPv4 space.

Taking credit for someone else's work very much is a useful analogy: if I can only speak through a third party, and I need that third party's permission to speak, let alone be spoken to, I quickly resent him or her. This is quite the case with current NAT solutions, with system administrators restricting "their" networks, making communication difficult for everyone else (by holding the only globally-routable address or "allocating" only a few; everyone else _is_ second-class).

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> This seems to take the long way around. I wonder why they didn't consider just using the firewall to control network traffic; unless you really need your applications to be completely unaware of IPv6, but so far, I've not experienced that problem.

Also, requiring root privileges for launch is a bit of a burden in some use- case scenarios.

<comment><USER9> Not exactly a common use case or linux related but my Red Alert 2 game patched for tcp/ip instead of IPX support it shipped with will only let you into LAN game lobbies on ipv6, you can't see any games to join. ipv4 and it works like a charm.

My point is just that I'm sure some very very weird stuff can happen with software.

</comment></comment><comment><USER10> Security aside, this is even useful for new deployments of IPv6. It seems a lot of people have IPv6 networks that are less favorable for some traffic than their IPv4 networks. Some are running their IPv6 through tunnels. As soon as you enable IPv6 on your desktop, suddenly Firefox or Chromium will prefer IPv6 for any website with a AAAA record, which adds a ton of latency and reduces bandwidth.

But theoretically, I could enable IPv6 for sshd (where I stand the most benefit) and leave it off for wget and browsers with this.

<comment><USER11> At home my HE tunnel is snappier than the plain ipv4 from rr. The weird thing is it is not just a case of user perception (mine). One of the upstream s1 ntp boxes I sync with has ipv4 and ipv6 and is relatively close. I set one of my s1s to sync with both the ipv4 and ipv6 addresses of the machine and the ipv4 clock displayed a lower delay and less jitter while the offsets were seasonably close.

<comment><USER11> argh, for posterity's sake the last line should read:

"the _ipv6 peer entry_ displayed a lower delay and less jitter while the offsets were reasonably close."

</comment></comment></comment><comment><USER12> On this subject, sort of, if I want to dump the commercial routers and replace with eithe a Linux install on a commercial router or a x86 box that is not too chunky / loud Where do I go for readme and instructions and community?

I can find some old Linux on linksys sites but not a lot recently

<comment><USER13> If you with the commercial router route, checkout OpenWRT / DD-WRT / Tomato. There are several forks, so depending on what features you want, and what hardware you have, you may opt for a different version. If you go with an existing x86/amd64, just install your favorite distro and start reading up on iptables configuration and management. While iptables to be quite powerful in terms of features, I find its' syntax is painful. There are several projets aiming to simplify this, by generating rules from another, simpler, DSL but it adds complexity. This is why I prefer PF, but that requires you to install OpenBSD instead of Linux. If you know your way around UNIX, and are ready to read up the FAQ and man pages, this shouldn't be a problem :).

</comment></comment><comment><USER14> This was interesting but got me thinking about a problem I encountered at work. Is it possible to create two bridges out of one namespace so that two separate programs can receive the same network traffic?</comment>


Friend-Watch - Analytics for your Facebook account - <AUTHOR> http://techcrunch.com/2011/08/23/new-friend-watch-app-lets-you-stalk-your-facebook-friends/

<comment><USER2> <plug>Got nothin' on wisdom.com.</plug></comment>


PA Supreme Court Rules Police Can't Force You to Tell Them Your Password - <AUTHOR> https://www.eff.org/deeplinks/2019/11/victory-pennsylvania-supreme-court-rules-police-cant-force-you-tell-them-your <comment><USER2> This is really interesting, and likely to end up at the federal level. It is also a large argument _against_ biometrics: you can replicate someone's fingerprints, but reading a password out of a brain is much harder.</comment>


Digg Relaunched; Is Almost A Pinterest Copy - <AUTHOR> http://digitizor.com/2012/08/01/digg-relaunched-is-almost-a-pinterest-copy/

<comment><USER2> Still not as much of a pinterest copy as sex.com

<comment><AUTHOR> Talking about Pinterest copies? Visit <http://www.pinspire.com/> !

</comment></comment><comment><USER3> Jesus,

Let it die.

Once again - what a terrible UI. I even posted to them recently: "...dont let too much UI get wrapped around the content.."

Ths is a puzzle page of content boxes, too much visual distraction; do I look at the pic, title, meta text etc. The grid doesnt align - no categories etc...

WTF.

Single long scroll between the elements?

WHO THE HELL APPROVED THIS UI

The lines between each section are too thin - the lack of contrast...

There are so many elements wrong with this.

This is a poor attempt at a quick flip of 500K on a house.

They bought this domain and had a super fast flip on the domain to be able to push this out while the word still had hype about the sale, and its an utter BS disgrace. Lets see if there is a deeper UX to be discovered that really works well for people, but purely based on initial launch ==failure.

DIGG IS DEAD!</comment>


Converting your website to static - <AUTHOR> http://jacek.migdal.pl/2014/12/07/all-things-static.html

<comment><USER2> Author here ;), happy to answer questions.

FYI: you can also use GitHub Pages instead of S3.</comment>


"Aaron's Law" Might Be Good - <AUTHOR> http://reason.com/blog/2013/01/16/congresswoman-names-law-after-dead-guy-b <comment><USER2> I think the musings of Philip Greenspun at [http://blogs.law.harvard.edu/philg/2013/01/16/aaron- swartz-f...](http://blogs.law.harvard.edu/philg/2013/01/16/aaron-swartz- followup-can-we-get-government-out-of-the-copyright-enforcement-business/) that we need to make copyright violation a purely civil matter would be a proper step.

<comment><USER3> Wouldn't that mean people without assets could do whatever they liked with copyrighted materials?

<comment><USER4> One thing they couldn't do is profit from the materials -- as they'd then have seizable assets.

But, yeah, a bunch of broke-ass teenagers sharing music amonst themselves? I'm down with that.

<comment><USER5> What about profiting at a lower price?

Playing devil's advocate here: I agree with "teens sharing music", but what if I sold them for, say, half iTunes price, and made $500. I would deprive an artist while profiting for enough money, and if I spend it, the money is gone. Seems broken.

<comment><USER6> Why would someone buy them from you at half price, when they can illegally download them for free?

</comment></comment></comment></comment></comment><comment><USER7> The eminently qualified Jennifer Granick calls it "a great first step" [https://cyberlaw.stanford.edu/blog/2013/01/thoughts-zoe- lofg...](https://cyberlaw.stanford.edu/blog/2013/01/thoughts-zoe-lofgrens- cfaa-bill-great-first-step)

See also her writings over the last few days <https://cyberlaw.stanford.edu/about/people/jennifer-granick>

<comment><USER8> Lawrence Lessig calls it "a critically important change that would do incredible good": [http://www.reddit.com/r/technology/comments/16njr9/im_rep_zo...](http://www.reddit.com/r/technology/comments/16njr9/im_rep_zoe_lofgren_im_introducing_aarons_law_to/c7xmx6j)

</comment></comment><comment> miw-sec-work A better AArons law would be to force publicly funded research be released freely and under a Creative Commons license. They did this in EU, why not the US?

Weakening the Computer Fraud laws, and naming it after Aaron trivializes his quest for free flow of information.

Of course this wont happen -- too much money tied up in selling research.

<comment><USER9> I have the same idea about open source software in governments. If it uses taxpayers money, then it should be open source, wherever possible (so in most cases). Taxpayers' money shouldn't be used to fatten corporations, if it can be avoided. And by that I don't necessarily mean that government software should be written by volunteers, but if they're paying them to write software, then that software should be made open source later. And they can keep paying companies for maintenance and upgrading it if they want. But at least the software would be available to anyone to use and tinker with it.

<comment><USER10> I asked before and was told that code written by the government is available by request due to the FOIA, no state secrets mind you, but it might be true of (a significant portion of) code written by contractors as well.

<comment><USER11> Does that really matter? The NSA faced scrutiny for writing its own software when commercial solutions were on the market. The US government contracts out almost all its software needs below the state secrets level anyway.

</comment></comment></comment></comment><comment><USER12> law professor orin kerr on aaron's law:

> A lot of people have wondered how to amend the computer crime laws in > response to the Swartz tragedy. So far I have seen a lot of interest in > this, but not a lot of sensible proposals. Already, Rep. Lofgren stepped > forward with "Aaron's Law," , text here, which would amend the statutory > definition of "exceeds authorized access." This isn't new text: It's just > the definition of "exceeds authorized access" that was passed by the Senate > Judiciary Committee last year to try to stop Lori Drew-like prosecutions. > This amendment is well meaning, no doubt, but I think it is a bad idea for > two reasons. First, it is weirdly disconnected from the Swartz case. Swartz > would still have faced exactly the same criminal liability under "Aaron's > Law" that he did without it.

the rest here (in section IV): [http://www.volokh.com/2013/01/16/the-criminal- charges-agains...](http://www.volokh.com/2013/01/16/the-criminal-charges- against-aaron-swartz-part-2-prosecutorial-discretion/)

</comment><comment><USER13> As an attorney I would like to give a quick two cents about the idea it is illegal, or volatile of the Computer Fraud and Abuse Act, to simply violate the terms of a websites terms and conditions or privacy policy. For example, if anyone remembers the Mom who created a fake MySpace profile spoofing her daughters friend, who ultimately committed suicide. See <http://en.wikipedia.org/wiki/Suicide_of_Megan_Meier> . If this women was acquitted who clearly violated MySpaces terms and conditions which possibly contributed to the suicide of a minor, then this charge does not have much bite to it.

Nevertheless a law such as this would prevent prosecutors with ice in their veins from charging who ever the hell they want, because lets face it we all somehow or someway are violating a terms of service. As a lawyer I can say with comfort that the law is catching up to technology, but generally an existing law is expanded into technology, what is crazy here is that there is no equivalent pre-existing law to support this kind of criminal charge. In other words, imagine you violating any non-tech terms of service and it being a federal crime, for example it would be crazy to imagine paying your credit card late or returning a movie late (as if Blockbuster still exists) puts you in violation of criminal law. The closest example I can think of is lying on a mortgage application which constitutes fraud under federal law but that is obviously distinguishable, and at least a mortgage application will tell you lying is fraud I have never seen such a disclaimer on any websites terms of service (but who reads those anyway?).

</comment><comment><USER14> Misdemeanor rather than felony would be a good direction.

</comment><comment> miw-sec-work Also, "Aarons law" was raised 1 year ago by the same Congressperson under a different name.

This is nothing but personal political quests using our martyr for justification. I'm insulted.

<comment><USER15> If she hadn't, would the complaint be, "why didn't anyone propose this change early enough to actually help?"

I suppose one could focus on retribution for this particular event, or recognize that the CFAA allows for many other unknown people to spend lengthy jail sentences with few people caring about them.

If the legislation is flawed, it should be changed, so that even people without celebrity are protected. Prosecutorial discretion has its place, but the idea of prosecutors selectively applying laws that are on the books with considerations about how popular suspects are in a poor model for a legal system.</comment></comment>


AOC Quit Facebook. The Media Bungled the Story - <AUTHOR> http://www.calnewport.com/blog/2019/04/17/aoc-quit-facebook-the-media-bungled-the-story/ <comment><USER2> I think the site is getting hugged to death. The quotes explaining the "bungled" bit is:

"Here's the thing: misuse of personal data and hateful content were not the reasons emphasized by AOC for why she quit Facebook. She instead called social media a "public health risk" that too often leads to "increased isolation, depression, anxiety, addiction, escapism.""

and

"In other words, the important story is not the fear that social media companies will improperly use our data; it's instead the fear that they'll subvert our primal drive to cultivate a meaningful life."

<comment><USER3> A PageDash archive: [https://app.pagedash.com/p/3bee2da8-7238-4681-9a5b-aeb180686...](https://app.pagedash.com/p/3bee2da8-7238-4681-9a5b-aeb180686bec/oq5FR5oV4sz391tawHIP)

</comment></comment><comment><USER4> I haven't logged into Facebook in almost two years. I can't imagine much has changed since my last visit. I do enjoy using Messenger for its convenience, but Facebook as a social platform is long dead for me.

<comment><USER5> Never used FB with the exception of creating a dummy account in high school to host a project page. Graduating university in a month and I may have to bite the bullet on it as my internship uses FB to connect the interns together. So far, I have enough distractions with Reddit, Twitter, and Hacker News (and various other forums) that I never felt like I was missing out on the FB punch bowl.

</comment></comment><comment><USER6> Does she think the same considerations apply for Twitter? (Serious question.)

<comment><USER7> Not to mention the irony of still being on Instagram. Guess being young doesn't really make you that technologically aware.

<comment><USER8> The parent was asking a serious question, as indicated - not mocking AOC's decision as ironic. And the question is reasonable. These are all different services. How do those differences affect how damaging they are, and in what ways? Was AOC engaged with them differently, or less than, Facebook? Does she plan to not leave those services, now or in the future? Etc, etc.

</comment></comment></comment><comment><USER9> mirror - [https://news.yahoo.com/aoc-skullduggery-social-media- faceboo...](https://news.yahoo.com/aoc-skullduggery-social-media- facebook-120249217.html)

this is in regards to Rep. Alexandria Ocasio-Cortez, not AoC Gaming or any other AOC acronym.

<comment><USER10> I'll admit, my first thought was Advent of Code, the delightful annual programming puzzle competition that is often mentioned here closer to December.

</comment></comment><comment><USER11> I don't think this is much of a story whether it's bungled or not.

Cal Newport wrote a book about quitting social media (I'm in the middle of the audio version of Digital Minimalism).

He is leveraging AOC's popularity for his own crusade. Which is fine, but considering AOC still uses social media extensively I don't think she's a good example for him to hold up.

</comment><comment><USER12> This article only references one news article published by the Washington Post[1]. It's not even completely clear to me after reading it that the writer was trying to do what the author of this post was saying.

To say the entire media has bungled the story is disingenuous. Go look at other articles that don't mention CA or any other political scandal[2]. This is a real non-story.

[1] [https://outline.com/rBfeXn](https://outline.com/rBfeXn)

[2] [https://abcnews.go.com/Politics/rep-alexandria-ocasio- cortez...](https://abcnews.go.com/Politics/rep-alexandria-ocasio-cortez- stopped-facebook-due-public/story?id=62412463)

</comment><comment><USER13> Her insta (a facebook property) is delightful though.

</comment><comment><USER14> Mirror? Not loading.

<comment><USER15> archive.is managed to get a snapshot [http://archive.is/YHpsh](http://archive.is/YHpsh)

</comment></comment><comment><USER16> I'm glad this article touched on 2 good reasons why people are quitting social media nowadays: data privacy concerns and public health risks.

But I think there's a third, equally important reason why people should consider quitting these platforms: public image management.

I find that a majority of people who use social media accounts simply do not know how to properly maintain a clean public image. Whether it be because they don't understand or refuse to tighten their privacy controls, or because other people take everything they say out of context for the sake of tarnishing their reputation, the things you post to social media can always be used against you. And for all intents and purposes, you can never take it back once you post.

For politicians, this rings especially true, and as far as I know for U.S. politicians, their social media accounts are 100% their thoughts, unsanitized or evaluated by a management team.

I suppose the old addage remains true: if you can't take the heat, stay out of the kitchen.</comment>


Open source production-ready app store enabled base stations in 2017 - <AUTHOR> https://www.linkedin.com/pulse/open-source-production-ready-app-store-enabled-base-stations-ectors <comment><USER2> Pretty amazing, looking forward to seeing more!</comment>


YouTube page load is 5x slower in Firefox and Edge than in Chrome - <AUTHOR> https://old.reddit.com/r/Windows10/comments/91jil3/youtube_page_load_is_5x_slower_in_firefox_and/ <comment><USER2> If true, then it is yet another example of anti-competitive behavior from Google. Reminds me of Firefox support in Hangouts.

<comment><USER3> The worst I've seen is that Google serves an ancient version of their homepage to Firefox mobile for "reasons". With an extension to spoof useragent to Chrome, you get the regular page and it works fine

<comment><USER4> Used to be the case with Google Maps as well, Chrome user agent would get you served with a faster version.

The worst I've seen though isn't by Google themselves. It's user-agent whitelisting on other web sites, that think it's okay to prevent users from using the browser they want because they couldn't spend time testing in the other browsers. Sometimes they even exclude Chromium. For that reason I have an user agent spoofing extension on each of my browsers.

<comment><USER3> IMO, Google put a new stage on embrace extend extinguish I call "neglect". Make everything open source and nice, then continue to update G specific functionality by simply not maintaining good support for anything else.

Google probably has the best engineers and greatest minds since IBM in the 60's, possibly ever. But great engineers are attracted to shiny objects, and if all of those are Google objects there's suddenly no reason to use anything else

</comment></comment></comment></comment><comment><USER5> It I remember when chrome only appeared, they deployed try-chrome banners, specifically seen on /. that froze both opera and firefox for a couple of seconds, but worked fine in chrome itself. No one believed me on local forums back then.

<comment><USER6> I think you messed up your example url.

<comment><USER7> I assume he meant [https://slashdot.org/](https://slashdot.org/)

</comment></comment></comment><comment><USER8> So apparently a misguided dependency on an obsolete version of a web framework that uses a dodgy polyfill on some browsers is now proof of a dark conspiracy since browsers won't have the same performance.

This a good discovery and certainly an unfortunate result, but do people not know how web development works? Mistakes like this do happen (though usually with less severe consequences).

<comment><USER9> HN hivemind: the only cure for a Google bug is... more regulation!

</comment></comment><comment><USER10> I'm not super quick to think they're doing something malicious here. I've experienced this exact thing at two of my jobs this decade, one of which I currently work at. Firefox and especially Edge are often just slower for some DOM manipulation use cases and style recalculations.

Although Pendo has been identified as the primary cause for Edge slowness :P

<comment><USER11> No, the tweet points out how this is specifically because Google uses a deprecated, older version of a standard that only Chrome supports. It's absolutely not a question of Firefox or Edge being "just slower," it is a question of using a feature that is literally only implemented in Chrome and that other browsers have agreed not to implement.

<comment><USER12> And the current version of the standard also isn't implemented in Firefox yet, so using that would change nothing?

<comment><USER13> It is implemented behind a preferred ATM, IIRC

</comment></comment></comment></comment><comment><USER14> The tweet that it refers to: [https://twitter.com/cpeterso/status/1021626510296285185](https://twitter.com/cpeterso/status/1021626510296285185)

</comment><comment><USER15> It's an interesting topic - but, the linked Reddit thread isn't all that interesting itself. It's most just a collection of unsupported conspiracy theories.

</comment><comment><USER16> The title implies two things:

\- Google is pushing Shadow DOM before other major browsers agreed to standardize the API

\- The page load in Firefox/Edge is worse than it would be without Shadow DOM

If either or both of these are true then it's pretty shitty on Google's part. If neither are true then it's just a load of FUD. Is anyone more familiar with the situation able to speak to these questions?

<comment><USER17> For the first one, this is pretty well known : Google "shadow dom v0"

v0 is the old stuff chrome shipped before there was consensus, and it's pretty different from v1, which is what the spec settled on. To be clear, "is pushing" isn't exactly correct, it's "pushed". I _think_ everyone is now on the same page wrt v1?

(Disclaimer: I do work at Mozilla but I've not been involved in this and this is just what I've read)

I'm surprised YouTube is still using v0 since v0 was deprecated a while back.

<comment><USER18> But v1 is not being implemented yet in Edge or Firefox? [1]

And then you get these kind of "crap" from twitter [2]

It seems we have been talking about Shadow Dom for years, grid, flexbox, web development still isn't anywhere near good enough.

[1] [https://caniuse.com/#feat=shadowdomv1](https://caniuse.com/#feat=shadowdomv1) [2] [https://twitter.com/Shihrer/status/1021891405633925120](https://twitter.com/Shihrer/status/1021891405633925120)

<comment><USER17> It's implemented, it's preffed off. These things take time to roll out properly. Chrome jumped the gun on v0, Firefox/Edge aren't "dragging their feet".

</comment></comment></comment></comment><comment><USER19> And now I know why Youtube always takes several seconds to begin working after page load. Usually it would sit there, showing me a thumbnail or even daring to play an add while basically every single UI element was a grey blob slowly pulsing while the javascript in the background was busy shitting data into a polyfill. And with that addon? Basically instant page load.

Why in all that is holy did anyone think this was a good idea and why did noone bother to test this on Firefox or if they did why did the engage in such shitty behaviour?

<comment><USER20> and yet, the saddest thing of them all, is that one can just... install Chrome!

it's not like the dark ages of IE6, where having to resort to use IE6 was also forcing to use Windows as well and forcing to use a dated, closed, non standards compliant browser.

Chrome is free, available essentially everywhere, standards compliant, fast and with cutting edge web features.

And that's a pity, that's what makes it just too easy to create a monopoly.

<comment><USER19> Chrome _IS_ the new IE6. Just consider the number of times that Chrome breaks features because Google thinks webdevs can't think. And Websites breaking because I'm not using Chrome is exactly why IE6 was shit. Websites worked in IE6 but not anything else. THAT was the true tragedy.

</comment></comment></comment><comment><USER21> I confirm that here too with debian and firefox 52.9. YT is damn slow and HookTube is faster even with the embedded YT player.

</comment><comment><USER22> I would hope that Googlers can make their sites work fastest on the browser that they own and control the internals of.

If I were building a browser, I'd be seriously tempted to do a lot of things for speed that don't seem to be on the table right now. For instance, I'd aggressively bundle common JS or CSS dependencies and short-circuit loading them over the wire. E.g. a site wants jquery 1.x, or Bootstrap 3? I can load that faster from even spinning rust than over fiber, and a few MB of JS or CSS is nothing to store.

<comment><USER23> > If I were building a browser, I'd be seriously tempted to do a lot of things > for speed that don't seem to be on the table right now.

Yes that's all fine, but we're talking about YouTube here as the offender not Chrome. It's fine to make your browser work faster, that's not the same as making your website work slower on other browsers.

> I would hope that Googlers can make their sites work fastest on the browser > that they own and control the internals of.

I would hope they'd do that by just optimizing for Chrome, not by deciding to base the whole website on an API that's deprecated and has been superseded and is ONLY still supported by Chrome.

</comment></comment><comment><USER24> Time for antitrust to break Google up. At the very least force the divestiture of both Chrome and YouTube. And obviously given the case in Europe, Android as well.

</comment><comment><USER25> Sure, on Firefox adblocking works pretty much solid, removing all kind of ads thanks to uBlock origin.

On Chrome, I get sometimes still ads across videos, plus Chrome is Google's Browser: obviously they test YouTube extensively with it and they just make it "work" for other vendors.

<comment><USER26> uBlock Origin shares the same code in Firefox and Chrome. Any differences you see have nothing to do with the browser.

<comment><USER27> They do, because there is an exploit for chrome that makes 3rd party look like first party. [https://github.com/gorhill/uBO- Extra](https://github.com/gorhill/uBO-Extra)

You will only get this behaviour on Chrome, Firefox is not vulnerable.</comment></comment></comment>