Pharo: An immersive programming experience - <AUTHOR> https://pharo.org <comment><USER2> I've found a good way to approach Smalltalk systems like Pharo and Squeak is to remember they're _whole systems_ , not just languages with a bolted-on IDE.

I like to think of each running instance as more akin to a Unix virtual machine than anything else. You end up using "in-image" version control systems like Monticello just the same way on a Unix server you use "in- operating-system" version control systems like git.

<comment> em-bee the only thing i miss is a terminal. some time ago i actually found one, but it needs porting to pharo.

greetings, eMBee.

<comment><USER3> Here's a new Terminal emulator that appeared this month - [https://blog.lxsang.me/post/id/24](https://blog.lxsang.me/post/id/24)

<comment> em-bee ohhh, you are my new best friend!

i love the serendipity of this topic coming up just days after that post you found when i have not looked into it for a few years after trying for a few days to get that original squeak terminal to run in pharo.

greetings, eMBee.

</comment></comment></comment></comment><comment><USER4> There is a Pharo MOOC currently running:

[http://mooc.pharo.org/](http://mooc.pharo.org/)

[https://www.fun- mooc.fr/courses/course-v1:inria+41010+sessio...](https://www.fun- mooc.fr/courses/course-v1:inria+41010+session03/about?platform=hootsuite)

The original course is in French (dubbed in English) with English, Spanish and Japanese subtitles.

<comment><USER5> There's also a website with a public archive iirc

</comment></comment><comment><USER6> Good to see this make the front page of HN. I only spend a few hours a month using Pharo (some working on my open source NLP project, but mostly just trying out new packages and demos released for Pharo).

Other comments here are right: think of Smalltalk as being the complete computer. When I use Pharo I usually put it in full screen mode and don't think about the macOS or Linux host.

<comment><USER7> It's not that Pharo discussions are rare on HN...

[https://news.ycombinator.com/from?site=pharo.org](https://news.ycombinator.com/from?site=pharo.org)

<comment><USER8> I've been visiting HN for quite a few years now, but somehow this is my first encounter of a topic on Pharo. The language seems interesting to me, so I am glad it was posted today.

</comment></comment></comment><comment><USER9> Squeak is the most delightful IDE I've worked in. I liked the idea that the IDE, including the interactive debugger, was available to the user to program. I hope Pharo gains more traction and becomes popular at least in teaching environments.

<comment><USER10> >. I liked the idea that the IDE, including the interactive debugger, was available to the user to program.

a similar concept exists in F# type providers. it doesn't always work out well in practice but there is a ton of promise there.

<comment><USER11> In what instances/how does it not work out well in practice?

I've only read the hype, and it sure reads well... :-)

<comment><USER12> I have a dream that some day we have statically typed language which would have first-class types with some compile time meta programming.

For given example data, type providers output just something and typically it's 95% what you want. But how to fix that 5%? You cannot just say that "Type.Property, could you please be an integer". Instead of you have to tweak example which in some case is just fine but on some cases it's not possible at all.

Example: you use these SQL libraries which are based on type providers. You have some clear schema and it's just works. Then some day you would like to use runtime defined columns but at that point you have to opt out from that type provider library (at least for queries related to that table).

I still love working with F# but I don't use much of type providers. I think they are like proof of concept that compile time meta-programming can be provide real value but it need still more freedom.

Edit: I use SQL library called Rezoom and while it has it's limits it has given great benefit during development. When you change DB schema every effected query and model transformation gives compile time error. Value there is real.

</comment></comment></comment></comment><comment><USER13> For anybody who's used an "immersive" programming environment, like Smalltalk to a Lisp machine -- what's the advantage? It seems like the ability to mutate the whole world would just be asking for mistakes. I can barely keep a clean Python REPL. Do you restart your whole machine to run your test suite?

<comment><USER14> You have two questions:

1\. What's the advantage?

2\. How do you manage the state of the environment?

The advantage is programming-as-teaching, as opposed to programming-as- carpentry. What I mean by "programming-as-teaching" is that old-fashioned Smalltalk (and Lisp) enviroments like Pharo are designed to support a style of development where you build a program by starting up the runtime and teaching it interactively, incrementally, how to be the program you want.

(What I mean by "programming-as-carpentry" is the more widespread and better- known model in which you are essentially working to build a planned artifact from a blueprint, and your development environment is analogous to a workbench or toolchest, rather than a student.)

If you don't like programming-as-teaching, then it's not an advantage. Most people don't work that way, but I think that's mostly because they don't even know it's an option. I'm not saying that everyone would prefer programming-as- teaching, but it gets so little exposure that I think it's a safe bet that more people would like it if they knew what it was.

I prefer programming-as-teaching over the alternative. I am dramatically more productive when I can work that way. I've been one of those fabled 10X programmers, as confirmed by VCS statistics, when working with such tools. It wasn't because I'm so brilliant or because I have amazing magical powers; it's because we were working with that kind of environment, and I knew how to use it and the other programmers didn't.

So, in brief, the advantage is that if you have such an environment and can use it effectively, you can be dramatically more productive. It changes the feel of programming from "let's build this variant and see how it turns out" to "reach over and correct that student's posture." The cycle of iterations becomes very fast, so fast that sometimes the duration of a cycle is imperceptible.

How do you manage the environment? First, let me acknowledge that modifying your working environment willy nilly is indeed a problem. The solution is that you don't do that.

Instead, you start from a known good system image. Environments like Pharo and other old-fashioned Smalltalk and Lisp environments offer a feature that can save the entire state of system memory to an "image" file. If you start the environment from an image, you get an in-memory state that is an exact duplicate of the state the environment was in when saved--right down to the positions and contents and activation states of all the windows. You start the image, and you're in the environment, exactly as it was when you saved it.

The way you manage incremental state changes is that you make a reference image that contains a known good state. A common choice for a reference image is a clean, unmodified release image. Then you make any site-specific modifications and customizations and save the result as a reference image as well. Then you save a working image based on that reference image and make your changes to the working image.

This practice enables you to easily recover if you get into a bad state. It's very fast to kill a bad session and restore a reference image. For example, a Pharo 6.1 image launches in under half a second on my older iMac. Clozure Common Lisp starts from an image in a similar amount of time.

When I've done ongoing serious work using this kind of system, we kept orderly collections of reference and working images--reference images that served as checkpoints in case of trouble; working images in which we had all the stuff we were working on loaded up and ready to go.

Besides making it quick and easy to resume yesterday's work, and to recover in cases where we messed up, images also served the useful purpose of enabling us to capture precise dynamic states of work in progress. For example, if a rare bug arises, I can save a working image with the problem on the screen, and give that image to my colleagues. They can start up the image and immediately see the problem state on their screens, and use the introspective capabilities of the development system to diagnose the problem.

But I guess that's more an advantage than a how-do-you-manage-it observation.

<comment><USER15> Where does testing fit into programming by teaching? I've worked with people who write some code, refresh the screen, write some more code, refresh the screen, etc. Is that programming as teaching? They definitely produce at least 10x more code than I do, but it is instant legacy code with no tests. Do you go back and write tests after everything looks like it is working?

<comment><USER14> Other comments have pointed out that test-driven design was actually born in the Smalltalk community, which sort of answers your question, by implication, at least: it's standard operating procedure to write a lot of tests as you go.

Indeed, in my experience, any expression I write in the repl is more likely than not to be a test, or part of a test.

The way I normally work is with a running Lisp image and one or more files open in editors. For greenfield development I usually create a project and write a loadfile to load any library dependencies I expect to use. Once that's loaded, I write some expressions to build some test data, and then some more expressions to operate on the test data. I evaluate the test-data constructors and then fiddle with operations on the data to assure myself that what happens is what I expect to happen. Basically, I start with an idea of how to accomplish something, sketch out the data structures and APIs I think I'll need, and start exercising them to see what I got wrong, what I need to add or change.

The contents of those source files gradually become the source code of my program and of the test suite that go with it. When the project is new and small, the sources are a few files with a small number of lines of code, and the organization is minimal and loose. As I flesh out the program, the source files grow more numerous and more formally organized.

As a project gets complex, I may choose to save working images at points where I'm confident that the state of the system is good. Such images enable me to start the Lisp environment with all of my code already loaded and ready to use. I can then work from that point in the same way that I've described.

The normal way I work on anything, unless the tools I have to use prevent it, is to have a work-in-progress running, and use variable references and function calls in the repl to query the and modify the dynamic state. For any expression more complicated than a short one-liner, I actually write it in a source file and copy it to the repl (actually, I usually use Emacs with SLIME, so "copying it to the repl" really means typing a keystroke or two in the source file).

One of the differences between old-fashioned Lisp and Smalltalk systems and newer systems with repls is that the old Lisp and Smalltalk repls are capable of doing every part of program development--defining and redefining functions and data structures, running and debugging code, inspecting dynamic state, catching, inspecting, and modifying erroneous code, and building the finished app for deployment--all by typing expressions at the repl.

To put it another way, the entire language and development tools and all of your application's internals are always in memory and completely accessible from the repl.

It's not that you would normally do all your development in the repl, although you could if you wanted. Rather, the important point is that, because the repl can do anything the language and development tools can do, there are no artificial limitations on what you can do from the repl. Outside old Lisp and Smalltalk systems, every repl I've used always imposes some restrictions--some things you can't do from the repl. Invariably the way I discover this is by needing to do something in the repl and finding that it won't let me.

<comment><USER15> Thanks for the detailed explanation. I haven't used smalltalk, and I think making the mental shift from "source files and an interpreter or compiler" to "working in the image" is something I'll need to experience to fully appreciate.

<comment><USER14> It's easier to grok if you have a friend or colleague who is familiar with it and who can show you the ropes.

Also, there's no guarantee that you'll find it congenial. Some people profess to prefer programming-by-carpentry, and that's fine, of course.

In fact, maybe it's better for your sake if you don't like it. The great majority of programming work is programming-by-carpentry. You might find that you really like programming-by-teaching, and then spend the rest of your life pining for the opportunity to work that way.

</comment></comment></comment></comment></comment></comment><comment><USER16> I left the Smalltalk world years ago (for Java ... hmm was it a win?) but the thing I miss the most is ... reading great code. Within the Smalltalk image/IDE, everyday you read code that other folks - many with brains much much bigger than mine - have written, and its great. A lot of the code I read now (Java,Objective-C) is ... err ... brain hurt :-(

<comment><USER17> > everyday you read code that other folks - many with brains much much bigger > than mine - have written, and its great.

I don't think that it's a language thing. It's probably popularity thing most of all.

In popular languages the quality of the _average_ line of code is pretty low. Plus, you know, real-world constraints of real developers working under real- world constraints....

</comment></comment><comment><USER18> I wonder if anyone familiar with both Squeak and Pharo could give a brief comparison between them, or link to one?

I don't have a specific goal in mind but have always been interested in Smalltalk, and am not sure which one to try out. Thanks!

<comment><USER19> Squeak is the original descendent from Smalltalk-80, where they used the last good image as a starting point for a bootstraped implementation of Smalltalk-80.

Pharo started as a kind of Squeak fork, where they wanted to pursue other ideas related to JIT, GC, compiler technology, traits and so on.

So basically both are Smalltalk-80 implementations, with Pharo trying to be a more modern version of it.

<comment><USER18> Thanks for the info! Very interesting about bootstrapping from the last good image. An old friend of mine (now retired) was involved in Squeak from before the beginning, and this reminds me that I've been meaning to catch up with him. I bet he has some interesting stories.

I tried out Squeak about ten years ago, but never could get past the awful bitmap font. Naturally, the first thing I noticed on the Pharo page was that the fonts look nice! So I installed it on my WQHD ThinkPad to try it out.

The first thing I noticed is the way it responds to the TrackPoint. It is not like any other application I have ever used. Instead of letting Windows control the pointer position, it's doing something very different, I just don't know what.

It's not so bad with the touchpad, but I don't use the touchpad except for scrolling.

It doesn't support high DPI either, so the text is all blurry. Way better than the old Squeak bitmap fonts, though.

I don't want to sound like a complainer, it just wouldn't be pleasant to program in this environment when every other app and dev tool I use supports high DPI and correct TrackPoint response, both on Windows and Linux.

<comment><USER20> The next-gen graphics system for Pharo, called Bloc, is still under development but making good progress. If you can find some of the demos [1] you'll see that things like text rendering are vastly improved.

[1] [https://twitter.com/AliakseiSyrel/status/1050781535983550464](https://twitter.com/AliakseiSyrel/status/1050781535983550464)

</comment></comment></comment></comment><comment><USER21> All is nice and well... until you need to code with a screen-reader. That kind of sucks, as those IDEs obviously use some non-native controls and the only thing the screen-reader can see is a blank window with no usable content. I was once very excited about such projects, but I'm not anymore because I can't really use them in any way.

With normal languages like Java etc. You can just use a different, accessible IDE (i.e. Eclipse instead of Net Beans). That creates some problems, especially when working with a team, but in the end, it can be done if you try hard enough. Smalltalk and smalltalky things that try to be GUI only, no command line at all, are completely hopeless, though.

<comment><USER22> That's an important point but lack of screen-readers (or whatever) support is forgivable at the early stages of a project that gives something conceptually interesting in exchange. I just hope they won't forget about it later when it gets clear that the project stabilizes and its future becomes clear.

<comment><USER23> Smalltalk has been around since the 1970s, though. Given the things the community is interested in, schools, video, sound, and so on, it's odd that accessibility never came up.

<comment><USER24> Most sighted persons really don't like thinking about blindness. It's not because they are bad people, it's just deeply scary and relatively uncommon.

This is an area where Apple really doesn't get enough credit. VoiceOver has absolutely revolutionized blind persons' ability to interact with consumer tech.

For more technically demanding applications, the Unix philosophy is inherently open to being made accessible. [https://pjb.com.au/blin/](https://pjb.com.au/blin/).

As an aside, I personally really enjoy accessible websites, because they focus on clear communication in the presence of a serious obstacle, rather than assaulting the visual cortex with attention grabbing nonsense.

Edit: Also one could argue that ed(1)¹ is the pinnacle of accessibility.

¹[https://www.gnu.org/fun/jokes/ed-msg.html](https://www.gnu.org/fun/jokes/ed- msg.html)

<comment><USER25> Here's a blind programmer who actually did argue, in all seriousness, that ed is the pinnacle of accessibility: [http://eklhad.net/philosophy.html](http://eklhad.net/philosophy.html)

<comment><USER24> Great read. Thanks!

<comment><USER22> Isn't it fascinating how posting a seemingly-useless comment that is guaranteed to be down-voted can provoke people to post interesting things in response? :-)

</comment></comment></comment></comment></comment></comment></comment><comment><USER26> It is still unusable with HiDPI displays. The bug is raised for 5 years, and as Pharo renders everything from scratch, it makes the UI pixelated.

OK, I will wait a few more years. I love Smalltalk, but I don't think I am qualified to fix this myself, unfortunately.

<comment><USER27> Not true, I work on a 4K screen and it is clean. Win10 latest. Windows VM, some settings to tweak in Pharo but it works.

Upcoming is Bloc, which uses Mozilla2D as rendering engine and is vector based.

Next Pharo UI will use it.

See [https://github.com/pharo-graphics/Bloc](https://github.com/pharo- graphics/Bloc)

<comment><USER18> I just installed it on my WQHD ThinkPad running at 225% scaling and everything is blurry. It's definitely not running as a high-DPI-aware app. Are you saying there are some settings I can tweak to fix it? That would be great.

The TrackPoint response is all wrong too, but one step at a time... :-)

<comment><USER27> Looks like this in Win10:

[https://imgur.com/a/X9PcgZe](https://imgur.com/a/X9PcgZe)

Should be better anyway, but it is usable.

I already got it better but the VM has changed since and the rendering is not as good (but is faster for some reason).

Vs IntelliJ, it sucks a tad.

</comment></comment></comment></comment><comment><USER28> Pharo and VR?

Pharo's code browser is visually compact. The code too. Current VR HMDs have only a small VGA-ish region of non-blurry pixels. Hmm...

Might be a good match. Emacs and IDEs with non-compact languages can be head- sloshing. And the HMD OEM market is being dysfunctional, so higher resolution is on the "I don't see a unicorn here - who cares about easily improving the world?" slowmo snail plan.

<comment><USER27> I have been coding in Pharo while in BigScreen on a Rift. Cool experience. Better with the font size set to medium in Pharo settings.

<comment><USER28> Neat. I (occasionally) use emacs in a trivial node.js-as-compositor stack on a Lenovo Explorer on linux. With a 7px font, a pixel-aligned window, no temporal supersampling (because low fps on an old laptop's integrated graphics), and WMR's handwave 500px-ish diameter pixels-not-blurred-together region, it's bearable, but far from competitive with a 1080p laptop screen. With subpixel rendering (Explorer has normal non-PenTile subpixels), a 5px font is ok (but not in emacs), but only across say 300px.

With better lenses and a better panel, an HMD might be competitive. But it's not perceived to be market, and there's pervasive confusion about the (minimal) GPU and (slow) fps needed. And there's only limited diy infrastructure.

But I enjoyed live-coding a vr environment (hot loading three.js and react.) At least with desktop mirroring, to flip-up to screen when the world broke.

Perhaps [https://pharojs.github.io/](https://pharojs.github.io/) ? Croquet Project, Open Cobalt, and OpenQwaq are all dead, and 3dicc seems to have gone closed-source commercial. Ah, ST - feels like the 1980's all over again. :/ But [https://github.com/ronsaldo/woden](https://github.com/ronsaldo/woden) looks alive. And someone apparently hacked a bridge to Unreal. I wonder what else is out there?

</comment></comment></comment><comment><USER20> Pharo is what I want my personal computer to be like.

</comment><comment><USER29> Yayy! More mentioning of ST (and Pharo) on HN. I'm just a beginner, but am slowly learning. So far the IDE is the only one I really like as it is one with the language rather than an entirely separate entity.

</comment><comment><USER30> A bit unclear from the landing page if it's an IDE with a REPL or if you can edit the IDE you're running in..

Kinda sounds like an object oriented Emacs? Or what the creator of Nightlight is trying to do with Clojure.

<comment><USER31> It's basically an open-source, interpreted, pure OO operating system. You can make changes to the system's code while it's running, any part of the system, and changes will take effect immediately.

<comment><USER2> The "interpreted" bit isn't really true anymore. Smalltalk has been bytecode- compiled since, uh, I think 1976, and the bytecodes themselves are JITted by the Pharo/Squeak virtual machine these days. They're usually only interpreted when you're using the debugger.

</comment></comment></comment><comment><USER32> Serious question though, what is immersion in the programming context? Is it being only in the programming environment and not leaving? Wouldn't using a web browser for documentation or stack overflow be a break in it? Why not just have the windows one needs and configure your ide to be how you like?

<comment><USER33> > Is [immersion] being only in the programming environment and not leaving? > The dream is never leaving, as expressed by the Dynabook concept. However, > in the full expression of that dream, the browser et al that you mention > would _also_ be inside the context.

In practice, things like using a browser for documentation is acceptable but not super common because viewing in the IDE is so much more powerful in certain ways (e.g. executable examples, finding "senders" of a message).

The idea is about having one's computer be "turtles all the way down" where you have the same power to change and explore every layer of the system without learning a whole new technology - from what we usually leave to the OS (which Dan Ingalls said "shouldn't exist") to what currently exists in apps (which Alan Kay and the creators of Smalltalk envisioned as services, which didn't stovepipe possibilities/power down to a small fraction of what's available).

<comment><USER32> Oh I see! That would be pretty neat, actually. Is looking up documentation a common feature in IDEs? I've never seen it before.

</comment></comment></comment><comment><USER34> Say I've developed (taught) an image - full of useful stuff. Now a new version of Pharo comes out. How do I move that stuff to the new version?

<comment><USER35> I'm pretty new, but I've found a few ways. Most useful to modern development is a package called Iceberg, which is a git interface...you can export your classes/code/etc, commit them, and pull them into different images. You get to choose what you export, so you can only export your code.

There's also Monticello which is similar, but uses different repository types than git.

You can also just "File out" a class or package, which exports it to the file system.

</comment></comment><comment><USER32> I swear to God if Aragon takes over Castille while I'm programming my immersion will be ruined! Needs to be 100% historically accurate!

</comment><comment><USER36> Can someone explain how a language can't have primitive types?

<comment><USER37> If you are familar with Java: Just imagine all integers are objects from class Integer and that int doesn't exist.

(Internally, they are actually represented as 31-bit numbers for performance reasons)

<comment><USER38> Or non-hypothetically, integers in Ruby -- 5.methods shows that 5 is an object just like anything else. But then Ruby has been called a modern Smalltalk for this reason.

<comment><USER19> With the difference that cruffy Smalltalk still has better JITs available.</comment></comment></comment></comment>


TileDB 2.0 and the Future of Data Science - <AUTHOR> https://medium.com/tiledb/tiledb-2-0-and-the-future-of-data-science-929cdcfe95ed <comment><USER3> I don't know if the creators of TileDB visit HN, but if they do - it would be nice to see a code example that showcases what makes TileDB unique. Right now, I don't see any benefit versus serializing a data frame in hdf5 and storing it in Redis or cloud blob storage.

(Reading [https://tiledb.com/developer](https://tiledb.com/developer), the Github, or the blog - none of it shows me anything amazing)

Edit: browsing around the site, I finally found out what tileDB offers, and I'm still confused (too much fluff language) To save people time:

[https://docs.tiledb.com/main/handling- dataframes](https://docs.tiledb.com/main/handling-dataframes)

Multi-column slicing: By defining any subset of the columns as dimensions and due to TileDB's tiling flexibility, you can increase the pruning effectiveness of multi-column slicing, thus leading to better overall read performance. Essentially, a TileDB array acts as a primary multi-dimensional index on the columns selected as the array dimensions.

Data updates and versioning: TileDB offers rapid, parallel, cloud-optimized updates. All the update logic is pushed into the storage engine and is completely transparent to the user. TileDB also exposes useful time traveling functionality, such as reading arrays at time snapshots, effectively implementing data versioning built into a single embeddable library.

Partitioning: TileDB enables balanced partitioning, without limiting each partition to single column values. Moreover, we will soon expose API functions for dynamically selecting different partitioning schemes (e.g., on different subsets of columns with different orders), without the need for reorganizing/rewriting the array.

Sorting: All sorting is taken care of by TileDB internally with multi- threading.

<comment><AUTHOR> I heard about TileDB because one of the authors commented at [https://news.ycombinator.com/item?id=22423632](https://news.ycombinator.com/item?id=22423632) . That discussion includes what you mentioned, plus a mention that it integrates with MariaDB, PrestoDB and Spark, including the ability to use an embedded MariaDB.

<comment><USER4> Thanks for the post!

</comment></comment></comment><comment><USER5> So, basically, tiledb is a file format like paraquet, avro and orc are formats?

And it comes with a nice library that makes accessing the data fairly storage- transparent?

<comment><USER6> Disclosure: I am a member of the TileDB team.

Yes you are correct, TileDB introduces a storage format, but is also a full- fledged storage engine that implements it. Having the complete storage engine is what allows us to make accessing the data transparent regardless of the API or application you use to access the data. We are able to integrate into different computational tools without having to reimplement or sacrifice features.

<comment><USER5> How do you compare to Iceberg?

<comment><USER6> Disclosure: I am a member of the TileDB team.

Iceberg, similar to Delta Lake, is another layer you have to add on top of Parquet, which limits your mobility of compute. You can currently use it either with Spark or Presto, but not directly via language APIs or other compute engines (e.g., pandas, Dask, MariaDB, etc). For example, the pandas "read_parquet" will not be able to read your dataframe, you will have to use pyspark or presto + sql query to fetch the data into pandas. TileDB has updates and data versioning built into its format and storage engine, therefore reading an array directly via a TileDB language API, or Spark, Dask, PrestoDB or MariaDB will have the same behavior. You will be able to see the dataframe after all the changes, or time travel and "activate" only some of the changes in time.

<comment><USER7> This isn't strictly true.

You can't arbitrarily reach into the data from the client with Iceberg or Delta Lake, true, but that's intentional. The service on top provides a lot of the functionality TileDB provides. There's nothing preventing you from writing a pandas integration, for example.

Notably, the most significant difference I can see is that, as far as I can tell, TileDB will not solve for failures due to S3 consistency, which are solved by Iceberg and Delta Lake. It's strictly necessary to have a central, ACID-ish place to record transactions, as probing S3 is not reliable.

At the very least, you are definitely suspectable to stale reads, which means users will see all sorts of bizarre failures at scale. I'd be pretty terrified of putting anything serious on TileDB.

<comment><USER6> Disclosure: I am a member of the TileDB team.

Several important concepts are being conflated here, so I'll elaborate on each separately.

S3 eventual consistency: TileDB is fully aware and designed around the eventual consistency guarantees of cloud object stores [1]. When an array is opened, only the committed writes (up to the specified timestamp if using time traveling) are seen by the reader (each write produces a timestamped "fragment", which is essentially a folder). There will never be partial reads, or corrupt reads. The array is always in a readable state with committed data. This works very similar to Iceberg's method of opening a table at a snapshot for reads [2]. I don't believe that TileDB is any more affected by stall reads than Iceberg (or Delta lake), the user must reopen or re-query a table to see data from a newer timestamp/snapshot.

Handling S3 write failures: TileDB is designed for a lock-free, multi-writer scenario. All writes produce a new timestamped fragment, which is immutable after completion of the write. TileDB performs an atomic write of an "ok" object, which signals when the fragment is complete. Any fragment which is missing the "ok" file, is ignored by the reader [3]. TileDB handles corrupt or incomplete fragments by erroring out on the read. In the future, we could offer a retry mechanism for failed reads, but the important thing is we will never return corrupt or invalid results. Incomplete fragments can happen because of S3's eventual consistency; it is possible for the atomic ok file to show up before an object in the fragment "folder".

Write serializability: TileDB's fragments are written at a (timestamp + uuid) fashion. In the event of a conflicting write at the same timestamp, the uuid provides uniqueness and guarantees that there are no errors by essentially randomly ordering the conflicting fragments. The effect in the end is similar to Iceberg's cancel and retry conflicting writes, except TileDB does not have the penalty of retrying. For Iceberg, if there are two simultaneous conflicting writes, it is effectively random which one would be accepted and which one would be retried.

ACID: TileDB does not support ACID intentionally, as it was not designed to be a transactional database. TileDB was designed with a lock-free multi- writer/multi-reader model, as our use cases up until now involved a one-off massive parallel write, and then multiple concurrent reads. Lack of transactions does not yield inconsistent data though, as stated above the read/write algorithms are specifically designed with this in mind. That said, we have recognized that there are workloads where transactions are important, and we have plans to eventually add transactions to our cloud product where we have the orchestration layer needed to manage them. It is important to keep the transactional layer modular and format-agnostic (Delta Lake is blending it with Parquet) and, therefore, we will build it on top of TileDB, not inside the storage engine.

Interoperability: Obviously nothing stops someone from writing a pandas integration for Iceberg or Delta Lake. An important part of our philosophy at TileDB is extreme integration with existing tools and frameworks, which is why we strive to support things like returning numpy arrays for the Python results and doing zero-copying everywhere possible. In other words, it is not trivial to add a tool integration, just the same as it is not trivial to just add an ACID layer.

[1] [https://docs.tiledb.com/main/basic- concepts/consistency](https://docs.tiledb.com/main/basic-concepts/consistency)

[2] [https://iceberg.apache.org/reliability/](https://iceberg.apache.org/reliability/)

[3] [https://docs.tiledb.com/main/basic-concepts/physical- storage](https://docs.tiledb.com/main/basic-concepts/physical-storage)

<comment><USER7> Thanks for the detailed response.

This is what I was getting at, mostly:

> Any fragment which is missing the "ok" file, is ignored by the reader [3].

As a client, how do I detect this? Say I just wrote some data and now I want to generate a new total. Could my data be silently missing?

> TileDB handles corrupt or incomplete fragments by erroring out on the read. > In the future, we could offer a retry mechanism for failed reads, but the > important thing is we will never return corrupt or invalid results.

But now I have to handle this everywhere I interact with TileDB. Most users don't expect essentially random errors. Retries are a stop-gap, as it can take arbitrary time for consistency to converge. I've observed O(hours) regularly.

> Incomplete fragments can happen because of S3's eventual consistency; it is > possible for the atomic ok file to show up before an object in the fragment > "folder".

Indeed! And from what I understand, mostly from conversation with the Iceberg team last year, they solve this entirely by using a central ACID service backed by RDBMS (which, aside, I find very amusing!! They never listen to Stonebraker...).

Indeed, as far as I can tell, the only way to remove consistency issues is to: \- store an index to objects somewhere central \- never list the bucket (or speculatively stat, etc.)

Until either a) you support a central index or b) S3 has a better consistency model (not that DynamoDB garbage Hadoop uses) I would be very reluctant to use TileDB if it wasn't using my own disks.

I think for small stuff, it'll work great, but at scale you're going to hit some roadblocks. I recently encountered splitting a bucket into to many buckets and prefixing UUIDs at the root of the bucket to reduce consistency problems caused by S3's rebalancing, and it ended up being a very expensive marginal improvement.

<comment><USER4> Folks, apologies, but I think we got a bit side tracked here, TileDB does not suffer from the consistency issues mentioned above.

Here is how TileDB performs a new (potentially concurrent with other reads and writes) write:

\- It creates a fragment folder (or "prefix" of a set of objects on S3 - <USER2> are no "folders" on S3) which is timestamped and carries a unique UUID. This fragment is self-contained and represents the entire write (e.g., all cells and all attribute values)

\- It writes all data objects under the fragment prefix. Note that TileDB never updates, it always writes new immutable objects.

\- After all the PUT requests succeed for the data objects, it creates an empty "ok" object.

Here is how TileDB performs a (potentially concurrent with other reads and writes) read:

\- It lists the array prefix to get the ok objects

\- There are two cases:

1\. The ok object is not there for some fragment. That fragment is completely ignored.

2\. The ok object is there. Since TileDB writes the ok object last, all the data objects it wrote have been committed and are all visible with GET requests. TileDB reads the data objects only with GET requests (not ListObject requests). Due to S3's read-after-write consistency model ([https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction...](https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel)), all those objects will be available for reading (now on all S3 regions) with GET and there will be no errors.

Therefore, TileDB follows the eventual consistency model of S3 without any errors and surprises. The user doesn't need to handle anything. Our customers have been using TileDB in production for a long time, storing hundreds of TBs of data on S3, and no consistency issue has ever come up.

Summarizing, what xyzzy_plugh is raising here is that TileDB does not have ACID guarantees. And that is true (we never claimed the contrary) and intentional. We are building a transactional layer outside of the storage engine. The reason is that this transactional layer indeed needs to be a constantly running distributed service, whereas we want the TileDB storage engine to be embeddable and used without performance regression even by applications that do not need ACID (that is, the majority of our data science applications).

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER7> This seems similar to the direction Apache Arrow is going in, but I don't see the use case for this once Arrow dataframes are prevalent.

<comment><USER6> Disclosure: I am a member of the TileDB team.

TileDB is designed primarily for persistent storage, similar to HDF5, Zarr and Parquet. TileDB is a cloud-optimized, dense and sparse multi-dimensional array storage engine, which is broader than Arrow's Dataframe. In addition TileDB also handles updates, time traveling and partitioning at the library level, which are not possible with the Arrow project's current choice of persistent format, Parquet. TileDB removes the need for using extra services like Delta Lake for updates or Hive for cataloging and partitioning, as you get all this functionality in a single, embeddable library.

<comment><USER8> I feel like you guys need to create some additional engineering-level documentation to convey all this. The landing page and few links I navigated to were all too fluffy.

<comment><USER9> For things that start as academic work, often the original paper is quite helpful. I find it an enjoyable read if you have a background in databases (it actually was a reading for my grad systems class)

[https://people.csail.mit.edu/stavrosp/papers/vldb2017/VLDB17...](https://people.csail.mit.edu/stavrosp/papers/vldb2017/VLDB17_TileDB.pdf)

The basic jist is: TileDB is substantially faster than alternatives for random writes/reads - this allows for more sophisticated parallel linear algebra algorithms as well as evaluation of predicates.

</comment></comment></comment></comment><comment><USER10> After reading it, it's not clear it is the future of data science.

</comment><comment><AUTHOR> The highlighted new features for version 2.0 are: 1) Support for heterogeneous dimensions in sparse arrays (that is, full dataframes), 2) Support for string dimensions in sparse arrays, 3) Google Cloud Storage and Azure Blob Storage support, and 4) Completely revamped R API.

They write: "These features mark an important step in our journey to deliver a universal data storage engine for all applications and data science tools."

<comment><USER11> i agree, the title and article is mostly hyperbole

<comment><USER4> Stavros form TileDB here. Here is a more verbose explanation. Up until before 2.0, TileDB was already powerful for the main applications we targeted at: geospatial and genomics. The support for both dense and sparse arrays and the way it handles data versioning made it quite unique vs. HDF5 and Zarr. But we noticed that most of the data scientists we were working with had a lot of data beyond genomic variants, LiDAR points and rasters. They had tons of dataframes. And they were using at least two storage engines, TileDB for arrays, and Parquet or a relational database for dataframes. If you are in a large organization, this a big pain.

In TileDB 2.0 we made a huge refactoring to support something seemingly simple: dimensions in sparse arrays that can have different types and that could even be strings. This allowed us to model any dataframe as a sparse array, effectively making TileDB act as a primary multi-dimensional index. In relational databases, this means that your data is sorted in an order on disk that favors your multi-column slicing enormously, so range search becomes rapid.

Therefore, what we are telling the community with this release is that you can have dense arrays, sparse arrays, and dataframes in a single embeddable library being integrated with pretty much every data science tool out there, so that data scientists never have to worry about backends, files, updates, or anything other than their scientific analysis. In other words, we believe the future of data science is more science.

<comment><USER12> Show us code usage examples please!

<comment><USER4> Full developer docs here: [https://docs.tiledb.com/main/](https://docs.tiledb.com/main/)

Specific dataframe examples coming up shortly.

</comment></comment></comment></comment></comment><comment><USER13> Looking at the TileDB docs, I like the idea that you can store and manipulate array efficiently. It works like a columnar storage, but organized in tiles. I am not clear how it could scale to multiple nodes and how to do queries. Maybe it should be integrated into postgres.

<comment><USER6> Disclosure: I am a member of the TileDB team.

TileDB fully supports multi-writer/multi-reader, so scaling compute to multiple nodes is mostly about choosing the computation layer. We have integrations with Spark, Dask and PrestoDB for distributed compute. In our cloud product we also offer serverless UDFs and SQL (via MariaDB) to allow for scaling out computations elastically without managing your own clusters.

On the postgres side, we are looking at eventually adding a postgres storage engine, now that the new storage support is in PostgresSQL 11. To start with we have built a storage engine for MariaDB, as MariaDB has an excellent storage engine API, which I have past experience with. We hope to upstream the storage engine to MariaDB after the MariaDB 10.5 release.

MariaDB integration: [https://docs.tiledb.com/mariadb/](https://docs.tiledb.com/mariadb/)

Spark integration: [https://docs.tiledb.com/spark/](https://docs.tiledb.com/spark/)

Dask integration: [https://docs.tiledb.com/dask/](https://docs.tiledb.com/dask/)

PrestoDB integration: [https://docs.tiledb.com/prestodb/](https://docs.tiledb.com/prestodb/)

Cloud Docs: [https://docs.tiledb.com/cloud/](https://docs.tiledb.com/cloud/)</comment></comment>


Under the Dome – Investigating China's Smog [video] - <AUTHOR> https://www.youtube.com/watch?v=T6X2uwlQGQM

<comment><AUTHOR> The film has been taken offline in China by the Communist Party's central propaganda department.

[http://www.nytimes.com/2015/03/07/world/asia/china-blocks- we...](http://www.nytimes.com/2015/03/07/world/asia/china-blocks-web-access- to-documentary-on-nations-air-pollution.html)

</comment><comment><USER2> There's quite an interesting story surrounding this documentary. Apparently it was initially pushed by state news outlets and the minister of environment and within days garnered over 100 million views in China.

Then later it was reconsidered, and the video removed and articles about it started getting censored.

</comment><comment><USER3> A neighbour came back from a trip there recently. He is old enough to remember the last of the pea soupers we used to see in the UK before they cleaned up the air (a bit).

Air pollution is no joke. They publish the number of people who will die, who will die early, and who will suffer serious complications as a direct result.</comment>


Tesla Model S Catches Fire: Is This Tesla's 'Toyota' Moment? - <AUTHOR> http://www.thecarconnection.com/news/1087401_tesla-model-s-catches-fire-is-this-teslas-toyota-moment <comment><USER2> No. "some 250,000 gas-powered vehicles catch fire in real-life settings every year." [Washington Post's WonkBlog, 1/9/12] [http://www.washingtonpost.com/blogs/ezra-klein/post/are- elec...](http://www.washingtonpost.com/blogs/ezra-klein/post/are-electric- cars-really-a-disappointment/2012/01/09/gIQArQVWlP_blog.html)

<comment><USER3> As this article points out, the facts aren't what's at stake here -- it's media/public perception.

<comment><USER4> To heck with that noise. Push the public and media to perceive that facts are important.

<comment><USER3> You make it seem so easy.

Let me ask you: How well did that work for Toyota?

<comment><USER5> Tesla's got two things going for it that Toyota didn't. It's an American company so it's not fighting against any xenophobic tendencies in the media or public. It's a singular incident, not a series of incidents. If more minor accidents similarly result in fires then they've got a problem, if they can show that most of the minor accidents (might be hard to get statistics on, but they can try and collect them going forward) result in nothing worse than needing repair work then they won't have an issue. It also helps that, in general, they're perceived as a very transparent company when it comes to safety.

</comment></comment></comment></comment></comment><comment><USER6> Previous discussion: [https://news.ycombinator.com/item?id=6488250](https://news.ycombinator.com/item?id=6488250)

</comment><comment><USER7> No, but it is an excellent time to purchase TSLA at a 6% discount.

<comment><USER8> Already ahead of you. Price drop? Time to buy!

<comment><USER9> 6%..? Wow, time to buy two!

<comment><USER8> I own several thousand shares of TSLA, so %6 saves me quite a bit between buying at ~$170 vs ~$190. I bought the initial investment at $17, but I keep looking for drops in price to increase my holdings.

</comment></comment></comment></comment><comment><USER10> No, at least not yet. This isn't on the front page of CNN, Fox News, NYTimes, LATimes, or the Washington Post.

Until this becomes a mainstream media story (rather than just a niche tech/auto story), Tesla's fine.

</comment><comment><USER11> No.</comment>


Learn 4 ways to impress your boss and team with your web development skills - <AUTHOR> http://www.supacoderz.com/blog/2013/11/learn-4-ways-to-impress-your-boss-and-team-with-your-web-development-skills/

<comment><USER2> I prefer to sleep on Saturday mornings

</comment><comment><USER3> For the most part, good points.

I'd add a #5: Be able to work with any team member. If you can get along well with most everyone, they (probably) won't say negative things about you to your boss behind your back.

</comment><comment><USER4> Visual Studio? Really? Do people still use that? Other points are good though.</comment>


The PC BiOS will be killed off by 2020 as Intel plans move to pure UEFI - <AUTHOR> https://arstechnica.com/gadgets/2017/11/intel-to-kill-off-the-last-vestiges-of-the-ancient-pc-bios-by-2020/ <comment><AUTHOR> Related previous discussion: [https://news.ycombinator.com/item?id=15730113](https://news.ycombinator.com/item?id=15730113)</comment>


Walking Naked into the Embassy - <AUTHOR> http://billt.posterous.com/walking-naked-into-the-embassy

<comment><USER2> Will they no longer keep restricted items at the guard station outside the embassy? It's a couple of years since last time I was there.

<comment><USER3> In Stockholm they _say_ they won't hold stuff in the instructions, but if you do have something that you can't enter with, they will. I guess they just don't want to be inundated with stuff.

</comment></comment><comment><USER4> This pointless, rambling, navel-gazing is not a harbinger of delightful insights from Digital Planet.

<comment><USER5> What you say might be true, but you're not making an argument so much as you are just name calling. If you don't find anything in the argument to respond to, it's better to not respond to it rather than fill HN with such vitriol.

<comment><USER6> The policy to not allow electronic devices, and expanding what this means to ludicrous levels might not be the smartest policy, but the point of this article is basically "I get bored without my toys".

I try to go on a longer vacation every year, and I never bring my laptop or go online when I'm gone. There are a lot of people I interact with over the internet on a daily basis, but it really is ok to let go for a while. If you stay off the internet for two weeks, they sky won't fall down, you won't have missed anything, and you really don't need to update your facebook status or twitter some crap every day. The author of this article should try it sometime, unless the extreme hardship of being disconnected for a few hours has scarred him for good...

</comment></comment></comment><comment><USER7> Having done this for my L1 visa, and jettisoned the USB key I forgot was on my key ring, I'm still puzzled by the author.

All the documentation is very upfront about this procedure. I think it should be obvious to anyone vaguely technical the range of small bugs that can be made now.

As such removing disambiguation is pretty sensible plan. Rather than airport security which makes a pretense of banning "dangerous" items at least in this case they can say you were warned and there is no room for misinterpretation.

</comment><comment><USER8> The U.S. Consul services offices in Shanghai have not allowed electronic devices for at least the last 5 years. They have lockers to store thing in prior to entering the waiting area. Its not something that bothers me but I do find it unnecessary.

</comment><comment><USER9> This is the first I've heard of electronics (CF card? what?) not being allowed into US embassies. Is this new?

Regardless, it's embarrassing.

</comment><comment><USER10> The US emb in Lon will hold things for you now. This is new in the past 12 months or so.</comment>


San Jose May Put License Plate Scanners on Garbage Trucks - <AUTHOR> http://www.mercurynews.com/bay-area-news/ci_28666531/san-jose-looks-at-using-garbage-haulers-catch

<comment><USER2> I don't have any plates on my 6 month old car. The penalty for not having plates is just a $20 fix-it ticket and no points on your record (you will never get pulled over just for not having plates, especially if the car looks new). The upside is that you can't be identified by cameras or passing cars (though when parked, the VIN is still visible unless you cover that too).

</comment><comment><USER3> More likely used to find cars that have significant parking fees, and/or expired tabs.

<comment><USER4> It doesn't matter what it's used for _today_ , and it especially doesn't matter what you or I _think_ it will be used for.

Data doesn't got away, and aggregation of data makes it more dangerous. The problem is when this data is combined with other licence plate databases in the future to produce a map of most people's movement data.

As we've seen with the recent Ashley Madison and OPM hacks, the person doing the aggregation may not be the people who originally collected the data. In fact, the obvious problem of someone finding the intersection of the Ashley Madison leaks with the security-critical people affected by the OBM hacks is a perfect example of this kind of risk.

Finding a way to properly secure a database filled with personal information will be necessary in some cases, but limiting the risks will always be a better idea. Data that isn't saved cannot be stolen in the future.</comment></comment>


Multiple IP Addresses for EC2 Instances - <AUTHOR> http://aws.typepad.com/aws/2012/07/multiple-ip-addresses-for-ec2-instances-in-a-virtual-private-cloud.html

<comment><USER2> Seems like this is only for Virtual Private Cloud (VPC) users, unavailable for us who uses EC2 on public cloud.

<comment><USER3> You can create your own public cloud on AWS VPC.

</comment></comment><comment><USER4> This is definitely a useful feature (even more powerful when combined with route53); and $44 a year per IP address isn't too shabby either. I'm curious if this is a way to try to stay ahead of Azure's plans to try to corner hosting with their Web Sites feature (eg [http://blog.ntotten.com/2012/06/08/three-more-things- about-w...](http://blog.ntotten.com/2012/06/08/three-more-things-about- windows-azure-web-sites/) )...

<comment><USER5> I read that link and I'm wondering what I'm missing?

This looks like some attempt by Microsoft to compete with low-end shared hosting, right? Amazon has never been in that market, and has never shown any interest in heading that way.

Additionally, it isn't clear why anyone would choose Azure for that if they were building a website from scratch.

The only compelling use-case seems to be people who have a .NET website they need hosting for (.NET hosting doesn't seem as competitive as other environments). Yes, Azure does support other environments, but there isn't anything particularly compelling about Microsoft's offering for anything except .NET. If you add in a reserved instance even the pricing isn't very good.

</comment></comment><comment><USER6> I wonder if Heroku will lower the prices in SSL now -> [https://devcenter.heroku.com/articles/why-does-ip-based- ssl-...](https://devcenter.heroku.com/articles/why-does-ip-based-ssl- cost-100-month)

<comment><USER7> FWIW, They already have: <https://addons.heroku.com/ssl> (I believe their new offerring just uses ELB under the hood)

<comment><USER8> Using ELB to achieve this means they are doing SSL termination on the ELB itself which means the request is no longer encrypted within heroku, unless of course they are using another cert for the connection between ELB->dynos.

<comment><USER9> Not necessarily, you can use an ELB to pass through the TCP connection and terminate at the instances

<comment><USER8> But if the instances also live on ec2 (which they do in the heroku case), you would still run into the issue of needing separate instances to terminate SSL on for each SSL enabled site (with the 1 IP limit). It would seem much more economical for them to terminate at the ELB level and only pay for an ELB per customer.

</comment></comment></comment></comment></comment><comment><USER10> Gah. When are they going to enable this for public cloud customers?

<comment><AUTHOR> You can create a VPC and then add an Internet Gateway to it, making instances on particular subnet(s) of your VPC accessible from the Internet.

[http://docs.amazonwebservices.com/AmazonVPC/latest/UserGuide...](http://docs.amazonwebservices.com/AmazonVPC/latest/UserGuide/VPC_Internet_Gateway.html)

Does this meet your needs?

<comment><USER11> One issue I have with VPC is that instances don't get public IPs by default like public cloud instances do. Instead you have to attach EIPs, which has extra complexity (esp. taking care to deallocate them if you don't care about persistence), and you're limited to only 5 EIPs by default. I know you can request to increase the limit, but I have no idea what the approval process is like, and it seems completely silly when you can get as many public IPs as you want by launching multiple public cloud instances.

(If you're curious, my architecture requires hosts outside of AWS to connect directly to EC2 instances. Using a NAT box inside VPC would likely introduce a bottleneck. I don't need multiple IPs per instance, but I just wanted to provide some feedback as to why VPC with an Internet Gateway feels inferior to public cloud EC2 for me. Thanks!)

<comment><USER12> We (AppHarbor) haven't had problem getting additional public IP addresses, just ask.

</comment></comment></comment></comment><comment><USER13> We requested it for public instances years ago. VPC doesn't help.

<comment><USER14> Why would you not want to use VPC?

<comment><USER8> The main reason I can think of is the single point of failure in regards to a NAT instance. With the only other alternative being giving EIPs to every host that needs internet access. I would like to see a more robust solution and HA for the NAT instance. Especially if you're doing any kind of heavy proxying, couldnt you max out the NAT instance's uplink at some point? Anyone have ideas?

<comment><USER15> Use ELB instead of a NAT instance?

</comment></comment></comment></comment><comment><USER16> i guess this was one of the most requested features of EC2

</comment><comment><USER14> i've wanted this for years, excellent news

</comment><comment><USER17> Am I the only one who thinks this would be a lot less complicated without NAT?

<comment><USER8> no

</comment></comment><comment><USER18> Anyone know where I can find the interface/ip limits per instance type?

<comment><USER19> here: [http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/in...](http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/instance- types.html#AvailableIpPerENI)</comment></comment>


Layers in Software: From Data to Value - <AUTHOR> https://blog.jessitron.com/2019/11/05/layers-in-software-from-data-to-value/ <comment><USER2> Summarized well by these lines:

> Back in the day, front end, back end, operations, and DBAs separated because > they needed different skills. Now we accept that a software team needs all > the skills. We group by responsibility instead — responsibility for business > value, not for activities

> We once layered by serving data to software. Now we layer by serving value > to people.

This is a nice narrative for the rise of developer products, where there used to not be a market.

<comment><USER3> Andy Grove's High Output Management(hate the name, like the book) also talks about this tension between teams that focus on the business unit and the horizontal teams that support(if I recall correctly he groups in marketing, finance, and others) the business units.

You get large advantages in having teams built around specialized functions, however at the cost of lower cohesion.

Business verticals will trade all sorts of advantages for that cohesion.

There's tension in finding the right mix, the book covers it much better than I can but it's interesting to see parallels in tech stacks as well.

<comment><USER4> > however at the cost of lower cohesion

Is that cohesion within the team, or across teams (in the company as a whole)?

</comment></comment></comment><comment><USER5> "Back in the day, front end, back end, operations, and DBAs separated because they needed different skills. Now we accept that a software team needs all the skills."

If the problem domain is adequately handled by generalists that's not a problem. But in situations requiring specialized skills it means needing to rely more on specialty contractors who keep your system from crashing when you go past the complexity threshold. As a specialist contractor it's very much in my interest to promote teams with "all the skills" because it means I won't be wanting for work.

<comment><USER6> IMO the argument isn't that you have 5 generalists, but rather that you might have 1 FE engineer, 2 software engineers a DBA and a Devops engineer (or at least 5 Software engineers that are more or less 'specialists' in those domains)

This allows you to have a self-contained micro-team that can approach each value-add opportunity from the perspective of the overall solution, without any of the bureaucracy of communicating with someone outside of the team.

</comment></comment><comment><USER7> Software is more than CRUD applications for business automation, but even for those, I'm not sure how accurate the characterisation here is, particularly regarding external dependencies. SaaS/cloud/this-week's-buzzword has done well for a few very successful businesses, particularly the handful of giants providing the infrastructure it runs on, but outsourcing everything you can think of is hardly universal.

</comment><comment><USER8> Fluffy words. The developer version of the agilistas discourse.

</comment><comment><USER9> > Interfaces between teams updated with every application change.

Dev products do not solve issues of bad architecture. It will be same struggle but in more strong form.

</comment><comment><USER9> Also this comes to mind. Most of successful IT companies do not use external services for core components and prefer inhouse solutions.</comment>


An Interesting Blog in Machine Learning and Artificial Intelligence - <AUTHOR> https://machinelearningmindset.com/blog/ <comment><USER2> Is this your blob? (Just asking. It's ok to submit your stuff here. But don't repost too much.)

It's much better to submit one post than the whole blog. Which one is your best article? I prefer articles that mix some relevant images with technical details.</comment>


An in-depth look at CVE-2018-8878 or why integer overflows are still a thing - <AUTHOR> https://blog.sqreen.io/buffer-under-read-ruby/ <comment><AUTHOR> As a former security researcher, I am amazed that integer overflows are still a thing in 2018, in the Ruby core - so probably everywhere... About 10 years ago, integer overflow vulnerabilities were trending in the security community. Plenty of nice vulnerabilities and exploits have been found with them - <USER2> in all PDF readers, ... I guess when the momentum of such vulnerabilities goes down, this class of bugs goes unnoticed. Secure programming is hard, it's too bad we don't keep the learning we had in the past.</comment>


German state of Hesse bans cloud services from schools due to privacy risks - <AUTHOR> https://arstechnica.com/information-technology/2019/07/germany-threatens-to-break-up-with-microsoft-office-again/ <comment><USER2> Dupe (different article, same topic): [https://news.ycombinator.com/item?id=20439545](https://news.ycombinator.com/item?id=20439545)

<comment><USER3> Thanks. Comments moved thither.

</comment></comment><comment><USER4> Headline is at least misleading, IMHO plain wrong. This is only in Hesse, one of the 16 states. This doesn't mean this interpretation of the Hesse Data Protection Office is valid for the whole of Germany. /pedantic

<comment><USER5> Also misleading because of this, right at the top of the article:

> What is true for Microsoft is also true for the Google and Apple cloud > solutions. The cloud solutions of these providers have so far not been > transparent and comprehensibly set out. Therefore, it is also true that for > schools the privacy-compliant use is currently not possible. > \- Hessian > commissioner of Data Protection and Freedom of Information</comment></comment>


DeepMind Safety Blog: Understand Agent Incentives via Causal Influence Diagrams - <AUTHOR> https://medium.com/@deepmindsafetyresearch/understanding-agent-incentives-with-causal-influence-diagrams-7262c2512486 <comment><AUTHOR> They call their preprint "Part I", but they don't seem to have published any subsequent part yet.

However, there already exists one preprint citing it:

[https://arxiv.org/abs/1903.05168](https://arxiv.org/abs/1903.05168) "On the Pitfalls of Measuring Emergent Communication"</comment>


The Choose Yourself Era - How to Get 100,000 Facebook Fans - <AUTHOR> http://www.jamesaltucher.com/2012/06/the-choose-yourself-era-how-to-get-100000-facebook-fans/ <comment><USER2> Awesome post - how James Altucher self published ebooks and used Twitter Q&A's to get 100k facebook fans</comment>


Google 2000 vs. Google 2011 - <AUTHOR> http://www.mattcutts.com/blog/google-2000-vs-google-2011

<comment><USER3> While I also agree that Google's search results have gotten better, I just feel (this is very subjective of course) that they should be far better today than they currently are. I don't think they're leaps and bounds better and I think they should be.

I'm not one to constantly point to yesteryear and claim they were better but I do look at where they are today and think they should be further along than they are.

I guess when I look at certain segments of technology in 2000, some other companies have stayed more focused on their primary goal and have made phenomenal achievements since then. For example, OS X wasn't released to consumers until 2001 and the iPod didn't exist. Yet look where we are today. I have an iPhone in my pocket with incredible computing power, a very good refined interface, and a fantastic OS. Apple has stayed very focused on OS and consumer device design and pushed the field forward very far, with everyone else scrambling to catch up. This is their main business, and while they have other side projects (like the AppleTV) they stay focused on their main thing which is making great devices for consumers.

While Google, whose primary business was search+, now operates in many different business segments. Their focus has widened greatly and now they have email, payment processing, online document editing, 2 operating systems, automated cars, and the list goes on and on. Now, I have great love for many of these products and services so I do not suggest that it was a bad idea for them to do them, but I think it is apparent that search has suffered as a consequence.

Had Google stayed more focused on search instead of all these side businesses, search could be far more developed than it currently is today. I think they got complacent while being on top and tried to find other places to allocate their resources. Which, from a business perspective, I understand, but from a search perspective it seems tragic.

Therein lies my disappointment with Google, not that they haven't been innovating or improving search, they have. But they have spent so much time and resources on other things that search is not as far along as it should be.

\+ Certainly it could be argued that their primary business is and always has been advertising and search is just one of the many vehicles for pushing said advertising, but the word Google is now synonymous with search and, marketing speak aside, I consider it their primary focus.

<comment><USER4> Well for 10 years I think the search results have only marginally gotten better. Besides, 10 years ago, what google was doing was revolutionary. No other search engine had such vast coverage or better intent determination. Now the competitionn is intense, Bing is almost as good as Google if not better in some scenarios, other guys are blekko are coming up with different approach (and have a better way to handle spam by avoiding it completely).

I think Google is spreading itself too thin. Search is what they do the best and its their bread and butter. They should stay focused on it.

Google is too engrossed in thinking about generating more ad dollars rather than making their core technology better.

<comment><USER5>
          Bing is almost as good as Google

It's easier to be a follower (or ahem, copy search results). Also if I'm going to switch away from Google it must be something truly better (like the difference I experienced when trying out Google after using Altavista).

          and have a better way to handle spam by avoiding it completely

I like Blekko, but avoiding spam is an impossible dream. If there are search engines (or any filtering method, automatic or based on peer reviews) out there with less spam, that's because right now only Google matters. The only way to fight it is to be a moving target, which is what Google is doing.

Unfortunately spammers are also very creative, so it's a tough battle.

<comment><USER4> Re: bing copying Google, it has already been profusely discussed all over the internet and has been shown as nothing more than a PR stunt by Google folks to avoid discussing spam

Re: fighting spam, its a tough problem indeed but so was indexing the ever expanding web. Both Bing and Google seem to have solved the indexing problem fairly well by now.

The issue with the the spam problem is not so much of how difficult it is to solve but the motivation. Clearly the spam generates a lot of cash for Google (and some for bing too) so they are less likely to be motivated to solve it.

Smartest of the minds are at work at Google so I find it hard to believe that they cannot solve it (or even make a very good attempt at it). It's their intent to not solve it and that's the reason I think they are trying to avoid discussing it.

<comment><USER6> _Re: bing copying Google, it has already been profusely discussed all over the internet and has been shown as nothing more than a PR stunt by Google folks to avoid discussing spam_

I'm assuming the downvotes are because of that part.

I keep seeing the argument that "Microsoft didn't answer all of Google's claims," but no supporting evidence. If anyone has some, I'd be curious to see it. (This is a legitimate request: I haven't been keeping score.)

<comment><USER7> At the risk of beating a dead horse, I'll try to summarize. Google believed that queries and clicks on Google search results were being used in Bing's ranking, so we ran an experiment. The experiment confirmed that clicks on Google search results are used by Microsoft in Bing's search rankings.

Things we don't know include:

\- the degree to which those clicks are used in Bing's rankings.

\- whether MSFT does Google-specific processing of the clickstream data they get, or whether clicks on Google are treated the same as some-random- website.com.

\- how long clicks on Google search results have been used in Bing's ranking (months, years, etc.).

Those are open questions that Microsoft could best answer.

<comment><USER6> Thanks for the reply, Matt.

 _Google believed that queries and clicks on Google search results were being used in Bing's ranking_

I thought it was the search _results_ that Google accused Microsoft of stealing. I guess I need to go refresh my memory.

 _Things we don't know include..._

Good summary, thanks. I don't guess anyone is very surprised that Microsoft isn't quick to offer up how it treats a single data stream.

<comment><USER7> "I thought it was the search results that Google accused Microsoft of stealing."

I'm trying to use very precise/neutral language to avoid the "copying/cheating" brouhaha. The crux of the issue is that clicks on Google search results are used in Bing's search rankings.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> Google 2k11 is a _radically_ better product than Google 2k:

1) Crawling is now continuous, and _fast_. You no longer have to wait months for an index update to see a new site on the Googles. Even pages well down the head of the importance distribution get crawled, indexed, and searchable in minutes.

2) Search in Japanese is ridiculously better. Just trust me on this. It has gone from being Hotbot2001 to (English) Google 2005 quality, which is impressive given the natural language problems Japanese throws at you, and the underdeveloped state of the link graph.

3) The Google symbiosis with particular sites on the Internet - <USER2> relevantly to my interests, Wiki and StackOverflow - has successfully incentivized the creation of sufficient basic and in-depth content that it is virtually inconceivable that you can come up with a generic English term and have a total failure on it. (e.g. the top ten for [manatees] will now exceed an eighth grade science report in informational content, which was a very hit or miss proposition in 2000.)

4) Google seems to have gotten over themselves in one critical respect: they used to vocally glorify non-intervention in the SERPs, as if G were the blind oracle of the net. This lead to truly stupid issues like a particular neoNazi group ranking at #1 for "Jew" and Google running an AdWords ad against "Jew" to explain that Google was content neutral between Jews and Nazis and in this case the Nazis just happened to be more loved by The Algorithm so there.

I don't know about internally, but on the basis of external evidence, Google is now willing in search quality on the behalf of users. This is a good thing. (Now if they were humble and cautious about wielding immense power, that would be a good thing, too. I get the persistent impression that they still think of themselves as the wee little underdogs, cuddly academics in the dorm room, rather than being Ma Bell.)

</comment><comment><USER9> I can't speak for others, but my disappointment is with the margin of improvement. I know that results often sucked back in 2001, but with 10 years of progress I would hope to get better results. In the case of "buy domain name" (<http://www.google.com/search?q=buy+domain+name>), 4 of the top 6 results I get are How To articles, not sources to buy domains.

Google is trying to solve a hard problem, and I appreciate that. I just don't have the patience I did 10 years ago to click through all the results to find a legitimate site. If I get a set of results that look like spam, I try to refine the terms, but often just give up.

I know that Google tries to solve this problem algorithmically, but I can't help but think that mixing in a human review would immensely improve results. And by human, I mean a Google employee, not a sure-to-be-gamed community review.

<comment><USER10> I think the issue here is that for you the correct results set would be reputable domain name providers, whereas for many other people, the howto articles are in fact what they're looking for.

Without having personal search switched on, they need to hedge their bets. (note: not having tried it I don't know how effective the personalized search is) In general these populist algorithms may tend to skew the results against techies, or those who 'know what they want', in favour of overall increase in perceived quality.

As far as I know there is a human component, and there is definitely significant weight these days towards CTR as determining quality. These would likely just reinforce the populist ranking factors.

One of the underlooked aspects of Google is the sidebar - using the 'fewer shopping sites' and time-based queries can be a huge help to relevance. I've for years used search modifiers (+, .., - etc.) in my initial queries with an expectation of what Google might expect. A search like "Buy Domain Names" still retains a lot of semantic ambiguity that their algorithm has to wrestle with. You have the tools to fix that yourself.

As such I tend to agree with Matt on this. We've just forgotten how bad it was.

<comment><USER9> _for many other people, the howto articles are in fact what they're looking for._

That's a fair point, but one of the top 6 when I searched was from fourhourworkweek.com, so a whole other level of bias set in on my part. :)

 _We've just forgotten how bad it was_

I still remember wanting to go to hotbot.com and accidentally typing hotbox.com. In a room full of people. So, some of my memories of early engine use are actually quite vivid.

<comment><USER11> Try misspelling a few other well known domains. That tactic is still in full force. I.e. <http://goggle.com/> (hint: this is not google, so fill it in at your own risk).

</comment></comment></comment></comment><comment><USER12> I was skeptical when Google Instant was going to "change the way people search" -- but in early retrospect, I think it really has for me. I now click on one result maximum (and often zero), relying instead on rapid search-term tweaking.

Even just a few years ago I'd expect to click through several results, or heaven forbid to the next page, for what I felt was a difficult query.

Between google instant and other improvements, I am increasingly trained to look ONLY at the top several results. When google doesn't deliver, the results seem worse then they used to when they aren't. I think we're forgetting how we used to have to search.

</comment><comment><USER13> I think Groucho Marx could sum up this article as "Who are you going to believe, me or your own eyes?"

I didn't keep a record of my searches from 2000, but I do remember that I was extremely impressed and satisfied with google back then. That is no longer the case. I am frustrated on daily basis by the results of many searches.

It's not really your fault though. I blame it on mono culture. Google has such a huge hold of the search market that it's probably not even worth SEO peoples time to bother gaming other search engines. There is a whole industry feeding off your success and we all suffer for it.

After the china attacks earlier this/last year google put out a decree that they would phase out windows machines among their employees. I thought it made sense. Windows is too easy a target because of it's success. The same is true in nature and the same is true in search engines.

The best we can hope for is a new search engine that will be insignificant enough in terms of market share to avoid the scammers. As long as google is the dominant search engine it will never get better.

<comment><USER14> > I am frustrated on daily basis by the results of many searches.

I have some speculation on this. Are you sure that your frustration comes from unusually poor results? Could it be that we're so used to things being perfect now, that a page of mediocre results looks like the end of the world?

Look at 2000. No Shazam. If you heard a song on the radio, that was the end of the story. There were some services you could call for $5 and a human would help you identify it. Now in 2011 you press a button and wait 5 seconds.

The web had a _lot_ less information on it. A few hundred million people were connected. Now, in 2011, a few billion people can connect. Huge shift in quantity. Very tough to keep the S/N going strong.

2000: No Wikipedia. Now in 2011 if Wikipedia isn't in your top results you might be upset, but in 2000 you were happy even though Wikipedia didn't yet exist!

So I pose the question to you: Are you absolutely, positively sure that Google's quality has declined and what you are seeing isn't just a side-effect of everything else being so awesome?

<comment><USER15> > Very tough to keep the S/N going strong.

I think there's also a problem that the quality of content that Google has to index is steadily declining. In the 90's, many people were keeping lists of links to good stuff they found on the internet. Google could crawl the links and make conclusions. Now, people hardly do that anymore because they can just google it.

So Google has to employ ever more powerful algorithms to maintain even the same level of search quality. I think here lies an existential threat to the search engines.

</comment></comment></comment><comment><USER16> I'm not sure that a single hand picked result shows much. If Matt has 40,000 results, how about opening them so we can get an idea of how things have changed for queries we care about, and letting people do an independent review?

</comment><comment><USER17> I guess the whole backlash against search results was Content Farms outsmarting Google (atleast temporarily) and distorting results.

Google's search results in 2000 were better than any competitor or Google itself 2 years before. But the same thing can't be said with certainty when comparing 2010's search results to 2008's.

</comment><comment><USER18> Google's spam filter is training the internet to produce better and better content. Over time a rising tide lifts all boats.

In 2000 they were working on not returning results from various subdomains off the same base domain.

This year they're working on content farms.

Next year we won't be talking about plagiarized or duplicated content. We'll be talking about Google giving us content that is superficially different but not unique enough.

Five years from now we'll be talking about how Google is giving us unique, relevant content in the first ten results, but it's not personalized enough. Et cetera.

Just like in the AI world, every time a computer does something we thought only a human could ever do, instead of admitting defeat, you say, "This just proves that chess|poetry|music composition|playing Jeopardy doesn't take _real_ intelligence." The truth is that you're not defining intelligence, you're moving goal posts. And if there's one thing Matt's post shows, it's that we moved the goal posts.

I think Matt likes the criticism. When Google's critics have to move to more and more sophisticated attacks, it shows that Google's facility is growing.

Google is not just raising the bar on the whole internet. You are raising the bar on Google.

</comment><comment><USER19> I think it's about content: Google has transformed the webs' spam gardens by promoting crap sites.

Because Google pushed low substance sites high in their search results, this created a vicious cycle where sites were rewarded for creating mass volumes of inane 'content', which has now become a huge percentage of the web

Google isn't just an observer to the system, search results impact the ecosystem as a whole. A complicated environment is something that is challenging to test and to predict, although that is what Google internally is geared for.

Personally I'm also annoyed by Google's Adsense policies that emphasize inane content: (They without warning blocked me for a post of a painting that included a nude figure).

</comment><comment><USER20> For me personally, google is much, much worse than before. I don't find what I need.

What I think is sad is that alternative search engines still work almost the same as google.

Of course he's going to find spam if he searches for the most spamable content. But that's now what I usually search for - mostly I search for programming related topics or trouble shooting. The old google would give me the right result straight away, the new google gives me some page on the main vendors site that is unrelated.

There are many ways niche search engines could come up to solve this problem, but none of them seem really interested.

<comment><USER21> Google isn't strict enough for programming queries anymore, for example this query yields no results:

        "hello-world" -"hello world"

It ignores the dash even though I'm using quotes.

</comment></comment><comment><USER22> I don't get people who say Google is getting worse myself. If any of you know search engine building so well and aren't working for another search engine (like DuckDuckGo) why not take your expertise to Google and help fix the problem?

<comment><USER23> FYI DDG just uses the bing API and then does some fancy stuff _on top_ of it, so _technically_ they're not a "better search engine" they're just a prettier bing.

<comment><USER24> FYI: no, actually we are a hybrid search eninge. We have our own crawling and indexing that is merged with about 15 external APIs.

<comment><USER23> "hybrid search engine" is just worthless words, what does it even mean? If you want people to stop saying you're just syndicating others data publish the amount of others data you do use, but when I've compared bing.com searches with yours they're _very_ similar whereas google is different, which would imply you're heavily taking from bing, whether or not you crawl yourself...

<comment><USER24> So I'm required to divulge information to stop you from spreading false claims?

You said " _they're just a prettier bing._ " I have ~100K lines of code in front of me right now that has nothing to do with UI, although I think UI is tremendously important and has been a primary focus as well.

Nevertheless, I've talked about all of this at length numerous times -- in videos, on HN, my blog, etc. Bottom line though is you can compare for yourself. We do certainly draw on Bing, but its importance will heavily vary per query and for many queries we'll look completely different. It really depends.

<comment><USER23> False claims? It's what you've said[1][2], albeit a long time ago, but because I can't find any up to date information that you've published (care to link to it?) I can't suddenly know something else.

You're not required to do anything, you can tell me to piss off if you like, but if you want people to claim things and for them to be factually accurate you should publish this information, and if you have then please link to it because I'm struggling to find it.

If you want people to stop spreading misinformation (intentional or mistaken) then spread the correct information, otherwise learn to accept that when you say you use the bing api but refuse to explain to what extent, people will assume your site is entirely powered by it.

[1] [http://www.gabrielweinberg.com/blog/2009/02/thoughts-on- yaho...](http://www.gabrielweinberg.com/blog/2009/02/thoughts-on-yahoo-boss- monetization-announcement.html) [2] [http://www.gabrielweinberg.com/blog/2009/03/duck-duck-go- arc...](http://www.gabrielweinberg.com/blog/2009/03/duck-duck-go- architecture.html#IDComment79453829)

<comment><USER24> Yes, false claims. You claimed that " _DDG just uses the bing API_ " to get results. That is false, and I don't remember ever saying that anywhere because it is false.

Those two posts do not say otherwise. They mention that we use BOSS, which is of course true. The distinction is that we do not _just_ use BOSS by any means.

I actually started out crawling the Web before BOSS even existed, and still crawl quite a lot. As for recent info, look no farther than the DDG FAQ, or my HN comments. This is from just 3 days ago: <http://news.ycombinator.com/item?id=2184873>

Note: no one was requiring you to make any claims, false or otherwise. So I still don't get why I need to divulge information to stop you from making false claims.

</comment></comment></comment></comment></comment></comment></comment><comment><USER25> Could it be that people who are having a hard time finding things on Google now are searching the same way they did in 2000? Google might (and arguably should) be attempting to cater to the way people are searching now, not the way they searched 10 years ago. Don't forget that people using a search engine need to evolve as well.

</comment><comment><USER26> It would be interesting if they set up a Google Retro and let people try and compare the old search and see if they still think it is an improvement over the newer systems.

<comment><USER27> it would be mostly useless. the content today on the web is very different from the content even a couple of years ago.

</comment></comment><comment><USER28> Here is what I've commented on matt's blog :

Well, you're Google is better, but, expectations are higher as is the number of Google employees. So when we see the same content farm again and again during weeks without any action being taken, it annoys. So when blaming "Google when the information just doesn't appear to be on the web at all" is clearly too high an expectation, fixing SPAM issues, be it case buy case, in matter of days instead of weeks is reasonable to me giving the resources you're _supposed_ to have.

So, HNers, my wondering is, is the search spam team under-staffed ? or is the problem elsewhere ?

</comment><comment><USER29> There weren't any content farms in 2000, and SEO wasn't as sophisticated.

Now both are quite pervasive, and having a marked impact on the qualitative assessments of Google search results.

On the quantitative side, Cutts has recently claimed:

 _... according to the evaluation metrics that we've refined over more than a decade, Google's search quality is better than it has ever been in terms of relevance, freshness and comprehensiveness._

How does the increase in SEO, content farms, and other questionable results (scraped content, etc.) figure into Google's evaluation metrics?

<comment><USER7> Good question. We've talked about evaluation a little bit before, e.g. [http://googleblog.blogspot.com/2008/09/search-evaluation- at-...](http://googleblog.blogspot.com/2008/09/search-evaluation-at- google.html)

When we evaluate our quality or a new algorithmic change, URLs can be rated as useful, navigational, etc. They can also be rated as spam. Useful/navigational sites URLs have higher scores, while a spam rating subtracts from the score. If an algorithm change tends to rank higher-rated URLs higher, that's good. If spam tends to rise in the rankings, that's bad.

What our metrics tell us is that Google has gotten better in overall search quality in the last few months, despite also seeing an increase in spam. It's safe to say that now we're putting a lot of effort into the spam side of things to get that back down to the levels we want.

</comment></comment><comment><USER30> I definitely agree that search results in '11 are better than in '00, although it would be cool if more search results from '00 were given since 1/40000 isn't the most representative of examples.

</comment><comment><USER31> Is the post suggesting that the perception of drop in quality is only because google was much much better than other search engines back in 2000?</comment>


How to Break Out of the Children's Menu Trap - <AUTHOR> https://www.nytimes.com/2019/10/23/well/eat/Children-food-health.html <comment><AUTHOR> _the children's menu is where you'll find the least nutritious foods: Chicken nuggets, grilled cheese, pizza and French fries are the standard "kid friendly" offerings.

But that wasn't always the case. In the 1920s and '30s, restaurants offered their youngest patrons their healthiest meals. Back then children's menus were designed to promote healthy eating habits...

"I think a lot of parents feel beaten down and hopeless about these issues"_

Aye. Yep.

</comment><comment><USER2> If at all possible, start at home, with small, simple and healthy meals. Share little nuggets of information with your kids on why you chose today's menu. Be ready to discuss any questions they might have regarding your choices.

<comment><AUTHOR> ... Peel back the duct tape in 1-inch increments with each stalk of asparagus consumed. Shackles may be loosened one quarter turn for each sporkful of broiled halibut verifiably swallowed. The iPad must not be returned until the plate is cleared.

<comment><USER3> meal time doesn't have to be torture.

use it to expand tastes, being open to new flavors/textures will go a long way in those kids futures</comment></comment></comment>


Advice for amateur mathematicians on writing and publishing papers - <AUTHOR> http://research.microsoft.com/~cohn/Thoughts/advice.html

<comment><USER2> This is good general advice for anyone with even a remote interest in writing academic papers (or even just good technical papers).

I just finished teaching a graduate course in computer architecture, and I was shocked at how unprepared they were in this regard. I literally had a student, after submitting a straight-up copy&paste of an ACM-published paper as his own (read: plagiarism), whine "but I've never had to write a technical paper before!". I had others, after having a week to work on an assignment, come to be the day before it's due asking "I can't find any papers.".

These are _graduate_ students. Holy crap. I wish I'd thought to write a brief thing like this and stapled it to the back of my syllabus.

<comment><USER3> At which school? In my experience, the quality of graduate students varies even more widely than the quality of undergrads, and is more closely correlated to the quality of the school.

<comment><USER2> I'd rather not malign the school. Let's just say they're overrun by populations of students that are underqualified - a condition I think is common among schools not in the "top tier".</comment></comment></comment>


Scans reveal intricate brain wiring - <AUTHOR> http://www.bbc.co.uk/news/science-environment-21487016

<comment><USER2> Raw data!

[http://www.humanconnectomeproject.org/2012/01/first- public-r...](http://www.humanconnectomeproject.org/2012/01/first-public- release-of-3t-connectom-scanner-data/)

<comment><USER3> It's unfortunate that you have to apply for access. Why force people to jump through hoops? Expose this data.

<comment><USER4> Maybe they do not want to happen to the public image of brain research what happened to the public image of AOL after AOL released to the public a few tens or hundreds or megs of search queries for research purposes.

<comment><USER5> You mean someone will combine this with the netflix database and will proceed to de-anonymize this based on what they were thinking during the scan?

The AOL data release was a total fiasco in that it wasn't anonymized properly and because they were not aware of the state of the art.

And it is not like they are not releasing it, they are releasing it! Just not on an instantaneous download.

As for AOL, I'm quite grateful they did what they did, I got quite a bit of mileage out of their data.

<comment><USER4> >You mean someone will combine this with the netflix database and will proceed to de-anonymize this based on what they were thinking during the scan?

There is a saying on the web site _Less Wrong_ that "nobody knows what science does not know." In other words, people tend to be too quick to leap to the conclusion that a particular feat involving science is impossible.

I also note that you have no personal stake in the question. If your scientific career or your ability to continue to win research grants were at risk, you might not be so confident in your belief that it would be impossible for anyone to misuse the data in a way that harms your reputation or the reputations of people you depend on or care about.

(There is a small chance I could be wrong in inferring that that is indeed your belief, in which case, ignore the above.)

Note that I am not asserting that the researchers _should_ limit distribution of the data or that doing so would have net positive effect on society. I am just saying that is a rational decision for researchers who care about their careers to make.

<comment><USER5> I think that being careful with medical data is an excellent idea. At the same time I can't see any harm coming from releasing this particular data as long as it has absolutely no meta data associated with it. Just the raw scans should be enough to satisfy anybody's curiosity. If they need more for whatever reason then they should have their credentials and needs checked on a case-by-case basis.

Mis-use of such data would require at least a hypothetical scenario, and stating that would bolster the reason why the data was not released without restriction.

But better safe than sorry.

</comment></comment></comment></comment></comment></comment><comment><USER6> "Only 50 such scans have ever been done."

"The aim the $40m programme is to map the entire human neural wiring system by scanning the brains of 1,200 Americans."

<comment><USER7> Look at what happened with DNA sequencing[1]. Linear extrapolation isn't always the best predictor of progress.

[1] <http://www.dnasequencing.org/history-of-dna>

</comment></comment><comment><USER8> Awesome, this kind of things really amaze me. I'm just curious what are the possibilities once they have everything scanned. Yes we can probably cure some disease or fix some stuff, but that's kind of old news, I am more interested in what they can actually do. Can they upload new language into your brain? or what else?

<comment><USER9> Perhaps you just worded that sloppily, but why do you consider curing "some disease", like, say, schizophrenia or Alzheimer's, would not be "actually doing" something?

<comment><USER8> Yes I did probably word it wrong, of course to cure these diseases is awesome, but we kind of know what the potential is there, I am more interested in what other potentials, apart from diseases, are there

</comment></comment></comment><comment><USER10> This TED video is a good explanation of the concept of a "connectome" for the layman: <http://www.ted.com/talks/sebastian_seung.html>

The simple connectome of c. elegans has been mapped, and there is an open source project to use this dataset and other data to run a full simulation of the worm in a virtual environment: <http://openworm.org>

</comment><comment><USER11> Pretty cool.

I wonder what fraction of the wiring in the brain these represent.

I guess these are only the largest "pathway" structures?

</comment><comment><USER12> Can't imagine what those few thick rendered wires represent in a brain. Do they stand for signals by strength? Why can't we see how they terminate? Why are the bundles seemingly unconnected with one another?

I didn't get any insight from that graphic. For $40M I hope they have some better results than that.</comment>


The Supreme Court takes on costume copyright - <AUTHOR> https://www.publicknowledge.org/news-blog/blogs/cosplay-goes-to-the-supreme-court/ <comment><AUTHOR> _1\. Under the Copyright Act, a "useful article" such as a chair, a dress, or a uniform cannot be copyrighted. 17 U.S.C. § 101. The article's component features or elements cannot be copyrighted either, unless capable of being "identified separately from, and ... existing independently of, the utilitarian aspects of the article." Id. Circuit courts, the Copyright Office, and academics have proposed at least nine different tests to analyze this separability. The Sixth Circuit rejected them all and created a tenth. The first question is:_

 _What is the appropriate test to determine when a feature of a useful article is protectable under § 101 of the Copyright Act?_

[http://howappealing.abovethelaw.com/13755846_1-StarAthletica...](http://howappealing.abovethelaw.com/13755846_1-StarAthletica- cert.petition%28as-filed%29.PDF)</comment>


Fork our company blog - <AUTHOR> https://github.com/Topguest/topguest.github.com#readme

<comment><USER2> You put your company blog on github? I don't see how this is original, and given your history, it looks to me like you're spamming your blog. Flagged.</comment>