Kafka Removing Zookeeper Dependency - <AUTHOR> https://www.confluent.io/blog/removing-zookeeper-dependency-in-kafka/ <comment><USER2> Finally! Travis Jeffery did this years ago in Jocko and also solved my other beef with Kafka at the same time by building it in Golang.

[https://github.com/travisjeffery/jocko](https://github.com/travisjeffery/jocko)

I've always found things built on JVM are are PITA to deploy (especially when using SSL) so the single Golang binary is a welcome advancement.

It's all the good things about Kafka (concept, API, and wire protocol) without all the crap (zookeeper dep, JVM foundation)

<comment><USER3> Agreed. We're working on this exactly - [https://vectorized.io/redpanda/](https://vectorized.io/redpanda/)

10x faster. API compat. No jvm.

I also know of other private impls of it. Just makes sense.

<comment><USER4> Nice job. I have always wondered why performant big data systems are written using resource hungry JVM.

<comment><USER5> Because the JVM is fast. And it's not resource hungry - your program might be resource hungry. Bad desktop Java programs misled a whole generation of programmers about the JVM.

Look at stuff like LMAX. Java can be lightning fast.

<comment><USER6> It's fast if you measure throughput or median latency but it tends to have pretty monstrous worst case latencies due to GC.

<comment><USER5> Depends. Java's GC can be tuned an infinitum. It's not a simple task and requires knowledge, but that's the beauty of it: if you need low-ish latencies, then you can tune the GC to target that instead of throughput. For example we're using relatively large heaps (partly due to inefficiencies in our code), but we still want to stay under 500 msec/request or so. So we told the G1 GC to target 150 msecs per collection, and then adjusted our heap size / code accordingly. It works well.

If you need really hard limits on collection then that's a tricky problem, but that's also tricky when you're managing memory yourself.

<comment><USER7> Once you start talking hundreds-to-thousands requests/second, 500ms is an incredibly long time and you're well past simple tweaks to GC. Tuning GC to a high degree is non-deterministic black magic, which is not what you're looking for at that point.

Simple tweaks can go a long way for a lot of developers, but GC performance has been a problem at the last 3 organizations I've been at - and I'm not in the valley or at a FAANG - so it isn't exactly an uncommon scenario for developers.

<comment><USER8> The JVM now ships with ZGC, a low latency GC targetted at exactly that. [https://wiki.openjdk.java.net/display/zgc/Main](https://wiki.openjdk.java.net/display/zgc/Main)

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> I couldn't be more excited about this! Having finished a Kafka project about a year ago, we had so many Zookeeper production and test environment issues that it was the running joke to check Zookeeper first if anything went wrong.

Honestly Zookeeper in theory is a great idea: Having a centralized service for maintaining config info saves a lot of heartache when dealing with an open source distributed systems project. But in practice, I've never had a smooth experience getting Zookeeper to run consistently, especially with Kafka.

For sure, part of it is that we treat it like oxygen, in that if it's gone for a few seconds everything just dies. But having dealt with similar systems both proprietary and open source, my opinion is that Zookeeper just hasn't risen to the challenges of its users in the past 5 years. If the next generation of software architects want to use open source streaming or distributed systems, Zookeeper needs to be rewritten or removed.

Also shout out to the confluent.io team: I never paid for your enterprise license, but without your blog posts, docker images, or slack room, I never would have been able to get Kafka working. Thanks again!

<comment><USER10> i am curious to know why people expect a raft library to be more reliable if embedded inside the kafka controller versus running inside a service like Etcd.

in the end broker will do RPC to a service (kafka controller/ etcd) and this service will use raft to replicate the state.

It should be exactly the same. And if anything knowing which node are running the raft algorithm help you be more careful with rolling restart and upgrade.

<comment><USER11> It might sound kind of trite, but it has less to do with the replication algorithm and more to do with the fact that Zookeeper's source is quite complicated vs, say etcd, so there are more opportunities for subtle bugs to appear. I encourage you to look at the bug section of the change log for zookeeper, and also the feature list of ZK vs etcd.

Also, etcd powers many critical open source projects, so there are many institutional eyes that actively contribute to its improvement. IME if we ever encountered an issue at work with ZK, we found it impossible to trace it down to a bug that we could fix and upstream. Etcd's been easier in this regard.

<comment><USER12> Agreed, etcd is rock solid.

With regards to Kafka, it's probably easier and more robust to add their own consensus layer rather than switching to etcd - Kafka is already a distributed system built by a team of distributed systems engineers. It makes sense for them to build their own consensus, deeply integrated with the replication mechanism, rather than relying on an external database.

</comment></comment></comment></comment><comment><USER13> I expect Pulsar to stay with Zookeeper. Kafka currently stores topic and partition info on ZK ( [https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data...](https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper) ), which can get to be a lot of data. But I think Pulsar only stores server names and basic config info on ZK ( [https://pulsar.apache.org/docs/en/administration-zk- bk/](https://pulsar.apache.org/docs/en/administration-zk-bk/) ), which is much more managable.

<comment><USER14> _But I think Pulsar only stores server names and basic config info on ZK ([https://pulsar.apache.org/docs/en/administration-zk- bk/](https://pulsar.apache.org/docs/en/administration-zk-bk/) ), which is much more managable._

Unfortunately, this is not correct. BookKeeper stores a lot of information in ZookKeeper. By extension, Pulsar (which is based on BookKeeper) also stores a lot of metadata there as well.

For example, from the BK documentation ( [https://zookeeper.apache.org/doc/r3.3.6/bookkeeperOverview.h...](https://zookeeper.apache.org/doc/r3.3.6/bookkeeperOverview.html) ):

 _An application first creates a ledger before writing to bookies through a local BookKeeper client instance. Upon creating a ledger, a BookKeeper client writes metadata about the ledger to ZooKeeper. Each ledger currently has a single writer. This writer has to execute a close ledger operation before any other client can read from it. If the writer of a ledger does not close a ledger properly because, for example, it has crashed before having the opportunity of closing the ledger, then the next client that tries to open a ledger executes a procedure to recover it. As closing a ledger consists essentially of writing the last entry written to a ledger to ZooKeeper, the recovery procedure simply finds the last entry written correctly and writes it to ZooKeeper._

</comment></comment><comment><USER15> I never know where I sit on stuff like this. On the one hand if you're confluent I think it makes total sense to own this part of your infrastructure. Especially if it lets you improve your operability story.

On the other hand I feel like projects should try and use open source "building-blocks", like etcd and zookeeper, when building their distributed systems. Not only does this help iron out correctness bugs, but it also means that more people are familiar with the quirks, limitations, requirements etc.... of these tools. For example, I think I would be frustrated to hear that K8s were implementing their own raft.

<comment><USER16> Counterpoint, if I need to deploy technology T to solve problem P, I don't want to have to also deploy a flotilla of support technologies because modularity or whatever. ZooKeeper was always an implementation detail of Kafka, the fact that you had to manage it separately was an abstraction leak that I'm happy to see fixed.

<comment><USER17> I feel like this just argues that ZooKeeper should be a library that can be embedded in the servers of other projects (even if it is conceptually a separate server in the same memory space listening on its own ports; but like, it is otherwise entirely hidden inside of the other program and is entirely managed and configured by it also).

<comment><USER10> 100 % agree we need good (etch/zookeeper) as a library.

I wrote something similar but that provide only (lock/lease) using paxos.

The problem with raft ETCD and zookeeper replicates state machine design is that : \- machine leaving and joining the ensemble dynamically is something hard to do correctly \- optimal quorum size is no more than 5, you can setup other node as observer but it's hard to decide which of 1000 node should be in the quorum ...

<comment><USER12> The etcd server can be embedded in other projects, and their Raft library is used by plenty of other projects like CockroachDB.

<comment><USER10> yes it can but it's a real pain compared to hashicorp version. I really hope they clean up the library to have a simple API.

This way it could become the standard for GO and we can stop wasting effort on multiple raft implementation.

</comment></comment></comment></comment></comment></comment><comment><USER18> This is cool. One thing I love about Cassandra is that you don't need Zookeeper, its just a bunch of homogeneous nodes.

<comment><USER19> Same with Elastic Search.

</comment></comment><comment><USER20> When will this be actually released and available for use?

I'd love to ditch Zookeeper (the weak part that falls over more regularly) in our current Kafka cluster.

</comment><comment><USER21> Here's the proposal in full.

[https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A...](https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self- Managed+Metadata+Quorum)

</comment><comment><USER22> Finally! Having had to deal with properly setting zookeeper up on multiple mesos clusters was so painful and recovering from outages via s3 backups so excruciating.. I feel somewhat ashamed that in two out of five cases we simply went with the managed Kafka by AWS because the team didn't feel confident enough to maintain the cluster.

<comment><USER11> Honestly this is probably the right thing to do. My cynical conspiracy is that the ZK dep was organized by cloud vendors to keep customers coming :)

It's been at least a year and a half since we've had severe data loss on our homebrew Kafka cluster, but after the first couple you never look at Kafka+Zk the same way... both iirc were due to leadership election bugs that had been reported many months ago with no progress on a solution.

I have no idea how AWS internally puts up with this. I wouldn't be surprised if they'd replaced the ZK dependency internally years ago.

</comment></comment><comment><USER23> I might be out of my realm here but would it be more efficient for the Kafka brokers to use a gossip protocol to distribute the metadata?

<comment><USER24> Gossip protocol is useful for propagating node failures or events that are fine with being eventually consistent.

It cannot be used for leader election as these events are time sensitive and needs to be consistent across the cluster within a short duration, that is the reason we have raft & paxos.

<comment><USER23> Ah I see, thank you for the clarification

</comment></comment></comment><comment><USER25> Amazing that people chose to operationalize Kafka with this completely unnecessary dependency. Good riddance!

</comment><comment><USER26> Well... it's just Kafssandra now.</comment>


ZeroDB, an end-to-end encrypted database, is open source - <AUTHOR> http://blog.zerodb.io/zerodb-open-source-announcement/ <comment><USER2> This is fascinating. Awesome work!

An encrypted server-side database with client-side decryption definitely seems like a great building block for private, secure applications! However, my ideal building block for the data-layer would take one step further: by only storing data on the client, and be able to synchronize between separate clients in a fully distributed manner.

This would open up the door to purely client-side applications with zero- knowledge sync clients that can be _optionally_ hosted on servers to improve availability without any compromises to privacy. It also has the potential to offer better-than-centralized UX due to the fact that all data is available locally, so there is no network roundtrip to be concerned with. A centralized solution like ZeroDB definitely still has it's place though, especially in applications that involve sharing and collaboration between multiple users, and in applications where the size of each user's dataset is too large to be managed fully locally.

My previous project, Toc Messenger [1], used a custom encryption layer over remoteStorage [2] to achieve something close to this, but remoteStorage is not a fully distributed protocol, and has to rely on the existence of a centralized storage server to function. For my next project, I'm hoping to experiment with something like Swarm [3] for the data layer instead, which uses CRDTs for conflict resolution and supposedly supports direct P2P sync in the upcoming 1.0 release [4].

[1] [http://toc.im/](http://toc.im/)

[2] [http://remotestorage.io/](http://remotestorage.io/)

[3] [https://github.com/gritzko/swarm](https://github.com/gritzko/swarm)

[4] [http://swarmjs.github.io/articles/2of5/](http://swarmjs.github.io/articles/2of5/)

<comment><USER3> Oh wow, thanks for the links, will need to go through that!

We've seen gundb [[http://gun.js.org/](http://gun.js.org/)] keeping data distributed between clients. Also, I'm even thinking if it is possible to run ZeroDB on top of SpiderOak's Crypton (which is inspired by ZODB) :-)

</comment></comment><comment><USER4> a very cool concept for doing a lot of projects where users need shared db.

</comment><comment><USER5> I wish i will have NaCL-sompatable encryption that can be used to easily implement your own clients.</comment>


Ctypes.sh – A foreign function interface for bash - <AUTHOR> http://ctypes.sh/ <comment><USER3> Neat...and on a somewhat similar note (bash enable -f hacks), some bash FUSE bindings I wrote a few years back: [https://github.com/zevweiss/booze](https://github.com/zevweiss/booze)

I initially wrote it basically just for giggles, but was rather pleased a few months ago when I realized I could use it for something I actually needed, and it was just the right thing (presenting a mount point that acted as a view of the differences between two rsnapshot-style hard-link trees -- only took a few dozen lines of very simple code).

</comment><comment><USER4> The problem with this project is that somebody might use it. I argue that if you need to do something in bash that involves more than a sequence of commands, including a simple if statement, then you should switch from bash to a proper scripting language (python, perl, ruby, nodejs, take your pick.)

<comment><USER5> I disagree. Shell scripts are great for automating system tasks — for sysadminy type stuff, including tasks with high complexity with nested loops and conditionals.

A scripting language like those you mentioned is great for if you need to run your script on systems you don't control, or you need it to be cross platform, or you'd describe what you're doing as creating software rather than automating tasks.

No scripting language can ever integrate with the host system quite as well as shell scripts can. Sure, shell scripts make it easy to shoot yourself in the foot, but then can't you say the same of at least perl and ruby?

I don't really see many good use cases for this library though. By the time you'd use it you should probably move on to something else.

<comment><USER4> Can you give a shell script example of something that perl/python/ruby/nodejs can't do?

<comment><USER6> > Can you give a shell script example of something that > perl/python/ruby/nodejs can't do?

Install perl/python/ruby _and_ nodejs -- requiring just the shell to be installed?

Snark aside (it's not _only_ snark - pretty much every system will have something like a posix shell), proper posix portable shell is hard - it's an old gnarly language -- but it's what we've got. And with a bit of discipline and good practices -- it doesn't have to be bad. That said, a lot of real- world shell scripts _are_ bad.

I tend to agree with the overall sentiment; shell isn't a great language. As soon as you start to mix awk (which awk is that, do you need GNU awk?), sed and perhaps a bit of egrep (or grep -E -- are both available? Does it accept only BSD-style parameters?) -- one should consider moving "up".

And for eg: setting up a python package/program -- I'd generally prefer a python script -- hopefully one that handles different file-paths (eg: / vs \ ) and other cross-platform stuff. If you already depend on python, why _add_ dependency on shell?

<comment><USER4> > Install perl/python/ruby and nodejs -- requiring just the shell to be > installed?

I just meant using any one of them, not all of them. And you can pretty much always rely on python and perl being installed.

But my point is that if you're doing something other than a one-off thing, then it belongs to some project and you're probably going to commit that script to that project's codebase. That means it has to be maintained. Do future you and whoever else has to maintain the project a favor and use one of the popular scripting languages that has reasonable syntax and semantics.

<comment><USER6> Many minimal distributions have only the shell installed - which make it (still) relevant for provisioning/bootstrap etc.

Personally I'd much rather maintain a shell script than a perl script - <USER2> that's just because I know shell better. Maybe shell is the first language people would program without learning it (js being the second)?

</comment></comment></comment></comment></comment></comment><comment><USER7> Some earlier discussion, people porting to FreeBSD and what not: [https://news.ycombinator.com/item?id=9959628](https://news.ycombinator.com/item?id=9959628)

</comment><comment><USER8> What could go wrong? /s

All sarcasm aside, a very interesting idea. I'm not sure what the proper use case is, but I'm sure someone will love this.

<comment><USER9> The best use-case I can think of is using bash as a REPL for C libraries. Many times in the past, I've made library calls that either misinterpreted the parameters or the result value. I would have loved the ability to prototype those calls in bash until I understood them enough to call them properly.

<comment><USER10> You can use gdb as an REPL for C.

<comment><USER11> This blew my mind the first time some one showed me, and is one of the reasons I currently prefer C to other mainstream compiled languages.

Edit: I suppose it is possible to get some version of a repl with C++ (Cling?), Java (Beanshell, Java 9?), C# (Mono, but not currently .Net?). But each of those seems generally a bit harder to get access to than simply calling gdb on a binary with debugging symbols enabled.

<comment><USER12> C# has REPL in the immediate window when the debugger is running; so it's kind of like using GDB as a REPL for C. Although it is somewhat limited in what you can do.

<comment><USER11> That's actually really good to know about.

Visual Studio's a little bit heavyweight, but if you're programming C#, there's a good chance that's what you're using anyway.</comment></comment></comment></comment></comment></comment>


Show HN: A directory of 50 COVID resources - <AUTHOR> https://stephsmith.io/covid-resources <comment><AUTHOR> I made this because I kept stumbling upon resources and thought "more people should know this exists". So, I tried to make a central directory of important resources, whether it be places to access important data, job openings and trackers, tools for people or businesses trying to access funding, and of course, resources for people currently with the virus.</comment>


Deploying Brotli for static content - grey-area https://blogs.dropbox.com/tech/2017/04/deploying-brotli-for-static-content/ <comment><AUTHOR> It may be interesting to look at a web content benchmark with the best gzip compatible compressors and peak memory usage:

[https://sites.google.com/site/powturbo/home/web- compression](https://sites.google.com/site/powturbo/home/web-compression) Only facts and numbers, no rumors, no speculation, no hype,...

</comment><comment><USER2> I wonder who will win in the long term between Brotli and zstd. [http://facebook.github.io/zstd/](http://facebook.github.io/zstd/)

<comment><USER3> ZSTD.

It is superior to Brotli in most categories (decompression, compression ratios, and compression speeds). The real issue with Brotli is the second order context modeling (compression level >8). Causes you to lose ~50% compression speed for less then a ~1% gain in ratios [1].

I've spoken to the author about this on twitter. They're planning on expanding Brotli dictionary features and context modeling in future versions.

Overall it isn't a bad algorithm. Brotli and ZSTD are head and shoulders above LZMA/LZMA2/XZ. Pulling off comparable compression ratios in half to a quarter of the time [1]. They make GZip and Bzip2 look outdated (which frankly its about time).

ZSTD really just needs a way to package dictionaries WITH archives.

[1] These are just based on personal benchmarks while building a tar clone that supports zstd/brotli files [https://github.com/valarauca/car](https://github.com/valarauca/car)

<comment><USER4> What use case do you have in mind for packaging dictionaries with archives? There is an ongoing discussion about a jump table format that could contain dictionary locations [1].

[1] [https://github.com/facebook/zstd/issues/395](https://github.com/facebook/zstd/issues/395)

<comment><USER3> For large files >1GiB a library + archive is often smaller then the archive on its own.

<comment><USER4> How are you compressing the data?

I would expect a dictionary to be useful if the data is broken into chunks, and each chunk is compressed individually.

If the data is compressed as one frame, I would be very interested in an example where the dictionary helps.

</comment></comment></comment></comment></comment><comment><USER5> It is interesting that Japanese, Russian and Thai benenefit more (30 %) from brotli, than latin languages (25 %). This is because of the utf-8 context modeling in brotli.

<comment><USER6> The first draft of the article actually had that reason, but there is also a strong correlation between the size of the dict (these dicts are almost 1Mb, while other languages are closer to 500kb) and compression ratio improvements. Therefore I've played it safe and attributed it to the window size.

Though for languages like Korean and Chinese (whose size is more inline with latin languages) we see 27.5% improvement, which is most likely due to context modeling.

Therefore I assume ratio improvement is split ~50/50 between these two. It was easy to verify that by compressing data with `brotli --window 15` and comparing ratios there, but I was lazy there. I'm sorry.

PS. I've also skipped NFC/NFD part of the post which is very interesting for Korean, where NFC normalized text occupies 30% less space. It also gives additional ratio 5% for brotli and 15% for gzip.

<comment><USER5> There is no Thai or Korean in the dict. The total size of the dict (including all languages) is 120 kB.

<comment><USER6> By "dict" I meant the data we are compressing: these are basically dictionaries for "English to X" translation.

What was saying is that there is a strong correlation between size of the data I was compressing and compression ratio improvements over gzip.

</comment></comment></comment></comment><comment><USER7> You can also deploy Brotli for dynamic content: at setting 4 it's both faster AND compresses more than gzip.

[https://certsimple.com/blog/nginx-brotli](https://certsimple.com/blog/nginx- brotli)

</comment><comment><USER8> I use brotli on my personal website, and usually notice 15-20% smaller files than the gzip equivalent. It's a bit of a pain to install a brotli compressor but then very easy to add a single build step to compress all static assets.

</comment><comment><USER9> Does someone know if IE11 will get brotli support? Or will IE11 only receive security patches?

<comment><USER10> Security patches only. IE11 exists on Windows 10 only for the benefit of enterprises that can't drop horrible legacy technologies like ActiveX. All new feature development is on Edge.

</comment></comment><comment><USER11> Seems like most of the image optimizations are undone by WordPress/CDN, saved >60% on the xkcd-styled charts.

</comment><comment><USER12> Why are their graphs xkcd-style?

<comment><USER13> I've seen it suggested before to imply a bit of a margin of error in the numbers. I personally like it a lot

<comment><USER14> Haha, I don't think the previous poster was referring to the error bars. That's not "XKCD-style", it's just standard practice in statistics: [https://en.wikipedia.org/wiki/Error_bar](https://en.wikipedia.org/wiki/Error_bar)

I think what he was referring to was the fact that the graphs appear to be hand drawn.

<comment><USER15> > I think what he was referring to was the fact that the graphs appear to be > hand drawn.

Yes, and so was your parent comment. Some people have suggested that the hand- drawn appearance communicates imprecision. For example:

> The rough, seemingly hand drawn nature of the graph provides a visual hint > as to the imprecision of the results.

[https://www.chrisstucchio.com/blog/2014/why_xkcd_style_graph...](https://www.chrisstucchio.com/blog/2014/why_xkcd_style_graphs_are_important.html)</comment></comment></comment></comment>


Growing evidence that the universe is connected by giant structures (2019) - <AUTHOR> https://www.vice.com/en_us/article/zmj7pw/theres-growing-evidence-that-the-universe-is-connected-by-giant-structures <comment><USER2> There's a theory out there that our universe is a pocket universe inside of a supermassive black hole. My relativistic gravity is pretty rusty and was shaky to begin with. If that theory isn't pure fantasy, and we were a pocket universe, _and_ large masses were spinning above the event horizon, we'd experience forces from those objects, right?

I'm still kinda fond of the Flatland theory myself. Us poor 4 dimensional beings can't figure out how to build tools that can take 5 dimensional measurements, so we see weird behavior we can't explain.

<comment><USER3> What if our sense of scale is off and the universe isn't curved we're just orbiting a black hole 10x the mass of the visible universe. It's not expanding that's us travelling through the gravity well at 99% the speed of light relative to it.

<comment><USER4> My pet hypothesis is that the universe we experience is _on the surface_ of the singularity of our black hole. This neatly explains some things:

\- The Great Attractor would simply be the volume deeper in singularity. The cone of influence for the entire universe tends towards the center of our singularity.

\- Dark energy, and the varying quantity of it, would simply be the amount of matter falling into our black hole. It would naturally start as a large amount (consisting of our parent star/mass) and later see variation as matter falls into it.

\- Dimensionless physical constants may have been fields in the parent universe. As an infinitely small point, our universe singularity would resolve a continuous gradient to a single value. Our child universes would have a single value of gravity, for example, which could change over time (but would be constant across the universe as they changed).

I've got a huge list of interesting consequences, but as an armchair physicist it's probably all quack science :)

<comment><USER5> Maybe you already know all this, but to other curious readers:

The great attractor is about 200 Mly away. Directly behind it, 600 Mly away from us, is the Shapley Supercluster, towards which the great attractor is itself moving. In the opposite direction is a underdensity, called the "Dipole repeller", and between the two, where we are, is a "dark flow".

[https://en.wikipedia.org/wiki/Shapley_Supercluster](https://en.wikipedia.org/wiki/Shapley_Supercluster)

[https://en.wikipedia.org/wiki/Dipole_repeller](https://en.wikipedia.org/wiki/Dipole_repeller)

[https://en.wikipedia.org/wiki/Dark_flow](https://en.wikipedia.org/wiki/Dark_flow)

<comment><USER6> I welcome our dark lord, Cthulhu. May his comings bless us for generations to come.

If i could write i would write a novel about the great flow. That was awesome and terrifying to read.

<comment><USER7> Someone is probably going to vote your comment down on the premise that Cthulu is unhelpful to the conversation. They shouldn't: it's not any more fantastic than the various laymans' pet theories here which have, for some indecipherable reason, been upvoted to the top of the discussion, despite having no connection to real physics whatsoever.

<comment><USER6> In a more serious note; i actually think the imagination and fantasy they is born in alot of people after reading about the mysteries of the universe is responsible for a many physics grad.

Taking things /seriously/ is, imo, what keeps people out of stem. Stemmers are, if anything, fantasy rich and colorful people, physicists in particular!

Im glad not many people chose to downvote me too :)

</comment></comment></comment></comment></comment></comment></comment><comment><USER8> I vaguely recall that the observable universe looks a lot like foam. And even more vaguely, that maybe this reflects inflationary expansion of random fluctuations in the early universe.

So maybe this apparent connectivity over implausible scales just reflects the fact that stuff used to be much closer together.

Edit: There's a great sequence on inflation in Takashi Miike's "God's Puzzle".

<comment><USER9> We used to think so, yes. But our telescopes have improved _a lot_ since then, and we're now able to see with much more detail, our 3D positioning is better too.

We've recently observed that the universe, at the largest scales of what's observable to us, looks like this: [https://i.stack.imgur.com/lFnDf.jpg](https://i.stack.imgur.com/lFnDf.jpg)

More picture and explanations on this short SE thread: [https://physics.stackexchange.com/questions/198138/is- most-o...](https://physics.stackexchange.com/questions/198138/is-most-of-the- matter-in-the-observable-universe-within-galaxies)

I'll let you ponder how much resemblance this bears to other kinds of structures...

<comment><USER8> That's what I meant by "foam". And remember that those are ~2D slices. In 3D, there are voids, with thin intervening walls, and stringy stuff where walls intersect.

It's dark matter that probably defines that structure. And so that dark matter could be moving, in coherent patterns at extremely large scales. But there's nothing necessarily supraluminal about it. It could just be moving in the same patterns that it did when it was hugely smaller. But then, I have zero clue about how conservation of angular momentum works during inflation.

If that were the explanation, we wouldn't expect to see perturbations in cosmic scale motion propagating supraluminally.

<comment><USER9> > That's what I meant by "foam".

Oh sorry I didn't catch that (not a native speaker). When thinking of "foam" I see a uniform, homogenous noisy structure and I was pretty sure this was the name given to our former such view of the universe. My bad! (although I'd say, if "foam" is used to described these pictures, it's a pretty bad choice of adjective by scientists imho. "Galactic tissue" or "tissue-like walls" makes more sense to the layman imho).

Indeed, it's not "probably" dark matter, it _is_ by the very definition of what we call dark matter. It's important to keep that in mind, when we name unknown phenomena it's just a 'placeholder' so to speak. Like X. So X explains the motion of galaxies, and we intuitively called X dark matter. Assuming there is an X, then there is dark matter (what it actually _is_ is another question, could be "white energy" or fluffly creatures from the 6th dimension for all we know, but we call it dark matter for now).

So based on this, I always wonder: is it our observation or maybe interpretation thereof that's flawed? Are there 'effects' that make us see and infer things that are slightly 'off' maybe? (like the discrepancy with dark energy measurements for instance, it appears recently that _there might not be accelerating expansion_ and the increasing redshift is due to errors in our calculations, i.e. we created "dark energy" out of a mistake and now it's possibly going to "disappear" as an idea because it's not needed anymore to explain anything).

As for inflation etc. These are also mathematical concepts that fit the observation, the clues; it however might not be the only possible model that works, we also might not have a correct interpretation (layman words) for it. Like we've got QED/QCD and we're still unable to describe in layman terms because it's illogical, paradoxical. Sometimes our idea of "simplest interpretation" is just that, an idea, and reality is a bit more complex, even if the simple equations may work as terms simplify (see how we "add" zero- value or unit-value arguments to some equations in physics to solve the math, it really begs the question if the "thing" you add actually exists, speaks of a real phenomena). I know Einstein was pretty unhappy about the picture painted by Λ... ;-)

About supraluminal anything (I assume you mean speed > _c_ ), I personally don't think it wise to postulate things that physics rule _out_ explicitely. Inflation doesn't do that though, since it's spacetime expanding and that's not constrained by c.

[1]: [https://youtu.be/blSTTFS8Uco](https://youtu.be/blSTTFS8Uco)

<comment><USER8> I'm no physicist, and foam is just my analogy. I could have also said bread, because it's basically foam. Or even soap bubble foam. And yes, biological tissue, which is fundamentally an emulsion of oil in water.

That's a great point about dark matter, that it's just a placeholder. There could be stuff that _only_ interacts through gravity. Or it could just be that there's a constant gravitational force that dominates at large distances.

TFA describes evidence that such properties as galactic rotational axes are correlated on extremely large scales. And if there are connections, they must be something that we can't see. So "dark matter" is the obvious candidate.

The other issue is how stuff can be correlated over distances that are too great for forces at the speed of light to correlate them quickly enough. Although TFA doesn't explicitly say that, I can't think of what else about those connections is so surprising. Given that the large-scale structure is so well known.

That's what got me thinking that inflation could account for it. I was thinking of stuff like shedding of vortices from rapidly moving objects in fluids. As they expand, features that arose at small scales in the boundary layer expand to far larger scales. Indeed, to scales that are large enough to be puzzling, if we had observed them in isolation.

That's simple minded, I know. But then, I'm no physicist.

<comment><USER9> I'm no physicist either. Maybe I wish I were, though. And no, your ideas here are everything but simple-minded, imho. On the contrary, being a non-expert lets you play with things with much more freedom than otherwise (like sci-fi authors), and then actual physicists can take over, sort out the testable and go with their own gut feeling.

(note: I assume TFA = Trend Filtering Algorithm, please correct if I'm wrong).

> galactic rotational axes are correlated on extremely large scales

I think "correlation does not imply causation". While this is a plague in economics, psychology, sociology, biology (non-exact sciences), it also applies in physics. We really don't know in this case, we just observe what you said but "correlation" is bias that typically hinders actual research, experiments — it's the idea that you'll see what you want to see and even design the experiment to show it, which is a very slippery slope, basically a form of "confirmation bias".

I want to stress that I in fact agree with you, in sentiment and analysis, but I'm just voicing my internal "devil's advocate" voice here: mental models and frameworks that are wrong, and not only make my intuition thus false, but deeper than that prevent me from seeing other solutions, from 'forking' every step of the way according to new data.

I guess I'm prefacing my conclusions here:

\- there is no "dark matter", there is X that explains _a bunch of discrepancies_ between theory and observation. Each such discrepancy could be explained by a different X (Y, Z...) or a sub-part of X (like mass relates to gravity, but also density, thus heat profile, etc).

\- there is always a "fundamental correlation" between literally _everything_ (aren't each and every organ of our body, every inch of this Earth the literal 'children' of supernovae and neutron stars explosions and black hole cataclysms?)

The hard part is to find first-degree (i.e. direct) relations, of the kind described by physics equations. It's not always discrete and binary, but it can be done — for instance I can tell you that "hunger" among the population is as certain a path as it gets to revolution. Not sadness, not lack of freedoms, not violence physical or mental, not even cataclysms have a firmer grasp on rebellion than the empty stomach of our children. It's as direct a path from A (hunger) to B (revolution) as you can get, statistically in all of history (you have _days_ before people rebel, a very few weeks at most).

The alignment of galaxies are obviously related to something, which might yet leave individuals otherwise independent — like metal rods align, each of their own "volition", with a magnetic field; there is no correlation between the rods but rather a macro phenomena. Functionally a vector space _wherein_ rods exist, not a function from rod _i_ to rod _j_ for any (i,j). Galaxies might work this way too — this is my intuition. No spooky supraluminal action at a distance, just a "gravitational polarity X" that averages to orienting space locally at large-enough scales. At a higher scale in 3D, it's probably more about tangents lines (i.e. said "polarity" is of the flow, the streams of galaxies, possibly "dark matter" which is a Y here, possibly distinct from the X phenomena) than a macro-universal sense of rotation (although it's a perfectly valid solution too, indeed many symmetries are broken for our cosmos to exist, namely anti/matter, causality, or chirality.)

Rotation discrepancy could be related to X and/or Y or yet another Z phenomena (my money is on the supermassive blackhole in the center, which I think we're not fully understanding all effects yet, far from it, but meh it's bias and subjectivity on my part).

[Btw, since you seem to like this kind of discussion, I'd strongly recommend you follow Anton Petrov on YouTube[1]. He's a wonderful young scientist who makes short 5-15 minutes videos on new astrophysics / astronomy papers every day or so. For me, it's just the best way I've ever had to 'quickly' keep track of the latest observations and theories, while hearing the explanation and questions of an expert. Dr Becky[2] is another newcomer in the field of intelligent vulgarization, and she's on a path to awesomeness as well.]

I've been into these things since my teenage years (late 1990s) and it's incredible how much more observation we have now, how much more material there is to think about. It's a truly great time for space amateurs.

[1]: [https://www.youtube.com/channel/UCciQ8wFcVoIIMi- lfu8-cjQ](https://www.youtube.com/channel/UCciQ8wFcVoIIMi-lfu8-cjQ)

[2]: [https://www.youtube.com/channel/UCYNbYGl89UUowy8oXkipC-Q](https://www.youtube.com/channel/UCYNbYGl89UUowy8oXkipC-Q)

<comment><USER8> Sorry. TFA is "this f... article".

I agree about the "correlation does not imply causation" plague. It's especially problematic for purely observational sciences, where experiments are impossible or unethical. Prototypically for epidemiology, but also for astronomy and cosmology. And yes, it's a slippery slope, even for experimental sciences.

I get what you say about the alignment of metal rods in magnetic fields. So yes, there could be some gravitational polarity over extremely large scales.

I see that I was getting stuck on the idea that changes can't propagate faster than light, no matter what the mechanism. But now I get how arbitrarily separated stuff can be similarly affected by a given source. For example, there was a recent article about hot gas "sloshing" in a galaxy cluster.[0]

And thanks for the YouTube links :)

0) [https://phys.org/news/2020-01-sighting-hot-gas-sloshing- gala...](https://phys.org/news/2020-01-sighting-hot-gas-sloshing-galaxy.html)

<comment><USER9> Aha, TFA, right :p

Yeah, the faster-than-light topic is actually a very, very tricky one, even for seasoned astro-peeps it seems.

I didn't mention PBS Spacetime[1] but it's probably the best place to start for best-in-class high level mojo in astrophysics. Tons of back-catalog. Really great quality production. Less 'newsy', more 'substance' than the other two I mentioned (these are my trio on the topic).

Case in point: they do a fantastic job at explaining, notably time shenanigans. They basically made it click for me, notably for FTL (Faster- Than-Light) space expansion, cosmic inflation, etc. Fantastic-everything about them, can't recommend enough.

One thing, though: my best way of framing the FTL limit is in terms of a speed limit ( _c_ ) to transfer "information". This is why entanglement is a valid principle, it does _not_ violate _c_ because entanglement cannot be _used_ to transfer _information_ faster than light[2].

So "structure" can change arbitrarily fast, from the perception of its objects, its contents; but "information", the contents, the objects, they are bound by _c_ and all the rules actually _defined_ by said structure.

I really don't know if that's the correct view, but it seems to work to explain (actually rule out) most light-speed paradoxes, so...

[1]: [https://www.youtube.com/channel/UC7_gcs09iThXybpVgjHZ_7g](https://www.youtube.com/channel/UC7_gcs09iThXybpVgjHZ_7g)

[2]: I think this vid does a good job at explaining why: [https://youtu.be/2_4l5_G3dnM](https://youtu.be/2_4l5_G3dnM)

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER10> The title (not the text, though) sounds a bit like the premise of The Flow from John Scalzi's The Collapsing Empire [https://en.wikipedia.org/wiki/The_Collapsing_Empire](https://en.wikipedia.org/wiki/The_Collapsing_Empire)

<comment><USER11> Any good?

<comment><USER10> Yep, not bad, but not in an epic sci-fi way like, say, Hyperion Cantos. It's funny and witty, kinda like a dramedy-political-thriller that just happens to be based in space.

</comment></comment></comment><comment><USER12> Is this really that surprising? I thought that the CMB distribution of matter displays a proto-structure that matches with what we observe today, meaning the large scale structure and congruent behavior we observe is just the result of the initial distribution and motion of matter in the early universe, rather than some mysterious force or effect we have yet to explain.

<comment><USER13> Or, it doesn't require mysterious force or effect - it could just be an emergent property of how matter interacts. For instance, basic rules of atomic forces lead to complex behavior in the aggregate, such as life, foam, planets, etc. Not sure why this wouldn't apply to universal size structure as well.

<comment><USER14> Right, but those forces and effects drop off with distance. The alignments we're observing are many, many, many orders of magnitude too far dispersed to be explainable by known forces.

It would be like finding a few dozen planets floating in interstellar space hundreds of lightyears away, much further away than the nearest stars, but on analysis finding that they all happen to be rotating around our sun even though our sun's gravity is utterly overwhelmed by the gravity of other objects much closer to them.

<comment><USER13> The same is true of a skin cell in your foot vs in your ear. You'd find remarkable similarity even though there is no apparent reason they would be similar until you comprehend the superstructure.

The idea is that the similarities across vast amounts of space would not be due to first order effects from forces, but rather much more complex and subtle interactions over time that happen to have structure rather than being purely random, just as life emerged from seemingly random interactions in primordial pools.

<comment><USER14> Well of course it seems likely some form of interaction at some point in the past or present is behind these effects, we just don't know what it was or is. If that's all your saying, sure.

</comment></comment></comment></comment></comment><comment><USER15> Is there no chance these are coincidences? It gives me "Bible Code" vibes: [https://en.wikipedia.org/wiki/Bible_code#Criticism](https://en.wikipedia.org/wiki/Bible_code#Criticism)

"information theory does not prohibit "noise" from appearing to be sometimes meaningful"

<comment><USER16> Indeed, or the result of some specific starting conditions in the early universe.

But I must assume that these cosmologists have already considered and discarded that. I'd just like to be able to read more about why that guess doesn't fit what they're observing.

<comment><USER14> 'Starting conditions in the early universe' is too vague to actually consider properly. To rise to the level of even thinking about accepting or discarding it, we'd need to know what sort of conditions we're talking about, how they might lead to what we're observing, and preferably also how they might have come about. It's a class of possible explanations, not an explanation itself.

</comment></comment></comment><comment><USER17> All hail his noodly appendage.

<comment><USER18> String theorists got it wrong: the universe isn't made up of small vibrating strings, it's made up of small vibrating noodles.

</comment></comment><comment><AUTHOR> One thought that occurs to me is that large-scale structures like sheets and black holes with parallel axes of rotation could be the result large directional explosions soon after the big bang rather than the result of a large force acting now.

<comment><USER19> Where would these directional explosions come from?

<comment><AUTHOR> While I'd note one doesn't necessarily need an set explanation for such directional explosions to formulate decent theory (look at dark matter, etc), I think electromagnetic forces often produce highly directional effects.

<comment><USER20> There are large-scale anistropies in the universe, and the origin of symmetry- breaking at cosmological scales is an unsolved problem. Either a force or type of matter is unaccounted for / incorrectly modeled, or the big bang was not symmetrical.

[https://en.wikipedia.org/wiki/Axis_of_evil_(cosmology)](https://en.wikipedia.org/wiki/Axis_of_evil_\(cosmology\))

</comment></comment></comment></comment><comment><USER21> Evidence? Observations from our point of view (including readings from space, still fairly close, on scale) are limited to our current position etc. So how can we really tell if it's just not curvature of space and blackholes that distort our point of view and readings? Interesting anyhow, but perhaps a bit soon.

</comment><comment><USER22> All i know is: I want stargate to be a thing

<comment><USER23> Stargate Universe in this case, as Destiny was launched to collect data all across the universe, because the Ancients had identified a pattern in the structure of the universe...

<comment><USER24> Not just "the structure of the universe", the Ancients identified structure in the cosmic background radiation.

The episode that revealed this aired within a month of a real-world discovery of a pattern in the cosmic background radiation [0]. Unfortunately, it wasn't nearly as enticing as what was in the show.

[0] [https://www.seeker.com/cosmic-rebirth-encoded-in- background-...](https://www.seeker.com/cosmic-rebirth-encoded-in-background- radiation-1765141336.html)

</comment></comment></comment><comment><USER25> Plasma filaments and other plasma structures can manifest long-range attractive and repulsive electromagnetic forces proportional to d^-1 instead of d^-2. So if those forces are operative at galactic and intergalactic scales (between plasma structures) and we're not fully taking them into account, then we may be overlooking the most powerful organizing mechanism in the universe.

</comment><comment><USER26> It seems that periodically there is evidence showing that the universe is this or that and yet nothing is and it will never be concrete because it's impossible to find the root of the universe. I see it as a more philosophical question. I think we should be more focused on exploring the universe, not understanding it.

<comment><USER13> We are able to explore the universe because of out attempts to understand it.

</comment></comment><comment><USER27> A long-term motion of large structures of dark matter may be one explanation. We still don't know too much about dark matter except it plays central role in building the cosmic web.

I hope James Webb Space Telescope will not fail to operate and its launch will not be delayed again (the project is 25 years old).

</comment><comment><USER28> What kind of movement is it?

Calling it a web of filaments doesn't paint a clear picture as to the movement itself.

Assuming it's synchronized, I think of child objects moving in sync with some kind of parent structure. That sounds ridiculous though.

<comment><USER14> I don't understand why you think it's ridiculous for child objects, which formed within a structure, to reflect characteristics of that structure.

The movements described in the article are mainly alignment and direction of axial rotation.

</comment></comment><comment><USER29> [https://www.youtube.com/watch?v=E4pWZGBpWP0](https://www.youtube.com/watch?v=E4pWZGBpWP0)

It would be pretty amazing if all this turned out to be true ^

</comment><comment><USER30> Another research about large scale structures: [http://irfu.cea.fr/dipolerepeller](http://irfu.cea.fr/dipolerepeller)

</comment><comment><USER31> This sounds a bit like Bill Gaede's Rope Hypothesis, but I'm not very well- versed in the topic to say for sure. Can anyone confirm/deny?

</comment><comment><USER32> This makes sense. Our roads look like arteries. The universe is fractal. So it makes sense that there are arterial-like structures connecting galaxies.

</comment><comment><USER33> The Coming of the Great White Handkerchief is near!

</comment><comment><USER34> > which is very slowly rotating in a counter-clockwise direction

What does this even mean? Would they rotate clockwise when looking from Australia?

</comment><comment><USER35> if quantum theory can make predictions about the patterns in the CMB, can it also anticipate these structures?

</comment><comment><USER36> Just a thought: maybe space is not sparsely populated with bits of stars and rocks. Space is filled with a form of matter we cannot yet detect.

<comment><USER37> Not matter but something is everywhere. Space isn't empty, nothing is empty, it just isn't always interactive.

[https://en.wikipedia.org/wiki/Higgs_boson#Higgs_field](https://en.wikipedia.org/wiki/Higgs_boson#Higgs_field)

</comment></comment><comment><USER38> might fit with the simulation hypothesis?

</comment><comment><USER39> Url changed from [https://bigthink.com/large-scale- structures](https://bigthink.com/large-scale-structures), which points to this.

</comment><comment><USER40> Can we teleport to those giant structures?

</comment><comment><USER41> Magnetic Universe

</comment><comment><USER42> No!

</comment><comment><USER43> I know another construct that has this structure: the brain.

</comment><comment><USER44> What happens when "God" pulls the plug on his/her experiment? I hope he backs us up.

<comment><USER45> Why? You wouldn't exist to be able to perceive that it happened at all even if he did

<comment><USER46> I just don't want to be left on some mislabeled Zip Disk is all.

</comment></comment></comment><comment><USER47> We know many things in nature are fractals: coastlines, tree branches, blood vessels...

What if the Universe is one big fractal, repeating itself at different scales: electrons go around a nucleus, planets around a star, stars/galaxies around blackholes, and so on and so on, building larger and larger structures.

<comment><USER48> Electrons don't rotate around the nucleus in the same way that planets go around stars, that's a simplification for high school text books. Supermassive black holes don't keep galaxies together either, galaxies rotate around themselves and even that isn't enough, hence dark matter.

<comment><USER49> Fractals are not necessarily the same at different scales. They can be self- similar at different scales.

</comment></comment></comment><comment><USER50> > "Such studies are statistical and a step forward would require a large > amount of polarization data, not easy to gather with current > instrumentation."

[https://www.urbandictionary.com/define.php?term=Blinker%20Sy...](https://www.urbandictionary.com/define.php?term=Blinker%20Sync)

</comment><comment> ur-whale Goes to show we just don't have the Math to understand large scale solutions to differential equations for the N-body problem.

</comment><comment><USER51> > as if they are connected by a vast unseen force.

LOL. Popsci is hilarious. And physics is in a desperate crisis. We need less science and more philosophy.

</comment><comment><USER52> We know very little about our universe. We can't even define the Hubble constant. Dark matter theory is a joke imo.

These nerds don't even understand refraction. Space isn't empty.

If space isn't empty you will never be able to account for the path photons actually travel.

Gas causes refraction, how could you be sure the photos that hit the lens in your telescope actually traveled in a straight line.

I know this will get down voted to hell. IDC.

<comment><USER53> As I understand it dark matter is predicted by one set of observations and confirmed by many others within calculated error bounds. It's not some kind of overfit model trying to explain a single observation. So to discount it you'd need to explain why it just so happens that many observations converge to the same results as that which you would expect from the predicted dark matter distribution. I don't think mere refraction can explain lots of different observations.

Dark energy on the other hand is a whole other story.

<comment><USER52> First thing to understand is the only evidence we have for the universe expanding is redshift.

Do I believe refraction is at all related to Dark mass/energy? Not all all. Refraction has nothing to do with that. Refraction has to do with redshift being miscalculated.

Compression waves are the culprit and the field that propagates those waves for dark garbage. baryonic acoustic waves give direct evidence as the perp.

<comment><USER53> I think you missed the actual point of my comment? Refraction was kind of irrelevant to it, I was trying to say something about the consistency of different observations and the strength of the evidence behind them. You could replace "refraction" with "squirrels" and it wouldn't really change what I was trying to explain.

<comment><USER52> Well you misunderstood that I was referring to refraction as being the reason for dark matter predictions. I was trying to imply that it can explain why the universe is expanding. Instead of the theory of dark matter.

I already gave you my reasoning for dark matter calculation, baryonic waves. Which you so happened to ignore?

</comment></comment></comment></comment></comment><comment><USER54> I have an unusual idea related to this. You might call it the interconnectedness of all things. (h/t DG) The more you look into quantum systems the more things seem to look like you are looking at the observable universe that we see in the sky. What if they were the same thing and space has somehow folded back upon itself? Of course it's not really a falsifiable conjecture. For example, we have absolutely no idea what's going on inside a proton and we'll probably never know.

<comment><USER55> > For example, we have absolutely no idea what's going on inside a proton and > we'll probably never know.

We actually have a pretty good idea of what's going on inside a proton.

[https://en.wikipedia.org/wiki/Proton](https://en.wikipedia.org/wiki/Proton)

<comment><USER54> Sure, can you explain it to me and cite your evidence? Because that's the wikipedia article about protons, not an answer. The truth is this is a controversial area of physics heavy on theories and light on experimental data. The simple fact of the matter is we lack the measurement technology to understand the inner workings of a proton. Theories are just that.

<comment><USER56> There is an entire theory that describes the strong interactions inside the proton. Please take a look at [https://en.wikipedia.org/wiki/Quantum_chromodynamics](https://en.wikipedia.org/wiki/Quantum_chromodynamics) and experimental evidence for QCD: [https://en.wikipedia.org/wiki/Quantum_chromodynamics#Experim...](https://en.wikipedia.org/wiki/Quantum_chromodynamics#Experimental_tests)

Furthermore particle accelerators have been smashing protons for a long time to probe what's happening inside. The standard model of particle physics is validated by evidence and experiments. I don't think it is fair to say that we don't know what is happening inside a proton. There are many questions in physics still left unanswered but I don't think anyone would classify this as such.

<comment><USER54> If we had such a "very good" understanding they wouldn't be spending billions to build the Electron-Ion collider at Brookhaven. Here's a good article about it, that's much more balanced than the QCD slanted wikipedia article: [https://www.sciencemag.org/news/2018/07/scientists- endorse-b...](https://www.sciencemag.org/news/2018/07/scientists-endorse- billion-dollar-collider-look-inside-protons-and-neutrons)

I especially like this bit about qcd:

> The mess is so complex that even basic properties of the proton remain > unexplained. For example, its three quarks account for less than 5% of its > mass, the rest arising somehow from energy of the virtual quarks and gluons.

<comment><USER57> This video explains where the mass of a proton comes from: [https://www.youtube.com/watch?v=Ztc6QPNUqls](https://www.youtube.com/watch?v=Ztc6QPNUqls)

Based on your comments above, though, I don't think watching this would change your opinion about anything. People have been more than polite in trying to help you fix your misunderstandings, but you refuse to absorb any new information or change your views when presented with new data.</comment></comment></comment></comment></comment></comment>


Peter Thiel Violates Core Principles of Silicon Valley - <AUTHOR> https://www.linkedin.com/pulse/peter-thiel-violates-core-principles-silicon-valley-michael-lazerow <comment><USER2> Trust and Transparency are two Silicon Valley principles ?

This mustn't be the same Silicon Valley that all of these big software companies are in then... because their core principle is making money.

Maybe things look different to the wealthy residents, or maybe they believe their own PR.

</comment><comment><USER3> I don't agree wth the author, but I do like his writing style.

Thiel did not break the law. Compare with other non-transparent transactions like the wholesale purchasing of political influence. Thiel's lack of transparency in going after Gawker seems unimportant to me.

<comment><USER4> The law is not valid a basis of ethics.</comment></comment>


Where Is Technology Taking the Economy? - <AUTHOR> http://blog.irvingwb.com/blog/2017/11/where-is-technology-taking-the-economy.html <comment><USER3> There's a lot of talk about how AI will "steal jobs". But I think one aspect that doesn't get enough attention is how our physical bodies seem to have major problems adapting (fast enough) to where technology is taking us. Take for example how immobile jobs , combined with bad food (industrialized processed food) and driving has all sorts of negative physical impact. To add to this, self-driving cars will remove the need of a driving skill. Or how address books removed the need to remember phone numbers. I remember in the 90s (4th grade) when could recall the phone number of everyone in my class. Today I honestly don't know anyone's phone number but my own.

How do we account for the fact that the human body and mind seem to need a certain amount/kind of stimulation to not entirely fall apart? Should we just settle with the new bodies technology gives us? Should we settle with simple compensatory measures like going to the gym for lack of moving otherwise?

<comment><USER4> > I remember in the 90s (4th grade) when could recall the phone number of > everyone in my class.

Seems like a waste of brainpower to me.

I remember in the 90s when I couldn't listen to music I wanted without paying £12.99 for a CD. Or when I couldn't do my maths homework because I lost my textbook which my parents had to replace at a cost of £30. Or when I was really upset after going to meet a friend who never showed up and couldn't talk to me to say why until the next day at school (his parent's car had broken down).

Technology is AMAZING, people will be fat even without self driving cars, it's greed and laziness that causes fat to accumulate, not tech.

<comment><USER5> > it's greed and laziness that causes fat to accumulate

Are Americans much greedier and lazier than they were 40 years ago? Are American animals?

<comment><USER6> All living things are greedy and lazy. It's not good, but it's also not evil. It's just physics. Humans aren't unusually greedy or lazy. They are just the only ones that beat themselves up over it. Particularly the American ones.

</comment></comment></comment></comment><comment><USER7> "In the distributive era free-market efficiency will no longer be justifiable if it creates whole classes of people who lose."

I lean towards the positive side of Universal Basic Income (my core argument is that no kid grows up thinking they want to be inactive) so it's heartening to see more of these arguments come to the fore.

<comment><USER8> UBI is lazy tech guilt thinking. Not to mention it's arrogant. The idea that we've solved all that needs to be done is remarkable. Just look around you. Do you think we are in a place without problems that need resolving? Or work to be done? It's truly a problem of goals of humanity that lead to this disaster of demoralizing pay people to exist UBI thinking. Pay people for education first. We aren't even doing this!! Pay people to resolve climate change. Pay for more research. Pay for xyz that drives humans to new levels while enabling them to valuably contribute to our society. I'm going to write further on this but the UBI mindset is nothing more than lazy arrogant tech people who have no historical frame.

<comment><USER9> The guilt thing is real. Every thoughtful person in tech knows that what we produce devalues human skill and produces net job loss.

<comment><USER8> Yes but the lack of jobs is a political problem. It's a problem of vision and goals. Just like a housing crisis. These can be resolved by changing the rules and goals of the group.

<comment><USER9> People are not things to be engineered. This is our largest collective blindspot. Some people may look at what you see as utopia and say "no, thanks." This can happen for something as simple as UBI. In fact, I'm sure it will.

</comment></comment></comment></comment></comment><comment><USER10> I think it makes sense to have something other than a pure universal basic income for redistribution of wealth. You want to design an incentive based system to encourage people to contribute to their community, pursue higher education and learning, as well as work.

Some kind of income supplement if you work at least 15 hours per week makes sense, along with a bonus for 'community contribution' of some kind. It makes sense to reduce the overall amount worked, but to still encourage work, and to improve the skillset of workers.

Pure universal basic income has dubious incentives. It's a solution, but I don't think it's close to the best one. Of course, any system, including the idea I'm proposing will be gamed, and have unfortunate side effects from its policies.

<comment><USER11> Contribute how? In the long run all jobs will disappear. The latest job to become obsolete will be scientific research.

<comment><USER12> Or artists, or prostitutes, or athletes.

</comment></comment></comment><comment><USER13> A 4 day work week is the answer. Ford changed the work week from 6 days to 5 days. It's time for another major corporation to set an example and make this move. Pay employees the same they make now for 5 days. For many professions, the same work can be done in 4 days, and for other menial jobs you give peole the opportunity to grow.

<comment><USER14> I'd like to read more on how Ford managed to change the work week from 6 days to 5 days. Would you kindly direct me to some articles/ books/ PDF's? Sincerely,

<comment><USER15> [http://www.history.com/this-day-in-history/ford-factory- work...](http://www.history.com/this-day-in-history/ford-factory-workers- get-40-hour-week)

In short, Ford likely realized in the 1920s that with the usage of mass manufacturing / shifts, worker effort was more important than hours spent working, and could afford to make the change.

This eventually became codified in the US via the Fair Labor Standard Act of 1938 [1], which defined overtime as anything over 40 hours a week.

[1] [https://en.m.wikipedia.org/wiki/Fair_Labor_Standards_Act_of_...](https://en.m.wikipedia.org/wiki/Fair_Labor_Standards_Act_of_1938)

</comment></comment></comment><comment><USER16> >Throughout the Industrial Revolution there were periodic panics about the impact of automation on jobs, going back to the so-called Luddites, - <USER2> workers who in the 1810s smashed the new machines that were threatening their jobs.

This is such a common misconception that the wikipedia entry for luddite mentions it in the first paragraph.

They destroyed weaving machinery for the same reason French taxi drivers shut down roads in 2016.

Revisionist historians depict luddites as anti-technology for the same reason Forbes chose to depict the French taxi drivers as anti-smartphone.

</comment><comment><USER17> Where it's alway taken us. Forward.

<comment><USER18> Widespread private car ownership, to take one example, has arguably _harmed_ average quality of life.

Pollution, reduced freedom to walk/bike, extra time having to work out as a result of those last two things (or simply reduced health), and, at an average income, total time lost to the car, including working to pay for it, for insurance, for gas, for maintenance, et c., being so high that it often results in a net _loss_ of time versus foot-powered transportation ( _without_ even considering how much denser cities would be without cars), all conspire to create a world where many are pressured to own a car (life is harder without it, areas with good schools may be inaccessible without it, geographical distribution of jobs in cities may be inconvenient to put it mildly, and public transit has largely gone to hell, all because of the side- effects of widespread car ownership) but they actually _lose out_ in terms of time-saved, and a number of other factors, compared with a world where car ownership was restricted.

Just setting technology loose, even with something as seemingly great as the automobile, isn't always a straightforward win for the average person.</comment></comment>


Common problems when translating games into Japanese - <AUTHOR> http://www.gamasutra.com/view/news/328996/Common_problems_when_translating_games_into_Japanese.php <comment><USER2> The problem with this article: pictures with text on it.</comment>


WeWork documents reveal it owes $18B in rent - <AUTHOR> https://www.businessinsider.de/wework-500-million-bond-sale-financials-analysis-2018-4 <comment><USER2> What's wrong with having $18B in rent? Seriously.

WeWork is in the same business as what Hertz is in for cars.

They rent out an expensive asset (eg commercial real estate and automobiles)

Hertz has $21B in liabilities [1], which is largely comparized of paying financing charges for all of those Toyota Camrys they rent out.

Yet they are still profitable.

[1] [http://financials.morningstar.com/balance- sheet/bs.html?t=HT...](http://financials.morningstar.com/balance- sheet/bs.html?t=HTZ&region=usa)

<comment> et-al The concern is that real estate fluctuates more than automobile prices.

If the market tanks and rents go down, then WeWork is locked into overpriced rates. They have contracts until 2022.

Yesterday's discussion [0] did point out that WeWork leases from an LLC subsidiary that leases from the actual building owners. This would shelter WeWork Corp in an odd corporate structuring sort of way (similar to McDonald's making money off real estate).

[0] [https://news.ycombinator.com/item?id=16913110](https://news.ycombinator.com/item?id=16913110)

<comment><USER3> There's some bargaining power on the lessee going on here too. The way I've heard it works is that they would lock in for a number of years at a reduced market rate. The longer a tenant is contracted in, the risk of the building being empty lowers for the owner. This allows a lower price per month to be negotiated because on average the building isn't going to be non-revenue positive.

Now, if property prices take a dive, I'm not sure that would impact WeWork's business all too much as their main clientele are solo entrepreneurs or small startups that couldn't otherwise afford an office fit out and/or the risk of taking on a lease. There is more value in WeWork than just the building.

</comment></comment></comment><comment><USER4> The article mentioned is [https://www.bloomberg.com/news/articles/2018-04-24/the-18-bi...](https://www.bloomberg.com/news/articles/2018-04-24/the-18-billion- rent-bill-the-numbers-behind-wework-s-empire)

At the very bottom,

> Since 2015, WeWork has also leased space in some buildings in which CEO > "Adam Neumann and certain of his immediate family members hold ownership > interests,"

Kind of weird, but maybe I'm missing something.

<comment><USER5> Sounds like a good way to get investor money into your own pocket even if a company isn't profitable.

</comment></comment><comment><USER6> Do they really have $18B in liabilities when they only generated $822M in revenue with $1.8B in expenses? Could someone explain the math of how this business makes or will make money?

<comment><USER7> They don't have $18bn in liabilities. Total liabilities as of 12/31/17 are $2.4bn, including $1.7bn in unpaid rent [0]. They've committed to paying $18bn in rent on space they're currently leasing [1]. But that's not carried as a liability on their balance sheet, and barring a collapse in demand (which _is_ a possibility that potential creditors should consider), they'll receive rent payments from clients to fund those payments as they come due.

As an aside, this sort of confusion is why it's important to link to either a primary source or a good secondary source whenever possible, in my opinion. The Bloomberg article is clear on this issue, while the Business Insider article linked is just a watered-down, ambiguous summary of that article.

[0] [https://ftalphaville.ft.com/2018/04/24/1524606395000/More- on...](https://ftalphaville.ft.com/2018/04/24/1524606395000/More-on-WeWork- and-its-bond-offering/)

[1] [https://www.bloomberg.com/news/articles/2018-04-24/the-18-bi...](https://www.bloomberg.com/news/articles/2018-04-24/the-18-billion- rent-bill-the-numbers-behind-wework-s-empire)

</comment></comment><comment><USER8> Can someone give ma a rough estimate of what $18B buys you?

Is that a 100 storey skyscraper in New York? 18 of them? (I am guessing that an apartment there must cost > $1 million looking at the prices here in Barcelona).

<comment><USER9> 10 Burj Khalifa's

<comment><USER10> 1000 whiz khalifas

</comment></comment></comment><comment> james-skemp Possible duplicate of [https://news.ycombinator.com/item?id=16913110](https://news.ycombinator.com/item?id=16913110) ?

</comment><comment><USER10> i don't understand a business concept that provides essentially a combination between what you get at a public library and what you get at a workplace for a fee.

they are a "salon" for techno-imperialists who think they are too good to stare at a computer at a library or starbucks

if they make money though, i guess they prove me wrong.

</comment><comment><USER11> They will never be profitable this way then.</comment>


RVM for Windows: vertiginous's pik at master - GitHub - <AUTHOR> http://github.com/vertiginous/pik <comment><USER2> Using a virtual dev environment (with vagrant) seems like a good move if you're stuck on windows. You get to develop with software that runs on linux using whatever tools you're comfortable with on windows.

</comment><comment><USER3> I only briefly looked through the README, but why is this necessary? Does The Real RVM (tm) not work (well) on Windows?

<comment><USER4> it doesn't work at all on windows afaik. Pik is a decent recreation, but not quite as good as rvm imo.

</comment></comment><comment> Ocho-Bits Is still Ruby and Rails slow as hell in Windows?

<comment><USER5> You can always use jruby

<comment><USER6> ruby startup speed is most annoying on windows. jruby is not a fix for this issue by design.

<comment><USER5> How long does it take for MRI to startup on windows?

<comment><USER6> interpreter startup with tiny script is ok but loading unit tests by rake command is slow.</comment></comment></comment></comment></comment>


How much does it cost to run your website - <AUTHOR> https://webion.io <comment><AUTHOR> Webion is a cloud-based website builder that allows anyone to build and host custom websites just by defining the user interface.</comment>


Ask HN: Will Microsoft ever become cool again? - <AUTHOR> <comment><USER2> They're doing a hell of a lot more cool things than Apple, these days. Windows is moving. macOS is terrible.

(Says the guy who's been a Mac user for >15 years and bought every new product Apple has come out with.)

Apple is doing this to themselves. As a dev platform, their appeal is waning. (Stop making shit laptops, and make your macOS stable again. ¯\\_(ツ)_/¯ )

Microsoft, under Nadella, is making the right moves, listening to their userbase. It was enough to get me to switch back.

<comment><USER3> Doesn't Windows have ads embedded w/in their OS?

<comment><USER4> I've been using professional since release and haven't seen an ad yet.

</comment></comment></comment><comment><USER5> If someone is on a windows machine, using the pre-installed IE11 or Edge browser, I tend to assume it's a 70 year old these days. Or possibly someone at a firm with some dated internal ActiveX page that needs IE.

That kind of association is extremely difficult to break. Car manufacturers have struggled with similar impressions. You can make the greatest, and cheapest, car in the world, and people won't buy it if it has that kind of negative association.

<comment><USER6> Talking of cars, Porsche sell stripped out, weekend racer versions for more than the more luxurious ones. But it is a niche market. Same with computers for nerds - we know what we are doing, but for the average consumer - which is certainly MS's target market - the more bells and whistles the more impressive it is in the showroom.

<comment><USER7> I see what you're trying to get at. But your example is not correct.

What Porsche model is a stripped down version of another? Porsche Turbo S —> GT3/2 RS? Cayman GTS —> GT4?

Because in those 2 examples, there's no stripping down.

</comment></comment></comment><comment><USER8> I think they're cool. The Surface Pro and Surface Book are the only computers since the iPad first came out that I actually wanted to buy.

They also put Linux in Windows, released visual studio for free, created one of the most popular text editors ever, and created 3 programming languages that are generally well liked (C#, F#, and typescript).

I'm not sure how cool they are compared to Google, Amazon, Netflix, Apple, and Facebook but I personally tend to compare them to enterprise companies like IBM and Oracle. Microsoft wins that coolness contest easily.

<comment><USER9> You left out that .NET Core is just as fast as Node.js [1] and .Net runs on linux. Its pretty cool that I can write Linux programs now even though I am terrible at c (to my neverending shame).

1 [https://stackoverflow.com/a/44093694/73804](https://stackoverflow.com/a/44093694/73804)

<comment><USER10> Being bad at C (and family) does not mean you can not write 'Linux programs', just pick any other language of choice and start writing. Recently the .Net family has been added, before that Mono made a valiant attempt at supporting it but those have been preceded (and might be succeeded) by a lengthy list of languages in which 'Linux programs' can be written, from BASIC to Haskell, from Python to Ocaml, from Fortran to Lisp, take your pick.

</comment></comment></comment><comment><USER6> Cool to whom?

As a nerd, I would like them to do what Apple did: release a new, clean, robust OS (based on OpenBSD would be smart, worked for Apple) and then provide a shim for backward compatibility. That is, allow all existing programs to still run but have a clean platform for future apps. Like ChromeOS I would prefer this to be truly secure, robust and updated with minimal intrusion.

As a consumer, I want systems that are reliable, robust, secure, free from malware, impossible to hack into, cheap and fast. In other words everything that Windows currently isn't.

<comment><USER10> The lower levels of Windows 10 are probably the least problematic so basing a 'new, clean, robust OS' on OpenBSD (which, by the way, Apple did _not_ do, they used parts of the _FreeBSD_ userland on top of a Mach-inspired kernel) would not do much good as they'd still have to include support for Win32 for it to be considered a replacement for Windows. It is there Windows really shows what it is made of: glue, tape, rubber bands, hastily erected scaffolding around sassing facades, heavy structures built on top of unstable foundations. Microsoft tries (and tries, and tries) to replace all that with something new but they have a hard time making up their mind what the replacement should be and where the focus should lie.

In this sense a good successor to Windows would be a lightweight VM host running one-off instances of older Windows versions to support the host of software made for that platform, allowing those instances to communicate with each other. That might be good for users but it does not offer a path forward for Microsoft.

In truth, there probably is no way forward for Microsoft in this space. Windows is a dead end, the 'information at your fingertips' idea which Gates spouted has come true but not by way of Windows.

</comment></comment><comment><USER11> Has it ever really been? I think that, as of today, Microsoft is cooler than it's ever been before.

<comment><USER12> I agree. To me, out of the 'big 4', they are the coolest currently.

</comment></comment><comment><USER13> Again or for the 1st time? They're focusing on the entreprise business and moved away from the consumer business. So my guess is that it's pretty hard to be "cool" in the entreprise world... Look at salesforce has been trying so hard, it doesn't change the fact that their products don't mean anything to the majority of people. So my answer would be... maybe?

<comment><USER8> Microsoft does plenty for the consumer or the non-enterprise ddeveloper. Visual studio for free, VS Code, Typescript, Surface Pro, Surface Book, Windows subsystem for Linux, etc.

</comment></comment><comment><USER14> VSCode and the whole open source organization around it is super cool in my opinion.

TypeScript and C# are great tools too.

</comment><comment><USER15> Does cool matter? They were under $40 a share 5 years ago.

</comment><comment><USER16> I was reading a comment stection of a win xp pirated edition. It had los of bug reports. 3-10 min after each there was a developer comment of the kind "thats terrible!" 10-30 min after that a comment containing a patch was posted.

Am i to understand this kind of service is impossible for a multibillion company? You dont have any questions, you dont have any problem and we are certainly not going to fix anything.

</comment><comment><USER17> No, they are another dull corporation just like IBM, Oracle, Apple, Google, etc. Don't fall for the PR of any of these companies. They aren't your friends.

</comment><comment><USER18> Again?

<comment><USER19> When IBM went to the PS/2 architecture, and the OS/2 software, to try to kill the PC clone market and lock up the PC hardware and OS market for themselves, Microsoft was the champion of freedom and openness. Really.

Windows 95 and 98 were when the GUI became cool. Well, cool _and available_ \- it was cool on the Mac, but nobody had Macs.

<comment><USER18> Yeah, I could see how finally getting their GUI system more or less working a decade after everyone else could be seen as cool. And yes, keeping the corporate drone market away from a technically better product, also cool.

<comment><USER19> "After everyone else"? Well, there was the Mac. And there was TopView. And there was... who, exactly, is "everyone else"? There were workstations like Silicon Graphics, but they didn't even pretend to be available (affordable) for the average user. They weren't competition for the PC. On commodity hardware, it was Windows and TopView, and TopView was... not very useful, to put it charitably.

The PS/2 was a technically better product. It was also IBM's attempt to put the genie back in the bottle, so that it could sell hardware at prices considerably above the commodity level. Technically better? Cool. Not having to pay several hundred dollars more for your hardware? The market thought that was even more cool.

</comment></comment></comment></comment><comment><USER20> No.

</comment><comment><USER21> MS hasn't been cool since the 80s and even then its coolness was debatable.

</comment><comment><USER22> No fucking way.

</comment><comment><USER23> Sure, as soon as they open-source Windows, Office, and Sharepoint and make the XBox platform Open Hardware.

<comment><USER18> Open source Sharepoint? Why would anyone want that?

<comment><USER23> They wouldn't, it's just the principle of the thing!</comment></comment></comment>


Avoid Eating At These Spots in San Francisco (Heatmap of 2013 Health Violations) - <AUTHOR> http://www.graf.ly/user_graphs/101

<comment><USER2> This is kind of gross to think about - the FInancial District, which is FULL of red, is supposed to be one of the classier areas to eat around the city? Right? I mean what about all the business lunches that go down here?

<comment><AUTHOR> It's one of those things you almost don't want to know the truth about.</comment></comment>


IRC bots for Laziness, Impatience, and Continuous Deployment - <AUTHOR> http://eng.wealthfront.com/2010/12/irc-bots-for-laziness-impatience-and.html

<comment><USER2> This is exactly why I wrote <https://github.com/thwarted/nodebot>, as a I ran into problems scaling and securing things like supybot, and integrating it with shell scripts. nodebot is announce only, and doesn't join the channels (so your IRC server needs to be setup to allow that), and since it's not centralized, high traffic from one node/service that ends up getting rate throttled by the server doesn't delay messages from other nodes/services. Each machine has an identity, but it can be configured to have a different identity (because I'd rather that git messages come from "git" rather than from the machine that ran the hook).

We use it at yelp for monitoring all manner of system status. There's separate channels for scribe log processing, nagios alerts, code deployments, and load balancer status. Our git hooks announce when developers push and merges are happening, and who's branches are being deployed. We get announcements when config files change. And it's all on-demand, if you're not interested in seeing it, you don't need to join the channel, IRC is perfect for these kinds of updates.

<comment><AUTHOR> Nice. Last.fm has a nice post at <http://www.metabrew.com/article/how-we-use- irc-at-lastfm>.

We also have separate channels for each message type. IRC is pub-sub, yay! Plus you can direct-message a bot and it will DM you back, to not pollute the channel you're currently in.

Do you have any bots that take commands? I think that's where it gets _really_ useful.

<comment><USER2> I've considered it, but that wasn't the use case for nodebot. In order to do some of that, you'd need to be able to run commands on the machine the bot is running on, or have the bot keep some log of state as messages are fed into it, both of which are somewhat non-trivial in terms of storage/query-ability or security. We have a few modified bots (supybot is the most common, I think) that developers have written and run themselves, but I don't think any of them are much more than playthings.

I could definitely see having the ability to allow a bot change the topic of a channel though, so, for example, the deployment status channel's topic shows the most recent version deployed, or the status of the current deployment, for those case where you're not sitting in the channel. Also, the ability to control nagios from IRC would be interesting, but we just use shell scripts for that right now (which is non-ideal in other ways).

</comment></comment></comment><comment><USER3> My complaint with IRC (and I freely admit that it could be that I am not using the "right" client) is that there is just this _sea_ of text on the screen. There is very little formatting to speak of, and it reminds me of everyone talking at once on a conference call.

 _slighty_ off-topic: I am proud to say that I was introduced to Wealthfront via HN, and I was so impressed by their engineering blog that I went to their website. And it sounded cool, so I joined.

And ever since then, I continue to see some awesome things from them, and so far their investment-matchmaking service has paid off, too.

Win-win if I've ever heard one.

<comment><USER4> I'm slightly biased, but this is what I aim to solve with Mibbit - make it a little more friendly and clean looking. Hopefully I'm headed in the right direction so far.

</comment></comment><comment><USER5> At StackVM we use an IRC bot to tell us when the servers are down. In our case the bot is just a DNode service so all the backend needs to do is connect to the IRC service, post its message, and disconnect.

<http://github.com/substack/rowbit>

It's a pretty great fit since we already use IRC a ton to coordinate development so we see the notifications right away.

<comment><AUTHOR> Sweet. We use Apache ZooKeeper for service discovery and have a #zk channel. We just did a deployment, here's the irc log: (deploying the 'PM' service, updating instance pm12 first)

[1:05pm] dm: PM leader changed from pm12 to (none)

[1:05pm] dm: PM leader changed from (none) to pm11

[1:05pm] dm: UNANNOUNCED: ServiceDescriptor[id=pm12,kind=class com.kaching.platform.guice.KachingServices$PM,uri=<http://10.99.110.140:8080,shards=[],status=UP>]

[1:05pm] dm: ANNOUNCED: ServiceDescriptor[id=pm12,kind=class com.kaching.platform.guice.KachingServices$PM,uri=<http://10.99.110.140:8080,shards=[],status=INIT>]

[1:05pm] dm: UPDATED: ServiceDescriptor[id=pm12,kind=class com.kaching.platform.guice.KachingServices$PM,uri=<http://10.99.110.140:8080,shards=[],status=UP>]

</comment></comment><comment><USER6> Interesting to see someone who is using IRC in this fashion. I personally have been an advocate of using XMPP in the same way. While IRC offers a good amount of features, XMPP provides the ability to scale and dive into multiple different realms. Just one example that comes to mind is constant updates from your XMPP bots, channels, etc. in a single location similar to Twitter, but for your development.

<comment><USER7> IRC is _easy_. Its a basic text protocol that requires a socket and some minimal parsing to work with. Whenever I use XMPP, its never that simple. The protocol is too verbose for its own good imho.

<comment><USER8> Xmpp allows you to be a bit more secure and fine tune the usage. If your client or library doesn't actually make this simpler, something has failed.

<comment><AUTHOR> See <http://en.wikipedia.org/wiki/End-to-end_principle>.</comment></comment></comment></comment>


Wikipedia became a battleground for racial justice - <AUTHOR> https://slate.com/technology/2020/06/wikipedia-george-floyd-neutrality.html <comment><USER2> Unsurprisingly, this doesn't seem to be a left vs. right issue but a far left vs. centre issue.

None of the edit suggestions or controversies were racist or biased. It is clear that ardent supporters of the BLM movement simply want to shape the historical record in the way they want to. And Wikipedia will probably side with them.

How is mentioning Floyd's past criminal record on his biographical page (not even on the "Killing of..." page) a detraction from his death? Are we just going to edit Wikipedia to remove anything that detracts from people we idolise? Wikipedia should present the facts (all of them) and that is it. If the facts detract, then maybe there's a reason for that. That doesn't change the fact that they are facts.

<comment><USER3> If you look at the linked talk page almost all votes are yes, include. It is currently included. You appear to be railing not against what is, but against an unrealized possibility which was in fact stopped by the very hive-mind you falsely assume would push, or at least accept it.

Call me slanted all you want, but this, the current state of comments here, that's what an agenda looks like.

<comment><USER2> I was basing my comments on the linked Slate article.

<comment><USER3> And the link I referred to is in that article.

</comment></comment></comment></comment><comment><USER4> This morning, I chanced upon the wiki page for "George Floyd protests".

I consider myself more liberal than many, and yet I was struck by the obvious left slant of that page.

Not that it has inaccuracies, but its selection of facts and how they are presented, are definitely not neutral. From the header ("protests" or "riots"?) to the prevalence ("George Floyd protests in Minnesota"[0] and "George Floyd protests"[1], etc[2], [3], [4], [5] ...) to the descriptions and topics....

IMO such bias erodes their trust, and the more centrist will have to choose between one echo chamber and another and decide their facts by emotion.

Unfortunately, I am beginning to feel that the more "woke" a subject is, the less likely the facts will be presented fairly.

[0] [https://en.wikipedia.org/wiki/2020_Twin_Cities_riots](https://en.wikipedia.org/wiki/2020_Twin_Cities_riots) [1] [https://en.wikipedia.org/wiki/George_Floyd_protests](https://en.wikipedia.org/wiki/George_Floyd_protests) [2] [https://en.wikipedia.org/wiki/George_Floyd](https://en.wikipedia.org/wiki/George_Floyd) [3] [https://en.wikipedia.org/wiki/Killing_of_George_Floyd](https://en.wikipedia.org/wiki/Killing_of_George_Floyd) [4] [https://en.wikipedia.org/wiki/List_of_George_Floyd_protests_...](https://en.wikipedia.org/wiki/List_of_George_Floyd_protests_in_the_United_States) [5] [https://en.wikipedia.org/wiki/List_of_George_Floyd_protests_...](https://en.wikipedia.org/wiki/List_of_George_Floyd_protests_outside_the_United_States)

<comment><USER5> I'm not sure I understand your point. By questioning the choice of "protests" vs "riots", are you suggesting that the actions taken by a minority of those participating should be used as the header for the whole article? Because that would seem to be a much more biased way to record the events.

<comment><USER4> Per that one example.

The URL of the article being linked to was "[https://en.wikipedia.org/wiki/2020_Twin_Cities_riots"](https://en.wikipedia.org/wiki/2020_Twin_Cities_riots") (it has now been changed).

Compare the two: "Twin Cities whatever" used in the URL is a lot less political than "Goerge Floyd whatever". And the "riots" used in the URL is obviously something people were searching Google for.

Wikipedia could have used the more conservative name, and acknowledged that "the protests, also known as the "George Floyd Riots" (as is obvious from the URL)....

Don't get fixated on that example though - the whole article reads like it was written by the press team of a large corporation with a narrative to sell.

<comment><USER3> The current name of the article is "George Floyd protests in Minnesota", and first paragraph contains "The course of events in Minnesota has also been referred to as the Minneapolis riots".

I disagree riots is a more conservative name - this would imply an article solely about the riots, necessiting an additional article about the protests, one which, being written in a way as to minimize overlap, would probably be accused of downplaying the very much existing riots. Or at the very least balkanize coverage.

</comment></comment></comment></comment><comment><USER6> Unfortunately it's been years since I can trust any wikipedia article on anything political or social related. Sad thing is these issues now taint topics like games, movies, software. Not being from US makes it even harder to swallow.

I guess it's unavoidable though. Having your ideological group stand on such a popular resource is incredibly powerful propaganda. It's like having millions of dollars lying in the street, you can't expect people not to touch it.

<comment><USER7> I'm of the opinion that it's not just 'like' money. It is money. Firstly, there are people from 'community organizations' that get paid to AstroTurf as their full time job. Secondly, those organizations wouldn't get funding in the first place if some big company didn't somehow depend on their political action to maintain their profits.

</comment></comment><comment><USER8> I get that Jimmy Wales acted questionably towards Larry Sanger, but I'm always a little saddened that Sanger has become a rent-a-quote to criticise something truly amazing that he helped to create. Wikipedia is for all its flaws a remarkable achievement.

<comment><USER9> Sanger, from what I've read of what he has been saying, doesn't appear to deny the remarkable achivement of Wikipedia. He's arguing it is increasingly losing a sense of - a culture of - neutrality that helped to make Wikipedia possible.

</comment></comment><comment><USER10> Maybe Wikipedia shouldn't allow certain articles to be added/edited until things cool off a bit. Writing history out of anger is probably not the best idea.

<comment><USER11> They do that. Articles are often locked to a certain level (e.g. closed for edits from new users, or even all users) for a certain period of time if they relate to a controversial topic or unfolding news event.

</comment></comment><comment><USER12> Wikipedia has sadly been in this state of subversion for years. Eventually you get a mental bullshit-filter, just like you easily train yourself to ignore ads online etc.

<comment><USER13> Sadly yes and it is very counter productive from my perspective. It is no secret that there are some groups that just blame people on their skin color or sex or whatever else. They have been there since the dawn of the internet. But recently they seem to be holding a lot of people in some kind of mental prison.

I wouldn't even want to read an article about Floyd because it won't net me believable information. I will just say that he was a victim of state violence. That is more severe than violence between any groups and everyone in a society is responsible to a degree and that social media or the internet in general is a bad place for reverence.

</comment></comment><comment><USER14> That is an unexpectedly balanced and informative piece from Slate. There are two main assumptions underlying progressive politics: systemic oppression and exploitation. Social Justice Activism takes these assumptions to be true by default. Anti-capitalist Black Bloc anarchists, for instance, take any opportunity to destroy corporate property as a positive move to eliminate economic exploitation. They don't tend to revisit their assumptions.

Bypassing due process and the presumption of innocence does not promote justice. Perhaps news stories should include subscription links to be notified of the key due process events associated with the criminal investigation.

I would hope that thoughtful protest could be patiently delayed until due process is complete.

</comment><comment><USER15> I don't think the ambiguity of the notability requirement should be used as a critique of the neutrality policy.

</comment><comment><USER16> For the love of all this good, please do not let the poison of political correctness infect Wikipedia.

<comment><USER17> You think Wikipedia isn't already poisoned? I have sad news for you, my friend...

</comment></comment><comment><USER18> Pikiwedia sounds better than Wikipedia.</comment>


Creating Database-driven Websites with PHP&MySQL   - <AUTHOR> http://www.hosting.vt.edu/tutorials/phpmysql/

<comment><USER2> Maybe I'm missing something... but what is the reason for submitting 8 years old PHP tutorial here?</comment>


Ask pg:  avg = karma per submission or karma per comment? - <AUTHOR>

Just wondering. <comment><USER2>
        ; Ignore the most recent 5 comments since they may still be gaining votes.
        ; Also ignore the highest-scoring comment, since possibly a fluff outlier.

        (def comment-score (user)
          (aif (check (nthcdr 5 (comments user 50)) [len> _ 10])
               (avg (cdr (sort > (map !score (rem !deleted it)))))
               nil))

</comment><comment><USER3> Pretty damn sure they're combined, despite being representative of entirely different values.

You can get more considered stats using <http://searchyc.com></comment>