CocoaPods downloads max out five GitHub server CPUs - <AUTHOR> https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomment-193772935 <comment><USER3> Note how perfect that response from mhagger is. A clear, honest sounding assurance of what Github wants to deliver. A perfectly comprehensible description of what is the problem, and where it is coming from. And then suggestion how to fix it the project actually can work on, plus mentioning changes to git itself that Github is trying to make that would help. It not only shows great work going on behind the scenes (and if that is untrue, it at least gives me that impression, which is what counts), but also explains it in a great way.

<comment><USER4> I was astonished at how selfish/myopic/whatever alloy's response was.

To be blunt, you're abusing the shit out of SOMEONE ELSE'S product that you're not even paying for. Your first question shouldn't be to see what Github can do for you to make it so you don't have to make changes. You should be falling over yourself investigating all available avenues for reducing load.

It's an incredibly entitled way to think about things and I would have a real hard time employing someone who's first response was like this.

<comment><USER5> I don't know, it sounded to me like he just didn't totally understand what Github was saying. By the end of the thread, it seemed like everyone was agreeing. I wouldn't be comfortable using words like "selfish" to describe any of what I read.

I certainly don't think the barb about your willingness to employ people who write things on Github issues threads that you disagree with is helping anyone understand any part of this situation. I understand the urge to find ways to be emphatic about how much you disagree with things, and I _often_ find myself compelled to write lines like that, but I think they're virtually always a bad idea.

<comment><USER6> > I certainly don't think the barb about your willingness to employ people who > write things on Github issues threads that you disagree with is helping > anyone understand any part of this situation.

It seems to be one of HN's go-to insults. "Look at this person's behavior, I would never hire them," as if everyone wants to work at your startup.

<comment><USER4> I didn't say I disagree with his statement. I'm saying, maybe more implied, that I'm not going to hire someone who displays a lack of interest in finding a real solution to a real problem that has to do very much with what they're trying to build. And on top of that shows a serious streak of entitlement and a lack empathy towards the very service they're essentially abusing and not even paying for.

I wonder how many times CocoaPods has ruined someone's day/night on some GH team. I wonder how many dinners some mom or dad has missed with their kids because their service alarms are going apeshit. I don't think it's hyperbole to say that if you are a top 10 repo at Github, you are responsible for ruining individuals days and taking time away from their families if you are hammering the system.

Now, these are entirely my opinion and I'm not saying alloy is bad at what they do. I'm saying that is a collection of attitudes that I'm not going to put on my team.

<comment><USER7> I think it is a hyperbolic statement to say they're responsible for ruining people's days.

Let's think this through and ask ourselves a few question.

1\. Did they go out there way to do damage? 2\. Are they responsible for deciding the infrastructure and how well it can handle with load? 3\. Did they force said people to work at GitHub? 4\. Is the open source culture and hosting a major part of GitHub business plan? 5\. Are they responsible for staffing to ensure people are scheduled to work when work needs some?

</comment></comment></comment></comment></comment></comment><comment><USER8> From CocoaPods.org:

> CocoaPods is a dependency manager for Swift and Objective-C Cocoa projects. > It has over ten thousand libraries and can help you scale your projects > elegantly.

The developer response:

> [As CocoaPods developers] Scaling and operating this repo is actually quite > simple for us as CocoaPods developers whom do not want to take on the burden > of having to maintain a cloud service around the clock (users in all time > zones) or, frankly, at all. Trying to have a few devs do this, possibly in > their spare-time, is a sure way to burn them out. And then there's also the > funding aspect to such a service.

\--

So they want to be the go-to scaling solution, but they don't want to have to spend any time thinking about how to scale anything. It should just happen. Other people have free scalable services, they should just hand over their resources.

Thank goodness Github thought about these kinds of cases from the beginning and instituted automatic rate limiting. Having an entire end user base use git to sync up a 16K+ directory tree is not a good idea in the first place. The developers should have long since been thinking about a more efficient solution.

<comment><USER9> It seems particularly galling that their response to GitHub was to essentially throw their hands up and say "We don't want to change anything, fix it for us". I think GitHub had a near perfect response to this, they analyzed the problem, came up with a set of changes that could be made to help fix it (both short and long term), and pointed to steps they've taken to help out. CocoaPods on the other hand (or at least one of their developers) did not handle this particularly well. When presented with the evidence of why they were seeing slow responses and long queues and suggestions of how to fix it, they complained that they didn't _want_ to fix it and didn't have the time or resources to do so.

Honestly if I was GitHub, I'd be tempted to just increase the throttling on CocoaPods and call it done, it isn't their problem if the users of that project have a bad experience. GitHub has provided solutions to the problem, it's CocoaPods that's resisting implementing those solutions.

<comment><USER10> I think that's pretty unfair. It's really obvious that the initial reply didn't really understand what was going on, and what was being explained. A couple followup additional explanations later, the same dev grokked the problem, CocoaPods' responsibility for the problem, and outlined a list of how they're going to solve it. Seemed to me to be a pretty nice example of professional and helpful candor between GH and an OSS project working to figure out a long-term solution.

<comment><USER11> I don't know, maybe I'm being overly pessimistic here, but to me it just screams of backpedaling once they saw the reaction they were receiving in this thread. The position shifted from "it's the way we architected things, how can you fix this for us" to "okay, here's some things we can do" pretty quickly and dramatically when the HN thread went up and people were reacting to the response. Cocoapods is using Github resources for free, so the appropriate response from the start should have been what it eventually came down to, not pushing back on Github because they don't want to invest in an actual CDN solution. But, as I said, maybe I'm being overly pessimistic in my analysis here, that's just how it came off to me.

<comment><USER10> I get where you're coming from. I also had a similar initial reaction. However, as I read through the subsequent discussion, it began to read as though the commenter was really not grokking the problem—and, more importantly, what to do to fix it. I thought it was very impressive that none of the GH participants reacted like some of the HN commenters here. Instead, they showed a great deal of patience and restraint in fully explaining the technical details, offering actionable solutions, and keeping everything very civil and supportive. Then the same guy who sounded like he was possibly being a jerk came back and sounded totally different because he seemed to actually know what to do to fix his project. Maybe the CP commenter read this HN thread and reacted to it, but I'll admit HN is the last place I'd think of finding one of my GH issues discussed.

Perhaps I'm just being too charitable. Either way, the project rather rapidly seemed to come to the right conclusion and jump on board fixing their problem.

On a related note, I feel like this issue could be turned into a great teachable moment for OSS projects; one agH could use as a tech blog and guides for how to be a good citizen and avoid things that can make your project get rate-limited without you knowing.

<comment><USER11> Yeah, I really just think it went a bit too far in the other direction and overcompensated somewhat, which is what was giving me that view. The comment with the heart emoji really stood out to me as a "huh, this might be because of HN" since it basically touched on exactly what was being criticized in here, that they weren't really appreciating what GitHub was providing for free. That said, I can totally see it just that alloy realized it on his own and wanted to make it clear. It's just that the timing of it all and the fact that it's hitting the same point kind of led me to believe that it was a reaction.

Obviously, that's not to say the sentiment isn't genuine. The eventual conclusion makes it seem that yeah, they do appreciate what GH is providing and are trying to make it less strenuous on the servers to get a better experience all round. Making it work well is really in their best interests since the users are seeing a degraded experience until something can be done about it. Definitely also happy that the right conclusion was eventually reached.

</comment></comment></comment></comment></comment></comment><comment><USER12> This reply: [https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomm...](https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomment-193810378)

 _" Not having to develop a system that somehow syncs required data at all means we get to spend more time on the work that matters more to us, in this case. (i.e. funding of dev hours)"_

In other words, using github as a free unlimited CDN lets them be as inefficient as they like. Such as having 16k entries in a directory ( [https://github.com/CocoaPods/Specs/tree/master/Specs](https://github.com/CocoaPods/Specs/tree/master/Specs) ) which every user downloads.

Package management and sync seems to suffer really badly from NIH. Dpkg is over 20 years old and yum is over a decade old. What's up with this particular wheel that people keep reinventing it seemingly without improvement?

<comment><USER13> Care to substantiate that last paragraph? Are you really suggesting OS X users use yum?

<comment><USER14> Actually it seems very likely that one or more of the popular linux distro package manager ecosystems would fare well on other OSs. Arch Linux's pacman was ported to Windows, e.g..

<comment><USER15> pacman has been ported to OS X a few times

      https://bbs.archlinux.org/viewtopic.php?id=53960
      https://bbs.archlinux.org/viewtopic.php?id=122544

</comment></comment></comment></comment><comment><USER16> I help run Mozilla's version control infrastructure and the problems described by the GitHub engineer have been known to me for years. Concerns over scaling Git servers are one of the reasons I am extremely reluctant to see Mozilla support a high volume Git server to support Firefox development.

Fortunately for us, Firefox is canonically hosted in Mercurial. So, I implemented support in Mercurial for transparently cloning from server- advertised pre-generated static files. For hg.mozilla.org, we're serving >1TB/day from a CDN. Our server CPU load has fallen off a cliff, allowing us to scale hg.mozilla.org cheaply. Additionally, consumers around the globe now clone faster and more reliably since they are using a global CDN instead of hitting servers on the USA west coast!

If you have Mercurial 3.7 installed, `hg clone [https://hg.mozilla.org/mozilla-central`](https://hg.mozilla.org/mozilla- central`) will automatically clone from a CDN and our servers will incur maybe 5s of CPU time to service that clone. Before, they were taking minutes of CPU time to repackage server data in an optimal format for the client (very similar to the repack operation that Git servers perform).

More technical details and instructions on deploying this are documented in Mercurial itself: [https://selenic.com/repo/hg/file/9974b8236cac/hgext/clonebun...](https://selenic.com/repo/hg/file/9974b8236cac/hgext/clonebundles.py). You can see a list of Mozilla's advertised bundles at [https://hg.cdn.mozilla.net/](https://hg.cdn.mozilla.net/) and what a manifest looks like on the server at [https://hg.mozilla.org/mozilla- central?cmd=clonebundles](https://hg.mozilla.org/mozilla- central?cmd=clonebundles).

A number of months ago I saw talk on the Git mailing list about implementing a similar feature (which would likely save GitHub in this scenario). But I don't believe it has manifested into patches. Hopefully GitHub (or any large Git hosting provider) realizes the benefits of this feature and implements it.

<comment><USER17> Wow, this is pretty cool. Reminds me of the performance optimizations Facebook has done with Mercurial: [https://code.facebook.com/posts/218678814984400/scaling- merc...](https://code.facebook.com/posts/218678814984400/scaling-mercurial-at- facebook/)

Mercurial was designed to be easy to extend, and it shows.

<comment><USER18> Git was created and designed to support Linus' workflow when developing the Linux kernel.

Hg was designed to be a DVCS system.

</comment></comment></comment><comment><USER19> Wow, really impressive response from GitHub. The right amount of technical detail coupled with balanced tone--halfway between "we support you" and "you make us crazy."

One correction to the post title: it's not maxing five nodes, but five CPUs.

<comment><USER20> Yeah, 5 cpu's is an order of magnitude difference. ;)

<comment><USER21> Wait, so the CPU isn't the big white tower sitting under my desk?!

<comment><USER22> According to my mother, that's "The hard drive".

</comment></comment></comment></comment><comment><USER23> I keep coming back to point #4 - who ever thought that 16k objects in a single directory would be a good idea? Ever since FAT that's been a bad idea, and while modern FSes will handle it without completely melting down it's still going to cause long access operations on anything to do with it.

Even Finder or `ls` will have trouble with that, and anything with * is almost certainly going to fail. Is the use-case for this something that refers to each library directly, such that nobody ever lists or searches all 16k entries?

<comment><USER24> I do think that your last sentence is the answer: if you're using a package manager instead of working with the directory heavily, this isn't a visible problem which is going to motivate people to work on it.

The other side to consider: "one directory per package" is a very simple policy and it feels right in many ways to people (e.g. Homebrew has a similar structure because it's a natural fit for the domain). If the filesystem and basic tools like ls work just fine (which is certainly the case on OS X, where even "ls -l" or the Finder take less than a second on a directory of that size), isn't there a valid argument that the answer should be some combination of fixing tools which don't handle that well or encouraging people to learn about things like `find` instead of using wildcards which match huge numbers of files?

<comment><USER23> One directory per package is completely sensible, just not all in one bunch. It's even fine if the mapping is to a flat namespace at something like the HTTP level - I can mod_rewrite /abcdefg to /a/b/c/abcdefg no problem. My only objection is to file- or directory-level structures that are this flat. I might be mentally deficient, but I can't even process anything that's structured this way.

As loathe as I am to admit anything about Perl is good, CPAN got this right. 161k packages by 12k authors, grouped by A/AU/AUTHOR/Module. That even gives you the added bonus of authorship attribution. Debian splits in a similar way as well, /pool/BRANCH/M/Module/ and even /pool/BRANCH/libM/Module/ as a special case.

Tooling can be considered part of the problem in this case. Because the tooling hides the implementation, nobody (in the project) noticed just how bad it was. I hadn't seen modern FS performance on something of this scale, apparently everything I've worked with has been either much smaller or much larger. Ext4 (and I assume HFS+) is crazy-fast for either `ls -l` or `find` on that repo.

It seems like tooling is part of the solution as well, but from the `git` side. Having "weird" behavior for a tool that's so integral to so many projects scares me a little, but it's awesome that Github has (and uses) enough resources to identify and address such weirdness.

<comment><USER25> My (perhaps naive) thoughts on this are - suppose a 16k-packages-in-one- directory solution were just as fast as a 16k-packages-sharded-by-prefix (the CPAN solution), then the former is conceptually simpler and so should be preferred. And the fact that you can mechanically transform one structure to the other means that the filesystem (or git) should be able to transparently do it for you (eg use the sharded approach as a hidden implementation, while the end user sees a flat directory). This seems to be similar to what ext4 does ([https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#Hash...](https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#Hash_Tree_Directories)).

<comment><USER26> The obvious question is how would you implement that. You might argue (as you should) that git has closer semantics to a filesystem than version control. But actually implementing this sharding would require git be a kernel module. Hardlinks and softlinks won't save you because they are both still dentries and thus have the same performance pathology. Maybe you could do it with fuse, but what have you gained by making your version control system even more annoying to use?

</comment></comment></comment></comment></comment><comment><USER27> The criticism against CocoaPods here seems awfully harsh.

Think about it from their perspective. GitHub advertises a free service, and encourages using it. Partly it's free because it's a loss leader for their paid offerings, and partly it's free because free usage is effectively advertising GitHub. CocoaPods builds builds their project on this free service, and everything is fine for years.

Then one day things start failing mysteriously. It looks like GitHub is down, except GitHub isn't reporting any problems, and other repositories aren't affected.

After lots of headscratching, GitHub gets in touch and says: you're using a ton of resources, we're rate limiting you, you're using git wrong, and you shouldn't even be using git.

That's going to be a bit of a shock! Everything seemed fine, then suddenly it turns out you've been a major problem for a while, but nobody bothered to tell you. And now you're in hair-on-fire mode because it's reached the point where the rate-limiting is making things fail, and nobody told you about any of these problems before they reached a crisis point.

It strikes me as extremely unreasonable to expect a group to avoid abusing a free service when nobody tells them that it's abuse, and as far as they know they're using it in a way that's accepted and encouraged. If somebody is doing something you don't like and you want them to stop, you have to tell them, or nothing will happen!

I'm not blaming GitHub here either. I'm sure they didn't make this a surprise on purpose, and they have a ton of other stuff going on. This looks like one of those things where nobody's really to blame, it's just an unfortunate thing that happened.

(And just to be clear, I don't have much of a dog in this fight on either side. My only real exposure to CocoaPods is having people occasionally bug me to tag my open source repositories to make them easier to incorporate into CocoaPods. I use GitHub for various things like I imagine most of us do, but am not particularly attached to them.)

<comment><USER28> I entirely agree with this. GitHub gets so much advertising + community from open source projects like this.

Also, I'm amazed this is even a problem. 5 CPUs is not a lot in the scheme of things (even if they mean physical instead of cores). TBs of bandwidth are also virtually free compared to a company the size of Github.

Even better: they are getting basically real world loadtested for free and finding loads of pain points, which may hit paying customers.

Unless I'm missing something, fire more metal at the problem. Many companies would love to be able to have every single cocoapod user (which is nearly every iOS developer) have to type github.com into their terminal for the cost of a bunch of servers + some bandwidth.

Pretty strange, unless this is hitting some really bad area of their service that can't easily be scaled out of (but i would be surprised)

<comment><USER29> >>Even better: they are getting basically real world loadtested for free and finding loads of pain points, which may hit paying customers.

I think their point is that it's using the system in a way that isn't intended or desired. How does that count as "real world" load testing?

And by that logic, shouldn't anybody who gets hit with a DoS attack just say "thanks"? It's tons of free load testing on your network infrastructure, and you'll definitely find some pain points.

</comment></comment></comment><comment><USER30> It's totally reasonable to host your code on github and to build a package manager that loads the content of a package from it's github repo.

What seems insane is to use a single github repo as the universal directory of packages and their versions driving your package manager.

There's a reason rubygems has their own servers and web services to support this use case for the central library registry, even if the source for gems are all individually projects hosted on github.

<comment><USER31> I assume they modelled it after Homebrew, which has been working fine doing exactly that for the last 7 years.

That only has 3,000 packages vs 15,000 for CocoaPods or 115,000 for RubyGems.

<comment><USER32> In case somebody is interested in such figures (I certainly am) - NPM has 249,838 as of today [0].

[0]: [https://www.npmjs.com](https://www.npmjs.com)

</comment></comment></comment><comment><USER33> > Scaling and operating this repo is actually quite simple for us as CocoaPods > developers whom do not want to take on the burden of having to maintain a > cloud service around the clock (users in all time zones) or, frankly, at > all.

The CocoaPods developers seem to be missing the entire point of git: it's a _distributed_ revision control system.

Setup a post-recieve hook on Github to notify another server, that is setup with a basic installation of git, to pull from Github so as to mirror the master repo. Then, have your client program randomly choose one of these servers to pull from at the start of an operation. Simple load balancer to solve this problem.

<comment><USER20> Rackspace is also known to sponsor significant resources for larger projects whom ask nicely. GlusterFS is one I used to be involved with doing this, and there are definitely others.

If CocoaPods reach out to Rackspace and/or other hosting providers, there's a decent chance they'll be able to pull together a good solution. :)

The downside though, is they'll need to figure out some way to keep it monitored/maintained. :/

<comment><USER34> Last I checked, Rackspace wasn't accepting any more projects.

</comment></comment></comment><comment><USER35> I find it amusing how GitHub's contact[1] form has (probably a recent addition):

> GitHub Support is unable to help with issues specific to > CocoaPods/CocoaPods.

\---

[1]: [https://github.com/contact](https://github.com/contact)

<comment><USER36> I think that contact page remembers the last repo you visited. I went to it in incognito mode and is wasn't there.

That's a pretty neat feature!

</comment></comment><comment><USER37> CocoaPods (and Homebrew) mainly exist because of a lack of tooling in the typical Apple ecosystem. So I would blame Apple for not supporting the community with money or tooling. Letting GitHub with its limited amount of funding pay the bill isn't a nice move. Apple dev relations should throw some money at GitHub so they can provide some dedicated resources or offer to pay the cost of other solutions (like a 3rd party CDN/AWS/Google Cloud/…).

<comment><USER38> CocoaPods exists because developers want to learn how to "build apps" but lack the resources to intelligently include and link to 3rd party code in their projects. CocoaPods doesn't enable anything not otherwise configurable via git submodules and Xcode project hierarchies / build settings.

Therefore, it's not Apple's problem. In fact, I've talked to a non-trivial amount of engineers (both in Cupertino and long time Cocoa devs) that disapprove of the shortcuts that Cocoapods takes all over, software architecture be damned. Reasonable parties can agree to disagree, but I do include 3rd party framework inclusion without a dependency manager as an interview screen for prospective iOS hires.

Since you mention developer relations, I'll assume you're not actually arguing that this is Apple's technical responsibility, but that they should throw around some $ to grease the wheels to make dependency inclusion better. As a platform vendor, funding hosting costs for some project that you don't agree with just to "support the community" is a bad idea. Better idea is to allocate resources to setup a structure that can fix the issue in a technically agreeable way while also benefitting from the independence a FOSS project provides. In doing so, you are correct that it'd be preferable for Apple to fund/use well-known FOSS standards, such as Github.

In conclusion, Apple should setup a FOSS project to address the current inconveniences associated with third party package inclusion and should involve and pay Github somehow.

Oh wait...[https://github.com/apple/swift-package- manager](https://github.com/apple/swift-package-manager)

<comment><USER39> Apple decided to block dylibs from the start of the iOS app store, and I think that friction point from how things are usually done in OSX, C & C++ land before iOS is what started the entire cocapods thing in the first place. The replacement with dynamic frameworks came about 8 years too late in ios 8.

</comment></comment></comment><comment><USER40> I've always found Github's business model interesting. What if a massive open- source organization (e.g. Fedora, Apache) decided to use it for all of their development, integrating it with continuous builds and all the associated pulls. Of course this isn't likely to happen for a number of reasons, but there are large open source projects that could put a significant load on their infrastructure if they chose to use Github as their main code versioning system.

<comment><USER41> > What if a massive open-source organization (e.g. Fedora, Apache) decided to > use it

This one is pretty big. [https://github.com/torvalds/linux](https://github.com/torvalds/linux)

<comment><USER42> Sure but it's just the kernel, and that's just a mirror. Linus does not use Github to manage kernel development. In fact he's been vitriolic in the past about how Github does pull requests.

I wonder how much traffic the Github Linux repo gets. Seems to me that people who want to use Linux, will go get a distro instead. And people who want to develop the kernel, will follow the kernel development process (which doesn't rely on Github).

<comment><USER41> For a period of time when Kernel.org was breached, GitHub was _the_ repository for Linux. [1] I remember reading a review of GitHub by him shortly after. He did not like how Pull Requests or patches worked on GitHub. I'd link it but I'm having trouble finding it.

1\. [http://www.theregister.co.uk/2011/09/06/linus_torvalds_dumps...](http://www.theregister.co.uk/2011/09/06/linus_torvalds_dumps_kernel_for_github/)

</comment></comment></comment></comment><comment><USER43> This bug report is a great step in the direction for GitHub. As of this comment there are 3 different GitHub staff members responding and providing feedback to the CocoaPods team. From the previous "Dear GitHub" messages and responses, this seems like perfect community involvement.

</comment><comment><USER44> I have been seeing this trend of GitHub getting "abused" for purposes other than hosting source code.

\- My school uses GitHub to host and track our software engineering project (which still can be argued as OSS).

\- People using GitHub issue system as a forum.

\- Friends uploading pdfs to GitHub.

\- Recently people posted on HN about using GitHub to generate a status page.

I think this is a really bad trend and people should stop doing that.

<comment><USER45> The school example and friends uploading pdfs to GitHub are both uses that GitHub encourages.

Using GitHub Issues as a forum and a source for generating status pages are both ok from a use/abuse perspective, but you may not have the best experience since that isn't what Issues is intended for.

<comment><USER46> The issue tracker on new repos has a 'question' tag by default, so Github are gently encouraging using issues as a forum. Though my inner cynic says that makes sense for them - issues tie a project to Github more than the git repo itself does.

It should be fine to come up with new ways to use Github, as long as it's not causing excessive load.

</comment></comment></comment><comment><USER47> GitTorrent: [http://blog.printf.net/articles/2015/05/29/announcing- gittor...](http://blog.printf.net/articles/2015/05/29/announcing-gittorrent-a- decentralized-github/)

Imagine a world where GitTorrent is fully developed, includes support for issue tracking, and has a nice GUI client that makes the experience on-par with browsing github.com.

I mention this not as an "Everybody bail out of GitHub and run to GitTorrent!!!" sort of statement, because I believe GitHub's response here was excellent and confidence inspiring. But it's an unnatural relationship for community supported, open source projects to host themselves on commercial platforms such as GitHub. GitHub primarily hosts them to promote its business. That's not necessarily a bad thing, but it results impedance mismatches like demonstrated here.

That isn't to say that a mature GitTorrent would replace GitHub. Rather, I envision GitHub becoming a supernode in the network, an identity provider, and a paid seed offering, all alongside their existing private repo business.

Honestly, once I scrape a few projects off my plate, I'm inclined to dive into GitTorrent, see where it's at in development, and see if I can start contributing code. It just seems like such a cool and useful idea.

<comment><USER26> My main issue with Free Software projects using GitHub is that it's proprietary, not that it's commercial. Admittedly, I think GitTorrent is a really cool idea, but I'm wondering if a distributed filesystem might be a more elegant solution than using both BitTorrent and Bitcoin.

</comment></comment><comment><USER48> I've never really understood CocoaPods. Dragging a framework into Xcode was never much trouble, and the amount of 3rd party libraries in a OS X / iOS project ought to be fairly small, so the gains are trivial.

The potential downsides seem much more annoying. Do you really want to have your dependencies on an overloaded central server somewhere?

<comment><USER49> Until recently iOS only supported statically linked libraries which could lead to issues if you needed to use multiple components that had a shared dependency that you needed to upgrade for one reason or another. You couldn't touch the embedded version. Additionally there was no native package manager which made sharing libraries a clumsy affair. Cocoapods makes both cases easier.

<comment><USER50> This is correct. Starting with iOS 8 they finally allowed linking against custom frameworks which is why Carthage is now becoming much more popular. CocoaPods solved a critical problem of getting dependencies linked in, while creating a new problem of having your xcodeproj file and build settings be managed by them. I'll be happy enough to drop them for new projects going forward.

<comment><USER51> > Starting with iOS 8 they finally allowed linking against custom frameworks

I don't keep up with iOS development enough to know if anything has changed with respect to static/dynamic linking in iOS 8, but it has always been possible to use custom frameworks in iOS (eg, frameworks you build yourself, unless the community has another definition for 'custom framework').

The framework directory structure is a bit unorthodox, but it's really just your statically built library (absent any '.a' suffix) alongside any header files in a Headers folder. Again, not sure if this has changed with any support for dynamic linking.

<comment><USER50> Sorry you're right I should have been more explicit, starting in iOS 8 you can use dynamically linked frameworks.

</comment></comment></comment></comment></comment><comment><USER52> What an unusually reasonable discussion. good on everyone.

</comment><comment><USER53> I love how this was like _the_ perfect storm of things that could go wrong, and how it seems like mhagger is just amazed more than anything else.

<comment><USER54> Hardly. Do they hit edge cases? Yes! Is using GitHub for your CDN a dick move and going to cause problems, regardless? Yes!

The response from mhagger is unnecessarily apologetic, and I predict we'll see an official update from GitHub on this soon.

<comment><USER55> > Is using GitHub for your CDN a dick move and going to cause problems, > regardless? Yes!

I don't know about that. Both oh-my-zsh[1] and emacs prelude[2] use git repos as their code distribution mechanisms, and that works really well. I think the real issues here are exactly what is called out in the issue: poor usage of git, and poor directory layout.

[1] [https://github.com/robbyrussell/oh-my- zsh](https://github.com/robbyrussell/oh-my-zsh) [2] [https://github.com/bbatsov/prelude](https://github.com/bbatsov/prelude)

<comment><USER54> They are using GitHub for their intended purpose, hosting their code. That is perfectly fine.

What is not perfectly fine is using GitHub as your package host, CocoaPods/Specs is the equivalent of Debians APT using one big GitHub repo to host all their packages. It has 92567 commits and 6872 contributors.

<comment><USER56> OTOH Homebrew also uses github as its package host and has a respectable 62000 commits and 5600 contributors and github seems to be just fine with it.

The big differences seems to be in the way they do their thing: I'm reasonably sure homebrew just git clones then updates the local repository normally[0], it has "only" 2500 files in Library/Formula, and because of its different subject it is way less write-active, CocoaPod has 1k commits/week which look to be increasing pretty much constantly, homebrew is around 350 with ups and downs.

Also not sure it matters, but homebrew has lots of commits updating existing formula, cocoapod changes are almost solely addition (publishing a new version of a package adds a new spec and doesn't touch the old one)

[0] which is exactly the bread and butter of github

<comment><USER57> The fact that Github added an API specifically to reduce the server load from Homebrew suggests that it wasn't "just fine". One of the Homebrew maintainers works for GH so they just had a much more direct route to solving the problem than with CocoaPods.

<comment><USER58> In this case it's actually more that there's a Homebrew maintainer who works for GitHub (me) who has been working on a bunch of improvements to Homebrew's update system (in my spare time). The desire for the API came from Homebrew's side rather than GitHub's and, as it reduces load for GitHub, it was a net win for both parties.

</comment></comment></comment></comment></comment></comment></comment><comment><USER59> I love GitHub's response, but I would urge the project more strongly to use modern CDN solutions. CDNs are dirt cheap and incredibly powerful nowadays, for the data sizes that we're talking about here.

<comment><USER60> How would you define "dirt cheap"? One of the most popular CDN's out there (Akamai) charges $3,500/month for 10TB/month. Who's going to foot that bill? :)

<comment><USER59> That's a list price that they charge those who don't care to negotiate.

CloudFlare starts at $0 and doesn't meter/charge for bandwidth. CloudFront charges 9 cents per GB and is integrated with other AWS APIs (which can be very useful). Both those solutions could be managed with a donation pool, I would try the CloudFlare free tier first.

<comment><USER61> CloudFlare's terms of service specifically forbid using it as a file hosting service.

</comment></comment></comment></comment><comment><USER62> Rust's cargo does something similar, however it looks like they were much more conscious of git-scalability (ex: limiting the directories in a single level, only appending lines to files to make diffs small).

[https://github.com/rust-lang/crates.io-index](https://github.com/rust- lang/crates.io-index)

<comment><USER63> For what it's worse, both of these characteristics indeed weren't an accident.

At the time, I wrote a script that hammered git commits into a repository using different strategies and looked at what the git repository would look like after 100,000 and a million commits. The "one version per file, nested in a flat structure" had serious issues.

There may still be scaling limits with the Cargo approach, but if we reach them, we have plans to create a new registry with a new initial commit and let the old registry age out, then rinse/repeat. At the moment, we haven't hit limits yet (with about 1/3 of the packages that Cocoapods has).

</comment></comment><comment><USER64> Amazing to me that people create inefficient systems like this and then complain when they are rate limited.

</comment><comment><USER65> Using Github as your CDN is a dick move. Kudos to GH for not banning the project out-right, but CocoaPods should seriously reconsider what they are doing.

</comment><comment><USER66> As a current maintenance developer/systems guy I can definitely feel the tempered annoyance from mhagger here. It's definitely nice to not remind yourself that it's not only your set of recurring issues in front of you that people have to deal with.

</comment><comment><USER67> Go's package manager, `go get`, also downloads from GitHub. I don't know the details of how `go get` and CocoaPods work, but I would be interested in learning why one is unscalable and the other seems to work.

<comment><USER68> Because cloning the dependency is not the issue here. CocoaPods is keeping its index in a git repo that is updated by the user as a way to get the latest index. This is the repo that incur a lot of requests from everyone.

Go get on the other hand doesn't keep any index. It just uses the url to download the dependency because of the mapping "url==project name" that exists with go projects.

</comment></comment><comment><USER69> In fact, they are maxing out five _CPUs_ \- not five nodes, big difference.

</comment><comment><USER70> I found the solution humorous. Ironically shallow clones are causing the problems, so fetch the max :)

$ git fetch --depth=2147483647

<comment><USER71> The problem is that the initial clone is depth=1, then subsequent fetches are depth=MAX. If you did clone depth=MAX in the first place it is faster to serve.

Shallow (depth=1) can be converted into a full clone with the above.

</comment></comment><comment><USER72> Has any consideration been given to Bintray[1] as an alternative store for this stuff?

1 - [https://bintray.com/](https://bintray.com/)

</comment><comment><USER73> Reading the issue, it seems that one of the problems is a single directory with lots and lots of files in it, which is something of a pathological case for Git. Now, this could be "fixed" by splitting the files in that directory into subdirectories, but the one giant directory will still exist in all the past commits. So would this actually fix anything, or just keep it from getting worse?

<comment><USER26> Filesystems also have the same pathology, which is why git's object store is of the form prefix/object.

<comment><USER73> In a filesystem, you can address the problem by reorganizing the directory structure. And you can do the same for future Git commits. But unless I'm mistaken, that colossal directory is stuck in the git history unless you actually rewrite that history, which would require everyone who cloned the repo to either rebase or re-clone the repo. Maybe it's not a problem because the cost is only paid when one checks out one of those old commits, which would happen rarely?

</comment></comment></comment><comment><USER74> Funny thing is that the repo is only 7 MB gzipped (or 4 with lzma). Not that surprising, since it's just metadata of course. They say they have about 1 million fetches/clones per week, so that would make about 16 TB per month. I'm not sure how much bandwidth costs, but wouldn't some sympathetic CDN host that for free, since they're OSS?

</comment><comment><USER75> I was up until 2am last night trying to publish my Pod [1] and Github kept timing out.

I had no idea it was just CocoaPods repo because my other repos were working fine. I accepted defeat, went to bed and everything was working great in the morning.

[1] [https://github.com/soheil/SwiftCSS](https://github.com/soheil/SwiftCSS)

</comment><comment><USER76> I would be interested to know what are the other top GitHub repositories. Afaik the Nix package manager uses a similar model (using a GitHub repo as a database), however they periodically release snapshots and the default configuration uses those instead of git.

<comment><USER77> I'm not sure how DefinitelyTyped works, but their repository [https://github.com/DefinitelyTyped/DefinitelyTyped](https://github.com/DefinitelyTyped/DefinitelyTyped) is huge and may be they use that github repository as a single distribution point.

<comment><USER78> People interact with DefinitelyTyped in a set of ways, from currently most common to least common:

1\. NuGet packages, which are built from the GitHub repository but then redistributed over a non-Github CDN (NuGet's)

2\. tsd management tool ([https://github.com/Definitelytyped/tsd](https://github.com/Definitelytyped/tsd)), which looks like it prefers Github's CDN raw URLs rather than full/shallow git clones

3\. typings management tool ([https://github.com/typings/typings](https://github.com/typings/typings)) with "ambient" typings searches (`--ambient` flag), which I believe also prefers Github's CDN raw URLs, as it essentially forked from TSD

That said, it's past time to move beyond the giant huge DefinitelyTyped repo, and I for one heartily recommend people migrate to typings which has better support for NPM and other module and package management systems, as well as smaller unit/module-focused Github repositories.

</comment></comment></comment><comment><USER79> Another reason I consider [https://github.com/Carthage/Carthage](https://github.com/Carthage/Carthage) a better solution of the dependency management problem.

<comment><USER80> And [https://github.com/apple/swift-package- manager](https://github.com/apple/swift-package-manager) once it's ready, which is similar to Carthage in structure (decentralized!).

<comment><USER81> Swift Package Manager will also have a centralised package index in the future, here's a quote from their [package manager proposal][1]:

> We would like to provide a package index in the future, and are > investigating possible solutions.

[1]: [https://github.com/apple/swift-package- manager/blob/master/D...](https://github.com/apple/swift-package- manager/blob/master/Documentation/PackageManagerCommunityProposal.md)

</comment></comment></comment><comment><USER82> When he says approaches similar to 'other packaging systems', which ones is he referring to? I can see why this is a bad approach but am unfamiliar with what would be considered a better practice (outside of just hosting a .tar on CloudFront).

<comment><USER30> RubyGems.org has their own web server and web services for publishing library versions and to allow the clients to fetch libraries and query the universal registry.

<comment><USER83> Same goes for almost any language-specific package manager you could name: Rust/Cargo, Clojure/Clojars, Node/npm, Elixir/hex, Python/PyPi.

No matter which way you slice it, what CocoaPods is doing is a bit daft, especially at their scale.

<comment><USER84> Small note: Cargo _does_ use an index (not the source code), in git, on GitHub. However, we're already doing the directory layout that they recommend in-thread, so we shouldn't have this specific problem.

</comment></comment></comment></comment><comment><USER85> Just last night, all my pod installs were timing out after ~30ish minutes. That explains it.

<comment><USER86> Should we infer from your behavior that such a timeframe is fairly normal for cocopod? If I ran an "npm i" (or "pip install", etc.) that didn't respond for a minute, I would suspect a problem and kill it. How can any development process that takes longer than that be practical?

<comment><USER85> Indeed, I was killing and restarting the process for a while, trying to get it to be more verbose, etc. Eventually I decided to try letting it run, and that didn't work either.

I don't know what's normal; last night was one of my first iOS projects.

</comment></comment></comment><comment><USER87> I just installed Cocoapods last night and tried to clone down the repo. It took about 5 minutes and I thought to myself "Is my 150MB/s connection slow?" This definitely clears up what was going on.

</comment><comment><USER88> Why aren't the packages distributed? Composer is incredibly distributed and likely doesn't cause nearly the same headaches for GitHub.

Seems like a poor design decision on the CocoaPods side.

<comment><USER89> What package? My project may have a dependency on a specific commit that's between point releases.

The problem is not packages, it's the index, containing 16000 subdirectories,

<comment><USER88> So the packages themselves aren't part of the repository?

<comment><USER74> No, only references to the repositories themselves are. The total Specs repo is only 300 megabyte, which comes to about 18 kilobyte per package on average.

It's still absolutely ludicrous to use git for it, or worse, to bother github with it.

</comment></comment></comment></comment><comment><USER34> It's difficult to run an open source project on a budget of $0. You're always relying on the goodwill of others.

</comment><comment><USER90> While I do not have much knowledge on the subject, why not using rsync?

</comment><comment><USER91> hopefully we can now move to using real artifact repositories.

</comment><comment><USER92> tl;dr: "Using GitHub as your [free-of-charge] CDN is not ideal, for anybody involved."

</comment><comment><USER93> Why is everyone talking about CocoaPods where the title is CoacoaPods anyway? :)

<comment><USER94> The title was misspelled and has been fixed. It's CocoaPods right on Github.

</comment></comment><comment><USER95> Only the Apple development community would think it's OK to have 16,000 subdirectories in one place and abuse GitHub as a free CDN instead of putting some actual effort in and develop their own repository infrastructure - <USER2> know, like almost every other package manager in existence.

<comment><USER96> It should be "Apple or Ruby development community", homebrew is using similar tech stack distributing the `Formula`.

<comment><USER95> I guess it's no coincidence that those two hispter communities are closely related.

</comment></comment></comment><comment> Const-me I don't think GitHub acts wisely here.

Short term sure, they're doing the right thing, implementing a nice way to manage the free rider problem without hurting them too much.

But long term it's different.

Financially, one average programmer = $80k/year, one average cloud server = $4k/year. And, GitHub has hundreds of millions of venture capital. More than enough to provision a few more servers, even if they will be installing new servers just for those pods.

The way they act now will lead to someone will develop a decentralized git+torrent hybrid. When that happens, sure, those pods will no longer consume precious GitHub's resources. However, for the rest of the github users, there will be no reason to stay on GitHub either.

<comment><USER26> The way CocoaPods is using git is bad, so they'd probably run into some problems no matter how they host it. If you know anything about filesystems, they start to have pathologies when you have many entries in the same directory (which is why git has it's object store in the format prefix/object). In addition, it looks like git has it's own pathologies with such large numbers of dentries. Having 16000 entries in your "specs" directory is not a good idea. No matter how you store it.

Not to mention that "just buy another server for this _one_ project" sounds like something CocoaPods should _pay_ for.

<comment> Const-me >The way CocoaPods is using git is bad

Very likely true, but I don't see how's that related.

>they start to have pathologies when you have many entries in the same directory

Only true for inefficient filesystems like FAT.

For NTFS, 16k entries is nothing, the performance fill start to degrade (due to directory fragmentation) at around 100k entries: [http://stackoverflow.com/a/291292/126995](http://stackoverflow.com/a/291292/126995)

>"just buy another server for this one project" sounds like something CocoaPods should pay for.

I don't think that's how 21 century economy works in this case.

Github's value is likely between $0.75B and $2B.

Bad PR caused by this story will exceed 10 years TCO of that extra server.</comment></comment></comment>


Show HN: My first xbox 360 game - piniq - <AUTHOR>

I think I mentioned in the past that I was working on an Xbox 360 game, and now, it's finally here! Hope some of you guys check it out and have fun with it. I had fun making it. Also, it's most fun playing with another human.
If you have any feedback after playing it, feel free to share. I intend to patch it after I find out what people think.
http://marketplace.xbox.com/en-US/Product/piniq/66acd000-77fe-1000-9115-d80258550a04 <comment><AUTHOR> Here's the clickable link.

[http://marketplace.xbox.com/en- US/Product/piniq/66acd000-77f...](http://marketplace.xbox.com/en- US/Product/piniq/66acd000-77fe-1000-9115-d80258550a04)</comment>


Texas Governor Talks About Taxes [2008 YouTube] - <AUTHOR> http://www.youtube.com/watch?v=0HdbBl9Ss94

<comment><AUTHOR> Just for a heads up, here is a 7 minute candid talk, by the Governor of Texas on you tube from 2008, about how Texas views taxes including its sales tax.

With all of the discussion about Rackspace and etc... servers creating a nexus to the state of Texas for e-commerce sales tax collection, I thought since we are in the Austin area I would give everyone a heads up on some of the Governors thinking behind it.

The Governors name is Rick Perry, and he is currently up for re-election right now, so he might be more willing to listen to your views now than later. Texas is very business friendly, and as Californians that is why we chose to locate here.

Austin is the heart of the Texas start-up community and the government here is well aware of it.

If you are concerned about the Red Tape or cost impact of the "Sales Tax" Nexus situation, here is the Governors contact page where you can send him a message and any suggestion you may have: <http://governor.state.tx.us/contact/></comment>


Intuitive differences: when to agree to disagree - <AUTHOR> http://lesswrong.com/lw/19v/intuitive_differences_when_to_agree_to_disagree/#more

<comment><AUTHOR> Kaj's article's points on intuitive differences, especially his specific example of unconscious, intuitive theories of modeling, seem to be very relevant to differences of opinion on how to go about programming. See the recent debate on Joel Spolsky's "Duct Tape

<http://news.ycombinator.com/item?id=840523> <http://news.ycombinator.com/item?id=848263> <http://news.ycombinator.com/item?id=848677> <http://news.ycombinator.com/item?id=849462></comment>


Tell HN: We're bringing back the "Work at a Startup" event on Saturday July 28 - <AUTHOR> Back in 2010 and in 2012, YC organized a "Work at a Startup" event, for founders of YC companies and prospective hires to meet each other in person. It got great feedback and we wanted to keep doing it, but we got stretched too thin for a while. This year, a bunch of us revived the project. Our goal is to fundamentally improve the experience both of joining a YC startup and of being an early employee in general. You may have seen some of the recent discussions about this, including a big discussion last week about concrete ways to improve things for early startup employees (see links below). As part of this, we've decided to bring back an expanded version of the original "Work at a Startup" event. That's what we're announcing today.
On Saturday July 28, we'll host our third Work at a Startup Expo: https://workatastartup.com/expo. We'll begin with a talk answering the most common questions about working at a startup. Then we'll have a series of rapid-fire presentations by 35 YC companies currently hiring engineers. After the event, we'll have an open house where you can talk one-on-one with the founders of any company that interested you.
New this time: we'll have a hardware demo area where companies building physical products can show them off. YC has more hardware companies than ever now, building everything from satellites to industrial robots.
For those not in the Bay Area, we will live stream all the videos, and if people are interested we can have an ongoing HN thread to discuss them as well.
If you're interested in attending, please apply at https://workatastartup.com/expo. Any questions or ideas, please comment below!
Links to previous discussions:
https://news.ycombinator.com/item?id=17286939
https://news.ycombinator.com/item?id=15916350
https://news.ycombinator.com/item?id=3676578
https://news.ycombinator.com/item?id=1346103
p.s. That recent thread about the pros and cons of working at a startup was one of the best discussions I've ever seen on Hacker News. I went through it carefully, tried to make sure that every comment got attention, and compiled a list of the main points, which got a thorough discussion at the last YC partner meeting. We're going to develop some substantive programs in response. It will take some time to responsibly make changes that affect the whole ecosystem, but we're going to work hard on it; it's an opportunity to make things better for everyone. Please stay tuned! <comment><USER2> I would be so highly interested to know the internal YC discussion and analysis about two main points mentioned in that previous thread you discussed, specifically:

\- moving away from open-plan offices to something that allows most works hours on most days to have a high degree of quiet & privacy

\- addressing the many points about how start-up compensation, even when discounting equity risks for unsuccessful outcomes or low exits, is just not competitive, even though offsetting factors like working on interesting projects, flexible work/life policies, etc., are generally no worse in traditional corps than start-ups, and quite often much better.

Any concrete feedback about the analysis that came from these points would be fantastic.

I think it's very exciting to hear that YC is directly discussing these things!

<comment><AUTHOR> Yes, that is what we're working on.

While it doesn't address the larger point about compensation (we're working on that), one thought about compensation specific to this event, is that the companies coming to the Expo are at a wide range of funding levels and sizes. Some of them would probably not even be considered "startups" by most people, like Instacart and Doordash, which are both hundreds of people. Many of them do already offer highly competitive compensation; so I would not let this be a deal-breaker for considering coming to the event.

</comment></comment><comment><USER3> I recall the event in 2012. It was interesting. Justin.tv basically was really looking for someone to help with the onerous task of operations -- but the presenter was upfront about such. I recall another startup, can't recall the name, where phrases like "we are a family", "we stay late to meet with our customers" etc were thrown around. Multiple people around me were shaking their head.

Both were honest and up front, but trying to play up the fact that "we live to work" played about as well as a piano falling off a crane onto Wiley Coyote.

<comment><USER4> > Both were honest and up front, but trying to play up the fact that "we live > to work" played about as well as a piano falling off a crane onto Wiley > Coyote.

I'd say the honesty is merely the minimum required level of courtesy and respect. The reaction is understandable, especially considering that, for example, by then JTV was 5 years old. If ops is truly still _onerous_ by that point, then candidates would be right to suspect there's something awry with management.

I do hope that this is something that's being addressed by YC, that some founders have the expectation of their early (and sometimes not so early) employees of having the same loyalty or buy-in as they do. If your entire employee option pool is 10%, it's not reasonable to ask even employee #1 for that.

</comment></comment><comment><USER5> I'm working for one of few >10b unicorns. $250k salary. $50k bonus. $1.3m stock with annual refreshers. I only have 7 years of experience and am a regular client developer.

I really want to be a part of a start up and learn things to get ready start to my own company at some point but all offers I got from startups can only make sense if they become Facebook.

<comment><USER6> Golden handcuffed!

</comment></comment><comment><USER7> I appreciate the work that you place into this. But again, what exactly are the startups you with* looking for? If they are hurting for talent wouldn't they take more risk in acquiring the talent just like they are an unknown company with a 90% failure rate?

You must have a rough idea of what the startups are talking about from their point of view. Just like you are trying to bring clarity into the startups maybe you could provide clarity into the recruiting/hiring.

*edit

<comment><AUTHOR> There's a whole range. Companies are hiring for new grad software engineers up to senior engineers and CTOs. There are also many roles, from full-stack web to back-end and data science and many tech stacks.

I'm particularly excited by how many companies are hiring for deep-tech roles like machine learning, computer vision, and embedded programming for hardware.

<comment><USER7> Thanks for the quick response. I guess I'll try to be more specific. What are they specifically looking for and how are they measuring that?

Granted, everyone wants the "smart, motivated" + some other synonym for a great person. But are they looking only at c.s grads from surrounding Bay Area schools/Ex-Google and the like?

<comment><AUTHOR> Something I've learned from working with a lot of YC companies, is that while they certainly all want people who are "smart and motivated", they measure and define it quite differently.

It sounds like your concern is whether companies are only looking for c.s. grads from brand name schools and companies, and I can specifically vouch for that not being the case. Depending on the company, they're likely to value open source contributions, independent projects you've built that are interesting, startups you've started or worked at, impressive technical work you've done at any company, and how candidates do on their own technical interviews (which vary greatly in substance), among many other factors.

</comment></comment></comment></comment><comment><USER8> It'd be nice to see a NYC version, too.

<comment><AUTHOR> Yes, a number of the NYC-based founders asked for that too.

</comment></comment><comment><USER9> Equivalent in London: [https://www.siliconmilkroundabout.com/](https://www.siliconmilkroundabout.com/)

Looks like they've dropped their emphasis on working at a startup, which was SMR's mission last time I checked. But I imagine there'll be some there?

</comment><comment><USER10> In general, what type/stage of startups would fresh grads (with internship experience) fit in best?

<comment><USER11> Based on the typical technical skills needed: almost nowhere.

Based on the typical compensation and culture: you describe the kind of employee they target.

And that is in a nutshell exactly what all the comments in the previous thread distill down to.

</comment></comment><comment><USER12> I'm working at my first startup. I really hope it's my last one. Best wishes to those who want to live the dream!</comment>


Building an LTE Access Point with a Raspberry Pi - <AUTHOR> https://snikt.net/blog/2019/06/22/building-an-lte-access-point-with-a-raspberry-pi/ <comment><USER2> Not mentioned in the article, but I'd also recommend force-setting DHCP Option 43 to "ANDROID_METERED" to let Android clients know you're on a metered connection (hotspot) instead of a normal landline-based WiFi network. [1] This inhibits some things like automatic updates.

I don't know of an equivalent for other OSes, unfortunately.

[1]: [https://www.lorier.net/docs/android- metered.html](https://www.lorier.net/docs/android-metered.html)

<comment><USER3> [https://apple.stackexchange.com/questions/199163/how-does- io...](https://apple.stackexchange.com/questions/199163/how-does-ios-and-os-x- detect-when-a-wi-fi-network-is-a-personal-hotspot) seems to think it's in the wifi beacon itself for OS X.

No detection method appears to be present in Windows 10, it's all geared towards LTE modems integrated in the machine.

I really wish this had been standardised.

<comment><USER4> Windows 10 appears to support one of the newer standards for announcing su h details in beacon

<comment><USER3> "Microsoft has defined a vendor extension to the 802.11 protocol."

[https://docs.microsoft.com/en-us/windows- hardware/drivers/mo...](https://docs.microsoft.com/en-us/windows- hardware/drivers/mobilebroadband/network-cost-information-element)

<comment><USER4> It also supports at the very least WiFi Alliance's Passpoint, based on 802.11u.

</comment></comment></comment></comment></comment><comment><USER5> I don't with to diminish the author's work but there's nothing all that special about the LTE portion. From the perspective of the Pi it's just another interface/network.

This is more akin to building a basic consumer all-in-one router -- which I think actually sounds cooler.

So the pieces that this article puts together:

* Bridging the Pi's ethernet and wireless interface and setting up DHCP pointing to a forwarding DNS server on the bridge with dnsmasq.

* Setting up Linux IP forwarding and the iptables rules needed to perform NAT over the WAN.

* Setting up a wireless access point on the Pi with hostapd.

It really is super cool that for $35 and an hour you can have a functioning, albeit not super fast, router that's basically ready to be plugged into a modem and work.

<comment><USER6> Yeah, I did this myself not long ago as part of a mobile office rig and ... I didn't write about it or share about it becuase it was mostly just putting USB male connectors into USB female connectors and sourcing the right kind of batteries to charge things.

Oh, and learning ethernet cables are a monstrous power draw and you're much better off using wifi for everything.

I mean, yeah it's cool to have. But I can't help but feel like it's a bit trivial.

<comment><USER3> Can you comment on Ethernet being more power hungry than Wifi? That's the first time I've heard that.

<comment><USER7> Ethernet usually requires isolation magnetics and transformers, while WiFi is basically just a wire sticking out of an RFIC.

<comment><USER8> It's not so much that the transformers consume more power than just radiating it into the air, but that Ethernet is not designed with power efficiency as a goal. Ethernet devices that need more power usually get it from PoE instead.

Without doing any analysis at all, I suspect the idle behaviour is a big difference; Wifi goes silent if there's no traffic, apart from AP beacons, but Ethernet continuously transmits "fast link pulses" every 16 miliseconds to maintain autonegotiation and detect connection lost.

<comment><USER6> Honestly I dunno why it was the case for me. What I know is that I thought I'd save power pushing pulses over copper rather than using a radio. I didn't. In fact it was consistently about 40% worse than wifi.

That might be unique to the RPi hardware I used. It might not.

Maybe I should have written that article. :)

<comment><USER9> The Pi does Ethernet with a seperate and pretty power hungry chip.

If the ethernet cable is disconnected, that chip mostly powers down.

I think it's a Pi related thing rather than being inherent to Ethernet. In fact, if I were to guess, the energy per bit per meter of ethernet is probably far far lower than WiFi.

</comment></comment></comment></comment></comment></comment></comment><comment><USER10> Title is missleading, i was hoping for [https://bellard.org/lte/](https://bellard.org/lte/) :(

<comment><USER11> yeah, likewise, I suddenly noticed it was just an access point, which seems a bit trivial to be hitting the #1 spot on HN.

But I might be bias, I'd really ike a CAT-M1 and NB-IOT base station

</comment></comment><comment><USER12> I had hoped this was a howto for building an LTE booster.

I know the spectrum isn't licensed for us, but wouldn't it be great if someone figures out how to build a booster for those of us with a weak signal.

<comment><USER13> [https://duckduckgo.com/?q=wilson+lte+booster&atb=v173-2_j&ia...](https://duckduckgo.com/?q=wilson+lte+booster&atb=v173-2_j&ia=web)

</comment></comment><comment><USER14> I did something similar with OpenWRT and a MikroTik box. It works fine. The interesting bits were getting OpenWRT to work on a then-unsupported device (it was very similar to other devices, though), picking a modem with good support for multiple US carriers, depending on the modem, disabling USB3 by taping over those pins, flashing carrier-specific firmware, and getting the modem to connect.

Then I went overboard and set up routing so bulk traffic (OS updates, backups) go over a slower DSL connection, while other traffic goes over LTE. And if LTE goes down, it switches to the DSL connection.

<comment><USER15> Your comment motivated me to check if I could make my home router fallback to my second connection (I flashed it with OpenWRT). After fiddling with using usb wifi dongle as second wan entry and looking over if any nearby electronic shop has usb lan adapters in stock, I discovered that my router has vlans and I can "rob" one switch port for second wan.

So I set up mwan3 and have more and more respect for what one can do with Openwrt and off-the-shelf router.

</comment></comment><comment><USER16> As for the Huawei stick, most of them support being switched to modem mode using usb-modeswitch. But I'd prefer the "virtual network interface" mode if I had the choice, usb-modeswitch and modem stuff isn't exactly the best documented thing on this planet.

<comment><USER17> What's the difference? From the sound of it, the VNI would be a USB device identifying itself as an ethernet adapter and usb-modeswitch does... mode switching, like the 'alternative protocol' thingy that USB3 can do?

<comment><USER16> The difference is that in VNI mode the stick runs its own OS including a router and other crap while in modem mode it directly passes through the modem to the host device, leading to better performance, working "server" connectivity (assuming you can get a publically routed IPv4 address from your provider, though) and less security issues as there is one stack less to worry about.

<comment><USER17> Ah, now I understand, thanks!

</comment></comment></comment></comment><comment><USER18> Read it hoping to find use of an SDR or direct use of LTE modem instead it's basically what any Linux user would have to do to get a USB modem working.

For some reason doing anything a normal Linux user would do but on a raspberry pi is instant clickbait.

</comment><comment><USER19> I have a iNet Slate, its effectively the same thing but a bit more purpose built and has a switch on the outside (which can be configured to do different things) that can turn on VPN so all connections are VPN'ed. It supports USB tethering and a USB modem. For the money, after you're done buying a case, you'd still need more ethernet ports, it probably doesn't work when borrowing local wifi.

[https://www.gl-inet.com/products/gl-ar750s/](https://www.gl- inet.com/products/gl-ar750s/)

</comment><comment><USER20> So how does it work? I assume the LTE USB modem operates on T-Mobile frequencies, so does that mean it accepts all T-Mobile customers?

How's authentication done? How does T-Mobile permit you to use their frequencies?

<comment><USER21> This is not about hosting an LTE 'access point'. It's about creating a access point which uses an LTE uplink. I agree, the title is somewhat misleading.

<comment><USER22> Correct. If you were to create an actual "access point" you will have the FCC and other agencies knocking on your door within minutes.

<comment><USER23> I don't think their response time would be nearly that quick. All the stories about Stingrays would imply that this actually isn't very well policed (FCC enforcement is reactionary. The carriers could be proactive, but then would discover the Stingrays. And if carriers were complicit with Stingrays, then Stingrays themselves wouldn't be necessary).

I would also think the widespread deployment of legit femtocells would provide a decent cover.

But yes, it is generally illegal. (Though this hasn't stopped OpenBTS from being developed)

<comment><USER22> Yes, we had the FCC (via FBI) knocking on our door within 30 minutes. This was about 10 years ago when we were doing research with an access point.

<comment><USER23> Sheesh! Do you care to share any more details? How dense of an area? How was it detected? If by "access point" you mean 2.4GHz, what was the actual problem?

<comment><USER22> It was a GSM antenna that registered with I think AT&Ts network in Philadelphia. It was part of a legitimate research project, so no charges or fines were placed on us.

<comment><USER23> Your description just makes me more curious! What do you mean by "GSM antenna" ?!

Should I really read that as you had a legitimate cell radio device, and just attached an unauthorized antenna? Or was the radio itself experimental/unapproved?

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER24> The author mentions power consumption issues in the post. I've seen the same when running RPi access points. Definitely use a decent power brick if you're going this route.

</comment><comment><USER25> I would be more interested if someone managed to build a GRE tunnel bonding solution with open source tools. Multiplexing a data stream over multiple LTE uplinks would be nice.

<comment><USER26> Holy grail stuff there. Would love that too.

</comment></comment><comment><USER27> Nice clear instructions and presentation style, well done.

<comment><USER28> I'm with you, it is a nice presentation and will follow it. Yanno' if you could get the ARM version of pfSense on the pi this would eliminate the Huawei modem, Maybe.

</comment></comment><comment><USER29> You should checkout EspressoBin, which should be more suitable for this task. Hardware-Switch, WAN-Port and so on

<comment><USER30> It has horrible reviews (booting by itself etc.). Any better alternative?

<comment><USER14> I've had decent luck with this: [https://mikrotik.com/product/RBwAPR-2nD](https://mikrotik.com/product/RBwAPR-2nD)

<comment><USER31> +1 for Microtik. They have their weird problems but are cheap and generally fairly reliable for what you get. Have a bunch of their hardware around, all served me pretty well.

</comment></comment></comment></comment><comment><USER32> Is there no need to do additional handling for hung modems/connections and safe reboots, or is all of that left as an exercise for the reader?

<comment><USER6> Usually you can just do this from the command line. With the right tuning Pi's boot quite fast, so I just used a Pi reboot and a 4g gateway with no battery to solve these issues.

<comment><USER32> To me, the resilience in having that stuff automated is the difference between "an access point" and "a computer that just happens to bridge connections."

<comment><USER6> True. I just didn't find that stuff very difficult. It almost never came up.</comment></comment></comment></comment>


Ask HN: When are large data structures used in practice? - <AUTHOR> Can anyone share some practical examples where it is a requirement to maintain 1M+ elements in a data structure when streams and iterators are not a viable alternative?<p>Using a vector as an example, where O(n) operations are actually very fast in practice when n < X, when might X be big enough in practice to warrant the use of other specialized structures like skip lists and radix-balanced trees? <comment><USER2> How about cases where the data structures need to support "interesting" query operations. E.g. data structures to support spatial queries: "find me all the neighbouring elements within a radius r of element x". Not super-specialised, but probably need something like a quadtree or a hash table or other indices to help partition the search space. There can easily be 1M+ elements if each element is an address in state / national address database, or a component of some utility network, or a location of an app user. There are use cases where you'd want to be able to execute these kinds of queries with low latency, so jamming everything into ram in an appropriate data structure could be a good fit.

</comment><comment><USER3> Any sort of DNA processing. There is a lot of research into concise data structures for a very specific set of operations for DNA processing since our ability to generate DNA data is accelerating much master than processing or memory speed.

<comment><USER4> > There is a lot of research into concise data structures for a very specific > set of operations for DNA processing

See: [https://alexbowe.com/fm-index/](https://alexbowe.com/fm-index/)

Impl: [https://github.com/shibukawa/fm- index.jsx](https://github.com/shibukawa/fm-index.jsx)

More: [https://news.ycombinator.com/item?id=22544718](https://news.ycombinator.com/item?id=22544718)

</comment></comment><comment><USER5> Here's one example:

In the IPv4 Internet, there are about 830,000 prefixes (network addresses) announced, and the number is growing [1]. Routers at ISPs and sophisticated customers maintain full ("default-free") routing tables with at least one entry for each prefix.

These tables are used for packet forwarding decisions, so lookups have to be fast. Traditionally, a radix tree is used, but some routers use other data structures or specialized hardware.

[1] [https://www.cidr-report.org/as2.0/](https://www.cidr-report.org/as2.0/)

<comment><USER4> > _Traditionally, a radix tree is used..._

Here's an excellent write-up on radix-trees usage by the Linux kernel for IP routing: [https://vincent.bernat.ch/en/blog/2017-ipv4-route-lookup- lin...](https://vincent.bernat.ch/en/blog/2017-ipv4-route-lookup-linux)

Also see: [https://news.ycombinator.com/item?id=22467251](https://news.ycombinator.com/item?id=22467251)

</comment></comment><comment><USER6> Not sure if this fits the example you have in mind: [https://medium.com/pinterest-engineering/an-update-on- pixie-...](https://medium.com/pinterest-engineering/an-update-on-pixie- pinterests-recommendation-system-6f273f737e1b)

> The ultimate goal is to fit the entire Pinterest graph in memory. Obviously > we can't fit the entire 100 billion edges in RAM, but once we prune down to > 20 billion edges, we end up with a graph that's 150GB.

</comment><comment><USER7> Do you mean specifically 1M elements in memory?

Because something like a relational database could have 1M elements in a B-tree but the whole data structure doesn't have to be in memory at the same time.

Or operating system page tables as another example.

<comment><AUTHOR> No just some arbitrary 80%-case "large" number.</comment></comment>


Computer Programming and Precise Terminology - <AUTHOR> http://www.ddj.com/cpp/208808373

<comment><USER2> Speaking of precise terminology, I doubt the OS is involved in assigning addresses to C variables.</comment>


Why Startup Hubs Work - <AUTHOR> http://www.paulgraham.com/hubs.html

<comment><USER2> <http://news.ycombinator.com/item?id=3077165>

<comment><AUTHOR> Oh, I probably missed that in the flood of submissions about Steve Jobs' death.</comment></comment>


Google needs Android to succeed in Africa. - <AUTHOR> https://iyinoluwa.posterous.com/google-hardware-and-africas-super-information <comment><USER2> I don't care about the topic, but I had to read the article to find out how to parse the headline.

Which, for the curious, is: To succeed in Africa, Google needs Android.

</comment><comment><USER3> I'm not sure I agree with the premise of the post. At this point, do most major cities (e.g. Lagos, Cairo, Kinshasa, Durban, etc.) even have infrastructure to support smartphones? Last I heard, 3G infrastructure was pretty spotty outside of select areas of Johannesburg.

Given that, wouldn't it be in Google's interest to hold off on pushing Android in Africa until said infrastructure is built?

<comment><USER4> > Last I heard, 3G infrastructure was pretty spotty outside of select areas of > Johannesburg

Perhaps a good few years ago. You should get HSDPA in all the major cities in SA. A lot of people actually use mobile broadband in place of fixed-line broadband.

</comment></comment><comment><USER5> In this situation, wouldn't Chrome OS be a better fit? You could have simpler computers, and therefore cheaper hardware, by moving features to the cloud.

<comment><USER6> But then, you'd be more dependent on working infrastructure, which is a major problem in the 3rd world. Mobile is a better fit than cloud to get started, and will provide the economic incentive for better infrastructure. The market will then shift naturally to cloud.

</comment></comment><comment><USER7> no one needs anything to succeed in Africa...</comment>


Herman Miller Just Redesigned Its Iconic Aeron Chair - <AUTHOR> https://www.wired.com/2016/10/herman-miller-just-redesigned-iconic-aeron-chair/ <comment><USER2> Ionic as in comfy? I find it ugly to be honest...

<comment><USER3> Iconic not Ionic. It's is an icon, as it is recognizable as a representative symbol of a modern office chair.

<comment><USER2> Hehe, sorry, still absolutely not my taste. I meant Iconic of course...</comment></comment></comment>


PowerPC Solaris on the RS/6000 - <AUTHOR> https://virtuallyfun.com/wordpress/2020/05/28/powerpc-solaris-on-the-rs-6000/ <comment><AUTHOR> So, this post (which I loved) brings back a lot of memories for me (having joined Solaris Kernel Development in 1996, when ppc was still a thing -- if barely).

If I may, a quick anecdote: back in MPK17 (now Facebook!) my office was next to that of Mike Sullivan, who was one of the lead engineers on on297, our internal name for Solaris 2.6. As part of his nature of running latest bits on everything around him, Mike had in his office (among many other things) a laptop. Over my years of working in Menlo Park, I can remember exactly one theft: one night, that laptop of Mike's -- and the laptop alone -- was stolen. The funny thing (and the reason this story made me think of it): it was a ThinkPad 850, an IBM PReP laptop running an internal build of on297 (to my knowledge the only one that ever existed) -- and I have often wished I could be there when the thief tried to unload what was surely the world's most arcane laptop, if not its most useless...

<comment><USER2> ppc is still a thing, my project computer is a power9. The only reason it isn't my daily driver is because I can't help but tinker with it. Being able to rewrite the supervisor chip code in order to get that fan curve just so, and having the support of unabridged and NDA free documentation - instead of grinding through reverse engineering... feels good. I highly recommend it, just stick to known working GPUs, the mmu actually does its job.

<comment><USER3> PPC is/was different from POWER. Technically related but they diverged enough that it was a different ISA for all intents and purposes

<comment><USER4> To be clear, you're talking about the original POWER ISA. The PowerPC ISA has subsequently been renamed the Power ISA (not to be confused with the POWER ISA). Confusingly, IBM's POWER CPUs (POWER3, POWER4, POWER5, POWER6, POWER7, POWER8, POWER9) since POWER3 implement the Power ISA (then known as the PowerPC ISA), not the POWER ISA. POWER2 was the last CPU to implement the POWER ISA.

[https://en.wikipedia.org/wiki/IBM_POWER_instruction_set_arch...](https://en.wikipedia.org/wiki/IBM_POWER_instruction_set_architecture)

[https://en.wikipedia.org/wiki/Power_ISA](https://en.wikipedia.org/wiki/Power_ISA)

<comment><USER5> And now the ISA has been updated for the POWER10 CPUs that'll be announced shortly.

</comment></comment></comment></comment></comment><comment><USER6> Posts like this, projects like these, and people like Neozeed are part of why I think the GNU project fundamentally is brilliant.

Getting GCC running on barely-produced prototype hardware system, on an ancient version of Solaris which itself was a particular pain to install, whilst entertaining and informing the reader about the history of the PowerPC migration circa 1994, is no mean feat.

What were the desirable/sexy features of OpenFirmware anyway? I remember typing occasional commands into it on old Macs (especially a Beige G3 running Mac OS 9 and, later, with varying degrees of grumpiness, OS X), mostly with the fear of god of bricking my system.

There was also that separate "crash" shell you could bring up once the machine had booted with a magic key combination involving the power button, and pressing something like 'R' would reboot it. I wish Apple was a bit more open about these programming interfaces – a bit like Solaris.

<comment><USER7> The beauty of OpenFirmware is that it is extended with Forth, which is extremely compact (so could fit on small PROMs in the computer and fcode for addon cards). If you are designing a PCI card, you can write a bit of fcode that will run on SPARC and PowerPC for instance to expose opaque RAID volumes as bootable disks.

Keep in mind the PC BIOS was and is a complete tire fire, and uses something like this [https://en.wikipedia.org/wiki/Option_ROM](https://en.wikipedia.org/wiki/Option_ROM) to approximate support for the above. UEFI is a bit of improvement but massive and obtuse in comparison to OpenFirmware.

<comment><USER8> _"you can write a bit of fcode that will run on SPARC and PowerPC"_

and x86, and ARM, and CPUs not even invented yet. Performance wouldn't be optimal compared to native drivers, but decent enough to boot a workable system.

</comment></comment></comment><comment><USER9> If you get access to an IBM RS/6000, another thing to play around with on it is the original AIX operating system.

Imagine a Unix that got an IBM working-over. When the RS/6000 workstations first came out, AIX was a big contrast to the more seat-of-pants workstation Unix of SunOS 4 (this was before Solaris 2).

Some of the AIX changes weren't only enterprise-y server management OS abstractions, but fun: AIX included a nice GUI hypertext browser for its documentation, in 1990 (possibly earlier).

<comment><USER7> I am always surprised by the negativity that surrounds AIX by layperson UNIX users. AIX actually remains ultramodern when the full timeline is taken into account of remaining UNIX-like OSes like Linux, Solaris, and the BSDs. There's a reason it is one of the few remaining commercial UNIX, and that's because it is good and runs on the best hardware for vertical scaling. Scaling on large SMPs is not an easy task for OS/kernel devs, and something even Linux struggles with to this day on any non-trivial NUMA system. Meanwhile IBM has been offering extremely large NUMA SMP, gobs of I/O bandwidth, and tightly integrated virtualization since the turn of the millennium.

AIX uses a database called the ODM for kernel and userspace configuration. This can be thought of something like 'nvlist' in FreeBSD and Solaris ([https://github.com/fudosecurity/nvlist](https://github.com/fudosecurity/nvlist)) but it has a more extensive object type system that is used almost everywhere in the system. This is in turn used to pass complex typed data between the kernel, userspace commands (like say logical volume management), and even an advanced daemon manager not unlike SMF or upstart or even systemd.

None of the above probably sounds particularly ground breaking in 2020 with systemd-ified Linux, but keep in mind most of this was in place 30 years earlier in the early 1990s with AIX 3.0. This was a time when, for instance, hardware support in other UNIX and clones still might mean custom compiling a kernel or at least invoking a linker to produce a new binary kernel.

Novice AIX users see smit and think "yuck I have to use a TUI to configure the system" but it is easy to do everything from the shell or automation. smit is just a rather gentle tool inspired by other IBM projects such as OS/400 and ISPF for menu driven user interfaces. Keep in mind systems like SCO also sported TUIs like this as it was a fad at the time to make UNIX seem more familiar to PC and minicomputer users.

One thing that is super bizarre, the kernel is also fully pagable. I suppose that made sense in the early '90s for diskless workstations where RAM might also be limited but it seems like unnecessary extravagance and complexity these days.

<comment><USER10> SAs have their own likes/dislikes. From an end-user dev point of view AIX is pretty harsh. XCOFF binaries mean almost no tooling works with them. It's 2020 and IBM finally came around to adopting LLVM/Clang to the platform. There's lots of strange deployment warts like no $ORIGIN support, everything is shared/PIC but you have none of the ELF naming support, shared libraries stay in a system-wide library cache even after the processes exit (i.e. updating the file on disk and starting a new process will not guarantee it is used), and the list goes on and on. It takes a huge amount of effort to keep all the usual open-source packages running there.

You have to do the open-source work, though, because the vendor side is as close to abandonware as you can come. The OS is basically not touched unless they have to. For example, they gave up trying to actually fix their system headers to have C++ include guards, so they just hacked the compiler to extern C _all_ system headers —- a terrible hack that GCC actually adopted as well to stay compatible. There's no versioning of user-space library additions or changes, so you can't easily make code forwards/backwards compat in the preprocessor. They gave up trying to follow modern C++ and pulled in a Clang front-end. User binaries remained broken for years until anyone complained about them. There are people there doing the bare minimum to add support for some new POSIX interface that comes out, but the headers are not "clean" in the same way that the Solaris ones are. Solaris headers and man pages are some of the best I've ever seen. They could constantly improve the system, but they don't. They do what customers ask for and no more.

It might be great at taking some Fortran simulation and cranking the nodes up, but it's not great at coming anywhere close to competing with Linux for general computing use.

<comment><USER11> AIX at least has LLVM, and IBM had unofficial GCC ports for years.

If you want a dumpster fire for development work, I'd highly recommend HP-UX. HP-UX's stdio wasn't very std back in 1990s. What I remember is a bunch of syscalls seemingly existing but not actually working despite being OK with the same code on Solaris and Linux.

Somewhere around 2003-4, it appears all development basically consists of security patches and new Itanium hardware enablement, and aCC barely supported anything C++ related; its more like trying to use Borland C++, and GCC was really iffy, although it had the advantage that post PA-RISC, HP did adopt ELF.

<comment><USER10> Yeah, I've used HP-UX as well. All the fun of an OS and compiler no one uses, with the added bonus of all the quirks of a CPU that _really_ no one uses. I wonder if the soldiers 1000s of libraries deep in the Oracle link line read HN and have some fun IA-64 porting tales.

</comment></comment></comment></comment></comment><comment><USER12> Years later there was another port of OpenSolaris to IBM Power computers called "Polaris": [https://www.theregister.co.uk/2006/01/10/solaris_on_power/](https://www.theregister.co.uk/2006/01/10/solaris_on_power/)

And even port to IBM z/VM ("Sirius"): [https://www.theregister.co.uk/2008/10/17/solaris_on_mainfram...](https://www.theregister.co.uk/2008/10/17/solaris_on_mainframe/)

<comment><USER13> > And even port to IBM z/VM ("Sirius"):

Unfortunately that died, because Oracle stopped being willing to cooperate with the port [1]. Of course, I understand why they might not see it as being in their commercial interests to support porting their OS to a competitor's hardware; but, I think their turn away from openness with Solaris accelerated its decline rather than delaying it.

[1] [https://www.theregister.co.uk/2010/03/29/oracle_solaris_z_ib...](https://www.theregister.co.uk/2010/03/29/oracle_solaris_z_ibm_mainframe/)

(Disclaimer: ex-Oracle employee, but I was never working on Solaris directly, just on higher-level middleware stuff which sometimes ran on Solaris, and I really don't know anything about this topic beyond what has been reported in the media.)

</comment></comment><comment><USER14> In the late 90's, I had Solaris running on my Toshiba laptop. I loved it. It was a great OS and development platform. At some point it became too difficult to keep up with software upgrades, otherwise, I'd still be using it.

</comment><comment><USER15> This article brought back a memory of a fellow high school student who had somehow obtained a windows NT PowerPC CD that we tried and failed to boot from in a PowerMac 6100.

<comment><USER16> hahahaha. I did this on an IBM prep machine, a really straightforward install of NT 3.51.

It even managed to run some x86 windows stuff - incredibly slowly (this was ~2005)

And it ran the windows 2000 shell preview as well! - though I may have installed NT 4 by then.

It ended up running Debian and being a mono test bench until it was retired to being an MP3 player for the home office.

Alas it was accidentally recycled along with a host of other stuff I was avoiding to deal with. Still it was fun to play with at the time!

<comment><USER16> it was probably a model 850

</comment></comment></comment><comment><USER17> I would love love love one of these machines to run Windows NT and some flavour or ancient Unix. I'm in Australia and desperately looking for an old PowerPC. Does anyone know where to dig these machines up? Yes I tried eBay but it is prohibitively expensive to post from overseas to Sydney.

<comment><USER18> Ex-government auctions have stuff occasionally. When I used to live in Canberra every two month or so one would be advertised could get lots of old hardware for dirt cheap. Mostly old Sun Sparc stuff but sometimes HP or IBM power based hardware too. I imagine Sydney would have similar auctions.

<comment><USER17> Can I ask on what web site you have found there auctions?

</comment></comment></comment><comment><USER19> And here I was, thinking I was a damned fool for wanting to port Illumos to run on Power 9. Well, okay, I probably am still a damned fool for wanting such, but knowing that the Solaris codebase has run on Power before gives me hope that it might actually be possible.

<comment><USER20> There were ultimately two ports of Solaris to PowerPC, this early one and a third party port of OpenSolaris/Solaris 10 to PowerPC. The latter, called Project Polaris, can be found in source form online [https://github.com/andreiw/polaris](https://github.com/andreiw/polaris)

Getting this running on ppc64(le) would be no small undertaking, but hey.

</comment></comment><comment><USER21> It'd be worthwhile to give pkgsrc a try.

</comment><comment><USER22> That is a real surprise !!! Life full of these.</comment>


Ask HN: What are some novel and niche database paradigms? - <AUTHOR> Outside of nosql, relational, graph, and timeseries are there any database paradigms that are defined? I'd like to explore the niche cases of database needs that aren't so straightforward or needed by the many. <comment><USER2> I'm not sure that it's a completely different paradigm but Datomic ([https://www.datomic.com/](https://www.datomic.com/)) is a pretty unique database if you're interested in thought provoking design. Here's a cool video of Rich Hickey (the creator of Datomic and Clojure) talking about the underlying data structures and design decisions: [https://www.datomic.com/](https://www.datomic.com/)

<comment><USER3> missing link to the video :)

<comment><USER2> Ooops, here it is. [https://www.youtube.com/watch?v=Cym4TZwTCNU](https://www.youtube.com/watch?v=Cym4TZwTCNU)

</comment></comment></comment><comment><USER4> Database as a term and concept has a lot of historical baggage and whether something is considered a "database" or not has a large marketing component.

The traditional function of minimizing slow disk accesseses in the absence of reasonable sized caches is largely gone as an interesting research area today. You may want to look at ways of storing data in memory and/or distributed systems generally. For example, asset data storage and retrieval in games, succinct data structures, systems like etcd.

Another tangential advice: look at some of the recent published research. Eg ACM SIGMOD proceedings.

The landscape tends to have relatively shortlived fads that leave behind them a narrow tail of applications that are actual fits for the idea in question. (Industry examples: "big data analytics" -> Hadoop, "web scale online db" -> MongoDB, etc).

</comment><comment><USER3> Versioned databases especially for tracking provenance and lineage for data science models might become of thing. See Apple's Entity Store [0] or my take on it [1]

[0] [https://www.youtube.com/watch?v=16uU_Aaxp9Y](https://www.youtube.com/watch?v=16uU_Aaxp9Y)

[1] [https://github.com/awesome-data- distribution/datae](https://github.com/awesome-data-distribution/datae)

</comment><comment><USER5> If you want a good high level overview I recommend Designing Data Intensive Applications by Kleppmann. You will walk away with a good understanding of the tradoffs of each paradigm.

</comment><comment><USER6> NoSQL is big, anything that's not relational is NoSQL. So key/val, document, column. There's also object oriented database. This probably covers almost what you're looking for - [https://en.wikipedia.org/wiki/NoSQL](https://en.wikipedia.org/wiki/NoSQL)

</comment><comment><USER7> Deductive databases are pretty neat

<comment><USER7> also probabilistic query systems like BayesDB

[https://github.com/probcomp/bayeslite](https://github.com/probcomp/bayeslite)

</comment></comment><comment><USER8> You should check out M, a database with a built-in programming language that was one of the original nosql transactional databases in the 60s.

YottaDB is a modern company proving an M db.

[https://learnxinyminutes.com/docs/m/](https://learnxinyminutes.com/docs/m/)

</comment><comment><USER9> Immutable ledgers (blockchain) and GIS (mixed systems) and search indexes (elastic) all come to mind as more niche databases. Some of these uses can be modeled in a variety of engines.</comment>


Founder's Paradise: Co-living Space in Koramangala, Bangalore, India - <AUTHOR>

Founder's Paradise — a proposed co-living space in Koramangala, Bengaluru, India.
What if you never had to end your conversation because you have to go home? What if you had a second chance at living a dorm life with amazing housemates? What if you found a home in a city you have just landed in?
Accelerate your growth. Understand a city faster. Bond with people, not just exchange business cards.
Help us validate this concept:
1. Will you be willing to pay around USD 400/ INR 25,000 per month and share the room with one more person? If you are a female, you'll get a female room mate.
2. This would include all amenities: high-speed Internet connection, fully stocked kitchen, purified drinking water, basic furnishings — beds and mattresses, wardrobes, geyser, fridge, TV, washing machine.
3. Bonus: A library with books on startups & entrepreneurs + board games + weekly dinners + weekly film screenings/ guest talks.
4. Duration of stay can be anywhere from 2–6 months.
5. Sign up to show your interest here: http://goo.gl/forms/vhQj7SFRrF <comment><USER2> I _am_ from Koramangala and work in a 'cofounder role' at a startup but would hate the idea of coming back home to more startup-conversations. Co-working spaces and meetups work well at solving most of the problem. The others aren't really problems- i have all of #2 and 3.

</comment><comment><USER3> Why do you need so much information if you're just looking to gauge interest?

I might be interested depending on the number of people in the space, the conditions of the space itself, and the available amentities of the surrounding area

</comment><comment><USER4> I pay £10 less than that a month with my own room, not in a third world country.

No thanks

</comment><comment><USER5> This sounds really great</comment>


Mix Parlay - <AUTHOR> http://PrediksiParlay.club <comment><AUTHOR> very clever blog, really like it. Thank you for let me join this blog</comment>


Show HN: Hasura Cloud: Managed GraphQL for your database and services - <AUTHOR> https://hasura.io/cloud <comment><USER2> I was a bit startled by the price (starts at $99/mo) but man, Hasura has been an absolute joy to work with. I'd highly recommend exploring the open source version on Heroku if you haven't tried it before.

Congratulations to Hasura on the launch!

</comment><comment><USER3> Linking to the announcement blogpost I wrote: [https://hasura.io/blog/announcing-hasura-cloud-managed- graph...](https://hasura.io/blog/announcing-hasura-cloud-managed-graphql-for- your-database-and-services/)</comment>