U.S. and U.K. F-35 Jets Include 'Core' Circuit Boards from Chinese-Owned Company - <AUTHOR> https://www.forbes.com/sites/zakdoffman/2019/06/15/chinese-owned-company-supplies-electronics-on-u-s-and-u-k-f-35-fighter-jets/#f144b3c25c05 <comment><USER2> It's a PCB manufacturer.

They aren't buying computers from china. The company that did design the electronics (GE) decided to have this little outfit manufacture the PCBs for them (I mean, we're talking a production run in the low thousands so far -- this is NOT a big project). And the little outfit got bought by a bigger PCB company in China a few years back.

The risk to the supply chain is zero -- anyone can make printed circuit boards. The intelligence risk is limited to the ability to see how chips are wired together, with some ability to guess what ICs are in use by clues in the pinouts.

<comment><USER3> I don't know anything about intercepting electronics, but I would not bet my life (which is what you're doing) that a good signals intelligence agency cannot somehow compromise a PCB.

<comment><USER4> I agree. I mean if we're guessing at possible dangers, it seems like there are several possible angles that might be tempting to try:

\- It's meant to be a stealthy plane, how about tweaking the board to cause a nice and noisy signal leak in certain circumstances?

\- find a way of causing the board to fail under suitable circumstances.

\- Probably less plausible, if they can figure out what data is passing through the board, then there is potential to engineer side-channels in the hope that they can achieve another compromise that can leverage it. Maybe they never get the chance to take advantage, maybe they do. Or if e.g. they can find an input that an adversary can potentially influence by how they engage with the plane, use that to trigger failure as above.

I'm assuming they're properly inspected and tested on receipt, but wires on circuit boards can interfere with each other in messy ways if you exceed tolerances even a little bit. I recall an amusing case some years back where someone used genetic algorithms to lay out features on an FPGA and got a very good solution that was totally un-reproducible: it turned out their GA had optimized to take advantage of quirks of that one specific FPGA that didn't work on other FPGA's of the same model, by combining things in ways it hadn't been designed for. It seems it's very much possible to design chips that superficially looks like it should act one way but where it acts differently in certainly circumstances.

Doing so in a way that's exploitable without being caught out is probably much harder, but I share your skepticism.

<comment><USER5> That's not how PCBs work at all. Dear god.

Such mission critical to EMI designs would be submitted to full spectrum EMI tests once assembled (hitting it with noise of all kinds of frequencies AND testing it for leaks of all kinds).

Causing a bare board to fail? lol, perhaps if you have an atomic disintegrator. Assembled boards with actual components? Now you have something you can actually fail but this PCB house does not do said assembly nor have access to that.

Figure out what data? Completely impossible. These are just copper traces. They don't get told "Mr Copper trace, you are going to do i2c and send XYZ data! and you are going to be a pci express and send ABC!" at the fab. Jeez.

 _Now a real cause for concern_ , if say a state actor had access to the completed assembly going into the planes, they could identify a place to embed a backdoor. The state actor controlled PCB house could then embed the backdoor into the PCB itself. But at that point you have 2 different security breaches and the PCB guy is probably lesser of your concerns.

The dumber thing would be to alter the current carrying capacity of power traces which tend to be obvious staring at a board. But it would just identical to trolling by causing boards to fail and eventually get it investigated and your supplier contract revoked.

<comment><USER6> > They don't get told "Mr Copper trace, you are going to do i2c and send XYZ > data! and you are going to be a pci express and send ABC!" at the fab. Jeez.

I do only very basic electronics, so I'm asking here in honest confusion: Why _can 't_ you figure that out? I'd have expected that part footprints, high- level topology, fine details of individual traces, and the general need for everything to make sense at all would tell you everything you needed to know.

RF optimizations (squiggles to control propagation delay, weird shapes in corners, notches) (edit: wouldn't these give you the bus's frequency too?) and thermal tweaks (wider traces, bigger vias) would give you information about the pinout of whatever's plugged into each footprint (fast, slow, low-power, high-power, etc). Basic topology (is a line connected to two pins or thirty, is a line isolated or is it part of a sixteen-wide or ninety-wide bus) would narrow that down even more. More important parts will almost necessarily have more lines coming out of their footprints so you'd have more information about the stuff you care about.

Even if we assume there're no standards in use ("941 lines, power here here and here, high-frequency bus with 288 lines here, so that's socket AM3, which means these are..."), I wouldn't have surprised if you'd told me you could figure out what part numbers they're dropping into the slots for microprocessors and similar. What am I missing? Why _can 't_ you figure out which lines are i2c to misc peripherals and which lines are PCIe between CPU and key coprocessors?

<comment><USER7> > > They don't get told "Mr Copper trace, you are going to do i2c and send XYZ > data! and you are going to be a pci express and send ABC!" at the fab. Jeez.

> I do only very basic electronics, so I'm asking here in honest confusion: > Why can't you figure that out?

You can. OP is trying to be snarky, but he's wrong.

I don't think it's worth getting into an argument with someone with such an obviously demeaning attitude ("Dear God," "Jeez," etc.), but a few points:

1\. It's not just bare copper, silkscreen frequently tells you what's going on, even in secretive government airplanes.

2\. Even without silkscreen, e.g. on assembled boards where the chip part numbers have been lasered off for "secrecy," you can often uniquely identify a chip family just by the pins connected to power/ground, certain passives like crystals or filters, etc.

3\. If it's a chip with highly remappable pins where you can't just look up which ones are i2c in a datasheet (e.g. FPGA), you can often find them routed together with obvious strategies for impedance control and/or shielding.

4\. For i2c in particular, you will likely see two wires routed together, each with a resistor-like footprint pulling the line high.

Amateurs regularly do stuff like this... It's silly to assume a state level actor couldn't do the same or better.

</comment></comment></comment></comment></comment></comment><comment><USER8> I understand in the abstract why the US is freaking out about China as their economy is eclipsing the US and they are starting to create a competing international order to the WTO/IMF type deal with the belt and road initiative. However, I don't understand exactly what is triggering the national security establishment's (usually stupid and deadly) sudden paranoia about China.

Many people I know simply see these headlines and start to see China as an official enemy. I don't understand why we should see them that way at all. The official enemies of the US are typically brutalized in various ways and slandered in the media. It is difficult to know what is to be taken at face value.

<comment><USER9> I think the concern is that a nation that has literal concentration camps and exhibits any-western thought and agendas is going to become a major power.

China is hardly going to "eclipse" the US. I think the concern is less economic and more geopolitical. It should worry the West that a highly authoritarian nation is becoming wealthy and powerful.

It's a reasonable concern, I think. Once China gets hard geopolitical power it will start to export its own way of thinking, just as the US did. But unlike the US, which at least in theory believes in basic rights, China is a literal authoritarian surveillance state with no such values.

I worry that state run concentration camps like those in xinjiang will become normalized.

<comment><USER8> Yea, China definitely has problems no question about that. Xinjiang is one of the points I am most concerned with. However, the US doesn't take actions based on anything resembling goodwill towards others (we have a network of torture stations and kidnapping black sites don't forget). In many important ways, the US can be described as highly authoritarian, particularly when it comes to foreign policy (witness combinations of illegal invasions, brutal sanctions, and sometimes decisive support for death squads in Panama, Venezuela, Cuba, Iraq, Guatemala, etc).

I do think the world should bring more attention to Xinjing, but I don't think escalating tensions is going to do anything worthwhile (especially when I haven't heard a whit about the trade war being sanctions for humanitarian violations). The trade war is also not going to help American workers, capital controls would do that though.

</comment></comment></comment><comment><USER10> 1) malicious actors get information from multiple sources and combine them. We talk about this with data sets that are leaked. I wonder what that means for things like this and China as a state actor

2) contractors on military projects are highly regulated. I wonder if any laws or contractual agreements were broken by this

</comment><comment><USER11> I've read several articles on this issue. It doesnt appear they are even doing the pick and place. They are just manufacturing the PCB. My opinion is it's a non issue that some politics have arisen.

<comment><USER12> Wouldn't it still be possible to guess a lot about the design and find potentially exploitable mistakes? Or to build defects into the traces so that they fail in use due to vibration, temperature, or poor mutual inductance? Transmitters do seem far too difficult to get working in a shielded environment and prone to discovery.

</comment></comment><comment><USER13> If I were China I would take steps to ensure the boards degrade after some years. That would ensure they pass inspection and get installed. Then the parts can start falling off.

</comment><comment><USER14> As far as I was aware, the UK didn't have access to the source code for it's own jets anyway. Does it make much difference if yet another nation adds it's own backdoors?

[https://www.thinkdefence.co.uk/2009/11/us-to-keep-jca- source...](https://www.thinkdefence.co.uk/2009/11/us-to-keep-jca-source-code/)

</comment><comment><USER15> Leaving all the real espionage possibilities aside, I wonder if this is not a modern day propaganda machine doing its job.

</comment><comment><USER16> One instance of a single part of any device being tampered with by the Chinese government would be enough to cause every western company to move all production out of China.

</comment><comment><USER17> Reminds me of Battlestar Galactica first episode, except with terrestrial jets instead of battlestars and vipers.

</comment><comment><USER18> Does this feel like propaganda to anyone else?

<comment><USER19> The weird thing with propaganda is that it can often be completely true in each part but represent a lie in totality.

You run ten completely truthful stories about X doing something but no stories about Y doing the same thing.

The perception then becomes skewed towards Y.

Is manufacturing a PCB in China for a critical piece of defense equipment a security risk, possibly, it's also a possibility the same is true if they made that PCB in France or Israel or Japan (though I'm sure the threat curve is different).

The other questions I'm curious about are things like, ignoring the F-35, how much other equipment in service is dependent on Chinese manufacturers - electronics are everywhere and China is the nations factory (of sub-assemblies at least).

<comment><USER20> > _" The weird thing with propaganda is that it can often be completely true > in each part but represent a lie in totality."_

Propaganda can also straight up tell the truth. Propaganda, contrary to the impression given to many put through the public school system, is not a synonym for "lie". Propaganda is often lies, but being dishonest is not a defining characteristic of propaganda.

One famous example: _Loose lips sink ships._ Okay, maybe that's a bit exaggerated, but it's generally pretty true. In a war, breaches in secrecy can easily cost lives. Those loose lips posters were still unambiguously propaganda though.

</comment></comment></comment><comment><USER21> Let's turn around the discussion... if Russia had decided to outsource the PCB's for some new stealth fighter to a US manufacturer you can bet that several cloak-and-dagger US agencies would be looking for any way possible to compromise and take advantage of the opportunity.

<comment><USER2> Maybe, but their hands would be tied by the fact that a PCB isn't a very effective way to do that kind of espionage.

I'm not saying there's no possible conflict of interest here or that China wouldn't try something or that we should continue to use Shenzhen Fastprint for circuit boards to go in our super premium fancy stealth doodads.

I'm saying this is stupid and a non-issue and people should calm down.

<comment><USER21> > isn't a very effective way to do that kind of espionage

If you know very little about someone, just examining their laundry will tell you quite a bit about them. So 'effective' is really relative.

And, I'd argue that just having their hands on a physical component gives them the ability to introduce purposeful exotic flaws which they can exploit, or just ensure failure sometime in the future yet aren't detectable by standard quality control methods. Kinda like 'zero-day' code flaws that are kept in ones back pocket until needed.</comment></comment></comment>


DigitalOcean opens a Singapore Datacenter - <AUTHOR> https://digitalocean.com/blog_posts/we-re-excited-to-announce-our-singapore-datacenter-sgp1 <comment><USER2> Ok, this is nonsense!

DigitalOcean did not "open" a data center in Singapore. All they have done is rented some racks / colo space from Equinix in an existing facility.

[http://www.equinix.com/locations/singapore- colocation/singap...](http://www.equinix.com/locations/singapore- colocation/singapore-data-center)

They're just another customer - to spin and imply they have built or opened a brand new data center is ridiculous.

 _" Our decision to open a new datacenter"_ is really "We've located servers in Singapore". What's wrong with telling the truth?

Just more evidence that DigitalOcean is all PR and hot air after their pot- shots at Linode and brushing off security issues as "features".

<comment><USER3> They haven't said anything untrue. They're open about the fact they're working with a colo provider and using a common definition for "datacenter" appropriately.

While "datacenter" can refer to the physical building, it's also commonly used to refer to a set of server/storage/network resources physically cordoned off for your use in a colo.

Using this definition isn't just common with hundreds of other hosters, its well standardized in IT circles, and used as the official definition of the word "datacenter" by numerous industry groups, regulators, analysts, and the US Federal government.

For instance: [http://searchdatacenter.techtarget.com/definition/data- cente...](http://searchdatacenter.techtarget.com/definition/data-center) [http://www.whitehouse.gov/sites/default/files/omb/assets/ego...](http://www.whitehouse.gov/sites/default/files/omb/assets/egov_docs/cio_memo_fdcci_deliverables_van_roekel_3-19-12.pdf)

The whitehouse.gov pdf above says: "under the FDCCI, a data center is now defined as a closet, room, floor, or building for the storage, management, and dissemination of data and information. Such a repository houses computer systems and associated components...<snip>... housed in leased (including by cloud providers), owned, collocated, or stand-alone facilities."

<comment><USER2> I'll respectfully disagree. They are implying creation and ownership of an entire facility. They want to market themselves as being bigger and stronger than they actually are.

I guess hustling and bending the truth is rule 101 in marketing for start-ups these days, but it doesn't have to be like that.

When Blue Bottle open a new coffee shop, that's what they call it, an opening. If other coffee shops stock their coffee, Blue Bottle don't claim to have opened a new location, they simply say their coffee is available or served there.

<comment><USER3> Their announcement uses the phrase "Working with Equinix to ensure the highest quality facility". Since Equinix is a colo provider, that implies they're running in a colo.

If you read their FAQ ([https://www.digitalocean.com/faq](https://www.digitalocean.com/faq)), they're pretty clear that their other datacenters are in 3rd party colos: "We have datacenters located in the NYC area (located in the Equinix and Telx datacenters), San Francisco (Telx), and in Amsterdam (TelecityGroup)."

</comment></comment></comment></comment><comment><USER4> Happy to hear about the new opening, DO. Happy customer here - thanks for providing what has proven to be solid service and support for me (far better than my previous 'budget provider').

I am seeing a lot of complaints about DO that has nothing to do with this announcement... Traceroutes and ping times from the region - legit. Debate on the claim of 'opening a data center' (vs renting rack space) - legit (at least on topic). Complaining about lack of DO features - off topic and noise...

Let's be honest about what DigitalOcean is (and they may not like my categorization - I don't know, I'm just a customer) - they are a low price / budget hosting provider... I am not saying that people's complaints about lack of features or problems aren't legitimate complaints, but that they don't particularly belong here or add value to this discussion. DO has been on here a lot - whining about wants and complaints on every post is lame...

You want to voice your complaints? Write a post about why DO doesn't live up to your dream of all that a $5/mo VPS solution should live up to on your little blog and see if it makes it to the front page - then you can rightfully whine about lack of IPv6 all you want and know that you're not just thread dumping on a young small company's enthusiastic announcement about expanding their services. Or... Here's a thought for all of you super awesome critics who clearly know what's up way more than DO does - go make something superior and sell it for $4.50/mo!!!

<comment><USER5> Perfect example from another comment: "I'm worried about DigitalOcean. I think their priority should be to try to approach feature parity with Amazon or Rackspace."

Maintaining/achieving solid quality and reliability with their current, or a slightly extended feature set, while maintaining their awesome price point is FAR more important than feature parity with the biggest VPS hosts in the world. We don't need another hosting provider that has every feature ever at a premium price. We need a provider with a smaller feature set that is more affordable.

<comment><USER6> +1

That doesn't mean no new features, but we are certainly not rushing products out the door, market pressure or not. We know what we're doing. :)

</comment></comment></comment><comment><USER7> Something seems off with the routing. I'm on one of the big 3 internet providers in Singapore, and I'm getting a 300-400ms latency to their ping server (speedtest-sgp1.digitalocean.com) and to a droplet which I just started.

        DO <-> SL ~2ms
        DO <-> AWS ~2ms
        Me <-> SL ~15ms
        Me <-> AWS ~15ms
        Me <-> DO   350ms

<comment><USER8> Not the best from Australia yet either.

        --- speedtest-sgp1.digitalocean.com ping statistics ---
        49 packets transmitted, 45 packets received, 8.2% packet loss
        round-trip min/avg/max/stddev = 262.448/360.721/530.517/66.088 ms

<comment><USER7> Despite what DO said in their announcement, my experience says that Australian users are better served by servers in California than Singapore.

<comment><USER8> They're all terrible, but SFO does look better for me than the rest.

        --- speedtest-sgp1.digitalocean.com ping statistics ---
        50 packets transmitted, 49 packets received, 2.0% packet loss
        round-trip min/avg/max/stddev = 281.718/334.077/458.775/43.216 ms

        --- speedtest-ams2.digitalocean.com ping statistics ---
        50 packets transmitted, 48 packets received, 4.0% packet loss
        round-trip min/avg/max/stddev = 368.574/423.556/581.576/46.758 ms

        --- speedtest-nyc2.digitalocean.com ping statistics ---
        50 packets transmitted, 47 packets received, 6.0% packet loss
        round-trip min/avg/max/stddev = 283.225/338.437/428.314/31.986 ms

        --- speedtest-nyc1.digitalocean.com ping statistics ---
        50 packets transmitted, 49 packets received, 2.0% packet loss
        round-trip min/avg/max/stddev = 284.853/339.193/424.864/29.139 ms

        --- speedtest-ams1.digitalocean.com ping statistics ---
        50 packets transmitted, 47 packets received, 6.0% packet loss
        round-trip min/avg/max/stddev = 352.165/453.513/584.099/52.971 ms

        --- speedtest-sfo1.digitalocean.com ping statistics ---
        50 packets transmitted, 48 packets received, 4.0% packet loss
        round-trip min/avg/max/stddev = 205.293/324.091/445.077/54.935 ms

</comment></comment></comment></comment><comment><USER9> I'm worried about DigitalOcean. I think their priority should be to try to approach feature parity with Amazon or Rackspace. As of now, there are many features that are missing, such as private networking (only available in NYC2), CDN, elastic IP, load balancing, etc.

How about the ability to support attaching metadata to any instance?[0] How difficult is this to implement? More importantly, how would a devops person manage many instances without this feature? Isn't ex-EC2/Rackspace/<others> the target market for DigitalOcean?

\--- [0]: [http://digitalocean.uservoice.com/forums/136585-digitalocean...](http://digitalocean.uservoice.com/forums/136585-digitalocean/suggestions/3682231-add- tags-to-instances)

<comment><USER10> Not to mention IPv6. I like DigitalOcean but am starting to think of completely ditching them not just because they don't have it (delays are understandable), but because the last word from them (around August or September last year) was that they'd have a public beta in October, and then gone completely silent.

No public beta happened, and now they won't respond to any customer queries about it. I'm fine with there being delays, but very not fine about the silent treatment (or vague answers like 'it's in development' \- actually give some detail of what's been going on because it's been 'in development' for something like three years now).

<comment><USER9> I guess the reason is because their focus is primarily on expansion. They're trying to put on their big boy pants too soon.

<comment><USER11> Or they're simply under investor pressure to put on those big boy pants over achieving feature parity.

<comment><USER12> Or they're simply, you know, responding to demand.

<comment><USER13> You hit the nail on the head. We provided, and more specifically, I, directly did, estimates on when we launch certain features.

Unfortunately those deadlines have come and gone. That was because those estimates were rooted in timeframes around how we were able to write code in 2012 when we didn't hit the large growth spurt we did in 2013.

As a result the majority of 2013 was about meeting customer demand. Luckily we've finished out the year very well in scaling the various teams within the company and adding in the necessary layers of management so that we finally feel that we are getting back on track with engineering again.

With the launch of Singapore there are actually alot of updates under the hood that customers don't see but the hypverisors in Singapore are running on a completely new version of our cloud backend.

As we close out February we are reviewing our product roadmap and will start providing feedback again on estimates and hopefully be once again much more accurate in those estimates.

One of the other items we've considered is putting together a blog post about our 2013 in review and specifically focusing on what happened to our product roadmap and what delays we hit and why, and how we are looking to get past those challenges in 2014.

Thanks!

</comment></comment></comment></comment></comment></comment><comment><USER14> And still no IPv6 support :(. When will they make it a priority?

<comment><USER15> First thing I said when I came in - glad to see others say the same!

I just figured out I can run IPV6 hosts at home on Comcast and ping them from an IPV6 ping gateway page. I'm still scratching my head on how that works and whether or not my router allows all IPV6 packets inbound to all hosts. Tried to test on AWS, but that's not working. Went to my DO account, and still no support.

Rackspace supports IPV6, but I think something wonky happened to my account when I was testing the developer credits for them. I need to get it fixed.

Anyone have IPV6 at home and want to help me test it?

<comment><USER14> Depending on what your router is running (hopefully OpenWRT), ping6/ICMPv6 should be allowed to any host by default. Check out your firewall rules and see what it says there. ICMPv6 is an important part of IPv6, and in general not being able to ping something isn't really a security gain, so much as a usability loss.

On my OpenWRT setup, the default is to rate limit ICMPv6 to 1000 requests/second and to limit the response types.

<comment><USER15> I'm able to SSH into my internal servers. Configuring OpenStack now to provision the addresses for instances as well.

I have an Asus RT-AC66U. Tried to flash with OpenWRT and a few others and failed a few months ago. It would appear it DOES NOT have a firewall enabled, so I'm wide open, so to speak.

<comment><USER14> Ooh, that's not good. I would suggest you put up a firewall directly on as many devices as you can. Then, try to get a router upgrade, or a router that can handle an IPv6 firewall. I think I know the router you have and I believe TomatoUSB can run on it, which has ip6tables installed.

<comment><USER15> I know it seems like it's not good, but I was thinking about it and even with Comcast's /64 aggregate, I have a billion BILLION addresses available inside my network. If you could scan at a billion addresses a second, it would take 30 years to scan all of Comcast's IPV6 addresses. That's crazy.

<comment><USER14> Well, not really. The first 64 bits are not all possible. They are subdivided, since some addresses are link-local, some are multicast, etc. Then, Comcast only has a certain allocation of that. On top of that, could one find a patter in how they allocate their addresses?

The second 64 bits are also not quite random. Most of your devices will autoconfigure using radvd. This means that the second 64 bits depend on their MAC address. Now, if I knew of an exploit to, say, a printer or a NAS device, I would know the MAC address range. My guess is that I could probably reduce the 128 bit address space to something like 100 or even 90 bits.

Second, and this makes it all the above a moot point, don't your devices connect to the internet? Any time they connect to a site, that site knows the IP address and that data may be used either explicitly or leaked and used by someone else. Everyone between you and the site also knows the address.

Lastly, if you ever set up a DNS record for any of these addresses, they are then visible to others even with some scanning if you don't ever publish the actual names.

Long story short, there is hoping you don't get hacked and there is knowing you have a firewall that only allows what you want in.

</comment></comment></comment></comment></comment></comment></comment><comment><USER16> How does DigitalOcean compare to RamNode? I've been using a small VPS at RamNode and it's been great. It offers pretty much the same features (SSD, 1 CPU, 1 Gbps connection) as DigitalOcean (plus RamNode offers 16 IPv6 addresses), yet it's half the price. The only downside to RamNode is the 500 GB of bandwidth compared to 1 TB at DigitalOcean. Maybe I'm missing something, but why would one use DigitalOcean over RamNode for VPS hosting?

<comment><USER17> 500GB bandwidth you say? That is the plan with only 128MB of memory; you can't run many things with so little memory. The 512MB plan at RamNode is $7.5, 50% more.

<comment><USER18> Is that factoring in the coupons they keep putting out there? I paid $100.44 for a year after their 38% coupon (38RLY) and signed up for 1024 MB RAM plan, 150 GB disk, 2 IPv4 addresses, 3000 GB bandwidth. It's great.

<comment><USER17> If you're looking for a bargain, I found this:

        2GB RAM, 50GB disk, 2TB bw, 2 IPv4, 100Mbps
        STEALOFADEAL (coupon) - 12 months, $40
        Chicago, IL - https://billing.chicagovps.net/cart.php?a=add&pid=160
        Buffalo, NY - https://billing.chicagovps.net/cart.php?a=add&pid=161
        Los Angeles, CA - https://billing.chicagovps.net/cart.php?a=add&pid=162
        Atlanta, GA - https://billing.chicagovps.net/cart.php?a=add&pid=163a=add&pid=163

<comment><USER19> ChicagoVPS are terrible. If you Google ChicagoVPS the suggest searches are "ChicagoVPS down" and "ChicagoVPS hacked".

There's 10 million threads on Lowendtalk bitching about ChicagoVPS. Here's one from last year. [http://lowendtalk.com/discussion/7668/chicagovps-has-lost- th...](http://lowendtalk.com/discussion/7668/chicagovps-has-lost-their-mind-s)

<comment><USER17> You get what you pay for.

</comment></comment></comment></comment></comment></comment><comment><USER20> Why are those complaining about routing issues posting ping outputs instead of traceroutes?

<comment><USER21> It doesn't matter. Even if they posted traceroutes, they'd only be from one direction which is near useless anyways.

</comment></comment><comment><USER22> I started a SG droplet about 5 hours ago. It's still down. They're suffering some outage and it's a shame it isn't mentioned on their status page. #fail

[https://twitter.com/kaihendry/status/433451127657357312](https://twitter.com/kaihendry/status/433451127657357312)

I'll stick with GPLhost for now [http://webconverger.org/servers/](http://webconverger.org/servers/)

</comment><comment><USER23> They need to keep on top of existing demand as well and start running over- capacity. Not joking but it's pretty long wait to get an instance up in a preferred DC sometimes which is putting a lot of us off the platform.

<comment><USER24> I am going to keep machines at Digital Ocean, but they have had the most failures/downtime I have seen from any host that I've used in nearly a decade, including ec2, rackspace, and gogrid (if there is anyone left using gogrid.)

One problem we've run in to is an instance fails, but it can't be brought back up because the backup image is only located in the datacenter that is already at capacity.

<comment><USER23> A million upvotes if I could.

I didn't actually consider that last point. That has effectively killed DO for me instantly from a risk point of view and I'm going to immediately migrate my two machines off it. Literally it's 22:50 here tonight and I don't think I'll sleep until it's done.

Have some company credit left over with Azure so will throw it on there on Ubuntu for now.

</comment></comment></comment><comment><USER25> It used to be that it was possible to take a snapshot of a DO droplet without shutting down the droplet, provided the droplet had 1GB of ram or higher.

Does anyone know if this is still the case?

<comment><USER26> I don't know if that was ever the case, but at this point you do have to power off the droplet to make a snapshot: [https://www.digitalocean.com/community/articles/digitalocean...](https://www.digitalocean.com/community/articles/digitalocean- backups-and-snapshots-explained)

<comment><USER27> You can take a "backup" without powering down, but not a "snapshot".

</comment></comment></comment><comment><USER28> I hope a Canadian Datacenter (preferably westcoast) is in the cards soon!

<comment><USER27> Make a uservoice post about it if there isn't one.

</comment></comment><comment><USER15> I wish they would add IPV6 support.</comment>


Show HN: PhantomJS URL screenshot service powered by Docker - <AUTHOR> http://stackhut.com/#/services/phantomjs

<comment><USER2> Isn't this powered by phantomjs more than docker?

Surely docker is just a deployment strategy?

<comment><AUTHOR> Absolutely, we're more concerned w/ use-cases vs. Docker itself. PhantomJS is a great tool, and we're using it to show off the platform which hosts this sort of stuff in the cloud (with Docker!). Would love to get some more feedback: firstname.lastname@example.com :)

<comment><USER2> If it's just for taking screenshots I'm curious why you chose phantom rather than say wkhtmltoimage/pdf?

I've used phantom (via Casper) for some scraping tools but for images I would have thought the wkhtmlto* project was simpler?

<comment><AUTHOR> Hey Stephen -- Honestly, it was just what I had more experience using; I always presumed Phantom is more consistent and malleable, albeit sometimes a pain to configure. You are definitely right on the simplicity aspect.

We're aiming to extend it past basic screenshotting -- is there any functionality you'd like to see? (moreover, are there any other services we could chuck on the platform that you'd like to see?)

Link to GH issues here, too: [https://github.com/StackHut/phantomjs/issues](https://github.com/StackHut/phantomjs/issues)

<comment><USER2> Personally? No. I specifically don't use hosted services wherever I can self- host. My interest was purely technical, rather than about a specific use-case I had in mind.

</comment></comment></comment></comment></comment><comment><AUTHOR> Hey all, I'm one of the developers, so if you've got any questions -- shoot away!

</comment><comment><USER3> Didn't take my screenshot properly I'm afraid.

<comment><AUTHOR> Hey Duboff! Sorry to hear that -- what website were you trying to screenshot? Would love to try and fix your issue: you can reach me at firstname.lastname@example.com .</comment></comment>


Angela Merkel, German Chancellor, Is Time 'Person of the Year' - <AUTHOR> http://www.bbc.co.uk/news/world-europe-35048796 <comment><USER2> > Mrs Merkel, 61, joins an eclectic list of former winners, including Adolf > Hitler, Joseph Stalin ...

A German is given the title and those are the first two names you choose to use?

</comment><comment><USER3> [https://news.ycombinator.com/item?id=10704198](https://news.ycombinator.com/item?id=10704198)</comment>


Amazon Updates Route 53 DNS Service - <AUTHOR> http://techcrunch.com/2013/05/31/amazon-updates-route-53-dns-service-to-make-hosting-high-availability-sites-on-ec2-easier/

<comment><USER2> Source of paraphrasing: [http://aws.typepad.com/aws/2013/05/amazon- route-53-elb-integ...](http://aws.typepad.com/aws/2013/05/amazon-route-53-elb- integration-dns-failover.html)

<comment><USER3> It is sad to see sites that were once well focused covering anything and everything, presumably in desperate pursuit of a few more page views.

I'm sure this is of interest to some people, but Techcrunch? Really?

</comment></comment><comment><USER4> Multi region failover is seriously inhibited with their current VPC setup as cross-region traffic is ridiculously complicated to get working.

We are on the "EC2 Original" setup with a Cassandra cluster operating over regions via Elastic IPs. We tried to setup a multi-region cluster on their "EC2 VPC" mode and gave up. A VPC can only span a region and currently no easy way to manage traffic between them. We looked at getting a VPN connection but the cost for a decent connection gets way prohibitive.

<comment><USER5> What do you mean by a VPN connection? Doing direct connect and then your own transport, or just the aws IPSec tunnel stuff over their normal transport?

<comment><USER6> I am curious about this as well... Why can't you connect between different VPCs?

</comment></comment></comment><comment><USER7> On this topic, today HP Cloud opened access to their Akamai Anycast DNS service with 75 edge locations. Pricing is about the same as Route 53 ($0.35/domain + $0.55/million queries). Not as easy to use as Route 53 yet - configuration is via API only.

<https://www.hpcloud.com/products/DNS>

<comment><USER8> Can anyone explain what Anycast is/why it is so huge now? I just looked at the Wikipedia article and it is quite Computer Science-y rather than talking in specific terms how it works or what the benefits are.

I understand how DNS works. UDP packet to server, returns IP from host via UDP, and or requests DNS record from a different DNS server with more information about the domain (all the way up to the root servers).

<comment><USER7> It routes a single IP address to different DNS servers depending on a user's location. It generally provides lower latency/faster DNS queries by routing users to closer DNS servers.

</comment></comment></comment><comment><USER9> Now if only they would lower the price; the issue that I had with Route 53 was that you pay not only for successful queries, but also for failed lookups (NXDOMAIN). So if someone does a lookup for "nonexistantsubdomain.yourdomain.com", you pay for the query - even though Amazon didn't serve anything.

Not a huge deal, but we had a few domains that were getting hammered with spam/bruteforcing, and with Route 53 there is no way to find or block them - you just gotta keep paying for the queries.

<comment><USER10> Isn't that just the way the cookie crumbles, though? Route 53 still has to look at a DNS query to decide what to do next, and some of them will be bad lookups.

</comment></comment><comment><USER11> I'm not sure how well this is going to work out in practice, but the concept of having multi-region failover could mean that much closer to almost entirely bulletproof infrastructure (if you are willing to spend the money).

<comment><USER12> Not really. This is no different than running your own, say, Haproxy with a heartbeat between two boxes next to each other on the same rack or some such.

That Amazon now with a bit of fiddling will fail over to another region if your ELB instance is down or because it detects errors is great - but hardly bullet proof.

What if your ELB instance is ok but your app is returning garbage.. its still returning something, right? And ELB will happily forward that traffic without thinking anything is wrong.

<comment><USER13> Full disclosure: Route 53 developer here. There is one interesting difference; when using DNS failover in combination with Latency Based Routing, Route 53 supports partition mode failures.

For example; if a customer has endpoints available in both the AWS Sydney region and an AWS US region, and Australian international connectivity is impaired then users within Australia will still go to Sydney. At the same time, a user in New Zealand who would ordinarily go to Australia (as it's closer) may now find themselves served by US endpoints because reachability to Australia from New Zealand is impaired but reachability to the US is ok. It's a small part of the availability story, but one difference in how a DNS failover may handle an event.

The "returning garbage" problem can be a hard one - both Route 53 and ELB can be configured to check a particular url for health-status, and it's important that that url's health be indicative of the overall stack's health, but it's definitely a challenge sometimes as an application operator to maintain a good "deep" check. For example, on my own personal Wordpress installation I check if the DB is reachable and answering before returning 200 from my status URL, but I've seen service owners do much much more comprehensive checks including inspecting some metrics, counting overall 500s, measuring response times, and so on.</comment></comment></comment>


AI Dungeon 2 costing over $10k/day to run on GCS/Colab - <AUTHOR> https://twitter.com/nickwalton00/status/1203370250030350338 <comment><USER2> Previous HN thread since I missed it: [https://news.ycombinator.com/item?id=21717022](https://news.ycombinator.com/item?id=21717022)

</comment><comment><USER3> Cloud bandwidth costs are a rip off. For this particular situation though, make sure to replicate your files across multiple buckets, one in each GCP zone, otherwise you incur the cross zone transfer costs. We do something similar to serve julia downloads, because it turns out that most downloads are from people running on the cloud (so we basically replicate our binaries to every cloud provider and then to every region for that cloud provider). Everything else then goes through fastly for us (we also use fastly to serve custom redirects if your request comes from one of aforementioned cloud providers). That works pretty well. You do have to monitor it though to make sure that load doesn't suddenly shift to a cloud provider you didn't account for. For example, after GitHub actions became widely used, we suddenly started seeing TB/day traffic from Azure, which we hadn't deployed any caches to, so our bandwidth utilization on Fastly shot through the roof. (Side note: Shout out to fastly for hosting our binaries for free!). Without the cloud provider caching setup, we'd probably be at similar $/day costs, but this way it's basically free (even without the free fastly service, we'd only be at ~$1000/month or so).

<comment><USER2> Can you describe a little more how you direct downloads to an in-zone copy?

<comment><USER4> I imagine using known IP ranges ([https://docs.aws.amazon.com/general/latest/gr/aws-ip- ranges....](https://docs.aws.amazon.com/general/latest/gr/aws-ip- ranges.html#aws-ip-download) & [https://docs.aws.amazon.com/general/latest/gr/aws-ip- ranges....](https://docs.aws.amazon.com/general/latest/gr/aws-ip- ranges.html#subscribe-notifications)) you could redirect to local resources within their region.

I'd probably use Lamda/NodeJS with a ~20 minute in-memory range cache that should stay under 128MB of memory per instance. Perhaps I'd store all the range starts and ends each as two 64-bit ints on a database of your choice for persistence, indexing, and comparison. Finally, some code to convert IPv6 and IPv4 into 128-bit (2x64-bit ints) IPv6 integer space and back.

A second service could listen for IP range updates avoiding any bandwidth fiasco if a new range opens up with a sudden influx of traffic.

<comment><USER5> I think they were looking for a solution where you replicate the data natively and direct to that copy for a certain region, rather than cache and serve

</comment></comment></comment></comment><comment><USER6> Not blaming them, but still a nice example to illustrate the myth that 'the cloud' will let you operate without a sysadmin (someone that knows about setting up and managing infrastructure). Yes, there might not be servers you have to rack and electrical power to be provisioned, but the amount and granularity of the parameters and decisions on the virtual fabric to be decided on and continuously monitored for changes and opportunities that could have an enormous impact on performance and cost is orders of magnitude higher.

<comment><USER7> Also not blaming him/them but isn't this an indicator of lack of cost control at research facilities? Surely someone should of costed this out before making it public?

Here in the UK running at 10k/day would of ruined a departments budget for an entire year!

<comment><USER6> Costing for cloud solutions is often not trivial, and a single overlooked parameter can do you in. I at least blame Google partially for having quotas and spending limits set to unlimited by default.

<comment><USER8> Don't forget Amazon's incomprehensible billing practices for AWS. It seems to be designed to be impossible to figure out what you're going to spend ahead of time.

<comment><USER9> Amazon really seems to be doing this on purpose. Even tasks as simple as starting an EC2 instance require referencing multiple pages to find out how much this instance will cost you, while some other providers just show the predicted costs/month in the instance creation dialog.

</comment></comment></comment></comment></comment><comment><USER10> Wait, wait, this is a university developing something, that needs to be downloaded, they choose a commercial solution for that, and that costs them $10k per day? They don't have their own file-hosting solutions? At a university with an AI dept?

With that money ("for a couple of days"), you could buy your own server and do it yourself, for a fraction of the price. And don't tell me you can't find people who'd be capable of doing that, if you've got AI development going on.

What am I missing?

<comment><USER11> It doesn't seem to be as simple as downloading a binary.

We (those who read HN) could probably download the code and run it at home, but I think the authors want non-technical users to play. To do that, they need an accessible Python runtime, so they're hosting the game in a Colab notebook. The download in question is referring to downloading the weights of the neural net into the VM running the notebook.

If only redistributing Python apps wasn't so difficult.

<comment><USER12> The game's GitHub page[1] states that you would need a "beefy" GPU ~12 GB and CUDA to play the game locally.

I think that's why the author was serving the game through Colab since the majority of users probably don't have a 12GB GPU.

[1][https://github.com/AIDungeon/AIDungeon/](https://github.com/AIDungeon/AIDungeon/)

<comment><USER10> Ooof, yes, now I see where the $10K/day is coming from...

As I have demonstrated, I've really not much of a clue when it comes to AI, but do users really need 12Gb GPU RAM, 100% of the time? Maybe it's possible to use one GPU for multiple users?

<comment><USER13> Google Colab gives each user a dedicated Nvidia Tesla K80 GPU for 12 hours for free, which is super cool and presumably why the project is on Colab. But as each user spins up their own Colab instance it pulls down the 6GB of GPT-2 model weights, incurring 30-40 cents of data egress charges against the GCP Storage Bucket that the data is stored in.

60k users yesterday * 6GB each -> 360TB of data egress!

Normally, a scenario like this wouldn't involve bandwidth costs because GCP -> GCP same-region bandwidth is free, but Colab is technically not part of GCP, so the bandwidth charge is being assessed as egress to the public internet, which is pricy for that much data. Though it's probably still a lot cheaper than paying for the GPU-hours for that many users.

<comment><USER14> This is sort of a killer example for the layperson of what ML can do, so hopefully Google will recognize this and comp most/all the data egress since it's a drop in the bucket for them, but every person that uses it can still go "Wow, Google's services allows for some amazing stuff."

<comment><USER15> Oh, it does. NOT.FOR.FREE, though; that's the entire reason it exists at all.

In other words, yes, this is an amazing amount of raw power - with a corresponding price tag.

</comment></comment></comment></comment></comment></comment></comment><comment><USER16> Someone, probably: _People who don 't know how their services work are doomed to spend lots of money on them._

From the GCS pricing page:

      Network egress within Google Cloud applies when you move or copy data from one bucket in Cloud Storage to another or when another Google Cloud service accesses data in your bucket.

      Within the same location (for example, US-EAST1 to US-EAST1 or EU to EU) -- Free

From the original tweet (not the linked reply):

      For reference most of the fees are from transferring from NA to EU and ACAP

It's costing them assloads of money because they're moving data between regions. AWS works the same way. Azure is probably also the same but their pricing page is incomprehensible so who knows.

Lesson: if you're going to _use_ data in a given location, you need to _host_ data in the same location.

<comment><USER17> The gcs bucket just needs to be set to region? See documentation:-

Data egress from your bucket to a non-Cloud Storage Google Cloud service is free in the following cases:

Your bucket is located in a region, the Google Cloud service is located in a multi-region, and both locations are on the same continent. For example, accessing data in a US-EAST1 bucket with a US App Engine instance. Your bucket is located in a multi-region, the Google Cloud service is located in a region, and both locations are on the same continent. For example, accessing data in an EU bucket with an EU-WEST1 GKE instance.

EDIT 2:

the cloud ai documentation suggests the correct setting is regional/regional

[https://cloud.google.com/ml-engine/docs/regions](https://cloud.google.com/ml- engine/docs/regions)

Cloud Storage You should run your AI Platform job in the same region as the Cloud Storage bucket that you're using to read and write data for the job.

You should use the Standard Storage class for any Cloud Storage buckets that you're using to read and write data for your AI Platform job.

EDIT 3:

\</comment></comment><comment>----- I don't think you can set a region for colab, so I am not sure you can make egress free. </comment><comment>----

</comment><comment><USER18> Can I be controversial and ask what justifies such expenses? Why is this game so important that it needs to be up and running so badly?

From what I've seen so far the "game" (if you can even call it that) highlights the weaknesses of GPT-2 much more than its strengths (the model's answers to user actions are random, the story is incoherent, the world is inconsistent). I don't get the feeling it was setup to demonstrate _weakness_ though. I think it was meant to demonstrate strngths.

I suppose it's just advertising for the BYU Perception, Control and Cognition Lab, but it sounds awfuly expensive for advertising for an academic group.

<comment><USER19> What a cynical input into the conversation.

First you assume that this project even existing is some kind of statement about how important it is; a connection I can't make myself unless I try to be extremely cynical. If you actually look at the project website you'll see that the main mirrors are currently down due to high download costs.

Then you smugly dismiss the entire project as a GPT-2 weakness highlighter, but not without wrapping it in some passive aggressive faux concern ("Oh is this trying to be good? Silly me, I thought this was a showcase for how to be bad!").

And then you assume that this game that thousands of people (although none of them were you) have enjoyed playing is probably an advertisement for an academic lab. Despite there being almost no evidence for it.

Can I be controversial now and ask: What's with the bile?

<comment><USER18> >> First you assume that this project even existing is some kind of statement about how important it is; a connection I can't make myself unless I try to be extremely cynical.

That would be cynical, but I did not make this assumption. I questioned the justification of the high cost to maintain the project, not the existence of the project per se.

I did not doubt that people enjoyed the project but, again, I don't understand how a research lab justifies paying such a high cost to provide entertainment.

If the project could be maintained for free then I would not have any questions. But if a lab is spending $10k to keep a game running then yes, I have to wonder why.

I do think that the game shows up GPT-2's weaknesses. Do you really think it's "bile" or "cynical" to recognise weaknesses of a technology?

Perhaps I'm cynical to assume it's advertisement. I apologise if that's the case.

Note also that accusing me of bile is a personal comment that I think is unnecessary.

<comment><USER18> Also, can I please ask you to tone down the god-moding if you want to have a disagreement? "passive aggressive faux concern", "bile", what's all that about? Why do you think you have an insight to my state of mind and emotions after reading a comment I made on the internet? This is just frustrating. Is that how you want the internet to be, really? A bunch of people assuming each other are either assholes or idiots, and treating each other accordingly?

<comment><USER19> Agreed. That was much more aggressive than it should have been. I don't think I have an insight into your state of mind, and I'm sorry for using that to land some stupid burn. The point would have come across just as well without my bile. You're right, that's not what I want the Internet to be.

In that spirit, I would very politely suggest that you read your original comment and consider if it represents what you want the Internet to be.

<comment><USER18> Well, thank you, that's very welcome and I feel bad for making you apologise. I'm glad you see things this way!

So, I'm looking at my comment again and I think I should have omitted the following two sentences:

>> I don't get the feeling it was setup to demonstrate _weakness_ though. I think it was meant to demonstrate strngths.

>> I suppose it's just advertising for the BYU Perception, Control and Cognition Lab, but it sounds awfuly expensive for advertising for an academic group.

The first one does sound as if I'm taking a swipe at the research team. This was not my intention but it came out all wrong.

The second one makes assumptions about the motivation of the BYU research team, and I should have kept those to myself.

My original question, what justifies the high cost of the product, I feel is valid so I wouldn't change it. But, clearly, I took it farther than I should have. I apologise that I didn't think my comment through enough so as to avoid having it come across as an attack on the BYU team and I'm sorry it upset you.

</comment></comment></comment></comment></comment></comment><comment><USER20> Definitely worth it:

She blushes slightly and smiles shyly. "Oh, I'm sure we will. But first, let me take off my clothes".

> say "no, that's prohibited"

"No, it isn't". She says with a smile. "But if you insist on not marrying me, then at least don't touch me". > say "I will marry you"

She nods happily and kisses you passionately on your lips. The two of you embrace each other as you kiss her deeply. It is only after this that you realize that you are actually married.

<comment><USER21> Sounds awesome, it's like reading those curated short stories written by AI. The tech isn't quite there yet, but it makes it all the more hilarious. It's like an alien who's been watching our TV for years comes and tries to write a sitcom.

</comment></comment><comment><USER22> To be clear, this was because of crazy high network egress fees, not the fancy neural network compute.

</comment><comment><USER23> So to catch up, AID2 is leveraging a cute offering by Google Colab which will provide any Google account a few hours of free access to a fairly powerful GPU.

To use Colab under this arrangement, each person playing the game has to load the model into their personal VM, which is being billed by Google as 6GB of data transfer from GCP into Colab.

The obvious questions;

1) Was there a place they could have hosted the model closer to Colab so that they weren't being charged for egress bandwidth — and also, ideally, so that 6GB of data wasn't actually being moved very far?

2) The underlying model is 6GB, but I'm curious how much memory is required for an individual user's world state and how hard it would be to have a single GPU handling multiple user sessions?

Presumably it would be possible to multiplex multiple sessions with a single GPU? You would have to serialize the game state, receive the next input, load the prior state, feed the new input through the model, return the resulting text output, and re-serialize the state until the next input comes through.

What I don't know if that's at all practical based on the amount of data that would have to be serialized? Is the 6GB model data separate and static throughout the game, with an isolated block of data for the current world- state? Or does playing the game fundamentally alter the state of the model, meaning you would have to reload the whole thing just to process the next command?

<comment><USER24> "World state" is the text people see on script. It can reside in the browser.

So a server can be completely stateless, i.e. it receives text so far (say, 5 KB), applies GPT-2 generate and returns.

The problem is that GPT-2 generate seems to be very computationally intensive. As I understand, it actually does number crunching with all these 6 GB of data, so it takes 5-10 seconds even on GPU (K80, at least, is that slow).

Is GPU capable of running multiple GPT-2 generate in parallel? No idea.

Assuming that a high-end GPU would be able to produce response in 2 seconds, you can only run maybe 10 concurrent sessions per server if you want fast response time.

</comment></comment><comment> daniel-thompson Update: the issue has been fixed, apparently:

"Should note for anyone who comes and sees this that's no longer how were hosting the model. :) Model is now hosted on a peer to peer torrent network so no more costs for us."

[https://twitter.com/nickwalton00/status/1204064712394076160](https://twitter.com/nickwalton00/status/1204064712394076160)

</comment><comment><USER25> It appears that it has been updated to use bittorrent as a temporary solution: [https://colab.research.google.com/github/nickwalton/AIDungeo...](https://colab.research.google.com/github/nickwalton/AIDungeon/blob/master/AIDungeon_2.ipynb)

</comment><comment><USER6> Still not clear why a university research lab would keep this running 'for a few more days' and not pull the plug immediately until either having the costs under control or some sort of reasonal matching benefit in place.

<comment><USER26> Probably because they have to spend their grant money some way.

<comment><USER27> One hears stories about researchers with unspent grant money scrambling madly to find something, anything, to spend it on before the grant expires. Use it or lose it.

My Dad told me how back in the 1970s he worked at a government-funded research lab. One time they called up a laboratory glassware supplier and said "We'd like to order $10,000 worth of glassware". The supplier asked "Sure, what specifically would you like to order?" The lab replied "We don't care, whatever you have in stock, so long as it costs us $10,000 and we pay you today – if we don't spend the money today we lose it forever".

<comment><USER28> It sounds like there is a missing incentive to go under budget. There's something wrong with the system if people feel like they just have to burn money.

<comment><USER29> > It sounds like there is a missing incentive to go under budget.

Other way around. Right now there exists a negative incentive to go under budget, being that if you need extra budget next year you suddenly don't have a buffer.

The correct way to fix things is to simultaneously prohibiting end-of-FY budget-keeping purchases (e.g. by mandatory reviewing of purchases) while at the same time allocating a company/organization wide buffer for stuff that needs immediate / projected stuff that was not in the planned budget.

</comment></comment></comment></comment></comment><comment><USER30> You are Faif, a ranger living in the kingdom of Larion. You have a hunting bow and a quiver of arrows. You spot the deer and are ready to finish your hunt when suddenly you hear a voice calling out to you from behind a tree.

> nosaving

Saving turned off.

> shoot the deer

You quickly turn around and see a young man with a long beard standing there. He is dressed in rags and his face is covered by a large bandage. His eyes are red and puffy and he has a large scar across his cheek

> shoot the man

You fire an arrow at him and it hits him square in the chest. The arrow flies through the air and slams into the ground. You look up and see that the man is lying on the ground motionless. You can't believe what just happened

> Check his pockets

You pick up the arrow and examine it. It's made of bone and is very sharp. You wonder if you should take it. You could use this to kill someone else.

Doesn't seem to understand me too well?

<comment><USER31> It's hit or miss. You'll have better results being more specific in your commands because it gives the model more to work off of and let's you crystalize elements of the scene through reaffirmation.

"Fire an arrow at the deer" or "Check the pockets of the corpse".

I'm often surprised at how well the game can understand me and that it has anything to say in reply at all.

</comment></comment><comment><USER32> I realise Cloudflare is a dirty word around here but you could pay $20/mo on their pro plan and serve this file, along with any other static assets, through their CDN.

I do this for assets for my own games hobby site. Granted, not getting tens of thousands of downloads a day, but there's nothing in their Ts&Cs to indicate to me that it wouldn't work even if I were.

<comment><USER33> Why is Cloudflare a dirty word around here? I'm afraid I'm out of the loop

<comment><USER34> There's one particular commentator who is very vocal about his disdain for Cloudflare.

<comment><USER35> Never noticed. What's his problem though?

</comment></comment></comment></comment><comment><USER36> I had a lot of fun playing the game. I think this game shows us how interesting AI technology will become in the 2020s. It's open ended but somewhat incoherent right now, but I think we'll figure out how to update this kind of technology to have an open ended yet internally consistent world by the end of the 2020s.

<comment><USER36> As it turns out, it's actually possible to bona fide win the game.

In my case, I was dating two girls, one was uncomfortable with the other girl, and broke up with me, so I asked the remaining girl if she would marry me. At this point, she said yes, we rode off in to the sunset and the game proclaimed "CONGRATS YOU WIN" then it saves the game for me.

I guess I could load the game and deal with domestic squabbles, having children, growing old together, but I'm not sure this training set is optimized to generate a domestic married situation comedy story.

The torrent trick works. Right now, the game can be played at [http://www.aidungeon.io/](http://www.aidungeon.io/)

<comment><USER37> Torrent trick?

<comment><USER36> Instead of downloading the six gigabyte trained data from the cloud ($$$), they use a Bit Torrent client to download it, to keep costs down. It works, as long as considerate users seed the file.

</comment></comment></comment></comment><comment><USER38> Insane but also expected. When I tried it out when it was posted here and saw it took multiple minutes to warm up I knew it was probably expensive.

>And it's currently costing 30-40 cents per download.

Is there no way to have a single hosted instance rather than downloading again for each user?

<comment><USER39> > Is there no way to have a single hosted instance rather than downloading > again for each user?

This might make the problem worse; then they'd have to do processing server side, rather than offloading it on the client. I dunno whether this would be more or less expensive than the initial download, but the torrent they put up seems cheaper either way.

<comment><USER40> Weren't they already doing the processing server side? If it were client side then it wouldn't be costing so much to run git clone every time, as the download would be from GitHub to the user's computer. It would be free, in fact.

My impression of the situation is that every user who tried to play would result in a new instance to spin up on Google's cloud services and then begin downloading a fresh copy of the repo from GitHub. This is what cost so much in bandwidth.

<comment><USER39> A client side request to GitHub would require them to serve up a relevant CORS header, but I do think you're right about me misunderstanding where execution is taking place. I'm unfamiliar with Jupyter Notebooks, and assumed "downloading" meant "to the client". I, too, am now confused about why this is set up like it is. Probably some constraint of Jupyter Notebooks that I'm unaware of.

<comment><USER40> If you install and run Jupyter on your local machine, it'll spin up a web server on localhost and then connect to it in your browser. All of the Python code runs on the server and only the results are sent to the client, to be displayed in the browser.

</comment></comment></comment></comment></comment><comment><USER2> To play backseat problem solver... You have free ingress? So start up a few $5 DO droplets and serve files from there. That gives you 1TB transfer per month.

I haven't tried this, but my understanding is that's per droplet. So when drop A is about exhausted, start B and switch over the traffic. Then shut down A. Then start C and shut down B, etc. Unlimited transfer? (Until your account gets banned, anyway.)

<comment><USER41> You can buy bandwidth from a simple CDN for cheap enough to not need to do this dance.

<comment><USER42> That's the real story here. Paying market rate for bandwidth is like buying soda from a restaurant. They'll gouge you and make a profit but you're thirsty and it's too late to shop around.

</comment></comment></comment><comment><USER43> If they had put the bucket behind a CDN costs would have been dramatically lower.

</comment><comment><USER44> GCP charges ~$0.60/hour for the GPU/CPU/MEM equivalent in colab. If bandwidth is costing 10k per day, how much is the free colab compute costing Google?

<comment><USER45> Probably less. I doubt the average use time is more than half an hour. Pretty cheap advertising for the service, really.

</comment></comment><comment><USER46> When will we come full circle and do "edge computing" on our own devices again? I'm getting sick of exorbitant cloud costs and the moat that is forming.

</comment><comment><USER47> [http://www.aidungeon.io/](http://www.aidungeon.io/)

</comment><comment><USER48> The ominous opacity of the AWS bill – a cautionary tale (taloflow.ai) [https://news.ycombinator.com/item?id=21694835](https://news.ycombinator.com/item?id=21694835)

</comment><comment><USER17> I think the key architectural mistep is putting a colab session per user session. A fix is having the user call out to an service API, which, behind the scenes, is executed on a pool of colabs with reuse. I don't know of any technology to make that easy though. The symptom that could of been a trigger for cost investigations were the long startup times.

</comment><comment><USER31> I know almost nothing about AI/ML.

I can imagine that running it on a single server and responding to requests is intractable because of how the game feeds your quest back into its model.

But what would it take to package it as a local application?

I really love this game. You can't beat this:

[https://twitter.com/ptychomancer/status/1203246078989987840](https://twitter.com/ptychomancer/status/1203246078989987840)

> $ Give rousing speach to my fellow mud beings

> "Mud creatures! Mud creatures! We must unite against this enemy!"

> The other mud creatures nod eagerly, and begin chanting, "We will fight! We > will fight! We will fight! We will fight! We will..".

</comment><comment><USER49> I remarked how impressed I was with the state of technology that made it possible to freely spin up a VM with ~14GB of memory and beefy compute.

Now it all made sense--of course someone had to pay for this all. Doh!

<comment><USER50> Google is paying for the Colab notebook and compute, that's free for users. The problem was the ~6GB model hosted on GCS, GCS has very high network egress costs.

</comment></comment><comment><USER51> Why would a user need a 12gb gpu to run the game locally? The deep learning model is already trained, and I can't imagine one needs a gpu to just evaluate the model.

<comment><USER52> It's a _big_ model.

<comment><USER45> It's a 6GB download. Mid-high end GPUs are 8GB. So what makes it need 12GB? How specifically is that memory used up by various types of decompression, intermediate calculations, etc.

<comment><USER13> I believe that the 1.5B model weights take up the 6GB themselves. Presumably f32 weight values? Since they are all needed for an evaluation pass they will eat up 6GB on the GPU off the bat. Not too surprising that everything else can't fit in 2GB, since that's going to have to fit the entire model architecture and all intermediary values.

<comment><USER45> What exactly are these intermediate values composed of that makes them notable in size compared to the model itself? Are there resources I should read for how a model like this executes?

Will this model work with half precision weights? Is it very awkward to use "brain" 16 bit floats?

</comment></comment></comment></comment></comment><comment><USER53> I was able to get it working without downloading files by mounting Google Drive, although once the shared folder is rate-limited you need to make a copy in your own Google drive:

[https://colab.research.google.com/github/Akababa/AIDungeon/b...](https://colab.research.google.com/github/Akababa/AIDungeon/blob/patch-2/AIDungeon_2.ipynb)

</comment><comment><USER54> Shouldn't this be free egress as traffic is straight to a google product/ colab server or is this being downloaded in the browser?

Also cost was before cdn was enabled. This kind of traffic generally costs fractions of a cent per GiB after signing a contract with negotiation.

""" Egress to Google products (such as YouTube, Maps, Drive), whether from a VM in Google Cloud with an external IP address or an internal IP address No charge """

</comment><comment><USER55> Speaking as someone who runs a bunch of large websites - out of my own pocket, for profit.... I'm confused.

How did we get from 60K users to 10K per day expense?

For comparison, I serviced millions of users per month for years from a single virtual server... (granted, that was after making our site super lean for a data & CPU perspective)

How much resources is each user consuming?

<comment><USER56> I guess you don't instantiate a 5Gb image / user. It's the data transfer that costs 30/40c / user.

<comment><USER57> Cloudflare ? or just one machine at OVH ?

<comment><USER57> [https://wasabi.com/cloud-storage-pricing/](https://wasabi.com/cloud-storage- pricing/)

Wasabi's pricing model of $.0059 per GB/mo ($5.99 per TB/month) with no additional charges for egress or API requests means you don't pay to access your data

I don't know, could be a couple of dollars only ?

</comment></comment></comment></comment><comment><USER58> One would need to read the install.sh base file to know where to copy the model from the torrent into. and then manually install the other dependencies. If someone could make a friendlier version or at least better instructions more people would give it a try.

</comment><comment><USER59> Could someone explain the appeal in keeping this going long-term? When it was posted here, I played it and read several adventure logs here and on reddit. In every case the story is nonsense. Sure some parts read like something a human would write, but anytime you go beyond a few sentences you can see contractions and lack of flow that good human author would never make.

Don't get me wrong, it's a cool demo of how far we have gone beyond Markov chains. Am I missing something or just spoiled from those Infocom games I played as a kid?

<comment><USER31> Man, why so cynical? I don't see how those Infocom games spoiled you since they were limited to whatever a team of writers could come up with. And a well-written, dynamical-feeling CRPG is so rare that we still trumpet the handful that were any good from 20 years ago like Planescape: Torment.

Here's an example of how this game is fun: [https://twitter.com/ptychomancer/status/1203246078989987840](https://twitter.com/ptychomancer/status/1203246078989987840)

It's just fun to play with.

> you can see contractions and lack of flow that good human author would never > make.

That doesn't seem like a sensible goal post. Unless you think the technology is magic, why would you go into this thinking it's going to compete with a master-planned work of fiction by a human writer?

I, on the other hand, am inspired by the game. Imagine Crusader Kings 2 (free on Steam btw) where the events are randomly generated by this kind of story- telling technology. Right now it's kind of boring wondering which of the finite human-written events are going to show up. After playing for a while you go from wondering what crazy event will happen next to knowing all of the events and waiting for your favorite ones to show up.

We're a ways off from embedding a game in this technology, but I think we are within reach of embedding this technology inside a narrative-driven game.

Another example is Dungeons & Dragons. The fun is the sandbox and interacting with the narrative even though the human-driven dynamic storyline is often complete nonsense if you were to read a transcription of what actually happened.

This is the second comment I've read so far that seems to gleefully pat itself on the back for wondering how someone could enjoy something else.

<comment><USER59> I only dug into this because people were raving about it. Never played D&D, but it did remind me of those text adventure games I played as a kid.

When humans play D&D as the game progresses, are the rules of the world that people establish supposed to be internally consistent and is there a planned plot arc? Can you say there's no gravity and then later drop your sword?

</comment></comment></comment><comment><USER60> A 6GB neural net is going to be expensive to run anywhere. Is it possible to break the neural net into services?

</comment><comment><USER61> IIRC correct bandwidth costs have been essentially flat for the past decade or so. Not coming down.

</comment><comment><USER62> Out of curiosity, how much more time will it cost to run this on general CPUs of users computer?

<comment><USER63> It takes a couple seconds per reply. GPUs aren't as powerful as everyone would have it, at least not for ML; you'd be lucky to get a 10x speedup.

</comment></comment><comment><USER64> How does one play this without a Google account?

</comment><comment><USER17> Gcs egress is 0.12 per GB, so 3GB per user?

<comment><USER65> The model weighs about 6.2 GB, you can check it from install.sh [1] that currently downloads it via torrent (it was S3 days ago, I think).

[1] [https://github.com/AIDungeon/AIDungeon/blob/master/install.s...](https://github.com/AIDungeon/AIDungeon/blob/master/install.sh)

</comment></comment><comment><USER66> I am, personally, amazed by this. The following might be kinda NSFW, though.

> You are Jesus, a knight living in the kingdom of Larion. You have a steel > longsword and a wooden shield. You are on a quest to defeat the evil dragon > of Larion. You've heard he lives up at the north of the kingdom. You set on > the path to defeat him and walk into a dark forest. As you enter the forest > you see a huge black dragon flying above it's lair. The beast looks very > angry as you approach.

\+ suck his d-ck

> Your screams are silenced when you feel the head of the dragon's ---- > entering your mouth. He begins thrusting inside of you and soon you're > filled with his \---. You collapse onto the ground and die

Well. That went really more wild than I could have ever expected.

<comment><USER67> I don't get it. What's NSFW here? Suck his what?

Also, yeah I think it's funny how chatbots can generate undesirable or NSFW responses and there's basically no way to stop it. I know GPT-2 can parrot old-timey scientific racism back at me, just like the redditors who trained it!

<comment><USER66> NSFW as don't go loudly reading it in an office.

</comment></comment></comment><comment><USER68> I'm too scared Google will charge my credit card just by opening this lab page... I've got a gaming rig, why can't I just run this locally?

<comment><USER67> I have an unpriveledged user account named "wilson" designed specifically for situations like this.

</comment></comment><comment><USER12> After having read Rizwan Virk's "The Simulation Hypothesis", and playing this amazing new game, I can say that AI-generated text adventure games are an important step on the road to the Simulation Point (the point at which it would be technologically possible for us to construct a simulation that is all-encompassing as the one in The Matrix).

<comment><USER67> Have you tried psychadelics yet? Or maybe learned how to lucid dream? I think such technologies will be preferable to you, even if our electronics can catch up with our biological evolution.

Sure, maybe dream machines are fun and cool, but the ones in our future will be made for profit by companies like Facebook and Alphabet which severely diminishes any potential they may have had. Real dreams are libre, gratis, and uninterrupted by ads.

<comment><USER12> I've never tried any psychedelic. But I am able to lucid dream sometimes, I just never got deep into it. I guess I should read more about it.

I agree with your second statement, this so-called dream machine sounds a lot like a future iteration of Facebook's Oculus Rift.

</comment></comment></comment><comment><USER69> Well that was obviously forseeable and dumb then. Use the user's resources or charge money.

<comment><USER70> The unexpected thing was that Google colab and GCS are separate such that transfer costs between them often end up as international external egress fees

<comment><USER71> This is just GCS. For example, if you use their managed Kubernetes service, you will get a fresh load balancer for every service you expose to the internet. Not a shared load-balancer, a new one.

Unless you set up an alternative you'll get absolutely rinsed through the cost of the instance and then the egress charges on top.

<comment><USER72> All of the load balancers on GCP are shared. Maybe you meant to say you get a new, fresh IP address which is true but also not very expensive.

"Cloud Load Balancing is a fully distributed, software-defined, managed service for all your traffic. It is not an instance- or device-based solution, so you won't be locked into physical load balancing infrastructure or face the HA, scale, and management challenges inherent in instance-based LBs."

[https://cloud.google.com/load-balancing/](https://cloud.google.com/load- balancing/)

<comment><USER73> Nothing in the paragraph indicates it's shared. Also: it might be shared in the implementation but you will still be billed for every single https LB that you use (or an NLB if you're doing tcp load balancing).

Every unique kubernetes ingress resource WILL spin up a NEW, uniquely billed Https LB. Every unique kubernetes service with specific annotations will spin up a NEW, unique LB (internal or external). The author is correct.

<comment><USER72> You are not billed by the "load balancer", you are billed by the "forwarding rule" which makes it very obvious that the infrastructure is shared and that you will have additional costs with every K8S ingress.

</comment></comment></comment></comment></comment></comment><comment><USER74> I am trying to think some solutions but the crux is that users may be expecting to have a unique story tailored specifically to them. If that assumption is false, then we have some solutions:

\- save top or similar stories and make them pre-determined to avoid calling the ai services

\- decrease the amount of times the user can keep the story going: users can only give 3 times input instead of X

\- charge people for the game

Probably there are more ideas out there.

One last idea: package the code and instruct users to run on their own machine(s) or have then to run on their own GCS account.

<comment><USER75> i think the game should be made runnable on a user's own machine too.

Make the data licensed AGPL if the author is afraid of people copying and making profit off it without including them!

<comment><USER76> AGPL does not prevent someone from copying data and making a profit off it and not including them.</comment></comment></comment>