Dubai Billionaire's Tech Startup Takes on Amazon - <AUTHOR> https://www.wsj.com/articles/dubai-billionaires-tech-startup-takes-on-amazon-1514480803 <comment><USER2> Who would want to work for someone like this? If I can get my work done efficiently, why do I have to stay at the office?

This billionaire should read deep work, maybe he would attract more talent.

>"I want people who sleep in the office," says Mr. Alabbar, the founder of the now three-month-old startup. "Want to go walk your dog in the afternoon and all that? I'm not the guy you work for."

<comment><USER3> Sounds like a modern day of slavery.

<comment><USER4> They actually have real slavery in Dubai and I'm guessing because this guy is in real estate that he is probably involved in that.

<comment><USER5> Amusing, considering that his employees are likely to have also been sourced from India.

</comment></comment></comment></comment><comment><USER6> I'm not comfortable with a company like noon being called a startup. They're not really cash strapped (billion dollars went right in), granted building a logistics company isn't cheap but it's not like they went through phases of operation and testing the market/product, they just launched straight in like a regular business.

On another note, their glassdoor reviews are outright scary. Granted there are only 7 reviews but it sounds like typical dubai: throw money acquiring the best (in this case talent) then screw up the execution.

<comment><USER7> [https://www.glassdoor.co.uk/Reviews/noon- Reviews-E1669856.ht...](https://www.glassdoor.co.uk/Reviews/noon- Reviews-E1669856.htm) if anyone is interested.

<comment><USER8> This worse than what I expected

</comment></comment></comment><comment><USER9> [https://archive.is/sPiBx](https://archive.is/sPiBx)

<comment><USER10> You're a great man

<comment><USER11> Redirect through FB gets you past the paywall. Put the following in a bookmarklet and click it when on a WSJ page.

javascript:window.location="[https://www.facebook.com/l.php?u="+encodeURIComponent(window...](https://www.facebook.com/l.php?u="+encodeURIComponent\(window.location.href\);)

</comment></comment></comment><comment><USER12> What a motivating call for talent! I'm applying right away, I love being a slave to my employer.

</comment><comment><USER13> Thought exercise. You're on the Fortune 500. Your goal is to be on the top of that list. So you decide you want to spend $1B to build a single venture to create something with stereo typical VC return expectations: >100x.

IOW, you're looking at potential trillion dollar markets. Not too many of those! Taking on Amazon may be one of the most accessible, IMO. Start by dominating a region that Amazon has trouble in, like the Arab world.

But like any startup execution and team is more valuable than idea...

</comment><comment><USER14> I'm sceptical, to begin with, they need to overtake Alibaba and co. who are the real elephants in the room there.

My father ran a construction materials wholesale business in Russia's East for like 20 years. Naturally, he does a lot of imports from China and he had an Alibaba outlet. Despite fact that it was written black on white in bold font that the product is warehoused in Russia and is imported from China, he was receiving regular orders from companies in the gulf countries who were saying something like "we care not where it is warehoused, just get it there for n amount of money by that date". Companies in the gulf shop stuff on Alibaba as if they buy cookies in a corner store, and they don't care much about money (you can understand now why and how Alibaba charges companies up to $1m and more for front page placement and first positions in search)

For anybody who ran an Aliexpress store, it is know that the only people who can order DHL delivery for <$1 items are guys from Saudi Arabia or UAE

It will be super tough for any newcomer to break the habit even if the talk is about B2C market.

Moreover => halfpastnoon's post below.

</comment><comment><USER15> This guy sounds like he has more dollars than sense.

He's made many foolish investments as stated in the archive link another user mentioned.

But what can I judge. He's rich I'm not.

<comment><USER16> "This guy sounds like he has more dollars than sense."

Well, that's Dubai for you.

</comment></comment><comment><USER17> Aside from the ambitious goal itself, the Arabic version of the website is executed well [https://www.noon.com/ar-sa](https://www.noon.com/ar-sa) which is somewhat rare. I couldn't read the whole article due to the paywall, but I don't think Noon is trying to compete with amazon.com directly. I think they're trying to compete with Souq[1], which was recently acquired by Amazon in an effort to enter the Middle East market.

1) [https://techcrunch.com/2017/07/03/amazon-souq-com- completed/](https://techcrunch.com/2017/07/03/amazon-souq-com-completed/)

<comment><USER18> The site is very fast and it offers a smooth experience.

A key missing aspect are product reviews and the shipping time is not great (5 days for a book)

<comment><USER14> Of course, because the must have microscopic inventories, and they order stuff on short notice from wholesalers in China.

This is and old old trick

</comment></comment></comment><comment><USER19> ex-employee here. throwaway for obvious reasons.

This isn't a three month old startup. People literally were sleeping under their desks. Wretched company. Multiple people burned out and one suicide attempt. Hired some of the best and brightest from all around the the globe. Used them. Burned them. Banned people from working in the UAE ever again. Zero transparency between investors and management. Zero transparency between management and product. Zero transparency between product and engineering. Missed three launch deadlines successively with Alabbar committing to hard dates for launch in the press. Alabbar looked like an idiot one too many times in the press and fired everyone including the CEO. Threw away all intellectual property. Purchased competitor JadoPado with zero due diligence. Intent was to repaint JadoPado with noon paint. Founder of JadoPado joined as CTO, left seven days later. Threw away JadoPado code base and created everything from scratch. Minimal lovable product was ready to launch nine months ago but didn't sign up enough suppliers or SKUs to actually launch.

[https://www.reddit.com/user/halfpastnoon](https://www.reddit.com/user/halfpastnoon)

[https://www.reddit.com/r/dubai/comments/6813vw/nooncom_is_no...](https://www.reddit.com/r/dubai/comments/6813vw/nooncom_is_now_ceoless/)

[https://www.reddit.com/r/dubai/comments/6c23nb/noon_closing_...](https://www.reddit.com/r/dubai/comments/6c23nb/noon_closing_operations_in_cfc/)

[https://www.reddit.com/r/dubai/comments/6elfio/nooncom_reshi...](https://www.reddit.com/r/dubai/comments/6elfio/nooncom_reshift_again_with_no_more_omar_kassim/)

[https://www.reddit.com/r/dubai/comments/69vjgt/can_confirm_j...](https://www.reddit.com/r/dubai/comments/69vjgt/can_confirm_jadopado_is_being_reskinned_to_nooncom/)

[https://www.reddit.com/r/dubai/comments/6fsmkb/exnoon_ama/](https://www.reddit.com/r/dubai/comments/6fsmkb/exnoon_ama/)

[https://omarkassim.com/whats-hiding-behind-noons- beta/](https://omarkassim.com/whats-hiding-behind-noons-beta/)

[https://blog.esanjo.com/the-jadopado-experiment-is-at-an- end...](https://blog.esanjo.com/the-jadopado-experiment-is-at-an- end-1afd427de16e)

[https://www.arabianbusiness.com/alabbar-s-noon-com-set- launc...](https://www.arabianbusiness.com/alabbar-s-noon-com-set-launch- within-weeks--668197.html)

<comment><USER20> 3 weeks, but who's counting.

Also, the codebase wasn't thrown out. Some bits definitely were (we had a fair amount of legacy given 7 years), but large parts of the same team (e.g. SVP of Product, VP of Engineering) that built our product by and large designed and built the product that noon launched with.

<comment><AUTHOR> Hi Omar, how can we reach Mr. Alabbar. I briefly reviewed your site and believe we can help you with Conversion Rate Optimization. Here's what we do: [http://blog.capitalandgrowth.org/wp- content/conversion_wizar...](http://blog.capitalandgrowth.org/wp- content/conversion_wizards_info_v2.pdf)

What is your email?

</comment></comment></comment><comment><USER21> Non-paywall link:

[https://www.fullwsj.com/articles/dubai-billionaires-tech- sta...](https://www.fullwsj.com/articles/dubai-billionaires-tech-startup- takes-on-amazon-1514480803)

Change link from wsj.com to fullwsj.com to bypass. Uses Facebook redirect.

<comment><USER22> only works for me sometimes. facebook redirect is a good hack!

[https://outline.com/AEnNh6](https://outline.com/AEnNh6)

<comment><USER23> Totally off topic, but I am genuinely curious what copyright law says with regard to these archive-type services, especially ones that alter the content, in this case by removing "clutter". Awesome service IMO, I'd consider running one myself but don't want to be sued.

Edit: I just read their DMCA section. It appears that they are saying "don't submit content you don't own" but then on their home page describe it as a service to remove ads (and other nuisances) from articles. In other words, they are trying to hide behind the DMCA while encouraging people to submit content that isn't theirs. Again, I love the service but I fear for their legal standing should WSJ or others decide to pounce.

</comment></comment></comment><comment><USER24> Real-estate tycoon, Dubai, 3-months-old startup, take on Amazon.

I get that he is a billionaire, but IMO this is not newsworthy.

</comment><comment><USER25> Good luck. Taking on amazon is basically taking on an arm of the USA directly now.

</comment><comment><USER26> That's really helpful.</comment>


Do we need a third Apache project for columnar data representation? - <AUTHOR> http://dbmsmusings.blogspot.com/2017/10/apache-arrow-vs-parquet-and-orc-do-we.html <comment><USER2> One of the lead Arrow developers here ([https://github.com/wesm](https://github.com/wesm)). It's a little bit disappointing for me to see the Arrow project scrutinized through the one- dimensional lens of columnar storage for database systems -- i.e. considering Arrow to be an alternative technology (i.e. part of the same category of technologies) to Parquet and ORC.

The reality (at least from my perspective, which is more informed by the data science and non-analytic database world) is that Arrow is a new, category defining technology. So you could choose to use it as a main memory storage format as an alternative for Parquet/ORC if you wanted, but that would be one possible use case for the technology, not the primary one.

What's missing from the article is the role and relationship between runtime memory formats and storage formats, and the costs of serialization and data interchange, particularly between processes and analytics runtimes. There are also some small factual errors about Arrow in the article (for example, Arrow record batches are not limited to 64K in length).

I will try to write a lengthier blog post on wesmckinney.com when I can going deeper into these topics to help provide some color for onlookers.

<comment><USER3> In fact, the conclusion of the article is more positive than the headline makes it out to be: "Therefore, it makes sense to keep Arrow and Parquet/ORC as separate projects, while also continuing to maintain tight integration."

<comment><USER4> I think he buried the lede a bit with the title - his answer being "yes, we should have a separate format for this." The way he phrased his points was a bit odd and seemed inimical at times even though in the end it wasn't - e.g. he mentioned the X100 paper and his C-store compression paper which both talk about lightweight in-memory compression schemes which would suit Arrow's use case well, but then he goes back and says "Arrow probably won't support gzip" (which is much more heavyweight but offers better compression ratio and is more suitable for disk based storage formats) - OK, so that's fine and to be expected then? It turns out, yep, it's expected.

His main idea I took away was dispelling the notion that we should have instead been putting effort into slightly repurposing an existing format for our in-memory data layout.

It's definitely exciting to see the data science world and the databases world finally interacting a lot more - I think each has a lot to learn from the other. Battle tested techniques on one side, and entirely new use cases to deal with on the other.

\---

For the curious: Abadi is a well-known name in the databases community, especially with regards to column stores. A few papers that he's co-authored that I like:

"Column-Stores vs. Row-Stores: How Different Are They Really?" \- speaks about how a different style of query execution is a big part of what drives column store performance, and not just the memory layout itself: [http://db.csail.mit.edu/projects/cstore/abadi- sigmod08.pdf](http://db.csail.mit.edu/projects/cstore/abadi-sigmod08.pdf)

And then there's the magnum opus "The Design and Implementation of Modern Column-Oriented Database Systems" which is a huge survey into the subject: [https://stratos.seas.harvard.edu/publications/design-and- imp...](https://stratos.seas.harvard.edu/publications/design-and- implementation-modern-column-oriented-database-systems)

<comment><USER5> You nailed it. The most exciting part about all of this is being able to move between a "data science" context and a "database" context (and back again) without pain or penalty.

</comment></comment></comment></comment><comment><USER5> I'm also a developer on Arrow ([https://github.com/jacques-n/](https://github.com/jacques-n/)), similar to WesM. It is always rewarding (and also sometimes challenging) to hear how people understand or value something you're working on.

I think Dan's analysis is evaluating Arrow from one particular and fairly constrained perspective of "if using Arrow and Parquet for RDBMS purposes, should they exist separately". I'm glad that Dan comes to a supportive conclusion even with a pretty narrow set of criteria.

If you broaden the criteria to all the different reasons people are consuming/leveraging/contributing to Arrow, the case only becomes more clear for its existence and use. As someone who uses Arrow extensively in my own work and professionally ([https://github.com/dremio/dremio- oss](https://github.com/dremio/dremio-oss)), I find many benefits including two biggies: processing speed AND interoperability (now two different apps can share in-memory data without serialization/deserialization or duplicate memory footprint). And best of all, the community is composed of collaborators trying to solve similar problems, etc. When you combine all of these, Arrow is a no brainer as an independent community and is developing quickly because of that (80+ contributors, Many language bindings (6+) and more than 1300 github stars in just a short amount of time).

</comment><comment><USER6> The Apache Software Foundation has no problem with hosting competing projects. _There is no top-level technical direction by the Board or any other entity._ There's no "steering committee" or the like where companies pay to play and push the organization to pump up their preferred project.

This is one of the fundamental reasons that the ASF is successful. Any time you see the ASF criticized for hosting competing projects without addressing this point, feel free to dismiss the critique as facile and uninformed.

<comment><USER2> Apache Arrow is not competing with Apache Parquet or Apache ORC.

<comment><USER7> and _even if it were_, the ASF doesn't mind. contributors are free to dedicate their time to whatever they choose.

</comment></comment></comment><comment><USER8> This point is interesting:

 _So indeed, it does not matter whether the data is stored on disk or in memory --- column-stores are a win for these types of workloads. However, the reason is totally different._

He says that for tables on disk, column layout is a win due to less data transferred from disk. But for tables in memory, the win is due to the fact that you can vectorize operations on adjacent vaules.

I also think the convergence of these two approaches is interesting: one is from a database POV; the other is from a programming language POV.

Column Databases came out of RDBMS research. They want to speed up SQL-like queries.

Apache Arrow / Pandas came at it from the point of view of R, which descended from S [1]. It's basically an algebra for dealing with scientific data in an interactive programming language. In this view, rows are observations and columns are variables.

It does seem to make sense for there to be two separate projects now, but I wonder if eventually they will converge. Probably one big difference is what write operations are supported, if any. R and I think Pandas do let you mutate a value in the middle of a column.

\-----

On another note, I have been looking at how to implement data frames. I believe the idea came from S, which dates back to 1976, but I only know of a few open source implementations:

\- R itself -- the code is known to be pretty old and slow, although rich with functionality.

\- Hadley Wickham's dplyr (the tbl type is an enhanced than data.frame). C++.

\- The data.table package in R. In C.

\- Pandas. In a mix of Python, Cython, and C.

\- Apache Arrow. In C++.

\- Julia DataFrames.jl.

If anyone knows of others, I'm interested.

[1] [https://en.wikipedia.org/wiki/S_(programming_language)](https://en.wikipedia.org/wiki/S_\(programming_language\))

<comment><USER9> For data frame implementations, you can also look at Spark's Dataset/DataFrame and Graphlab Create's SFrame.

The company/team behind Graphlab Create was bought by Apple and the open sourced components haven't been updated since then. Because of that, I wouldn't use it in production, but if you are just looking for functioning implementations to compare, that gives you one more.

<comment><USER8> Thanks, I never heard of Graphlab Create. It is a substantial piece of code! It says it's "out of core", which means it's probably more similar to Parquet/ORC than Arrow. But still interesting.

[https://github.com/turi- code/SFrame/tree/master/oss_src/sfra...](https://github.com/turi- code/SFrame/tree/master/oss_src/sframe)

For comparison, dplyr and arrow:

[https://github.com/tidyverse/dplyr/tree/master/src](https://github.com/tidyverse/dplyr/tree/master/src)

[https://github.com/apache/arrow/tree/master/cpp/src/arrow](https://github.com/apache/arrow/tree/master/cpp/src/arrow)

C++ does seem to be useful for stuff like this.

</comment></comment></comment><comment><USER10> A pretty glaring flaw or omission in the analysis is using a table that's just 6 columns wide. Tables used in data analytics workloads are much wider, 50-100 columns or more is common. That number of columns means scanning significantly more data for the row-oriented storage.

<comment><USER11> Depending on the details, I'd wonder if you'd just end up pulling in a single cache line from the row, regardless of how wide the row is. Even in his benchmark, you're wasting a good 5/6ths of the memory bandwidth you're using (pulling a 24 byte row and only examining 4 bytes of it).

</comment></comment><comment><USER12> I would argue that the fact that Arrow has been integrated into so many projects over the last year is proof that a separate project made sense. Dremio, PySpark, Pandas, MapD, NVIDIA GPU Data Frame, Graphistry, Turbodbc, ...

<comment><USER13> Here's the conclusion of the article:

 _It makes sense to keep Arrow and Parquet /ORC as separate projects, while also continuing to maintain tight integration._

You might enjoy reading it to see why.

</comment></comment><comment> d--b Before people get all negative about the article, be sure to read it, because the author does answer that yes, it does make sense.

<comment><USER14> > Therefore, it makes sense to keep Arrow and Parquet/ORC as separate > projects, while also continuing to maintain tight integration.

I wonder how many people will be lost before getting to the last line.

</comment></comment><comment><USER15> FWIW it should be possible to vectorize the search across the row store. With 24 byte tuples (assuming no inter-row padding) you can fit 2.6 rows into a 64 byte cache line (a 512 bit simd register). Then it's just a matter of proper bit masking and comparison. Easier said than done, I figure because that remainder is going to be a pain. Another approach is to use gather instructions to load the first column of each row and effectively pack the first column into a register as if it were loaded from a row store and then do a vectorized compare as in the column-store case.

All of that to underscore it's not that one format vectorizes and the other doesn't. The key takeaway here is that with the column store, the compiler can _automatically_ vectorize. This is especially a bonus for JVM based languages because afaik there is no decent way to hand-roll SIMD code without crossing a JNI boundary.

<comment><USER16> This isn't that hard. A sane person would do 3 cache lines at once as 192 bytes = 8 24 byte tuples. You would do 3 AVX512 loads, a masked compare at the proper places (actually, I think the masked compare instructions can take a memory operand, which might get you uop fusion so the load+compare is one instruction) yielding 3 masks each with 16 bits (of course, most of the bits would be junk). The 16 bit masks can be shift+or'ed together (whether as "k-regs" or in GPRs) and the correct bits can be extracted with PEXT.

The downside of this is that you are still reading 6 times as much data. A straightforward implementation of this should not be CPU bound IMO. If a Skylake Server can't keep up with memory doing 32-bit compares I'll eat my hat.

Gather is _not_ a good idea for this purpose. Gather is very expensive. It's really mainly good for eliminating pointer chasing and keeping a SIMD algorithm in the SIMD domain.

</comment></comment><comment><USER17> I've built a columnar in-memory data transformation engine [1] and wrote [2] about the need in common columnar data format to avoid re-compression. The problem I have with the Apache projects is that they all require strongly typed columns (if I'm not missing something). In our case, the app is actively used for processing spreadsheets and sometimes XML files which requires supporting mixed data types (e.g. timestamps and strings) in a column.

Another issue is storing 128-bit decimals. They are more preferable for financial calculations, but are not supported in the Apache projects.

So may be we need a forth standard for columnar data representation. Or expand the existing ones to make them more versatile.

[1] [http://easymorph.com/in-memory-engine.html](http://easymorph.com/in- memory-engine.html)

[2] [http://bi-review.blogspot.ca/2015/06/the-world-needs-open- so...](http://bi-review.blogspot.ca/2015/06/the-world-needs-open-source- columnar.html)

<comment><USER5> Arrow supports a union type for heterogeneous columns (we use it for random json in Dremio) and a 128-bit decimal.

</comment></comment><comment><USER16> It's an interesting article but surely a modern instance should be able to manage more than 4 u32 compares per cycle? Any machine with AVX2 (or beyond) should either be able to do 2x256 AVX2 VPCMPEQD instructions (or AVX 512 can do 1x512). The code to marshal up the results of this compare and do something useful with would push out to another cycle or two but IMO we can surely manage to compare an entire cache line (64 bytes) in 1 cycle worth of work.

This doesn't invalidate his point, and there are even more interesting things that can be done with packed data formats - and of course if you're searching a 6-bit column for some data item (or set of data items) you might be even happier to be using a columnar form (especially if the row contains lots of stuff: the 6:1 ratio in the article might be understating the advantage of working at the column level).

</comment><comment><USER18> I don't have any strong feelings about this either way, but the main question in my head after reading this is:

\- Are the differences SIGNIFICANT enough to support two (three?) different codebases? A lot of your points seem to be more about building in configurations/plugins versus a completely different (core) storage format. For instance, adding in separate plugins for different dictionary compressions, or being able to specify the block read size or schemes. I would just think you may be spending a lot of time reinventing 80% of the wheel and 20% on 'memory specific' functionality.

(I'm naively oversimplifying, but its something to think about).

</comment><comment><USER19> If the CPU test is simply reading 1/6 of the data, then it should be memory that is the bottleneck and not the CPU (even unoptimized). Something smells very wrong about his adhoc test. The next piece of data is in the cache-line seven times out of eight. And its reading 1/6 of the data into memory. Should be way faster even without -O3. And if there's something fundamentally broken about the code without -O3, then _why post that benchmark at all_? Seems dishonest.

Be good to post the code when making such claims.

<comment><USER20> I thought it made the point that if you aren't vectorizing your code then it hardly matters if you use an optimized data format.

SSE makes a huge difference. With AVX and multiple threads you can actually exceed the memory bandwidth of a CPU socket, which looks funny in performance graphs as adding threads to cores suddenly stops scaling.

</comment></comment><comment><USER21> Columnar data representation can be optimized and designed for a wild range of query and data types that have non-aligned constraints. So yes, you can have 3, or 12 for that matter. I think we'll see more & more purpose-specific database technologies as gains from Moore's laws and its analogues start slowing down, and some previously niche application areas become high-$ areas with larger user bases.

</comment><comment><USER22> Not sure if this is a good place to ask, but how do Apache Arrow and Parquet compare to Apache Kudu ([https://kudu.apache.org/](https://kudu.apache.org/))? Seems like all three are columnar data solutions, but it's not clear when you'd use one over the other.

Kind of surprised the article didn't mention Kudu for that matter.

<comment><USER23> This covers the distinction a bit better. [https://www.slideshare.net/HadoopSummit/the-columnar-era- lev...](https://www.slideshare.net/HadoopSummit/the-columnar-era-leveraging- parquet-arrow-and-kudu-for-highperformance-analytics)

<comment><USER22> Right on, this is perfect. Thanks!

<comment><USER5> One quick note to make on this. Kudu is a storage implementation, (similar to Parquet in some ways). Arrow isn't about persistence and is actually built to be complementary to both Kudu and Parquet.

Also note: Kudu is a distributed process. Arrow and Parquet are libraries that can be embedded into your existing applications.

</comment></comment></comment></comment><comment><USER24> > However, a modern CPU processor runs at approximately 3 GHz --- in other > words they can process around 3 billion instructions a second. So even if > the processor is doing a 4-byte integer comparison every single cycle, it is > processing no more than 12GB a second

This is so laughably simplified.

</comment><comment><USER25> Does arrow itself supports and optimizes for multicore processing? Or is that a responsibility of a higher layer like dremio? If so, do such layers optimize Arrow queries execution to utilize multiple cores as much as possible?

</comment><comment><USER26> I wonder if one could have an FPGA attached to the CPU, load a piece of code into it for pipelined decompressing and processing a piece of compressed column store, and then vroom! It would process data really fast.

<comment><USER27> Like Netezza?

<comment><USER26> Yeah, but I want to have it in my home computer. Any application could use this thing.

</comment></comment></comment><comment><USER25> How can I query Apache Arrow without entire Hadoop stack? It seems it could be a great in-memory OLAP engine, if only there was an efficient way to slice and dice it?

<comment><USER5> We built Dremio (github.com/dremio/dremio-oss, apache licensed) entirely on top of Apache Arrow specifically for the purposes of creating a high speed analytical capabilities including MOLAP like work as well as other forms of caching/acceleration for analytical workloads. Other products/projects are also starting to adopt a similar technical architecture.

<comment><USER25> Dremio looks very interesting indeed. What would you recommend for interacting with Arrow with more control, as a library? I'm interested in creating new Arrow-based data sources, not using it as an intermediary to other data sources.

On a side note - what other products/projects did you mean?

<comment><USER5> The Arrow project itself is a set of libraries. One of the things we'll do is try to add more algorithms over time to it so if you want say, a fast arrow sort or arrow predicate application. Full SQL is always far more complex and I can't see the project itself .

The engine inside of Dremio is something we call Sabot (a shoe for modern arrows, see sabot round on wikipedia). We hope to make it modular enough one day to use a library but it isn't there yet.

In regards to your other question re projects/products: Arrow contributors are actively trying to get more adoption of Arrow as an interchange format for several systems. We've had discussions around Kudu (no serious work done yet afaik). Parquet-to-Arrow for multiple languages is now available. Arrow committers include committers from several other projects such as HBase, Cassandra, Phoenix, etc. The goal is ultimately to figure integrations with all.

In most cases, these data storage systems are saddled with slow interfaces for data access. (Think row-by-row, cell-by-cell interfaces.) Arrow, among other things, allows them to communicate through a much faster mechanism (shared memory--or at least shared representation if not node local).

<comment><USER25> How does dremio differ from PrestoDB? As far as I know, PrestoDB can also virtualize access to many data sources and join data between them. We didn't go deep with PrestoDB because our basic tests for multi-source joins ran very slowly, and it seemed to pull all data from both joined tables into one place. I'm not a Prestodb expert, so maybe there's a better way to do it (all suggestions welcome).

What's the differentiator? Is dremio smarter somehow and avoids copying all data to perform a simple join? Or does it copy the data the same way but Arrow lets it be faster than Presto? What's on your roadmap?

<comment><USER5> PrestoDB is similar to Impala, Hive and other SQL Engines. Each is designed to do distributed SQL processing. Dremio does embed an OSS distributed SQL processing engine (Sabot, built natively on Arrow) as well but we see that as only a means to an end. Our focus is much more on being a bi & data fabric/service.

At the core of this vision are: very advanced pushdowns (far beyond other OSS systems), a powerful self-service UI for managing, curating and sharing data (designed for analysts, not just engineers) and--most importantly--the first open source implementation of distributed relational caching for all types of data. You can see more details about this last part in a deck I presented at DataEngConf early today: [https://www.slideshare.net/dremio/using-apache- arrow-calcite...](https://www.slideshare.net/dremio/using-apache-arrow- calcite-and-parquet-to-build-a-relational-cache-81440786)

</comment></comment></comment></comment></comment></comment><comment><USER28> Well, there is another one here: [https://carbondata.apache.org/](https://carbondata.apache.org/)

</comment><comment><USER29> Let me give benefit of the doubt instead of simply doubting -

Can somebody provide a justification why performance-centric implementation- details justify an entire new project? Couldn't this be done as simply a storage engine? For that matter, couldn't all columnar datastores merely be storage engines?

<comment><USER5> It's more than an implementation detail because we're also targeting interoperability between multiple separate technologies. One of the key things that the article didn't fully cover is that Arrow serves two purposes: high performance processing and interoperability.

A key part of the vision is: two systems can share a representation of data to avoid serialization/deserialization overhead (and potentially copying in a shared memory environment). This is only possible if the in-memory format is also highly efficient for processing. This allows the processing systems (say Pandas and Dremio) to share a representation, both process against it, and then move the data between each other with zero overhead.

If you shared the data representation on the wire and then each application had to transform it to a better structure for processing, you're still paying for a form of ser/deser. By using Arrow for both processing and interoperation you benefit from near-no-cost movement between systems and also a highly efficient representation to process data (including some tools to get you started in the form of the Arrow libraries).

</comment></comment><comment><USER30> I think the author comes across as tone deaf for two reasons:

1\. Burying the lede, by tacking the affirmative conclusion on as the final sentence,

2\. This quote reads as pompous, not humble:

> I assume that the Arrow developers will eventually read my 2006 paper on > compression in column-stores and expand their compression options to include > other schemes which can be operated on directly (such as run-length-encoding > and bit-vector compression). I also expect that they will read the X100 > compression paper which includes schemes which can be decompressed using > vectorized processing.</comment>


Tales of Cybersecurity - <AUTHOR> https://medium.com/swlh/tales-of-cybersecurity-fafea221213 <comment><USER2> I'm sorry to say but the article isn't well written, the structure is incredibly off: the intro says it will be about one thing, then it just goes off in a different direction. Also, the conclusion is somewhat similar to - never use the Internet</comment>


DeepMind and Blizzard Open StarCraft II as an AI Research Environment - <AUTHOR> https://deepmind.com/blog/deepmind-and-blizzard-open-starcraft-ii-ai-research-environment/ <comment><USER3> A lot of people here seem to be underestimating the difficulty of this problem. There are several incorrect comments saying that in SC1 AIs have already been able to beat professionals - right now they are nowhere near that level.

Go is a discrete game where the game state is 100% known at all times. Starcraft is a continuous game and the game state is not 100% known at any given time.

This alone makes it a much harder problem than go. Not to mention that the game itself is more complex, in the sense that go, despite being a very hard game for humans to master, is composed of a few very simple and well defined rules. Starcraft is much more open-ended, has many more rules, and as a result its much harder to build a representation of game state that is conducive to effective deep learning.

I do think that eventually we will get an AI that can beat humans, but it will be a non-trivial problem to solve, and it may take some time to get there. I think a big component is not really machine learning but more related to how to represent state at any given time, which will necessarily involve a lot of human-tweaking of distilling down what really are the important things that influence winning.

<comment><USER4> I wonder if we will see any advanced cheese strats come out of this. I'm assuming some implementations will eventually develop micro control that is far beyond any human player's capabilities, which would make things like all- in probe rushing much more viable. Instead of playing the normal meta in a computer-vs-human, I imagine an advanced AI would simply send all of its workers off the mineral line as soon as the game starts, and attempt to out micro the human opponent before they can build an army-producing building.

<comment><USER5> I know this isn't the exact same as the article, but when genetic algorithms were introduced to solve for build orders, the "seven roach rush" was in vogue, something that was unexpected at the time and "discovered" using GA.

I think there is a space for finding strategies that have more leeway in execution and thus are more suitable for humans to pilot rather than have machine level micro.

<comment><USER6> I love the story of the Seven Roach Rush. To quote the linked article, "The most interesting part of this build, however, is how counter-intuitive it is. It violates several well-known (and well-adhered-to) heuristics used by Starcraft players when creating builds."

I'm fairly certain that this application of machine learning will present some surprising strategies.

[http://lbrandy.com/blog/2010/11/using-genetic-algorithms- to-...](http://lbrandy.com/blog/2010/11/using-genetic-algorithms-to-find- starcraft-2-build-orders/)

</comment></comment></comment></comment><comment><USER7> _Related_ : Today I learned that a group of AI researchers has released a paper called: _STARDATA: A StarCraft AI Research Dataset_. According to one of the authors: "We're releasing a dataset of 65k StarCraft: Brood War games, 1.5b frames, 500m actions, 400GB of data. Check it out!"

> Article: > [https://arxiv.org/abs/1708.02139](https://arxiv.org/abs/1708.02139)

> Github: > [https://github.com/TorchCraft/StarData](https://github.com/TorchCraft/StarData)

<comment><USER8> The great thing about this is that it includes the game state throughout the game. It's been pretty easy to find lots of Starcraft replays, but the replays only include enough information to recreate the game (basically just the player actions). If you wanted to know what was happening in the game at the time the player made an action, you had to load up Starcraft and simulate out the game until that point. This dataset has already run the game for you and provided the data!

<comment><USER9> Is it that much computation to simulate an entire game? You obviously don't need to render the graphics or anything, it should just be a list of events that occur, which doesn't seem all that slow to process.

<comment><USER10> Until today's release of the headless Linux client, you still had to run the full StarCraft program, which gets expensive fast. And it massively complicates the workflow to have to play through every game serially to recreate the state rather than simply reading random rows of data from a 300GB dataframe on disk.

<comment><USER9> Oh I see, thanks, I didn't know. But man, 300 GB per game sounds completely nuts!

<comment><USER11> I believe the 400GB is the total amount for the 65000 different game replays

<comment><USER9> Oh but how does that work? That's ~6 MB per game which sounds like just a list of actions rather than precomputed data per frame. Is it compressed somehow?

<comment><USER12> "The full dataset after compression is 365 GB, 1535 million frames, and 496 million player actions." \- Yes

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER13> The API Blizzard is exposing is really nice. Sadly most of the advantages AI had in SC1 were just due to the fact that an automated process could micro- manage the tasks the game didn't automate for you (a lot of boring, repetitive work). SC2 got rid of a lot of that while still allowing room for innovative and overpowered tactics to be discovered (MarineKing's insane marine micro, SlayerS killing everyone with blue flame hellions, some more recent stuff I'm sure from the newest expansions). Hopefully the API lets AIs converge on optimal resource management and get to exploring new and innovative timings, transitions, army makeups, etc.

<comment><USER14> > SC2 got rid of a lot of that

You think so? My impression is that SC2 had a lot more of repetitive tasks you had to do. E.g. wall off the ramp, send a worker scouting, ... and you have to perform certain actions every X seconds (like using chrono boost). A lot or mastering the game is rote learning, and polishing a build order. Another big part is constantly scouting and reacting to what the enemy is doing.

Due to those reasons I found SC2 a bit tedious (it was still fun, just felt more like work than SC1). Granted, this is maybe because I played SC1 more on LAN, and there wasn't all the metagame going on. But I think SC2 really does focus on "grinding" and rote learning to get better, probably this was chosen to make it more "eSports"-like.

If I would get to design a SC2.5 or SC3, I would remove all the rote - <USER2> actions you always have to perform - and I would give the player the opportunity to trade off more between macro and micro.

Actually, it would be cool if you could "research" certain AI features in game for a cost. For example, have one upgrade that micros your marines like a pro, or positions your units in sensible formations. Another player could counter this with a "radio jam" ability, that would make your units in an area take bad formations, or be controlled by a very simple AI. And if you are good at micro, you could save the update, or invest it in an update that makes macro simpler. And so on, I think there are a lot of things one could explore there. Maybe opening SC2 to AI exploration can lead to such gameplay innovations.

<comment><USER15> I really like the idea of a helper AI that takes high level commands and handles the manual dexterity aspect of the game. That would make RTS games a lot more attractive for someone like me who prefers simple, thoughtful games rather than complex games with a steep learning curve.

<comment><USER16> There are actually 2 parts to the complexity of a modern RTS game like starcraft:

1\. Memorizing certain well known strategies and counters very well and recalling them immediately.

2\. Having decent speed with the mouse/keyboard to actually execute those strategies within a very short period of time.

I think what you're talking about it automating 2)... which I completely agree with. How to do it... that is more complex though...

</comment></comment></comment></comment><comment><USER17> This seems all in good fun but I wonder if it's come too late.

Starcraft 2 is at its twilight.

The biggest leagues of South Korea have disbanded. [1] The prolific progamers who transitioned to Starcraft 2 have gone back to Broodwar. [2]

Blizzard itself has scrubbed all references to Starcraft 2 on the very home page of Starcraft. [3] Except for the twitter embed, it has only only one "2" character... in the copyright statement.

My take is that the future for the Starcraft franchise will be through remastered and potential expansion packs following it.

Starcraft 2 had a good run but, with the entire RTS genre stagnating [4], I don't think Blizzard wants to bet on anything less than the top horse.

[1] [https://www.kotaku.com.au/2016/10/the-end-of-an-era-for- star...](https://www.kotaku.com.au/2016/10/the-end-of-an-era-for-starcraft- and-south-korea/)

[2] [http://www.espn.com/esports/story/_/id/18935988/starcraft- br...](http://www.espn.com/esports/story/_/id/18935988/starcraft-brood-war- glory-days-jaedong-best-bisu-talk-starcraft)

[3] [http://starcraft.com](http://starcraft.com)

[4][http://www.pcgamer.com/the-decline-evolution-and-future- of-t...](http://www.pcgamer.com/the-decline-evolution-and-future-of-the-rts/) (Aside from MOBAs)

<comment><USER18> I don't quite agree, FWIW.

SC2 does seem to be at its twilight _in Korea_ , and I agree progamers and fans there are super interested in Remastered.

But I don't think Remastered will be very popular outside KR. The SC2 "war chest" promo appears to have made more money than expected, as measured by hitting its funding ceiling within a few days.

So I don't think it's "Remastered replaces SC2", I think it's a divergence into KR playing Remastered and non-KR playing SC2, and the number of progamers and players doesn't have to be zero-sum: it could enlarge the population playing either game, too.

<comment><USER17> I agree that Starcraft 2 won't suddenly drop dead. People do play it and FWIW, I liked it! I played all the expansions, online, and even the arcade mode. It was a good game.

But I disagree that Blizzard has faith in Starcraft 2 for America or any other country.

The removal of Starcraft 2 from Starcraft's English-Speaking homepage is one sign of finality. In-universe, Blizzard has also ended the main dramatic arc of Starcraft 2's story, leaving room only for half-hearted spin-offs.

Numbers-wise, we're seeing 50% drop-offs in user activity the last 2 years alone. Even with the release of "Legacy of the Void", the number of daily games played for 1v1 since 2015 have gone from 321,000 to 138,000. The new, much-advertised, much-worked-upon, Archon Mode has gone from 11,000 games a day to a measly 1,000 [1]. Not just because of Korean disinterest, we're seeing players leave across the board in all countries.

In 6 years, Starcraft 2 went from millions of players concurrently to an average of 20k a day.

Compare with the lifespans of League-of-Legends, Dota, Counterstrike, even the original Broodwar, and the reason for remastered becomes more obvious.

Blizzard knows Starcraft 2 won't lead to the resurgence of the RTS genre, so they're trying another route.

[1] [http://www.rankedftw.com/stats/population/1v1/#v=2&r=-2&sy=c...](http://www.rankedftw.com/stats/population/1v1/#v=2&r=-2&sy=c&sx=a)

</comment></comment></comment><comment><USER19> It's a bit too bad they're having to move towards supervised learning and imitation learning.

I totally understand why they need to do that given the insane decision trees, but I was really hoping to see what the AI would learn to do without any human example, simply because it would be inhuman and interesting.

I'm really interested in particular if an unsupervised AI would use very strange building placements and permanently moving ungrouped units.

One thing that struck me in the video was the really actively weird mining techniques in one clip and then another clip where it blocked its mineral line with 3 raised depots...

<comment><USER20> They can always finetune using RL later. Superversied training was the first step at making AlphaGo work.

</comment></comment><comment><USER21> I also want to see the algorithm win on unorthodox maps. Perhaps a map they have never seen before, or one where the map is the same as before but the resources have moved.

Don't tell the player or the algorithm this, and see how both react, and adapt. This tells us a great deal about the resiliency of abilities.

<comment><USER22> I am considering a random map generator for just this reason.

</comment></comment><comment><USER23> When Watson won at Jeopardy, one of its prime advantages was the faster reaction time at pushing the buzzer. The fairness of that has already been hashed out elsewhere, but.....

We already know that computers can have superior micro and beat humans at Starcraft through that(1). Is DeepMind going to win by giving themselves a micro advantage that is beyond what reasonable humans can do?

(1)[https://www.youtube.com/watch?v=IKVFZ28ybQs](https://www.youtube.com/watch?v=IKVFZ28ybQs) as one example

<comment><USER24> My understanding is that in a full match, AIs still have no hope against humans, since even though they can crush humans at micro, their macro is still abysmal [1]. I'm not aware of a match where any AI has beat a pro human player at Starcraft -- I'd be interested in learning otherwise!

[1] [http://spectrum.ieee.org/automaton/robotics/artificial- intel...](http://spectrum.ieee.org/automaton/robotics/artificial- intelligence/custom-ai-programs-take-on-top-ranked-humans-in-starcraft)

<comment><USER25> would you love to be proven wrong?

<comment><USER26> Of course.

</comment></comment></comment></comment><comment><USER27> Blizzard should put in an AI-assisted play mode where players are limited to X lines of code that can be launched with keyboard commands.

<comment><USER28> I know that, as a player, the high mechanical limitations of Starcraft are part of why it's such a difficult, high-skill-ceiling game. But.. I've tried to enjoy watching SC2 on Twitch, and while it's _kinda_ fun, it's just so disappointing when a complicated strategic game is thrown away because a player doesn't react fast enough to workers being sniped or a drop being shot down.

I wish the individual units had some automatic behavior -- for example, marines would could run in spread out formations near tanks or banelings; workers would flee from hazards; flying units would avoid turrets unless specifically directed to fly over them. It would require a lot of rebalancing, of course, but it would make the game so much more tactical and strategic and (imo) enjoyable to watch.

<comment><USER27> Yeah I can even imagine a thriving "marketplace" for specialty code that top players would keep secret.

And it doesn't have to just be for micro. For people who are bad at macro, maybe code can be written to consistently maintain X workers at all bases.

The difficult part here would be how to balance the AI-assistance. Is lines of code (or number of characters) a good proxy for complexity? What's the number- of-character to benefit ratio?

I guess that's ultimately determined by the individual player's strengths and weaknesses. If a player sucks at macro, then the macro script is worth the number of characters.

</comment></comment></comment><comment><USER29> Are there any known arbitrary code injection for starcraft? Like how you can use a regular controller to reprogram super mario world to play pong?

[https://www.reddit.com/r/programming/comments/1v5mqg/using_b...](https://www.reddit.com/r/programming/comments/1v5mqg/using_bugs_in_super_mario_world_to_inject_new/)

[https://bulbapedia.bulbagarden.net/wiki/Arbitrary_code_execu...](https://bulbapedia.bulbagarden.net/wiki/Arbitrary_code_execution)

Is this how we are going to accidentally let AGI loose into the world!? /s

On a more realistic note I think this will degenerate into a game of who can fuzz test for the best game breaking glitch. Think of all the programming bugs that turned into game mechanics in BW that we haven't discovered for SC2 yet: [http://www.codeofhonor.com/blog/the-starcraft-path- finding-h...](http://www.codeofhonor.com/blog/the-starcraft-path-finding-hack)

</comment><comment><USER30> The StarCraft 1 BroodWar AI scene has been thriving for a few years now: [https://sscaitournament.com/](https://sscaitournament.com/) You can watch 24/7 live AI vs AI games on Twitch at: [https://www.twitch.tv/sscait](https://www.twitch.tv/sscait) Support for voting on who to play next and even a betting system are in place, too. For those who wish to get their feet wet with BW AI development, here are the Java / C++ tutorials: [https://sscaitournament.com/index.php?action=tutorial](https://sscaitournament.com/index.php?action=tutorial)

<comment><USER30> Some thoughts and analysis on why Starcraft AI by one of the active AI developers Dan: [https://dangant.com/2017/08/09/why-starcraft- ai/](https://dangant.com/2017/08/09/why-starcraft-ai/)

</comment></comment><comment><USER31> The SCAI bots I've seen are more hardcoded tactics engines rather than machine learning models. They're still impressive, but their logic isn't quite 'learned' it's hand coded which is a crucial difference.

</comment><comment><USER32> That's surprising. I thought Bliz didn't want anyone near sc2 but approved of sc1 being used for this purpose.

<comment><USER33> SC1 really doesn't make sense for this, 80% of the skill is just keeping on top of the mindless but mechanically intensive stuff, which is trivial beyond trivial for an AI.

SC2's automated away most of this (pretty much everything but production cycles), which makes it a better measure for AI vs human.

<comment><USER34> > SC1 really doesn't make sense for this, 80% of the skill is just keeping on > top of the mindless but mechanically intensive stuff, which is trivial > beyond trivial for an AI.

If that were true, then AIs would be dominant in BW instead of still bad at the game.

</comment></comment></comment><comment><USER35> I thought this was already happening. Right after AlphaGo beat Lee, I remember hearing about it. Did they give up on having their AI playing SC2? I wondered if that would work, since it seemed to take turns in Go at the same speed as a normal player, I wondered if it was trying to compute the most likely winning move each turn and the late game implications of those moves. If it tried that in a fast paced game how it would deal with the speed. It obviously would need to develop a pattern of pre-baked strategies that would win it the game. Would it play the same build every round or would it realize that changing things up each match wins it more games?

</comment><comment><USER36> It's a bit too bad they're having to move towards supervised learning and imitation learning.

I totally understand why they need to do that given the insane decision trees, but I was really hoping to see what the AI would learn to do without any human example, simply because it would be inhuman and interesting.

I'm really interested in particular if an unsupervised AI would use very strange building placements and permanently moving ungrouped units.

One thing that struck me in the video was the really actively weird mining techniques in one clip and then another clip where it blocked its mineral line with 3 raised depots...

</comment><comment><USER37> There's something funny about a company that is actively developing bleeding edge AI technology, but who can't design a webpage that works on mobile without crashing.

<comment><USER38> Just goes to show how complicated web tech is, even ai researchers can't get it right!

</comment></comment><comment><USER39> When I used to play a lot of StarCraft, and then later with Total Annihilation, I wished for the ability to customize the AI.

So then BWAPI came along ... and ... AI is hard. The best SCBW bots are still pretty pathetic compared to a human player, never mind an expert human player.

</comment><comment><USER40> I'd be really interested in how differently tiered data sets (ladder rank) would work as sources for teaching.

Is it possible that training on diamond players is less effective than training on, say, silver? Is that actually even an interesting thing to look at?

</comment><comment><USER41> Any predictions for how long it will take for an AI to win against the world's best player?

<comment><USER42> Awhile. This just isn't like Go or Chess. The gap from perfect information to imperfect information is quite a chasm, and from turn-based to real-time is even more vast.

I play _Age of Empires 2_ semi-competitively, and I just can't imagine the research progress that would have to be made for a pro to lose to an APM- limited AI agent. So much of the game comes down to intuiting what your opponent is planning without being able to see what they're doing, and more importantly _intuiting what your opponent isn 't ready for._

The biggest difference, though, is the "RT" in "RTS"\-- real time. This isn't turn-based anymore, where at a given moment you have a single choice to make, a single piece to move as in Chess and Go, and can then wait for the singular and visible reaction your opponent makes before making your next choice.

My understanding it that the moves a program like AlphaGo makes are not interconnected-- it picks each move individually as an ideal move for that board state. It could take over halfway through the game for someone else and would make the same move that it would have made at that point if it had been in control the whole time and arrived at that board state on its own.

But that doesn't work in a real-time game, since you and your opponent are now moving simultaneously and the "board" is never static. Your moves must be cohesive and planned and flow continuously without time to ponder, each connected to the last. There is no "one" move for a given state.

Another facet of real-time play is the idea of _distraction_. It's very important in RTS's to keep your opponent distracted, to disrupt their plans and their focus, by coming from unexpected directions at unexpected times, sometimes concurrently with other operations against them. This can't happen in Chess or Go, where the demands on your focus are far less urgent and two things can't happen at once in a literal sense. Can an AI agent learn to appreciate the power of distraction? Can it learn to intuit what will be most disruptive to a human, and what won't be disruptive at all? How can you teach a computer to _learn_ to be annoying?

I will say, of course, that nobody saw AlphaGo coming. And I hope it's the same with RTS's. That would be _so_ exciting. I would love to see an AI blow us away with previously unthought-of strategies. That would be the coolest thing ever. So I hope it happens. But I'd be astonished. RTS is just such a whole new level of thinking for AIs.

<comment><USER18> On the other hand, bots are starting to beat professionals at (thousands of hands of repeated) Poker, so I think we can't say that imperfect information is something that's especially intractable for maching learning algorithms.

[http://spectrum.ieee.org/automaton/robotics/artificial- intel...](http://spectrum.ieee.org/automaton/robotics/artificial- intelligence/texas-holdem-ai-bot-taps-deep-learning-to-demolish-humans)

<comment><USER43> Yeah but compare the search space of Poker vs Starcraft.

<comment><USER42> Exactly. And poker is, again, turn-based with a single move to be made. And it may not be "perfect information," but it again is a game of static board state where there is probably a statistically optimal move for a given state provided you have memory of the other players' previous moves so far in a round.

</comment></comment></comment></comment></comment><comment><USER44> > even strong baseline agents, such as A3C, cannot win a single game against > even the easiest built-in AI.

Then, why not release code for the built in ai, and improve on it ? Or is the built in ai cheating ?

<comment><USER43> The built in ai is scripted whereas they're trying to teach agents to learn the game from scratch with some sort of reward-based/machine learning approach.

<comment><USER44> still, why not use machine learning to tweak the script instead of starting from scratch.

</comment></comment></comment><comment><USER45> Someone needs to link this to FB's ELF platform (An End-To-End, Lightweight and Flexible Platform for Game Research). That was specifically made for RTS games like SC.

</comment><comment><USER46> great they opened it up. I'm sure reinforcement learning / Deep learning will solve this. It has been a tough problem before, but honestly doesnt seem that tough compared to all the harder AI problems.

<comment><USER43> Such as?

</comment></comment><comment><USER47> This gives me great ideas

</comment><comment><USER48> I think I know what my final year project will be.

</comment><comment><USER49> YESSSSSSS!!!!!!!!

\--why are there not more fanboy comments?!

<comment><USER50> Probably because they don't contribute much to the conversation.

</comment></comment><comment><USER51> "so agents must interact with the game within limits of human dexterity in terms of "Actions Per Minute"."

I am really glad they are limiting APM because otherwise things just get stupid.

<comment><USER52> IMO there should also be a precision limit. The timing of actions should include human-typical jitter and the wrong action should sometimes be activated to simulate misclicks/fat-finger keypresses — e.g., messing up a control group by assinging a unit to the wrong number key. The bot must also not be able to act faster than human reaction times (~250ms), this could be enforced by adding a fixed delay to the observations.

I wouldn't be surprised if human Starcraft II play isn't so much limited by decision-making as by the translation of decisions into mechanical actions, which in turn dilutes the attention devoted to actual decision making.

<comment><USER53> ideally they'd train it on real keypresses rather than actions

<comment><USER54> Why would that be ideal? Wouldn't that just make ML at the strategy layer harder without doing anything to make the discoveries more valuable?

<comment><USER53> to emulate human handicaps at the interface layer. I didn't say it would be free

<comment><USER54> But why is that desirable? Why would we want to emulate the human physical handicaps in our quest to advance AI at a strategy level?

<comment><USER53> For the same reason the APM are limited: to ensure that what we are doing is really focusing on advancing strategy rather than brute mechanical skill. If I played against an AI using nothing but the rendered frames and sound of a game as input, I might not even make the stipulation on reflexes. I'd be humbled if I lost.

As it stands now, most of the games I like have bad AI. Sure, it can be fun to play a hack and slash against lots of little, dumb minions, but FPS, RTS AI these days still don't cut it as savvy opponents. Often they have inhuman perception, direct knowledge of game state, or higher starting resources, but they make abysmal decisions.

Yes, I realize these are unlikely, expensive goals and incremental progress is how things are done. I just want to know if it's possible or desirable to emulate actual human reaction time.

Do you disagree this would in principle help separate strategy from godlike reflexes?

<comment><USER54> This is not the same AI you normally face in a game. Most (all?) of those AI opponents use rules written by the game developers to make decisions and some of them simply cheat to be competitive ( _cough_ Mario Kart 64 _cough_ ).

This blog is about creating AIs that interact with the game the same way humans do, the computer plays by the same rules and has no special access to the game state beyond what the player would have. With these constraints there are no existing bots for StarCraft or StarCraft 2 that can even beat the built-in rule-based AI. They aren't even close to beating professional players.

If the strategy abilities are so weak today that we can't even beat the tutorial AI then why introduce further arbitrary handicaps on the bots? How do those handicaps advance the state of the strategy layer? The AI has many potential advantages over the human player beyond just reaction time. Should we also limit the amount of data the bot considers to emulate the amount of inputs a human player can process? What about emulating human memory, can a human really learn from 60,000+ games? What about 1.5 million?

I do not think it is desirable to emulate human limitations in AI unless you are trying to create an artificial human. I think the advantage of creating an AI is to do something people _can 't_ already do so why should we impose our physical constraints on them?

I do not think it is important to separate reflex from strategy. Since every player has a different APM ability some strategies are more valid than others for each individual. If I do not have the reflexes of a professional player there are strategies I cannot employ. As long as StarCraft is not imposing APM limits on human players to maintain competitiveness the bots should also not have a limit.

<comment><USER53> Okay, thanks. I appreciate the counterpoint. I guess I'd like to see it both ways: bots limited to human speed and bots not.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER55> It's not like this is going to create fantastic AI.

Keep in mind there's been an amateur AI project for broodwar for almost 7 years now. Even after such a long learning period, the games are very primitive, and the AI's still couldn't pose a threat to even a beginner human player. Sometimes the games take hours. Trying to build strategy and decision making into an AI is incredibly complicated. There have been teams working at the SSCAIT for many years now, and the product is still fairly primitive.

So what CA did was instead write up a simpler AI that mimics strategy and decision making. We all know it's not great, but I'd be really skeptical that 3rd parties would magically create an AI that can think strategically.

</comment><comment><USER56> Novice here: I really want to try this Starcraft API but I don't know how to start. I believe this uses more reinforcement learning and agent-based models (which honestly I am not familiar with yet) What are good papers to get started on this?</comment>


Invest in Lines, not Dots - <AUTHOR> http://www.bothsidesofthetable.com/2010/11/15/invest-in-lines-not-dots/ <comment><USER2> Why is reading about _raising money_ always such a time suck?

I swear, if we all spent as much time on our products, and less time reading about fundraising....well, we'd probably force AOL to shutter TechCrunch in a month.

</comment><comment><USER3> This could well apply to recruitment of people: lines not dots.

</comment><comment><USER4> Surely someone will invest in the 'dot' and he'll miss the chance. He seems to assume there is no competition. Or maybe what is a dot to him was already a line to someone else (because they met earlier).

<comment><USER5> I don't assume that at all. I'm arguing in favor of investors getting out and spending time with entrepreneurs now that they might want to fund later so that your line becomes someone else's dot.

</comment></comment><comment><USER6> For me, the real takeaway for entrepreneurs who want to get funded is to better "manage expectations." In other words, contact VC / angel early, set artificially low expectations initially (ones you know you can hit, so you build momentum), give yourself time, outperform, communicate out performance...rinse and repeat.

</comment><comment><USER7> Can anyone give a better explanation of "performance" as shown by those graphs. What situation corresponds to performance going down? If performance relates to "speed", why would you expect the speed to continue increasing, rather than staying the same?

<comment><USER5> As I wrote in the article the interpretation is left to the investor. It could be product features shipped. It could be biz dev deals signed. It could be talented people hired. Whatever. The point is that you get to see how the person accomplishes things that they set out to do. If you meet them only briefly over a 2-week period you don't have enough data points to decide.

</comment></comment><comment><USER8> I thought this was going to be about investing in companies that connect entities together rather than standalone entities.

</comment><comment><USER9> Huh. I thought it was an article about vector graphics vs bitmaps.</comment>


Get Hacked, Don't Tell: Drone Base Didn't Report Virus - <AUTHOR> http://www.wired.com/dangerroom/2011/10/drone-virus-kept-quiet/

<comment><USER2> The plan to build a single centralized overseeing system doesn't seem to be the smartest choice security-wise.

<comment><USER3> Amen. But it will probably take a full-scale war to make people (and nations) aware of the security implications of armed remote-controlled robots. Some of them we can only guess at.

Future war: nations will hack each other's drones and use those drones to attack their owner? (or a third party, to provoke counter-attack)

<comment><USER4> Maybe we can have special drones to hack and take over other drones. That'd be pretty cool in a command and conquer sort of way.

</comment></comment></comment><comment><USER5> Does anyone else think this is a windows infection, and that this OS should be banned in sensitive areas ?

<comment><USER4> That would be security theatre. Any actually dangerous attack on systems such as these is likely to be bespoke, like stuxnet. Therefore the platform doesn't matter.

Also, it shouldn't be too hard to create a virus that could spread via windows PCs but still carried a linux payload, for example.

As soon as you're up against bespoke, targeted attacks, the whole game changes.

<comment><USER6> You are right, but Linux is superior here, because it is open source and you know everything about it. For Windows you can simply buy a 0day and that's it. When you write something for Linux, you can't be sure the targeted program is even on the distribution.

The NSA has excellent experience when it comes to secure UNIX systems. They published a very long RedHead guide somewhere, was posted on HN some time ago.

But in reality there is no way they share their knowledge, totally different organization and different contractor so synergies are lost for sure. How much energy would be saved when you don't try to reinvent the wheel every time.

<comment><USER7> I believe you mean this: <http://www.nsa.gov/ia/_files/os/redhat/rhel5-guide-i731.pdf>

Regardless, if its a _targeted_ attack you can probably figure out whats on the system you are targeting.</comment></comment></comment></comment>


Is Swift loosing adoption ? - <AUTHOR> https://trends.google.com/trends/explore?date=all&q=%2Fm%2F010sd4y3,%2Fm%2F0_lcrx4,%2Fm%2F09gbxjr,%2Fm%2F0h52xr1,%2Fm%2F0n50hxv <comment><USER2> No, Swift isn't like other langugaes which start from nothing. Most languages come out quietly and build a userbase slowly but steadily, which results in more organic growth, so they will have a steadier curve. Existing developers however have no real pressure to move to the new language, and can do so at a comfortable pace IF they decide it is even worth learning.

Swift however was released to a huge population of existing iOS and Mac developers, in releasing Swift Apple basically set a timer on the end of life of Object C and it meant that all developers at some point or another HAVE to learn swift to continue having a healthy career as iOS/Mac developers or be relegated to maintaining legacy code (which usually doesnt last long for client applications). So they release it, instant exposure to all existing Objective C programmers, instant interest from nearly everyone in their target audience (existing developers), and they're all in a rush to begin learning about it as staying relevant depends on it. However once a large subset of existing developers have already learnt about it, interest dies down again and it evens out at a rate you would expect from any other language based on interest from NEW developers.

<comment><AUTHOR> That would explain the initial large ups and downs. But i don't explain the general slow downward tendency starting from april 2017 and going on ever since.

You can compare that with kotlin (suddenly adopted officialy for android), typescript (suddendly adopted by major javascript frameworks, and sponsored by microsoft), etc.

I just don't explain the trend other than either language loosing market share on some fields (such as server side development), or ios native development loosing market share in general. I don't think people are reverting to objective-c...</comment></comment>


Founder of Silk Road could have been the 280th richest person in the world now - <AUTHOR> He had amassed 600,000 bitcoins when he was arrested, worth about 6 billion dollars today.  If he had kept going, he could have easily doubled that to 1.2 million bitcoin, getting him close to top 100 richest <comment><USER2> Satoshi Nakamoto hasn't touched the account in years. There can be multiple reasons:

1\. He's doing this for the good of the ecosystem

2\. He values privacy over money

3\. He's dead.

4\. He's confined somewhere without access to the Internet, like prison.</comment>


The Landscape Of Front-end Development Automation - <AUTHOR> http://updates.html5rocks.com/2013/11/The-Landscape-Of-Front-end-Development-Automation-Slides

<comment><USER2> Very useful. Thanks.</comment>


The Amazing Bureaucracy of Burning Man - <AUTHOR> https://www.citylab.com/life/2017/08/the-many-rules-of-burning-man/538389/ <comment><USER2> A good deal of attendees are socialists at heart. They tend to enjoy bureaucracy if it's presented as coming from and benefiting the people. They call the place "home" and the bureaucrat fills the role of a loving parent that actually cares.

<comment><USER3> Please explain how Burning Man attendees are "socialist" because they go to an event with rules

</comment></comment><comment><USER4> Burning Man is bureaucratic because it is big. If it were smaller it could be more casual about all sorts of things. No one cares if one person poops in the wilderness. But 50,000 people doing it is rather a different matter.

That said, I'm a bit surprised that people who make a big deal about independence and thinking different put up with the bureaucracy, whether necessary or not. You'd think they would prefer smaller gatherings that don't need all that formal process.

<comment><USER5> Since BM reached 50k people, the majority of the people coming aren't the same people as those who started it, generally. It's become much more expansive, for better or for worse, and that means that far fewer of those 50k people than before have the strong ideological bents that earlier folk did.

And that's ok, sometimes you just want to party in the desert.

</comment></comment><comment><USER6> TLDR: East coast academic goes to burning man without doing any research and discovers the concept of MOOP and that Burning Man is not a lawless Madmax event where anything goes.</comment>


Front-End Microservices - <AUTHOR> https://jobs.zalando.com/tech/blog/front-end-micro-services/ <comment><USER2> I've worked on a product that was built using "Front-end Microservices". Probably one of the harder development platforms I've ever run into.

It takes a lot of diligence and uncommon knowledge to do it successfully. For instance, when I started on this project, each "microservice" was deployed individually, meaning if you went from /account to /product you were loading an entirely different app. This gave users an initial page load several times during a typical workflow.

I currently work on a massive monolith which ties our entire backend and frontend together in one app. It's a little clunky but overall it makes it a lot easier for my team and me to ship reliable code.

</comment><comment><USER3> It's like these companies live in a totally separate universe where none of the normal software engineering principles apply any more. The micro services cargo cult permeates everything.

<comment><USER4> This front-end micro-services architecture looks a lot like how Amazon renders its pages. Amazon's rendering framework takes in components from various teams, renders their components, assembles it all into a final page and then sends it to the client. In theory, this allows all the the benefits promised by the article: teams can use whatever frameworks they want, they can architect their part of the page in a manner that suits them, etc.

In practice, it's a mess. Just go to Amazon.com and open the page in the DOM inspector, or view-source. There's no consistency between any of the page components. Things are duplicated. Things are coupled to each other in weird ways. It looks like an absolute nightmare to debug.

<comment><USER5> Amazon has invested significantly in tooling that makes debugging this possible.

I would not recommend anyone go down this route unless they are also willing to invest heavily in the tooling support necessary.

<comment><USER6> So the "remedy" to bigger spaghetti is a bigger debugger?

<comment><USER5> No, I was answering the statement "it looks like an absolute nightmare to debug".

Amazon's page request and rendering pipeline, I would estimate, is necessary complexity. Due to the large engineering force and the speed at which they want to move, it's imperative that the infrastructure support distributed teams delivering at their own pace while minimizing integration points via shared repos but instead integrate via defined APIs. This is a fallout of that need. Do most companies have that need? Probably not.

<comment><USER4> It's arguably necessary complexity for Amazon. I would argue that the necessity of the complexity arises from the way that Amazon treats its teams as individual standalone business units, which prompts teams to treat their part of the website as a standalone application which communicates with other parts of the website as if the were third parties. This has advantages and disadvantages, which have been outlined elsewhere.

However, even assuming that this is necessary complexity for Amazon, it does not mean that this is necessary complexity for your organization. In fact, I would argue that this is very likely unnecessary complexity for the vast majority of organizations. I concur with others in this thread who say that this framework is the result of an assumption that splitting everything into tiny components communicating via API is a good idea for user interfaces. The main problem I have with this approach is that it encourages divergence in user experience for different parts of the web site or web application. This is definitely a problem that Amazon has, and it's a factor that should be kept in mind before moving to a UI pattern that's modeled off microservices.

</comment></comment></comment></comment></comment></comment><comment><USER7> I appreciate the skepticism in the comments because this is a level of complexity that just is not needed in most projects.

This solution adds value when you have an engineering org large enough to have different teams responsible for different parts of your app that are bundled into a single experience for the user. For example, on Airbnb, one team might own the listing details page itself while another team owns the booking form and flow.

<comment><USER8> Fully agree. A monolithic architecture works for most cases but not ALL cases. Large engineering orgs are one such usecase where the monolithic architecture starts to become less appropriate.

</comment></comment><comment><USER9> I gave a quick look at the article so it's probably my fault - can anyone explain to me why this are not just a new iteration of the concept of Portlet (1) which was supposed to totally renovate etc. etc. ...

(1) [https://en.wikipedia.org/wiki/Java_Portlet_Specification](https://en.wikipedia.org/wiki/Java_Portlet_Specification)

<comment><USER10> I had the pleasant of designing and implementing a portlet application back in 2005 - 2006 circa in Java with JSR164 (I may put the wrong JSR here). Initially, I thought portlet concept was very cool and could be a game changer. After a year or so working with it, my opinion changed: it was so convoluted, needlessly. If the goal is to shift the rendering responsibility to the portlet service owner then you could have used client side rendering with AJaX as well (remember AJAX?)

In the end, I think what really killed the portlet development is the complication it causes when you have to maintain states among the portlets.

</comment></comment><comment><USER11> Server-side Includes or Edge-side includes, which would include either statically generated (but routinely updated) or dynamically generated (but cached for a short while) assets work just as well if not better, but they're "old" technologies, I guess?

</comment><comment><USER12> > The best solution I've seen is the Single-SPA "meta framework" to combine > multiple frameworks on the same page without refreshing the page (see this > demo that combines React, Vue, Angular 1, Angular 2, etc).

I have very little front-end experience (most of the web dev work I've done was hand-written javascript). But having the overhead of a big, comprehensive framework such as React, Angualar already seems like an overkill to me. But having several such frameworks on the same page seems outrageous, it cannot be considered good engineering. Where are the engineering good practices of efficiency, craftsmanship etc?

<comment><USER13> I'm even thinking about the human issues here. If you've got different teams using different technologies, over time you're going to end up with a tangled, poorly standardized codebase where the same problem is solved 6 different ways in 6 different places, and people are afraid to make changes because it's hard to predict their impact.

I have had the pleasure of working at a microservices shop that had everything working very, very well. The system was a pleasure to work on, development was easy, the operations team was more in control than anywhere else. But I think they accomplished it by unilaterally banning a lot of the things that people think microservices will let them do. Only two programming languages were allowed (a low-level one for the most performance-critical stuff, and a high- level one for the rest), only one flavor of database was allowed, communication protocol changes and any new 3rd party libraries had to be approved by a Star Chamber tribunal that basically said no to everything, lines of communication were strictly controlled (viz., none of this "everyone talks to everyone else through Kafka"), etc. etc.

It was glorious. It really was. Easiest codebase to work in ever. Did I get to use my favorite languages or libraries? Nope. Was that holding the company back? Double nope.

</comment></comment><comment><USER8> You would never start out building a website with microservices. They have all the downsides people have already commented on.

But are people taking into consideration team/employee size?

If you have 100+ engineers working on a site, in different countries with different managers and different leaders with different product goals. Then the microservice architecture becomes a necessary evil.

<comment><USER8> In my experience I've seen good productivity out of a monolithic architecture up to 50 developers. Have others seen the monolithic architecture scale higher?

<comment><USER14> Google?

<comment><USER8> They have a monorepo - not monolithic app.

<comment><USER15> I think the expectation is that the app is relatively monolithic, as well. If you can force everyone into a single repository, why not force them to a single architecture to push data in?

</comment></comment></comment></comment></comment><comment><USER16> What is the benefits of doing this? Micro-service evangelists never care to explain that.

They link to another blog article that say:

> The monolithic approach doesn't work for larger web apps

> Having a monolithic approach to a large front-end app becomes unwieldly.

But in my opinion breaking up the frontend (or backend for that matter) they way that they say you should is much more "unwieldly".

Seriously, is there any benefits at all? Job security?

<comment> jf- At some point in the development of a large app you run into irreducible complexity. The logic and moving parts are just necessarily complicated and nobody is going to fully understand the system. It doesn't matter what architecture you use, hard is hard. I think people tend to forget this and build a house of cards trying in vain to correct it.

<comment><USER6> Okay, but there are a lot of ways to split up & scale projects and teams besides microservices. What's needed is a guide for scaling that considers all the common options and their tradeoffs, and how to measure those tradeoffs against your own shop.

</comment></comment></comment><comment><USER17> The hardest part is, again, client-side assets fetching/caching. The architecture seems scalable for large projects. It's hard but will be easier if frontend frameworks go to this direction.</comment>


MountainWest RubyConf 2010 Session List + Registration - <AUTHOR> http://mtnwestrubyconf.org/2010/

<comment><AUTHOR> I'm on the advisory board and I'm speaking at this conference, but MWRC has been consistently kick-ass. And it's $100 for 2 days of hard-core Ruby talks.</comment>


Ask HN: Of the startups from YC's Demo Day (day 1), which would you invest in? - <AUTHOR> <comment><USER2> Amazed by some of the choices YC made. A social network for colleges??? in 2016? Random phone calls with strangers? Notification (spam) platform? The freakin Uber of birth control?

Oh my... the majority of these startups might not survive 2016. I think they should be more picky and not necessarily pick a team first. Yes you can pivot to another idea anytime and reuse the same people but I bet you a lot of the rejected applications could be worked around in the first place if the idea is really cool but a single founder. etc. (I've never applied BTW).

Thought, I have to say that they have 2 great ones like by-the-glass wine delivery and the personalized beauty products. That's what I'm talking about. Can't wait to see Day 2, hopefully it'll gradually go up in quality. It has to...

</comment><comment><USER3> None-- all of them seem too temporal, nothing really ground breaking or innovative.

<comment><USER4> In the past which startups were ground breaking and innovative on demo day?</comment></comment>


Twitter for iPhone Gets Push Notifications - <AUTHOR> http://mashable.com/2010/11/16/twitter-iphone-push-support/ <comment><USER2> Lots of Twitter users already have push notifications set up for mentions and DMs from texts, emails, Boxcar, Notifo, etc.</comment>


Ask HN: Should I make my app freemium - <AUTHOR>

So on Saturday I launched BugMuncher (http://bugmuncher.com), my Google+ inspired website feedback tool.
I've had a lot of positive feedback, but unfortunately not much in the way of sign ups. I'm wondering should I create a free version, with limited functionality, and if yes, how should I limit it?
I was thinking the free version would not have the blackout functionality, and maybe ads on the preview page?
Also, while I've got your attention, have you got any other thoughts on how I could increase sign ups (is my design all wrong, price too high/low, etc)?
Thanks,
Matt Bearman <comment><USER2> I don't think pricing is your fundamental problem.

* Your primary market is where diagnosing rendering problems, error conditions conditions etc. are not merely a difficult problem, but an _impossible_ one. You need to be targeting visualizations, games, or similar. With the kind of limited web development I do, basically hacking wordpress templates, your product solves a non-problem for me.

* You're going to need some sweet explanations for braindead users. As a developer, _I_ know how to use the product, but will my users? Probably not. You need a 1-2-3 flyover to explain WTF is going on.

Alternatively, you could pivot and focus on targeting freelancer frontend web designers/developers. I would imagine that the freelancers have a need for a simple "markup the website" tool so a client can annotate the site with their feedback, "change the font of this title", etc.

<comment><AUTHOR> Thanks for the feedback, some really good points there. The pivot angle is one I had briefly considered, maybe I'll give it a bit more thought.

I've got a screencast showing how to use BugMuncher, maybe I should link to that directly from the help link in the tool itself?

</comment></comment><comment><USER3> I just realised you could make this a lot more useful if you allowed users to load any website via bugmuncher.com and provide feedback to the site owners. To be clear: I go to bugmuncher.com, type in a url (say nytimes.com), the page loads, I use your tool to highlight the bug and hit send, you email or tweet it to the appropriate address.

You could also look at it as more than a bug reporting tool. It could just be a playground for people to visually comment on sites, sort of like <http://canv.as> does for pictures.

This could get you some traction and then you charge site owners for premium features, like tracking on their own site.

PS You should put your email in your HN profile's about section.

PPS You should let users submit feedback without highlighting anything and without previewing.</comment>


Ask HN: Why are finance managers paid more than software developers? - <AUTHOR> I work in a famous technology company. Today, one of the finance managers by mistake left her printed salary slip on my desk while passing me some internal documentation/policies. I was shocked to find that she earned much more than I do.
For context, I am an experienced software engineer, working in a senior position. In fact, the work my fellow S/W engineering colleagues and I do make the backbone of my company. If we quit for some reason, the company is fucked.
However, this isn't just specific to my company - a quick google search returns that this is the case in most companies.
I slog my life away working 12 hours almost, while this chick hardly works for 6 hours. Most of the time, she's checking her Facebook or changing tracks on Spotify and so forth. I understand the work she does is important, but is it more important than the guys who actually built the product itself?
Please help me understand if I'm missing something, or is it time to jump ship to somewhere else where the gravity of work I do is respected and appropriately compensated. <comment><USER2> LOL. Why is a software developer paid more than a teacher. If for teachers quit for some reason, the schools are fucked.

Context – I'm a software development manager; with 15 years experience as a software developer

</comment><comment><USER3> >Please help me understand if I'm missing something, or is it time to jump ship to somewhere else where the gravity of work I do is respected and appropriately compensated.

Maybe you are not fully understanding the gravity of someone else's job and do not respect it. If finance workers would all quit for some reason the company would be fucked too.

</comment><comment><USER4> Another opinion than the others: I'm french and in my country, the position of software engineer or developper or any technical position in the making of a software product / services is still a relatively new job.

I am working in a startup and until now we had more non-technical employees (it changed after a big pivot and big layoff).

We were the one making the product but every other positions did not have any considerations for us except treating us like weird wizards accepting any requests.

As their understanding were limited, they usually had more respect for other job like finance, or sales, without regards for the actual responsabilities of the employees.

There is a strange cultural bias that do not recognise the capabilities of a software engineer as more complex than the work of a young salesman.

</comment><comment><USER5> it would be good to take a step back and have a global perspective on this.

If you consider yourself as a product in the market what value would you ascribe to yourself . And it also depends on the goals you have for yourself. the demand and supply of the market dictates your value , if what you do is easy and anyone can replace you or you can be replaced by software , then you should probably start to retrain and better position your self .

Having said that a company is also made of people and some people know where the skeletons are buried and how to manage people (which anyone who has tried, will tell you is non-trivial) these skill are valued separately based on the organisation size and age/fame .

</comment><comment><USER6> Bringing value to the company is more than just the hours you see a person doing the work.

For example, if you were able to complete your tasks in 50% less time and spend the rest of the time reading Hackers News, should your salary go down by 50%?</comment>


Show HN: WhatWhen - Tidy timelines for freelancers (built w/ Meteor) - <AUTHOR> http://whatwhenapp.com

<comment><AUTHOR> What is this? WhatWhen is a tool freelancers can use to keep their projects on schedule and clients up to date. A timeline of dates/times related to a project: payment due dates, work days, meetings/reviews, etc.

Why? First, it's part of my process. All project's are given a rough timeline in both the proposal and contract stage. Before now I did all of this in InDesign. This is an effort to streamline that process.

Second, I think others can benefit from it. If you've ever done freelance and had a client nag you about an upcoming meeting, deliverable, whatever, you know how stressful it can be. Hopefully you can use this as a tool to let clients know where a project is at and what needs to be done when (ha).

Business Model? I'm working on a larger app called Proper (<http://properapp.com>) that looks to solve the problem of managing client expectations in the form of proposals/contracts. I'm still working out the idea, but the goal is to create a tool that makes it easy to deliver a scope, timeline, cost, and terms (the contract part) to a client. WhatWhen is 100% free to use (you get up to 5 timelines).

What I'd Like from the HN community? Solid feedback. Nothing erroneous like "this sucks, my app is better," but good, honest feedback. Looking to connect with any freelancers who see value in a tool like this and have some insight. I'd also like you to just give the product a try. Yes, you have to sign up to use it, but considering the product is free, I think it's cool to give a fellow designer/developer a hand and try out their stuff. If you tweak out about filling out sign up forms, I'd pack up early.

Discussion Beyond This Thread, Bugs, and Feature Requests? Just shoot me an email: email@example.com and we can chat.</comment>


An attempted heist at Coinbase was scary good, even though it failed - <AUTHOR> https://www.technologyreview.com/s/614094/an-attempted-heist-at-coinbase-was-scary-good-even-though-it-failed/ <comment><USER2> I suspect the larger the prize and potential to succeed to get it, the more one will be willing to spend to get it.

Practice defense in depth - you simply cannot protect yourself from 0days.

</comment><comment><USER3> TLDR: Two zero days. Complex social engineering including spoofed university emails and identities. Not a lot of detail.</comment>