Ask HN: Side income while still highly invested in full time job? - <AUTHOR> I'm a software engineer with a job that I really enjoy with a company I want to succeed. At the same time I'd like to start earning some side income. I also have a young family so I need to be careful with my time commitments. Any suggestions? <comment><USER2> It's hard to assess your situation but I would rather look for a better job (or better-paying job). If you still want to try it keep in mind that:

* You need to be extremely careful with IP and legal: are you going to use your work machine for a side project? Are you going to do any side work from your current company offices? Is there conflict of interest? All those things need to be carefully planned.

* You're going to get distracted, so your performance on both jobs will be lower.

* If your management is going to figure it out, you'll lose trust. If I'd figure that one of my employees does contracting on the side, I wouldn't be happy. If you really need more money for a serious reason, I'd prefer people to chat with me about that during 1:1. We've had an employee couple years ago who had a brain tumor and we covered his medical expenses on the company, with no obligations, which was more than his annual salary. He recovered successfully since.

* Having two jobs will impact on your quality of life and your family. Maybe not instantly but at some point it will. Be careful not to burn out.

<comment><USER3> While I agree to most of what you said, the attitude of paying based on what someone needs doesn't seem to be a fair basis for negotiation to me. The case you mentioned is surely an extreme that deserves an exception to be made and I highly appreciate that you helped your employee in this situation, but this shouldn't be the general way to assess what someone is worth to you as an employer.

Also someone working on the side (within limits) may actually benefit you as an employer. It's a way to gain experience that you otherwise couldn't get your hands on. I agree that things need to be made transparent and agreed on before, but I'd rather not recommend a general negative attitude towards this.

<comment><USER2> I'm assuming the case when one gets paid fairly, regardless of their location. In my previous comments on one of GitLab posts I expressed my disagreement with the way their salary calculator works: it calculates salary based on location and I'm against that.

When discussing those things first thing to keep in mind are legal complications for both parties, and being open about this lets everyone deal with risks properly.

<comment><USER4> "Fair" in a capitalist economy is based on supply and demand. If you want to get a developer locally to Silicon Valley, you're going to have to pay more based on the pool of qualified individuals. If your pool is "the entire United States" or the "entire English speaking population", as employer, you can pay less. As an employee, you're competing with a much larger pool.

Why would s company pay more than they have to?

</comment></comment></comment></comment><comment><USER4> Suggestions? Don't do it.

Your family comes first. Why would you work full time at a company _and_ do side work at the expense of your family?

Do you have equity in the company? If not, your employment with the company is solely a business arrangement. You give them X number of hours per week and they give you Y dollars.

If you can make the Y larger without changing the X, why wouldn't you? Why would you sacrifice time with your family for a company? Do you really think if times got hard and they could lay you off and survive they wouldn't?

Unless you have a decent ownership stake at a company, never confuse a company that you work for with _your_ company.

</comment><comment><USER5> Check any agreements you have with your company they may contain wording that anything you create is their IP or other restrictions.

If that checks out I would look in to building income with products and eventually a SaaS app.

Listen to the archives here:

[https://StartupsForTheRestOfus.com/archives/](https://StartupsForTheRestOfus.com/archives/)

Rob and Mike cover starting a business on the side and lots of topics that will be of interest to you.

This is inspiring too: @DHH Startup School Talk 2008 [https://www.youtube.com/watch?v=0CDXJ6bMkMY](https://www.youtube.com/watch?v=0CDXJ6bMkMY)

Keep your focus on your family and on your day job.

There are hours here and there when your family is asleep or busy that you can put that time in to building products. If you enjoy this type of work you can build your own products instead of watching TV two hours a night. Follow Rob's stair step approach. Like DHH says it's still hard, but it's definitely doable.

Good luck.

</comment><comment><USER6> I generally don't do long answers but I've done what you are suggesting so stick with me.

I worked in a school and created a saas product as a sideline. This was very manageable in the world of school. I joined an IT company 8 years ago and brought my saas product with me. My product requires very little support so the new company were happy to let me do my full time work and let me look after product. The company even thought they may get involved with my product but it differed too much from their vision. 8 years on I now work 3 days a week for the company and 2 days a week on my on own saas product and I'm very happy. The company have been very supportive.

Some people here are saying don't do it, think of your kids. I started my sideline as I wasn't making enough to save any money and wanted to put some money aside for my kids. My kids and I (who are now adults) are very close. Do what you think is best.

I think it's important to be honest. If you go down this route make something that requires as little interaction with your users as possible. It was always important for me not to be the guy sneaking out to take calls.

I reckon a good starting place would be setting aside an evening or something and sticking to a fixed amount of time. I've been down the road of a full time job and sideline taking up my whole week. I won't go back there again. I don't work on Saturdays. I don't really work on Sundays either but I'm very involved in my church which is another type of commitment.

Have a good think about why you want to do this? Having my own thing which I made and try to grow fires me up. Know what your reasons are so you don't end up disappointed.

</comment><comment><USER7> www.moonlightwork.com is an agency specific for what you're looking for, or toptal and list yourself as weekends or nights/mornings only.

They also hire interview phone screeners all the time, and whenever I've talked to one of these guys they sounded like they were either driving in a car or in a coffee shop lineup so seems like it's something you could easily juggle on the side. They're always looking for experienced developers to do phone interviews. These positions are usually on github jobs and other remote job boards [https://github.com/lukasz-madon/awesome-remote-job#job- board...](https://github.com/lukasz-madon/awesome-remote-job#job-boards)

As for time get up early and finish by 11am if it's a flexible enough side gig

</comment><comment><USER8> I started doing /r/flipping as a side hustle. It works, but it scales with how much time you're willing to spend on it. But that's also a benefit of it-- no hours, no deadlines, nobody to answer to. And failure is painless. Obviously it makes less money than contracting on the side would, but I couldn't find a way to do contracting without signing up for large chunks of time in advance.

</comment><comment><USER9> I have a slightly different take on this. I know that the OP said "start earning some side income". However, I think there can be value from a side project other than income. It can be a great hobby the allows one to explore other creative avenues.

As a manager (and founder), I encourage my team to have outside projects. However, they should not be done on the company's dime. I also encourage them to use most of the cool tools we have.

For me, I'm currently building solid wood word clocks. A coworker made a bunch of educational web tools for teachers (spouse is a teacher). One sw dev was learning electronics and borrowed some nice soldering equipment and an o scope.

Some of us make a little money doing these things to make it fun and cover our hobby's expenses. The point is these are more of a hobby (that might earn some money) than a job. By encouraging this with my employees, they get a breath of fresh air and some alternative perspectives.

FYI, the minimum stint for engineers at my company has been about 5 years. Our most valuable engineers have been with me for more than 15 years.

</comment><comment><USER10> Your family is your side job. The thing your kid wants most from you is your time.

</comment><comment><USER3> Are you aiming at passive or active side income?

I find active work on the side to be much more manageable than passive. E.g. you could start doing contract or consulting work on the side. Keep your employer in the loop. It's important that they know about your side activities or otherwise you might get into trouble.

Passive side income is much harder to do if you don't have experience with this kind of work. Especially if you have a family and cannot commit to investing a lot of time. You need to get really good at self management and make sure you don't commit months worth of your time into something that later turns out to not earn you any money. Start as small as possible. Don't make the mistake of thinking that a product consists of code only. From my experience at least two thirds of product success is outside of engineering (design, marketing, sales, finance, management).

Good luck!

</comment><comment><USER11> Man, it is sooo hard for me to put mental energy anywhere besides what I'm working on full time.

My advice is to first ask if you're in shape. Maybe you could work out instead.

I recommend having a fixed weekly (or monthly) time budget on this. It's perfectly reasonable for you to have a hobby like making money.

</comment><comment><USER12> > I also have a young family so I need to be careful with my time commitments

Find a better paying job if needed. Taking time out of family time when kids are young is something you will definitely regret later - as do some of my friends who took up high paying management consulting gigs when their kids were young and missed the whole childhood of their kids.

</comment><comment><USER13> How are your investments doing? Your salary can help put away a ton of money you can grow using vanguard funds.

</comment><comment><USER14> Day trading, Landlording, freelancing : some of the things I do.

But if you have young family, you might want to invest your time in the family and build a solid foundation. That could help you spend more time away from them being ambitious in the future.

<comment><USER15> Off topic but are you from nepal?

<comment><USER14> :) No, I am not. I trekked to Mt Everest base camp last year and I stayed there the first night after starting at Lukla.

Are you from Nepal?

<comment><USER15> I am not either but I love the country and its people. I have trekked before around the Annapurnas.

</comment></comment></comment></comment><comment><USER16> Codementor.io. You can generally work as little or as much as you want (bulk of work is helping others via videoconferencing). Oftentimes mentoring calls you do can turn into more long-term side work, if that's what you want.

</comment><comment><USER17> Check out indiehackers.com it is a great community for those building profitable side projects.

</comment><comment><USER18> > I also have a young family so I need to be careful with my time commitments. > Any suggestions?

Spend time with your family? Time is the only thing you can't buy. How are you going to start a side business and earn income if you aren't willing to trade time with your family? There's only 24 hours in a day.

If you are a software engineer who is so poorly paid you need to look for side income, maybe look for another job? Unemployment is at all-time lows and companies are desperate for tech talent.

I can understand baristas looking for side income, why would a software engineer need one?

<comment><USER19> ^ This is the right answer. Find another, better paying day job. That shouldn't be super hard if you're in the US. Do not steal time from your family, you will 100% regret it eventually, guaranteed.</comment></comment>


The Fortran Whetstone programs were the first general-purpose benchmarks - <AUTHOR> http://www.roylongbottom.org.uk/whetstone.htm <comment><USER2> I remember those, followed by Dhrystones for integer performance [1] which were widely used in the minicomputer industry for the shorthand MIPS performance characterizations of systems.

[1] [https://en.wikipedia.org/wiki/Dhrystone](https://en.wikipedia.org/wiki/Dhrystone)</comment>


Ask HN: When Amazon.com will accept bitcoin? - <AUTHOR>

I am asking it because I can see bitcoin articles almost everyday here in HN. I'm sure Amazon.com will accept bitcoin some day, but when? <comment><USER2> I'm not going into the _flaw_ that, I believe surrounds Bitcoin and its algorithm (in particular that I believe Bitcoin isn't acting like a currency as its incentivising hoarders & due to that this will lead to a recession and because the money supply does not expand it will evolve into a depression) but rather the issue around Amazon adopting Bitcoin today.

First of all let's put some perspective here – Bitcoin's market cap has only recently passed $1 Billion mark[1] whereas, Amazon comprises of 12-20% of online sales (depending on where you look) which makes them the 15th largest US Store.

One of the problems with Bitcoin is despite its low fees, transaction speed is _much_ slower than you're standard debit/credit card & considerably slower than the 1-click card you already have on file with Amazon. Whilst you may not consider this important, considering that Amazon is trying to get to same-day delivery (especially as eBay already do this[2]) then having to wait for the transaction to verify significantly affects shipping times & Amazon's intentions to do same-day delivery.

Likewise, going back into my earlier point about Bitcoin only recently reaching the $1B market cap[1] no one really knows exactly how much "financial bandwidth"[3] that the Bitcoin market can carry & and since Bitcoin is highly volatile, if Amazon entered the market at this point it would add even more volatility to the market which in turn makes it more difficult for vendors – who value stability. Moreover, $1 Billion isn't a significant amount of money (you can argue otherwise but look at Global GDP etc) and Bitcoin is still maturing which, means they still have to demonstrate resiliency to attacks from organized-crime/hackers, spam transactions etc as well as, the fact that it will be _high-risk both legally and financially_ for Amazon especially as, Bitcoin will have a sudden large influx in market cap (after all it is really volatile).

Additionally, Amazon are already looking at Virtual Currencies[4] which suggests that they will probably be looking at them closer over the long-term.

[1] [http://techcrunch.com/2013/03/28/bitcoin-how-an- unregulated-...](http://techcrunch.com/2013/03/28/bitcoin-how-an-unregulated- decentralized-virtual-currency-just-became-a-billion-dollar-market/)

[2] <http://now.ebay.com>

[3] <http://financial-dictionary.thefreedictionary.com/Bandwidth>

[4] [http://www.ft.com/cms/s/0/5d46a9fc-6fbc-11e2-956b-00144feab4...](http://www.ft.com/cms/s/0/5d46a9fc-6fbc-11e2-956b-00144feab49a.html)

<comment><AUTHOR> So, in your opinion the main problems are: 1. transfer speed. 2. high volatility. Is that right?

</comment></comment><comment><USER3> 1\. When USA government accepts bitcoin for taxes.

2\. When Amazon decides WoW gold is good enough too.</comment>


Decentralization and What Section 230 Means - <AUTHOR> https://www.coindesk.com/decentralization-and-what-section-230-really-means-for-freedom-of-speech <comment><AUTHOR> The only problem with Twitter doing fact-check was they were unilateral and biased. If they were neutral, probably they could to the fact-checking without problems.</comment>


SpaceX Thales Mission Launch - <AUTHOR> http://livestream.com/spacex/events/3995038

<comment><USER2> "Falcon 9 is holding due to weather"

<comment><USER2> countdown has restarted!

</comment></comment><comment><USER3> Video keeps interrupting.

<comment><USER2> seems good to me

<comment><USER3> Consider me jealous :(</comment></comment></comment>


Things you can do to really support Open Access - <AUTHOR> http://phylogenomics.blogspot.com/2013/01/10-things-you-can-do-to-really-support.html

<comment><USER2> When open knowledge systems will run programs to drill into archives of information, discover patterns and make predictions, how long until the open access initiatives will clash with the growing interest of companies to hold monopoly on those programs? That's just one of the big questions raised by this issue.

<comment><AUTHOR> I think the raw data/publications can be open and the algorithms/search can be private. Isn't google a great example of this?</comment></comment>


Intel Prepares to Graft Google's Bfloat16 onto Processors - <AUTHOR> https://www.nextplatform.com/2019/07/15/intel-prepares-to-graft-googles-bfloat16-onto-processors/ <comment><USER3> ISA: [https://software.intel.com/sites/default/files/managed/c5/15...](https://software.intel.com/sites/default/files/managed/c5/15/architecture- instruction-set-extensions-programming-reference.pdf) Look for anything marked with AVX512_BF16 CPUID feature flag.

Numerical details: [https://software.intel.com/sites/default/files/managed/40/8b...](https://software.intel.com/sites/default/files/managed/40/8b/bf16-hardware- numerics-definition-white-paper.pdf)

Support for bfloat16 is already present in MKL-DNN ([https://github.com/intel/mkl-dnn](https://github.com/intel/mkl-dnn))

Disclaimer: I work for Intel

<comment><USER4> Please take my bug report:

Dropping denormals is a huge mistake. This is easy to see if you draw out a number line for a very tiny floating-point format, for example with a 2-bit exponent and a 2-bit fraction. (do this on a sheet of graph paper) Without denormals, there is a huge gap surrounding zero.

Strangely, the infinities were kept. Treating these as NaN is far less harmful than dropping denormals. Treating -0.0 as 0.0 and never producing -0.0 would be less harmful. (the PDF didn't say what happens) Even treating NaN values as normal numbers is probably less harmful than screwing up the denormals.

IEEE floating point has lots of crazy stuff to annoy hardware vendors. Most of it isn't all that important, but denormals matter.

<comment><USER5> bfloat16 is only really intended for ML purposes, and denormals don't really matter there, especially given they accurately multiply-accumulate into 32 bit floats.

<comment><USER6> Would it hurt ML usage if they added them, would it make the implementation much harder? If not, could really increase use.

<comment><USER5> I'm not an expert in the nitty gritty, and I've heard conflicting information, but my rough understanding is that it would be affordable and not too difficult for Xeon processors, but relatively more expensive for the dedicated Nervana neural network processors.

<comment><USER4> Sure, I can see that being the case. My point is that the Xeon should support denormals.

The Bfloat16 format does allow for denormals. Intel's implementation mangles them, changing them to 0.0 on both input and output.

<comment><USER5> But then you need either new denormal-accepting instructions or, worse, a new global state bit enabling bfloat16 denormals, all to support use cases probably over two orders of magnitude less common than ML. What's the compelling reason to bother? Note that you need to support the denormal- disabled case because you'll want compatibility with Nervana.

<comment><USER4> Intel did add a global state bit. It just isn't useful because you can't modify it.

Nervana is discontinued, isn't it? Compatibility doesn't matter. It's pretty compatible anyway, as long as you aren't demanding bit-identical output.

<comment><USER5> > Intel did add a global state bit. It just isn't useful because you can't > modify it.

You mean the CPUID bit? That's free. Toggling denormals isn't.

> Nervana is discontinued, isn't it? Compatibility doesn't matter. It's pretty > compatible anyway, as long as you aren't demanding bit-identical output.

Nervana isn't discontinued according to their website[1], and bitwise compatibility does matter, certainly more than denormals do.

[1] [https://www.intel.ai/ai-at-ces/](https://www.intel.ai/ai-at-ces/)

<comment><USER4> I mean the bit to toggle denormals, not the one to identify support for the opcodes.

Denormals are far more important than bitwise compatibility. To be clear, you would still be able to load a Nervana-produced number into a processor that supports denormals, and the other way would work too. You'd just avoid mangling numbers that are near zero.

If you still think denormals don't matter, seriously do what I suggested: draw it out on graph paper. They matter.

<comment><USER5> > I mean the bit to toggle denormals, not the one to identify support for the > opcodes.

bfloat16 doesn't handle denormals, why is there a bit to toggle it? What's it called (so I can CTRL-f for it)?

> If you still think denormals don't matter, seriously do what I suggested: > draw it out on graph paper. They matter.

No, I get how denormals work, I know what you're pointing at. But ML genuinely doesn't care, neural nets don't give a damn about mathematical purity[1]. In contrast compatibility matters because ML doesn't give you any guarantee that it's not depending on the behaviour at small values, and minor differences in rounding does cause issues. For example, Leela Chess Zero had difficulties with reproducibility because different GPUs round floats differently.

[1] Fun but relevant aside: [https://openai.com/blog/nonlinear-computation-in- linear-netw...](https://openai.com/blog/nonlinear-computation-in-linear- networks/)

<comment><USER4> It's two bits, DAZ and FTZ. (seems like "denormals are zero" and "flush to zero")

Bfloat16 obviously can handle denormals. The encoding is possible. There would be no need to handle the issue if the encoding did not exist.

As hex, these would be denormal: 0x0001 to 0x0080, and 0x8001 to 0x8080. It's the same as plain old 32-bit IEEE, with half the bits lopped off.

ML is all about difficulties with reproducibility. I don't see a reason to get upset about denormals when a 3D-printed turtle can be confused with a rifle.

<comment><USER5> > It's two bits, DAZ and FTZ. (seems like "denormals are zero" and "flush to > zero")

You mean the standard ones for normal floats? You certainly wouldn't want to reuse that for bfloat16s.

> ML is all about difficulties with reproducibility. I don't see a reason to > get upset about denormals when a 3D-printed turtle can be confused with a > rifle.

These are different things, despite the similarity in terminology.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER7> I met Naveen Rao after Intel bought Nervana. He seemed pretty adamant about getting stuff shipped fast. In contrast, the Xeon folks own all the politics and seem to want the transition to be very gradual. Plus the Phi folks get phased out. They had done a Nervana trial at Facebook but then flaked on other trials. Clearly Intel is trying to desperately manage their books.

Having Nervana and friends on a Xeon chip could be a huge positive change for software. Not only could we toss out the issue of GPU memory transfer, but Nvidia GPUs aren't so great with concurrency, and here with the linux kernel we might have a chance to beat Nvidia. Naveen sure would like that... Nervana once had a Maxwell compiler that was better than Nvidia's.

<comment><USER8> They had an assembler where one person wrote kernels that were faster than cublas in a lot of cases. Afaik, nobody ever released anything else with that assembler, and Nvidia caught up to that performance quickly. In talking with the cublas devs, it seemed more that maxas kernels were highly tuned for specific sizes, whereas cublas/cudnn had to be more general.

Nowadays it's really a moot point with Nvidia's Cutlass being open source.

<comment><USER7> True story-- the Nervana Maxwell stuff didn't go very far-- but it was noteworthy because they had both that small win as well as their own hardware platform.

One other thought about the Nervana-Xeon convergence is that the support for more memory (thru DDR, Optane, or even just mmap'ed NVME) will be a big win for modeling and large-minibatch SGD. For example, the minibatch fetching could be pushed to the hardware / OS instead of Tensorflow (or the crazy guy behind Tensorpack) using a threadpool. A lot of training is still I/O bound at some level, and processors only support so many PCI-e lanes...

<comment><USER9> A V100 GPU gets 900GB/s of memory bandwidth. I am less of a CPU expert but afaict you'll be lucky to get much more than 10% of that out of a CPU. This is going to make a huge difference that Intel can't make up with bigger execution units.

bfloat helps with this because the data is half as large. But of course if you're doing Nvidia you're probably already doing (IEEE) fp16.

<comment><USER10> That sounded a bit low. This

[https://en.wikichip.org/wiki/intel/microarchitectures/cooper...](https://en.wikichip.org/wiki/intel/microarchitectures/cooper_lake)

says "Higher bandwidth (174.84 GiB/s, up from 119.209 GiB/s)"

I don't know if memory bandwidth matters for this type of job, though.

<comment><USER11> Is that intel ARK published spec sheet bandwidth, or actual usable bandwidth? There is a difference.

I've found I get about 75-80% of the advertised bandwidth both from my real app (TLS crypto) and a toy memory copy benchmark using AVX256 instructions. The toy memory copy benchmark is how I realized that my bottleneck was actually memory bandwidth and not CPU horsepower on Broadwell based servers.

<comment><USER10> I haven't a clue. I just got it off the link quoted. It's a good question and when there's a difference, you know what marketing will say.

To make a stab, I suppose it might depend on whether all requests are coming from a single memory bank or spread evenly across all memory banks, assuming fully populated (again from the link "Octa-channel (up from hexa-channel)")

</comment></comment></comment></comment></comment></comment></comment><comment><USER12> > At this point, Intel doesn't have bfloat16 implemented in any of its > processors, so they used current AVX512 vector hardware present in its > existing processor to emulate the format and the requisite operations. > According to the researchers, this resulted in "only a very slight > performance tax."

Why implement bfloat if you get just slightly less performance emulating it with AVX512, which already exists? Maybe it's an "us too" claim?

<comment><USER13> They did not specify what the tax was relative to. Maybe they meant relative to 32-bit float?

AVX512 is expensive. I believe if you have an AVX512-heavy workload it can cause the processor to throttle.

<comment><USER14> Yes, there's a bios setting to control this. It basically under clocks the core while AVX units are under load.

<comment><USER15> Taking my 7980xe as an example: When it runs non-avx512 loads, I currently have it set to run at 4.1 GHz (all-core). When running avx-512 heavy loads, it instead runs at 3.6 GHz -- and tends to get much hotter (70-80C instead of 50-60C). 3.6 GHz is a mild overclock; Silicon Lottery reports 100% can achieve that speed for avx512 loads.[1]

Running programs doing the same thing (eg, Hamiltonian Monte Carlo where the likelihood function has or has not been vectorized), the avx512 version is far faster than scalar, and routinely 50%+ faster than avx2.

The avx512 instruction set itself also provides conveniences that make it easier to explicitly vectorize, even if most compilers don't take advantage of them on their own. Masking load and store operations in particular (they're better about masking to handle branches).

On why avx512 vs a graphics card: I need double precision, and my code routinely has maximum widths smaller than the 32 or 64 a graphics card would want to computer in parallel.

[1] [https://siliconlottery.com/pages/statistics](https://siliconlottery.com/pages/statistics)

<comment><USER16> Yeah, people tend to completely exaggerate the impact of throttling from AVX512. It's only an issue when you do short bursts of AVX512 and the rest is not AVX512. If you do math and your math can be done in AVX512, even with throttling it's going to be substantially faster. That it runs hotter doesn't concern me at all. Intel's claimed safe Tjunction is something like 105C. EEs tend to take the published component specifications seriously (e.g. your 1000v diode is guaranteed to withstand at least 1KV of reverse voltage), so I trust Intel when they say things are fine up to that temperature. Even beyond that it won't burn out, it'll just thermal throttle.

<comment><USER17> Maximum Tjunction for an STM32F303 (just happened to have datasheet open) is 150C, as is most other ICs I've seen.

So is 105C just a very conservative number, compensating for the probe location, or are there process specific things which brings it down to 105C?

<comment><USER18> From what I understand, the newer very-high-density procsses are far more sensitive to voltage and temperature than the older larger ones.

<comment><USER17> Makes sense. The STM32G series, which still has 150C Tjmax, is ST's first 90nm MCU[1] so yeah.

[1]: [https://blog.st.com/stm32g0-mainstream-90-nm- mcu/](https://blog.st.com/stm32g0-mainstream-90-nm-mcu/)

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER19> does anyone know what is the story on the software side - CUDA is basically industry standard now.

Tensorflow OpenCL support bug [1] has been open for FOUR years now (with the discussion devolving into an Intel PlaidML flame war).

AMD OpenCL is now ROCm ?

At the end of the day, I cant run ANY accelerated workloads using Intel graphics or AMD ....because there's simply no software support anywhere.

OTOH, if you have a nVidia stack... boom. you get accelerated python [https://developer.nvidia.com/how-to-cuda- python](https://developer.nvidia.com/how-to-cuda-python)

Are you running containerized workloads on kubernetes ? it has BAKED-IN support for nvidia-docker ([https://kubernetes.io/docs/tasks/manage- gpus/scheduling-gpus...](https://kubernetes.io/docs/tasks/manage- gpus/scheduling-gpus/))

Is anything going to change ?

[1] [https://github.com/tensorflow/tensorflow/issues/22](https://github.com/tensorflow/tensorflow/issues/22)

<comment><USER20> Huh, that's strange. I remember in the infancy of cryptocurrency mining (back before specialized ASIC hardware), OpenCL was far superior to CUDA, and AMD cards were doing 10X the hashrate for the price as OpenCL cards. What changed in the interim? Is SHA256^2 just a completely different workload than Tensorflow, or has Nvidia pulled ahead?

<comment><USER21> It's a different workload. Afaik, AMD cards are still better hash-per-dollar for the coins that are gpu-mineable.

</comment></comment></comment><comment><USER22> This is super exciting! Brings a bit of competition to NVIDIA for ML-related tasks, while being more "open" (to some extent) than the TPU ASICs (because you won't have single-cloud lock-in). In any case, good to see Intel finally waking up.

<comment><USER23> Is Intel planning TPU's or GPU's?

I don't see how GPU with AVX512 can compete with TPU's or GPU's, BF16 or not.

</comment></comment><comment><USER24> I wonder why they chose it over facebook's 8 bit posit: [https://code.fb.com/ai-research/floating-point-math/](https://code.fb.com/ai- research/floating-point-math/)

<comment><USER25> What does the "int8/32" mean in that paper?

<comment><USER26> 4x8-bit values in a 32-bit value? SIMD does this to perform an operation on 4 8-bit values in a single 32-bit register. There are other configurations, IIRC.

[https://en.wikipedia.org/wiki/SIMD#Software](https://en.wikipedia.org/wiki/SIMD#Software)

</comment></comment></comment><comment><USER27> They're not really "grafting" it, they're implementing it as a first-class data type.

<comment><USER28> Definitely feels click-bait-y.

Ditto for them calling it "Brain Floating Point" [1]. I mean it appears to be a typical floating-point numeric data type with some of the precision truncated to reduce its cost.

I guess they might be trying to make it sound super-fancy to preempt people seeing it as cheap or/and low-tech?

[1]: [https://en.wikipedia.org/wiki/Bfloat16_floating- point_format](https://en.wikipedia.org/wiki/Bfloat16_floating-point_format)

<comment><USER29> It's called brain floating point because it was developed by Google's "Google Brain" machine learning project.

<comment><USER28> Yeah, I get the marketing perspective; just, we typically describe primitive data types in terms of what they are from a technical perspective rather than a marketing perspective.

<comment><USER30> fp16 & fp8 refer to IEEE 754 floating point which has notable differences from bfloat16 & make them have worse performance for machine learning.

<comment><USER31> Yeah the point is 'reduced precision FP16' would make a lot more sense - <USER2> it actually describes what it does. 'Brain' floating point makes you wonder how Neuralink are planning to use this datatype.

</comment></comment></comment></comment></comment></comment><comment><USER10> Newbie question: what is the typical and extreme values (excluding +/-infinity, and are these used too?) that can occur in training/running of NNs? Also, what level of accuracy is needed?

It may well be a stupid q but I really don't know, and always assumed they would be [-1..+1] and that fixed point would suffice. Clearly not.

<comment><USER32> Nope, no standardization at that level.

</comment></comment><comment><USER33> 7 bits of precision.... or 2.1 decimal digits.

So not quite as good as a 6-inch slide rule.

Edit: voters in this thread seem pretty uptight.

<comment><USER34> What would be the wattage of a typical six inch slide performing calculations as quickly as a top-of-the-line Intel microchip? Or if it would be physically impossible because of speed of light considerations, what would be the wattage of n slide rules in parallel performing such computations such that it adds up to the throughput of a microchip?

Speaking of boiling the ocean . . .

</comment></comment><comment><USER35> I am sorry for Intel. Perhaps John Gustafson's 16 bit posits or unums would have made a better choice.

<comment><USER36> Why? Google has certainly researched their floats before committing an entire line of silicon chips. It's easy to just enumerate all possible float16 configurations in a simulator to see which one performs best on a wide range of neural network applications. Then pick the best one. Big data driven organizations do this all the time (brute force through an entire line of solutions, pick best results).

<comment><USER22> I know nothing about ASIC or CPU simulators but I suspect that it's not as easy as you make it sound: for machine-learning related tasks, performance doesn't only come from raw compute numbers: you'll also want to model the actual data movement costs across the caches hierarchy and registers. Because a lot of time training is not necessarily compute-bound: the relative cost of data transfer (VS compute) can be quite high, or even dominate.

<comment><USER37> fp16 and bfloat16 are the same size, so there's no difference in data transfer rates.

</comment></comment></comment></comment><comment><USER38> That's surprising, I wouldn't guess it is something they would do so easily, especially considering Intel processors isn't something you generally use to train NNs.

But I'd be rather glad if they implemented unums already.

<comment><USER32> I absolutely train on Intel from time to time at work. If your model fits, cpu is a dream. The driver never breaks, they never crash and lock up your display...

</comment></comment><comment><USER39> It looks somewhat similar to the minimal size of compact float [1] (a format I developed for data communication), but with 2 more bits because I use them for sizing.

[1] [https://github.com/kstenerud/compact- float/blob/master/compa...](https://github.com/kstenerud/compact- float/blob/master/compact-float-specification.md)

</comment><comment><USER40> Does bfloat16 have any other uses than deep learning?

<comment><USER41> It isn't really very good for machine learning - machine learning doesn't need 8 bits of exponent.

For weights during training, 7 bits of mantissa also seems a bit low - it's common for weights to adjust much less than 1% during a single batch of training, which this couldn't represent.

I think this is more a "we want something which is faster but is compatible with existing code written for fp32's".

<comment><USER9> Bfloat is specifically designed for ML. It is the native type in Google's TPUs. It is quite good at ML; most models that work with fp32 work with bfloat with no adjustments; that's in contrast to IEEE fp16.

You're right that the mantissa is small. The trick is that you always accumulate into fp32 and then truncate down to 16 bits at the end. You'd do this for any 16-bit floating type.

Source: I work on this at Google.

[https://cloud.google.com/tpu/docs/bfloat16](https://cloud.google.com/tpu/docs/bfloat16)

<comment><USER42> Have you evaluated alternatives such as posits, Kulisch accumulation, and zfp? [https://arxiv.org/pdf/1805.08624.pdf](https://arxiv.org/pdf/1805.08624.pdf) [https://arxiv.org/pdf/1811.01721.pdf](https://arxiv.org/pdf/1811.01721.pdf) [https://insidehpc.com/2018/05/universal-coding-reals- alterna...](https://insidehpc.com/2018/05/universal-coding-reals-alternatives- ieee-floating-point/)

In particular the latter describes a generic framework that can be used to generate a lot of different number systems. Could hardware implement this, allowing us to compose and choose the number system by just setting some simple flags?

</comment></comment></comment></comment><comment><USER43> Is there a principled way to not overfit? Or is the best we can do not overtraining or reducing the precision of the calculations?

<comment><USER23> regularization, dropout, early stopping etc.

</comment></comment><comment><USER16> Why "graft"? It's not something that's foreign to them. This promises to essentially double the performance of Intel chips on an increasingly important workload, and also simplify the modeling work, because the models don't experience any accuracy drop when simply converted to bfloat16, unlike with quantization, where it's model dependent and finicky AF. I'd much rather do fp16 or bfloat16 at inference time, without constant pain that is quantization. I hope ARM, AMD and RISCV pay attention and implement this in the exact same way, so that models could be portable.

<comment><USER29> I read this article as saying, "hey we can emulate bfloat16 pretty well in software on top of our existing hardware features". That's what "graft" and "minimal impact" mean to me.

Intel (for better or worse) takes a very experiment-results-driven approach to choosing which features to implement in hardware. So this result -- that software emulation of a feature works almost as well as a hardware implementation would -- probably makes Intel less likely to implement the feature in hardware.

ARM, AMD, RISCV etc, will probably come to similar conclusions.

<comment><USER25> The article links to another saying that bfloat16 is coming to Xeons. Also the title of the current says it too. There's nothing implying less likely if you read the articles.

[https://www.nextplatform.com/2018/12/16/intel-unfolds- roadma...](https://www.nextplatform.com/2018/12/16/intel-unfolds-roadmaps-for- future-cpus-and-gpus/)

<comment><USER29> Ouch, you're right! I misunderstood.

</comment></comment></comment></comment><comment><USER44> Could we get a Bfloat32?

<comment><USER45> You already have that; it's more formally called IEEE 754 single precision. Requires double the memory bandwidth (bad) for extra precision that back propagation would have corrected for anyways.

<comment><USER46> Seems like BFloat32 would have the same exponent range as Float64 (double precision), and a correspondingly shorter significand.

</comment></comment></comment><comment><USER47> Ohh intel is awesome

</comment><comment><USER48> I desperately want Chinese companies finally begin producing and designing general purpose CPUs, GPUS, and other types of accelerators. Current situation is terrible duo- and mono-polies, slow pace of innovations, and low reliability. We need more players.

<comment><USER49> Finally Bloomberg will be able to report on real hardware backdoors!

</comment></comment><comment><USER50> Something about Google being able to influence features in consumer grade CPUs rubs me the wrong way.

<comment> m-p-3 Not surprising though, new features shows up where there's money to make.

</comment></comment><comment><USER51> English is not my first language. I have never heard the term "Graft", even if I consider myself quite literate in English. So here you go, for everybody else in my situation:

Graft, as understood in American English, is a form of political corruption, being the unscrupulous use of a politician's authority for personal gain.

Edit: by the way, I really couldn't fit the term with the article. And realized I was probably looking at the wrong definition. This one might be much more apt:

a shoot or twig inserted into a slit on the trunk or stem of a living plant, from which it receives sap.

<comment><USER52> That definition definitely does not apply to this usage. You are looking for "to join (one thing) to another as if by grafting, so as to bring about a close union." (etymology 1, verb, definition 4 on Wiktionary [1]).

[1] [https://en.m.wiktionary.org/wiki/graft](https://en.m.wiktionary.org/wiki/graft)</comment></comment>


Have a Flash SWF File? convert it to HTML5 with Google Swiffy - <AUTHOR> http://www.google.com/doubleclick/studio/swiffy/ <comment><USER2> If you're thinking about using this, you might also consider Gordon (<https://github.com/tobeytailor/gordon>) or Smokescreen (<https://github.com/cesmoak/smokescreen>). Gordon's simpler -- very easy to follow code. Whereas Smokescreen, like Swiffy, offers support for audio and AS2. I integrated Gordon into my HTML5 game yesterday to render a walk cycle SWF and the framerate didn't drop. I'm going to do the same thing with Smokescreen next.

</comment><comment><USER3> Swiffy saved the day for us recently on a project where we needed an animation to support IE7+ as well as iPad. We were able to have our designer create a flash animation and upload to swiffy to test that the conversion worked. We then used the swf as a fallback for when canvas was not available.

The one HUGE drawback for me with swiffy is there was no standalone compiler that we could use and integrate into the site build. So we now have this manual step of converting the files on the (hopefully still functioning) swiffy website every time a change is required to the animation.

<comment><USER4> I guess it shouldn't be too hard to write a tool which automates that. You could probably even just use curl. I also trust that Google has enough interest in HTML5 adoption that they will make sure that this tool will continue to function.

</comment></comment><comment><USER5> Moderately complex intro animation from an old e-learning course I did converted successfully (failed on earlier versions of this tool so obviously they've been improving things).

No sound, not much scripting but lots of timeline animation with masked images.

swf: 205kb html: 352kb

(not including the swiffy runtime.js)

Not bad considering all the images have to be base64 encoded.

Generated code looked... generated... But not too awful. You could probably do something with it if you needed to.

However - it's no worse than trying to modify a swf file. You just wouldn't unless you ran it through a decompiler first and I don't see why that wouldn't be possible here also.

In most cases you'd go back to the FLA file or the XML equivalent and modify that instead.

<comment><USER4> Are those sizes before or after gzip compression ?

<comment><USER5> Before. A quick test by just zipping them up gave:

html: 209kb swf: 205kb

Does anyone know whether 'on the wire' gzip would give similar benefits?

<comment><USER6> Two notes:

1\. Zip uses a different algorithm than gzip - the results are not necessarily equivalent.

2\. If the HTML is static, which seems to be the case (since it's generated manually from the SWF), you can pre-gzip them instead of forcing the server to do it on demand - that'll reduce latency and CPU usage. Therefore "on the wire" is not really relevant.

<comment><USER7> Zip and gzip both use DEFLATE.

</comment></comment></comment></comment></comment><comment> guard-of-terra Tried to recode random Масяня cartoon, it wouldn't play and would not support audio even if it would.

</comment><comment><USER8> Literally yesterday I had conversation about the coming wave of flash to HTML5 conversion tools and services. Expect to see alot of effort going into conversion automation. Yes there will always be a need for hand crafted conversion but a lot can and will be automated.

<comment><USER9> > flash to HTML5 conversion tools and services

Which is a horrible thing. Images there are like 20 canvas elements on a page drawing animated ads. I am sure Macbook fans would roar higher than Flash

</comment></comment><comment><USER10> Machine generated HTML5.. hmm something in the back of my head tells me if you file isn't massive, its worth doing it yourself because it might produce bloated code. i.e it will be kind of difficult to work on that code later yourself.

</comment><comment><USER11> Swiffy can't handle files input SWF files larger than 1024 MB and will display an error to that effect if you try. It might be more user-friendly if that fact were displayed on the file upload page. Just a suggestion!

<comment><USER12> Larger than a gigabyte? I should hope that you don't have too many Flash files that big...

<comment><USER11> Hehe... Thanks for catching my error. That should have read: "1 MB" or "1024 KB".

</comment></comment></comment><comment><USER13> Too bad this doesn't work in all browsers yet.

</comment><comment><USER14> Now if they could just port playerglobal 11 to Javascript, we can run Molehill games in Canvas. Think how happy everyone will be to get that CPU-intensive Flash plugin out of their browser, when they find that now every website chews up 400% more processing power. A cynic might think they were promoting Javascript over AS3 so everyone would have to buy a round of newer, faster phones once these SVG animations start hitting the web.

<comment><USER15> You already have tools that do that, like <http://jangaroo.net>

Just for large project switching to an interpreted language like javascript with another layer of abstraction to emulate the flash API is too much of a performance hit.</comment></comment>


Nginx Patch: SPDY/3.1 protocol implementation - <AUTHOR> http://mailman.nginx.org/pipermail/nginx-devel/2014-January/004890.html

<comment><USER2> With help from some companies:

> Three companies that like Google's approach -- Automattic, MaxCDN, and > CloudFlare -- are funding Nginx developers to update its SPDY support to > version 3.1, CNET has learned. Under the deal, SPDY 3.1 should arrive in > Nginx 1.5 in January, a source familiar with the partnership said.

[http://news.cnet.com/8301-1023_3-57616193-93/nginx- upgrade-f...](http://news.cnet.com/8301-1023_3-57616193-93/nginx-upgrade- funded-by-fans-of-googles-spdy-web-protocol/)

<comment><USER3> Just out of curiosity, are they paying Nginx Inc or Nginx contributors?

<comment><USER4> The developer who announced this works for Nginx the company, so I would assume (emphasis on the assume!) that Nginx Inc. was paid to undertake this.

I don't think this is a problem given that the patch is being openly pushed into the "public" source immediately. It'd also be extremely challenging to pay "the community" for a patch like this (how do you divvy up the funds? per SLOC?), although technically you could always directly pay a single contributor or two.

Directly engaging Nginx Inc seems to be both the surest way to get something delivered _and_ avoid a lot of PR risk, if you ask me.

<comment><USER5> Normally if a company wants to take that route, they would hire one of the developers as an employee, and the employee's job would be to work on that open source project.

</comment></comment></comment></comment><comment><USER6> Good initiative! Just a heads up that server push does not seem to be implemented.

Server push is one of the spec items that got much wanted clarifications in SPDY3 and would be awesome to see nginx support that. Apache mod_spdy, jetty and netty do have it.

Server push makes it handy to provide super low latencies for full page loading by pushing out all the files that "server thinks" browser might need for a page without waiting for the requests.

Server push is not a magic bullet, it can take more bandwidth compared to the regular visit with a browser that has required resources (images, css, js etc) cached. But still better than inlining in html because the resources from server push can be cached.

If you care more about page loading speed (especially for first-time visitors) and less about actual bandwidth consumed, SPDY with server push can be great. Hint: deciding on whether visitor might benefit from "a little push" as a fresh visitor could be done with a cookie existence check or smth similar.

<comment><USER7> Last year, I had spent about a week in benchmarking SPDY. Server push didn't show any improvements to page load time for even first-time visitors. The whole benchmark was scripted with Chrome running in a dummy X server. The round-trip time was artificially constrained to about ~100ms.

Even if it were to show an improvement in some situations (different round trip times or network speeds), it would be in the microseconds to couple of milliseconds range; with a large variance.

Given that it degrades performance for second-time visitors, I would recommend not enabling it without further benchmarking.

<comment><USER8> We have a lot to learn on how to use server push effectively. That said, let's analyze some actual use case..

a) Page A currently inlines half dozen small assets on every page. These inlined resources inflate every page and are delivered at same priority as HTML. By contrast, a "dumb push" server delivers these same assets as individual resources via push. Net benefit? Basically same performance since inlining is a form of application-layer push. However, a smart server can at least prioritize and multiplex push bytes in a smarter way... Now, let's make the server just a _tiny_ bit smarter. If it's the same TCP connection and the server has already pushed the resource, don't push it on next request. Now we're also saving bytes... <insert own smarter strategy here>.

b) Page B has two CSS and one JS file in the critical rendering path. Without push you either inline all three (ouch), or, you roundtrip to the client and get it to parse the HTML to discover these resources and come back to the server... With push, the server can avoid the extra RTT and push those assets directly to the client -- this is a _huge_ performance win. How does the server know? Well, either you made it smart.. or you use some adaptive strategy like looking at past referrers and building a map of "when client requests X, they also come back for Y and Z" \- this is already available in Jetty.

The fears of extra bytes are also somewhat exaggerated (they're valid, but exaggerated). The client, if they desire, can disable push. The client can also use flow-control to throttle how much data can be transferred in first push window (both FF and Chrome do this already). Lastly, the client can always decline and/or RST a push resource.

Some additional resources: \- [http://chimera.labs.oreilly.com/books/1230000000545/ch12.htm...](http://chimera.labs.oreilly.com/books/1230000000545/ch12.html#HTTP2_PUSH) \- [http://www.igvita.com/2013/06/12/innovating-with- http-2.0-se...](http://www.igvita.com/2013/06/12/innovating-with- http-2.0-server-push/)

<comment><USER7> These ideas sound fine on paper, but for first time visitors (on a warmed up browser), server push didn't show any improvements to page load times in my benchmarks. My question is why enable server push at all?

I am no longer with the company for whom I performed the benchmarks, that's why I can't publish them. Perhaps there are other benchmarks out there, that show server push is more performant. If so, I would be happy to see them.

<comment><USER8> I don't know the details of your benchmark methodology or setup (server / client), so not sure I can offer a meaningful response... short of: let's not confuse "my benchmark failed" with "the future doesn't work". Anything from a poorly implemented server (broken multiplexing, flow control, prioritization, etc), to bugs in past versions of Chrome...

Jetty guys had a couple of nice demos they showed off at various conferences. Here's one: [http://www.youtube.com/watch?v=4Ai_rrhM8gA](http://www.youtube.com/watch?v=4Ai_rrhM8gA)

</comment></comment></comment></comment></comment><comment><USER9> I've been having issues compiling the last 2 versions of nginx. Anyone else seeing this?

      make -f objs/Makefile
      make[1]: Entering directory `/home/newman314/src/nginx-1.5.9'
      cc -c -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g -march=native -Ofast -fomit-frame-pointer -fstack-protector -D_FORTIFY_SOURCE=2 -I src/core -I src/event -I src/event/modules -I src/os/unix -I objs \
    		-o objs/src/core/ngx_string.o \
    		src/core/ngx_string.c
      {standard input}: Assembler messages:
      {standard input}:1253: Error: no such instruction: `vfmadd312sd .LC5(%rip),%xmm1,%xmm0'
      make[1]: *** [objs/src/core/ngx_string.o] Error 1

Configure flags used: ./configure --with-http_ssl_module --with- http_spdy_module --with-http_gzip_static_module --with-cc-opt='-march=native -Ofast -fomit-frame-pointer -fstack-protector -D_FORTIFY_SOURCE=2'

<comment><USER10> I cloned from [http://hg.nginx.org/nginx](http://hg.nginx.org/nginx) to try and reproduce with your configure arguments, but i actually error out elsewhere;

        src/http/ngx_http_request.c: In function 'ngx_http_set_virtual_server':
        src/http/ngx_http_request.c:1955:32: error: 'cscf' may be used uninitialized in this function [-Werror=maybe-uninitialized]
        cc1: all warnings being treated as errors

which i'm not interested in following up. My `ngx_string.o` did build successfully though. Using gcc 4.7.2 on Debian 7

Are you using a new CPU with an old(er) gcc version? Try specifying -march explicitly instead of native.

<comment><USER9> Doh. I recently moved this VM from a C2D to an i5 machine. And this is on Ubuntu 12.04 which is old enough not to know about this newer CPU. Still, it should not crap out like that...

<comment><USER9> Just as a follow up, I tried this with no -march flag and it compiled just fine.

This is with gcc 4.6.3.

</comment></comment></comment></comment><comment><USER11> Thats great. I have been waiting for this since Firefox and Chrome stated that spdy will no longer be supported on next versions.

I hope that nginx will maintain spdy2 and serve version 3 to compatible clients. My company just upgraded to Firefox 14...

<comment><USER12> Firefox 14? If that's correct, does your company understand it's vulnerable to numerous security vulnerabilities by being so far behind (especially ones that could remotely compromise your entire system)?

If they need to better manage Firefox updates, I recommend they look at swapping to Firefox ESR (Extended Support Release). See [https://www.mozilla.org/en- US/firefox/organizations/](https://www.mozilla.org/en- US/firefox/organizations/) for more information.

</comment></comment><comment><USER13> Is this connected with Dropbox testing SPDY [1], or they're on HN front page by chance? Just curious...

[1] [https://news.ycombinator.com/item?id=7132202](https://news.ycombinator.com/item?id=7132202)

<comment><USER8> Seems coincidental. Just highlights that we need HTTP/2, and the sooner the better... Inventing your own app-layer multiplexing on top of HTTP (via base64 chunks - ugh), is not an interesting problem to be solving in 2014.

</comment></comment><comment><USER14> is QUIC[0] going to be supported anytime soon?

[0] [http://en.wikipedia.org/wiki/QUIC](http://en.wikipedia.org/wiki/QUIC)

</comment><comment><USER15> Version 3.1? OK.

So HTTP 1.1 lasted for 20 years without significant changes, but SPDY which has been around for a year or so and is already on version 3, with extra point-releases for shit and giggles.

If you're going to create a protocol and want it to be used internet-wide can you at least plan up front and keep things stable once released?

Glad I don't have to maintain and deploy any code related to this. Jesus christ.

<comment><USER16> A version number is just an arbitrary point in time, in the case of SPDY, with a certain feature set. Just because SPDY/2 doesn't support everything SPDY/3 supports doesn't mean it's unstable.

This is why I like date-based version numbering, such as Ubuntu 13.10 or C++11.

<comment><USER17> But date-based numbering provides no information other than the date the code was released on. Semver, on the other hand, conveys much more.</comment></comment></comment>


Ask HN: Need help with websocket / redis  and system design? - <AUTHOR> When user connects to server via websocket. Socket connections are stateful, the server it connected with, is the only server that would be able to communicate over that socket.
I'm designing a chat. There are n servers, each server containing n users/sockets connected to it.
Chat has a feature, connect to random, that connects you to a random user chosen from across all servers.
Currently using PUB/SUB via Redis to communicate among servers.
- Is Redis PUB/SUB the best way to handle communication between servers when most of the communication can be between two servers not 1 to N.
user x is in Server A, user y is in Server B. All N servers are subscribed to 'message' channel. x wants to send a message to y. Server A publishes on 'message' channel. All N servers are lit, when only Server B needed it. Can I keep track which server the partner of x, the y, is connected to using Redis pub/sub and only send to that server.. Broadcast vs Emit?
- How do I design protocol of partnering up users? First to describe the problem...
A -> Req (need partner for user x)
B <- Req
C <- Req
B -> Offer (user y available for user x)
C -X (no one available, doesn't do anything)
A <- Offer (process the offer)
C <- Offer (doesn't have user x, doesn't do anything)
A -> Accept (assigns user x's partner to be user y and let other server know)
B (server crashed no longer exists, problem)
How would, Server A, know that, Server B, with y has crashed / or y has left. And it should continue to search for online user. Right now it's just sitting, thinking that it has paired its user x with another live user. <comment><USER2> I suggest you develop with such a tool instead of using redis [https://github.com/ClusterWS/ClusterWS/blob/next/.github/REA...](https://github.com/ClusterWS/ClusterWS/blob/next/.github/README.md).

<comment><AUTHOR> I checked it out, i think it's just a library for websocket. I find native functionality to be good enough on browser side, and on server I'm just using ws module.</comment></comment>


Tracing Summit 2017 Videos - <AUTHOR> https://www.youtube.com/watch?v=m8gig4BBGP0&index=1&list=PLuo4E47p5_7bfeZyYIyNYM-f-2tmr0neu <comment><AUTHOR> Complete schedule with abstracts and biographies available here: [http://tracingsummit.org/wiki/TracingSummit2017#Schedule](http://tracingsummit.org/wiki/TracingSummit2017#Schedule)</comment>


Ask HN: What do you think of my startup idea? - <AUTHOR>

I will keep this short and sweet. Basically Mint for businesses that will pull data from Quickbooks/Freshbooks/etc...<p>Would you use it? To what extent? <comment><USER2> Uhm:

<http://news.ycombinator.com/item?id=1482178>

<comment><AUTHOR> I've seen this however this doesn't take advantage of all of the work that is done by the business owner in categorizing/itemizing inside of Quickbooks for instance. This is simply scrapping your CC and Banking account information. If this could pull from the ledger do you not agree it would provide a lot more value? I already have my business accounts in Mint, not sure what inDinero is really doing to differentiate itself.

<comment><USER3> I am playing with InDinero and it is frustrating that it is not tied to Quickbooks. This is a deal-breaker since I already have to have Quickbooks for my accountant.</comment></comment></comment>


We Changed YugaByte DB Licensing to Apache 2.0 - <AUTHOR> https://blog.yugabyte.com/why-we-changed-yugabyte-db-licensing-to-100-open-source/ <comment><USER3> >However, the reality is that CockroachDB is not yet at the levels of adoption where AWS would be interested. So why did Cockroach Labs make the change?

Because it's too late once they get to that stage. The licence needs to be changed ahead of time. Because companies generally try to plan ahead more than a few months into the future.

I was following along with you until this point, but now I feel like you're trying to hard to paint these companies in a negative light and not give any benefit of the doubt.

I use CockroachDB and I have looked into your database recently. If I recall, your database dashboard was in the enterprise version only. I may look at it again now. I'll try to give you the benefit of the doubt (that you're not intentionally misrepresenting your competitors) even though you don't seem to be so charitable with your competitors.

Edit: Looks like dashboard is still proprietary. Also FYI, this page is unusable on my Pixel 2XL:

[https://www.yugabyte.com/platform/#ee-1](https://www.yugabyte.com/platform/#ee-1)

<comment><USER4> Hi @jazoom,

Thanks for your comment... had a couple of clarifications. Our take on the open-source licensing of CockroachDB/MongoDB has no implication on the features of these products. So in essence, you raised two separate points (one about the licensing, the other about the dashboard/features), happy to address both.

> Question about the dashboard

The portion of the dashboard that is not related to the orchestration is going to be open-source. This is a work in progress (it needs engineering work to separate these, and we're almost there). In fact, would love to have you beta test this if you're interested - please join our community slack and holler at me.

Note that we already had the core-DB open source - now we are making previously enterprise-only features open (like encryption at rest/on wire, distributed backups and read replicas). For comparison, CockroachDB (and MongoDB) had enterprise features - which were closed (and remain so). They made changes to the core-DB as well to prevent competition, effectively making it non-open-source (if you go by the definition of open-source).

Note that we're keeping "managed service" portion called _YugaByte Platform_ under a closed license - this is the part that "manages" the cloud experience like creating nodes automatically, configuring security groups, etc. Not sure if these other companies have an equivalent product to this.

>> However, the reality is that CockroachDB is not yet at the levels of adoption where AWS would be interested. So why did Cockroach Labs make the change?

> Because it's too late once they get to that stage. The licence needs to be > changed ahead of time. Because companies generally try to plan ahead more > than a few months into the future.

> I was following along with you until this point, but now I feel like you're > trying to hard to paint these companies in a negative light and not give any > benefit of the doubt.

We named changes done by 4 companies (Elastic, Confluent, MongoDB, CockroachDB). Two of these are positive changes (Elastic and Confluent made only enterprise features closed, while the core is still open). MongoDB and CockroachDB closed their core as well with restrictive licenses. So we're simply calling that out.

MongoDB is being offered as a service by Azure CosmosDB _and_ AWS. The licensing change done by MongoDB did not deter Amazon. Similarly, the enterprise features in Elastic were rebuilt by AWS and open-sourced into a fork. So the lesson here is - if a cloud provider wants to, they will build it anyway. Additionally, in the case of CockroachDB and YugaByte DB - AWS has Aurora, which is over a billion dollars in _yearly revenue_ offering the same API as Postgres, so no net new functionality here. The few extra features offered by both these products can easily be replicated by AWS given the above. AWS is in fact more likely to build on Aurora rather than taking any of these services. So in the light of this, the above point of view seems fair.

PS: Will share internally about the link not working, thanks for pointing out!

</comment></comment><comment><USER5> We currently use CockroachDB in production and this is definitely going to make us take a serious look at YugaByte. We love Cockroach, but not really being able to do a distributed backup / restore operation without paying a million dollars is really annoying.

</comment><comment><USER6> So are they not worried AWS will offer yugabyte as a service and edge them out of their own market? Maybe AWS wouldn't because of Aurora. And I guess the other's aren't large enough to matter.

<comment><USER7> One way for YB to fight off the competition from AWS would be to include a clause that any forks of the DB should always be in sync with the upstream(master developed by YB), so hypothetically speaking this would mean companies that are using the AWS YB could always have the freedom to switch to using the OSS YB and only risk loosing the features that AWS has contributed to its fork and might actually force the OSS to keep up.

So goodluck YB.

<comment><USER4> (founder/cto of YugaByte)

This is a very insightful suggestion, thanks for raising that! We had considered many of these variants until finally, we concluded that fully open is the best way.

PostgreSQL (which is the database on fire right now) got to this spot by being fully open and permissive - and embracing all forms of competition. In fact, PostgreSQL got rewritten from Lisp to C (which begs an interesting question - what _is_ a database? The code or the query language? Anyway I digress).

We felt if we want to build something as foundational as PostgreSQL for the cloud, then we need to be as open.

<comment><USER8> Can you explain more about the rewrite from Lisp?

<comment><USER9> Postgres was first written in Lisp and had to be re-written in C because of slow performance [1]. YugaByte DB reuses the same C-based Postgres query layer but runs it on a Google Spanner-inspired distributed document store written in C++. The goal of this new "rewrite" is to add horizontal write scaling, native failover/repair and geo-distribution. As long as users love the language they interact with, the underlying software will continue to see such re-writes in order to meet the needs of the current times.

[1] [https://thenewstack.io/the-slow-climb-of-postgres-and-the- va...](https://thenewstack.io/the-slow-climb-of-postgres-and-the-value-of- persistence/)

</comment></comment></comment></comment></comment><comment><USER10> Great !

[https://blog.yugabyte.com/rise-of-globally-distributed- sql-d...](https://blog.yugabyte.com/rise-of-globally-distributed-sql- databases-redefining-transactional-stores-for-cloud-native-era/)

I have found this article on their site to be useful. The article compares different SQL databases under Monolithic SQL Vs NewSQL Vs Globally Distributed SQL databases

</comment><comment><USER11> How does YugaByte compare to CockroachDB?

<comment><USER12> There's a feature matrix here [1] comparing YugaByte to CRDB, TiDB, Aurora, CosmosDB, Spanner, MongoDB, FoundationDB, Cassandra and Dynamo. No idea about performance characteristics though.

[1] [https://docs.yugabyte.com/latest/comparisons/](https://docs.yugabyte.com/latest/comparisons/)

edit: there is a relatively recent blogpost comparing the two here: [https://blog.yugabyte.com/yugabyte-db-vs-cockroachdb- perform...](https://blog.yugabyte.com/yugabyte-db-vs-cockroachdb-performance- benchmarks-for-internet-scale-transactional-workloads/)

Key summary from the post:

 _In a nutshell, YugaByte DB delivers an average of 3.5x higher throughput and 3x lower latency compared to CockroachDB. Following are the detailed performance characteristics at scale (millions of rows) for internet-scale transactional workloads:

5x more insert throughput, 9x faster 4x more query throughput, 3x faster 4x more distributed transactions throughput In addition, YugaByte DB offers additional features such as read replicas (for timeline-consistent, low- latency reads from the local region) and automatic data expiry (by setting a TTL at table level or row level)._

Obviously to be taken with a grain of salt as with all benchmarks doing head- to-head comparisons.

<comment><USER13> Going to vendor's website always have comparisons that claims, it can do everything that the competitor's can not. What I'd also like to know is limitations. Where it's not a good idea to use a product? This would help me make informed decision and build relationship with vendor based on trust.

Best of luck with new licensing.

<comment><USER4> Thanks for your wishes @nishantvyas!

This might help answer some of your questions: [https://docs.yugabyte.com/latest/introduction/#what-are- the-...](https://docs.yugabyte.com/latest/introduction/#what-are-the-trade- offs-involved-in-using-yugabyte-db)

Please look at the trade-offs in the above section (vs SQL, vs transactional NoSQL, vs eventually consistent NoSQL).

(founder/cto at YugaByte)

<comment><USER14> Joins and transactions across nodes, very cool. Inner joins/right joins too? Do these have similar time complexity, latency/bandwidth aside, as a comparably indexed traditional RDBMS?

<comment><USER4> Yes, we support all types of joins (inner, right included) and indexes. In fact, we recently also enabled stored procedures with plpgsql :)

Tuning to perform these efficiently in a distributed manner is a work in progress (this is wrt the time complexity question). The hints used by a traditional RDBMS would not be effective in the distributed case - <USER2> this requires changing the optimizer and doing more "push-downs" to optimize the queries. Currently, depending on the query, these may or may not be fully optimized - but we hope to have good coverage soon!</comment></comment></comment></comment></comment></comment>


The Argument for Real-Time REST APIs - <AUTHOR> http://blog.programmableweb.com/2013/03/05/the-argument-for-real-time-rest-apis/

<comment><AUTHOR> Author here, happy to answer questions. At Zapier we've seen lots of suggestions/specs for HTTP subscriptions, webhooks, and more but it seems the most obvious way to enable real-time in API is via another simple REST resource the encapsulates webhook/subscriptions.

It definitely has its drawbacks (mainly that there is no ubiquitous specification) but the advantages are most folks already have a heavily used REST API, and we already have lots of variation in REST so its nothing particularly jarring.

I'd love to hear thoughts on this though.

<comment><USER2> Hi Bryan,

Enjoyed your post. First-time posting here on HN so pardon any noobish remarks that follow ... I definitely like the idea of exposing webhook/subscriptions via REST in this way. Do you have any thoughts on how rich client-side apps (e.g. a Backbone app) could get the real-time benefits of such an API? It's my understanding that real-time updates will be POST-ed to a user-defined hook URL, presumably hosted on a server that is maintained by the API consumer. Maybe I've got it all wrong?

Thanks again for your post.

<comment><USER3> A great way to get webhooks to the client is through some sort of real-time push service. Either homegrown or something like Pusher. You'd have to listen at a webhook endpoint then re-create the event in Pusher, but the client would be simple. Pusher might even support creating the event directly from a webhook, I can't recall.

If you don't want to maintain the webhook listener, create a script on webscript.io to listen and create the Pusher event.

<comment><USER2> Nice suggestions. I'll have to give Pusher a try one of these days. I've actually worked with the homegrown approach before, having my client connect to websocket endpoints that subscribe them to a redis channel where real-time updates are published as events occur within the system.

One thing that's often bothered me about that approach is how I've handled both websocket and RESTful HTTP endpoints from a single server. The solution mentioned by OP could go some way to allowing me to move the websocket/push handlers off to a dedicated server. This 'push server' might consume the REST API exposed by my HTTP server to set up webhooks. It could then forward information coming in on the hooks to the client, using some push mechanism, be that websockets, Pusher, whatever.

Thanks for your reply :)</comment></comment></comment></comment>


Ask HN:Is anyone from the community able to live strictly from donations? - <AUTHOR>

Hi, I have a problem. I find myself unable/unwilling to work on things that I don't consider important, and by important I mean things that can improve the human condition, that optimize in some way our existence on this planet, that deliver true value. At some point in the past this made me quit my job and never wanting to come back again. I had a couple attempts later but no use. I just couldn't take anymore the inefficiency, the misplaced or lack of interest, the greed or just plain stupidity. I could swear a lot of people don't have a clue what they are doing, what is the impact of their actions and don't really care. I know I may be no better and that I'm just too idealistic but this is what I feel. I also find it very difficult to join other projects even if I like the general idea. It just has to grow inside me somehow in order to make me act..<p>In the same time, the things I work on and do consider important I just want to give away for free. The only way I find acceptable to get something in exchange of the potential value that I provide is donations. So I was wondering if anyone of you guys manages to live strictly from donations? <comment><USER2> Tarn Adams, the creator of dwarf fortress, is the canonical game programmer example.

</comment><comment><USER3> I came here to say dwarf fortress but someone beat me to it. I will also suggest you look at The Wotch. It is a web comic. At one time the main author was getting enough in donations for it to be their sole or primary source of income. (I do not know the current status.) Another site that at least at one time supported the author in spite of their longish list of limitations is Hyperbole and a Half. Kind of webcomic-ish, kind of unique. She did not take donations or use ads. She was only comfortable selling merchandise and, at least for a time, that was her source of earned income.</comment>


Massacre at IBM - <AUTHOR> http://steveblank.com/2012/08/03/massacre-at-ibm

<comment><USER2> _Find and Fix Bad News. Bad news early is good news, if you find and fix it early._

This is my immediate takeaway from this article. I am just learning to not be shy about showing people my projects early, to start the validation process as soon as possible. I have been surprised to see that people not liking some of my ideas actually feels good, because it frees me to either improve the ideas, work on something else, or push hard and try to show that my idea does have merit.

</comment><comment><USER3> tl;dr: if you ignore your customers, pretty soon they will ignore you.

(Almost all of their customers said that they needed a particular feature. Not having it was costing them production time. The few customers who didn't explicitly say they needed it were considered evidence that nobody really needed it. Selective deafness.)

</comment><comment><USER4> That was a really horrible way to give a story a linkbait headline, intentionally or otherwise. Flagged.

A former IBMer.

</comment><comment><USER5> envelopE. the word is envelope. please. "envelop" is a verb and completely throws the reader. twice.

</comment><comment><USER6> I'm currently reading _The Strategy and Tactics of Pricing_ by Nagle et al. And something similar is jumping out at me -- in terms of going deep into the customer's world and finding out how your product adds value.

Talking. The best thing since pointy sticks.

</comment><comment><USER7> What an amazingly tasteless headline.

</comment><comment><USER8> bad headline.

bad spell checking.

shameless self-promotion.

gathering requirements during dinner and compressing the schedule from 18 months to 9. I'm sure it was great to work in engineering there and keep those managers honest.</comment>


Don't Post The Evidence Unless It Supports Your Case - <AUTHOR> http://www.techcrunch.com/2008/08/24/dont-post-the-evidence-unless-it-supports-your-case/ <comment><USER2> _"Let me help Mr. Arrington with disclosure.

Marc Andreessen co-founder of Ning is on the panel of experts for his upcoming TechCrunch conference. <http://www.techcrunch50.com/2008/panel-of-experts/>

C'mon, Mike you need to disclose this is you are going to bash WL. "_

Ouch. The comments war that came after Arrington's little fit is kinda interesting.

</comment><comment><USER3> Everyone does it. Most people ignore evidence that doesn't meet their preconceptions.</comment>


TextMate-like ⌘T & ⇧⌘T in Google Chrome Dev Tools & Other New Features - <AUTHOR> http://www.elijahmanor.com/2012/02/textmate-like-t-t-in-chrome-dev-tools.html

<comment><USER2> Odd title; confusing for those of us who haven't used TextMate and unnecessarily specific since the feature in question isn't specific to TextMate at all. Surely the title should be "'Go-to-file' and 'Go-to-symbol' shortcuts in Google Chrome Dev Tools...".

<comment><USER3> What is funny, I use TextMate for more than a year already and thanks to the title I learned what cmd+shift+t does, did not know:) I only knew cmd+t.

<comment><AUTHOR> Ohh yeah, the find symbol feature is very helpful. I primarily use Sublime Text 2 these days (which has a similar feature)

</comment></comment></comment><comment><USER4> Not a while ago, I used to switch to Firefox whenever I was working on any webapp or website because Firebug seemed so much more intuitive than Chrome Dev Tools; but for a few weeks now I don't feel the need to do it. In fact Chrome Dev Tools do provide some features that are not available on Firebug that I think are nice and should be ported soon:

1\. Computed Style.

2\. Timeline - It helps to know what triggers are fired when, without putting log statements.

<comment><USER5> > 1\. Computed Style.

They've been available since... well just about forever: <http://i.imgur.com/GGw22.png>

<comment><USER4> I apologize, I simply forgot that they did, even when I have used it enough. In fact when I just checked Firebug again, I realized how easy it feels to use as compared to Dev Tools.

</comment></comment></comment><comment><USER3> Absolutely Awesome! One more thing is needed, however. The ability to be able to save the modified sources to the server (with appropriate security of course, dev-env only). Also the ability to fetch the server sources from the server, if e.g. I'm running a Node.js and I want to debug and edit the JS in Chrome - _along_ with the client side JS, side by side.

The close we programmers are to the "living code", the better. The file- watching, auto-reloading and re-executing BDD's (or TDD's, etc.) like vows (<http://vowsjs.org/>) get pretty close to it, but it's not the same.

When I'm coding a function, I want to be able to run it anytime (when the syntax is OK of course), and in the environment of my program.

Great stuff. Also similar in-client code editing efforts, Firefox: <https://github.com/Gozala/sky-edit>, general: <https://github.com/ajaxorg/ace>

<comment><AUTHOR> You are in luck! There is an autosave Chrome extension that will save any changes you've made to the JavaScript or CSS from Dev tools to your local file system... [http://addyosmani.com/blog/autosave-changes-chrome-dev- tools...](http://addyosmani.com/blog/autosave-changes-chrome-dev-tools/)

</comment></comment><comment><USER6> Dock to the right is amazing

<comment><USER7> This seriously excites me the most. I can't wait until this reaches the stable build.</comment></comment>


Interview with Ren Zhengfei, Founder and CEO of Chinese Telecom Giant Huawei - <AUTHOR> https://www.youtube.com/watch?v=Nl2jCWDwE8w <comment><USER2> This was an interview? Seems more like a very good PR piece for Huawei's CEO.

<comment><USER3> Jeez; What a disingenuous comment insinuating that there can never be any decent news about the "big bad" Chinese company and its people.

<comment><USER2> What is disingenuous about my comment? I think you're reading a lot into it. I just don't think this video qualifies as an interview in my opinion. If Trump or some other polarizing figure were given a platform to say whatever he wants with no scrutiny, most would (rightfully) not consider that an interview either.</comment></comment></comment>