Apache NiFi - <AUTHOR> http://nifi.apache.org/index.html <comment><USER2> I've used it a fair bit, though not for a couple of years. Few points, some of which may be out of date:

* I've seen customers fall into the trap of thinking they don't need expensive developers because you can drag and drop, just people who can use a mouse can crack on with NiFi.

* It persisted its config to an XML file, including the positions of boxes on the UI. Trying to keep this config in source control with multiple devs working on it was impossible.

* Some people take the view that you should use 'native' NiFi processors and not custom code. This results in huge graphs of processors with 1000s of boxes with lines between you have to follow. Made both better and worse by being able to descend and ascend levels in the graph. The complexity that way quickly becomes insane.

* You're essentially programming with it. I've no doubt you could use it to write, say, an XMPP server if so inclined. Which means you can do a great many things of huge complexity. Programming tools have developed models for inheritance and composition, abstraction, static analysis, etc. which NiFi just didn't have. The amount of repeated logic I've seen it's configuration accumulate is beyond anything I've seen from any novice programmer.

I ended up feeling like it could be an OK choice in a very small number of places, but I never got to work on one of those. The NSA linking together multiple systems with a light touch is possibly one such use case. For most everyone else, I couldn't recommend it.

<comment><USER3> I have not found that argument persuasive to the managers who believe that coding is inherently wasteful if what you're trying to do is _technically possible_ in a workflow builder GUI.

<comment><USER4> That's true but, personally, I just avoid working for managers like that.

<comment><USER3> I can avoid this at the line manager level, but Directors love these things.

<comment><USER5> No, we don't.

</comment></comment></comment></comment></comment><comment><USER6> We used NiFi...one of the worst experiences.

It installs like an appliance and feels like you are grappling with a legacy tool weighed down by a classic view on architecture and maintenance.

We had built a data pipeline and it was for very high-scale data. The theory of it was very much like a TIBCO type approach around data-pipelines.

Sadly the reality was also like a TIBCO type approach around data-pipelines.

One persons experience and opinion and I am super jaded by it due to some vendor cramming it down one of our directors throats who subsequently crammed it down ours when we warned how it would turn out. It ended up being a very leaky and obtuse abstraction that didn't belong in our data-pipeline when you planned how it was maintained longer-term.

I ultimately left that company. It had to do with as much of their leadership and tooling dictation as anything else, NiFi was one of many pains. I am sure there are places that are using NiFi who will never outgrow the tool so take it with a grain of salt.

Said company ultimately struggled for the very reasons those of us who left were predicting (the tooling pipeline was a mess and was thrashing on trying to get it right, constantly breaking by forcing this solution, along with others, into the flow. Lots of finger-pointing).

Sucks to have that: "I told you so..." moment when you never wanted that outcome for them....I just couldn't be a part of their spiral anymore.

<comment><USER7> Can you elaborate by what you mean on a TIBCO like approach? I haven't used their tools, but would like to know more about the issues you ran into. What were examples of the leaky abstraction>?

<comment><USER8> I'd like to second this request. I have encountered event buses and ETL in a number of places over my career - I don't understand what the heck TIBCO does beyond something simple like RabbitMQ/ZeroMQ. How is this different from Pub- Sub (and its variants). Any pointers to books or blog posts would be really appreciated.

<comment><USER6> TIBCO is very much providing queuing/caching to shuttle data from one point to another.

The goal is even more-so to be the interconnect for all systems across a varied enterprise at a higher level. It's all pub-sub underneath the hood. Think cheap butts in seats doing the same work for a "negligible hit on performance".

In the same way you can plug random devices into outlets around the house all served by some powerplant you don't know (or even need to care about), TIBCO attempts to provide that same interface.

Data does need some restructuring, whether these are aggregations, transformations, etc. So they provide steps in the process where you can perform these operations through a drag and drop UI.

There is an input defined and an output defined in XML that you don't have to code, but is managed and can be seen. The engine beneath provides the lower layers of routing, bytecode, implementation letting you just drag blocks around on a screen "connecting things".

The goal is very pure: I have many people in my organization that know how data flows, not all of them are developers. How can I enable them to connect my organization without everyone needing to be a developer.

In theory and in practice are always the interesting observations. What I had seen happen (as was mentioned somewhere else) is that very strong developers became weak by relying on this tool (or merely left for adequately challenging work). When the world moved on to something else, so much had changed it was almost a career change to get back into development.

They went from understanding Java 3/4, JEE to Java 11, Spring, DI Frameworks....I saw a lot give up or move over to product management roles. This only made the tension between on-premise infrastructure teams and public cloud teams more divisive and toxic. I don't think it's anything uncommon in other areas, just feel like we've reached a full revolution in this particular space (and not the first revolution either).

<comment><USER7> Thanks for a clear explanation without dismissing the product as garbage. (it's in that space where techies hate it, but it must provide value since it's so expensive!)

Why do non-technical people need to understand the data flow? It seems like documentation (data dictionaries) would be preferred. Or, are they useful for very non-technical people, while TIBCO data flow understanding is useful for people who are data savvy but not tech savvy?

<comment><USER6> It is a butts in seats equation.

If you can have less expensive operators driving and mapping the world and place all the smarts in the pipes, you can drive down opex and divert cash to capex for competitive advantage.

Linux and much of the streaming software world is smart people, dumb pipes.

If you invert that you have more automation, predictability, control at lower cost. The risk is a lot of eggs in one basket and when the market turns, if the company you are buying from mismanages tech, if they can't keep pace with change...you go along for their ride. Every company big and small falls into this technical debt. I have maby opinions on why as I am sure many do.

There is a lot baked into that comment but the constant tug-of-war every CIO is trying to wrap their head around....how do we do more with less and gain an advantage.

</comment></comment></comment></comment></comment></comment><comment><USER9> NiFi's biggest strength is that it is a 2-way system - it is not Storm, it is not Flink, it is not Kafka, it is not SQS+Lambda.

I like to think of it like Scribe from FB, but with an extremely dynamic configuration protocol.

The places where it really shines is where you can't get away with those 3 and the problem is actually something that needs a system which can back-pressure and modify flows all the way to the source - it is a spiderweb data collection tool.

So someone trying to Complex Event Processing workflows or time-range join operations with it, will probably succeed at the small scale, but start pulling their hair out at the 5-10GB/s rate.

So its real utility is in that it deploys outside your DC, not inside it.

This is the Site-to-Site functionality and MiniFI is the smallest chunk of it, which can be shrunk into a simple C++ something you can deploy it in every physical location (say warehouse or grocery store).

The actually useful part of that is the SDLC cycle for NiFi, which lets you push updates to a flow. So you might want to start with a low granularity parsing of your payment logs on the remote side as you start, but you can switch your attention over it to & remove sampling on the fly if you want.

If you're an airline flying over the arctic, you might have an airline rated MiniFI box on board which is sending low traffic until a central controller pushes a "give me more info on fuel rates".

Or a cold chain warehouse which is monitoring temperature on average, until you notice spikes and ask for granular data to compare to power fluctuations.

It is a data extraction & collection tool, not a processing and reporting tool (though it can do that, it is still a tool for bringing data after extraction/sampling, not enrichment).

</comment><comment><USER10> Incredible piece of software. I've used it in production at my last two jobs. You can build almost anything in NiFi once you get into the mindset of how it works.

A good way to get started with NiFi is to use it as a highly available quartz- cron scheduler. For example, running "some process" every 5 seconds.

Disclaimer: I'm an Apache NiFi committer.

An article you might find interesting about it's ability to scale.

[https://blog.cloudera.com/benchmarking-nifi-performance- and-...](https://blog.cloudera.com/benchmarking-nifi-performance-and- scalability/)

Disclaimer v2: I used to work at Cloudera

<comment><USER11> This page is classic Apache project in that I read it and have no idea what it does. Can you high level explain what this thing is really for?

<comment><USER12> Agreed. So here's an attempt to describe NiFi at a high level.

Fundamentally NiFi is a "dataflow engine", a system that can be used to automate data transfer from different and varying types of sources and sinks. It has a fairly usable UI that enables a "dataflow manager" (end user) to perform transformation, routing and delivery of data using a "drag-n-drop" configuration approach.

Getting data into or out of your application/system, or performing simple schema transformations, is a common (maybe tedious) task that most developers face. NiFi helps connect the dots, so to speak, and decouples the receipt/delivery of data away from your application. NiFi comes with a set of "batteries included" connectors for almost every transport protocol you would generally need. And it's modular so you can create your own processing components as well.

NiFi is fundamentally modeled after what's called "Flow-Based Programming"[1], which is a style of programming that facilitates composition of black-box processing units. It can run at an enterprise or IoT level, depending on where that decomposition best fits into your architecture.

[1] [https://en.wikipedia.org/wiki/Flow- based_programming](https://en.wikipedia.org/wiki/Flow-based_programming)

(disclaimer: I'm affiliated with the NiFi project)

<comment><USER13> Is this a layer on Apache Camel [1] or something completely different?

[1]: [https://camel.apache.org/](https://camel.apache.org/)

<comment><USER10> It was built from scratch at the NSA and open sourced a few years ago.

<comment><USER14> I'm curious how you would compare it to Apache Camel - when would you use Camel and when would you use this?

We use camel with DSLs to make programmatic workflows that connect data flows together. However Camel itself doesn't typically carry the data. Sometimes it SFTPs files around etc., but mostly, it is just a messaging layer.

Is that the main difference here?

</comment></comment></comment></comment></comment></comment><comment><USER12> NiFi at first glance sometimes just looks like a glorified GUI for building out a data-delivery application. But NiFi doesn't just compile an application to be deployed on your network. Instead, the "power" of NiFi is that it allows an operations staff to perform the regular day-in-day-out task of monitoring, regulating and if needed modifying the delivery of data to an enterprise.

NiFi gives insight to your enterprise data streams in a way that allows "active" dataflow management. If a system is down, NiFi allows dataflow operations to make changes and deal with problems directly, right at tier 1 support.

It's often the case that an enterprise software developer has an ongoing role of ensuring the healthy state of the applications from their team. They don't just develop, they are frequently on call and must ensure that data is flowing properly. NiFi helps decouple those roles, so that the operations of dataflow can be actively managed by a dedicated support team that is more tightly integrated with the "mission" of their dataflow.

NiFi additionally offers some features that most programmers skip to help with the resiliency of the application. For example:

\- the concept of "back pressure" is baked into NiFi. This helps ensure that downstreams systems don't get overrun by data, allowing NiFi to send upstream signals to slow or buffer the stream.

\- data provenance, the ability to see where every piece of data in the system originated and was delivered (the pedigree of the data). Includes the ability to "replay" data as needed.

\- dynamic routing, allowing a dataflow operator to actively manage a stream, splicing it, or stopping delivery to one source and delivering to another. Sources and Sinks can be temporarily stopped and queued data placed into another route. Representational forms can be changed (csv -> xml -> json, avro), and even schemas can be changed based on stream.

Anyone can write a shell script that uses curl to connect with a data source, piping to grep/sed/awk and sending to a database. NiFi is more about visualizing that dataflow, seeing it in real-time, and making adjustments to it as needed. It also helps answer the "what happens when things go wrong" question, the ability to back-off if under contention, or replay in case of failure.

(disclaimer: affiliated with NiFi)

</comment><comment><USER15> NiFi is vey good at reliably moving data at very high volumes, low latency, with a large number of mature integrations, in a way that allows for fine grained tuning, and i've seen first hand that it is very scalable. It's internal architecture is very principled: [https://nifi.apache.org/docs/nifi- docs/html/nifi-in-depth.ht...](https://nifi.apache.org/docs/nifi- docs/html/nifi-in-depth.html)

Out of the box it is incredibly powerful and easy to use; in particular it's data provenance, monitoring, queueing, and back pressure capabilities are hard to match; custom solution would take extensive dev to even come close to the features.

It is not code, and that means it is resistant to code based tooling. For years it's critical weakness was related to migrating flows between environments, but this has been mostly resolved. If you are in a place with dev teams and separate ops teams, and lots of process required to make prod changes, then this was problematic.

However, the GUI flow programming is insanely powerful and is ideal when you need to do rapid prototyping, or quickly adapt existing pipelines; this same power and flexibility means that you can shoot yourself in the foot. As others have said, this is not a tool for non technical people; you need to understand systems, resource management, and the principles of scaling high volume distributed workloads.

This flow based visual approach makes understanding what is happening easier for someone coming later. I've seen a solution that required a dozen containers of redis, two multiple programming languages, zookeeper, a custom gui, and and mediocre operational visibility, be migrated to a simple nifi flow that was 10 connected squares in a row. The complexity of the custom solution, even though it was very stable and had nice code quality, meant that that solution became a legacy debt quickly after it was deployed. Now that same data flow is much easier to understand, and has great operational monitoring.

Some suggestions: \- limit NiFi's scope to data routing and movement, and avoid data transformations or ETL in the flow. This ensures you can scale to your network limits, and aren't cpu/memory bound by the transformation of content. \- constrain the scope of each instance of nifi, and not deploy 100s of flows onto a single cluster. \- you can do alot with a single node, only go to a cluster for HA and when you know you need the scale.

</comment><comment><USER16> Phew! Happy to have read the comments here. They say a lot. I will go with Apache Airflow for all my workflow needs from now on. I wasn't entirely sure if this was the best bet, but after seeing all of this I am now.

I know a massive installation [0] which is about to be open sourced, where Apache NIFI is used in the middle of the stack as a key component. No dismissal of the capabilities this package offers intended.

[0] [https://sikkerhetsfestivalen.no/bidrag2019/138](https://sikkerhetsfestivalen.no/bidrag2019/138)

slides [slide #32]: [https://static1.squarespace.com/static/5c2f61585b409bfa28a47...](https://static1.squarespace.com/static/5c2f61585b409bfa28a47010/t/5d76acacbff92a46b71dfdba/1568058561846/ELK- i-solnedgang.pdf)

</comment><comment><USER17> For the love of god, don't use NiFi to trigger an Airflow DAG.

<comment><USER18> Can you expand? We just set this workflow up and it seems to be working fine.

<comment><USER17> NiFi is meant for stream processing and Airflow for batch processing, if your NiFi triggers an Airflow DAG that means that your entire process is batch processing and you shouldn't use NiFi in the first place. If you still want to do stream processing then use Airflow sensors to "trigger" it.

</comment></comment></comment><comment><USER19> If you're considering Apache NiFi, you should also look at Apache Airflow and Uber Cadence to decide what model would work best for you.

<comment><USER20> They do have totally different use cases, so it should be fairly quick to decide which one is for you.

<comment><USER21> Can you explain what the different use cases are? In particular, what is Airflow good at that Nifi is not?

I have used Nifi a little bit and Airflow not at all. Reading the home pages of the two products, it's hard for me to know when it would be more appropriate to use Airflow than Nifi.

They both schedule jobs and move data according to control flow topologies that you build in a GUI, right?

<comment><USER22> From watching the presentation on Youtube ([https://youtu.be/sQCgtCoZyFQ](https://youtu.be/sQCgtCoZyFQ)), it seems NiFi is geared towards acquisition of data, gluing/batching/massaging the flow between systems and providing the necessary interfaces to downstream systems;

\- Would love to see the ability to develop custom NiFi processors in Go/Rust/Elixir etc.

\- XML is a big pain in the rear.

\- Being container-aware is big win. Stateless is even better.

I see a good opportunity there for users like me to explorer NiFi's capability in the future.

> They both schedule jobs and move data according to control flow topologies > that you build in a GUI, right?

Airflow on the other hand is designed to run scheduled jobs (whether it be batched or otherwise). The 'job' can really be anything - build / data processing pipelines, system configuration management pipelines and so on. In Airflow parlance, one can create connected DAG's as pipelines that massage the data in a way you intend it to.

They both share some commonalities but I do gravitate towards their use cases being subtly different and an important one highlighted above.

</comment></comment></comment></comment><comment><USER23> Can someone explain what this is? I can't find anything on the website that explains it

<comment><USER24> Sure, I've worked with it. It let's you visually build data pipelines. It's extremely useful for getting work done quickly--you just drag and drop prebuilt connectors to things like elasticsearch, s3, or twitter and you have a data pipeline, including automatic backoff and the ability to inspect the data at each step. It's visual so it's easy to tell what's going on. Biggest downside is it's not automatically distributed. You can set it up to be distributed, but you have to do the plumbing yourself on the nifi graph by dropping nodes for routing tasks between nifi servers. Overall, perfect tool for quickly building a pipeline that can be easily shown to the business and in which you can visually see all data and errors at each step.

<comment><USER25> Why would one consider distributing it in the first place? IMHO that would require distributed transactions, which would make it hugely complex.

</comment></comment></comment><comment><USER26> We are using NiFi as our dataflow engine for real time data ingest. We are using a current version, 1.11.4, and have several instances running including a development instance. The interface provides our team the ability to do quick iterative development and testing. An example of one of our use cases is we have 2 dataflows that ingest data from 2 different vehicle location/status systems and pump them into SQL Server. At the same time another dataflow merges the data from SQL Server and sends the data to Azure Event Hub. These dataflows were easy to setup, test and extend. This replaced a process that was written in Go.

</comment><comment><USER27> Nifi is a good (not great) tool, mostly because of all of the functionality you get out of the box. It comes with almost any kind of connector you would need for moving data. There's a pretty steep learning curve, but once you push through that, creating a new data flow from scratch is quick and easy. It sucks that other people in this thread have had bad experiences with Nifi, and I can't say that I haven't. However, it has generally been a positive addition to my team's stack.

</comment><comment><USER28> Had some second hand opinions on running NiFi in prod and all of them were rather negative, some saying it was a mistake. That was around one year ago. I wonder if things have changed since then.

</comment><comment><USER29> I have never heard of this before, and I'm sad that profit-driven, marketing speak has taken over even non-profit product pages.

> An easy to use, powerful, and reliable system.

This is the title. That's the most important sentence, and it's absolutely meaningless.

It's bad enough that everything has to "sell" \- just describe to me what your product does and I'll decide if I need or not. Don't try to convince me.

If you have to sell, do it by differentiating yourself from your competitors. No one is calling themselves "Difficult to use, weak, and unreliable", so saying the opposite is not differentiation.

When did we accept that marketing-speak was default communication. Can't we have some landing pages that are essays? Or even a few paragraphs instead of trying-to-be-catchy bullet point phrases in large font?

<comment><USER30> > This is the title. That's the most important sentence, and it's absolutely > meaningless.

Well, yeah, it's meaningless if you cut off the second half...

> ...to process and distribute data.

That's what it does. The adjectives before it aren't the meat of the sentence.

</comment></comment><comment><USER31> I have experience from multiple projects with NiFi and it was the main reason for me and others quitting the company. Somehow management were convinced by some salesmen that this would be the golden bullet, however, all of their deliveries were delayed. We experienced issues debugging flows with performance problems, and even basic version control was problematic due to ids being replaced every time.

</comment><comment><USER32> NiFi is a fantastic tool for a certain set of organizational constraints.

* It doesn't need much in the way of dependencies to run. If you can get Java onto a machine, you can probably get NiFi to run on that machine. \- That is HUGE if you are operating in an environment where getting any new dependencies installed on a machine is an operational nightmare.

* It doesn't require a lot of overhead. Specifically, no database.

* You can write components for it that don't require a whole lot of tweaking for small changes to the incoming data. So, if I have a machine processing a JSON file that looks like XXYX and another machine processing a nearly identical JSON file that looks like XYXX, the tweaks can be made pretty easily.

So, if you're looking for a lightweight, low overhead, easily configurable tool that may be running in an environment where you've got to run lots of little instances that are mostly similar but not quite, NiFi is great.

If you are running a centralized data pipeline where you have a dedicated team of data engineers to keep the data flowing, there are better options out there.

</comment><comment><USER33> No more XML. Check out NiFi 1.11.4, it does everything you need for easy ingest. If you are reading some files putting them into Kafka or S3 or a database or MongoDB or Hbase or Hive or Impala or Oracle or Kudu or ..., it's genius.

[https://www.datainmotion.dev/](https://www.datainmotion.dev/)

<comment><USER34> As far as I see, your link has nothing to do with 1.11.4 release. May be you wanted to link to a specific page?

<comment><USER33> [https://www.datainmotion.dev/2019/11/exploring-apache- nifi-1...](https://www.datainmotion.dev/2019/11/exploring-apache- nifi-110-parameters.html)

</comment></comment></comment><comment><USER35> Having used NiFi in production, my biggest issue with it is handling source control and multiple environments. As the "IDE" is effectively also the runtime, the lines between "local", "stage", and "prod" are easy to blur.

They have a built-in source control product called "NiFi Registry", which can even be backed by git. The workflow for promoting flows between environments feels clunky though, especially as so much environment-specific configuration is required once your number of components gets high enough.

Moving our Java, Ruby or Go code between environments or handling versioning and releases was a piece of cake, in comparison.

</comment><comment><USER36> Do I understand what this is: general purposes SSIS-type data integration, pipeline, and workflow tool?

If so, how does it compare to SSIS, dbt, and other projects (please name!)?

Otherwise, what is an analogous toolset?

</comment><comment><USER37> I have been working on a new product which competes with NiFi, providing streaming data transformations.

Think, if order value > 100 and the customer has ordered 3 times in the last hour and the product will be in stock tomorrow.

Kafka streams, Flink and Dataflow are super powerful and I think there is room for a GUI tool.

Would be great to hear experiences of NiFi in this domain or discuss the space with any experienced users. Will add contact details in my profile.

<comment><USER38> Have you heard of BPEL? :D

<comment><USER39> From wikipedia:

> the open BPMS movement led by JBoss and Intalio Inc., IBM and Microsoft > decided to combine these languages into a new language, BPEL4WS. In April > 2003, BEA Systems, IBM, Microsoft, SAP, and Siebel Systems submitted BPEL4WS > 1.1 to OASIS for standardization via the Web Services BPEL Technical > Committee

if that doesn't scare you, then oh boy...

Just look at the list of organizations mentioned there, I don't know if you could make it more enterprisy.

</comment></comment></comment><comment><USER40> For those who were around during the mid-2000s, is this basically another revival of SOA (Service Oriented Architecture)?

I watched one of the explanation videos and it brought back memories.

My dislike of the phase back then, which I hope they've addressed now, is that while everything looked find and dandy while designing things on a UI, when something broke it was a whole heaps of generated XML no one could read.

</comment><comment><USER41> So Apache has at least a handful of software packages that do about the same thing, but with different interfaces and connectivity?

<comment><USER42> This has sorta been my experience with a lot of Apache projects recently. The differences between them are becoming quite nuanced.

I'm trying to piece together the main reason why someone would pick this over Camel, or vice-versa. I know they're different - but not night and day.

</comment></comment><comment><USER43> So, the comments here have mostly ranged between neutral to negative regarding their experience.

I have a problem where I want to stream data to an ML layer and then stream that to a web app (e.g. Laravel or Django).

Reading the docs here, this seems like this would solve this problem, but was wondering if people had alternatives given that people seem to think poorly of this application.

<comment><USER44> Nifi is fantastically good at one thing, which is dataflow. Where you've got data coming in at point A, but you need it at point B, and for some reason can't convince either A or B to connect directly.

It's not a message bus, nor is it a data processing framework, nor a scheduler, nor an ETL tool. If you try to use it for one of those you're in for a bad time.

What you're describing sounds like you might need a message bus (think ZeroMQ, Kafka, etc). Assuming you're writing the software yourself and want to connect it together.

<comment><USER45> could you elaborate on the difference between what op is asking for and what nifi/airflow does? to me the use case of moving data through a couple of different services could be solved by a message bus OR nifi.

<comment><USER44> Nifi is designed to handle the problems that crop up when two systems can't talk directly to each other. It puts a buffer between them to allow one process to keep sending data when the other isn't quite fast enough, it can do some basic transformation when the data isn't quite in the right format, etc.

However if you just need something to send messages then you're better off using a tool that does just that, you don't need the overhead of a system that can connect arbitrary applications that talk in incompatible protocols you just need a single protocol that allows your applications to send each other data.

In their docs Nifi calls itself a dataflow tool and calls dataflow a necessary evil. It's the band-aid you need when you've got a mismatch between the way data is generated and the way it is consumed. It would be insane to deliberately create such a mismatch just to use Nifi.

<comment><USER45> ok that makes sense. but in that case wouldn't be easier to write your own adapters for each data source?

<comment><USER44> Well no, Nifi has _a lot_ of adapters that you can use out of the box, making them yourself isn't easier.

</comment></comment></comment></comment></comment></comment><comment><USER46> I like the way how it buffers messages. You basically stop a process and it will continue when it left off. It is easy to create a distributed cluster. It has hundreds of different connectors for external sources. On the other hand is very bulky. Making it to work with https was horrible. You cannot just put it behind a reverse proxy.

</comment><comment><USER47> We only use NiFi on the edge of our data lake. It's very good at bulk loading, pulling log files/sensor data from hundreds of systems into our systems.

However, it does not handle small records well, and deploying custom processors is a pain, so don't use it to replace your stream processing framework.

</comment><comment><USER48> Our team found this adapter to integrate ML with NiFI pretty handy: [https://dev.to/tspannhw/easy-deep-learning-in-apache-nifi- wi...](https://dev.to/tspannhw/easy-deep-learning-in-apache-nifi-with- djl-2d79)

<comment><USER33> Thanks! I wrote a few more of those. [https://github.com/tspannhw/ExecuteClouderaML](https://github.com/tspannhw/ExecuteClouderaML)

</comment></comment><comment><USER49> It reminds me of LONI Pipeline[1], which was created for the need of a neuroscience lab to process images of brain scans.

[1] [http://pipeline.loni.usc.edu/](http://pipeline.loni.usc.edu/)

</comment><comment><USER50> I am new to this site, why is there just a link to Apache NiFI on the front page? Is this somehow news? Sorry not trying to be rude just confuses me a bit since NiFi has been around for some time.

<comment><USER51> Someone found it interesting, I guess. :) Personally, I'm finding the discussion very interesting, since I haven't seen it here before.

Sometimes Wikipedia articles hit the front page. It's fine. Usually.

</comment></comment><comment><USER52> Is this an open source self hosted equivalent to [https://aws.amazon.com/datapipeline/](https://aws.amazon.com/datapipeline/)?

<comment><USER53> Yes, basically. AWS Data Pipeline includes spinning up and tearing down VMs to run the pipelines.

Nifi has a much more extensible architecture though, you can pretty much do anything you want in a data pipeline in Nifi, and open source of course.

</comment></comment><comment><USER54> Reminds me of the early 2000s when we were all into BPM, graphical or otherwise. The drawbacks are pretty obvious. I bet the engineers who built it had fun, tho.

<comment><USER55> As a Java programmer back then, watching with horror as consultants brought in their flavor of the month (MQ/Spring/Camel), I could only pad my resume, and pretend to say "never look a gift horse in the mouth".

</comment></comment><comment><USER56> For someone not involved in the web stack, reading through that page tells me nothing. Can someone tell me what use cases this is meant to solve?

<comment><USER12> I tried to give an answer here. Does this help?

[https://news.ycombinator.com/item?id=23147150](https://news.ycombinator.com/item?id=23147150)

</comment></comment><comment><USER25> Does NiFi still use XML output for its source code? It makes it very hard to put under source control. Overall it's a nice tool and very fast.

</comment><comment><USER57> Has anyone replaced Mirth with NiFi for HL7 slinging?

</comment><comment><USER58> Our devs hated it due to ease of accidental flow fsck ups. And good luck scaling it or trying to put it behind an LB!

</comment><comment><USER59> > An easy to use, powerful, and reliable system to process and distribute > data.

But what is it?

</comment><comment><USER60> Brought to you by the NSA ;-)

<comment><USER12> I mean, yes. Paid for by $US Tax Payers. I'm for one glad to see at least some of my investment returned, regardless of your opinions about the NSA.

</comment></comment><comment><USER61> inb4 this shit gets bloated to hell like apache httpd

</comment><comment><USER62> We have a team using this at work. They had built a process and needed it to be put on a VM and run periodically. They said the requirements were a dual core machine and 8gb of ram. The "binary" was like 1.8gb. I'm sure it included a jre and a full nifi runtime, but god damn that is ridiculous. Had this process been built using go or crystal or something like that it probably would have been less than a megabyte and able to run with 512mb of ram.

<comment><USER63> From the spec (8gb) it appears that NiFi was not needed. Were you co- locating zookeeper on the same nodes?

Also you say it needed to be run periodically? It's supposed to be a long running service. If you need something that you can spin up and shutdown in a container or VM then it is probably not the solution.

Use it if you have high volumes of data you need to transport from A -> B. We run on a cluster with 256gb ram, 128 core per a node.</comment></comment>


What We Owe the MythBusters - <AUTHOR> http://www.nytimes.com/2015/11/05/opinion/what-we-owe-the-mythbusters.html?src=recg <comment><USER2> I agree with this article whole heartedly. It made science accessible to the general public.</comment>


Insights Your Urine Can Offer - <AUTHOR> http://www.huffingtonpost.com/leslie-spry-md-facp/personal-health-_b_5069148.html

<comment><AUTHOR> I shortened the title from its original version: '5 Key Health Insights Your Urine Can Offer'. A bit linkbaity, I know, but this is HuffPost we're talking about. Feel free to change it.

Big takeaway from the article: pay attention to what your body is telling you. Unless your urine is clear or very light yellow most of the time (perhaps not when you first get up in the morning), it is likely that you are dehydrated. More info:

[http://www.mayoclinic.org/diseases- conditions/dehydration/ba...](http://www.mayoclinic.org/diseases- conditions/dehydration/basics/symptoms/con-20030056)

Solution? Drink more water. Not a crazy amount of water, but more than you already drink. The next few times you urinate, pay attention to the color. Watch the urine stream, not the water in the toilet. Is it much lighter/clearer than before? You were probably not drinking enough water before. Not convinced that there is a causal relationship between your hydration and urine color? Do a 'washout period' where you go back to your previous hydration level for a day or two, then observe urine again, then increase hydration and observe again. Do _not_ trust your thirst as a gauge of hydration. If you are thirsty, chances are you're already dehydrated.

An exception to the 'brightly colored urine is bad' heuristic is riboflavin:

[https://en.wikipedia.org/wiki/Riboflavin#Toxicity](https://en.wikipedia.org/wiki/Riboflavin#Toxicity)

Excess dietary riboflavin is harmless. Other B vitamins can do this too, apparently, so you can expect your urine to look funny if you've taken a multivitamin recently.

More broadly, my advice is to keep an eye out for strange, unexplained changes in your normal physiological function, because these changes might mean trouble.

(Warning: possible 'TMI moment' ahead.)

It amazes me when I walk up to a urinal and find that the previous person who used the urinal not only neglected to flush it (damn you!), their pee is a horrible dark brownish yellow color. This can't be healthy. (The behavior that led to the brown urine, I mean, not my habit of noticing other people's urine.)

Oh, and painful urination is definitely a concern (as in, tell your doctor), but I think you knew that already.

/end public service announcement</comment>


What's Powering the U.S. Economy? It's a Mystery, Frankly - <AUTHOR> https://www.bloomberg.com/opinion/articles/2020-02-06/u-s-economic-expansion-shows-no-signs-of-ending <comment><USER2> "while the Dodd-Frank financial reforms and the scars of that disaster probably are holding back financial institutions from piling up excessive risks"

My impression has been that Dodd-Frank was quietly undone not too long after it was put in place. Is this not true?

<comment><AUTHOR> My impression is that pieces are still in place (the CFPB still exists, the Volker rule still holds for now: [https://www.msn.com/en- us/finance/markets/wall-street-ventur...](https://www.msn.com/en- us/finance/markets/wall-street-venture-fund-curbs-to-be-eased-in-volcker- revamp/ar-BBZnrbM) ) but that it's been slowly undone over the years.</comment></comment>


Top-tier companies that use Elixir programming language - <AUTHOR> https://curiosum.dev/blog/5-top-tier-companies-that-use-Elixir <comment><AUTHOR> What connects Pepsi, Spotify and Discord? They all use Elixir in their tech stack. Why is Elixir so attractive to the top-tier companies in the world?

<comment><USER2> > Why is Elixir so attractive to the top-tier companies in the world?

It isn't, which is why so few of them use it.

The funny thing about Elixir is that even though in theory it has lot of the things you would want in a modern multi-core world, the reality is it would have been more attractive as a language a decade ago.</comment></comment>


Show HN: Implement Posixish threads one bite at a time - <AUTHOR> https://github.com/ljanyst/thread-bites/blob/master/README.md <comment><AUTHOR> What is the best way to understand something well? For me, it's always been trying to build something similar from scratch. This project is my attempt to learn what it takes to implement pthreads on x86_64 Linux. I've learned quite a bit doing it, and it may be useful to other people. I am also interested in your feedback, especially concerning the synchronization routines. PRIO_INHERIT mutexes seem to be a particularly demanding beast. In the limited time I've had for this project, I was not able to decipher what glibc does to implement them.</comment>


Ask HN: Do something you love, or do something you are good at? - <AUTHOR> With the epidemic happening, and in the time of Lent, I've given up playing videogames which has always been a past-time for me and a frequent escape for real world stress. Besides realizing that this isn't healthy (some escape is healthy, just not as much as I was doing), I've began to go back on games I greatly enjoyed playing when I was younger.
Then it hit me, I'm not old yet, but I am getting older and started to think of how I should be spending my time on Earth here.
My strengths are in software and web development, but I'd love to create a game that I used to enjoy playing (Guild Wars) - except that, I have no experience! The question really burned in my mind, do I do something that I am good at, or do I take the chance at doing something I'd enjoy more?
What are your thoughts? <comment><USER2> Do something you love.</comment>


The application paradigm is unsustainable - <AUTHOR> https://medium.com/@miguelrochefort/the-application-paradigm-is-unsustainable-4071f392d2af <comment><USER3> This is a pet peeve of mine.

We were told that with HTML5 it would have been soon possible to use a browser to do almost anything (BTW 90% or maybe 95% of which was already doable before) and even the greengrocer around the corner (not really, just an hyperbole) wants me to install an app (which needs to come into two flavours/platforms, IOS and Android).

And now we have (just as we had before) an endless number of half-broken sites (in the sense that there are no two browsers rendering them exactly the same and the usual patches related to identify a specific browser and serving slightly different content are still everywhere) and ON TOP of that an endless number of very "vertical" or "strictly branded" apps.

<comment><AUTHOR> My post used the term "application", but I could have written the same thing using the term "website" instead.

I don't think using 1,000 different websites is more sustainable than using 1,000 different applications. The same problems I listed apply in both cases.

We need to completely dissociate the client/interface from the service provider.

<comment><USER3> That may be a further step, but right now the difference (which I see as huge) is that a link/bookmark (that can actually be re-discovered in no time - <USER2> not saved - via Search Engine) takes a few bytes of the (precious, limited in size) storage of my device while the stupid app takes Kilobytes or Megabytes.

Moreover, whilst the site can be found 99.99% of times through a quick search on a search engine, the installed app cannot (unless I miss something) if you don't remember how the stupid icon/logo looks like.

In any case I see lots of people swiping pages and pages before - maybe - finding the stupid app they are looking for.

BTW, it is not an entirely "new" thing, on more traditional desktops each and every stupid program (actually its installer) will try to convince you to add an icon/shortcut on the desktop, I have seen desktops with (literally) several tens of such shortcuts, possibly more, each with a "distinctive" icon.

Android or IOS do the same, but due to the limited size of the screen you can have at the most - say - 16 or 25 icons per page, you need to swipe to the next page, visually scan for the icon, miss it, go ahead, go back, etc.</comment></comment></comment>


China is using AI to predict who will commit crime next - <AUTHOR> http://mashable.com/2017/07/24/china-ai-crime-minority-report/?utm_cid=mash-com-fb-main-link#IOwDV7tESqqj <comment><USER2> [https://www.bloomberg.com/news/articles/2016-03-03/china- tri...](https://www.bloomberg.com/news/articles/2016-03-03/china-tries-its- hand-at-pre-crime)

[https://arstechnica.com/information- technology/2016/03/china...](https://arstechnica.com/information- technology/2016/03/china-is-building-a-big-data-plaform-for-precrime/)

</comment><comment><USER3> Just watch minority report to see what a bad idea this is

</comment><comment><USER4> This is a slippery slope.</comment>


Suffering-oriented programming - <AUTHOR> http://nathanmarz.com/blog/suffering-oriented-programming.html <comment><USER4> "With Storm, I distilled the realtime computation problem domain into a small set of abstractions: streams, spouts, bolts, and topologies. I devised a new algorithm for guaranteeing data processing that eliminated the need for intermediate message brokers, the part of our system that caused the most complexity and suffering."

As someone who has gone through the storm source in very fine detail, let me tell you how he did this. He hashed each tuple that needed to be processed then XOR'd it into a variable that started at a value of zero.

When the piece that needed to be processed was complete it would get XOR'd back into the variable. Once the variable hit 0 he knew everything was done! Pretty neat if you ask me.

<comment><USER5> Isn't it problematic that 11, 01 and 10 XOR together to get 00? And why not use a simple counter?

<comment><USER6> Random 64 bit ids are used in the process. So the probability of accidentally completing a tuple is very, very small (1 / 2^64 for every ack).

Counters don't work because of the asynchronous nature of Storm. For example, consider a topology that looks like this:

A -> B -> C

          \-> D

Let's say A emits 2 tuples (+2 differential), B processed those and emits 2 to C and 3 to D (+3 differential), C processes 2 tuples (-2 differential), and D processes 3 tuples (-3 differential).

Everything's asynchronous, so the acker could receive the acks in this order: A, C, B, D

The counter would then look like this: 2, 0, 3, 0

So it would think that the tuple was complete before it actually was, which means the counter algorithm doesn't work.

<comment><USER7> "So the probability of accidentally completing a tuple is very, very small (1 / 2^64 for every ack)."

Which means, due to the birthday paradox, that you should expect your first collision around 2^32 tuples. Process a few million tuples a day (that's only about 25 per second) and you should expect your first error within your first year.

EDIT: I'm wrong, see the explanations below.

<comment><USER6> The birthday paradox doesn't apply. All that matters it the value at the time the ack is applied, so it's always 1/2^64 (because the xor of any number of random numbers is still random).

<comment><USER7> No, the birthday paradox very much applies. The chance of success for your first ack is (2^64-1)/2^64. The chance of success for your first and second ack is the chance of success for your first ack times the chance of success for your second ack, ((2^64-1)/2^64)^2. And so on for your third ack, and your ack.

By the time you reach your 2^32 ack, your chance of having all successes is ((2^64-1)/(2^64))^2^32, which is < 0.5.

EDIT: My test program seems to indicate that I'm wrong, but I can't see the flaw either in it or in my reasoning. Can you explain why the birthday paradox doesn't apply here?

EDIT': In the birthday paradox, it would be ((2^64-1)/2^64)) * ((2^64-2)/2^64) * ((2^64-3)/2^64), etc.

<comment><USER6> The birthday paradox concerns the chance of 2 random elements in an __entire set __being the same. There's no set here, so there's no birthday paradox.<http://en.wikipedia.org/wiki/Birthday_problem>

Hilariously, you got your equation right (which isn't the birthday paradox equation btw) but got your result wrong. [http://www.wolframalpha.com/input/?i=%28%282%5E64-1%29%2F%28...](http://www.wolframalpha.com/input/?i=%28%282%5E64-1%29%2F%282%5E64%29%29%5E2%5E32)

So the chance of success after 2^32 acks is >0.999999999, not <0.5.

It takes about 2^60 acks before there's a significant chance of a mistake. That's a lot of acks, so it will take an insanely long time for a mistake to be made.

</comment></comment></comment></comment></comment></comment></comment><comment><USER8> The biggest problem I have with this approach (and believe me, I love this approach), is that it makes it hard to finish things.

For example, over Christmas, I built a small pretend-natural-language CLI controller for iTunes. I made a working version in something like four hours, spent a few days adding in crazy half-thought-out features like speech recognition and a web interface - and then I basically stopped development.

I didn't stop development because it got boring - I stopped development because I'd solved my own problem. Not beautifully (certainly not from a coding perspective), not efficiently, but the problem I had was solved.

The problem, then, is that once the "suffering" is gone, or sufficiently lessened, there is no real reason to keep building.

(oddly, my password for my old account no longer seems to work. I was hebejebelus)

<comment><USER9> "The problem, then, is that once the "suffering" is gone, or sufficiently lessened, there is no real reason to keep building."

Then how is the project incomplete? If it's not a product that you're planning to sell, put your code on Github or the like and others will add any features that you're missing.

<comment><USER10> > put your code on Github or the like and others will add any features that > you're missing.

No they won't, because he hadn't even _started_ on the "make it beautiful".

When I want to solve a new-ish problem, I can't imagine grabbing some barely working cowdung from some guy's github repo.

If he hasn't even _tried_ to make it clean or readable, it'll take me more time to make sense of the mess than just rebuild it myself.

<comment><USER11> It depends on just how useful, and how ugly, it is. See: <http://dreamsongs.com/WorseIsBetter.html>

<comment><USER10> WiB is more about appreciating barebone low-level tools, like C/Unix in the posted example. These tools are crude but _technically polished_ , otherwise no-one would bother to use them.

<comment><USER11> It depends on what you mean by "technically polished." Unix handles interrupted signal calls by returning an error code that means "I was interrupted." This technique bunts on the hard problem of rolling back OS operations. It is not "technically polished" in that it does not solve all of the hard problems in front of them. But it's still useful, and it caught on because people used it.

Gabriel's insight is not about low-level tools. It's about at what point can you bunt on the hard problems, have an ugly work-around, but still be useful enough that no one will use "the right thing" when it eventually comes about?

</comment></comment></comment></comment></comment></comment><comment><USER12> _It's a mistake to try to anticipate use cases you don't actually have or else you'll end up overengineering your solution._

I wish more people would think like this before sabotaging their perfectly good APIs with noise. I'm a huge fan of the Pareto principle in that regard. First and foremost, expose the 20% that allows me to be 80% effective. The rest can be figured out as we go, but at least, what you'll teach me today, I'll learn fast and will know well.

<comment><USER13> I agree with you 100%.

This is also known as the YAGNI principle: <http://en.wikipedia.org/wiki/You_aint_gonna_need_it>

</comment></comment><comment><USER14> This resonates:

> The most important characteristic of a suffering-oriented programmer is a > relentless focus on refactoring. This is critical to prevent accidental > complexity from sabotaging the codebase.

I can't tell you how many times I've seen accidental complexity creep in because someone adds a new feature without taking the time to refactor to the simplest set of abstractions. But - and here's the flaw in the approach - <USER2> have to be an expert in the code to produce such a set of abstractions, which is a potential bottleneck when your codebase is big enough to require multiple developers. Not everyone has the time/capability to be an expert.

<comment><USER15> What's a good solution then? Have code reviews where the newer developers have to get their changes approved by older developers?

</comment></comment><comment><USER16> This is really an extension of the advice to make sure you build a product that scratches an itch you have. Extend it to developing frameworks, and you have "suffering-oriented programming".

Of course, the real problem is that there are certain spaces that wouldn't really be serviced if that's all we did. Education is an excellent example: the people who really feel the pains of education, students, won't really start having the ability to help solve those pains until they're further along in their education, at which point the earlier pains don't necessarily apply as well. In a similar vein, learning to program is a problem that's been on the map for a long time, and, while the situation is constantly improving, we're dealing with the problem of, by the time we've got the skills to really help solve it, we've already learned to program, and have lost some amount of sight of how learning could be improved, because we're no longer in the proverbial trenches.

So while "scratch a personal itch" is awesome advice, and almost certainly helps in product development and framework development, sometimes it isn't enough. Sometimes just following your curiosity or intuition and exploring a space that doesn't produce regular hurt for you can lead to an equally good result. At that point, what you need to make sure you have is external feedback from the people whose itch you're trying to scratch. Maybe the chances of failure are greater in these cases, but the opportunities for success are also probably greater.

<comment><USER17> I'm not sure I (fully) agree with your example of education.

In education the scratchable "itch" is felt by educators and parents in their offering of a service (education). The extent to which we as a society expend resources providing tools/resources for educators and allowing them to build/purchase solutions demonstrates the extent of (or lack of) our societal valuation of education. The itch in this is scratched by the service providers, not the consumers.

<comment><USER16> The trouble is that the scratchable itches educators and parents feel are not the same ones that the students feel. You need to be able to address all three itches in many educational products, but it's fairly difficult to be in all three roles to gain the understanding you need.

</comment></comment></comment><comment><USER18> "First make it possible. Then make it beautiful. Then make it fast."

This mantra closely resembles the rule attributed to Kent Beck that one should "Make it work. Make it right. Make it fast".

<comment><USER19> Agree, was disappointed to see the paraphrase go unattributed.

</comment></comment><comment><USER20> _"First make it possible. Then make it beautiful. Then make it fast."_

Alas, the guy writing the checks all too often doesn't see past step 1.

<comment><USER21> This. I've yet to see a client development project that made it past step 1 on anything other than the most trivial features. Sucks, but I got paid. What can you do?

<comment><USER22> I think the advice is intended for products you build for users, not products you build for clients. The right strategy depends on who your customer is. When you find a ripe market, you'll discover that your customers—i.e., the users—just want a _solution_ , even if it isn't pretty. When your customer is the client, then you're selling to him, not to his users, which means you rationally should care less about the product. This can be painful, which is one reason why many startup founders prefer to avoid client work.

</comment></comment></comment><comment><USER23> Well written, I especially loved the definition: _Suffering-oriented programming can be summarized like so: don't build technology unless you feel the pain of not having it._

Having said that, it would be cool to hear from a "devil's advocate". I am particularly thinking of cases where "making it possible" gives an 80% solution but a fundamental flaw or limitation makes the remaining 20% prohibitive, basically requiring a complete rewrite to move forward. Is this a problem in practice or are 80% solutions usually "good enough"?

<comment><USER24> If your spec is good, it's rare, because you're most likely to see a showstopper appear 10% or 20% of the way into making the 80% solution - <USER3> that's early enough that you can rearchitect and continue.

The cases where it becomes almost impossible tend to come from the "reuse something written to a different spec" projects where you're forcing things through the wrong architecture. The warning against over-focus on genericity at the end of the article is directed at exactly this - when you aim for generic you can often find yourself building to a spec without concrete goals, thus the actual problems are poorly dealt with.

</comment></comment><comment><USER25> Great read, I have a question:

I would very much love to find out what kind of techniques you folks used to ensure some degree of quality (or sanity) in the "make it possible" phase.

I'm basically wondering if you used anything like CI / TDD / Code coverage or any other quality ensuring techniques at such an early stage. One might argue that it's counter-productive, since you're going to likely rip-out most of that code anyway during the "make it pretty" phase. Others will state that keeping high code quality throughout even prototyping will make you waste a lot less time on debugging and therefore make you overall on the long run.

To me it feels like it'd have to be a very delicate balancing act.

</comment><comment><USER26> A few other commenters have suggested that they don't get a chance to get past _"First, make it possible."_ to making things beautiful, then fast. The person writing the checks doesn't care if the implementation is horrible, as long as it works, etc.

Thing is, both points of view are correct, in different ways.

The kind of long development cycle Nathan alludes to here is not something a lot of people can do, but is one of the risks you can take / benefits you gain if you're your own boss. I wonder if it's more common in larger companies, who can afford to polish some projects before putting them into production? With a startup, it's more likely that there's one or two projects, and both are P1-CRITICAL.

For people wanting to explain it to a boss, I think it's best put in terms of technical debt[1]. This project will be expensive, but by spending more time/resources up front, there is a much lower cost to adding features or fixing bugs in the future; this includes getting new coders up to speed on the codebase, turnaround of critical features, etc. The cost is initial time to market - it will be longer before you see version 1 going out to people who can use it.

Taking on technical debt is a perfectly valid _business_ decision under the right circumstances. If time-to-market is critical and you're planning on having enough money when it's successful to pay off that debt in the future, then maybe you want to throw out making it beautiful / performant just to get something in front of people.

Planned technical debt is a business decision. It's the folks who buy up front and ignore their debt that end up with problems, but that much is something we're all familiar with. One way or the other, that debt is paid eventually, whether with cash or time.

Even if you're in the unfortunate position of turning out software that sucks for people who don't care though, there's light at the end of the tunnel! The best thing about programming is that the more you do it the better you get at it. The faster you get at it. This means you can build refactoring into your projects and estimations without making a big hoo-hah (as it were) about it.

Open source projects are another outlet people have to scratch their perfectionist itch; if you start a project, you decide the timelines, features, level of polish, etc.

[1] [http://www.codinghorror.com/blog/2009/02/paying-down-your- te...](http://www.codinghorror.com/blog/2009/02/paying-down-your-technical- debt.html)

</comment><comment><USER27> The art of thinking about a problem appears to be lost. What happened to getting it fast, beautiful and possible first time by thinking about it?

I come from an engineering background and "right" is the only way to do it.

<comment><USER14> In software engineering the "thinking beforehand" approach is probably riskier than the "think as you go" approach. For example, often the requirements aren't well-defined when the project starts.

Contrast this with physical-stuff engineering (which I think is what you're referring to by saying "engineering background"), where the requirements tend to be better-defined and the cost of experimentation/refactoring is a lot higher.

<comment><USER27> Well electrical engineering turned out to be 50% software (much to my dismay) so I'd say it was right between tangible and non-tangible engineering.

</comment></comment></comment><comment><USER28> _Suffering-oriented programming rejects that you can effectively anticipate needs you don't currently have._

I do anticipate, however, I never do it in code. Rather, I find that just thinking through some scenarios helps me to see where the code might start evolving, and to make sure that that part of the code is isolated enough to change without having to change everything else, if the time ever comes.

(Perhaps Nathan includes this in his "make it beautiful" phase, but I felt it deserved to be made explicit. I've too often heard YAGNI used as an excuse to not even think about a problem, let alone code for it.)

<comment><USER29> I agree, as some agile guru said, a plan is useless but planning is essential. An important thing along with avoiding making useless features is also to avoid building a design that is hostile against features we do need to add. Building a good design without having any anticipation of what will come is in my point of view impossible. The tricky part is to keep the anticipations and assumptions to a reasonable level.

</comment></comment><comment><USER30> Awesome. I have this exact same methodology...Now can you write an article about how to explain how this works to Business people/non-tech people, who have no experience in "iterative program development" or why the programming process MUST be iterative?

Everytime I mention code refactoring to a non-tech person the response is similar to "Why would you program the same thing again?" And your article is the exact reason why. You can't know everything you need to know in the beginning of developing some brand new technology. If that was the case, then every startup would be successful.

</comment><comment><USER29> So in agile terms, besides splitting down the work by user stories it can also be split down by function (make it work), design (make it beautiful) and last performance (make it fast). Quite a natural approach to prioritize that order depending on what stage the project is, early, mid or late. It requires a continuous refactoring but in the business man's point of view it's rework. But the end result will probably look more like a second generation product rather than a first gen with mediocre function, design and performance.

</comment><comment><USER31> LOL. The title really attracted the attention

</comment><comment><USER32> Storm sounds awesome. Can you make something exactly like it but built on Node.js (CoffeeScript if possible but not important) instead of Java?</comment>


Kaby Lake Systems with Intel Optane SSDs Coming Soon - <AUTHOR> http://www.anandtech.com/show/10932/kaby-lake-systems-with-intel-optane-ssds-coming-soon <comment><USER2> The Optane details seem like a false start, at least for consumers. It may have use in enterprise just due to the endurance for WALs and ZIL etc.

Performance of the NVMe Samsung 960 Pro is to the point where enterprise HHHL cards were not much more than a year ago. These are affordable for mid-high end notebooks, and the 960 Evo delivers close to the same amount of performance for an actual consumer workload not running benchmarks that will easily hit the rest of the mid range and down into low end at lower capacity.

According to Micron, there isn't much demand for large consumer SSDs. 1TB will be held for a while as the sweet spot.

So, where do you need 16GB of cache but for HDDs? And where is an HDD actually useful in a modern notebook? I can see people wanting a few TB in a portable USB enclosure. But as a boot/OS drive, you really want all flash.

<comment><USER3> I'd rather a big, fast SSD than an Optane+HDD combo. An HDD in a laptop is just yuck. But Optane does seem to out-perform the top-end PCIe enterprise SSDs by a factor of 5-7x, at least judging by the benchmarks Intel has published against it's own top-end SSDs.

<comment><USER4> It seems like they are putting those drives on x2 PCIE slots there is no way it's going to outpace any NVME drive.

Their capacity is also very small this looks to me like 2005 when Lenovo had SSD cache addin cards for the pc/express card slots.

Give me a 2tb drive or go him until the the 960/960 pro is all the rage.

</comment></comment></comment><comment><USER5> One of the criticisms of the Microsoft Surface Studio a few months ago was that it used a hybrid drive rather than a big SSD. This suggests why Microsoft might have chosen that direction for its hardware's architectural roadmap. It's the kind of customer that would have early access to Intel's pipeline via _quid pro quo_.

</comment><comment><USER6> The use of a hard disk instead of ssd to back the Optane suggests the performance differential between SSDs and Optane is not great enough to justify another level in the cache hierarchy (i.e., sticking the optane in front of an ssd doesn't measurably improve performance, or improves performance less than adding RAM for the same cost).

Maybe they will have something more compelling (more pcie lanes or more capacity) next year, and then they'll start backing these with cheap TLC, or start using them for primary storage.

</comment><comment><USER7> Optane SSD should have latency 10 to 100x lower then even the best SSD. And much higher QD1 throughput.

The problem is how much will this translate into real world experience. Are we still limited by I/O?

And why HDD? It is a waste of space in laptop.

<comment><USER5> There are still reasonable use cases for mass storage in a laptop: photographs, video, music, etc. And there's a tradeoff of overhead and risk with putting stuff 'in the cloud' in the form of management overhead, network connectivity, and the evaporation of a service upon which someone depends.

Laptop minimalism is just a set of tradeoffs. A Thinkpad X-series or Chromebook isn't for everyone. Neither are Xeons nor Atoms.

<comment><USER7> Yes, but the market for Mass Storage in a Laptop? That is even more of a niche. My understanding is that even PROs want 1TB SSD rather then 4TB HDD.

<comment><USER5> I used 'mass storage' to refer to the largest element of the local persistent storage stack. In the end, what matters is not the technology but the performance: what demanding users require is a performance level that is currently provided by SSD's, how that is achieved doesn't matter (marketing and faster alternatives not withstanding).

</comment></comment></comment></comment><comment><USER8> If Optane achieves the latency benchmarks claimed in [1], I would be happy to use it as swap for large-scale computing. For IO-heavy tasks, it could start to blur the decision to scale up vs scale out.

1 = [https://www.extremetech.com/computing/237817-first-intel- opt...](https://www.extremetech.com/computing/237817-first-intel-optane- hardware-could-be-coming-to-consumer-desktops)

<comment><USER9> so you would prefer 16GB Optane drive as swap instead of ... you know, actual 16GB DIMM?

<comment><USER8> In which universe is that a serious question?

<comment><USER9> In the same one where Lenovo announces brand new Kaby Lake laptops with '16GB Optane cache'.

<comment><USER8> To be clear, then, I would not use a 16GB DIMM of RAM for swap. I would use all the RAM I had as RAM.

Having something faster than an SSD for swap would be a win; hence my original point. Specifically, if the latency is only 10x higher than RAM (as shown in the image I linked to), then it would allow for trade offs to be made from an engineering perspective regarding scaling up vs scaling out.

</comment></comment></comment></comment></comment><comment><USER10> We should wait and see what they release before being too critical.

</comment><comment><USER11> I have little idea of how 3D XPoint works but if it is that fast and non- volatile what prevents it from replacing RAM+Disk? Cost I presume or are there other reasons?

<comment><USER12> This isn't necessarily holding 3DXpoint back, but one place where RAM is more useful is security. You don't want TLS encryption keys on a non-volatile medium. Who knew that RAM's quickly lose contents when off property would turn out to be a feature?

</comment></comment><comment><USER13> 16GB SSDs are way too small to be usable in that case, especially when paired with an HDD... They should be paired with an slower SSD.

</comment><comment><USER14> A white ThinkPad. Kill me now. Or at least Lenovo should release the Retro to atone for their sins.

</comment><comment><USER15> me thinks it's time to start making fused CPU/RAM monster chips. Monster CPURAM chips and PCIe SSDs! nothing else makes sense! then make it smaller - use crystals for a 3D process! START USING BLOOD TO COOL IT AND FEED IT. CALL IT MITTENS. ehm.. got carried a way there.

the days of an Octacore sitting at the helm of 128GB of RAM on a single die are soon upon us, me brethren/ettes.

for the daily joe, all of this doesn't mean much really. if anything, there's some serious improvements coming to our backbone infrastructure and laptop industries. considering improvement in lithium, this might kill off the desktop entirely, if done right.

</comment><comment><USER16> Consumer based SSD's should be avoided. They are all a big lie when it comes to write performance. Some SSD's are hardly faster than regular hard drives for write operations.

[https://blogs.technet.microsoft.com/filecab/2016/11/18/dont-...](https://blogs.technet.microsoft.com/filecab/2016/11/18/dont- do-it-consumer-ssd/)

<comment><USER17> The link talks about consumer SSDs and server workloads, makes a valid point regarding those two things (don't mix them), but doesn't say anything with regards to your comment. Also: Optane is a completely different technology. So, what's your point?

<comment><USER16> Server workloads is a bad excuse. Most browsers and a lot of applications use SQLite. Firefox and Chrome is known to write a lot more than they read.

[http://www.phoronix.com/scan.php?page=article&item=samsung-9...](http://www.phoronix.com/scan.php?page=article&item=samsung-960-evo&num=2)

<comment><USER18> That's still ten to twenty times faster than a hard drive, and browsers are not exactly slamming their history database. It's a trickle of writes, and a consumer SSD can handle it fine.

<comment><USER16> You are absolutely right that a consumer SSD can handle it just fine, just like you can surf the web fine with 2GB ram and a single core CPU. But some people pay more bucks because they want better overall performance, and it sucks to pay for a top consumer SSD model that underperform the enterprise models by a huge margin for some basic tasks. Almost everyone who review consumer SSD, with a few rare exceptions, never benchmark sync write performance / latency. Those number matters a lot more than many believe.

I've already seen to many people who use consumer drives with VMware, ZFS Intent log, SQL servers and so on, that gets frustrated because those applications are behaving a lot slower than what they expected and they feel really bad when I tell them that they have been fooled to believe that they bought "fast" SSD drives.</comment></comment></comment></comment></comment>


Indian teens make washing machine with junks, runs without power - <AUTHOR> http://www.siliconindia.com/shownews/Indian_teens_make_washing_machine_with_junks_runs_without_power-nid-71376.html

<comment><USER2> Almost no useful information can be found in the article. Basically it's about a couple of students who constructed a washing machine out of "junk material"; the machine uses "kinetic energy" to run. There is no explanation of how the kinetic energy is provided to the machine, nor what materials were used apart from a plastic bottle used as the "box to wash the clothes".

</comment><comment><USER3> It's not a perpetual-motion machine; it uses a "kinetic energy" input. Could be an electric motor, or a hand-crank.

But, then, that technology's been around for quite a while: [http://www.cleanairgardening.com/portable-washing- machine.ht...](http://www.cleanairgardening.com/portable-washing-machine.html)</comment>


Show HN: Tablight – Spotlight search for Chrome tabs - <AUTHOR> https://fouad.co/tablight/

<comment><USER2> This extension can be made without a content script and with many fewer permissions. You should look into [https://developer.chrome.com/extensions/commands](https://developer.chrome.com/extensions/commands) to see how to properly integrate keyboard shortcuts, and [https://developer.chrome.com/apps/app_window](https://developer.chrome.com/apps/app_window) for the floating UI.

Unfortunately, Chrome extensions are not allowed to use chrome.app.window.create (which you will need to make the floating UI without modifying the content page), so you will need to convert your extension into an app to use this API.

</comment><comment><USER3> That's a great idea. Two suggestions though:

\- this only seems to search the page title. As such it provides a limited value;

\- I'd love to be able to navigate through results using my keyboard;

<comment><USER3> Also, I think you should be explicit about using tracking (Google Analytics).

<comment><AUTHOR> You're right! I was in a hurry building this; just made an issue [https://github.com/fouad/chrome- tablight/issues/2](https://github.com/fouad/chrome-tablight/issues/2)

</comment></comment></comment><comment><USER4> A screenshot or demo on the page would be very valuable, so I know what's going to happen before I install!

</comment><comment><USER5> Sweet. I've been using Vimium ([http://vimium.github.io/](http://vimium.github.io/)) which has something similar but it shows bookmarks and history and what not. I wanted something to do the same but for the current tabs. Thanks.

Can you make it so I don't have to click on the dropped down tabs? Instead I want to just use my cursor to select one.

Also, I noticed if I am in a different window, it will show tabs in the dropdown for tabs in other windows but if I click them nothing happens. Is that just me, or not possible?

ESC button to exit the searching.

(Notice it's open source but don't have time right now to dig into it, but maybe in the future!)

<comment><USER6> You can search your current tabs on Vimium using shift-t.

<comment><USER7> I didn't know that either and that made me look at all the other options and features that Vimium offers. So many things I didn't know about! :)

Thanks for making me look! (it's like vim, there's always more to learn)

</comment></comment></comment><comment><USER8> Nice! For my own purposes I once started on a Chrome extension like this: TabSwitch ([https://chrome.google.com/webstore/detail/tab- switch/jibghim...](https://chrome.google.com/webstore/detail/tab- switch/jibghimhedaoojdpdianpomdmgdhogce)), that uses a Most Recently Used ordered list and searches for domain and page title.

I really like that you have the interface on top of the web page itself!

</comment><comment><USER9> Looks nice. For similar functionality I currently use Quick Tabs:

[https://chrome.google.com/webstore/detail/quick- tabs/jnjfein...](https://chrome.google.com/webstore/detail/quick- tabs/jnjfeinjfmenlddahdjdmgpbokiacbbb?hl=en)

</comment><comment><USER10> This is nice. But the shortcut you've given 'Cmd/Ctrl + O' should be 'Cmd + Ctrl + O'

<comment><AUTHOR> thanks! will add customizable shortcuts [https://github.com/fouad/chrome- tablight/issues/1](https://github.com/fouad/chrome-tablight/issues/1)

<comment><USER11> Awesome plugin and yes! please do

</comment></comment></comment><comment><USER12> Awesome, thank you!</comment>


Samsung Foundry unveiled aggressive roadmap scaling down to 4nm - <AUTHOR> https://semiengineering.com/samsung-unveils-scaling-packaging-roadmaps/ <comment><USER2> Note that semicon roadmaps nowadays are primarily a marketing tool, There's a strong disconnect with current and near future state of the art engineering capabilities. the newly spun-out Samsung foundry company is going head to head with TSMC and Intel, and they will promise anything as all parties are aiming to lock in as many as possible customers into their proprietary workflows.

</comment><comment><USER3> What are the nm even measuring anymore -- or is it just a numbering thing, so 4 is better than 10 is better than 14 is better than 22?

<comment><USER4> nm is short for nanometers. In this case, it's a measurement of the precision to which the transistors can be etched into the chips. The higher precision you have, the more complex you can make the chip.

<comment><USER3> But what is it _measuring_?

The +/\- on specified dimension, the minimum width of a feature, or....what?

<comment><USER5> Typically it's "feature pitch", or an overall scale of the various features involved in producing a transistor like width and height.

You can see more on the breakdown on pages like this: [https://en.wikipedia.org/wiki/10_nanometer](https://en.wikipedia.org/wiki/10_nanometer)

It's more a marketing term since no particular component is exactly 10nm, but it's based on a composite of the features involved.</comment></comment></comment></comment>