Ask HN: Where do I get started on ASICs, FPGA, RTL, Verilog et. al? - <AUTHOR> I actually want to understand the chip manufacturing process - design, prototyping (using FPGAs, etc), baking process in foundries. And also at least a basic understanding of how IP is managed in chip industry - like "IP core" is a term that I frequently hear but due to the myriad interpretations available online, I don't really understand what an "IP core" really means. Hoping to get useful advice from veterans in the chip industry. <comment><USER3> I wouldn't get too hung up on the phrase 'IP core', it's basically the equivalent of a software library. A reusable chunk of silicon or verilog.

If you want to know about how chips are made then I'd highly recommend the book "CMOS Circuit Design and Simulation" by Baker. It's starts off telling you how silicon is etched to make chips, then goes through how MOSFETs work and how to simulate them using SPICE. By the time you're half way through the book you'll know how a static CMOS logic gate works (down to the electrons).

If you'd rather learn something that you'll be able to apply yourself (without building a chip fab) then the place to start is Verilog (or VHDL). asic- world.com has some good tutorials. You can simulate what you've written using Icarus verilog and look at the results using GTKwave. If it works in simulation and you want to put it onto a real FPGA it's then just a matter of fighting the Xilinx/Altera/Lattice tools until then give you a bitstream. If you have enough money (a lot) you could even get a physical ASIC manufactured.

<comment><USER4> > If you want to know about how chips are made then I'd highly recommend the > book "CMOS Circuit Design and Simulation" by Baker.

This is a ~$120 book. Do you (or anyone else) have a recommendation for something a little easier to tell hobbyists they should get?

<comment><USER5> The book PDF is here: [https://www.u-cursos.cl/usuario/9553d43f5ccbf1cca06cc02562b4...](https://www.u-cursos.cl/usuario/9553d43f5ccbf1cca06cc02562b4005e/mi_blog/r/CMOS_Circuit_Design__Layout__and_Simulation__3rd_Edition.pdf)

Or buy a used second edition at $20: [https://www.amazon.com/gp/offer- listing/047170055X/](https://www.amazon.com/gp/offer-listing/047170055X/)

</comment></comment></comment><comment><USER6> Nand 2 Tetris[1] is a good starting place. You should get a good view of how the different level interact. Coursera have two cources [2][3] that cover the same material as the book

[1] [http://nand2tetris.org/](http://nand2tetris.org/) [2] [https://www.coursera.org/learn/build-a- computer](https://www.coursera.org/learn/build-a-computer) [3] [https://www.coursera.org/learn/nand2tetris2](https://www.coursera.org/learn/nand2tetris2)

<comment><USER7> I second this course. It's awesome. The next step from there is probably Coursera's VLSI course, starting with [https://www.coursera.org/learn/vlsi- cad-logic](https://www.coursera.org/learn/vlsi-cad-logic). It's all about how real-world VLSI CADs work.

<comment><USER8> Thanks! I finished Nand2Tetris and was wondering where to look for a good next step.

</comment></comment></comment><comment><USER9> Here's my collection of discussions with recommendations that I've saved just in case I decide to take a single step in this direction someday:

Open Source Needs FPGAs; FPGAs Need an On-Ramp | [https://news.ycombinator.com/item?id=14008444](https://news.ycombinator.com/item?id=14008444) (Apr 2017)

GRVI Phalanx joins The Kilocore Club | [https://news.ycombinator.com/item?id=13448166](https://news.ycombinator.com/item?id=13448166) (Jan 2017)

What It Takes to Build True FPGA as a Service | [https://news.ycombinator.com/item?id=13153893](https://news.ycombinator.com/item?id=13153893) (Dec 2016)

Low-Power $5 FPGA Module | [https://news.ycombinator.com/item?id=9863475](https://news.ycombinator.com/item?id=9863475) (Jul 2015)

</comment><comment><USER10> Its quite late here, so I'll be brief. You mentioned the words "how IP is managed in chip industry" \- so I'm going to move past the bookish knowledge and the tutorials and the open source code.

The chip design and EDA industry are very closeted and niche - there is so much knowledge there that is not part of any manual.

For example for a newcomer, you wouldn't even know what testing and validation in chip design would be - or how formal verification is an essential part of testing.

You wouldn't know what synthesis is, what is place and route, and GDS masks for foundries.

There is seriously no place to learn this. The web design or the AI world works very different - you can be a very productive engineer through Udacity. Not with ASIC.

You need to find a job in the chip design or EDA industry. There is seriously no other way.

If I had to make a wild parallel - the only other industry that works like this are people who make compilers for a living. Same technology, same problems, similar testing steps I guess.

<comment><USER11> I think your compilers example doesn't fit. They are actually a rather straightforward thing to build, many undergraduates build one as part of their studies, some hobbyists build compilers that get used by thousands of people and in fortune 500 companies core infrastructure. There's little rigor involved.

Among the few software systems that need rigor are control systems for physical installations and trading/finance systems for example.

<comment><USER12> Also many production-grade compilers (GCC/G++, Clang, OpenJDK, V8, and almost every new language that's come out since the 90s) are open-source. You can go read the commit logs & source code to see how they work, if you're diligent and willing to slog through them. There are certainly tricks that professional compiler writers use that aren't covered in textbooks (the big ones center around error-reporting, incremental compilation, fancy GC algorithms, and certain optimizations), but you can always go consult the source to learn about them.

I thought the thread was really about domains where the bulk of knowledge is locked up in industry rather than being about rigor, but I'd put control systems in that category as well. Also information retrieval (Google's search algorithms are about 2 decades ahead of the academic state-of-the-art...the folks at Bing/A9/Facebook know them too, but you aren't going to find them on the web), robotics, and aerospace.

<comment><USER10> im generally talking intel c++ compilers, etc. When I was doing EDA, we used to fork out a lot of cash for these compilers and these guys used to work very closely with us to optimize for certain kinds of code styles - for example loop unrolling on HP-UX, etc. I dont know if this is still a thing, so i might be mistaken.

</comment></comment></comment></comment><comment><USER13> As a veteran from the chip industry, I should warn you that all these suggestions about FPGAs for prototyping are not really done that much in the ASIC industry.

The skills to do front end work are similar but an ASIC design flow generally doesn't use an FPGA to prototype. They are considered slow to work with and not cost effective.

IP cores in ASICs come in a range of formats. "Soft IP" means the IP is not physically synthesised for you. "Hard IP" means it has been. The implications are massive for all the back end work. Once the IP is Hard, I am restricted in how the IP is tested, clocked, resetted and powered.

For front end work, IP cores can be represented by cycle accurate models. These are just for simulation. During synthesis you use a gate level model.

<comment><USER4> As a veteran from the chip industry, I can tell you my experience is completely the opposite.

Nobody in their right mind would produce an ASIC without going through simulation as a form of validation. For anything non-trivial, that means FPGA.

<comment><USER13> I don't agree. If it's non trivial, I don't have the more advanced verification tools such as UVM if I prototype via FPGA.

The ability to perform constrained randomised verification is only workable via UVM or something like it. For large designs that is arguably the best verification methodology. Without visibility through the design to observe and record the possible corner cases of transactions, you can't be assured of functional coverage.

While FPGAs can run a lot more transactions, the ability to observe coverage of them is limited.

I have worked on multiple SoCs for Qualcomm, Canon and Freescale. FPGAs don't play a role in any SoC verification that I've worked on.

<comment><USER14> That's a false dichotomy -- you can do FPGA verification in addition to simulation-based verification. And yes, there are ASIC teams that have successfully done that.

<comment><USER13> At the SoC level, I don't think so.

The reasons are numerous. I already gave a few. I will give another. Once you have to integrate hard IP from other parties, you cannot synthesise it to FPGA. Which means you won't be able to run any FPGA verification with that IP in the design. You can get a behavioural model that works in simulation only. In fact it is usually a requirement for Hard IP to be delivered with a cycle accurate model for simulation.

I'll give another reason. If you are verifying on FPGA you will be running a lot faster than simulation. The Design Under Test requires test stimulus at the speed of the FPGA. That mans you have to generate that stimulus at speed and then check all the outputs of the design against expected behaviour at speed. This means you have to create additional HW to form the testbench around the design. This is a lot of additional work to gain speed of verification. This work is not reusable once the design is synthesised for ASIC.

I can go on and on about this stuff. Maybe there are reasons for a particular product but I am talking about general ASIC SoC work. I got nothing against FPGAs. I am working on FPGAs right now. But real ASIC work uses simulation first and foremost. It is a dominant part of the design flow and FPGA validation just isn't. On a "Ask HN", you would be leading a newbie the wrong way to point to FPGAs. It is not done a lot.

<comment><USER15> As another veteran in the ASIC industry: we are using FPGAs to verify billion transistor SOCs before taping out, using PCBs that have 20 or more of the largest Xilinx or Altera FPGAs.

It's almost pointless to make the FPGA run the same tests as in simulation. What you really want is to run things that you could never run in simulation. For example: boot up the SOC until you see an Android login screen on your LCD panel.

A chip will simply not tape out before these kind of milestones have been met, and, yes, bugs have been found and fixed by doing this.

The hard macro IP 'problem' can be solved by using an FPGA equivalent. Who cares that, say, a memory controller isn't 100% cycle accurate? It's not as if that makes it any less useful in feeding the units that simply need data.

</comment></comment></comment></comment></comment></comment><comment><USER16> There are three things here that you've intertwined.

 _Process_ \-- This is the science of creating circuits on silicon wafers using lithography, etching, and doping. There is a large body of knowledge around the physics involved here. Materials science and Physics and Silicon Fabrication are all good places to start.

 _Chip Design_ \-- This is creating circuits which are run through a tool that can lay them out for you automatically. HDLs teach you to describe the semantics of the hardware in such a way that a tool can infer actual circuits. Generally a solid understanding of digital logic design is a pre-requisite, and then you can learn the more intimate details of timing closure, or floor planning, signal propagation and tradeoffs of density and speed.

 _IP_ \-- Clearly all of the intellectual property law is a huge body but most of the IP around chips is patent law (how the chips are made) and copyright law (how they are laid out).

<comment><USER17> Yes that's true, but there's actually even more like packaging, testing, etc. I took a free online course from Stanford called "nanomanufacturing" but it really was mostly about about chip manufacturing, packaging etc. Even though I worked in the semiconductor industry for 12 years (mostly bench testing preproduction ASICS) I still found it really useful. Not sure if you can still view the archives here if you sign up for an account (I can but I took the class)

[https://lagunita.stanford.edu/courses/Engineering/Nano/Summe...](https://lagunita.stanford.edu/courses/Engineering/Nano/Summer2014/info)

No substitute for learning the physics, but at least it kind of gives you some idea of what's involved. In addition to all the crazy technology involved in fabricating the chips, the packaging technology has gotten really sophisticated. It can be very confusing about what's the difference BGA, WLCSP, stacked dies, etc. Anyway the course covered a lot different types of processing with examples.

<comment><USER16> That is awesome. The link doesn't work for me but I didn't really expect it to. My first job in the Bay Area was working for Intel and about 6 months in I was offered some 'counterfeit' or grey market Intel DRAM chips. (as a microcomputer enthusiast, not as an Intel employee) I took the offer to security, who gave me the cash to buy a tube of them, which I did, and they disassembled them to figure out where in the packaging pipeline they had gone missing.

Sadly I never got to hear the full story on how they came to be but I did get a good look at the packaging pipeline that Intel used at the time. It was extensive even then with half a dozen entities providing steps in the path.

</comment></comment></comment><comment><USER18> An ASIC is pretty expensive unless you've got Google money [1]. Start with an FPGA dev board [2] and probably just stick with FPGA. Hell, Amazon has an FPGA instance [3]:

[1] [https://electronics.stackexchange.com/questions/7042/how- muc...](https://electronics.stackexchange.com/questions/7042/how-much-does-it- cost-to-have-a-custom-asic-made)

[2] [https://www.sparkfun.com/products/11953](https://www.sparkfun.com/products/11953)

[3] [https://aws.amazon.com/ec2/instance- types/f1/](https://aws.amazon.com/ec2/instance-types/f1/)

<comment><USER3> You can actually get an ASIC manufactured for a few thousand dollars via CMP or Europractice. So not quite Google money. The difficulty is in paying for the software licenses you need to go from Verilog to DRC checked GDSII files (which is what you need to send to them).

In fact personally I think this is a much better route for open source hardware. Reverse engineering FPGA bitstreams impressive, but you're swimming against the tide. If we had good open source tooling for synthesis/place-and- route/DRC checking and good open source standard cell libraries (and these things exist, eg. qflow, they're just not amazing currently) then truly open source hardware could actually be a thing. Even on 10 year old fabs you'll get much better than you could do on an FPGA (you just have to get it right first time).

<comment><USER19> I'm very interested in this. I read that one small design (which I knew was very small, but not quantitatively so) cost approximately $5k per small run.

How is the cost calculated? I presume size of final wafer (ie, number of chips produced) at least; does transistor count per chip influence anything too?

Finally, is it _possible_ to produce and maintain a fully open-source design that's the chip-fab equivalent of the book publishing industry's "camera-ready copy"? I get the idea that this is specifically where things aren't 100% yet, but, using entirely open tools, can you make something that is usable?

<comment><USER3> The cost depends on the technology. More modern processes are dramatically more expensive (don't even think about something like 28nm). If you're looking at something ancient like 130nm, 0.35um, etc then it's going to be something like $1-2k per mm^2. Normally there's a minimum cost of a few mm^2. Expect to get a few tens of chips back. Transistor count has no effect, they couldn't care less what you put inside your part of the wafer, so long as it follows the design rules (for example minimum % coverage on each layer to prevent it collapsing vertically), but obviously if you need more transistors you might need more area.

Yes it's 100% possible to do an open-source design, qflow has been used to make sub-circuits of ASICs, but it's going to be extremely difficult. There are lots of things missing which you'd have to take from the fabs PDK or design yourself, for example open source I/O pads (sounds boring, but actually lots of work with ESD, etc). Combined with huge missing feature-sets in the open source tools, like extraction of designs back to SPICE circuits with parasitics and complete DRC checking, you're not going to have a fun time.

<comment><USER19> Hmm, I see. Thanks for this info, and particularly the bit about the pricing.

</comment></comment></comment></comment></comment><comment><USER20> > And also at least a basic understanding of how IP is managed in chip > industry - like "IP core" is a term that I frequently hear but due to the > myriad interpretations available online, I don't really understand what an > "IP core" really means.

I'm by no means a veteran but my understanding is that "IP core" refers to a design you buy from someone else. Say you want a video encoder on your smart fridge SoC. You can either spend a whole lot of time, manpower and money developing one yourself or you can license the design from someone else who already has one and just dump it in.

You'd only do this when you want to integrate the design into your own (likely mass manufactured) chip. You can also often buy a packaged chip that serves the same function for much less but doing that is a tradeoff. You can do it at very low volume and cost but you potentially lose a bunch of efficiency in terms of space and power.

<comment><USER21> That is my understanding as well. In a commercial setting, reinventing the wheel is economically a bad idea. For the company licensing the IP core, the licence revenues are another form of return on investment for the design effort. Companies, like ARM, are "fab-less", i.e. they create IP cores and license them to semi manufacturers.

</comment></comment><comment><USER19> I've gotten curious about FPGAs myself of late, particularly with video capture. A hopefully-on-topic question of my own, if I may:

I've seen that some FPGA boards have HDMI transceivers that will decode TMDS and get the frame data into the FPGA somehow. That got me thinking about various possibilities.

\- I want to build a video capture device that will a) accept a number of TMDS (HDMI, DVI, DisplayPort) and VGA signals (say, 8 or 10 or so inputs, 4-5 of each), simultaneously decode all of them to their own framebuffers, and then let me pick the framebuffer to show on a single HDMI output. This would let me build a video switcher that could flip between channels with a) no delay and b) no annoying resyncs and c) because everything's on independent framebuffers I can compensate for resolution differences (eg, a 1280x1024 input on the 1920x1080 output) via eg centering.

\- In addition to the above, I also want to build something that can actually _capture_ from the inputs. It's kind of obvious that the only way to be able to do this is via recording to a circular window in some onboard DDR3 or DDR4 (256GB would hold 76 seconds of 4K @ 144fps). My problem is actually _dumping_/saving the data in a fast way so I can capture more input.

I can see two ways to build this

1) a dedicated FPGA board with onboard DDR3, a bunch of SATA controllers and something that implements the equivalent of RAID striping so I can parallelize my disk writes across 5 or 10 SSDs and dump fast enough.

2) A series of FPGA cards, each which handles say 2 or 3 inputs, and which uses PCI bus mastering to write directly into a host system's RAM. That solves the storage problem, and would probably simplify each card. I'd need a fairly beefy base system, though; 4K 144fps is 26GB/s, which is uncomfortably close to PCI-e 3.0 x16's limit of 32GB/s.

I'll admit that this mostly falls under "would be really really awesome to have"; I don't have a commercial purpose for this yet, just to clarify that. That said, my inspiration is capturing pixel-perfect, perfectly-timed output from video cards to identify display and rendering glitches (particularly chronologically-bound stuff, like dropped frames) in software design, so there's probably a market for something like this in UX research somewhere...

<comment><USER22> The capture thing appears to exist: [https://www.blackmagicdesign.com/products/hyperdeckshuttle/](https://www.blackmagicdesign.com/products/hyperdeckshuttle/) \- presumably it applies lossless compression. Lossless encoding within the h264 container should be possible.

<comment><USER23> That Blackmagic stuff is really awesome. Expensive as ..., but worth every penny.

<comment><USER9> Pardon my ignorance, but I thought Blackmagic's claim to fame was always being on the low end of the pricing for their tech.

</comment></comment></comment></comment><comment><USER24> Regarding the ASIC part: There are several good courses online that explain the whole process from hardware description in a hardware description language to GDSII file, which you could send to a foundry in some detail. See for example this course [https://web.csl.cornell.edu/courses/ece5745/](https://web.csl.cornell.edu/courses/ece5745/), there are also very good courses available from Berkeley and MIT. What is usually missing is the more gory details of the backend flow, which can become very involved and complicated depending on your design and process.

Unfortunately someone who has no access to the EDA tools by cadence / synopsis or standard library files from foundries, cannot really follow along all that far, you are limited to working at the RTL level.

There are several good open source RTL simulators available, I have personally used mostly verilator, which supports (almost) all synthesizable constructs of system verilog and has performance close to the best commercial simulators like vcs. It compiles your design to C++ code which you can then wrap in any way you like.

You should also check out [https://chisel.eecs.berkeley.edu/](https://chisel.eecs.berkeley.edu/), which is a hardware description language embedded in scala, the nice thing about it is that it has a relatively large number of high quality open source examples, designs ([https://github.com/freechipsproject/rocket- chip](https://github.com/freechipsproject/rocket-chip)) and a library of standard components available, something which can't really be said of verilog / vhdl unfortunately. As an added bonus you can actually use IntelliJ as an IDE, which blows any of the commercial IDEs available for system verilog or vhdl out of the water.

Another thing I can recommend is to get yourself a cheap FPGA board, some of them are programmable purely with open source tools, see [http://www.clifford.at/icestorm/](http://www.clifford.at/icestorm/). Alternatively the Arty Devkit comes with a license for the Xilinx Vivado toolchain.

<comment><USER25> I wouldn't start with Chisel as there's a lack of good documentation online. When I used it for a Berkeley class, sometimes you would feel like you hit a wall. Verilog or SystemVerilog will have much more in the way of stack overflow type of documentation.

</comment></comment><comment><USER26> For what its worth a lot time working towards Carnegie Mellon's undergraduate (and graduate) degree in computer engineering revolved around verilog and FPGAs and ASICs. After learning all sorts of principles and design skills we learned verilog so we can actually build decent size projects and simulate them. Then we would build real world projects with an FPGA and then the advances classes has us designing and simulating ASICs. Then the really advanced classes has us studying manufacturing ASICs.

</comment><comment><USER27> I wanted to learn about Verilog development and to get a better understanding of what's happening on chips. To that end I bought a MiST FPGA-based computer: [https://github.com/mist-devel/mist-board/wiki](https://github.com/mist- devel/mist-board/wiki)

It's Altera Cyclone III-based. The MiST wiki says the free "web edition" of the Altera "Quartus II" development environment is sufficient to develop for the unit (albeit I haven't actually gotten around to doing anything with it yet).

I can't say how the MiST board stacks-up to dev boards from FPGA manufacturers. I may be going about this the most wrong way possible, but here was my rationale: I was attracted to MiST because tutorials were available for it ([https://github.com/mist-devel/mist- board/tree/master/tutoria...](https://github.com/mist-devel/mist- board/tree/master/tutorials)), and because the device could be usable as a retro-computing platform if it ended up being unusable for me for anything else. (Chalk that up to rationalization of the purchase, I guess.)

</comment><comment><USER28> First of all you must understand logic gates, flip-flops (D-FF, T-FF and so on) and multiplexers. All of them are built on logic gates. To verify that you really understand the concept, try to implement a digital clock. (This one: [https://sapling-inc.com/wp-content/gallery/digital-clocks- fo...](https://sapling-inc.com/wp-content/gallery/digital-clocks-for-sbl-sbt- sbw/Sapling-404-Wall-Mount-White-1230-H.jpg))

Logic gates are essential of learning digital circuits. After you understand logic gates, you can use it to build many things that are really related to the application. There are many tools to verify the logic gates works as designed.

Then based on the project requirement ($$, time, performance, ...), you can choose to use FPGA or ASIC to implement logic gates. FPGA use array of logic gates while ASIC uses CMOS to implement the logic gates. FPGA is easier to learn and much cheaper. You can buy some development board which costs only several hundred dollars. While ASIC needs much domain knowledge and people involved. ASIC needs you to understand the electronics in order to build something that is useful. You need to understand how the CMOS are implemented (= how semiconductor becomes conductive), how the resistance and capacitance affect the performance, the number of wafer layout which affects the cable layout and more. And don't forget manufacturing can introduce defeats which cause the IC to malfunction in unexpected ways. Each step in ASIC needs a specialist for them

<comment><USER29> For 5 seconds I was like "but the clock is the analog part of the circuit, this does not make any sense". Then I clicked the link.

</comment></comment><comment><USER30> You may find the book _Contemporary Logic Design_ by Katz and Borriello to be interesting. Its what we used in my college digital logic class.

For my computer architecture class (i.e. the class where we learn how to design basic CPUs -- and ultimately implemented one in Verilog), we used _Computer Organization and Design_ by Patterson and Hennessy. That might also be of interest.

</comment><comment><USER31> You can start by buying a Papillon board and going through the free range VHDL book. That's the high level.

When you want to understand CMOS and integrated circuits, you need some electronics experimenter kit, and a lot of practice in ohms law. Then read up on multiple gate transistors and (here my experience stops) lithography and small scale challenges (tunneling loss is a thing, I suppose?)

Of course, "ip core" can mean different things, might be some Verilog source, might be some netlists, might be a hard macro for a particular process. You really need to work with it to get the specifics. (Subscribing to EETimes, going to trade shows, and otherwise keeping up might help)

But at the end of the day, you're asking "how can I become and experienced ASIC engineer," and the truth is that it takes time, education, and dedication.

<comment><USER32> The board you're thinking of is the Papilio. The Papillon is a dog breed. :)

</comment></comment><comment><USER33> I studied an MEng in Electronic Systems Engineering, and really enjoyed the courses in IC design.

However, I couldn't find a chip-design job in a country other than the US or UK that wasn't related to military applications.

Now I work in Taiwan, and I see the chips being made! But my work is related to control systems for the testing equipment, which is software instead of hardware design.

I got a Virtex-II FPGA board from a recycling bin, and I wanted to find a good personal project for it. Even now, I'm at a loss for ideas. I can do everything I need with a Raspberry Pi already.

Please can someone suggest some good projects I could only do with an FPGA?

<comment><USER34> You could see what the folks at Hackaday.io are doing.

[https://hackaday.io/list/3746-programmable-logic- projects](https://hackaday.io/list/3746-programmable-logic-projects)

</comment></comment><comment><USER35> Opencores.org has a very large collection of opensource IP cores.

</comment><comment><USER36> [http://www.clifford.at/icestorm/](http://www.clifford.at/icestorm/)

open source FPGA workflow. ASICs is more tough

</comment><comment><USER37> I'd start with learning a hardware description language and describing some hardware. Get started with Verilog itself. I'm a fan of the [Embedded Micro tutorials]([https://embeddedmicro.com/tutorials/mojo](https://embeddedmicro.com/tutorials/mojo)) -- see the links under __Verilog __Tutorials on the left (they 're also building their own HDL, which unless you own a Mojo board isn't likely of interest). Install Icarus Verilog and run through the tutorials making sure you can build things that compile. Once you get to test benches, install Gtkwave and look at how your hardware behaves over time.

You can think of "IP cores" as bundled up (often encrypted/obfuscated) chunks of Verilog or VHDL that you can license/purchase. Modern tools for FPGAs and ASICs allow integrating these (often visually) by tying wires together -- in practice you can typically also just write some Verilog to do this (this will be obvious if you play around with an HDL enough to get to modular design).

Just writing and simulating some Verilog doesn't really give you an appreciation for hardware, though, particularly as Verilog can be not- particularly-neatly divided into things that can be _synthesized_ and things that can't, which means it's possible to write Verilog that (seems to) simulate just fine but gets optimized away into nothing when you try to put it on an FPGA (usually because you got some reset or clocking condition wrong, in my experience). For this I recommend buying an FPGA board and playing with it. There are several cheap options out there -- I'm a fan of the [Arty]([http://store.digilentinc.com/arty-a7-artix-7-fpga- developmen...](http://store.digilentinc.com/arty-a7-artix-7-fpga-development- board-for-makers-and-hobbyists/)) series from Digilent. These will let you play with non-trivial designs (including small processors), and they've got lots of peripherals, roughly Arduino-style.

If you get that far, you'll have discovered that's a lot of tooling, and the tooling has a lot of options, and there's a lot that it does during synthesis and implementation that's not _at all_ obvious. Googling around for each of the phases in the log file helps a lot here, but given what your stated interest is, you might be interested in the [VLSI: Logic to Layout]([https://www.coursera.org/learn/vlsi-cad- logic](https://www.coursera.org/learn/vlsi-cad-logic)) course series on Coursera. This talks about all of the logic analysis/optimization those tools are doing, and then in the second course discusses how that translates into laying out actual hardware.

Once you've covered that ground it becomes a lot easier to talk about FPGAs versus ASICs and what does/doesn't apply to each of them (FPGAs are more like EEPROM arrays than gate arrays, and for standard-cell approaches, ASICs look suspiciously like typesetting with gates you'd recognize from an undergrad intro-ECE class and then figuring out how to wire all of the right inputs to all of the right outputs).

Worth noting: getting into ASICs as a hobby is prohibitively expensive. The tooling that most foundries require starts in the tens-of-thousands-per-seat range and goes up from there (although if anyone knows a fab that will accept netlists generated by qflow I'd love to find out about it). An actual prototype ASIC run once you've gotten to packaging, etc. will be in the thousands to tens of thousands at large (>120nm) process sizes.

<comment><USER38> If i correctly understand what you are saying, it cold pe possible to make a custom small chip for doing some crypto capable of a little more than what those smartcards offer for under 10k$? That would be awesome, from a trust perspective, at least if you could realistically compare the chip you get back with what you know to expect, using an electron microscope.

<comment><USER37> Not for an ASIC without spending a LOT on tooling, and really $10k is awfully optimistic even if you had all of that tooling (I probably should've just said tens of thousands).

For <100k, yes, you can absolutely do a small run in that range.

Honestly, you might be better off just buying functional ICs (multi-gate chips, flip flops, shift registers, muxes, etc.) and making a PCB, though. Most crypto stuff is small enough that you can do a slow/iterative solution in fairly small gate counts plus a little SRAM.

<comment><USER31> If you do that, why wouldn't you use a FPGA or just a fast CPU? Microcontrollers and CPUs are blending in performance, and are cheap enough to plop on a board and call it done for many applications.

<comment><USER37> Sure, but if really want to avoid trusting trust (and you're of the mind to build your own hardware), FPGAs and µcs offer a lot of room for snooping.

Given the GPs suggested use, it seemed trusting trust was not on the table.

Certainly even a tiny FPGA can fit pretty naïve versions of common crypto primitives, as can any modern micro-controller. Assuming you only need to do a handful of ops for whatever you're looking to assert/verify, that is by far simpler than building a gate-level representation :)

<comment><USER38> I was thinking about a chip with only sram for secret storage that could be bundled into a ID-1 sized card with some small energy storage for the sram (there are affordable .5mm LiPo Cells that fit inside such a card), and then use the card to fit some display capable of giving some little data out, as well as a touch matrix,possibly by just using a style similar to carbon- contacts on cheap rubber membrane keyboards, but gold plated like the smartcard interface. But it seems like you can't afford to store one decompressed ed25519 or dare rsa, so the idea is moot by virtue of requiring sub-100nm technology to fit at least some sram.

</comment></comment></comment></comment></comment></comment><comment><USER39> You can start at the beginning of VLSI revolution and read from the horses mouth 'Introduction to VLSI Systems'. If you are still serious about it get into MITs EECS.

Btw you ask about 4 almost totally separate areas(RTL UVM TB etc), only managers/execs/veterans/architects know the whole process from raw silicon to packaging.

</comment><comment><USER40> Shameless friend-promotion: [http://tinyfpga.com](http://tinyfpga.com)

I spoke to the creator today and he's planning a tutorial/example IP series - probably open to suggestions if there's anything you're particularly interested in.

</comment><comment><USER41> Check out EDA playground. You can run verilog on their web interface and can bring up waveforms from simulations.

[https://www.youtube.com/user/edaplayground](https://www.youtube.com/user/edaplayground)

</comment><comment><USER42> You just described a few different sub-fields of computer engineering:

1\. Processes, which involves lots of materials science, chemistry, and low- level physics. This involves the manufacturing process, as well as the low- level work of designing individual transistors. This is a huge field.

2\. Electrical circuits. These engineers use specifications given to you by the foundry (transistor sizes, electrical conductance, etc.) and using them to create circuit schematics and physically laying out the chip in CAD. Once you finish, you send the CAD file to the group from #1 to be manufactured. Modern digital designs have so many transistors that they have to be laid out algorithmically, so engineers spend lots of time creating layout algorithms (called VLSI).

3\. Digital design. This encompasses writing SystemVerilog/VHDL to specify registers, ALUs, memory, pipelining etc. and simulating it to make sure it is correct. They turn the dumb circuit elements into smart machines.

It's worth noting that each of the groups primarily deals with the others through abstractions (Group 1 sends a list of specifications to group 2, Group 3 is given a maximum chip area / clock frequency by group 2), so it is possible to learn them fairly independently. Even professionals tend to have pretty shallow knowledges of the other steps of the process since the field is so huge.

I'm not super experienced with process design, so I'll defer to others in this thread for learning tips.

To get started in #2, the definitive book is the Art of Electronics by Horowitz & Hill. Can't recommend it enough, and most EEs have a copy on their desk. It's also a great beginner's book. You can learn a lot by experimenting with discrete components, and a decent home lab setup will cost you $100. Sparkfun/Adafruit are also great resources. For VLSI, I'd recommend this coursera course: [https://www.coursera.org/learn/vlsi-cad- logic](https://www.coursera.org/learn/vlsi-cad-logic)

To learn #3, the best way is to get a FPGA and start implementing increasingly complicated designs, e.g. basic logic gates --> counters --> hardware implementation of arcade games. This one from Adafruit is good to start: [https://www.adafruit.com/product/451?gclid=EAIaIQobChMIhKPax...](https://www.adafruit.com/product/451?gclid=EAIaIQobChMIhKPaxajQ1gIVnbbACh1rkgZSEAQYASABEgKKcvD_BwE), though if you want to make games you'll need to pick up one with a VGA port.

Silicon design & manufacturing is super complicated, and I still think that it's pretty amazing that we're able to pull it off. Good luck with learning!

(Source: TA'd a verilog class in college, now work as an electrical engineer)

<comment><USER43> There's also the field of "Computer Architecture", which is a level above the ones you've described.

</comment></comment><comment><USER44> Get yourself an FPGA devkit and start making little hardware bits

<comment><USER45> What's the best way for a novice to determine what their needs are? One thing that's been a barrier to entry for me has been fear of vendor lock-in from point of sale with regards to upgrading in the future; ie: purchasing a starter kit by company X but once determining that only company Y supports the feature set. Where there would then require a non-trivial amount devtime would be sunk in re-tooling your code both across hardware and design environments. I was originally really excited to see AWS hosting FPGA instances, but a friend told me that they were charging a heavy premium and only had a limited number of manufacturers.

<comment><USER37> There are basically two companies in this business, Altera (Intel) and Xilinx.

I would not worry about vendor lock-in for now -- there are some quite affordable dev boards (like the Arty ([http://store.digilentinc.com/arty-a7-artix-7-fpga- developmen...](http://store.digilentinc.com/arty-a7-artix-7-fpga-development- board-for-makers-and-hobbyists/)) I've mentioned elsewhere), and no matter what you pick there's a ton of tooling. The concepts from the tools will translate between vendors, though, even if the commands and exact flows change.

<comment><USER46> To add to this:

Xcell is the name of Xilinx's self-published journal(s), and it's free. Here's the Xcell portal:

[http://www.xilinx.com/about/xcell- publications.html](http://www.xilinx.com/about/xcell-publications.html)

Here's a short article published in Xcell by Niklaus Wirth after he designed and prototyped a RISC CPU on a Xilinx FPGA:

[https://issuu.com/xcelljournal/docs/xcell_journal_issue_91/3...](https://issuu.com/xcelljournal/docs/xcell_journal_issue_91/30)

And finally, it's worth noting that the most recent post on the Xcell blog is one titled "Found! A great introduction to FPGAs", which was written just a couple days ago and is a glowing recommendation for the textbook "Digital System Design with FPGA: Implementation Using Verilog and VHDL".

[https://forums.xilinx.com/t5/Xcell-Daily-Blog/Found-A- great-...](https://forums.xilinx.com/t5/Xcell-Daily-Blog/Found-A-great- introduction-to-FPGAs-Digital-System-Design-with/ba-p/797803)

</comment></comment></comment></comment><comment><USER47>
      This might help you:
      MOSIS Integrated Circuit Fabrication Service

[https://www.mosis.com](https://www.mosis.com)

</comment><comment><USER48> Okay, okay, hang on now, how a chip is manufactured and the design flow are really two different things.

If you could tell us your background it might be helpful to get started.

</comment><comment><USER49> This is a huge topic. You can spend your entire career in just each part of your question - fabrication in a foundry, FPGA synthesis, HDL design, ASIC place and route, etc..

I've actually done all-of-the-above, from making SOI wafers to analog circuit design for CMOS image sensors to satellite network simulation in FPGAs to supercomputer architecture and design to XBox GPU place-and-route.

It will honestly take you at least a few years to be able to understand all of this, and I can't even begin to tell you where to begin.

My track started with semiconductor fabrication processing in college - <USER2> of chemistry, lithography, physics, design-of-experiments, etc.. I guess that's as good a start as any. But before that I did get into computer architecture in high-school, so that game me some reference goals.

What are you ultimately trying to do? Get a job at a fab? That's a lot of chemistry and science. Do you want to design state-of-the-art chips, as your IP-core question hints at? That's largely an EE degree. Do you want to build a cheap kickstarter product, as your FPGA question suggest? That's EE and Computer Engineering as well.</comment>


What is this colored fiber in my chicken? - <AUTHOR> http://cooking.stackexchange.com/questions/79674/what-is-this-colored-fiber-in-my-chicken <comment><USER2> Fake meat cannot come soon enough - poor bird was encouraged to grow in an unhealthy manner resulting in dead tissue inside it while it was still alive. I wonder if it was painful for the bird having this tough dead tissue at the core of its breasts.

<comment><USER3> This is just the tip of the iceberg in how animals are treated. Industrial meat production practically runs its margins on increasing the suffering of animals in a manner that narrowly avoids this kind of visible anomalies in the product.

As many others have noted in the past, slave labor and concentration camps operated on a similar industrial calculus of forcibly squeezing out value from living beings before putting them to death.

Waiting for "fake meat" to fix the problem is like saying: "I can't wait until we can grow zombies to pick cotton, then all the slaves can finally go free."

<comment><USER4> Not if we're talking about advances in textured vegetable protein. When I read "fake meat" I take it to mean that it's not animal derived. Otherwise it'd just be meat.

<comment><USER5> Still, we can do something about it right now. We may need meat, but we don't need meat in every meal; if we reduced the consumption of meat, we could conceivably switch to more expensive, less efficient, more humane ways to grow animals.

If you can afford it, you can also only buy meat from non-intensive farming. Or stop eating meat all together.

I just don't think it's morally right to wait for a solution that may not ever come, or not for a long time. I eat veggie fake-meat often, and sometimes it's great, but it's not close to real meat and I don't know that it'll ever be accepted by most of the population.

<comment><USER6> "We may need meat, but we don't need meat in every meal"

Is it still a common belief that we 'need meat'? I thought it was well established that humans in normal health do not _need_ meat to thrive - especially with our current technology and industry. We can easily meet all of our nutritional needs without any animal flesh.

<comment><USER5> I agree we don't need to eat meat, and I'm a vegetarian, but I'm around... 80% sure.

I'd rather use the less-strong statement I need to make a point. It is still a common belief that not eating meat is in some way less healthy, because plants have lower-quality proteins.

</comment></comment></comment></comment></comment></comment><comment><USER7> Animal welfare aside, I find people in North America really in favour of chicken breast, much more than other parts, say chicken thigh. Yet I myself think chicken thigh tastes much better, especially with the skin (yet again it is usually skinless in supermarkets here, unfortunately). Is it because of nutrition (percentage of fat/protein etc.)? On top of that, chicken feet are considered unacceptable by many...

<comment><USER8> On the bright side: I live in the US, I love the "dark"/fatty meat on chickens, and I don't mind that everyone else here prefers "white"/dry chicken meat because that means a) chicken thighs are often much less expensive, and b) I rarely have to fight over the thighs and other fatty meat when sharing a full bird.

Edit to add: As for _why_ other people prefer white meat, people usually invoke that it tastes better, but I've always figured that this was an _acquired_ taste and that decades of pressure to reduce fat in our diets pushed people continually over to breast meat.

<comment><USER9> It's the effect of decades of advertisement. We've been duped into thinking that white meat is normal and good, when white meat in chicken breast shows a sedentary existence. Truly free range chickens do not have breast meat as white as those from factory farms.

It's the same way we've been duped into believing that we need to eat that bowl of cereal in the morning.

<comment><USER10> > We've been duped into thinking that white meat is normal and good

It's also why McDonald's McNuggets are now made with "all white meat" \- except it isn't breast meat. Its a mixture of meat and meat by-products that have been blended and bleached. It has a mouth-feel unlike either dark or white meat chicken; it's a unique combo (I am not against McDs or their nuggets, btw).

I remember when McNuggets first came out - there were three distinct shapes. Only the "round" shape was white meat - and actual all-breast meat. The other two shapes were dark meat. Today, there are only two of the same three shapes that I recall, and both are of the blended "white" meat.

Which is a shame - I liked the variety more before.

</comment></comment></comment></comment><comment><USER11> This is not a problem caused by antibiotics. Is a problem of genetic nature that happens because broilers are inbreed for growing big and fast. The same birds in true range with plenty of food would face exactly the same problem, with or without antibiotics. They are too heavy and often tend to have cardiac diseases, but they live short lives and are delicious so they are the most sucessful bird in the planet.

On the other hand, we are a paradoxal species. Able to feel horrified by this, whereas happily petting our distorted-faced bulldogs, persian cats, caesarean born bullterriers, extra-dwarfed toy Yorkshires, ponies and toy mini pigs, without any trace of moral conflict...

<comment><USER12> "the most successful bird on the planet" is an interesting line considering that they are all killed and eaten.

I guess the obvious line of reasoning is to talk about how GDP is less than perfect for measuring economic success, etc. Sometimes large numbers are not a sign of success.

<comment><USER11> Not all are killed. Millions of captive chickens live much longer and peaceful lives than its wild counterparts (They do not need to care for the danger of being burned alive in an indonesian fireforest for example). Many laying and pet hens die for natural causes after ten years or so roaming around some garden or farm. This is a lot of sun-bathing, clucking and scratching time to enjoy for a middle sized bird with plenty of predators that normally would not reach its first birdday. Even broilers live longer than most wild chicks.

<comment><USER13> Agreed. Folks forget that the wild population, if stable, implies that _almost all hatched chicks are horribly killed and eaten by predators in their first year_.

Of course the moral issue of "should I cynically raise billions of birds because they're tasty" is real.

</comment></comment></comment></comment><comment><USER14> Brown meat is much much more delicious; why would anyone prefer white meat is beyond me.

And the irony is that, in buying "heavy breasted chicken", customers pay for something they can't consume (assuming chicken is priced by the pound in the US).

<comment><USER15> Maybe because studies in the past two years suggest that red meat is actually quite dangerous with frequent consumption.

<comment><USER16> No they haven't. Red meat is not dangerous. Cooking red meat the way Americans tend to (disgustingly fried everything) is, but disgustingly frying anything is bad for you.

<comment><USER17> People fry red meat ?!

I have spent much time in the US, but I think I have never seen that...in restaurants at least. And I'm kinda glad I didn't.

<comment><USER18> It's not as common, but yes. One example found in Southern cuisine / Texas / etc. is called either chicken fried steak or country fried steak. Although often pan fried, some restaurants deep fry them.

<comment><USER19> With mashed potatoes and white gravy[1], it's delicious, too.

[1] People who put brown gravy on chicken fried steak are heretics who should be deep fried themselves.

</comment></comment></comment></comment></comment></comment><comment><USER20> People just eat way too much chicken. The numbers of chicken consumed every year in most western nations is astounding. And those poor birds, the way they are packed to the point of not even being able to walk while they are raised, it's really a lot more disgusting than the final product shown in this article.

<comment><USER21> Do you think red meat is better? I've been trying to avoid red meat lately for health reasons, and it is very difficult because for many restaurant major of the items seem to contain red meat. I end up eating lots of chicken and fish.

Going vegetarian would be probably best...

<comment><USER22> Chicken is also significantly better from a carbon emissions perspective too I believe.

Going off [http://media.treehugger.com/assets/images/2011/10/Greenhouse...](http://media.treehugger.com/assets/images/2011/10/Greenhouse20Gas20Emissions20from20Common20Proteins20and20Vegetables20.jpeg.650x0_q70_crop- smart.jpg) chickens are far better than almost all other animals.

So if you're going to eat meat, chickens are a good choice in this way at least.

<comment><USER20> I typically stick just to freshly caught fish from the coast. Looks like even canned or farmed fish is just as emissions-bad as chicken.

</comment></comment></comment></comment><comment><USER23> This is kinda terrifying. I can't put my finger on why exactly but dang.

<comment><USER24> cows being raised for meat aren't exactly healthy either. Corn is not what they're designed to eat, but it is what we feed them.

[http://www.pbs.org/wgbh/pages/frontline/shows/meat/interview...](http://www.pbs.org/wgbh/pages/frontline/shows/meat/interviews/pollan.html)

<comment><USER25> I just spent a half-hour reading this very well written article. Very cut and dry explanation of the livestock system. Highly recommend everyone to read.

Interesting tidbit: "But a lot of it has to do with the way we grow our food, and the fact that we mix 100 different cows in a single burger. We never used to do that. The butcher used to take the scraps from that one animal and make his hamburger right in front of your eyes. ... Now, you get one infected carcass, and that meat can spread all around the country, because we have this centralized national system."

</comment></comment></comment><comment><USER26> Answer: it's an oxygen shortage in the overly thick muscle, causing tissue death

Solution: oxygen masks for the chickens

</comment><comment><USER27> if this disgusts you for moral/guilt reasons there is a good solution. Eat less meat! You don't have to go full vegan.

I think people employ black and white thinking about this -- either I eat all meat or I become vegetarian/vegan and eat no meat. That's hard, and you're likely to fail (hard cutting to vegan would be quite difficult, regression rates after a few months are sky high).

Instead, simply resolve to eat less meat. Make Thursdays meatless. Opt for the vegetarian option when eating out. Slowly reduce your meat consumption to a level you're happy with. Learn about meat alternatives (which have gotten really, really good in the last few years).

If everyone in the US took one day a week to have no meat, the whole industry would change dramatically. Baby steps!

<comment><USER28> If I wasn't vegetarian already, this probably would have done the trick!

</comment></comment><comment><USER29> Certainly one of the weirder things I've seen on HN.

</comment><comment><USER30> Your concern for animal welfare will never be enough for the animal rights people (just read the other comments here).

<comment><USER31> Umm, I've seen plenty of balanced views in this thread. I'm a vegan, partially for animal rights but I still encourage meat-eaters that are concerned to go for the better stuff. I don't believe it is the best solution, but I applaud anyone willing to attempt to make things better.

"the animal rights people" is a pretty broad brush, I know that quite a few can be completely impossible to discuss the nuances with but honestly, there are so many vegetarians for ethical reasons around that just don't say anything or go "cool, that's better than industrial farming". I also encounter a good part of meat-eaters that become incredible defensive and almost hostile if they find out about my dietary choices.

Always frustrating to see black and white grouping when the truth is incredibly nuanced.

<comment><USER32> > I also encounter a good part of meat-eaters that become incredible defensive > and almost hostile if they find out about my dietary choices.

Yeah after someone found out after kinda knowing them for a couple years they said "Fuck You" and we haven't spoken them since then. I wonder what their problem was ¯\\_(ツ)_/¯

</comment></comment></comment><comment><USER33> I have a solution, (downvotes are welcome), eat an organic and mostly vegatarian diet.

<comment><USER34> From the guidelines: [https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

> Please don't bait other users by inviting them to downvote you or proclaim > that you expect to get downvoted.

(I also don't think your statement is particularly controversial.)

<comment><USER33> My bad, apparently they followed my advice anyways.

Jokes aside, eat more veggies for your health, the health of those poor little things that are fed shit and antibiotics daily in very bad environements, and last but not least for the health of the earth.

<comment><USER35> Remind me what plants are fed, again?

<comment><USER33> you don't get the end of chain thing do you ?</comment></comment></comment></comment></comment>


Arachne: a GPL self-contained web browser - <AUTHOR> https://en.wikipedia.org/wiki/Arachne_%28web_browser%29 <comment><AUTHOR> HN seems to be looking at niche browsers with their own rendering engines today (NetSurf: [https://news.ycombinator.com/item?id=10417807](https://news.ycombinator.com/item?id=10417807), Dillo: [https://news.ycombinator.com/item?id=10417007](https://news.ycombinator.com/item?id=10417007)), so I felt bringing up Arachne made sense.</comment>


The Evolution of Tech Companies Logos - <AUTHOR> http://www.neatorama.com/2008/02/07/the-evolution-of-tech-companies-logos/

<comment><USER2> It's hard to believe that someone paid $300,000 back in 1972 for the current IBM logo.

PS - the new xerox logo is atrocious.</comment>


Why you should move that button 3px to the left - <AUTHOR> http://www.designstaff.org/articles/design-details-2011-11-29.html

<comment><USER2> Is this quote really about design?? " have certainly been in a bad mood, gotten confused by a product, and found myself repeatedly smashing a button to no effect. In my frustration, I try the same thing, just harder. But it doesn't help me accomplish my goal."

This sounds more like something that is broken. I wish there was a better example provided of bad design affecting the users mood.

</comment><comment><USER3> Why do you think you're at the mercy of the programmers? Why can't you learn to build and deploy the code yourself? Seems like a bit of a bottleneck to put all your changes through a programmer.

<comment><USER2> Everyone has their skills. Not all designers are coders, even knowing how to efficiently create HTML and css isn't something that everybody needs to be great at.

I agree it is helpful that designers know the basics of html and css, but that isn't their job.

It's like asking a developer why they aren't also a designer. Why not just learn how to design.

Having said that I've met a few designers who are AMAZING programmers as well (or vice-versa) but it is a very rare and valuable combination.

I'm learning some design now through designlab (just started), but I likely won't become an amazing designer, I just want to get better and possibly improve my abilities to see what the designers see and I'm missing, and hopefully improve my ability to work and communicate with designers.</comment></comment>


Fippet allows you to pass on snippets of info via a 4 character code - <AUTHOR> http://fippet.com We have just launched - would love your feedback! <comment><AUTHOR> Hi Krap,

Thanks for the feedback, you recon logging in to post a snippet should be optional? It's just once you post a snippet we like to add it to your profile, so you can find it again.

4 characters is easy to remember :)

</comment><comment><USER2> Why the limitation of 4 characters?

And also why do I need an account to post a snippet?

Also the wood background makes the text really hard to read.

</comment><comment><AUTHOR> If you have any questions let me know :)</comment>


$21 Billion Orbiting Solar Array will Beam Electricity to Earth - <AUTHOR> http://www.physorg.com/news172224356.html <comment><USER2> Does this tell us anything more than the item over at <http://news.ycombinator.com/item?id=799806> ??

95 comments there already ...

<http://searchyc.com/solar+beam?sort=by_date></comment>


Ask HN: What app would you make, if you have access to all mobile operator data? - <AUTHOR> <comment><USER2> Cafe', Provider & Press (journalism outlet) polling and poll autoawesome /SS7#Slack (or React; something to hand)?

Broadcast channels to the edge. (ATSC, DTMB, DVt data like NTSC vertical blank had/has!)

Wine tastings right at the colocation racks.

</comment><comment><USER3> Free geofencing APIs.

</comment><comment><USER4> Mobile sending app</comment>


Ask HN: Do any of the 'Do Funky Stuff X – You don't need to code' work? - <AUTHOR>

Is this a healthy productive trend the industry is moving towards or is it time we stop these gimmicky claims.... <comment><USER2> You mean like creating a website using Wix for instance? Well the answer is always the same: They work as long as you do thing exactly as they intended it. As soon as you want something a bit different, you will have to either forget it or start to hack around it. But for instance for a barber shop, it would be nonsense to pay a developer to create a website when they can just use a premade solution and customize it a little bit. Now, if you want to do something a bit more serious, you will definitely need to code.</comment>


Ask HN: Learning a foreign language, for programmers? - <AUTHOR>

I'm curious if anyone here knows of a resource for learning foreign language that's suited for programmers?
I figure we're all pretty used to learning languages (programming) - is there such a thing as a foreign language course modeled after coding courses?
If not - what are your favorite resources for learning foreign languages? <comment><USER2> Without exception, Anki is the way to go.

[http://ankisrs.net/](http://ankisrs.net/)

I've found the philosophy extolled on these blocks to be especially conducive to the programmer mentality: [http://alljapaneseallthetime.com/blog/](http://alljapaneseallthetime.com/blog/) [http://www.hackingchinese.com/](http://www.hackingchinese.com/)</comment>


SpaceX wins its First competitively-bid Air Force satellite contract - <AUTHOR> http://www.bizjournals.com/denver/news/2017/03/14/spacex-wins-its-1st-competitively-bid-air-force.amp.html <comment><USER2> SpaceX, despite all its geek-cred, is still solidly an enterprise B2B provider. At this point, however much I'd like to comment, I have nothing to say except that if the been-counters in the DoD (and the other satellite companies) approve, they probably know what they're doing.</comment>


Insurer Claims Man Comitted Arson via Remote PC Login - <AUTHOR> http://i.stuff.co.nz/national/crime/83868063/Northland-man-denies-burning-down-house-but-insurer-refuses-to-pay-out# <comment><USER2> Funny this should come up now.

I'm on holiday at the moment and have been having geek fun by SSHing into a raspberryPi back at my home [in a different country], taking photos with its camera and uploading them to Google Drive where i can view them from here.

It did idly cross my mind that this kind of setup had the potential for "remote control" crime. I expect we'll start seeing more of this sort of thing, in future.</comment>


NLP Challenge: Find semantically related terms over a large vocabulary (1M)? - <AUTHOR> http://metaoptimize.com/blog/2010/11/05/nlp-challenge-find-semantically-related-terms-over-a-large-vocabulary-1m/

<comment><USER2> This doesn't make much sense....

From the website, here are some example "documents":

      roger heavens trinity cricketers willie sugg early cambridge cricket giles phillips cambridge university cricket

      roger hornetttom pridmore 020 7399 4270 collins stewartalan geeves 020 7523 8800 buchanan communicationscharles ryland /isabel podda 020 7466 5000

      roger hubbold aig .1 force-field navigation

Now, what is the point in trying to generate any kind of "meaning" from those documents if they consist of completely meaningless gibberish?

As I was reading this challenge, I immediately thought of spam filtering / youtube comment classification ("smartness" classification) / etc as a potential _useful_ application of this technology.

For example if each "document" is a youtube comment, then in theory you could write an algorithm to examine each comment and output a "smartness guess" for each. Then you (as in, you personally, by hand) would look at the results and specify your own "smartness rating" for a few comments. Then you'd run an algorithm to look at the difference between your specified "smartness rating" and the "smartness guess". Then, using that difference, it would tweak the settings in the original algorithm until it outputs a "smartness guess" that more closely fits your "smartness rating". If you repeat that process enough times, and your original algorithm has enough modifiers to tweak, then you might wind up with an algorithm that can make a pretty good guess about whether any given youtube comment is retarded or not.

And that's just one example of a practical application for this kind of thing.

That said, if the input "documents" are _completely and utterly meaningless_ , then there does not seem to be any point in trying to build "meaning" from those inputs. (Garbage in, garbage out.)

<comment><USER3> Semantically related != meaningful.

Also, they do note that, "'Semantically related' is purposefully left vague."

<comment><USER4> I'm sorry, but "Semantics" is "the study of meaning" (at least according to Wikipedia). How's that possible that "Semantically related != meaningful"?

<comment><USER5> You left out a word - 'related'. Consider:

Tennis Playpen

Semantically _related_ yes, meaningful no.

</comment></comment></comment></comment><comment><USER6> That's somewhat similar to my second mini-project of my NLP course (that I just delivered like one week ago).

What we had to do was, first to gather a large corpus (in this case) of portuguese. Then, produce the frequencies of each unigram and bigram. Then, process those frequencies and output the top 10 and bottom 10 collocations of the corpus. Collocation definition from wikipedia: "Within the area of corpus linguistics, collocation defines a sequence of words or terms that co-occur more often than would be expected by chance".

The method used to calculate the collocations was based on a chi-square test. The values with higher chi-square were the most probable collocations (if above the critical value that would reject the null hypothesis).

So, I actually have Ruby code done that I could change just a little bit for this, but there are probably more clever methods.

</comment><comment><USER7> This sounds interesting but the main challenge here seems to be to define the question: What is a semantically related word. I can think of three classes - variations (singular/plural), synonyms (thesaurus), contextual (occur in the same sentence/paragraph). Once the definition of "semantically related" is established, it should be easy to find a clever algorithm to produce results. However, out of context the problem is a little bit boring - in order to find a good solution I'd like to know the real world problem the list of "semantically related" words is going to solve.

<comment><AUTHOR> There are two aspects of the problem that I consider particularly interesting: a) defining the problem, because being able to define the problem and justify your answer is half the puzzle in a lot of NLP work. b) scaling to a corpus of this size (vocabulary of 1M), because this scale is tricky but useful in many web problems.

Semioticians typically distinguish between _paradigmatic_ and _syntagmatic_ axes of semantic relatedness. Paradigmatic means that two words occur with similar other words, e.g. these two words typically have the same word immediately to their left, like "blue" and "azure". Syntagmatic means that the words typically co-occur in usage, like "blue" and "sky". Check out the image on this page for another illustration of these axes: <http://www.aber.ac.uk/media/Documents/S4B/sem03.html>

Regardless of whether you choose to do a paradigmatic or syntagmatic analysis, it's interesting to see how you motivate your approach and if you can scale it to 1M different vocabulary words.

<comment><USER8> _scaling to a corpus of this size ... it's interesting to see how you motivate your approach and if you can scale it to 1M different vocabulary words_

It looks like the sparsity of the matrix is going to be a much bigger challenge than the scale.

I understand about the focus being primarily on the approach, that makes sense; how are you intending to evaluate the results files?

<comment><AUTHOR> _It looks like the sparsity of the matrix is going to be a much bigger challenge than the scale._

Sparsity is good. The sparsity is the only reason that you can keep a matrix with this many dimensions in memory.

 _I understand about the focus being primarily on the approach, that makes sense; how are you intending to evaluate the results files?_

For any submission, I will post for a random subset of vocab words each entry's 10 related terms. I'll then ask people to vote blind.

<comment><USER4> I think the more fair comparison would be to show THE SAME random subset for each entry. I.e. same X words for all result sets.

Otherwise, it might happen that a superior result would just show words that don't even have good results, while an inferior subset would get better covered words.

<comment><AUTHOR> Yes, it will be the same random subset for all participants. Sorry if that wasn't clear.

</comment></comment></comment></comment></comment></comment><comment><USER9> My first idea would be the following:

If I had enough time for the problem I would first use a morphological and syntactical analyzer to try (at least with a high probability) to determine the morphological structure of the words and I would tag them with their syntactical category (part of speech). So I would treat for example 'work' and 'worked' as being the same word. On the other hand I would treat 'can:aux' and 'can:noun' as different words.

Now I would completely eliminate all the words which are not nouns not adjectives and not verbs. (the, a, can:aux, may, etc...) These are 'asemantical' words.

Maybe eliminate extremely rare words also (with 1 or 2 occurences)

After this preprocessing now comes the semantical relatedness analysis.

A very primitive solution:

\----

Compute the ten best B for A where the following is maximal:

f(AB) / (f(A) * f(B))

f can be the frequency. f(AB) can be the frequency of occuring together in a document or within a distance (like 5 words). But you can fine tune it: weight it with the actual distance.

You don't have to maintain the full matrix. As you go through the documents just maintain max 100 words for each word for which this measure is maximal, and in the end just take the 10 maximal from the 100.

\----

A more sophisticated solution for the semantically relatedness analysis:

Let's put all the possible words into a 2 dimensional map with totally random x,y coordinates. Now as you go through the documents imagine that the words which occur near each other attract each other on the map so their distance is made *=0.8. They gravitate. Imagine that the mass of the word is its frequency: the distance is made smaller by small frequency words going towards the big frequency word a lot, and the a big frequency word goes just a litle bit. Just like gravitation. Of course time to time we change the scale: we blow the whole map up a bit to not make everything gravitate into one point. This algorithm is a 'global algorithm' in the sense that it takes into account indirect things like synonims which are semantically really related even though they not really occur in the same sentence. In the end determining the 10 nearest neighbours fast is easy: you can use a simple grid as 'space partitioning'.

Edit: A 2 dimensional map s not necessarily optimal, it might be too tight. I would also experiment with setting the map dimension to 3,4,5 etc...

<comment><USER10> try it and see :D

I think I'm going to be attempting a solution, too (though it's not the same as yours)

</comment></comment><comment><USER11> I checked in a simple co-occurrence frequency based implementation for this here <https://github.com/tgflynn/NLP-Challenge> with a short discussion on my blog: <http://cogniception.com/wp/>.

</comment><comment><USER12> Solution: [http://groups.google.com/group/metaoptimize-challenge- discus...](http://groups.google.com/group/metaoptimize-challenge- discuss/browse_thread/thread/48bc7de317e79576)

</comment><comment><USER13> Hmm. Without any thought given to computability: Rearrange the words of every document into alphabetical order, then sort the whole list of documents into alphabetical order. Now, start reading the documents, and for each one make it a tree with each word a leaf, BUT if there is already a tree with a fragment of the same phrase, then that tree is linked to as a node of the new tree.

So for the document 'christmas day' the tree

______d1______.

__/_________\\__.

christmas___day

is formed. But for the document 'chistmas day special', the tree

_____d2______.

___/____\\____.

d1_______special

is formed. Note: word order doesn't matter. For any given document, how will it know if there's already a tree that uses some of the same words? When a tree is created from a document, give it a hash for the full set of words, plus hashes for sub-strings. These can be stored in a big lookup table, so new documents know where to look for relevant trees (or -- use whatever solution is well established for this sort of problem). Might also need to algorithmically redefine older trees as we go along, ie. divide a tree into smaller phrases

For each leaf let it keep track of what trees its part of (including grandparents and above). Then to find the top-k, for each leaf object (if it's in the vocabulary), print out all the other leaves of all the trees its part of. Like

 _if leaf.__str__ == 'day':

___ for tree in leaf.trees

______ print tree.leaves_

Then do a simple frequency count to find the top-k associated words.

(sorry for the n00b answer... IANACS)

</comment><comment><USER14> One way of doing this:

1\. Get some million tweets from twitter API (each tweet is exactly like what the article describes as a 'document')

2\. After ignoring common words (e.g. 'a', 'an', 'the' ... ) in a tweet , start assigning some kind of relevance rank to all the other word pairs found in the tweet e.g. its likely that 'Cricket' and 'Sachin' (Or 'NBA' and <top NBA player) will both increase each other's relevance rank, WRT to each-other.

3\. Process all the tweets like this, while maintaining the output of step 2 in a most suitable data structure. You would also need to start dropping word- pairs (to avoid having the problem of storing Million-C-2 words! ) based on some logic/heuristic.

4\. If we have a good logic for having reasonably not-big storage (by avoiding the million-c-2 explosion), then what we have is at the end of processing: A simple look up of a million words, where each word has its 'top' max_allowed(k) semantic words.

PS: The most complex piece in the approach is to come out with a solution for dropping word-pairs (in step 3)

<comment><USER15> I have a preliminary implementation of this, geared for map/reduce-style parallelism in Perl, up at <http://github.com/spencertipping/metaoptimize- challenge> (in the fcm directory). It may be a start to solving step 3 -- using reduce-by-two on each step and dropping low-relevance words (I think this is valid, though I'm not 100% certain).

<comment><USER14> Hi! Thanks for reading my suggested approach. I can understand how Map-reduce can be used to process the things faster (by processing them in parallel and later using reduce to aggregate the results??)

      But can't imagine how reduce (of map-reduce) can help in dropping of word-pairs (i.e. in Step 3)

<comment><USER15> I think it's an imperfect solution, though it might still be solid. It depends on your reduction strategy (basically breaking associativity). If you do a fold-left, then you'll accumulate such high relevance that you'll quickly start discarding every word in the right-hand data set. On the other hand, if you binary-split the folding process you're more likely to be OK.

I used a binary split and due to the sparseness of the data (and the uninsightfulness of my algorithm) didn't run into too many spurious drop cases. But the implementation I posted is very basic and shows my lack of background in NLP-related pursuits -- I'm lucky it did anything useful at all :)

</comment></comment></comment></comment><comment><USER16> I will no bother with this challenge, since there is no a clear good solution. I would find more interesting working with semantically rich documents in which your work can be tested to measure the quality of your solution. That is, a good work must always gain enough recognition to be pursued.

</comment><comment><USER17> Couple of approaches that I can think of from papers of which combining in some way would probably mean decent results. Matching left and right fragments of n length, adding POS tagging to the matching fragments and using wordnet relation closeness for terms that exist in wordnet.

</comment><comment><USER18> I encourage anyone to use Extractiv (<http://www.extractiv.com>) for this challenge. We're rolling out an On-Demand (local document analysis similar to Calais) Semantic Conversion service that could be helpful.

<comment><USER19> Slow down the rollout a little: your 'take a tour' page inline link to Semantic On Demand 404s, as does the inline link from your pricing page:

<http://www.extractiv.com/tour-semantic-on-deman.html> \- missing letter in '...deman.html'

<http://www.extractiv.com/tour-semantic-on-demand> \- missing '.html'

Also, the pricing page says users need to buy a plus or premium subscription for access to Semantic On-Demand, but confusingly says the basic plan includes 'On-Demand access to 1,000 documents / day' - I presume you meant to emphasize the unlimited usage available with the premium plans.

<comment><USER20> also, the example from here <http://www.extractiv.com/tour-semantic-on- demand.html> does not work.

I second, this seems interesting (and I would be really interested in seeing what you produce as "relations")</comment></comment></comment>


Using Graph Processing for Kafka Stream Visualizations - <AUTHOR> https://www.confluent.io/blog/kafka-graph-visualizations <comment><USER2> Hi! I'm the original author of this post. Happy to field any questions if there are any.</comment>