Go Module Mirror and Checksum Database Launched - <AUTHOR> https://blog.golang.org/module-mirror-launch <comment><USER3> Vendor. Everything. Always.

I appreciate the fact that Google is investing in infrastructure to make the Go ecosystem robust, I really do.

Having said that I encourage everyone to do a simple experiment.

Check out your code into some random directory. Copy the directory to a USB drive and walk it over to an air gapped machine (no wi-fi, no ethernet, clean dev environment installed). Copy the directory to the box, make some small code change and try to build your binary.

If your build fails you're doing it wrong.

In real life one of the following will happen[1]:

\- Your network connection to the Internet will be down

\- Your network connection to the corporate network will be down

\- The services you depend on (Github, Google proxy, Docker hub) will be down

\- The services you depend on will throttle you

\- The services you depend on will serve you wrong or corrupt data

\- The services you depend on will be offline for maintenance

\- The certificates on the service you depend on were not renewed in time

\- The account you use on services you depend on will be inexplicably suspended

Murphy's law being what it is, some or all of the above will happen when your SaaS is on fire, your customers are screaming, and you're trying to roll out a fix. Meanwhile your company is losing approximately your personal lifetime earnings in revenue every half an hour.

Vendor. Everything. Always.

(and have backups, lots of backups)

[1] Based on highly educational career experience

<comment><USER4> I don't see any of those as reasons to commit dependencies to the same VCS repo your source control lives in.

Your first point is particularly laughable. If your network connection to the internet is down and you're a SaaS provider, you're probably going to need to fix that before you can roll out a fix to your hosted software.

That said, they're all good points to consider when building out a deployment pipeline, and in general mirroring your dependencies is a great idea. But committing a vendor directory is a poor solution/mitigation for most of those risks.

<comment><USER3> > Your first point is particularly laughable. If your network connection to > the internet is down and you're a SaaS provider, you're probably going to > need to fix that before you can roll out a fix to your hosted software.

Don't assume that your SaaS runs on the same network as your build machines. In most cases this is not the case - or at least it shouldn't be. Your SaaS might be broken in some way, you're trying to create a build to fix it on your secure corp network and your connectivity to the Net disappears. And it's 3 AM in San Francisco and everyone who can fix it is asleep, and meanwhile your European customers are not happy.

Let's be clear. You can mitigate the effect of all these problems one way or another. But the mitigations are easier if you have less moving parts and less dependencies on systems you don't control.

Lets face it, ultimately it all comes down to limiting your business risk at least cost (because money). One simple way to do that is if your build environment travels with your code down the years.

<comment><USER4> > Don't assume that your SaaS runs on the same network as your build machines.

That wasn't the assumption. It's laughable because under any condition of where your hosting and build infrastructure live, you'll need to restore your internet connection before you can deploy whatever software fix you're trying to ('deploy' in the sense that your customers have access to the deployed fix).

</comment></comment></comment></comment><comment><USER5> For anybody wondering if vgo is production ready, we've been using it since it came out in go 1.11 and so far didn't have any problems.

Existing workflows don't break because there's the vendor command and dependency management is instantaneous. You can also roll it out gradually as it doesn't have any problems depending on non-module repositories (using commit hashes and dates)

There's a great blog post series too if you're interested in the internals of go modules: [https://research.swtch.com/vgo](https://research.swtch.com/vgo) (it's also one of the good counterexamples to people claiming go ignored x years of PL research and history)

<comment><USER6> I found the editor support to be still somewhat hit and miss though. VsCode go extension still does not work as good on module enabled projects for example. Otherwise we did not face any issues either.

<comment><USER7> That's a great reason to try out Goland, which has had excellent module support since day 1.

<comment><USER8> If you can afford that $100-200/year subscription.

<comment><USER9> I bought it and stopped using it. It does work with modules, but I don't like the IDE itself and went back to vs code even though it breaks on me regularly. The UI/UX is just too foreign for me in GoLand. I had to search Google to find out how to search the project. Everything I wanted to do was non-discoverable for me.

<comment><USER10> CMD+SHIFT+F just like any other editor (vscode, sublime etc) out there?

Also you have action manu on CMD+SHIFT+A that will give you any command IDE can offer.

It's as discoverable as vscode if not more (you can even use IDE Features Trainer plugin if you want to)

<comment><USER9> Pretty sure cmd+shift+f (maybe it was just cmd+f) would search the open file.

<comment><USER11> With Shift: search whole Project. Without: search in open editor

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER12> I like the simple plain text signing format for the API calls.

For example: [https://firstname.lastname@example.com](https://email@example.com)

Is signed with:

      — sum.golang.org Az3grobHchAJWrV4M34o1kLnZV4vrGSfFA+2Q9VClbmWqBjsnN4GzK1xB1RaYGSo0jIjWH9GDcR3Tja5sadw2ESoKwg=

The format is documented here: [https://godoc.org/golang.org/x/mod/sumdb/note](https://godoc.org/golang.org/x/mod/sumdb/note)

Also, as an aside, I am building a transparency tool for arbitrary binary downloads with the rget project: [https://github.com/merklecounty/rget](https://github.com/merklecounty/rget)

<comment><USER13> I really like rget, btw.

I posted a trivial little command-line demo of a client and server for some others who were thinking about a transparency tool. It implements the cacheable GET API we designed for the checksum database but obviously it applies to arbitrary key-value pairs. It might be a better foundation than stuffing things into CT logs (or not, you do get some interesting infrastructure that way).

[https://rsc.io/tlogdb](https://rsc.io/tlogdb)

<comment><USER12> Thanks!

I am almost certain I am going to need to move off of the CT log thing but I hope to onboard a few projects first[1].

But, tlogdb looks like a great starting (sent you a PR if you have a moment). I am trying to understand how all of the parts fit together at the moment particularly if I need to use Trillian or how far a direct SQL store to Spanner/CockroachDB like tlogdb implements is sufficient for awhile.

The big advantage of implementing something like tlogdb is that it becomes a URL map which means that developers don't need to publish additional metadata (at the expense of the service needing to download entire files to calculate the digests).

All of that to say thanks for all of the work on sumdb. Great starting point.

[1] [https://github.com/merklecounty/rget/issues/10](https://github.com/merklecounty/rget/issues/10)

</comment></comment></comment><comment><USER14> Is there a justifiable need for the proxy? It seems like we're just adding a centralized point of failure, the lack thereof something I have absolutely touted as a feature.

For verification we already have the go.sum file validating the hash. We've got vendoring if you're concerned about dependencies disappearing.

The touted speed feature seems like a non-issue? The absolute largest Go package I've ever pulled took maybe a minute to grab its dependencies, and then I had them and never had to again.

The cynical side of me feels like this is a way for Google to collect analytics on the Go ecosystem. The slightly less cynical part of me thinks people from other ecosystems demanded it due to cargo culting the idea.

The fact that it makes it more difficult to use private modules is pretty irritating as well as I am going to need to make sure all our developers and build systems are aware when we move to 1.13. Having to set GOPRIVATE on _each machine_ is absurd. It should be definable in the go.mod so it can be known via git.

<comment><USER15> > The cynical side of me feels like this is a way for Google to collect > analytics on the Go ecosystem

> other ecosystems demanded it due to cargo culting the idea

And there it is, the hallmark of a typical comment on a thread like this

* Cynical

* Expresses concern about data being collected even though this data can't possibly be used for anything bad. (If you disagree, go to crates.io and tell me what concerns you have).

* Expresses disapproval of a large corporation even though they haven't done anything wrong here.

* Pithy dismissal of other programmers by calling them names ("cargo cult"). This change improves speed of downloads by 6-7x on poor connections and 3x on good connections. Clearly this is helpful for situations like CI which typically involve clean builds. What's the issue here exactly?

* Furious attack on a free product that's completely optional to use. If you want to keep vendoring your dependencies, go ahead and do that. Why are you angry that some people are going to use this proxy?

<comment><USER14> "Furious attack". For serious? I gave a light hearted criticism of adding a single lynchpin to the entire Go ecosystem. I think that's a legitimate source of concern.

I was by no means pithy, I was by no means being dismissive. Thinking you need something just because you had it elsewhere is the definition of cargo culting - and is something the Go developers have otherwise largely avoided.

Just because something is _faster_ doesn't make it better or worth it, particularly when it comes with tradeoffs and is already really fast - <USER2> is my argument - that the tradeoffs aren't worth the benefit.

<comment><USER15> This is not a single point of failure. It's by no means a legitimate concern. Go read the proposal. If you don't want to use proxy.golang.org, use goproxy.io. Or host your private one. Fairly sure you'll be able to use github soon as well. Do whatever.

I notice you couldn't back up your FUD about analytics being collected.

Go programmers are happy that they have a solution where their dependencies (and specific versions of those dependencies) will be available forever without them having to take the trouble of vendoring. Your dismissive response? Just vendor. They're happy that clean builds are faster, saving CI time. Your response? It doesn't need to be faster.

<comment><USER14> You're doing the same thing you're criticising, being dismissive of my concern.

> If you don't want to use proxy.golang.org, use goproxy.io

Yes, I can do that, but as I mentioned in my original post having to get an entire team of people and fleet of CI systems to use a non-standard configuration which is not communiciable in the project repo is a minor PITA.

If and when proxy.golang.org has a bad day, builds will fail. Go developers will have a bad day. Having the ability to work around it doesn't make it not a lynchpin.

As for the "FUD about analytics being collected" it was an offhanded quip. I thought it read pretty clearly as _not entirely serious_ hence the prefix "The cynical side of me feels" and not "I feel". It doesn't merit the effort of defending.

</comment></comment></comment></comment></comment><comment><USER16> "Below is an example of such a tree"... and there is a picture of a binary tree, with no indication as to how the tree is used.

I appreciate trying to explain the interesting technology behind authenticating modules, but this diagram appears to explain nothing at all.

<comment><USER17> There was a talk at gophercon that I belive goes into more detail about how this works and the cryptography behind it (I only skimmed it so far) [https://youtu.be/KqTySYYhPUE](https://youtu.be/KqTySYYhPUE)

</comment></comment><comment><USER18> This is probably a dumb question, but here goes... I'm testing go modules with an internal project that uses a private github.com repo. It depends on other public go packages and modules is nice to manage those but the general public will probably never see or want to use this code.

Are modules in private repos impacted by this in any way? Would they show-up in the index somehow?

<comment><USER19> If the private one is the main module, you don't have to do anything, it doesn't get sent anywhere because it's just what's on your disk. Indeed, you can call it "module foo", which is not a go-gettable name.

If you _depend on_ private modules, you need to follow these instructions: [https://tip.golang.org/cmd/go/#hdr- Module_configuration_for_...](https://tip.golang.org/cmd/go/#hdr- Module_configuration_for_non_public_modules)

In any case, private modules never show up in the index, mirror, or checksum database.

<comment><USER20> ...but references to them are stored by google in logs and used per the privacy policy at [https://proxy.golang.org/privacy](https://proxy.golang.org/privacy)

</comment></comment></comment><comment><USER21> Also I would like to bring up since JFrog's module proxy, GoCenter was publicly hosted first, they will have more than likely been first to host a module version. That means you are now open to the situation where someone uploaded v1.0 of a module to Go Center, and v1.0 of a module to Google's Mirror, and they contain different code.

I am not sure how the Go team plans to handle this, but the only obvious solution would be for Google's proxy to reach out to GoCenter first to see if they have a copy of a specific version, download and cache, and serve that. Does anyone know if that happens?

<comment><USER22> Why would two versions - specifically tied to a given commit - of say "v1.2.3" contain different code?

<comment><USER21> Well you mentioned it in your comment, tags are not specific commits. Go Modules standardize around SemVer, and fallback to hashes only when it needs to. The reality is anyone can git push -f and change what a tag points to currently.

So the scenario goes like this: 1\. Someone requests a module from GoCenter that is currently not in it's caches. 2\. Gocenter downloads the requested version v1.2.3, and stores that. 3\. Bad/Careless/Ignorant/Evil (depending on circumstances, I guess) git push -f's 4\. Module is requested in Google's module proxy, which now loads a "v1.2.3" into it's local caches, but it contains different code now than "v1.2.3" of GoCenter

Outside of git push -f, there are other hacky techniques you can use to "update" the code a tag points to.

<comment><USER22> Hmmm, maybe my understanding of tags is incorrect.

When I tag something in git, that _is_ for a specific commit isn't it?

</comment></comment></comment></comment><comment><USER23> [https://proxy.golang.org/privacy](https://proxy.golang.org/privacy)

A Google privacy policy that's actually reasonable.

</comment><comment><USER24> Does anyone know if the Go Proxy server using the original download protocol[0] defined by vgo?

        GET baseURL/module/@v/list fetches a list of all known versions, one per line.
        GET baseURL/module/@v/version.info fetches JSON-formatted metadata about that version.
        GET baseURL/module/@v/version.mod fetches the go.mod file for that version.
        GET baseURL/module/@v/version.zip fetches the zip file for that version.

[0] [https://research.swtch.com/vgo-module](https://research.swtch.com/vgo- module)

<comment><USER25> Yes, see also [https://youtu.be/WDbbIS7m9bU?t=298](https://youtu.be/WDbbIS7m9bU?t=298) (sorry, slide deck not yet available it seems)

</comment></comment><comment><USER26> This looks great.

I'm interested in the possibility of running a private mirror. Does anybody know if they've release the source for proxy.golang.org? I can't seem to find it in the golang github org.

<comment><USER12> [https://github.com/goproxy/goproxy](https://github.com/goproxy/goproxy) as well</comment></comment>


How does a graphic designer reach early-stage startups for branding projects? - <AUTHOR>

I'm a graphic designer looking to reach early-stage startups to offer my brand identity creation services.
I have used Twitter and LinkedIn to get my name out there, and the activity so far is really encouraging, but I haven't found anyone ready or willing to sign me up yet, so I'm looking for other avenues as well.
I have been trained in "old-school" marketing techniques as far as self-promotion, but the marketing landscape is changing and I don't want to come off as pushy or spamming; I DO want to be noticed, and by the right people at the right stage.
The startup scene is so different from the way "small business" has traditionally worked. It is very exciting, and a little new to me but I think it's really the best "new frontier" for graphic guys like me to make any significant impact in our profession/industry going forward.
My challenge is understanding the startup culture so I feel comfortable presenting my services and talents to startups, and they feel comfortable using them.
I live in Phoenix but really want very much to work with startups nationwide and globally, and not limit myself to any one region.
Eventually I want to build a brick-and-mortar design/branding/marketing firm with real employees supporting me, but that will take clients and funding.
I have seen similar comments to the effect that AngelList and f6s are good places to submit interest, do you think those will work?
What works best? How can I best put my best foot forward? I'm excited about the possibilities, but feel like just having a website and hoping folks will come by, is not making enough of a marketing effort. <comment><USER2> Your goal is unrealistic. Start local and small. Find a local small business that needs a branding refresh. Do an amazing job. From this you will get the best kind of marketing - word of mouth. With the referrals you get from this, do an amazing job again and again.

Don't limit yourself to startups - this is why you are failing. Take any business that needs design (they all need design). Never turn down work. Treat every job as if your life depends on it (it does).

If you do all of the above consistently for a couple years, then and only then will you get the client base you desire. When that happens, still take the small jobs.

<comment><AUTHOR> With all due respect, I will disagree with your advice to a large degree. I've tried what you are suggesting, literally for years. No success, and it's not for lack of talent or skill on my part, aesthetically speaking.

In my local area (I've lived several places), there is so much competition for the "small jobs", and so few local small businesses that actually care about the value of amazing design (or who can afford to pay the fees needed for a designer to actually be in business) - no matter how much you try to educate them - that it's really not worth it to go after such small fish.

If I'm going to spend my time and effort educating potential clients on the value of and need for branding, I want it to be on those businesses that 1) have growth potential, 2) are at the beginning stages of their lives, when the need is greatest and 3) run by people who have the greatest probability of "getting it", i.e., actually able to comprehend that there is a value to excellent, well-thought-out creativity and that sometimes that value justifies a little bit higher price than $99.

Joe's Body Shop or Jim's Plumbing or Karen's Flower Shop or Martha's Nail Salon are frankly not an adequate target audience for most designers who aspire to make some major impact, and in most cities those kind of businesses are at least 50% of small business (excluding national franchises/chain operations which dominate in many cities). They are perfectly content to have their name plastered on the side of their truck and leave it at that. I want to create solutions for businesses that are a little bit more meaningful. Once I do a couple of those, I think the word of mouth will build from there.

And maybe some of these other kinds of businesses will see what I do, and that it's awesome, and that it has the potential to help drive revenue by helping build respect and credibility with the public, and those businesses will then try to convince ME that their business is worth MY time, instead of the other way around.

I am encouraged by my self-promo efforts so far, but I'm always looking for ideas from people who have a knowledge of the startup scene.

</comment></comment><comment><USER3> Do you have a link to your site that works? Your Twitter one is down.

<comment><AUTHOR> [http://micmac99.wix.com/msreeddesign](http://micmac99.wix.com/msreeddesign)</comment></comment>


'It was me. I know it was me.' - <AUTHOR> https://www.washingtonpost.com/nation/2020/05/30/coronavirus-daughter-to-mother-contagion/ <comment><USER2> For anyone who doesn't want/can't click on the link, it's a first person article about someone who believes they passed the Coronavirus on to her mother.

<comment><USER3> That's like saying the Mona Lisa is a painting. It is a deeply sad real story, with many details in it. It cannot be summarised in a sentence.

Link: [https://outline.com/wcZMzs](https://outline.com/wcZMzs)

<comment><USER2> Sorry if I came off as cavalier. I just wanted to give people some idea of the article contents since I didn't have any idea what the article would be about when I clicked on it.

<comment><USER4> You did the right thing. Thanks.

</comment></comment></comment></comment><comment><USER5> > What if I catch it all over again, or what if I can still give it to someone > else? The doctor told me that's not factual, since they cleared me as > recovered. He says it's paranoia and anxiety. He wrote me a prescription and > told me to take two tablets every time I leave the house, but it's easier to > stay here. If I'm by myself, nothing else can go wrong.

mental health always falling through cracks in the system

</comment><comment><USER6> The mother seemed to be trying to catch it. Hard to understand.

<comment><USER7> Is English your third language? Did you read the story upside down wearing someone else's glasses?

I guess you won't get the subtext there, since you're so bad at reading comprehension, so I'll just plainly state that somehow you completely misunderstood what you read, like you have no context for humans and you're a robot trying to blend in.

<comment><USER6> I'm sorry, I can't help but feel a little angry at the mom, the daughter runs downstairs while delirious, the mom hugs her knowing what would happen and now she has to live with the guilt.</comment></comment></comment>


This Guy Just Found a Faster Way to Multiply - <AUTHOR> https://www.popularmechanics.com/science/a29514208/faster-way-multiply/ <comment><USER2> No he didn't, he just proved the wellknown and widely implemented Schoenhage–Strassen conjecture from 1971 to be correct. Which was O(n log log n), now implemented for O(n log n).

Schoenhage–Strassen is used for multiplying large numbers with >30.000 digits. Below Karatsuba is used, and for >trillion digits Fuerer could be used. What he did was to improve Fuerer's method from 2007, nobody is using yet.

[https://en.wikipedia.org/wiki/F%C3%BCrer%27s_algorithm](https://en.wikipedia.org/wiki/F%C3%BCrer%27s_algorithm)

</comment><comment><USER3> From my understanding, this won't have any implications on the way ALUs are designed, correct? I was taught that Russian Peasant multiplication[1] was the most straightforward method for implementing multiplication efficiently on hardware. I would have to imagine that performing a Fast Fourier Transform (FFT from the paper) would add significant overhead. Can anyone more familiar with ALU design confirm or deny this?

[1] = [http://mathforum.org/dr.math/faq/faq.peasant.html](http://mathforum.org/dr.math/faq/faq.peasant.html)

<comment><USER4> Is my math off or is RP already nlogn?

Edit: I see my error. N is number of digits, not size of number. By our usual definition of n, table multiplication is log10(n)², which is much smaller.

</comment></comment><comment><USER5> The paper:

[https://hal.archives-ouvertes.fr/hal-02070778/document](https://hal.archives- ouvertes.fr/hal-02070778/document)</comment>


Jsfx; js sound effect generator - <AUTHOR> http://www.egonelbre.com/js/jsfx/

<comment><USER2> Hit the jackpot of sound whilst playing with this ( just hit play ):

[http://egonelbre.com/js/jsfx/index.html?load=%5B%22square%22...](http://egonelbre.com/js/jsfx/index.html?load=%5B%22square%22,1.0000,0.4000,0.8400,2.0000,0.2610,2.0000,1178.0000,1332.0000,183.0000,0.7640,0.7180,0.0000,3.1294,0.8219,0.7720,1.0000,1.0000,0.5000,1.0000,0.8000,1.0000,1.0000,1.0000,1.0000,0.0000,0.0000,-1.0000%5D) ( if thats not a fantastic teleport sound I don't know what is )

Amazing work!

</comment><comment><USER3> That stuff is insane

Btw, this is kind of a close to <http://thirdcog.eu/apps/cfxr>

(watch the vid)

</comment><comment><USER4> That's the kind of stuff I want to read here on HN!

</comment><comment><USER5> What a great project.

FYI: the sliders don't show up in FF5, you should maybe add some browser compatability messaging.

</comment><comment><USER6> Extremely cool. I had no idea sound was possible with Javascript.

Let the great experiment begin!

<comment><USER7> Continuously generating-and-playing audio in JS isn't yet possible, except via a few experimental browser-specific APIs (Webkit-based browsers and Firefox have two different ones). But, if you want to generate a discrete portion of sound (e.g. a few seconds), you can produce a .wav file, shove it into the DOM somewhere, and then play it back. So something like this works, but a version that changed the tones in real time as you dragged the sliders wouldn't.

</comment></comment><comment><USER8> the randomize button is a lot smarter than it looks. is there or can there be a 'download .wav' button?

</comment><comment><USER9> that is amazing!</comment>


Ask YC: Learning to hack... - <AUTHOR>

One of the common pieces of advice given by the hackers on this site to the non-hackers is to "learn how to hack." In the past few weeks, I have come to the realization that I, too, would like to "learn how to hack." Here's the problem: Not having an extensive technical background, this piece of advice seems to be quite vague. With which languages should one start in order to learn to hack?
   I sent a similar version of this question to PG about a month back and he referred me to the creators of Appjet, a YC startup he said was working on this very problem. After looking at the site and receiving an email from one of the founders, the answer I gleaned was to start with learning Javascript. I would like to know from this community whether I'm on the right track to "learning to hack." <comment><USER2> Saying you want to learn how to hack is incredibly ambiguous and in the strictest sense of the word pretty difficult to achieve. A "hack" is essentially a clever solution to a programming problem or a solution to a problem that's outside the norm and requires extensive domain knowledge. Judging by the comments it seems the term has been diluted to pretty much be a synonym with "programming" or "coding."

If you want to learn how to code, the first question is what do you want to code? Do you want to make websites? Create PC games? Create mathematical algorithms? Use math to create images... the possibilities are endless. Learning Javascript would be useful if you want to create websites, obviously, but outside of that it's not a general enough language with enough (any, really) general applications to be a good place to start. It's essentially an odd combination of Object Orientation and Functional programming which is cool in that it introduces you partially to two separate kinds of languages simultaneously. It can also be confusing for the same reason.

If you want...

A multi-purpose language that will let you rapidly prototype programs and is both extremely readable, easy to understand, and has modules for just about everything: _Python_

A language that forces you to learn about more fundamentals of programming and doesn't keep you from royally screwing up: _C_

A language with extremely high-level features, infinite flexibility and power: _Lisp_

A language that lets you create extremely stable applications built for concurrency: _Erlang_

A strictly web-only language with a huge supported base in the online hosting business and about 1,000 frameworks: _PHP_

The latest craze in web development: _Ruby_ with obligatory framework _Rails_

\</comment><comment>---

I could go on and on and on... hopefully one of these descriptions scratches your itch. Keep in mind these are merely suggestions and there are many other languages that would fill any of the descriptions I gave.

EDIT: I am a firm believer that regardless of your end goals to really learn programming one must start at the beginning. Learn about computer architecture, how CPUs and memory work; learn the pain of programming Assembly and managing memory bit by bit even though you'll never do it again. Then move onto C; learn how even the most fundamental of abstractions made programming possible for lesser mortals. Keep moving forward and come to comprehend that features such as garbage collection, while taken for granted, are in reality really freaking awesome and will help you keep your hair longer.

Move up the ladder, rung by rung, until you have the confidence and knowledge to use a language that is highly abstracted and has all the "batteries included," like Python. At this point the language/interpreter/compiler will do most of the work for you but you'll have the experience necessary to understand, more or less, what's going on behind the scenes. From there, if you want to specialize, choose a language created for the domain that interests you; PHP, Erlang, Matlab, etc.

A brief 5/10/20 years later, you're a hacker. Nobody will give you a diploma, though.

<comment><USER3> "I am a firm believer that regardless of your end goals to really learn programming one must start at the beginning."

Absolutely - I've never met a hacker who didn't know the fundamentals.

<comment><USER4> I definitely agree with this -- although maybe not that you have to _start_ at the beginning. I started with lots of python/Java, but then dropped down to where I had been staring at enough assembly that moving back up to C was a huge relief.

I think it's nice to get to learn programming and just get going with something like python, where you can take advantage of all the libraries like other people have mentioned, and maybe have more fun right away and avoid being turned off by it all -- before learning all the lower-level stuff -- not that you shouldn't learn it though!!

<comment><USER2> Right. I guess the real step one is to find out if you even _enjoy_ programming. You're correct that it's much easier to determine this realistically if you start by playing with one of the higher-level languages you will eventually end up using.

Good call.

</comment></comment></comment></comment><comment><USER5> learning html and javascript could be a good place to start since you can make things right away with those.

i think that is very important when you start, to get instant feedback, making stuff is fun :)

for this reason i would recommend python as your first language. it is very highlanguage and have extensive libraries. it has clear sparse syntax and is very easy to write and read.

but that doesnt mean it is a weak language, ont he contrary it is a very powerful language that is used by a lot of big companies and small startups.

so not only is it a good language to start with but it is also something you will have great use for in general as a skilled hacker.

while paul graham and a lot of others rave about LISP(right fully, its a powerful, flexible language with some very special features) i wouldnt start with it. it lacks a standard and you start working on a higher level of abstraction right away, there is not a lot of libraries that you can just install and start having fun with right away as you can with python. i definitely think you should learn it later on but not at the start.

with python you can add the pygame library and start making simple games right away, it is very easy. you can write webapps with webpy/django without much knowledge needed.

then as you evolve you should pick up something like the art of programming(or a similar book that has code in a language you know) and learn about all basic algorhitms and datastructures. if you want to be a serious hacker you need to know the theory.

python however lets you create cool things right away without knowing that much.

while this is a rant for python it is not fanboyish, i just think python is very suited for programmers of all skillevels. there was an article on here some days ago about pythons suitability as a beginners language.

there are other options obviously, ruby for example his fairly similar to python in a lot of aspects but when python tries to keep things simple(there should preferrably be one obvious way to do things) rubys philosophy is there is more than one way to do it.

you could ofc learn java(java and javascript only have the name in common) which is wide-used language that you will run into sooner or later anyway. on this board youll probably find that it often gets critisized for being a language for average programmers though. it is fairly verbose and in some peoples opinion too objectoriented and dont offer any options.

the downside with python and ruby are that they can be pretty slow but that isnt an issue for most applicatiosn and certainly not for the apps youll write a s a beginner.

ruby also has a very popular webframework called rails(ruby on rails).

c/C++ are probably the fastest(executionwise, not developmentspeed) and are also good to learn but are a little more difficult to start with and c is fairly low-level. good to know but wouldnt recommend it to start with. devc++ is a good development environment if you do though.

<comment><USER6> Good place to learn Python:

<http://www.diveintopython.org/>

<comment><USER7> Not for a non-hacker. That book is great for programmers who want to learn python, but someone new to programming would need something more like "Wade Into Python."

<comment><USER6> Good point. I stand corrected.

"Beginning Python - From Novice To Professional" is good starting point for a newbie. Very clear and smooth.

<comment><USER8> For a complete newbie, the Zelle book is probably even better. I think it is called "An Introduction to Computer Science Using Python." MIT and a few other schools use it now, but it is a very gentle introduction.

HTDP.org is also really good if you go the Scheme route instead.

<comment><USER6> Thanks.

</comment></comment></comment></comment></comment></comment><comment><USER9> Your best bet is to get a project and make something. It doesn't matter what the project is or if anyone besides you will ever use it.

There will obviously be a lot of debate (and possibly some downmodding) over this, but I would say that the language doesn't matter. I think the important part will be finding resources that you think will be helpful for you (a book, a tutorial you find on google, a friend who knows programming). In your case you want to spend as much time coding and as little time on set up and administration as possible.

Figure out how to do something easy, then see if you can start tweaking it and adding small things. You'll be a regular hacker in no time

</comment><comment><USER10> <http://news.ycombinator.com/item?id=149482>

<http://news.ycombinator.com/item?id=127952>

<http://news.ycombinator.com/item?id=123903>

<http://news.ycombinator.com/item?id=90782>

<comment><USER11> Along those lines: [http://www.gabrielweinberg.com/startupswiki/Ask_YC_Archive#t...](http://www.gabrielweinberg.com/startupswiki/Ask_YC_Archive#toc38)

</comment></comment><comment><USER12> One thing I would like to point out, that you probably won't like to hear ..

If your purpose is to learn how to "hack" to the point where you can make real webapps, you should be aware of two things.

1\. You will never become a good hacker if you are only doing it for the money or for the recognition. You have to be able to love the code for the sake of the code itself. You have to be able to see the beauty in the code. Otherwise you will never really understand it. It is the difference between being able to copy a poem out of a book and being able to write a new poem. 2\. Assuming you are a true genius, the process will still take you many years. Working hard will not be enough. You will have to learn how to work smart to learn all the stuff you will need to know.

If your purpose is to understand hacking to the point where you can better understand technical people so that you can work with them better .. that is a much more realistic and reachable goal. For that I would recommend you start with Ruby or Python and try to make a full-scale production webapp, and from that experience you will learn a lot that will explain how a technical person thinks.

<comment><USER13> I really agree. I think there's probably a lot of people out there who read about Gates and Brin and Page and Zuckerberg, and say, "omg I want to be a haxor it wud be fun and I can make $$$!". I hate to say it, but if you _really_ wanted to hack, you would have done it before you read about how much money haxors are making. You would have started like us, in QBASIC, because you were exhilarated at the thought of controlling that computer screen...

</comment></comment><comment><USER14> I think I understand the underlying reasoning behind the question. If like me, you have read a lot of the posts on this site, and the essays of Mr Graham, and have a desire to somehow start a start-up, or become your own boss etc., you search for ways to achieve this/these goals.

Most of the advice is fairly general, and I was often left with the feeling that many people just repeat the mantra: 'learn how to hack', 'do something you want, and users will like' etc etc. All good advice I may say, but nothing too concrete on how you actually start!

As hackers, most people will say: "you need to code, and business people, although relevant aren't required straight away."

I sort of agree and disagree. What does not work though is doing nothing, or procrastinating(in the wrong areas).

For me, I decided to at least do SOMETHING. I came from a systems engineering background (read:good Microsoft/Cisco techie, and reasonable sales/presentation sort of guy) - not a hacker though.

So, I decided to attempt to learn Java about a year or two ago. I have done OK with this, but unless you use a language in anger most days, you don't get very far. My latest attempt is Python. I like Core Python Programming 2nd edition by Wesley J. Chun.

What I guess I am trying to say is, you need to work out where you want to be (successful business), and take small steps towards it. That maybe learning business stuff, and it maybe learning to hack. Whatever. Just do something that is a step towards where you want to be, and continually re-evaluate how it's going.

Good luck.

</comment><comment><USER15> Learning the essence of programming is more important than learning a specific language, once you master one of them it's very easy to pick up the rest. So I'd pick an easy one such as PHP and use online resources like w3schools.com to get me started.

Also, I find it very effective to learn by looking at examples on line first and create similar problems to solve. That way, my head remain active to think through the problem solving process. It also helps memorizing the syntax because you see the purpose of the code instead of just passively taking it all in.

</comment><comment><USER16> You are trying to learn how to program, but many people are suggesting that you should pick a language based on what you plan on doing with the language. I don't think that should be your primary consideration. The primary consideration should be the quality of introductory material in that language. I think you'll find most material to be in Java, followed by C++ and Python.

Assuming you want to do web programming, I would suggest you start with Python as your first language. It has quality introductory material[1] and is widely used for web programming, as well as many other tasks. There is less complicated syntax that you have to ignore while learning how to program compared to Java and C++, and is generally regarded as a Better Language. Even if you weren't planning on doing web programming, Python would still be my suggestion for a first language. I'd only suggest anything different if your main concern was getting a generic programming job, in which case Java would be a better idea.

I feel like JavaScript would be a particularly poor choice as a first language since most introductory JavaScript material isn't trying to teach how to _program_ ; it typically teaches how to manipulate web pages. As a new programmer yourself, it will be harder to judge what texts you should pick from those that are available.

If you start out trying to learn how to program web applications instead of learning how to program in general, you'll be selling yourself short. Your code will be worse and it will be more difficult to figure out how to solve atypical problems that a web framework hasn't solved for you. Don't do it.

[1] Or so I hear. I haven't read introductory material in years, and then it was C++. Try _How to Think Like a (Python) Programmer_ : <http://www.greenteapress.com/thinkpython/>

</comment><comment><USER17> I've always liked Eric Raymond's advice:

<http://www.catb.org/~esr/faqs/hacker-howto.html#skills1>

BTW, it's notable that Eric used to recommend Java as a second programming language, and that he no longer does.

The only thing I'd add to his recommendations is a pitch for Ruby. I think it's a really good first language.

</comment><comment><USER18> Teach Yourself Programming In Ten Years <http://www.norvig.com/21-days.html>

</comment><comment><USER19> you should find some small thing you need to do for yourself : for example searching through all the images on your computer, and seeing which one is of a particular format (jpg, bmp for example) or which one was created/modified in the last couple of days. only then should you approach learning a programming language (i recommend python for starters, because it's sweet and has a lot of libraries, letting you concentrate on what you want to do instead on arcane problems) and then learning the techniques necessary to solve your problem (you'll see that most tasks you can solve programaticaly have a certain template).

It's important to start with a real task, and not only do the simple training programs you'll find in every manual/tutorial. I can't stress enough the difference between grasping a concept and actually being able to apply it in the field.

Second, after you learn your first language, don't ever think of sticking only to it. Learn your second, learn your third and your Nth language and never stop learning them. Sure, most of them are similar (python is like ruby, perl, lisp, c is like c++,c#,java) but there are some oddballs like prolog and haskell which will make you a better programmer just by knowing that they exist and what they're about). Thus you'll have a tool for every job.

Also, after a while, learn assembler (x86) and some hardware theory (how a computer works, what's in a CPU or a GPU etc.).

Finnaly, have fun! Don't feel pressured to know anything you don't need or don't think will make you a better programmer. Don't learn Java and it's assorted libraries because every job wants you to know it. Don't learn LISP because the cool kids are doing it, and don't write programs you won't use.

</comment><comment><USER20> The nice thing about the Javascript recommendation is that by using the "View Source" option in your browser, you can get lots of real-world examples to read. Reading code is a great way to learn how to hack (and the way we used to do it, back in the day, when access to computers was very limited).

I'd suggest the following: find a web page that interests you, view the source, and try to understand what the source code is doing. Start with getting a vague overview, and keep drilling down into each layer of abstraction, asking questions repeatedly, until you truly grok every line.

When you can do that, you'll be well on your way.

<comment><USER21> You might want to add that the "View Source" option is only in Firefox once the Web Developer add-on is installed. Now that you mention it, I would suggest installing this add-on for many reasons, but I found personally that the way a majority of code is presented in books and the way it is actually implemented in the real world are two different things. This hurdle has been the greatest for me with regards to CSS. Books, at least the ones I have been exposed to, tend to fail at actually showing the real world applications of those principals. The most progress I have made in that past has been through getting your hands dirty and digging through other individuals code.

To the author, I would suggest finding a website that you feel accomplishes something you would like to create (I would start small, like a personal blog just because the amount of code might be overwhelming with larger sites). Borrow their code using the add-on, and change certain features to see effect the change makes to the product. I would continue the process for a while until you get a feel of how all of the code works together. Only after this point I would suggest going into books. Maybe you will learn differently that I do, but reading a book without actually being able to have hands on experience with what I am attempting to grasp has a hard time sinking in.

<comment><USER22> > You might want to add that the "View Source" option is only in Firefox once > the Web Developer add-on is installed.

No, it's a standard part of Firefox. "View Selection Source" is too, which can be useful.

I agree that Web Developer (or similar plugins) can be a great way to learn.

<comment><USER23> Web Developer and FireBug for FireFox are the best way I know of to discover the inner workings of a web page.

FireBug even has a javascript REPL built in!

</comment></comment></comment></comment><comment><USER24> I suggest _why's lovely Hackety Hack (<http://hacketyhack.net>)

Reasons why H-ety-H is a good choice: 1\. Ruby - Ruby is a beautiful, easy to learn, modern language that's both powerful and easy for beginners. 2\. Hand- Holding - It walks you through all of the interactive examples 3\. Community - Hackey Hack encourages users to share their programs with others

Besides, if eigth graders can do it, so can you!

</comment><comment><USER25> What is it that you want to hack? What do you want to make?

For me, I was a kid and I started hacking basic on an Atari 400 with a whopping 2k of ram. I was trying to make a dungeon crawl game. I learned about variables, conditionals and looping. Then I learned about I/O.

Your environment is, of course, vastly different than mine was, but the process is the same.

</comment><comment><USER26> Learn JavaScript, Python and PHP. It will take time but eventually it will pay off. Moreover, once you learn/master any language please always pay attention to everything that comes with the package. In linguistics, the ultimate goal is to become proficient enough to start _thinking_ in a language. Being a non- native speaker, I know how hard it's for me to start thinking in French and English (despite knowing a few things). Develop a sense of dissatisfaction. Always find ways to optimize things. Build something that lies between "Hmmm, I guess I cannot build it" and "Well, I guess I can do it given I spend time doing it" (read: Psychology of Flow). Do a product, albeit small. I've observed that you hack with altogether a different mindset when you're developing a product with a belief that it can some day pay you back somehow. And please once you're an expert JavaScript programmer always capitalize the S in JavaScript ;)

</comment><comment><USER27> Frankly, I would advice against learning Javascript. When you try to interact with HTML, it's very inconsistent, and it's difficult for you to get a grasp of the rules and what is supposed to happen where.

I would suggest starting with Java. Java is a modern language, you can get a free compiler and IDE, there are tons of excellent books, and you will learn all the concepts of object oriented programming without having to deal with the 'interesting' parts of the machine.

Set yourself a goal, for example to create a GUI app that organises your MP3s or so. Then work on that as your way of learning Java.

After you know Java, the other imperative languages are mostly about syntax. There are no real new concepts to learn.

Now, I do not use Java personally. I use C++, and I started with VB. It's useful to learn C++ early, but only attempt this if you are a dedicated and patient person.

</comment><comment><USER28> i hadn't noticed Eric Raymond keeps updating "How to become a Hacker". No mention of the in-vogue ruby, haskell, erlang, scala, scheme, smalltalk, OCAML, etc, but good stuff.

<http://www.catb.org/~esr/faqs/hacker-howto.html>

</comment><comment><USER29> Like schtog said, I think python is a great choice. You can easily write programs that actually _do_ things without having to get too deep into the ugly details.

Most of all, write programs that are fun. For my third year software engineering class we wrote a 3D implementation of bomberman. It was a blast to write and I still hack on it now and then to add crazy over the top powerups.

Also, it looks like you've found a good place to get your news from. Being aware of cool new projects / languages / ideas is always a good thing.

</comment><comment><USER30> Adding to what everyone else said, you need a framework for the programming. These are skillsets & you can only pick them up if you practice extensively. You need something to practice at.

Install ubuntu 8.04 & explore the add/remove apps. Install netbeans & write a hello world app. Install quanta & edit some saved web pages. Get mediawiki working & use it for notes. Install a simple mediawiki extension. Edit it, or write your own.

</comment><comment><USER31> this is a good thread for me to read. i have found it difficult just to realize what i need to know to start working on projects that i have in mind. i have so far been working with html css and some java script to design static web-sites but am really unsure as to where to go to make a simple web-app with a database to store user information that produces a dhtml page.

i was thinking of picking up a mysql book for dummies yesterday but decided to put it off till i finish the one on javascript. though even then im not sure if that's what i need to know or if its the right choice

with so much buzz about rails everywhere i was contemplating picking up something on that but then there is python too, and i havent a clue what exactly they do. then there is the new google app engine. will that help me? i signed up, but am yet to fully delve into exactly what it does for a developer. i guess there is alot of ways to cook an egg. but this forum really helps alot.

</comment><comment><USER32> See <http://news.ycombinator.com/item?id=135494> for a previous discussion on this topic and <http://news.ycombinator.com/item?id=135575> for my previous recommendation.

</comment><comment><USER33> It really depends on the domain you want to work in. Do you want to create a web service? Do you want to create systems? Do you want to create a tech start-up? I assume the answer to the last is yes, but even then it is not necessary for every founder to be a hacker, but it certainly does help.

</comment><comment><USER34> For all intents and purposes, you will need to learn (at least) 2 technologies, one for the client and one for the server.

On the client, you have 2 primary choices: HTML,CSS, & Javascript or Flash/Actionscript. I would definitely opt for HTML,CSS, & Javascript because it's open source and it's everywhere.

On the server, you have many more choices for languages and databases. One of those languages is Javascript, although not too many people use it on the server.

I think you're on exactly the right track. Learn how to build web pages first with HTML, CSS, and Javascript. Then when you're ready to "connect" to the server, ask the next question, "What is the best language/data base to use on the server." But stand back because sparks may fly from your terminal screen as we hackers fight with each other trying to answer that one :)

Good luck.

<comment><USER35> "For all intents and purposes"

Not quite - there are other applications of programming than web and web services...

<comment><USER34> Right. I was just making a few assumptions and trying to keep it simple and helpful.

</comment></comment></comment><comment><USER36> as a novice, but language bigot it depends a lot on what you want to make.

Low level, kernel, command line C is king. Python for slightly slower but faster coding.

For an GUI program use C++ (qt is a good toolkit).

For Web Pages learn html/css and then add javascript. After that I would say mysql (or postgresql. I like postgresql better) and php. you can trade php for perl/python/ruby if you like.

Web means learning a lot of languages but to me seems the easiest.

</comment><comment><USER37> [http://iamelgringo.blogspot.com/2008/05/teach-yourself- you-t...](http://iamelgringo.blogspot.com/2008/05/teach-yourself-you-to-hack- in-6-9.html)

</comment><comment><USER38> Ruby and Python seem to be in vogue for web applications. I'm a Ruby guy because I got tired of reading through flame wars and decided to pick one of them randomly.

</comment><comment><USER39> <http://norvig.com/21-days.html>

</comment><comment><USER40> angel tech is a good book

</comment><comment><USER41> Good knowledge of OS is a must..then some C, Perl will some handy...If you know unix shell programming, u can become a master in hacking...

and also, if you are a DB guy like me, then even with ordinary SQL, u can hack a lot of stuff...

<comment><USER42> [comment that looks archaic once parent is modded up]

Which is to say a hacker also needs a taste for recursion.</comment></comment>


Show HN: Knowyourself.ai – infers your personality from your Tweets and HN posts - <AUTHOR> https://knowyourself.ai <comment><USER2> My favourite part of KnowYourself.ai is the word cloud image of my most used words. I also love seeing other people's word clouds.

I wish that the default share text included a hash tag e.g. #KnowYourself or #MostUsedWords so I could easily find what other people are sharing.

I was a beta user of Peepeth.com and later a crowd fund supporter which KnowYourself grew from. It was as I used Peepeth that I learnt to fully appreciate that my public words on social media could be analysed by advertisers, employers and governments.

KnowYourself.ai gives me the power to analyse my public posts for myself on platforms such as Twitter and Peepeth. Also to see how my messaging can change over time. Participating in the crowd funding campaign for Peepeth gave me credits for KnowYourself.ai which I am enjoying using (I need to work out how to gift). My word cloud shows that auto posting things like the weather can dominate the words in my feed. (Maybe it's time to stop that IFTTT feed).

As a long term user of Peepeth I got to know and become friends with Bevan. He is a passionate maker. Pouring so much of his energy into making great products. I have loved using Peepeth and attending the retreat in Bali that Bevan hosted.

I recommend trying out KnowYourself.ai and seeing what your public posts show about you and then feed that in to how you post in future.

Looking forward to my word cloud on a coffee mug. (Though maybe once I stop with the weather posts)

<comment><AUTHOR> Thanks Andrew!

I've added the #KnowYourself hashtag in the share messages.

Still working on the gifting functionality for credits; will let you know when it's ready.

Would also like to create a filter so you can recreate your word cloud without certain words (your weather words, for instance).

Thanks for the feedback!

</comment></comment><comment><USER3> Why does this tool require me to log in with my twitter account instead of just giving my account name and letting it pull my tweets from the public feed? I am curious what it concludes about my tweets but am not going to give it (limited) access to my account (which says a lot about me, I know).

<comment><AUTHOR> Thanks for asking- KnowYourself only lets you query your own content, and authing with Twitter ensures that.

The app only requests read permissions, and the only non-public data it asks for is your email address.

</comment></comment><comment><AUTHOR> Hey all,

Wanted to show you my latest product. KnowYourself.ai infers your personality from your social media posts.

I built KnowYourself to be a fun and easy way to learn more about yourself, and to make the web a bit more self-aware.

It's a freemium model- there's a free Twitter report, and paid premium reports (monthly plans and one-off credits).

I'd appreciate your feedback. How could it be improved? Who / which groups would get the most value out of this?

KnowYourself also supports Hacker News (secret feature), and you can have a full HN report for free. To find out your HN personality:

1) Sign up on KnowYourself.ai (by getting your free Twitter report, or registering here: [https://knowyourself.ai/hn](https://knowyourself.ai/hn))

2) Then, visit [https://knowyourself.ai/me?hn=true](https://knowyourself.ai/me?hn=true)

3) Follow the on-screen instructions.

Thank you!</comment>


Ask HN: Data Scientists / Statisticians of HN, how do you verify your numbers? - <AUTHOR> Hi everyone. I'm in need of guidance as to how to verify the correctness of my work.
I'm a trained software engineer, and spent most of my career programming applications, of various kinds: mobile apps, websites, backends... About a year ago, I got a new role that requires me to sift through quite a lot of data and extract numbers from it. Basically, we have a big SQL database full of events, and I'm responsible for building reports based on those events.
That's a lot of fun! The database has grown to a size where it actually becomes a challenge to run those big aggregations, and I must tweak my queries carefully. 100% recommend to anyone interested in... well going through a lot of stuff.
However there's one aspect of it that's a lot less fun: I find it hard to get a "yes, those numbers are correct.".
At first, I assumed I could simply write unit tests, reproduce the data that we should have in the production database, and verify my numbers on smaller datasets.
The reality, however, is far from that. While the events respect established schemas, some were lost. Others didn't arrive in order. Some are duplicated, and others seem related but actually are not. Also, due to the sheer volume of records, I can't just start counting them manually.
I realized, on multiple occasions, that some queries were incorrect. Of course the resulting numbers were incorrect too. But we didn't notice that immediately, and some reports had to be recalled/corrected. No harm was done, but I did lose some sleep over it.
So there's that. While I very much enjoy working with that data, I wish I could find a way to validate my work, and finally be able to click on a "resolve" button knowing that my results truly are correct.
Is there any magic trick I'm missing? Any tool that could help me? Any workflow that would be my data unicorn? <comment><USER2> The trick is to realize the act of verifying the accuracy of the data "is" the science. I come from High energy physics background. People think the point is to find sexy new particles or make ground breaking discoveries. No the real goal is to understand the inner workings of every part of your data acquisition pipeline (from the hardware detector to the software analysis) so well that you eliminate all possible other explanations when you spot a deviation. That then is your discovery.

I was a DS for a couple years in NYC area. My advise is the same as in science, treat the act of verification as the end goal itself. If you understand your experimental apparatus, in this case, how the DB is populated then you understand all subsequent of the "physics". Anything you can't understand should be carefully quantified as random and systematic errors.

Sorry I don't have any magic bullet or standard recipe for how to go about it. Every case is different.

<comment><AUTHOR> That does help, thank you. I will look into making the verification a core part of the process.</comment></comment>


Canada considering internet link tax - <AUTHOR> https://www.theglobeandmail.com/business/commentary/article-heritage-minister-raises-possibility-of-link-tax-for-internet/ <comment><USER2> I can understand why Canada and almost any other country would be upset about the huge rivers of money flowing out to a foreign tech company, but this tax looks like it'll just make things more expensive for the Canadian consumer. I think a better way to protect Canadian interests would be to require that personal data, servers, and support staff serving Canadian consumers be located within the country.

<comment><USER3> What happens when every country you operate in wants this?

<comment><USER4> If you're company is based in US/India/China you can just go "fuck it" and only offer your services internally, and still becoming absolutely ludicrously filthy rich because your home country is the size of a continent with individual states/provinces producing more economic output than some old world nations.

If you aren't based in an absurd world superpower than you just go bankrupt.

<comment><USER3> Kinda reminds me of this podcast / transcript about one person's ideas on the coming geopolitical reorganization

[https://www.chartercitiesinstitute.org/post/charter- cities-p...](https://www.chartercitiesinstitute.org/post/charter-cities- podcast-episode-6-peter-zeihan)

</comment></comment></comment></comment><comment><USER5> Internet and service are already garbage here compared to nearly every where else in the world. I've got two choices for home internet where I live and 'two' choices of phone company, both owned by the same parent company. I pay twice as much for half as much data as most other places in the world. My carrier subjects me to random throttling i've still to figure out entirely what's affected and when after years, Canada's internet policies have favoured big business for as long as they've existed with the consumers being constantly screwed over.

To be honest, this has been going on longer than there's been internet with phone and cable providers. This is just another push in the favour of big businesses who can afford to pay over others who can't, while the government takes its, generous, cut.

</comment><comment><USER6> > Facebook this week told the Australian regulator that removing Australian > news content from its site would not materially affect its revenues, adding > that it sent more than two billion clicks to Australian publishers in the > first five months of 2020 worth hundreds of millions of dollars.

How do they get that? Is it based on what Facebook would charge an advertiser for a click? I don't know anything about it, but 2 billion clicks being worth hundreds of millions of dollars doesn't pass the smell test for me.

<comment><USER7> Why not? The cheapest click you can get is $0.05, which is nearly impossible. But let's say it was possible, it'd be 100 million.</comment></comment>


Going Solar: Six Month Update - <AUTHOR> http://www.extremetech.com/article2/0,2845,2338834,00.asp

<comment><AUTHOR> The original solar system selection process is documented here <http://www.extremetech.com/article2/0,2845,2308674,00.asp> and covers details like the up front costs</comment>


Je Suis Circonflexe: Anguish from Changes to French Spelling - <AUTHOR> http://www.economist.com/news/books-and-arts/21693551-why-minor-fiddling-french-spelling-causes-such-anguish-je-suis-circonflexe <comment><USER2> This makes me sad. I'm currently learning French and with all the problems around spelling and homonyms with French in general, the circumflex words are the only ones I can reliably remember/learn! Because I'm familiar with Spanish, the circumflex helps to guess the meaning of new words because the circumflex is usually present to indicate a missing "s".

For example, feste -> fête, hostel -> hôtel, and isle -> île.</comment>


The Assignment Problem and the Hungarian Method (2005) [pdf] - <AUTHOR> http://www.math.harvard.edu/archive/20_spring_05/handouts/assignment_overheads.pdf <comment><USER2> [http://zafar.cc/2017/7/19/hungarian- algorithm/](http://zafar.cc/2017/7/19/hungarian-algorithm/)

Article above describes a very short implementation of the most efficient form of the algorithm (40 lines of code).

</comment><comment><USER3> One interesting use of the Hungarian algorithm is in git-tbdiff [1] (topic branch diff). It uses it to compute which patch in an older topic branch match each patch in a newer topic branch.

[1] [https://github.com/trast/tbdiff/blob/master/README.md](https://github.com/trast/tbdiff/blob/master/README.md)

</comment><comment><USER4> For those who learned point set topology from Munkres's book, you may be interested to know that this is also sometimes called the Munkres algorithm: [https://en.wikipedia.org/wiki/Hungarian_algorithm](https://en.wikipedia.org/wiki/Hungarian_algorithm) .

</comment><comment><USER5> I recently used this algorithm in a project. For anyone looking for a high- quality Java or Clojure implementation, I would recommend:

[https://github.com/timothypratley/munkres](https://github.com/timothypratley/munkres) (Clojure)

Which wraps:

[https://github.com/kevin-wayne/algs4](https://github.com/kevin-wayne/algs4) (Java)

</comment><comment><USER6> Can somebody explain the reasoning behind step 5?

It's clear that the algorithm works (assuming it converges), because it only uses manipulations that preserve optimality. Why, though, should I choose these manipulations instead of others that also preserve optimality?

<comment><USER7> You are simply making another zero (because you don't have enough to cover all rows and columns) without making any other cell negative. If you subtracted from uncovered columns and added to covered rows you would get the exact same result. Every uncovered cell is subtracted from and every line crossing is added to.

The only arbitrariness is in the selection of covering rows and columns. There I guess the argument is that if a cell has another zero on its row and another zero on its column (i.e. a line crossing in any possible covering) then it is not going to be in the optimal solution.

</comment></comment><comment><USER8> Anybody have a year for this?

<comment><USER9> Based on the parent url and content I think it's 2005:

[http://www.math.harvard.edu/archive/20_spring_05/](http://www.math.harvard.edu/archive/20_spring_05/)

<comment><USER8> Thanks!</comment></comment></comment>


Homomorphic encryption implementation - <AUTHOR> https://hcrypt.com/ <comment><USER3> According to [https://hcrypt.com/scarab-library/](https://hcrypt.com/scarab- library/), the functions that you can do on encrypted texts are

      Add cyphertexts (= XOR)
      Multiply cyphertexts (= AND)
      Add with carry in and carry out
      Add with carry out

I'm not really familiar with homomorphic theory, but it doesn't seem like you could build that much on top of these primitives.

<comment><USER4> You may want to start here:

[https://en.wikipedia.org/wiki/Group_homomorphism](https://en.wikipedia.org/wiki/Group_homomorphism)

The way I've understood it is that only addition has been possible so far. With a multiplication operation, you can get a field, and do "everything" with the real/complex numbers.

<comment><USER5> Well a homomorphism applies to a group and a group can only have one operator by definition. That won't change probably. Rings have two operators, fields are rings that support cancellation and integral fields I believe support inverses as a rule and where you'd get "everything" I believe.

<comment><USER6> "Well a homomorphism applies to a group"

Not so. The notion's far more general: [https://en.wikipedia.org/wiki/Ring_homomorphism](https://en.wikipedia.org/wiki/Ring_homomorphism)

This gives us a notion of homomorphisms between fields.

[https://en.wikipedia.org/wiki/Homomorphism](https://en.wikipedia.org/wiki/Homomorphism) for more general notions of homomorphisms.

<comment><USER5> You're right. My bad. Thanks for correcting me.

</comment></comment></comment></comment></comment><comment><USER7> If I were Google I would fund the heck out of homomorphic encryption research.

As Google tries to gobble up even more of everyone's data and every waking habit as they try to improve Google Now and other services, the privacy concerns are only going to grow bigger. It will be not just a drain on their public image (just like it is on Facebook's image), but it would also give competitors a lot more opportunities to take jabs at them. Think Scroogled, but 10 fold.

Homomorphic encryption would pretty much fix all of that, and they wouldn't even need to give up their data collection (or not as much).

<comment><USER8> Unfortunately not. HE has some performance issues (that might be solveable with time) but the primary issue is that Google do actually want to see exactly what you're doing. It's important for basically everything Google does that makes it money. Furthermore, the privacy concerns are nowhere near great enough. All of the mainstream privacy concerns have come from Google releasing your information to others (Google Buzz, that recent ECHR case etc) and not Google keeping information themselves. Furthermore, since I assume that what you're really talking about here is emails, it would be impossible to practically implement a HE system for emails that would allow for searching (because one could derive meaning from emails by sending them the right queries). It'd be no better than client-side encryption, and no one really wants that.

No, the people for which this is really useful is large non-IT companies like Boeing or healthcare companies etc. They have huge data processing or storage requirements, and frequently have to run around hoops in order to ensure data is not transmitted in a way that would breach commercial agreements or confidentiality requirements. The problem is that frequently these workloads (especially for companies like Boeing) will involve much data, and HE tends to come with a large size increase and performance drop - so it's often too costly, which is a shame.

<comment><USER9> Can you expand a bit more on the type of commercial agreements and confidentiality requirements of Boeing? In particular, what kind of data would the agreements pertain to, and who would the other parties be? Would the 'hoops' occur only for internal data transmission, or is it because they need to share it with 3rd-party organisations? I'm a researcher in the area, and I'm familiar with stuff like that from the healthcare domain, where you have e.g. lots of different hospitals that want to share certain data for research purposes, but I'm curious as to the use cases more commercial organisations might have.

<comment><USER8> Yup - Ars can explain it better than me! [http://arstechnica.com/information- technology/2014/04/how-bo...](http://arstechnica.com/information- technology/2014/04/how-boeing-merges-its-data-centers-with-the-amazon-and- microsoft-clouds/) Clearly, though, this would be a case where homomorphic encryption would not be required - because they've solved the problem without it.

<comment><USER4> Boeing's technique sounds more like steganography than cryptography.

<comment><USER8> It's neither. They make the assumption that their combined cloud partners aren't all in cahoots, and distribute their data such that the probability that a) one can intercept enough chunks and b) one can infer meaning from the chunks is very low.

The cynical part of me wants to say that it's very similar to a standard mapreduce, and the security properties came for free/very little.

</comment></comment></comment></comment></comment></comment><comment><AUTHOR> Can anyone else verify the existence of a truly homomorphic encryption scheme?

This would be great for an encrypted file synchronization project I am working on; being able to diff an encrypted file would be amazing.

<comment><USER10> I can verify their existence, but you don't need to take my word for it - <USER2> of the research on fully homomorphic encryption is publicly available through the IACR's online preprint database, [http://eprint.iacr.org](http://eprint.iacr.org) . I doubt FHE is a good solution for your project, though. The technique is not quite ready for primetime due to seriously slow performance. It gets better every year but it will be a while yet before you'll be able to diff an encrypted file using FHE.

EDIT: Though now that I think about it you might be able to code a custom diff circuit for FHE that borders on feasibility. It depends on the algorithm used, really. File diff seems spiritually similar to some of the bioinformatics work being done with privacy-preserving edit distance computation on encrypted DNA sequences.

EDIT2: Here's a paper that discusses an FHE 'hardware platform': [http://eprint.iacr.org/2014/106.pdf](http://eprint.iacr.org/2014/106.pdf)

<comment><AUTHOR> I was not aware of the lackluster performance. That is really too bad.

The consistent feedback I get about file sync (from techies) is they really don't want to be uploading a whole file every time they change something. I agree, but to have client-side encryption such that you're data is protected from the server host as well as people who gain access to said server seems to preclude efficient sync with current tech. Which is again too bad.

I am no crypto expert so rolling my own / building off this FHE is not really possible. I do understand usage of things like AES and RSA well enough though to know that adequate security precludes diffing/efficient sync.

<comment><USER11> > seems to preclude efficient sync with current tech.

I disagree. You can deterministically chunk the large file with one of the existing rolling checksum schemes, and then use the chunks as your primitive instead of whole files. The server only sees encrypted chunks. The client knows to only upload changed chunks.

That still leaves important choices to be made about cipher mode, key management, etc. But it's not intractable.

<comment><AUTHOR> Assuming you are using AES with CBC, best case you will have to resend every block after and including the first one that changed.

My understanding is that CBC is among the most secure forms of AES encryption because it is essentially impossible to have patterns in data (unlike EBC). Practically speaking then one must assume that it is common to upload most of a file. Any software that boasts this security cannot effectively 'diff' your files.

EDIT: Formatting

<comment><USER12> You don't have to apply the CBC mode to complete files. If it is secure for a 1 MB file, I don't see why it would be insecure for 100 parts of a 100 MB file.

If you manage to merge small files into the same blocks, you even gain some privacy because the server can't even tell the number of files anymore.

[1] also has a discussion of the trade-offs of the different modes of operation for whole disk encryption. That seems related here because nobody wants to rewrite the whole disk after changing the first byte.

1: [https://en.wikipedia.org/wiki/Disk_encryption_theory](https://en.wikipedia.org/wiki/Disk_encryption_theory)

<comment><AUTHOR> I understand now. that is a clever idea; I like it a lot.

</comment></comment></comment></comment></comment></comment></comment><comment><USER13> Would anyone comment on [http://www.wisdom.weizmann.ac.il/~oded/PS/obf4.pdf](http://www.wisdom.weizmann.ac.il/~oded/PS/obf4.pdf) ?

"On the (Im)possibility of Obfuscating Programs"

"In this work, we initiate a theoretical investigation of obfuscation. Our main result is that, even under very weak formalizations of the above intuition, obfuscation is impossible."

I haven't spent too much time meditating on the implications of the paper, so I was hoping to hear from someone who has. Does this paper strike a mortal blow to FHE? Or is this paper pointing out a result that can be ignored in practice?

When I get a few hours I'll spend some time trying to come up with examples of what the paper is talking about and hopefully discover the truth for myself, but if someone here has already done that, please chime in!

<comment><USER10> This is talking about a completely different primitive (indistinguishability obfuscation) so FHE is fine. In fact this is talking about an exceptionally strong characterization of IO (virtual black-box) which is not used in current research on the subject.

<comment><USER13> Would you expand on this?

From the abstract: "Informally, an obfuscator O is an (efficient, probabilistic) "compiler" that takes as input a program (or circuit) P and produces a new program O(P) that has the same functionality as P yet is "unintelligible" in some sense"

That sounds like an exact description of FHE, and the paper seems to claim something is impossible. So I'm trying to figure out: If the paper isn't claiming FHE is impossible, then what is that "something" and how does it relate to FHE?

<comment><USER8> So the obvious difference is that if I run an obfuscated program, I'd expect to get the same output as the original program - the obfuscated program is to an extent expected to perform as a normal program, but be unreadable.

With HE, you operate on encrypted data and what you get out is still encrypted data - you never have knowledge of what the data is, you just know you've done something to it. So I could perhaps add two encrypted numbers together to obtain something I know to be the sum (but have no idea what the sum is).

<comment><USER13> Ok, so if I understand correctly, it's impossible to write a program which hides its intent? That is, reverse engineering of what a program is doing will always be possible?

Something doesn't seem quite right, though. Programs are data. If it's possible to perform operations on data without knowing the data, then shouldn't it be possible to perform (useful) computation without revealing the algorithm? If not, why not?

<comment><USER8> No. It's because they're different things. The idea here is that we perform the operation, but the output is not known either. So my data can pass through my system with me doing whatever I want to it, and when it comes out it can be decrypted again.

The closest analogy (I think you are trying to get to this) is that one might be able to make a series of operations that would execute an encrypted program. The problem is that this would result in encrypted output.

<comment><USER13> _The closest analogy (I think you are trying to get to this) is that one might be able to make a series of operations that would execute an encrypted program. The problem is that this would result in encrypted output._

Thanks for your time. That's what I was trying to get to, yes. Would you help me understand why encrypted output would be a problem for the interpreter? If the input data is encrypted, and the data defines a program which can be executed, and the result of that execution is more encrypted data, then (for example) why can't that encrypted data be fed back into the interpreter as further input? Or transmitted over the network to a computer with the decryption key (so that the encrypted output can be used in a meaningful way, without revealing to the original computer what was computed)? In other words, why is encrypted output any more of a problem than operating on encrypted data in the first place?

<comment><USER14> Obfusticators are pretty different from FHE. Instruction based obfustication takes common instructions like mov eax, 0xff and replaces them with more complex instructions that do the same thing but in a very indirect way while other common methods include mostly name mangling, compression, and encryption. To modify the code or dump it, you need the unpacker to somehow decrypt the code and load it into memory.

The point of FHE, however, is to encrypt private data to send it to a third party to perform operations on that data without that third party having the keys necessary to decrypt the data (and thus see what it is). With FHE, the third party can modify the data but must then send the encrypted result back to the client who then uses the original keys to decrypt the result and look at it. The client can't see exactly which operations were performed and the third party can't see the original data.

I don't know if a FHE based interpreter is possible but since you need to have the original keys to read from an FHE payload, I don't think so.

<comment><USER13> Let's try a direct question: why can't the third party perform the operation "is x equal to 5"?

The point of HE is that you can do add and mul. FHE extends HE to do arbitrary computation. "Is x equal to 5?" is an arbitrary computation, so it seems like FHE must support it.

If comparison ops are supported, then you can make an FHE based interpreter. Thus far, no one has been able to explain why _specifically_ it wouldn't be possible. And if it's possible, then programs are naturally obfuscated: the interpreter is operating on encrypted bytecode input. But the original paper I cited says this is impossible! So there seems to be an interesting mystery here.

<comment><USER15> > why can't the third party perform the operation "is x equal to 5"?

They can; the output is an encrypted boolean.

> And if it's possible, then programs are naturally obfuscated: the > interpreter is operating on encrypted bytecode input. But the original paper > I cited says this is impossible!

No it doesn't. Isn't your paper talking about an obfuscated program Ob(P) that does the same thing as P? P takes plaintext to plaintext, so Ob(P) takes plaintext to plaintext.

Homomorphic encryption is a different thing. Hom(P), if it exists, takes ciphertext to ciphertext. (Or possibly plaintext to ciphertext if it's a public-key cryptosystem.)

<comment><USER13> How is the output to a conditional an encrypted boolean? A conditional is "if this, do that." That means if the conditional is true, the branch is executed. The output wouldn't be an encrypted boolean; the output would be whatever the branch does. Right?

<comment><USER16> I am sure they meant that the output of a conditional _expression_ would be an encrypted boolean.

A conditional _branch_ would become an encrypted conditional branch. That means, you wouldn't be able to infer the branch from the encrypted output.

Let's take a simple example. I have an algorithm X which takes two integers and returns an integer.

Let's say I want to run this algorithm on a third-party VPS. I obfuscate the algorithm X, in order to hide the operations that X does. Let's called the obfuscated version OX.

I host the OX algorithm on the third-party server, and start supplying it pairs of integers. The third-party is observing the set of inputs and outputs:

      (4, 4) => 8
      (3, 2) => 5
      (1, 0) => 1

From these observations, the third-party would be able to infer what OX does. (simple addition in this case).

Now, however, if I use HE, I will get an algorithm (HX) which takes encrypted input and spits out encrypted output. A third-party will see the corresponding log of inputs and outputs like this:

      h3830120cjfakj, 102309123clals => 293sdlzlxdf
      94ka3lkc.zdkf, 102kksdllz => 1923939nddd

From these observations, it is impossible to know what HX does (as per the theory).

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER17> HE is supposed to be the holy grail of cloud computing, but last time I looked at it, it was dog slow. I glanced at the site, but saw no performance figures...

</comment><comment><USER18> Thank you so much for posting this now! I was writing a paper about this topic and this arrives just in the right time! Awesome.

<comment><AUTHOR> Glad I to be of help, I would be interested to read your paper when it is publicly available!

</comment></comment><comment><USER19> Sometimes I am really glad that Bootstrap exists.

</comment><comment><USER20> I was hoping for some homophobic encryption.

</comment><comment><USER21> Totally read this as homophobic encryption...</comment>


MPAA Chief Suggests Backroom Negotiations On New SOPA Are Well Underway - <AUTHOR> http://www.techdirt.com/articles/20120405/13292918393/chris-dodd-suggests-backroom-negotiations-new-sopa-are-well-underway.shtml <comment><USER2> > _"But he won't [having a public discussion], because Dodd is a coward and a > DC-insider who only knows how to cut deals, not how to actually respond to > the public's best interests."_

SOPA was a malignant offshoot of a dying industry, but that doesn't justify being irrational.

Chris Dodd is no longer a Senator, he's now the chairman of an industry trade group - he isn't responsible for the "public interest" any more. Cutting deals in public would dilute the MPAA's negotiating leverage. He isn't being private because he's a coward but because he's not a clueless negotiator.

It's the Lamar Smiths who are the clear and present danger. The sitting representatives and senators supporting this crap are the ones violating their responsibility to the public interest, they're the ones warping the chess board. Dodd? He's another player.

</comment><comment><USER3> 1\. Hollywood is dwarfed by Big Internet companies. Big Internet is more important to the economy of both the U.S. and the world than Hollywood is.

2\. The MPAA declared war on Big Internet with SOPA.

What's really going on is that, thanks to that SOPA wake-up call, companies like Google and Comcast now have lobbyists on the ground in Washington greasing palms and promising campaign contributions, and they can offer a lot more cash than the MPAA can. The MPAA has grown accustomed to being able to buy politicians but now they're being outbid.

<comment><USER4> The MPAA represents the interests of companies that are owned by media conglomerates that control the mainstream news media. They strongly influence who wins elections. Don't underestimate their power.

<comment><USER5> The influence of the mainstream media is waning. All they do is gather information into a centralized location, editorialize, and distribute the information. How much longer will people listen to them instead of their friends on Twitter?

<comment><USER6> Long enough to to let the Internet get fucked over.

And I'm sorry, but outside tech people I know almost nobody who really uses Twitter. I know a ton of people who have Twitter accounts and almost none of them ever use it.

The influence of mainstream media is not waning. The influence of proper journalism is waning though, and that's pretty scary.

</comment></comment></comment></comment><comment><USER7> Of course he says this, he's trying to peacock some power. I'm guessing the "negotiations" aren't going so well if he's broadcasting this, but one of the side-benefits to him and his talking like this would be to discourage activism. Once again, the __AAs act undemocratically.

Update: original: [http://thehill.com/blogs/hillicon- valley/technology/220181-d...](http://thehill.com/blogs/hillicon- valley/technology/220181-dodd-says-sopa-can-live-again)

</comment><comment><USER8> Big Internet is going to have to spend a lot more money if they want to win the war. Big Media & friends are outspending Big Internet when it comes to lobbying. According to Opensecrets[1]: Computers/Internet spent $125.6M in 2011.

OTOH, TV/Movies/Music spent $123.2M and Telecom spent $58.9M. But that's not all of their allies. If you look at the SOPA supporters list[2], many of them produce brand name goods, like Tiffany. They're worried about counterfeiters. Let's call this potential group of supporters, Misc Business. In 2011, they spent $491.2M.[3]

It boils down to incentives. For them, there's limited downside to supporting SOPA-like legislation, and a lot of upside in deterring pirates.

1: [http://www.opensecrets.org/lobby/indusclient.php?id=B12&...](http://www.opensecrets.org/lobby/indusclient.php?id=B12&year=2011)

2: [https://sites.google.com/site/boycottsopasponsors/home/list-...](https://sites.google.com/site/boycottsopasponsors/home/list- of-supporters-and-sponsors)

3: [http://www.opensecrets.org/lobby/indus.php?id=N&year=a](http://www.opensecrets.org/lobby/indus.php?id=N&year=a)

<comment><USER9> It is sad to see that laws are essentially being auctioned like this. Is this what you would call Free Market Politics?

</comment></comment><comment><USER10> I thought they said they've learned their lesson after the SOPA backlash. Clearly they haven't.

<comment><USER11> Like everyone who's ever been caught doing something bad, they learned an important lesson: don't get caught.

</comment></comment><comment><USER12> This might be relevant: testpac.org

They were doing an AMA on reddit today.

Basically its a super PAC run by redditors. Their first goal is to unseat Lamar Smith.

<comment><USER13> For clarification: Their goal is actually to just decrease his victory to 70% of the votes. He's basically been unchallenged for decades and has had over 80% of the vote forever. By damaging his success by even 10%, you get to the real reason for doing this...

Not because it will solve all the world's problems, but as a demonstration of power. Showing that the movements can be organized by the internet, and they can have real world effects.

</comment></comment><comment><USER14> I think that in the end, "we" cannot win this one. This is happening in the US, and the EU is up to the same game. We all know that when business and governments want some thing, they get it. People will be harassed and jailed so that failing businesses can be kept on life support so that they may influence "democracy" on behalf of politicians. Yes, its corrupt and vile, but its is also inevitable.

<comment><USER11> But most businesses don't want SOPA. It's one tiny minority of businesses that control most of the media that wanted SOPA. So while it may _sound like_ business and government want SOPA, it's simply not true; the media companies are merely telling everyone it's true.

(That's why they got so mad when the Internet companies, which have a lot of reach, fought SOPA. It resulted in an epic loss for them. And it will continue to do so, because the government isn't in the business of protecting tiny industries at the expense of much bigger ones.)

</comment></comment><comment><USER15> It is a good thing to have tech companies talking to content companies out of the glare of the press. They'll be more likely to understand each other and find common ground.

Tech and content are not enemies or even competitors, they are inextricably linked pieces of the economy who must work together. That's why they fight so much; it's like brothers sharing a small room.</comment>