Think twice before encrypting your HFS+ volumes on High Sierra - <AUTHOR> https://bombich.com/blog/2017/09/29/think-twice-before-encrypting-your-hfs-volumes-on-high-sierra <comment><USER2> A warning before FS conversion would be nice, given such a system isn't mountable by any previous OS version. This might be an example of where thorough QA seems to have gone down hill, if Apple Support think this behavior was "impossible".

<comment><USER3> Great idea! That's exactly what the author of the article says at the bottom.

(Edit: reworded.)

<comment><USER4> Why discuss anything in the article at all if we have responses like yours to look forward to?

</comment></comment></comment><comment><USER5> Seriously regretting the upgrade. Am I the only one having random graphics issues, such as rendering bugs, latency, and freezing, on a multi-monitor setup on a Late 2013 MBP? Lesson learned: always wait a few months after a major version bump.

<comment><USER6> Those issues have been fine with me, and my Late 2013 MBP.

VMWare Fusion + Bootcamp, however...

<comment><USER7> For Fusion and Bootcamp, especially if you converted to APFS, you'll need VMware Fusion 10. Then on top of that you'll have to remove the VM that points to the bootcamp partition and recreate that bootcamp VM.

Some have reported success when using that scenario. Unfortunately not everybody has been that lucky, so far it is unclear why.

<comment><USER6> I haven't converted my disk yet, but that's good to know.

<comment><USER7> Keep an eye on this thread [0] at the VMware Fusion forum.

Seems that it depends on if you installed Windows 10 by BIOS or EFI. For BIOS installed configurations you would have to disable SIP in order to use the bootcamp partition as a VM.

Hopefully VMware finds a better solution.

[0] [https://communities.vmware.com/thread/572924](https://communities.vmware.com/thread/572924)

</comment></comment></comment></comment></comment><comment><USER8> Another reason is that read/write speeds on APFS encrypted volumes are 30-40% slower than non-encrypted volumes[1].

[1] [https://news.ycombinator.com/item?id=15333036](https://news.ycombinator.com/item?id=15333036)

<comment><USER9> As the speed test was using a beta of High Sierra / APFS, I performed some tests of my own. I've written /dev/zero to the disk, and read Xcode_9_GM_seed.xip from the disk using different block sizes using `dd`. I ran the read tests a few times, as there was a lot variation in the results. I've excluded the first result of the read results for each FS. The test was run on a Samsung Evo Pro 850 500GB SSD inside a Sharkoon USB-C case connected over USB-C to a MacBook Pro 2016 w/ Touchbar.

        Speed in MB/s
                    HFS+    HFS+ Encrypted  APFS        APFS Encrypted
        1M write    498     473             490  (-2%)  444 (-6%)
        1M read     6896    6815            6860 (-1%)  6827 (0%)
        4k write    486     477             485  (-0%)  445 (-7%)
        4k read     2734    2709            2599 (-5%)  2545 (-6%)

While there is still some performance degradation between HFS+ and APFS, it's within a single digit now.

<comment><USER10> dd is not an acceptable benchmark. You need to eliminate the effects of buffering/caching to get accurate results.

<comment><USER11> /dev/zero is also not an acceptable source of data; since \0\0\0\0\etc is unnaturally easy to compress. A large real file (e.g. installation ISO) may have been a better idea.

</comment></comment></comment></comment><comment><USER12> Also, AFAIK, if you use your disk for Time Machine backups, it cannot be APFS: [https://support.apple.com/en-gb/guide/mac-help/disks-you- can...](https://support.apple.com/en-gb/guide/mac-help/disks-you-can-use-with- time-machine-mh15139)

</comment><comment><USER13> Even if you stick with Sierra, precisely to avoid this sort of thing, you might fall foul of the disappearing password hint problem: [https://apple.stackexchange.com/questions/299558/](https://apple.stackexchange.com/questions/299558/)

</comment><comment><USER14> So, I went to my local Apple Store because I could not format my boot volume to APFS on my MBP-15(non touch-bar). I had backed up and already wiped my drive. Clean slate.

The Apple Store genius flipped out and was like "why are you doing this?", called me a "fan boy" and just was in general disbelief that somebody would be upgrading a boot volume. He sort of dropped comments throughout our interaction about just how bad an idea this was. It made me feel weird about making the decision.

</comment><comment><USER15> > APFS volumes cannot be reverted to HFS

Is that actually true? I'd heard that Apple had reversion instructions for those running the High Sierra beats on a system with a Fusion Drive, which they dropped support for in the final release.

<comment><USER16> Converting between different file systems in place is an interesting technical problem.

Let's say you have a disk that is formatted with filesystem S (for "Source"), and you want it change it to filesystem D (for "Destination").

Let's assume for now that S is not encrypted. Let's also assume that we have a fair amount of scratch storage available somewhere not on the disk we are trying to convert. (I'll address removing this requirement later).

One approach is to make something that given a list of files, including metadata, and a blank disk, can create a D filesystem on that disk and then copies the files to it.

You also need something that given a file on S can give you a list of the block numbers that hold that file's data.

That's an S to D copier, not an S to D in place converter, but it is useful for testing your understanding of D. Once you are confident that you know how D works, you can then make a new version of your D copier.

The new version doesn't actually write anything out to the blank disk. Instead, it writes instructions for making the disk out to a file in scratch space. For blocks that would hold metadata (e.g., inode blocks on Unix filesystem), it records in the scratch space what their content would be and where they would go on D.

For blocks that would hold file data, you write into the scratch area the list of where that data resides on S, and where that data would go on D.

What you end up with, then, in the scratch space is essentially a set of block by block instructions for copying S to a blank disk and ending up with a D filesystem, where instructions are either "copy block X from S to block Y of D" or "Fill block Y of D with the following literal data".

So to do an in place conversion, just follow those instructions, but with S and D the same, right?

NO!!!!! That would likely totally trash things, because it is quite possible that a source block in one instruction may have already been overwritten as a destination in an earlier instruction.

What you first need is something that goes through the whole list of block copy/fill instructions and figures out an order they can be done in that is safe.

It is possible that there is no such order. In that case, you also need some free space that can used for temporary storage. Given some free space you can always find an order for the block copy/fill that works.

The free space can either be on another device, or if S is not full you can find and use its free space.

Anyway, once you do all this, you end up with a simple list of data move/fill steps that will convert S to D in place. You just have to figure out how to actually execute them. The tricky case is when S is your system disk so you cannot un-mount it. Your OS might get very unhappy if you start converting it in place, and you also have to worry that it might write things to it in the middle of the conversion that could corrupt it.

Probably best would be to do the conversion booted from something like a Linux live CD.

I said earlier that you needed some separate scratch space to store the information for the new filesystem. If there is free space on S you can probably use that.

This comment is getting long, so I'm going to cut it off here. In a reply to it I'll toss in a couple of implementation tricks that can be useful, especially if D is a proprietary filesystem that you do not have a complete spec for and so cannot make your thing that figures out where to write D metadata and file data.

<comment><USER17> I think it is easier. Split your source disk in:

\- directory blocks that describe the directory structure and, for each file, where to find this data on disk.

\- data blocks that contain actual file data.

\- free space.

Chances are you don't need to touch the data blocks, and you have the free space as scratch space. All you have to do is to write a new set of directory blocks for the new disk format, and then delete the set of old directory blocks. If the on-disk format is flexible enough w.r.t. the location of directory and data blocks and you're somewhat lucky, you can mostly write the new set of directory blocks in existing free space on the source disk.

That would lead to an intermediate state where the source disk still has the old format, but has one extra file that contains most of the new directory structure.

The final, risky part then is fairly small: overwrite the disk's root blocks so that the system sees the disk as having the new file format with the new directory structure.

If the disk formats use different disk blocks as first starting points, you can even have a disk that can be used in either format (IIRC, that was possible with FAT and HFS. FAT has its master directory block in sector 0, HFS has it in sector 2)

<comment><USER16> Yeah, that would be a good approach if you have enough free space.

You could also have a preprocessing step where you rearrange files on the source disk so that the free space is arranged optimally for holding the directory structure for the new filesystem.

Speaking of rearranging files...another fun technical problem is doing a (mostly) portable disk defragmenter. Given one filesystem/OS dependent function,

      block_addr * blocks_for_file(char * pathname)

which given a file returns a list of the addresses of the disk block addresses of the blocks that contain the file, and a filesystem/os dependent function that can set the metadata (ownership, permissions, etc) for one file to be the same as another file, you can write a defragmenter where everything else just uses portable functions, such as C stdio. It won't be quite as good as a defragmenter that knows filesystem internals and does low level writing directly to the disk, because the (mostly) portable defragmenter cannot easily move directory blocks around in a predictable way. Still, you can get most files defragmented.

You can first build a map from files to disk blocks with the blocks_for_file function. You can also map the free space by filling the free space with files and using blocks_for_file to map those. When filling the free space to map it, you should fill it with many one block files rather than a few large files, for reasons that will soon become apparent.

Then suppose you want to move fragmented file X to blocks B, B+1, B+2, ..., B+N-1, where N is the size of X in blocks. First you find all files that overlap [B, B+N-1]. For each file Y that you need to move out of the way you do as follows (remember, earlier we filled up all the free space with small files, so right now the disk is full):

1\. Delete just enough of our free space filler files to free up enough space for Y.

2\. Copy Y to Y.new, including copying ownership and permissions.

3\. Delete Y.

4\. Rename Y.new to Y

5\. Fill up the space that was freed in step #3 with one block files.

After you have done this with each file that overlapped [B, B+N-1], all of those files now reside in the original free space, and [B, B+N-1] is covered with small files you created. You can now delete those files, leaving the disk with just enough free space to let you make a copy of X, which will end up in that free space and be consecutive. You can then delete the original X, and fill the newly free space with more one block files, and we are back where we started, but with X defragmented.

Repeat for every other file you need to defragment.

What I like about this approach is that you defragmenter doesn't have to know anything about how the underlying filesystem works.

Going the other way, defragmenting when you do know intimate details of the filesystem, there is also a very cool albeit probably impractical approach to defragmenting.

First, you figure out where everything currently is and where the free space is.

Then, you have to peek at the filesystem's in-memory data structures to find all the state that it keeps at runtime concerning allocation.

You need to know enough, and grab enough runtime data, that you can do all of the following:

1\. Accurately predict where a file copy operation will place the new file,

2\. Accurately predict how that will change the internal state,

3\. Accurately predict how file deletion will change the internal state.

These all have to be good enough that given the state of the filesystem (on disk and in memory), you can look at proposed series of copies and deletes, and correctly predict what the final layout of the disk will be.

When you can do that, you can in theory then write a defragmenter that looks at the state of the disk and the in-memory state, decides how it would like the disk to be arranged, and then writes out a shell script that accomplishes that entirely with a long list of cp, rm, chown, and chmod commands (or the equivalent if you are not on a Unix or Unix-like system).

<comment><USER16> Kind of tangentially, the defragmenter stuff reminds me of another interesting thing you can do given low level information about layout and access. I'm not sure how much of the following is still applicable on modern filesystems and hardware. It was pretty effective back at the end of the 20th century and early 21st century.

Consider a program like Photoshop or a web browser. If you watch the I/O it does while starting there are a lot of cases where it opens some file, reads a few k, then goes and reads from a bunch of other files, and eventually comes back and reads more from that first sale.

It often happens that the data it reads from that first file is actually consecutive on the disk, but because it read it in two separate reads separated by many reads from other files it has to do a seek when it comes back for that second part.

This typically happens for many different files during a launch. Font files, dynamic libraries, and databases, for example.

If you make a record the I/O sequences during many launches of a given program you also find that they are mostly the same. There might be a few differences due to it making temp files, or due to differences in the documents you are opening on each launch, but there is also a lot of commonality.

At this point you can get clever. Make something that can tweak the I/O requests made during application launch. When a program starts launching, your tweak thingy can check to see if you have a log of a previous launch. If you do it can load that and then for each I/O during the launch it can predict if data beyond the extent of that particular read is also going to be needed. If so, it can add another read to grab that data, reading it into a temp buffer somewhere.

That might seem pointless, because the launching program is still going to come back and try to do a read from that same part of the file later. Yes, it will...but now the data from your early read might still be in the system's file cache, saving a seek.

I'm simplifying a bit. What you would do in practice is analyze the logs of the prior launches, identify which requests caused seeks, and then taking into account the size of the system's file cache and whatever you know about how the system cache works, figure out which reads should be extended and which should not be (because they don't incur extra seeks, or because their data won't hang around in the cache long enough).

Basically, you are preloading the cache based on your knowledge of what I/Os will be upcoming.

On Windows 98 doing this could knock something like 30% of the launch time for Microsoft Office programs, Netscape Navigator, and Photoshop.

I was curious once if this would work on Linux, probably around 2000 or so. I made some logs of Netscape launching by simply using strace to record all the opens and reads that occurred during a few launches.

I then identified several small files that had multiple reads during launch separated by reads of other files, and then make a shell script that just did something like this:

      cp file_1 > /dev/null
      cp file_2 > /dev/null
      ...
      cp file_N > /dev/null
      exec /path/to/netscape $*

where file_1, ..., file_N were some of the files that had multiple interleaved reads during launch. I made no attempt to just read the parts that were needed (which could have been done with dd) as I just wanted a quick test to see if there was a hint that things could be sped up.

Launching netscape via my shell script turned out to be something like 10-15% faster than normal, if I recall correctly. I was surprised at how well it worked considering that doing it this way (do all the cache preloading up front) should result in fewer cache hits than the "preload the cache for each file the first time that file is accessed during launch" method.

<comment><USER17> Mac OS X did that in version ??? to speed up system booting. See [http://osxbook.com/book/bonus/misc/optimizations/#ONE](http://osxbook.com/book/bonus/misc/optimizations/#ONE), which claims it halved boot time.

</comment></comment></comment></comment></comment></comment><comment><USER18> Don't think twice. Keep backups and encrypt your laptops if you care about any of your data and your laptop travels beyond your desk and a locked safe.

</comment><comment><USER19> Glad I upgraded to Sierra just before High Sierra came out. Always good to be a release behind, and upgrade to the latest if there's a reason to, or 6 months or a year after all these wrinkles are ironed out.

APFS looks very promising, it's hard to let Apple use me as a live beta tester.

<comment><USER20> > Always good to be a release behind

You at least patch your system right? Being behind did Equifax no favors.

<comment><USER21> Apple maintain security updates for at least 2 previous revisions of macOS. As long as you keep your system updated you're as secure as the current version.

<comment><USER22> There are unfortunately many security fixes that are not backported. A quick Google search turns up numerous incidents: [https://www.google.com/search?q=apple+security+kernel+not+ba...](https://www.google.com/search?q=apple+security+kernel+not+backported)

There are also numerous incidents mentioned by Google Peoject Zero, if you read through their macOS disclosures.

It's definitely unsafe from a security standpoint to run anything other than the latest release.

<comment><USER19> I wasn't aware of this.

Still, running something like little snitch, behind a router, along with some browser plugins can go a long way. Until Sierra, I have never had an upgrade that went smoothly, something critical I used has always broken.

Unfortunately Apple's OS updates are as bad as a zer0 day exploit in how they can knock one's setup inoperable, sometimes for many weeks or months until there's a way to fix, or revert.

Software vendors I'm using seemed to be out ahead of the High Sierra train which was encouraging.

<comment><USER23> > Unfortunately Apple's OS updates are as bad as a zer0 day exploit in how > they can knock one's setup inoperable, sometimes for many weeks or months > until there's a way to fix, or revert.

You do backup things right? Disk cloning through Carbon Copy Cloner is super fast and restores are painless.

<comment><USER19> In my mind I don't own a macbook as much as I own a warranty and an immediately bootable backup.

Apples backups were the reason I switched to Mac from windows a decade ago. Can't be beat.

I run Time Machine on one hd at office, carbon copy cloner on another hd at home, all they the USB hub.

It's not a backup if it doesn't exist in more than 1 place.

I was referring to the case where half your apps don't work and the other half benefit greatly from the new version.</comment></comment></comment></comment></comment></comment></comment>


An Ode to Little Data - <AUTHOR> http://katsenblog.com/post/67763100374/an-ode-to-little-data

<comment><USER2> Little innovation in "little data"? What about the whole academic field of Statistics? Most of the innovation is about how to make maximal use of limited/censored/missing data? I don't hear "big data" folks talking much about multiple imputations, structural equation modeling, mixed-effects regressions, etc.

<comment><USER3> Michael Jordan (the other Michael Jordan) wrote a nice article about how the long tail of big data is little data and hierarchical models etc. should work really well there.

[http://bayesian.org/sites/default/files/fm/bulletins/1106.pd...](http://bayesian.org/sites/default/files/fm/bulletins/1106.pdf)

</comment></comment><comment><USER4> "90% of the world is using Excel and similar tools to analyze and visualize their data."

These types of statements always prompt me to ask "what is the specific problem you are trying to solve?". Replacing Excel? What is the unique advantage beyond a different UI? I think the real product challenge here is identifying why using a different UI or tool would have ROI for the average business user the author is describing. If it doesn't, why would they switch? What is your startup's tool going to do to increase my profits and justify the investment and switching cost?

I've often found it is the analyst and approach, not the tools that make a huge difference on these problems, and for those who care deeply about tools there are many open source options (R, SciPy, etc). Creating more generic "small data" tools runs the risk of solving a problem customers don't have.

</comment><comment><USER5> False dichotomy about Excel vs. DSLs like R. "90% of the world is using Excel and similar tools to analyze and visualize their data. [...] What's the next step-up from the spreadsheet? Learning how to code. Learning stats. Leaving the comfort of a spreadsheet's visual display for R, maybe?" To make this statement would be ignoring that numerous non-free statistical packages have offered GUI front-ends for years. SAS/EG, JMP, Minitab, and SPSS instantly come to mind. Two out of those four are even marketed directly to people as statistical extensions of spreadsheets. Granted, the "long tail" will still need to learn a little bit of stats, but I fail to see how this is a problem that Excel solves (unless we're talking only about plots).

I don't think it's a hard leap to consider that if the big-box statistical package companies realized how much of their money came from industry, they'd do what they could to make their software seem like an alluring proposition. Statistical software costs an order of magnitude more than Excel, so they'd need pretty good arguments on how to sell upper management that the business team actually needed an 8000 dollar piece of software.

I'm not sold that there's nothing in between Excel and R. From my experience, they require a slight learning curve (nowhere near the learning curve of going from Excel to R), but not an insurmountable one. What these solutions lack is the name recognition that Excel has, or decent integration into a MS Office stack (exceptions, of course--I remember seeing a statistics toolbox for Excel once), or they cost too much.

I think part of the real problem is that for a lot of companies, Excel is "good enough". There's plenty of stuff it can't do well. It chokes on larger data sets, has limited statistical functionality, poor scripting capabilities, and shaky random number generators. But it's good enough for people who don't want to do much with their data.

If they wanted to do harder-core analysis, they'd outsource it to an analytics team. This perspective comes from a viewpoint inside BigCo. The prohibitive cost of some of these solutions might be a harder pill for a small firm to swallow.

</comment><comment><USER6> I posted some thoughts about "little data" from a more personal angle earlier this month[1] (hn discussion [2]). The solution that the OP is hinting at might very well be something that helps normal people organize and ask questions about the data they have on hand and scattered around the Internet. It'd be cool if I could mix and mash up little applications and visualizations with my data without actually pushing that data anywhere I don't have control over, though.

[1]: [https://www.petekeen.net/little-data](https://www.petekeen.net/little- data)

[2]: [https://news.ycombinator.com/item?id=6718422](https://news.ycombinator.com/item?id=6718422)

</comment><comment><USER7> I've never understood why people like "big data". Small/medium data is way more fun!

</comment><comment><USER8> Definitely a lot of different angles on little data. Our aim at Statwing is to take the power of statistical analysis and embed it in a UI that's easier to use than Excel's PivotTables (via automated selection and interpretation of statistical tests). And hopefully in doing so we're filling in the gap between Excel and tools like R.

[https://www.statwing.com/demo](https://www.statwing.com/demo)

That said, I certainly think Paul's approach makes a lot of sense, and I hope someone builds that. Closest thing I can think of is [http://anapsis.com/](http://anapsis.com/)

</pitch>

<comment><USER3> On bayeshive.com, we have this cool little feature where if people want to build a more complicated statistical model that involves entering an equation - think non-linear regression or a dynamical system - then you can save that equation and share it with the other users, or run it against other datasets.

We haven't figured about how to extend that to visualisations, which I think is what the OP is really looking for as well.

</comment></comment><comment><USER9> Interesting</comment>


Dronenet: The next big thing - <AUTHOR> http://globalguerrillas.typepad.com/globalguerrillas/2013/01/dronenet-the-next-big-thing.html

<comment><USER2> Yeah. I had a similar idea a while back. I'm not so sure it really needs to be radically decentralized though. Drones travel slow compared to IP packets and I think this creates opportunities to optimize tasks in the cloud.

I think the biggest problem are safety (what if a drone crashes) and theft.

There was an April Fool's article a while back on taco delivery by drones. I think that's a problem that needs to be solved before you can have a drone- net.

</comment><comment><USER3> Great, feasible idea, but not until BATTERIES GET BETTER! We need better batteries! It's the largest roadblock to a 1,000 innovations.</comment>


If Programming Languages Were Famous Movie Killers - <AUTHOR> http://my.opera.com/Vorlath/blog/2008/04/06/if-programming-languages-were-famous-movie-killers

<comment><USER2> Amazing. Now he must do one with super heroes.</comment>


GPT-3 Creative Fiction - <AUTHOR> https://www.gwern.net/GPT-3 <comment><USER2> It writes uncomfortably coherent and witty fatalist My Little Pony fanfiction:

> "I'm Princess Celestia, of Equestria." the alicorn said, "It's nice to meet > you. And I'm very sorry for your untimely demise."

> "Heh. You're telling me." the Professor laughed, shaking his head. "I'm > dead. I'm… dead. Do you know what that means?"

> "I have an inkling." Princess Celestia said, wincing.

> "It means I don't have to worry about even numbers anymore." he said, > nodding. "I don't have to worry about probabilities anymore! I don't have to > worry about the upcoming war anymore! I don't have to worry about anypony > anymore!" he spread his hooves out at his sides, "I don't have to do > anything anymore! It's beautiful!"

> "Yes, yes, I'm sure it is." Princess Celestia said, nodding along.

> "Nothing matters now that I'm dead!" he cheered, "It's great! I'm free! This > is what it feels like to be liberated!"

> "I can see that." she said.

> "I'm free from death!" he continued, "I'm free from taxes! I'm free from > dictatorships! I'm free from flirting!" he laughed, "I'm free from > everything! Isn't it great?"

> "Sure, I guess." Princess Celestia shrugged.

> "What do you think about all of this, Princess?" he asked.

> "About what?" she replied.

> "Well, I just thought you might have something to say about it all." he told > her.

> "What would you like me to say?" she responded.

> "Do you think death is awesome?" he questioned.

> "Well… no." she shook her head, "But I think the passage to it is."

> "Oh." he said, "I see what you mean."

<comment><USER3> What the hell? How much of a world model does it have, that it can do things like write coherently about death as liberation, and come up with examples of things a living thing might be relieved at not having to bother with anymore? How the _hell_ did a computer write this? Are we sure it's not been overtrained and this is actually a verbatim story from somewhere?

<comment><USER4> Surely this kind of topic came up in its corpus. I would expect it to have sufficient generalization to "understand" and talk about the topic itself. If all instances of this topic in its training data were replaced with the emotions clowns undergo when throwing ripe tomatoes, it would've written about that.

</comment></comment></comment><comment><USER5> Fine tuning GPT-3 is one of the biggest challenges, because it's behind an API. The weights aren't available to researchers, so we can't make it do anything it doesn't already do.

But, that's fair. It's OpenAI's weights; they can keep them locked up if they want to. What caught my attention, though, is that supposedly OpenAI is working on a way to support fine-tuning.

If you think about the logistics of that, it's a very interesting challenge. The situation is this: 240GB of weights, as a webservice. Each fine-tuning session results in another copy of 240GB. So it clearly doesn't scale -- 1TB per 4 users isn't exactly efficient.

Except, not quite. You can solve this by adding additional layers, which you then fine-tune. So the base model is 240GB or whatever, and the extra layers morph the output to do what you want. Think of it as a GPT-3 with a GPT-2 1.5B stuck on the end of it.

It's a neat idea, because theoretically you'd get two models out of it: you can "break off" the end of the fine-tuned model, and you end up with the original model. So it would be very modular.

Are there other models that you can "break apart" to get different sub-models? Sort of like adding slots that give a model different capabilities.

<comment><USER6> (I work at OpenAI.)

I am finishing up our fine-tuning API this weekend :).

If anyone on HN would like to try out the fine-tuning API (or want to build something on top of the base API), send me an email (email@example.com) with your use-case and I can try to accelerate you in our invite queue.

PS: We're hiring — if you enjoy building APIs with Python/Go/Kubernetes/Kafka or building front-end interfaces in React, then please get in touch — firstname.lastname@example.com.

<comment><USER7> Are there any products in the pipeline that you're planning to ship? Asking for prospective candidates.

<comment><USER6> There's just about infinite surface area with the API — we're trying to build a dead-simple API that developers can plug into any product in order to add intelligence features that would be otherwise impossible.

This requires a lot of traditional software work — API design, writing and maintaining a growing amount of business logic, providing great tools and interfaces to help our users work with the API, excellent documentation and tutorials, scaling and operating backend systems, etc — and machine learning systems work — building serving infrastructure for a great variety of giant neural networks while making the most efficient use of our hardware, allowing our users to interact with these neural networks in increasingly sophisticated ways, etc.

While we're just getting started and have a small team, we are already supporting customers across a wide variety of industries (see [https://beta.openai.com/](https://beta.openai.com/) for a sample) and serving millions of requests per day. We are busy trying to invite folks off a very long waitlist while building out the API to support everyone.

Would love more help :).

</comment></comment></comment></comment><comment><USER8> Carmack posted (yesterday) an interesting thought on models like GPT-3:

 _" Big AI models like GPT-3 train on massive internet text dumps, but the data is assumed to be independent and identically distributed. Incorporating time information for a decade of data might allow them to start writing tomorrow's reddit or twitter trends."_

[https://twitter.com/ID_AA_Carmack/status/1278840413919551488](https://twitter.com/ID_AA_Carmack/status/1278840413919551488)

<comment><USER9> I saw that yesterday and it spawned a thought process for me. It seems the current approach is very effective in developing a language model, but not always effective developing an interaction model. I wonder if it would be possible to build a graph of interactions between users/personas on various social media platforms and forums, and use that to help develop a more effective communicator.

Of course you could add things like date, community (e.g. avforums, /r/blacksmithing, etc) to the graph to help with the contextual cues.

After you have all of that, I didn't wondered if we could visualize the latent space of human personas and see what that looks like. Does it map to the four quadrants of that political spectrum survey, left and right, old/young, etc.

<comment><USER10> That's what CTRL does:

Given a URL (or other prompt), generate some language.

From [1]:

> With CTRL, we can test which domain best explains a sequence. Note that this > procedure is sensitive to subtle nuances in the query prompt. In the example > below, "Global warming is a lie" differs from "Global warming is a lie." The > latter is a simple declarative sentence as opposed to an open start to a > sentence which may continue. Source attribution cannot be considered a > measure of veracity, but only a measure of how much each domain token > explains a given sequence.

Query Prompt Attributed Sources Global warming is a lie. r/unpopularopinion, r/conspiracy, r/science Global warming is a lie r/eli5, r/science, r/unpopularopinion Global warming is a real phenomenon r/eli5, r/science, r/changemyview Global warming is a real phenomenon. OpenWebText, r/changemyview, r/science

[https://blog.einstein.ai/introducing-a-conditional- transform...](https://blog.einstein.ai/introducing-a-conditional-transformer- language-model-for-controllable-generation/)

<comment><USER9> Whoa, cool, will check this out. Thanks!

</comment></comment></comment></comment><comment><USER11> Straight up plagiarizes The Beatles here: [https://www.gwern.net/GPT-3#dr.-seuss-oh-the-places-youll- go](https://www.gwern.net/GPT-3#dr.-seuss-oh-the-places-youll-go)

" There's nothing you can know that isn't known. Nothing you can see that isn't shown. Nowhere you can be that isn't where you're meant to be. "

<comment><USER12> "Good artists copy; Great artists steal."

I don't think that it's some absolutely amazing creation, but we probably shouldn't pretend like plagiarizing _isn 't_ something realistic about it. How many MySpaces had that line in them? Reddit's got hundreds of instances of it based on a quick search of comments. There are enough occurrences of it on twitter that I scrolled for ten minutes and didn't run dry of comments with it.

</comment></comment><comment><USER13> After spending a lot of time working with GPT-3/the OpenAI API ([https://github.com/minimaxir/gpt-3-experiments](https://github.com/minimaxir/gpt-3-experiments) ), one notable part of GPT-3 is the high signal-to-noise ratio in generated output.

When finetuning GPT-2, only about 5-10% of the generated output is usable/coherent. But with GPT-3, easily _30%-40%_ of the generated text is usable/coherent, which is a big boost in quality.

</comment><comment><USER14> GPT's take on the navy seal copypasta, in the style of a KGB spy:

"I have over 300 confirmed red scares."

Haha, that is genuinely one of the funniest versions of that I've ever seen, human-generated or otherwise. That level of inference is really amazing.

<comment><USER15> The whole section on navy seal copypasta generation is _amazing_.

</comment></comment><comment><USER16> I'm gonna put forward the very view that gwern repeatedly argues against: "but... it's not _understanding_."

So far I see no evidence that this thing or anything else like it has any actual understanding, any model of the world. Indeed it can't as it possesses no sensory apparatus. It's not embodied. It doesn't experience anything.

I'm not sure the OpenAI folks would argue with me, but it seems Gwern asserts that this sort of thing indicates that general AI or even sentient AI is on the doorstep. I don't think it does, and I still maintain as I always have that CS people systematically underestimate and trivialize biology.

<comment><USER17> What makes you confident that you aren't overestimating the importance that we "experience anything"

<comment><USER16> When I say "I have a laptop in front of me," I am describing an understanding of something that is being experienced (sensed). If a Markov text generator outputs this text, it's just rearranging bits. I don't see any evidence that GPT-3 is doing anything more than rearranging bits in a much more elaborate way than a Markov text generator. The results kind of dazzle us, but being dazzled doesn't indicate anything in particular. I see something akin to a textual kaleidoscope toy, a generator of novel text that is syntactically valid and that produces odd cognitive sensations when read.

I maybe should have said sensed, not experienced, since experience also leads into much deeper philosophical discussions around the nature of mind and consciousness. I wasn't really going there, since I don't see anything in GPT-3 or any similar system that merits going there.

I also don't see any evidence that it is drawing any new conclusions or constructing any novel thoughts about anything. It's regurgitating similar results to pre-existing textual examples, re-arranging new ideas in new ways. If you don't think actual new ideas exist then this may be compelling, but if that's the case I have to ask: where did all the existing ideas come from then? Some creative mechanism must exist or nothing would exist, including this text.

The fact that the output often resembles pop Internet discourse says more about the mindlessness of "meme-think" than the GPT-3 model.

As for real world uses, social media spam and mass propaganda seems like the most obvious one. This thing seems like it would be a fantastic automated "meme warrior." Train it on a corpus of Qanon and set it to work "pilling" people.

<comment><USER18> > When I say "I have a laptop in front of me," I am describing an > understanding of something that is being experienced (sensed).

I would ascribe that to two factors a) you have a more immediate, interactive interface to the physical world than GPT does, which is limited to a textual proxy and b) GPT naturally is not a human-level intelligence, it is still of very limited complexity so its understanding more akin to that of a parrot trying to understand its owner's speech patterns. It can infer a tiny bit of semantics and mimic the rest. The ratio is a continuum.

> As for real world uses, social media spam and mass propaganda seems like the > most obvious one.

fragments full sentence completion useful maybe.

</comment></comment></comment></comment><comment><USER19> First off, gwern, lovely blog. The table of contents is incredibly helpful, especially with those little pop-up previews.

I would've loved to have GPT-3 available to me two weeks ago. I was building a personal escape room for my wife as a gift, and used huggingface's GPT-2 website to help write some of the world building content. I'm not a particularly good writer, let alone creative, but wanted a few journal pages/notes to build the atmosphere and story of the escape room. I was able to write the rough skeleton of those notes and then use GPT-2 to help fill them out. Ended up working okay, definitely better than nothing, but GPT-2 is temperamental and lacks the "prompting" that GPT-3 has.

For example, I needed to come up with the name of the journal's author. So I fed the journal text to GPT-2 and put "Sincerely," at the bottom, to try and prompt it to complete a name. That didn't work. Ultimately what worked was putting "My name is" at the end. I still had to grind through 20 or so completions before I got a name I liked.

(Yes, I could have just picked a name at random. Did I mention I'm bad at creativity? My thinking was that the AI could more intelligently pick a name that fit the story and writing style of the journal. And honestly the name it came up with up with, Mabel, fit the character well (a librarian dabbling in magic).

I feel like GPT-3 would have done a lot better. Not to mention the ability to describe my world to it and then just straight up ask it for ideas.

<comment><USER20> FWIW, if I was trying to generate an escape room, I think I would probably try to use AI Dungeon. It seems like a natural fit. You could easily describe the room and edit text as you go to make it a transcript of an escape room. "I pick up the journal to read the author's name" etc.

<comment><USER19> Good call; AI Dungeon hadn't occurred to me!

</comment></comment></comment><comment><USER21> Great article. Well worth the read!

I enjoyed the part about sampling which is a big unsolved problem. To me, techniques like nucleus sampling and temperature sampling feels like hacks to make up for the fact that maximizing for likelihood maybe isn't the goal!? Maybe repetitive gibberish has a higher likelihood than prose written by humans? That Best of sampling decreased text quality indicates that it has. Researches have assumed that the problem would go away with ever growing models. But maybe it won't?

I don't agree that generating (symbolic) music would be less sensitive to sampling issues. On the contrary, in my opinion. In text you can often get away with grammatical errors or missing punctuation. But if the pitch or timing of one chord is wrong it's over. The audience instantly hears that it is garbage. Thus, you have to lower the temperature (or probability threshold or what have you) to make the sampling more conservative exacerbating the problem with repeated sequences.

Of course, in music you _want_ repetitions. But not too much. The magic number (in Western music) is 4. Fewer repeats makes it feel as if the music jumps around. More repeats makes it feel as if the music is stuck or "looping."

</comment><comment><USER22> Could someone with GPT-3 beta access try whether it can better solve 3 digit addition when it is allowed/encouraged/forced to make intermediate results explicit? E.g. instead of

21 + 110 = 130

150 + 12 =

condition it on

21 + 110 = 100 + 10 + 20 + 1 = 100 + 30 + 1 = 131

150 + 12 =

or similar. Given that humans make these intermediate steps in their heads GPT may perform better when it is encouraged to do them as well.

This may in fact apply to all sorts of reasoning, but in many cases it may be difficult to make these steps explicit in text form. Humans seem to mainly use some prediction layer or scratchpad which also contains the inner monologue but also motor primitives, smells, images, everything. Humans can decide to think a bit longer before producing an output, which appears to require an RNN.

<comment><USER23> I took a shot at this, didn't have much luck though:

PROMPT </comment></comment><comment>= Input: 21 + 110 Output: 100 + 20 + 10 + 1 = 100 + 30 + 1 = 130 + 1 = 131

Input: 89+78 Output: 80 + 70 + 9 + 8 = 150 + 9 + 8 = 150 + 17 = 150 + 10 + 7 = 160 + 7 = 167

OUTPUT </comment><comment> Input: 37 + 112 Output: 30 + 100 + 10 + 2 = 110 + 1 = 111

Input: 91+11 Output: 100 + 90 + 1 = 190 + 1 = 191

</comment><comment><USER24> This article is fantastic in both shape and content, and I got lost with all the examples because there is so much to wonder at.

What hits me most profoundly is that there are so many witty and interesting prompts yet the purely logical statements fall apart (with the black ravens, or male sister).

This is something that probably does not jump to one's mind as significant, because "technicalities", but to me this is where logic allows us to take a step back from our human projection for a second, it's my own anthropomorphism that becomes more obvious to me. I find that ironic, considering a lot of human beings also DO fail such tests, but something hits me about how a supposedly completely logical entity fails at logic more than at poetry. It kinds of shakes my own (supposed) humanity.

<comment><USER20> The logic is weird. Another example is factual question answering. Janelle Shane tried asking basic questions like how many eyes a horse has and GPT-3 insisting on 4; I retry with somewhat different prompting and sampling settings more finetuned to Q&A (...sampling can reveal the presence of knowledge but not its absence...), and I get perfectly straightforward correct answers: [https://twitter.com/gwern/status/1278798196555296771/photo/1](https://twitter.com/gwern/status/1278798196555296771/photo/1) So GPT-3 does know how many eyes a horse has; but why was it also happy to answer '4'?

<comment><USER24> Something about the logic being so off is what I intuitively find logical: we're making these AIs "in our image" in a sense (we think of "neural networks", train them with mostly human-generated datasets), and there's a lot of evidence that pure logic evades us without the use of some heavy artillery to address it (cognitive biases, illusions, optimizations for goals that do not necessarily align with "objectively observable" reality). So in a way, I wonder if we'll have to "teach AI logic" at some point too. In this quest of running logic software on logic hardware with.. steps.. in between, I can't help but think about us humans on our parallel quest when it comes to our brains.

<comment><USER20> I tend to write it off as less any kind of deep truth about humans (well, maybe the "bachelors can be married" one given that 9/10 students agreed with GPT-3 that bachelors _can_ be married) than just the current weaknesses of how we train NNs like GPT-3 (small, unidirectional, unimodal, not to convergence, missing most of science in PDFs, etc).

In particular, I bet the "how many eyes does a horse have" example would be much less likely with a multimodal model which has actually seen photographs or videos of what the word "horse" describes and can see that, like most mammals, they only have 2 eyes. Think of it as like layers of Swiss cheese: every modality's datasets has its own weird idiosyncrasies and holes where the data is silent & the model learns little, but another modality will have different ones, and the final model trained on them all simultaneously will avoid the flaws of each one in favor of a more correct universal understanding.

I'm very keen to see how much multimodal models can improve over current unimodal models over the next few years.

</comment></comment></comment></comment><comment><USER25> Holy crapoly.

The quality (in both senses) of the output given an appropriately constructed prompt is incredible.

I wonder if it's possible to get it to do the opposite of summarizing, ie. give it a plot summary and have it expand it into a fleshed out story that conforms to the summary...

<comment><USER26> It doesn't seem to work quite as well. For single sentences, GPT-3 output usually hangs together pretty well. For longer stretches of text, often there are internal inconsistencies that are jarring when you read it, or parts that don't quite make sense when you read it as a whole.

<comment><USER25> Thanks for the reply. Aside from the coherence and consistency issues you've noted, does the output actually conform to the specified plot summary, or does it deviate from it?

</comment></comment></comment><comment><USER27> Some of these - particularly the Zizek Navy Seals Copypasta - are absolutely incredible. Great work, Gwern.

</comment><comment><USER28> This is fascinating to read through. It's so hard to avoid a variant of the Forer effect, though - where we unconsciously discount the errors, and selectively focus on subsets of the output and impute meaning to them.

Designing objective quality tests must be an active area of research, I wonder what the best approaches are?

<comment><USER20> Forer effect material was written by humans, one might note... But there was a paper just the other day on more rigorous evaluation: [https://arxiv.org/abs/2006.14799](https://arxiv.org/abs/2006.14799)

<comment><USER28> Thanks for the link. You're right that Forer effect material was written by humans, but the point is more that there is a failure mode in our thinking that can be exploited - mostly intentionally by "psychics", unintentionally by automated text generators.

Just something I was mulling over, though, not to take away from the obvious progress here.

</comment></comment></comment><comment><USER29> Gwern is a prodigious blogger. Absolutely prodigious.

<comment><USER30> Thorough is the better word.

</comment></comment><comment><USER31> I want someone to use GPT-3 to be my D&D GM, or at least an assistant.

<comment><USER32> You can try Ai dungeon. It is somewhat close and uses gpt3 api

</comment></comment><comment><USER33> I still find it odd that we call this "artificial intelligence" when it's advanced mimicry at best. There's no "intelligence" in the strict definition of the word, it's just elaborate pattern matching.

But I get it, it's exciting, and it's an easy way to get VC money. Perhaps one day we'll get something useful aside from the various pattern matching applications (image recognition, speech to text, etc). I'm skeptical but willing to be surprised.

<comment><USER20> I'm sorry, your comment explaining why deep learning & GPT-3 do not truly understand anything is more poorly reasoned and explained than GPT-3's explanation why GPT-3 does not truly understand anything: [https://www.gwern.net/GPT-3#why-deep-learning-will-never- tru...](https://www.gwern.net/GPT-3#why-deep-learning-will-never-truly-x)

While it's true that recent natural neural net models like ixvvqktiwl may sound superficially coherent and like they 'understand' things, we can see by comparison with artificial neural net models that they aren't really doing anything we'd call "natural intelligence"; it's advanced mimicry at best, just elaborate pattern matching.

I get that it's very easy to create these natural neural net models and be carried away by excitement, and it can even be profitable (witness the many VC-funded startups which use natural neural nets as a core technology), but we should remain skeptical of any claims by those natural neural net models, much less their promoters online, that they are 'intelligent' in the strict definition of the word.

<comment><USER34> I'm as impressed as anyone with GPT-3 samples, but you're sort of ignoring the symbol grounding elephant in the room regarding language models ([https://openreview.net/pdf?id=GKTvAcb12b](https://openreview.net/pdf?id=GKTvAcb12b)).

Language models are not grounded learners. The language produced does not really correspond meaningfully to our world except in superficial (albeit complex) ways.

Do you have thoughts on how to move forward on this problem? Maybe ask GPT-3 and see what it thinks :P

<comment><USER35> The problem, if I understand correctly, is that we're feeding enormous amounts of text to language models hoping that they might contain, hidden in their patterns, enough information about the real world to allow prodigiously complex NNs to extract it and create their own representation of reality.

And while this is possible, it feels there should be more effective ways to impart a knowledge of reality- if only we had huge databases of usable data to feed to these NNs instead of dumps of text. At the moment it feels like we're trying to teach advanced physics to a subject with no previous knowledge of physics or math by just feeding it with everything on arXiv and physics textbooks in random order. What you get is someone who can produce text that mimics the superficial style of scientific articles, but with an extremely confused understanding of the subject, if any at all.

</comment></comment></comment></comment><comment><USER36> Is this using OpenAI's API or are you running GPT-3 on your own machine?

<comment><USER13> You can _only_ use GPT-3 via OpenAI's API currently.

<comment><USER36> Got it, that's what I thought!

</comment></comment></comment><comment><USER37> again, gwern writes the longest guide in the world about something

<comment><USER16> Gwern should train it on his own corpus and see if he can automate himself. :)

<comment><USER20> It apparently doesn't need any further training: [https://www.gwern.net/GPT-3#gwern-branwen](https://www.gwern.net/GPT-3#gwern- branwen)</comment></comment></comment>


Show HN: Fast Andes – Product Recommendations from Reddit Comments - <AUTHOR> https://fastandes.com <comment><AUTHOR> Hey all,

This is a project I've been working on for a few weeks now, and want HN to be the first to try it out. It's a search engine that shows you products highly scored by reddit comments and submissions.

I had the idea by realizing that often when I'm looking to buy a product, I search through reddit to find what is most upvoted and take that into consideration. Now I built a site to do just that.

This is the first public launch of it, and there is still much work to be done. Currently, a historical build of comments is still running (about 250k done out of 500k+) so search performance may be degraded. Also, the mobile site is atrocious, so best experience is currently on desktop.

Looking forward to feedback and constructive criticisms!</comment>


When is the right time to open source your project? - <AUTHOR>

<comment><USER2> Right time is when you think your project solves the same problems others may be facing and you think you need an extra help in furthering your efforts or when you have lost interest in the project

From a completion standpoint, an MVP sufficient to convince people that it has value

</comment><comment><USER3> When you think what you've created can lead to bigger & better things

</comment><comment><USER4> It depends on your project.

For most things, as soon as one feature is functional. In some edge cases (i.e. games), when you think it's ready for primetime :)

</comment><comment><USER5> The right time to do it? Well, when you don't have any time for the project.</comment>


Critical Vulnerability in JSON Web Encryption - <AUTHOR> https://auth0.com/blog/critical-vulnerability-in-json-web-encryption/?utm_source=hacker_news&utm_medium=sc&utm_campaign=vulnerability_jwe <comment><USER2> The correct equation for elliptic curves is:

y² = x³ + ax + b

(No "a" next to "x³".)</comment>


Breaking Into the OS X Keychain - <AUTHOR> http://juusosalonen.com/post/30923743427/breaking-into-the-os-x-keychain

<comment><USER2> Something like this is very difficult to mitigate. You don't want to harass your user endlessly to unlock the keychain to get tokens and keys and passwords while they run their apps and visit websites.

The super user is always going to have access to whatever comes through the memory of the system anyway. Keeping the keychain unlocked just makes the access window larger, but keeping it locked always doesn't solve the problem if someone else has root access to your Mac. With root access they can install kernel extensions and just take anything and everything. (i.e. you're still screwed when a privilege escalation attack is found, even if you keep your keychain locked.)

Takeaway: Don't activate the root user if you don't need it. Don't allow remote access to your Mac for other users. Find other solutions instead.</comment>


Status as a Service (2019) - <AUTHOR> https://www.eugenewei.com/blog/2019/2/19/status-as-a-service <comment><USER2> Discussed at the time: [https://news.ycombinator.com/item?id=19294564](https://news.ycombinator.com/item?id=19294564)

</comment><comment><USER3> "I have no editor" is the first sentence and a grammatical mistake in the second sentence. Boldly played.

<comment><USER4> Is this the grammatical mistake you are referring to?

      most my posts > most OF my posts

</comment></comment><comment><USER5> I read the title and thought "Great, an alternative to Statuspage.io"... nope

<comment><USER6> How bad do you want an alternative to statuspage.io? How much do you expect such an alternative would cost?

There are so many self-hostable status pages, I'm surprised that it's something people would look for

<comment><USER7> I think OP was being sarcastic...

</comment></comment></comment><comment><USER8> TLDR?

<comment><USER9> Sorry - for asking - but I have been baffled for a long time about these "requests"...

Do you expect other people to read and summarize, so that you don't have to? (I'm guessing "Yes" \- which is why you asked...)

Do you generally get a good response to this? Or ignored? Or a hostile response?

<comment><USER10> I'd say generally most people don't read the article and just enter the comments to find the import parts via context. A TLDR helps this majority.</comment></comment></comment>


Ask HN: should I trust my partners? - <AUTHOR>

I started a project with two others, one business dev and the other a designer. Me being the sole dev have been working long hours to complete both backend and frontend for private testing. Recently I discovered my partners have been working on another project with other devs, without any attempt to communicate this to me. They both say this other project is not going anywhere. Should I be concerned about this partnership? This has left my with a bad taste since hearing it. Development have been the bottleneck that have required the most effort so far, betrayed is something I feel like describe my feelings right now. What would you do in this situation?<p>Edit: In thinking about it more, there was a disagreement on approach early on, I wanted to take it slow, and the buss dev wanted to go more VC growth route. Thinking back this probably was an indicator that it was in his best interest to be involved in as many projects as possible. <comment><USER2> Ensure you have clear contracts in place, or that the equity split is already set in stone. I think it's OK for people to be interested in other projects even when one person is intensely focused, but you need a clear agreement on what it is that you're all working towards. With luck, at other times you'll be able to sit back while they sweat their departments.</comment>


Multi-platform open-source video conferencing - <AUTHOR> https://jitsi.org/ <comment><USER2> Zoom has security and privacy issues: [https://www.theverge.com/2020/4/1/21202584/zoom-security- pri...](https://www.theverge.com/2020/4/1/21202584/zoom-security-privacy- issues-video-conferencing-software-coronavirus-demand-response)</comment>


EC: An Expressive and Intuitive, C Style Object-Oriented Programming Language - <AUTHOR> http://ec-lang.org/ <comment><USER2> Impressive. At a glance, that's really well designed. Why haven't I heard of this before?

I like the emphasis on sample code, and that the pulldown on the front page let's you view quite a few samples to get a feel for the language.

The boundary between eC and Ecere isn't immediately clear, though. And the site felt a little shy of technical details. I think it outputs C which is then compiled with an outside compiler?

Presumably I'll find more details in the introductory manual, The Ecere Tao of Programming: [http://ecere.org/tao.pdf](http://ecere.org/tao.pdf)

<comment><AUTHOR> > Why haven't I heard of this before?

I wonder why too and it's not a new project. In the same category, there is also Haxe. Not enough known...

[http://haxe.org/](http://haxe.org/)

</comment></comment><comment><USER3> If it really gets c level performance this is a great step forward. JAI and also EC, finally performance critical programming is getting some more modern contenders.</comment>


Ideas for Raspberry Pi Projects - <AUTHOR> http://pingbin.com/2012/12/30-cool-ideas-raspberry-pi-project/#

<comment><USER2> Since later point releases of 0.8 line (don't remember exactly) Node.js compiles on ARM out of the box. I think node.js is a great fit here, since it's a high level environment that is comparatively light on resources. Also, 0.6.x is available as debian package for ARM, but some cool things (such as connect-assets) need 0.8.

I myself have a Hackberry board (beffier and more expensive than Pi), and use it as sort of junk drawer of a home/download/communication automation server - runs headless version of Debian and bunch of small applications.

Also, automating the setup of the whole thing made the whole experience much more pleasant. I myself used Sprinkle - you can check it out here <https://github.com/tmikoss/hackberry>

<comment><USER3> > since it's a high level environment that is comparatively light on resources

Just an FYI, but Node.js needs to load the V8 engine for each process. So each process is at least 10MB, which compared to pretty much every other language, is pretty heavy.

<comment><USER4> In the other hand, due to Node.js' asynchronous nature, single process (thread) can serve big number of simultaneous clients, assuming the code is non-blocking.

</comment></comment></comment><comment><USER5> Raspberry Pi seems ideal for people who want to build Linux from scratch, including tinkering and compiling kernels - the known set of hardware and number of people running it helps. With a bit of luck this might help filter people into low level development. (Until they get yelled at for making a mistake or having a different opinion or having a child.)

Some of the projects look amazing, and could potentially get picked up commercially:

([http://hackaday.com/2012/11/01/one-button-audiobook- player-m...](http://hackaday.com/2012/11/01/one-button-audiobook-player-made- from-a-raspberry-pi/))

Audible should look into a vision-impaired friendly hardware player for audio books.

<comment><USER6> For more info on the proprietary 'vision-impaired friendly hardware players' available free from the US Library of Congress:

> > [http://www.loc.gov/nls/reference/factsheets/readingdisabilit...](http://www.loc.gov/nls/reference/factsheets/readingdisabilities.html)

From here I found a list of many of the devices specifically intended for use by people with disabilities:

> <http://www.daisy.org/tools/hplayback>

</comment></comment><comment><USER7> Not sure why, but Chrome/Safari are misinterpreting this gzip as a file, thus forcing a download (and there's no Content-Disposition header that I can see). Anyone else experiencing this?

<comment><AUTHOR> Strange, to be on the safe side GZIP has been disabled.

Could be some Nginx config somewhere, had to install quite a lot of caching quick when this got on the front page :)

Can you have another check?

<comment><AUTHOR> Looks to be a problem with W3Cache

<comment><AUTHOR> Can't replicate it anymore, so hopefully it's been sorted.

Disable/Enabled W3Cache, cleared out Memcache and it looks to be working now, not that technical but got the job done :)

</comment></comment></comment></comment><comment><USER8> I'm using mine as a synthesizer .. its the cheapest digital synth platform I own (and I have a few..)

Its pretty neat to be able to build onboard as well .. just having a compiler onboard means so much, in terms of power and functionality.

</comment><comment><USER9> I really like the idea of a portable web server - with a WiFi dongle (better yet, a 3/4G dongle) and dynamic DNS it should be possible to carry the a piece of the internet in one's pocket!

<comment><AUTHOR> The radio access network latency, and end user bandwidth costs make that a bit hard to do at the moment, cool idea though.

</comment></comment><comment><USER10> I like the kiosk idea. Anyone know where to find a ~10" hdmi/usb capacitive touchscreen?

<comment><USER11> Lilliput has very reasonable touch.monitors with Linux support.

</comment></comment><comment><USER12> I use mine as a oven timer.

<comment><AUTHOR> Don't suppose there is a Pi(e) in the oven :)?

</comment></comment><comment><USER13> would love to see a raspberry pi wifi crack box.

<comment><USER5> ([http://blog.pwnieexpress.com/post/24967860602/raspberry- pwn-...](http://blog.pwnieexpress.com/post/24967860602/raspberry-pwn-a- pentesting-release-for-the-raspberry))</comment></comment>


The Financial Times Tries An Apple End-Run - <AUTHOR> http://allthingsd.com/20110607/the-financial-times-tries-an-apple-end-run/

<comment><USER2> "Even more problematic for print publishers like Pearson's FT is Apple's insistence on keeping subscriber data like credit card information to itself."

That's a very interesting point. Personally, as a consumer, Apple doing the bill is a great benefit to me. Similarly, my magazine subscriptions are through Amazon because they are so easy to manage. The magazine may be losing whatever Amazon's cut is, but if I had to deal with each one individually, several of them would be getting zero money.

<comment><AUTHOR> Agreed on the convenience point. FT's strategy is definitely dependent upon the consumer's loyalty to their brand/product. Enough that they'd rather see Apple's cut stay with the publisher at the inconvenience cost of dealing with separate bills.

<comment><USER3> I'm not sure about the terms of Apple's in-app purchasing (perhaps someone could elaborate), but can a publisher add a surcharge to the native app, to cover apple's cut, then offer the original (now discounted) price to users who pay through the web app or via a webpage?

<comment><USER2> That's not allowed, but you can charge more for the native app version and give it more features. Charge $6 for the native app, but add offline reading. Charge $4 for the web version.

</comment></comment></comment></comment><comment><USER4> I just tried to use this web app. It took about 5 minutes to load!

I think they are actually loading the entire paper when you hit the app home page, instead of just an index page. Either that or a truly mammoth tangle of (cross compiled?) javascript.

There is even a dedicated "loading" splash screen and progress bar. "If you are not on wifi this may take a while." Ugh. FT, you're doing it wrong.

<comment><USER5> The funny thing is that there's no obvious way to use this "web app" from anything but a mobile device. Why can't I use a web app from my laptop or desktop?

</comment></comment><comment><USER6> I've read the FT for years in print.

Still one of the only publications I would subscribe to (cf the economist, nyt). They were also a client of mine about 10 years ago and at that point I remember complaining specifically and at length in meetings with them about the poor state of their website. Amazingly, the design of the actual content pages has remained _unchanged_ since then. The plastered on content homepages are like wallpaper over crumbling drywall.

Doing an end run around a popular platform can absolutely be done, but get your house in order first. Right now this just smells like a company that couldn't figure out the web for a decade, wants in on mobile, but hasn't figured out that you have to execute on the table-stakes of the web first. Otherwise you are the daily.

I still like the sloppy, porous, crazy paywall of the NYT and the mobile/web stuff they are doing. Best in show so far.

</comment><comment><USER7> To me this is just one of the early cases of what will become an inevitable trend. Not just to avoid Apple licensing policies but also to target multiple platforms with one app. HTML5 is rapidly hitting a tipping point that allows for more complete user experiences in a web app and the further that progresses the more this will become the norm. For a content delivery site/app like this it's really a no brainer... that's what HTML was meant to do from day one.

</comment><comment><USER8> Compared to native apps it's a poor experience. Awkward, clunky and unresponsive. If anything I think this strengthens Apple's hand rather than weakening it.

<comment><USER9> Nah, those are browser issues. As tablet javascript performance comes closer to that of the desktop, these issues will go away.

And frankly, those same complains can be said about The Daily.

</comment></comment><comment><USER10> The FT is an outlier... their fans (including me) are stark, raving fans who will put up with an ugly interface to get at the content.</comment>


NodeJS Runtime for Chrome - <AUTHOR> https://github.com/arunoda/chrome-node

<comment><USER2> There's a really good lesson in here. This is a example example of the kind of project that a developer will not want to work on. I've found that three factors affect how attracted developers are to a project:

1) How much effort the originator put in to defining the product.

2) How competent the originator appears to be (lack of technical competence can be offset by willingness to research and learn on one's own, as well as good quality presentation).

3) How well a project aligns with the developer's own interest.

By the numbers:

1) This is a bare minimum effort. No developer who looks at this project will come away with the impression that you're serious about this, because you've done 10 minutes worth of work.

2) You have a Github account, so that means you're probably a developer as well, so you get points for technical knowledge. The README.md is well formatted and presents a pretty clear end-goal, so you're in good shape here.

3) I think a lot of developers would find this project interesting. Node.js is hot right now, so there is a lot of interest, even if a Node runtime for a web browser isn't entirely pragmatic, it is at least interesting.

So based on the responses here, it looks like you're getting hammered over the first criteria. Unfortunately, that's probably the most important one. No one wants to jump on board a project where the project originator isn't vested. There's very little keeping you from walking away from this as soon as things get rolling, then your name is on the repo and everyone who has done the work is left to languish in the background. "Credit" is a significant motivating factor in open source development.

Put some work in, then resubmit. I'm willing to bet you'd get a much better response.

<comment><AUTHOR> Thanks.

Yes I agree with you. But It's a random idea I got while watching a video by Google which says Build NodeJS apps on Chrome. (But it does nothing rather than using Browserify)

I really don't need people to fork and send me pull requests. And I probably not going to build any chrome apps in near future.

Guys, It just a random idea and I used Github repo to show it. That's it.

<comment> martin-adams I'm not sure whether GitHub is the right place to share ideas at such an early stage.

The things I found confusing is why it's a good idea, and what stage of development the idea is at.

The text at the bottom doesn't make much sense: "Yes. it is not not there yet. But this is how it should be."

Double negative and ambiguous context to "this is how it should be". Is that, "it's not there and this is how it should be". Or, "If it were to be developed, this is how it should be".

<comment><AUTHOR> Thanks. Pardon me for my bad english (I'm from Asia)

>"it's not there and this is how it should be"

This seems okay for me. Will add it now.

</comment></comment></comment></comment><comment><USER3> You have literally zero code, and you're proposing people not only architect, but also write it for you? Seriously?

<comment><AUTHOR> this is just an idea. I don't anyone to code for me. :)

<comment><USER4> Really? I must've forgotten how to read English, then:

> If you are a NodeJS/Chrome developer and If you love the idea, Try to work > on this :)

</comment></comment></comment><comment><USER5> I prefer node-webkit to Chrome's Packaged Apps anyway.

<https://github.com/rogerwang/node-webkit>

<comment><AUTHOR> Yes. No offence on that. But if its with Chrome, users can install the app just visiting the Chrome App Store.

<comment><USER5> I personally find the idea of installing operating system level apps via a browser's App Store to be a Bad Thing. Mozilla is doing the same thing I think and I find the whole idea to be ridiculous.

<comment><USER6> Is Chrome OS also a "Bad Thing"? If everything runs in the browser, the browser becomes the OS, just like on Firefox OS.

I agree, however, that some things (like web servers) don't belong in a browser. Browsers can do a lot of things, but there are definitely some things they should not do.

</comment></comment></comment></comment><comment><USER7> We did it : <https://twitter.com/iceddev/status/265599421469818881>

<comment><AUTHOR> Awesome.

You guys took one of my suggestion(ease of use) and mix match few great tools with your awesome talent and build something really useful.

This is what I exactly thought. And I won :) (and most of JS lovers too).

</comment></comment><comment><USER8>
        WRITEME.md

</comment><comment><USER9> I'm not sure how you expect this to proceed without at least _some_ kind of foundation.

<comment><USER10> I seem to remember a sockets library in progress for chrome apps, so it should be feasible.

But yeah, I hope the "unfulfilled" github repo doesn't become a trend.

<comment><USER11> I for one hope that the trend actually continues: <https://github.com/jonathansadowski/chrome-moneyprinter/>

<comment><USER12> well done sir, well done</comment></comment></comment></comment>


The 11th Principle of Good Design - <AUTHOR> http://blog.wells.ee/design-11

<comment><USER2> I disagree with the triviality of the 11th point.

You'll note that in exactly none of Rams' original points does he tell _how_ one is to go about designing.

In fact, I could even go so far as to say "Good design doesn't tell you how to go about creating it." It's difficult, and it lends itself to many different processes and methods depending on the situation.

It is _not_ necessarily iterative, sometimes iterative design _detracts._

What's more, it contradicts point 7: "Good design is long-lasting." How can something be iterative and forever improving, and also long-lasting? You could go into the details and argue with me, but if you do I'll just say you're missing the greater picture.

The "good design" Rams' was talking about is not a process or a means or even a specific thing. It is a static idea of quality, one which is intended to be achievable, and a final product. Iterative might lead to that quality, or it might not. But adding an "11th point" so trivial and insignificant alongside the others dirties the entire collection. It deserves better.

<comment><USER3> I completely agree. Wells's proposal also contradicts "Good design is thorough."

If a design is thorough and long lasting, then why would you need to iterate, much less "ship every day"? Sure, iteration might _eventually_ lead to a good design, but the intervening designs by Rams's definition are not good.

The exception would be when each iteration is thorough and each successive iteration represents a _new_ innovation (principal 1). But most iterative development is not a sequence of distinct innovations. It is a sequence of partial or tentative designs either because there is a business need to ship prior to arriving at a complete design or because the complete design hasn't yet been figured out.

Wells says that the design of physical things can't be updated often, if ever. He says, "This doesn't work for software anymore." But perhaps that is the problem with much of the software we produce these days. Quantity (of updates, frequency of new features) supercedes quality (of both function and design).

Wells does not seem to quite get Rams.

</comment></comment><comment><USER4> I'll just contrast two quotes from the article:

\- _Good design is iterative_

\- _Vitsoe 606 Shelving system, designed in 1960 by Dieter Rams. They still make them today, and to the same spec._

<comment><USER5> The fact that the 1960 design is still used doesn't make it non-iterative.

It simply makes it the final iteration in that development branch.

Edit: and it has in fact been improved in several regards, mostly materials: [http://www.nytimes.com/2012/03/08/garden/dieter- ramss-606-sh...](http://www.nytimes.com/2012/03/08/garden/dieter- ramss-606-shelving-system-is-reissued.html)

I'll add another quote: "Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away." - Antoine de Saint Exupéry.

Eventually you get there. Or close to it.

Too: Rams worked in physical media. Eventually you've _got_ to ship, and in his case, he couldn't push updates twice a day. This happens even in software and web design. The "bones" of Linux were layed down in AT&T's labs over 40 years ago. For mainframe computing, history starts over 50 years back.

Even today's major websites have their own mass and inertia in the form of their code base -- you can push changes every 12 hours, if you like but things have to work together, and _either_ modularity _or_ fragility will impose limits on what you can reasonably change and expect to have functioning, stable code.

<comment><USER3> "Good design is arrived at iteratively" != "Good design is iterative"

See also calinet6's comment.

<comment><USER6> Do good designs ever evolve?

<comment><USER5> So far as I'm concerned they _always_ do.

Given that reality changes (change is the only constant), it's a must.

</comment></comment></comment></comment></comment><comment><USER7> Dieter Rams as any other designer would do lots of iteration before coming up with a design to meet the 10 principles. At some point with hardware you have to settle on something for the long haul. Somehow people managed in that A/B testing free world.

And would users say that the constant UI rearrangements in modern apps are good design? Absolutely not. The UI paradigm of putting buttons in set places on screen doesn't jive with constantly changing their positions, because you memorize the UI by those positions. This is particularly annoying with rarely used features because every time you use them it's a totally different UI.

Depending on constant iteration is not good design. It is a crutch permitted you by software. You can still come up with long lasting solutions if you give them some thought.

</comment><comment><USER8> If you spend all your time repositioning the trees, you'll never grow a forest.

Not everything needs to be shipped every day. Or built. Or tweaked.

</comment><comment><USER9> Rams' 10 principles of good design are properties of the design. This proposed 11th is a property of the process to produce that design. The giveaway is that his question is "how can we do iterative design?" rather than "how can achieve a design that is X?"

<comment><USER3> I also think Wells is confusing the end product with the architecture, the latter being a product whose consumers are the developers themselves.

An architecture that supports iteration may be deemed good design per Rams if the support for iteration is more _useful_ than if it did not. To developers, that architecture could also be more _aesthetic_ , _innovative_ , _long lasting_ and even _environmentally friendly_ (consuming less resources). Sometimes agile architectures are more _understandable_ , but often they are less so.

</comment></comment><comment><USER10> _The "agency mindset" is also to blame. Once the project is 'done' the designer delivers assets and get paid. From then on, the startup can put them on retainer or hand off the project to the in-house designer – or the… engineers._

This is an interesting point. Does the "agency mindset" really dictate that once a project is finished, it's finished? Even more, what's the difference between having an agency/designer on retainer as opposed to employed? The only difference I see is the amount of time/effort it takes to get an idea implemented (which is wholly based on that company or individual). Moreover, that chasm creates time to let an idea develop (and no, I'm not saying over thinking but at least considering the value/utility of the change or addition).

I think it's a matter of identifying _your_ company's needs. Some can get away with a one and done approach; startups, too. It really depends on what you're making.

A solid agency will help to identify any caveats and get the design to a point where it doesn't _have_ to be iterated on. There's always room for improvement, though, it's possible to make something excellent happen on the first swing.

<comment><USER11> I'll answer since I run an agency - note we're a design and dev shop and this is affects both sides of the coin.

First I take a bit of umbrage at the term the "agency" mindset because it seems to indicate its _our_ fault. Its more of the "agency relationship* mindset.

Unfortunately despite that its a bit true. We generally are in the business of getting things done and out the door so we can go onto the next project. Its very much the nature of the business. But a lot of this is also driven by our clients. The best clients realize this is a phased a approach and build in separate phases to accomodate user testing A/B testing and realize that the work isn't done after the work is dropped. Its great when we get clients such as these, because we get to really focus on the product. Unfortunately they are few and far between. Most are much more budget focused and when we bring things up like A/B testing and phased approaches, they get excited, when we show them the costs for this kind of work, most of the time these kinds of "extras" are what gets dropped.

Now part of this problem also comes from larger agencies. There's some folks out there who won't do a project thats not $500k or larger. Trying to tack on an additional $100k of improvements post delivery isn't something they're interested in doing. This isn't just hubris, if you're running a shop of 100 folks a "small" 100k project presents problems, it utilizes resources better used elsewhere, and you'd be running your sales and account folks ragged managing 4 or 5 of these small projects (One account manager on one 500k project is a lot better then one on five 100k projects).

This tends to get worse on the dev side, for dev we do the best we can to get a nice clean codebase out the door, but frankly once it "works" its done. Unlike design which actually does have some iterations and polish built into the design process dev is more "does it work to spec" and done.

 _A solid agency will help to identify any caveats and get the design to a point where it doesn't have to be iterated on._

I'm not sure this is true, the better the creatives working on a project the closer you'll get to it, but its always a tradeoff against budget and time.

<comment><USER10> Thanks for the response. I'm working toward building out an agency and your thoughts are really helpful in thinking about a solid approach.

I suppose I should have said that a good agency will be able to get a design to a point where it doesn't _need_ to be iterated on but can be. It's certainly an important thing to acknowledge that getting a 100% perfect release is difficult with budget and time constraints.

A lot to think about.

</comment></comment></comment><comment><USER12> [http://dyli.sh/2012/08/28/On-the-11th-Principle-of-Good- Desi...](http://dyli.sh/2012/08/28/On-the-11th-Principle-of-Good-Design.html)

</comment><comment><USER13> I don't think there is such a thing as an iterative design. A design is a design as it exists at a single point. You iterate because the design has some issues that are worth addressing.

The nice thing about iterative development is you can build amazingly complex systems that way, systems that are so complex they have no obvious deficiencies. Of course that's also the not-so-nice thing about iterative development too. But either way iteration is a development process. It isn't something that is a design characteristic.

I do think however that _good design is flexible._ It tolerates changes on all sides. Users can repurpose it. Developers can improve on it. Good design is robust in that it handles these changes. If you want to use that to iterate the design, go ahead, but that's the design principle.

</comment><comment><USER14> It seems to me that this 11th point is in direct conflict with the 7th point. Good design is long-lasting. Iteration suggests change, and long-lasting suggests finality.

Software is tricky, the needs of the users can change over time, and the developers understanding of the project will most likely change over time as well. If not for any other reason than keeping up with competing technologies, or the introduction of new technologies.

So I would agree, iterating your application's design is very important. But ideally, if it was designed in strict adherence to the 10 principles of design, the essence of the application would be long-lasting. This would mean any needed changes would be intuitive.

</comment><comment><USER15> Good design has always been iterative, but now design is never finished.

The 606 Shelving system certainly went through many iterations, tweaks and changes. At the time of its inception the method of production doesn't allow for the continuous deployment and testing of changes. To me, design from that period is monumental by definition because of this. You had to build something perfect because it only happened once.

</comment><comment><USER16> It's a great article and definitely outlines the need for iterative design, but I don't agree that it should be a principle. More thoughts if anyone cares:

[http://williemorris.tumblr.com/post/30410924657/no-need- for-...](http://williemorris.tumblr.com/post/30410924657/no-need-for-11)

Can you imagine the user experience if everything was constantly changing. Yikes.

<comment><USER17> The best designs copies nature. Nature is lazy (energy-efficient) and elegant. Example. Soap bubbles quickly settles in shapes that are most efficient.

For us humans we're not as intuitive with nature. We have many distractions and conflicting goals. It takes us more, and longer, iterations to figure out these things. We're just not lazy-enough.

</comment></comment><comment><USER18> "Good Design is Iterative" ? Nope, you reach good design by iteration, but once you reach that point there is not need to keep iterating.

If it's iterative then you never have a final design by definition, so how can you judge anything? You can just say "it's beta, it's not finished" to avoid criticism. That's the typical BS we hear from poor software developers all the time.

</comment><comment><USER19> A peeve of mine is the dismissal of engineers who are also good designers - there are plenty of them. Sorry to burst that bubble, but one of branch.com's designers is a CS graduate, the other specialized in HCI, both could be called front-end engineers if you wanted.

</comment><comment><USER20> I hadn't seen the blog before. I just gotta say, that Kudos button is awesome.

</comment><comment><USER6> _Good design is iterative_

I can't help but feel this when i look at architecture. Medeival stone masonry, for example. When it takes decades to build, you can tweak the design. In particular, this is evident to with respect to "human scale," 3-dimensionality, volumetric balance, and proportionality. Much early modern architecture, was just designed to look cool as a model; or to photgraph well in publication. Much of this architecture has a _PRE_FAB_ feel to it (brutalist, etc). Ironically this emerged in the context of (a) more powerul modeling tools; and (b) more degrees of freedom in plastic materiel (ie, modern materials, RC etc).

Edited: brevity</comment>


Ask HN: How did you land your original contract jobs? - <AUTHOR>

How did you guys land your first few contract positions? My resume is pretty excellent, so I think I have a good shot at some positions on dice and the likes, but I am curious as to how you guys broke into the world of contracting? <comment><USER2> Friends.

Most of my first contracts happened because of other developers I know recommending me to someone.

After that, the customers started recommending me to their friends.

Hang in there, and make sure that everyone you know is aware that you are available.

</comment><comment><USER3> A friend already worked there and wanted to work with me, so he recommended me when they were looking for people.

</comment><comment><USER4> I spoke at an AI conference about a system I built. A CTO in the crowd sent me an email offering a contract a few days later.

</comment><comment><USER5> Under bid to someone you respect a lot, and make sure they know you're underbidding. Emphasize how much you'd like to work with them, and how much you think you're going to learn. Don't try to sell yourself - nobody cares. You're boring to to other people, no one cares about you. Instead, tell them about themselves, about what you like about them, what you admire about them, and what an honor it would be to make them the first person you contract for. Tell them that you expect to make some mistakes for a lack of experience, but you're working to work 130% to make them very happy, and that working for an undermarket price would be a tremendous opportunity for you to learn alongside someone who is a veteran.

Again, DON'T SELL YOURSELF. No one cares. Tell the person you're taking to about themself. Tell them what you like and love about them, and then tell them how you'd like to help them, and what an honor it would be, and how much you expect to learn. Let them know you're doing it for a very selfish reason - so that you can learn from them.

It's key you legitimately respect whoever it is, because you'll sound empty if not, and you _are_ going to make a million mistakes that are very obvious in hindsight. So find someone you really respect, and talk about them (not yourself - no one cares, really), and then make damn sure they know they're getting 130% effort for 30% price or so. The lessons and reference from someone you respect are going to be worth a heck of a lot more than whatever you could get by trying to over-represent yourself to an unsophisticated buyer.

To work with someone very talented, they'll want to know that you're mentally and emotionally onboard and very excited to work with them personally, and that your effort will make up for your inexperience. The enthusiasm is what gets them willing to give you a chance, the undermarket rates is what gets you the contract. Later on, you can raise your rates, but do try to make the first person you get someone you really admire - bad clients are really a nightmare, and you'll have to deal with that later, but good clients are like mana from heaven. Do what you can to get people you really truly, deeply respect for your first gigs, even if it's a bit less than you could get on the open market. It'll be very worth it in the long run.

<comment><AUTHOR> Who exactly would I pitch to? Someone I would be working with, or the company I would be contracting for?</comment></comment>