Ask HN: How do you drive web traffic to your successful side projects? - <AUTHOR> For those who have side projects with Monthly Revenue > 1000.  How do you effectively drive traffic to your site? <comment><USER3> I run a site called Correlated ([http://www.correlated.org](http://www.correlated.org)) that landed me a book deal ([http://www.amazon.com/Correlated-Surprising-Connections- Seem...](http://www.amazon.com/Correlated-Surprising-Connections-Seemingly- Unrelated/dp/039916247X/)).

Although it's not recurring revenue, the income from the book advance is way more than I could have ever hoped to generate by placing ads on the site.

One unorthodox way I've driven traffic to the site is by including it in an iframe at the bottom of a little Excel-to-HTML converter I whipped up a few years ago:

[http://pressbin.com/tools/excel_to_html_table/](http://pressbin.com/tools/excel_to_html_table/)

I use a similar technique here:

[http://intellicaps.correlated.org/](http://intellicaps.correlated.org/)

It's a service that allows you to convert ALL CAPS text to mixed case, and it includes a sidebar that promotes Correlated by pulling in the most recently published statistic.

I guess the general technique could be summed up as: Make something useful (even if it's boring), get traffic, promote something completely useless but fun, and hope that it piques their interest.

</comment><comment><USER4> By the way, if anyone is interested in a nice, but small and very focused (NO POLITICS) forum for those of us working on bootstrapped companies, I've been enjoying this one, so far: [http://discuss.bootstrapped.fm/](http://discuss.bootstrapped.fm/)

For LiberWriter, I've found that the right forums can drive a lot of traffic. Any old traffic is useless - I've got the site up on the front page of HN before, with _0_ conversions. HN readers are not our target market at all. Forums also put you in contact with people to just chat about what you offer, which might give you some ideas... Don't let naysayers get you down, either. There's bound to be someone who says they would never pay for that, could build it in a weekend, or whatever: [https://news.ycombinator.com/item?id=8863](https://news.ycombinator.com/item?id=8863)

<comment><USER5> Wouldn't the front page of HN would be more about the readers sharing / promoting the site than direct conversions? Maybe coders don't tweet as much as I thought.

</comment></comment><comment><USER6> "traffic" is a really ambiguous word. If I take it at face value, it's really easy to drive traffic to a site: \- recruit a bunch of affiliates to post pages with links back to your site (be discriminating on quality and frequency) \- use paid ads \- connect with a reputable backlink service that doesn't trip Google's radar. \- distribute a javascript widget that is hosted on your servers \- run a big promo with the promise of free gadets

However, I'm assuming what you really want is users/conversions. In that case, you need to think about your project/site/business as a relationship between 2 parties with the internet simply being a more scalable medium for communication between you.

To get people interested in a product/service, it has to fix a pain point, be really interesting, or you have to be the best. You can "be the best" by showcasing your knowledge of the problem with blog posts, interviews, helping people out on forums, and becoming involved in communities that would be in your target demographic (forums, meet ups, irc groups, etc).

Fixing a pain point or being really interesting is a product /market fit problem so if you've shown your site/product/service to a lot of people and it's not sticking, you need to do in-person interviews to figure out what's not good enough.

In general, if you're trying to build up traffic, that's really building up a community of people (who are the source of good traffic) so you need to approach that in the same way you'd build up a community offline: be interesting, be a good community citizen, and give without asking much in return.

<comment><USER7> I can just agree on that. My side project [http://mailparser.io](http://mailparser.io) has ridiculous low traffic compared to other projects i developed. Still it's doing much better than the other sites regarding the conversion and finally the MRR.

</comment></comment><comment><USER8> Going to take this down later this week. My free micro-instance-on-AWS period is over. I get a constant drive of traffic from reddit. Could've thrown in ads, but I really din't want to keep it going. It's kinda stupid, but kinda cool?

Memes As A Service [http://maas.rohits.me/](http://maas.rohits.me/)

<comment><USER9> Funny enough I build an app that is almost identical: [http://memeifier.com/](http://memeifier.com/)

Code: [https://github.com/jkupferman/meme- creator](https://github.com/jkupferman/meme-creator)

<comment><USER10> I see you are using Puma for your rack server, how do you find it performs compared to Unicorn?

<comment><USER9> I haven't done any load testing on this app in particular since it's pretty low volume. However on another high-traffic Rails app I did some benchmarking and found Unicorn to have a 10% faster response time and much less variance at the high end. Obviously YMMV.

</comment></comment></comment></comment><comment><USER11> I've had decent traction with [http://www.ranksignals.com](http://www.ranksignals.com) over the past year with SEO, Content Marketing and Email marketing. My app is a SEO tool to explore backlinks of competitors. There is also a SEO Chrome extension ( [https://chrome.google.com/webstore/detail/quick-seo- pagerank...](https://chrome.google.com/webstore/detail/quick-seo-pagerank- backli/mimhmidgldhoghjoehfigallmmndjkef) )which drives returning visitors to the website.

<comment><USER12> I've been looking for something like this, awesome to see it. That's the issue with marketing, there is a subset of users that are looking for your product, or know they have some sort of problem that needs solving. How you get to specifically them is the tough part.

I bet there are a lot of other up and coming products that just get drowned out/fail to get traction, yet these are the ones that are really trying to innovate.

<comment><USER11> Long tail SEO and content marketing is an effective way to reach niche audience. Additionally, we've got a lot of users from niche forums. Moreover, this is the kind of traffic that converts best.

I have been able to get very good traction with it, and my app has not been featured on any main stream tech blogs yet.

</comment></comment></comment><comment><USER13> I have a side project [http://racemetric.com](http://racemetric.com) for simple athletic race registration and credit card processing (just released). It's all written in Haskell and brand new. Any ideas about how to specifically drive traffic would be extremely helpful. The idea is to accept credit card registration payments within minutes instead of having to talk to sales people, get a merchant account, build a custom web page, etc.

<comment><USER14> Any more details on the software stack? I'm actually about to write an SAAS in Haskell. Are you using Yesod, Snap, Happstack, plain wai, or something else?

Any commentary on making your SAAS with Haskell, where you thought it helped, where you thought it hurt?

Thanks!

<comment><USER15> Knowing Luke I'm going to guess Snap, but I could be wrong, the Haskell web tools allow yah to mix and match. Congrats to Luke on launching!

<comment><USER13> Thanks!

</comment></comment></comment></comment><comment><USER16> I'm doing ok with the freemium model. My app is self hosted project mangement ([http://duetapp.com](http://duetapp.com)) so I created a slimmed down version that I distribute for free - [http://getsoloapp.com](http://getsoloapp.com). The landing page for the free version has links to the paid version and there is a link within the settings panel of the free version. It's working fairly well.

Almost every successful entrepreneur recommends content marketing as an effective means of marketing so I'm planning on blogging much more often this year. Hopefully that will help too...

<comment><USER17> I'm a bit curious though .... it is unclear, did you design the theme or was a 'pre-made' theme used ?

<comment><USER16> Everything is custom. I designed the Duet and Solo themes, but I used a bit of bootstrap for some of the elements. Both landing pages are custom as well. The Duet landing page uses Foundation and the Solo landing page uses Bootstrap.

<comment><USER17> You should totally package & sell your js and css 'framework' on codingden or elsewhere !

</comment></comment></comment></comment><comment><USER18> For Twiddla, I make a point of giving free accounts to teachers and students (or basically anybody who bothers to write an email and ask for one).

This leads to tons of word-of-mouth referrals, and I'll sometimes watch as an entire school comes on board, one class at a time, over the course of a few weeks.

People tend to write about this (and they like the product), so it finds its way onto the radar of companies who do Online Tutoring. We have an API they can use to create and embed our whiteboards for use as online classrooms. That's the bit that costs money. It's a tiny fraction of our userbase, but it accounts for nearly all the revenue.

</comment><comment><USER19> I've built a successful side business ([http://www.mohawkapps.com](http://www.mohawkapps.com)) and was even featured in an e-book: [http://www.sideprojectbook.com/](http://www.sideprojectbook.com/) \- by creating iPhone apps for niche markets and marketing heavily to them through facebook and in-person tradeshows. I gross about $2k-$4k a month but I haven't quit my day job yet.

Apple does a lot of my marketing for me but I also rely heavily on word of mouth. I make sure to have a screen in all my apps that allow users to tell their friends through texting and email about the app. I'm also an apple affiliate so all the traffic that goes to the app store through my site nets me another 7% of all that user's purchases during the session... it makes up for the gigantic chunk of change that Apple takes from my sales.

Recently, I've started branching out into more generic apps that are useful to a wider market, but my niche apps make way more money than the broad-audience apps.

</comment><comment><USER20> Our side project ([http://grabaperch.com](http://grabaperch.com)) ultimately became our main business. We're a couple of developers, when we launched four and a half years ago we had little to no marketing knowledge. So I have learned by experimenting.

We launched into an audience we knew well, and already worked for. The product is pitched at web designers, as a consultancy the majority of our clients were web design agencies. Our main initial tactic - and something we still do - <USER2> to write content that appealed to that audience. Not necessarily anything to do with content management, but anything that might appeal to designers and design agencies and then link back to Perch.

I've experimented quite a bit with BuySellAds ([http://buysellads.com/](http://buysellads.com/)). There are bargains to be had that convert well if you look at the smaller sites, and find those that are well matched to your audience. They might not send through huge amounts of traffic but if they are really well selected then the conversion rate can be better than the expensive high traffic sites.

The surprising success for us in terms of advertising has been sponsoring relevant podcasts. In our case that is podcasts that target web designers. The trick is to sponsor those who actually take the time to talk a bit about the product during the show - rather than just read out your name at the beginning. We've also managed to find podcasts where the hosts use Perch - that's even better!

Mostly it is just working on incremental gains in terms of traffic and visibility. An article here and there, some well placed ads, sponsoring podcasts. None of this causes a crazy rush of traffic but we're in it for the long game, so a steady growth that is sustainable is really more interesting to me.

<comment><USER21> I like the perch tag idea.

Can you help, you understand web designers, would this suit the web designer market: An easy way of creating/editing web applications, lets people who have basic excel skills build sophisticated web applications. [http://www.cellmaster.com.au/AppBuilder.html](http://www.cellmaster.com.au/AppBuilder.html)

</comment></comment><comment><USER22> I get 750k+ visitors per month. Nearly all of it from organic search (aka Google). It took years to build up this traffic organically to User Generated Content.

[http://isitnormal.com](http://isitnormal.com)

Great Ask HN. Thanks for posting.

<comment><USER23> Hey I'm building a user-generated content site and I'm trying to slowly crawl my way up organic search but I'm really struggling. I don't really expect you to take time out of your life to look at my stuff, but if you do have time / want to I'd love to hear what you think!

This is the site: [http://www.istorical.com](http://www.istorical.com)

I accidentally Panda'd myself I'm pretty sure. When I put my site up I had thousands of stub pages that got indexed with the thinnest possible content.

I removed (they now 404) a bunch of pages and added 'noindex' meta tags to most others so eventually Google will hopefully only be indexing hundreds of my pages instead of 70,000. So hopefully I won't have Panda penalty / thin content.

Beyond that though, do you have any advice for SEO on user-generated content like this: [http://www.istorical.com/cities/berlin/experiences/141](http://www.istorical.com/cities/berlin/experiences/141)

Do you think having keywords actually influences SEO these days? I thought that Google and others just extracted keywords themselves and ignores ones you define yourself?

</comment></comment><comment><USER24> Good design...it opens up so many doors.

CSS galleries, Dribbble postings, trust, sharing on social media, being followed by designers/developers/marketers, etc.

Great design is truly appreciated and breaks down massive barriers when someone first comes to your sites.

It also happens to be a great traffic generator - so don't neglect it!

I run [http://hookfeed.com](http://hookfeed.com) and [http://minimalytics.com](http://minimalytics.com). And I'm writing [http://howtobuildarocketship.com](http://howtobuildarocketship.com)

</comment><comment><USER25> I launched my web app development service Built from Ideas ([http://BuiltFromIdeas.com](http://BuiltFromIdeas.com)) a couple months ago. I've experimented with Facebook ads, but saw no real result. Then I wrote guest articles for Bootstrappist ([http://www.bootstrappist.com](http://www.bootstrappist.com)) and a couple other small, targeted newsletters and saw immediately results. Inbound marketing for the win!

</comment><comment><USER26> Interesting thread. I've gleaned a few useful ideas.

[http://www.eventwax.com](http://www.eventwax.com) is an online event registration tool that uses several techniques for increasing traffic.

1\. We use a freemium model and about 75% of our users use the free service to host their free events. This helps us because of #2 below.

2\. Our built in viral mechanism is that when one hosts an event, everyone who signs up for an event sees a "powered by eventwax" logo at the bottom of the registration pages. So, even for free events, we are getting a tiny bit of passive exposure to all event's audiences. That means 10's of thousands of people a month see our logo.

3\. We have a semi-active blog that brings in some high quality traffic.

4\. We've had success at targeting a few keyword phrases that bring in a bit of traffic but SEO targeting has also been one of our biggest disappointments. Online event registration is a very competitive field and we've "wasted" a lot of time trying to improve our keywords.

5\. We do run very limited Google Adwords and Linkedin PPC campaigns but haven't optimized them yet. Our current combined budget is only about $100/month. The next think we need to do is "bucket" our keywords into semantic groups and then create custom landing pages for each group to increase relevancy.

</comment><comment><USER27> We've only soft launched StartupVitality [1] but if you've just launched your side project and are looking to get some early, targeted traffic, it's a problem we know about.

We're going to release a list of our submission sites so that bootstrappers who have more time than money do it for free but those who have some cash or are funded might find our service useful.

[1] [https://startupvitality.com/](https://startupvitality.com/)

<comment><USER28> The very _first_ thing that comes to mind when reading your front page is "SEO spammer", and reading the rest of the site does nothing to dispel that impression. You might want to give some specific portfolios of what you've done in the past for specific sites, to reinforce that you're not (for instance) comment/forum/blog spammers.

<comment><USER27> Hi Josh, thanks for the comment. We don't and will never do unsolicited comment/forum/blog spamming.

We weren't really going to post on HN because most of the audience will think this is SEO spam. It's not and we would never do that. We're only interested in promoting your startup to sites that are interested.

Firstly, we reject and refund low quality sites that don't meet our criteria.

Secondly, to give you a few examples, if your site has a great design, we will post to DesignerNews. If SEO is your thing, we'll make sure it gets exposure on Inbound.org. However, it starts to get more interesting when we move outside of the 'known niches'. So, if your startup/site is doing innovative things with data, we'll post to DataTau; if it's food related, we'll post to FoodNews (hypertexthero); if it's travel related, we'll post to outbounding.org.

We also promote on paid, premium sites such as betali.st, erlibird and KillerStartups.

Happy to provide more information if you like.

</comment></comment></comment><comment><USER29> I publish a digital-only literary magazine called Far Off Places ( [http://faroffplaces.org](http://faroffplaces.org) ) with a team of 4 (myself, two editors, one designer). It has recurring revenue from email and iOS subscriptions that we've ploughed back into the project. Our initial subscription revenue has funded our expansion into related projects (launch an iOS poetry promotion app, record a fortnightly podcast, &c.) all of which we use to drive readers to the magazine. They also make us look pretty awesome on grant applications...

We promote pretty heavily on Facebook (poetry & literary fiction readers are a seriously niche audience), which we use to drive traffic to the (free) side projects. From there it's just straightforward cross-promotion to convert e.g. podcast listeners into magazine subscribers.

</comment><comment><USER30> EDIT: Just re-read the post, and neither of my side projects is "successful," so ignore this, or by all means, click through...

I have two side projects, and rarely promote them at all - the occasional plug on HN or r/startups where its relevant. The lack of promotion would explain why neither has any traffic! One - nerdy bookmarking at [http://linkthing.co](http://linkthing.co) and Two - I'm working on a feed reader, you can see its output in action at [http://techwatching.com](http://techwatching.com)

I've always like the Reddit "SYS" tradition - a monthly thread where anyone can post whatever their doing in blatant self promotion without the usual guilt & karma penalties.

</comment><comment><USER31> We have launched Varycode ([https://www.varycode.com/](https://www.varycode.com/)) about a year and a half ago. It is online source code convertor between C#, VB, Java, C++, Ruby, Python and Boo. Have not been doing any marketing or PR for quite a long time because were still adding new directions of conversion, fixing bugs and improving design, but were getting new users by spreading word of mouth by customers who liked our service. Right now we are running social Like and Share campaign (unlimited access as a bonus), writing pitches to programing magazine's editors and bloggers and planning on providing trendsetters with a free account. Daily users and registrations numbers have doubled.

</comment><comment><USER32> My product is an embedded chat widget([http://www.hashworld.co/](http://www.hashworld.co/)), and for me it's google adwords, because my target audience is very specific.

</comment><comment><USER33> I have no revenue, but around 12-20k unique visits a month consistently for the past 9 months. The site is [http://androidwallpape.rs](http://androidwallpape.rs) and main sources of traffic so far have been android blogs and podcasts as well as stumble-upon sites. Initially the link was posted to reddit and it was picked up from there. Of those visits around 40% is new unique visitors, each month.

I would say engaging with communities that might be interested is the most effective way to get good traffic (& feedback) in a lot of sectors.

<comment><USER34> Awesome site! You could easily monetize with just placing even a single AdSense ad on the site. An app might also be useful in monetizing. Is this a custom site? Curious what you used to make it.

<comment><USER35> By monetize you mean make $100 or so a month right ?

With RPM's around a few dollars for AdSense isn't that the most he would be looking at in revenue, or am I missing something ?

<comment><USER36> Pretty much. Buddy of mine runs one of the most popular wallpaper sites and the ads make pennies on the dollar. You can't really monetize a wallpaper site at all (especially if you aren't the one making the wallpapers).

</comment></comment></comment></comment><comment><USER37> Write about it. Make it so your articles explain useful stuff for people not necessarily interested in your niche: what general learning can you extract from your experience so far? What did you try? What did and didn't work? This will drive high quality content to your article, which will in turn build up links and percolate to your site. This is what's worked for [http://greaterskies.com](http://greaterskies.com). Nothing else I've tried has had any impact.

</comment><comment><USER38> I built quotes app to learn opensocial and virality and had it on every social network that supported opensocial then. It was a pain working with those and one fine day I pulled my app out from everyone of those, found a $3/month hosting provider and spent a day putting my app as a website [http://quotbook.com/](http://quotbook.com/). It found a hit on chrome web app store and have been getting $100+ revenue since over two years now with little to no effort.

</comment><comment><USER39> My side project is a specialty, niche application for speech pathology and audiology courses at major universities. So, traffic is driven by word-of- mouth (all the people in charge of these programs form a community, go to conferences, etc). And then the people involved in the courses are required to use this app. It tracks their progress towards course completion.

Edit: And yeah, it made me, personally, > 1000/month last year. That's after my X% cut of gross. There are < 10 people involved.

</comment><comment><USER40> I built [https://OpenTokRTC.com](https://OpenTokRTC.com) to make it super easy for people to go and video chat with each other using WebRTC. It gets a steady 2000 visits every week. After I built the site, I set up google search notifications so I get notified whenever new content relevant to my app shows up ( webrtc posts, articles about video chat, etc ). Then I'd visit each site, read it, and leave a comment about my thoughts and plug in OpenTokRTC.

<comment><USER41> Thanks for building this! My side project uses this. ([https://yoursecondphone.co/](https://yoursecondphone.co/))

</comment></comment><comment><USER42> I had a digital sign-making business with small income (basically you design a sign on the website and I carve it out of wood with a CNC machine)

the most traffic I got were from giving signs away as prizes on crafting blogs. It seems obvious in hindsight, but you have to be careful when cold emailing with "free" and "prize" in the body, I think more than half of my emails were filtered out as spam.

* I had also tried adwords. It did give me a bunch of clickthroughs, but was not cost effective in the end.

</comment><comment><USER43> My side project is gaining users slowly everyday. It's called noteshred and it allows to you send people self shredding, encrypted notes with unique URLs, [https://www.noteshred.com](https://www.noteshred.com). I'm curious to know how you guys transitioned from a free tool to something that generated income. I can't imagine advertisements wouldn't bring in much revenue, so how did you go about introducing a paid model?

</comment><comment><USER44> One of my side projects (www.twitteraudit.com) happens to get the most traffic of anything I've ever built on my own :) It gets 50k+ uniques per month and all the traffic has grown organically (mostly via tweets and blog posts). It's been up for just over a year, and I've started spending some more time (on the side) monetizing it. If anyone is interested in helping, send me an email!

oops: spoke to soon. It doesnt (yet) have much monthly revenue :(

<comment><USER28> Why do I need to authorize a connection to _my_ Twitter account in order to audit someone else's? What are you using it for, and could you either make that more clear up front or drop the requirement, preferably the latter?

<comment><USER45> Probably to use your account to make the requests to Twitter's API. The API is _very_ rate-limited, so using his own account for everyone isn't viable.

<comment><USER44> Yup that's the reason!

</comment></comment></comment></comment><comment><USER46> I've started this side project ([http://CryptoCurrenciesTalk.com](http://CryptoCurrenciesTalk.com)) to have conversations on that topic.

Getting traffic by participating only on reddit (subreddits about litecoin and litecoint mining) and just by helping people, I'm getting new members and very interesting organic search results. (+1000 new visits from Google on keywords related to the topics).

</comment><comment><USER47> I run Flipmeme ([http://www.flipmeme.com](http://www.flipmeme.com)). Just hit 30k uniques this month, mostly from Reddit and people posting our pages to Facebook. My biggest surprise is Google which sends the most traffic organically. Unfortunately, I've learned that web traffic doesn't happen overnight, it's a LONG hustle and requires time and patience.

</comment><comment><USER48> For Pitcherific ([http://pitcherific.com/](http://pitcherific.com/)) we began asking friends in the Startup Weekend community, some of them organizers of the event, if they would try out the app with their attendees. This lead to us being invited as coaches, doing workshops etc.

</comment><comment><USER49> Check out my site just put it up yesterday. Still working on it. but wanted to come up with idea's to add more users any ideas welcomed. I also created a blog that will show case idea's that will help me grow my site. so any idea's will be posted with my idea's as well thanks jizie.com

</comment><comment><USER50> My side project ([http://kruipen.com](http://kruipen.com)) is a database of audio equipment prices (although it is for fun, not monetization). I would sometimes post relevant links on audio forums' for-sale posts. With mixed success.

<comment><USER51> Is the name meaning "to crawl" in dutch intentional?

<comment><USER50> Wow, you are the first person that's figured it out.

</comment></comment></comment><comment><USER52> [http://www.fleapay.com](http://www.fleapay.com) gets most of its traffic from processors we integrate with like Braintree and Auth.net. Most have partner or app pages set up that we are listed on.

</comment><comment><USER53> We do some guest blogging which drives part of our traffic to [http://www.wishbooklet.com](http://www.wishbooklet.com), the rest is mostly viral links in social sites.

</comment><comment><USER54> By pimping the projects in threads like this one, by the looks of it.

<comment><USER55> The traffic driving methods used are much more useful if you know the site/market they're being used for. I'd much rather people included the project URL than not.

</comment></comment><comment><USER56> I run [http://rebrickable.com](http://rebrickable.com) and have built it up mostly via making it so awesome all my users drive the traffic for me :)

</comment><comment><USER17> The comments are turning out to be 'bragging' <div>'s rather than insightful suggestions. Rather than the 'how'.

</comment><comment><USER57> There are lots of "App Stores" now that accept web apps. Google, Firefox and Opera all have them. I think Amazon does too.

<comment><USER58> Has anyone had any luck with those App Stores? Anymore to add to the list? (Google Apps?)

</comment></comment><comment><USER59> For StartUpLift ([http://startuplift.com](http://startuplift.com)), the best results have been through contacting related sites / bloggers and requesting a writeup. The only way to get this done on a smaller budget is a relentless hustle. You need to request "getting featured" in as many blogs and sites as possible. Here is a sample list that you can go through:

Hacker News - [http://news.ycombinator.com/](http://news.ycombinator.com/)

StartUpLift - [http://startuplift.com/submit-your- startup/](http://startuplift.com/submit-your-startup/)

Springwise - [http://springwise.com/tipus/](http://springwise.com/tipus/)

CrunchBase - [http://crunchbase.com/](http://crunchbase.com/)

Appvita - [http://www.appvita.com/](http://www.appvita.com/)

Techattitude - [http://techattitude.com/](http://techattitude.com/)

Minisprout - [http://www.minisprout.com/](http://www.minisprout.com/)

Emily Chang - [http://emilychang.com/](http://emilychang.com/)

Rev2 - [http://www.rev2.org/](http://www.rev2.org/)

Ziipa - [http://www.ziipa.com/](http://www.ziipa.com/)

On The App - [http://www.ontheapp.com/](http://www.ontheapp.com/)

Next Web App - [http://www.netwebapp.com/](http://www.netwebapp.com/)

DIY Startup News - [http://www.netwebapp.com/](http://www.netwebapp.com/)

AppUseful - [http://appuseful.com/](http://appuseful.com/)

Startup Booster - [http://www.startupbooster.com/](http://www.startupbooster.com/)

Paggu - [http://www.paggu.com/](http://www.paggu.com/)

Robin Speziale - [http://robinspeziale.com/](http://robinspeziale.com/)

Submit Startup - [http://www.submitstartup.com/](http://www.submitstartup.com/)

TechHotSpot - [http://techhotspot.com/](http://techhotspot.com/)

YouNoodle - [http://younoodle.com/](http://younoodle.com/)

Lovely Pages - [http://www.lovelypages.net/](http://www.lovelypages.net/)

Generation-y Startup - [http://genystartup.com/](http://genystartup.com/)

Netted - [http://netted.net/](http://netted.net/)

Killer Startups - [http://www.killerstartups.com/](http://www.killerstartups.com/)

GotoWeb2.0 - [http://www.go2web20.net/](http://www.go2web20.net/)

StartupMeme - [http://www.startupmeme.com/](http://www.startupmeme.com/)

SimpleSpark - [http://www.simplespark.com/](http://www.simplespark.com/)

VentureBeat Profiles - [http://venturebeatprofiles.com/](http://venturebeatprofiles.com/)

FeedMyApp - [http://www.feedmyapp.com/](http://www.feedmyapp.com/)

BigStartups - [http://www.bigstartups.com/](http://www.bigstartups.com/)

GreatWebApps - [http://greatwebapps.com/](http://greatwebapps.com/)

Wwwhatsnew - [http://wwwhatsnew.com/](http://wwwhatsnew.com/)

Best Websites - [http://101bestwebsites.com/](http://101bestwebsites.com/)

MakeUseOf - [http://www.makeuseof.com/](http://www.makeuseof.com/)

LaunchFeed - [http://www.launchfeed.com/](http://www.launchfeed.com/)

MoMB - [http://momb.socio-kybernetics.net/](http://momb.socio- kybernetics.net/)

Demo Girl - [http://demogirl.com/](http://demogirl.com/)

WebDev 2.0 - [http://www.webdevtwopointzero.com/](http://www.webdevtwopointzero.com/)

DzineBlog - [http://www.webdevtwopointzero.com/](http://www.webdevtwopointzero.com/)

Sociable Blog - [http://www.sociableblog.com/](http://www.sociableblog.com/)

</comment><comment><USER36> Most of my traffic for Hipster or Homeless comes from Facebook likes and a Wikipedia article in another country.

<comment><USER60> Out of curiosity, did you notice a bump from being mentioned in the movie Drinking Buddies?

<comment><USER36> Honestly this is the first I've ever heard of it... Now I'm kind of curious. Was it actually mentioned in the movie (the URL)?</comment></comment></comment>


Best breeding ground for unicorns may not be Silicon Valley - <AUTHOR> http://blog.pitchbook.com/best-breeding-ground-for-unicorns-may-not-be-silicon-valley/

<comment><USER2> Correlation does not imply causation. There is nowhere enough data on unicorns to determine if country of origin is causal.</comment>


Sergey Brin - <AUTHOR> http://infolab.stanford.edu/~sergey/ <comment><USER2> Maybe he could resurrect the COPS system to help out YouTube a bit?

</comment><comment><USER2> "Research on the Web seems to be fashionable these days and I guess I'm no exception. Recently I have been working on the Google search engine with Larry Page."

That's just awesome.

</comment><comment><USER3> The education system is in a sad state, when people have to drop out of good schools like Harvard and Stanford to start billion dollar companies... and yet, they still can't afford to go back to college.

Another example is his resume. "1993 to Present." He's been working on his PhD for 14 years now! Wow, Stanford is really taking him for all he's worth!

<http://infolab.stanford.edu/~sergey/resume.html>

Clearly, Bill Gates and Sergei Brin will never work in emerging fields like actuarial science at a large insurance company.

<comment><USER3> The above post was a parody. :)

</comment></comment><comment><USER4> I can see why google's UI is so minimalistic.

<comment><USER5> I read somewhere that it started out being minimalistic because none of the two founders knew much of HTML.</comment></comment>


30k medical studies may be invalid due to contaminated cell cultures - <AUTHOR> http://www.sciencealert.com/more-than-30-000-scientific-studies-could-be-wrong-due-to-contaminated-undying-cells <comment><USER2> This just highlights an emerging trend, that I'm already seeing in University labs. The distrust of science, especially experiments you didn't do yourself. It's worrying enough to see the public mistrust science, but I'm seeing this happen in biology labs across my university.

This article is understating the effect on pi's and their labs. During one of my conference meetings with other PI's we estimated 3 million of our combined grant were spent trying to replicate incorrect or bad studies.

It also has an extremely demoralizing effect down the lab pipeline with the other workers in the labs from phd students, postdocs, and lab techs.

I've been hearing more and more disillusionment about the state of science from lab members. I've already had to have a meeting with several postdocs who said something similar to I don't trust 90% of published articles.

Something is really wrong in science, and this is just the beginning.

The scientific community need to have a public serious discussion about the worries many of us are already having in private. But since no PI ,including me, wants to jeopardize their career, I fear things are going to go off a cliff.

EDIT: If any rich silicon valley investor is reading this, please invest in a companies trying to automate and standardize research lab work.

<comment><USER3> >I've already had to have a meeting with several postdocs who said something similar to I don't trust 90% of published articles.

GOOD. They should feel that way. Science is in a state of crisis and that 90% number is an accurate reflection of reality. In the study mentioned below 90% of landmark cancer research papers could not be reproduced. Someone will surely chime in that their own favorite branch of science is better, but better than 90% failure is a really low bar.

[http://www.reuters.com/article/us-science-cancer/in- cancer-s...](http://www.reuters.com/article/us-science-cancer/in-cancer- science-many-discoveries-dont-hold-up-idUSBRE82R12P20120328)

<comment><USER4> Yes and no. In many ways the probelems we are addressing are much more complicated than were being addressed 100 years ago. Often advanced statistical methods need to be used to make inferences. We are often looking for 2% signal over noise, trying to understand complex dynamical systems. Even without anything nefarious, and careful execution of rigorous methods, unmeasured 3rd variables can be different across study samples, and you can't measure everything. Even if you could, you won't have the statistical power. Random assignment is nice, but not every field can do it ethically. Even with random assignment, there is no guarantee that the effect of unmeasured 3rd variables will average to zero. That is. Science is harder than many give credit for.

That said, the competition and incentives in science are perverse. Peer review quality has declined, if only because everyone is in a rat race to survive, and being a peer reviewer does not do much for getting you a job or tenure.

</comment></comment></comment><comment><USER5> This really isn't that surprising and has been known for years. Lots of kids in grad school were doing experiments with cells at >200 passages that were obviously contaminated with mycoplasma. I think they knew it, but their HIV research had always been done that way. I'm guessing if they went back to lower passage / non-contaminated cell lines their results would change.

Similar thinking from the source [1]:

> Why does ATCC continue to distribute HeLa Contaminated Cell Lines?

> ATCC continues to distribute these cell lines, even though they have been > shown to be contaminated with HeLa, because researchers need them for > purposes beyond use as models for specific disease/original source tissue.

Beyond that, certain cell lines started out as contaminated [2]:

> In the earliest stocks available, the level of contamination was 0.6%.

[1] [https://www.atcc.org/Global/FAQs/3/6/HeLa%20contaminated%20C...](https://www.atcc.org/Global/FAQs/3/6/HeLa%20contaminated%20Cell%20Lines-1207.aspx)

[2] [https://www.atcc.org/Products/All/CRL-1593.2.aspx#characteri...](https://www.atcc.org/Products/All/CRL-1593.2.aspx#characteristics)

</comment><comment><USER6> I'm imagine that Step 0. of Materials & Methods in new papers going forward will be: "We verified that our cell line was made of intestinal cancer cells with PCR and genomic expression tests..."

</comment><comment><USER7> This contamination has happened before with HeLa cells and the original "War on Cancer". It was an absolute disaster back then, and I'm aghast that the medical industry hasn't learned from this mistake. I actually am quite shocked and horrified that this has happened again. This means that many many human lifetimes of work are invalid.

There's a great documentary by Adam Curtis on this exact topic. "The Way of All Flesh" [https://www.youtube.com/watch?v=C0lMrp_ySg8](https://www.youtube.com/watch?v=C0lMrp_ySg8)

Again, I can't express just how sad this makes me.

</comment><comment><USER8> > The scientists can carry out a genetic test before starting their research > to detect misidentified cells. But that takes time and money. "The > scientists I spoke to said that was the biggest problem," says Halffman.

Would performing such tests be a business opportunity? A researcher wants to use a certain cell product, but first sends a sample to our testing facility to see if it's a quality product or a botched one.

</comment><comment><USER9> This is overstating the issue in the present day. An immortalised cell line is after all a very convenient but very contrived model of a real system. For example, the greatest revolution in cancer treatment in the last 10 years has been immunotherapy, which was developed using more realistic models than 'cell line in a dish'.

<comment><USER10> This is what you always hear from biomed/psych/etc. It simply is not important to get details right. Misinterpret the results of your analysis (p-values), use the wrong cell lines, measure the wrong thing, etc. It rarely seems to affect the conclusions.

If anyone who points out errors is being pedantic, it makes you wonder about the point of doing all that. Just come up with an idea A, say "heads = conclusion A", "tails = conclusion not A", and flip a coin.

<comment><USER11> I think one of the things that is actually happening is that for many examples, while the _quantitative_ answer changes, the _qualitative_ conclusion isn't altered by the errors.

In graduate school, as part of a class on survival analysis, we subjected the same data set to increasingly sophisticated analysis techniques to account for all kinds of things.

Each time, the effect estimate changed.

At the end, while discussing them, the professor asked a very simple question: "Do any of these suggest that HAART is a bad idea?"

Similarly, during the Ebola epidemic, when everyone was fussing about the various forecasts missing the mark, etc., what they were actually predicting was "This is a serious crisis that needs international intervention".

<comment><USER10> I'm sure that HAART is a good idea sometimes and a bad idea other times. Also, this will change as time goes on (HIV mutates, demographics change, other treatments are available, etc)

I'm also sure that there are many things going on at any given moment that should count as "serious crisis that needs international intervention". The real question is whether it is a more serious crisis than other things currently happening.

To deal with both these issues you are going to need deeper understanding than "good idea vs bad idea" or "is crisis vs not crisis".

But the real problem with being unable to quantify your understanding is that you are left unable to make precise predictions, thus you can never perform any stringent tests. If all you can predict is something vague (eg, HAART will increase 5 year survival), there will be many ways to misinterpret the data in support of your explanation even if it is totally wrong.

<comment><USER11> The point is not that it will change over time - it was that _in this setting_ with _this information_ different methods can give you different answers, but that may not actually matter. If you're HIV+ in this country right now, you want HAART, whether I did some fancy marginal structural modeling or not.

> To deal with both these issues you are going to need deeper understanding > than "good idea vs bad idea" or "is crisis vs not crisis".

This is not, actually, how the response to Ebola worked. It was very much "crisis vs. not crisis". And a huge amount of clinical decision making is "good idea vs. bad idea".

The suggestion was not that you're not able to quantify your understanding. The suggestion was there are errors that are possible to make that, while changing your effect estimate, do not change whether or not you do a thing.

To return to the HAART example, imagine you're an HIV+ patient and I've told you that a major study failed to control for time-varying confounding, and that upon re-analysis, instead of doubling your 5 year survival, it only increases it by 87%.

Or, for a form of "is this repeatable?" that I particularly despise, that it still doubles your chances of survival, but the p-value has gone from 0.047 to 0.062.

Do you want to stop taking the drug?

<comment><USER10> >"To return to the HAART example, imagine you're an HIV+ patient and I've told you that a major study failed to control for time-varying confounding, and that upon re-analysis, instead of doubling your 5 year survival, it only increases it by 87%.

Or, for a form of "is this repeatable?" that I particularly despise, that it still doubles your chances of survival, but the p-value has gone from 0.047 to 0.062.

Do you want to stop taking the drug?"

This obviously depends on the relative costs. Cost of side effects, buying the drugs, time going to treatment, etc. That all requires accurate quantification. But cost-benefit isn't even what I meant.

To begin with, if they can't nail down a quantifiable effect that is stable from study to study, who knows what is going on? Why would you have confidence in their estimates of effectiveness if they are inconsistent with one another?

<comment><USER11> "This obviously depends on the relative costs. Cost of side effects, buying the drugs, time going to treatment, etc."

I'm going to suggest if you're facing death from an AIDS-related illness, a relative risk of 2.00 vs. 1.87 will feel very, very similar to you.

"To begin with, if they can't nail down a quantifiable effect that is stable from study to study, who knows what is going on? Why would you have confidence in their estimates of effectiveness if they are inconsistent with one another?"

Define "stable" \- because even if the effect of something is fixed in the same way a physical constant it, it's invariable sampled with error.

Again, if I give you five studies that suggest that HAART improves survival by:

100%, 102%, 87%, 94% and 96%

are you really going to suggest that we don't know that HAART improves survival?

<comment><USER10> > _" I'm going to suggest if you're facing death from an AIDS-related illness, > a relative risk of 2.00 vs. 1.87 will feel very, very similar to you."_

I at first thought we both understood there would be some uncertainty about these values no matter what but we were leaving that out out for simplicity's sake. Roughly, I was assuming it is at least +/\- 10-20%. The reason these values "feel very, very similar" is that I would not expect medical data to be able to distinguish between them.

> _" Define "stable" \- because even if the effect of something is fixed in > the same way a physical constant it, it's invariable sampled with error."_

Ok... so you _are_ implicitly considering uncertainty.

> _" Again, if I give you five studies that suggest that HAART improves > survival by:

100%, 102%, 87%, 94% and 96%

are you really going to suggest that we don't know that HAART improves survival?"_

Not enough info.

\- Where did the uncertainty go?

\- What methods were used to generate these values? Even basic strategies like blinding the people collecting/processing/analyzing data is often still missing.

\- What population/frame do those numbers refer to, and how much error can we expect from extrapolating for other situations in the future?

You need a reliable quantification, doing this hand wavy "things are better/worse, significant/insignificant" is an awful idea.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER12> I wonder how many cervical cancer deaths are parasitic cancers from HeLa cell lines? Similar to [https://en.m.wikipedia.org/wiki/Devil_facial_tumour_disease](https://en.m.wikipedia.org/wiki/Devil_facial_tumour_disease)

<comment><USER6> Huh. I'd take a bet that it's less than a dozen. I think it would be difficult to transfer it-- you'd have to have an OB-GYN who was also do research who didn't follow proper cleaning protocols between researching and doctoring.

Unlike Devils, we don't bite eachother's cervices as a way to say "hello".

<comment><USER13> I would make a stronger bet and say it's exactly zero.

<comment><USER6> Well, does a non-malignant HeLa tumor that's been artificially transplanted count? Because if so, we may have both lost our hypotheyical bets. :] [https://www.quora.com/Is-it-possible-that-the-HeLa- immortal-...](https://www.quora.com/Is-it-possible-that-the-HeLa-immortal- cell-line-could-cause-an-infection-or-cancer)

(I can't find numbers of how many tumor masses lasted, but hundreds of prisoners were injected.)

</comment></comment></comment></comment><comment> fly-swatter If only we could convince more physicists, chemists, and software engineers to do biological and medical research.

<comment><USER11> Having met a number of physicists and software engineers working in biological and medical research, they have their own, unique brands of fail in addition to the usual slate.

</comment></comment><comment><USER14> This problem has been cheerfully ignored for a long time - [http://discovermagazine.com/2014/nov/20-trial-and- error](http://discovermagazine.com/2014/nov/20-trial-and-error) :

> In June 2007, all that changed. Ain attended the annual Endocrine Society > meeting in Toronto, where Bryan Haugen, head of the endocrinology division > at the University of Colorado School of Medicine, told Ain that several of > his most popular cell lines were not actually thyroid cancer. One of > Haugen's researchers discovered that many thyroid cell lines their > laboratory stocked and studied were either misidentified or contaminated by > other cancer cells.

[...]

> But rampant contamination is not the shocker in this story. Ain retired all > the lines; he never sent any of them out again. He also sent letters to 69 > investigators in 14 countries who had received his lines. He heard back from > just two.</comment>


Neat approach to link shortening - <AUTHOR> http://www.guuda.com

<comment><AUTHOR> We just added something to our Twitter client that you might find interesting. Just paste any link into the tweet field and it shortens on the fly (with some simple animation). It also runs validation to check for dead or mistyped links.</comment>


A car-sized asteroid made the closest Earth flyby a space rock has ever survived - <AUTHOR> https://www.space.com/closest-asteroid-flyby-of-earth-recorded.html <comment><USER2> Another approach from sunwards, another new record, another reminder how much more important it is that we put effort into making sure we can prevent avoidable asteroid impacts.

</comment><comment><USER3> ...ever? That's a bold prediction.

</comment><comment><USER4> Close! And if it had impacted?</comment>


Literate SQL - <AUTHOR> http://modern-sql.com/use-case/literate-sql <comment><USER2> It's critical to have a good understanding of SQL, but once you do, ORMs with a functional syntax solve this problem for many of us. I know it's not possible for some people to use ORMs as they can't risk the ORM making a performance mistake, but I think the solution is to improve ORMs to the point that writing raw SQL is akin to writing assembly instead of using a higher level language.

That said, we're not there yet. Using an ORM in 2017 without understanding what SQL you want it to produce IS NOT OKAY. Please learn SQL if you're going to use an ORM, and understand the SQL you are producing with the ORM.

Summed up, if you can, use an ORM to solve this problem.

<comment><USER3> I love Django -- but what it does is mostly magic to me. That being said... I'm fairly competent in being effective in Django -- although not for advanced and efficient querying.

I want to get strong in SQL -- where/how do I start?

<comment><USER4> Write raw SQL queries. My feeling on the Django ORM is that it's useful for defining models and managing migrations, but I prefer to write raw SQL for all queries.

The PostgreSQL documentation is really good too.

<comment><USER5> Meh. My feeling is that Django and other web frameworks are too magical -- I always worry they might be doing something inefficient. That's why I write all my web applications in hand-tuned assembly. It even has reusable abstractions for when I need them, and nice hand-formatted assembly is super readable!

<comment><USER4> You're seriously comparing SQL to assembly? There's a point where it doesn't make sense to add another layer, another dependency, etc.

<comment><USER5> In both the "never use an ORM" and "always use assembly" cases, the argument is based entirely on having full control of what's going on and not trusting any intermediate layer to get it right or eke out every last femtosecond worth of performance.

So it's a perfectly fair comparison.

</comment></comment></comment></comment></comment></comment><comment><USER6> I wish CTEs ("common table expression", i.e. a "with clause") had the same performance as a subquery in PSQL. I always assumed they'd be implemented as a kind of macro that expanded to a subquery. Is there a good reason for this distinction?

For that matter, having some kind of SQL-oriented macro/preprocessor language would be fantastic. I guess GPP (General Preprocessor, [https://logological.org/gpp](https://logological.org/gpp)) is always an option.

<comment><USER7> For the uninitiated, what is GPP ?

<comment><USER8> Closest guess: General-purpose PreProcessor

[https://github.com/logological/gpp/](https://github.com/logological/gpp/)

<comment><USER6> That's it, sorry I wasn't clear.

</comment></comment></comment></comment><comment><USER9> Love the literate sql style. A lot of times we have to use temp tables instead of with statements though. In Vertica, all the withs of a query have fit in memory together. Using temp tables instead to create the logical sequence, you only need enough memory for each temp table and final query on there own. The cognitive effect is still the same. Define A, Define B based on A, Define C based on B, etc. but sequentially rather than recursively. When the queries are smaller, withs are more direct, a little less verbose, and usually more performant. However, with the datasets I'm working with, the queries are usually too big or I just don't know how big they will be as I start out writing. So I typically just use temp tables from the outset vs writing with withs only to have to rewrite it once I get "too big to fit in memory" errors.

<comment><USER10> We do the same thing with Redshift. Using temp tables also gives us more direct control over the query plan, since we can specify distribution and sort keys. On top of that, we can run unit tests on the temporary table (explained in another comment) and fail the query if tests fail.

</comment></comment><comment><USER11> I see that this is about a way of writing SQL, but as Literate Programming is cited in the title, I wanted to point out that you can already use the true Literate Programming tool noweb to write SQL with or without "with" statements.

The advantage is that you'll be able to write full documentation amongst the SQL, present it in any order, and reuse chunks.

The disadvantage is that it outputs to stdout, so if that's no good for your task then it's no help.

<comment><USER12> Well, stdout can be redirected in any sane environment, so that's not much pf a disadvantage. However, thanks for making the point that what they're describing (good variable naming practices, and the use of 'with' to prefix them) is hardly what Knuth was talking about when he coined "Literate Programming".

</comment></comment><comment><USER13> It took me a long time to learn SQL well enough to appreciate its beauty.

Simple queries, you know, "SELECT foo, bar FROM baz WHERE lastChange > CURRENT_DATE" are easy.

But if you are facing the database of your ERP software (as I often am) whose vendor is very reluctant to tell you about its internal structure and how that interfaces with the ERP system (my gut feeling, though, is that we're lucky - SAP and Oracle are probably much less friendly to people poking around in their databases to create custom reports, hehe), using SQL and its interactive nature to explore the database is a lot of fun as long as the database design is relatively sane. Thank God our ERP vendor's programmers were not creative enough to do insane things.

(Well, they did one crazy thing - there are NO foreign keys to be found anywhere in that database, instead it is all faked with triggers. I think that's how people used SQLite before it supported foreign keys. But we're talking about Microsoft freaking SQL Server here; being derived from another enterprise-y RDBMS, I find it hard to believe that it would at some point have lacked foreign keys. Since the triggers DO check for referential integrity, why on earth did they no just use Foreign keys? What were they thinking?)

It gets a little mind-bending at times, but in a good way.

But explaining SQL queries of the non-trivial kind to somebody is intimidating. One of our accountants at one point expressed interest in learning SQL, because she would bug me with questions that I answered by running a few carefully worded queries. For some reason I find SQL relatively easy to understand but really, really hard to explain.

</comment><comment><USER14> Why not just reverse the SELECT and FROM clauses?

        FROM table1, table2
        SELECT col1, col2
        WHERE ...

or even better

        FROM table1 JOIN table2 ON condition ...
        SELECT ...
        WHERE [additional join conditions]
        HAVING [filter conditions]

Then the compound version looks like:

        FROM (
          FROM tables...
          SELECT ...
          WHERE
        ) SELECT...
        WHERE ...

or

        FROM (
          FROM tables ...
          SELECT ...
        ) JOIN [whatever]
        SELECT ...
        WHERE ...

Actually, putting the WHERE before the SELECT makes even more sense.

<comment><USER15> Because that's invalid SQL?

<comment><USER14> So was the WITH clause until someone decided it should be valid and added it.

Every improvement to every language ever started out with someone saying, "Hey, here's an idea..." And none of those ideas were ever "valid X" at the time they were proposed.

</comment></comment></comment><comment><AUTHOR> I came to this page looking for ways to parameterize and re-use raw SQL. _with_ (and _create view_ for multiple queries) fulfill the case where the unit of reuse is a filtered view on one or more tables.

<comment><USER10> To make progress on this problem, I wrote a light SQL preprocessor to support INCLUDE statements. It enables me to write code like this:

        WITH frequently_bought_together AS (
          INCLUDE frequently_bought_together.sql
        )
        SELECT ...

This allows way better isolation and reuse of business logic than before. In Redshift, I combine this with an assert user-defined function to enable writing unit tests in raw SQL.

With all that together, I can trust analysts to update complex data assets and I can ask them to take any data issue investigation they've done and turn it into a re-usable test. Tests end up looking like:

        CREATE TEMPORARY TABLE frequently_bought_together AS
        INCLUDE frequently_bought_together.sql
        ;

        SELECT f_assert(COUNT(*) > 0, 'Table is empty');
        SELECT f_assert(COUNT(DISTINCT item_bought || item_recommended) = COUNT(*), 'Table is fanned out');
        ...

It has made a huge difference in how we write SQL.

<comment><AUTHOR> It took me a second, but I can see why this would be handy. Whether you just use it in a _with_ , or create a view from it, you have a single definition of the query that anyone can use. And you can test the _INCLUDE_ d query. Neat.

</comment></comment></comment><comment><USER12> The use of meaningful labels and "with" to prefix them may be good practice, but that's barely related to what Knuth meant by "Literate Programming".

</comment><comment><USER16> It's great to see highly legible SQL in a tutorial. These days it seems as though everyone on the internet is secretly running a SQL obfuscation competition.

</comment><comment><USER17> Nice article, thx. I think I'll start using WITH more.</comment>


The Big List of Naughty Strings - <AUTHOR> https://github.com/minimaxir/big-list-of-naughty-strings <comment><USER2> Solid list for a quick SQL injection and XSS reference with lots of examples. Even unicode/accents/two-byte characters etc are super useful to check handling on all the way from the front-end to the persistent storage solution (DB, etc).

Lost it laughing at "Human Injection" section:

> # Strings which may cause human to reinterpret worldview

> If you're reading this, you've been in a coma for almost 20 years now. We're > trying a new technique. We don't know where this message will end up in your > dream, but we hope it works. Please wake up, we miss you.

<comment><USER3> That was a bastardization of the original one on 4chan's /x/. This one is the real one:

> It has been reported that some victims of torture, during the act, would > retreat into a fantasy world from which they could not WAKE UP. In this > catatonic state, the victim lived in a world just like their normal one, > except they weren't being tortured. The only way that they realized they > needed to WAKE UP was a note they found in their fantasy world. It would > tell them about their condition, and tell them to WAKE UP. Even then, it > would often take months until they were ready to discard their fantasy world > and PLEASE WAKE UP.

<comment><USER4> > This one is the real one

Are you sure? I think this idea has been explored in several novels and films over the past several decades.

<comment><USER5> Never seen it before on film. The Manchurian Candidate isn't it. Can you give me some pointers?

<comment><USER4> Doesn't The Matrix explore this idea?

<comment><USER6> Wake up, Neo... The Matrix has you...

</comment></comment></comment></comment></comment></comment><comment><USER7> This is deiciously ironic:

> Also, do not send a null character (U+0000) string, as it changes the file > format on GitHub to binary and renders it unreadable in pull requests.

<comment> Seb-C For long I had the same problem with Japanese language files being shown as binary in the GitHub diffs, and it is solved by having something like this in the .gitattributes file

        *.php diff

Overall I am amazed that everything shows properly in GitHub. [https://github.com/minimaxir/big-list-of-naughty- strings/blo...](https://github.com/minimaxir/big-list-of-naughty- strings/blob/master/blns.json)

</comment></comment><comment><USER8> See also:

2018 [https://news.ycombinator.com/item?id=18466787](https://news.ycombinator.com/item?id=18466787)

2017 [https://news.ycombinator.com/item?id=13406119](https://news.ycombinator.com/item?id=13406119)

Show HN from 2015: [https://news.ycombinator.com/item?id=10035008](https://news.ycombinator.com/item?id=10035008)

</comment><comment><USER9> OK, seeing "﷽" [1] was unexpected :). For those who does not know, it's very important for muslims and It's all over the Quran

[1] [https://github.com/minimaxir/big-list-of-naughty- strings/blo...](https://github.com/minimaxir/big-list-of-naughty- strings/blob/9c25300f66fd968ad863412608025908c8aa5efd/naughtystrings/internal/resource.go#L899)

<comment><USER10> what does it mean?

<comment><USER11> [https://www.urbandictionary.com/define.php?term=%EF%B7%BD](https://www.urbandictionary.com/define.php?term=%EF%B7%BD)

Fun fact: it's a single Unicode character.

<comment><USER12> Yup, you can put 280 of it into a single tweet.

<comment><USER13> I don't _think_ that's right. I looked into the way Twitter counts characters when I was trying to work out the largest prime number that could be written out in full, in base ten, in a single tweet[1]; the rules are more complicated than you might expect, and have changed several times.

The current rule seems to be that all Unicode characters count as two, except for the ranges 0–4351, 8192–8205, 8208–8223 and 8242–8247 which count as one.

[1] In case you're wondering, I think it's, arguably: [https://twitter.com/robinhouston/status/1197294154738544641](https://twitter.com/robinhouston/status/1197294154738544641)

<comment><USER12> Good point! Still, I could swear I saw someone (@FakeUnicode?) do exactly this once, but of course I can't find that tweet any more, partly because it turns out that search engines don't handle ﷽ well at all, and I don't feel like testing it on my own followers somehow.

Edit: it looks like it might count it as two characters, so that's only 140 per tweet.

<comment><USER13> That's definitely possible! @FakeUnicode mentioned in the discussion that, when 280-character tweets were first introduced in September 2017, it was possible to tweet 280 single-codepoint emoji using TweetDeck.

[https://twitter.com/fakeunicode/status/1197282221503041537](https://twitter.com/fakeunicode/status/1197282221503041537)

There are several amusing examples in the thread linked from this tweet.

</comment></comment></comment></comment></comment></comment></comment><comment><USER14> I encountered an amusing instance of this recently watching my six-year-old son playing music on the kitchen Alexa. Alexa felt it was necessary to censor the name of a children's song entitled, "Pussy Cat, Pussy Cat."

<comment><USER15> When I saw the title I thought it was a list of profanity that one might want to filter out from an open web application (i.e. a list that also includes swear words from multiple languages).

</comment></comment><comment><USER16> Also tangentially related: the big list of usernames that should be disallowed in any online system: [https://github.com/forwardemail/reserved-email- addresses-lis...](https://github.com/forwardemail/reserved-email-addresses- list)

<comment><USER17> Ugh, that list might be why my email address mail@[personal domain] is forbidden more and more often.

<comment><USER18> I use a catch-all and mark some handles as spam once a website has been hacked, or the email has been caught by spammers.

I trust you've probably adopted a similar workaround for problem websites.

For throwaway websites (where I use a password I have no intention of keeping track of), I often sign up as "spam@domain".. This is surprisingly blocked in a lot of instances.

</comment></comment></comment><comment><USER19> Almost related: [https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene- and...](https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and- Otherwise-Bad-Words)

<comment><USER20> What? only 151 russian words? The russians have an own dedicated sub-language which consists solely out of bad words. No idea or concept is too complicated to be expressed in bad words alone. They switch from normal russian to bad words russian as soon as the situation allowes it.

<comment><USER21> [citation needed], please, I wanna read about this beautiful gem of a sub- language!

All I can find with some quick searching is Wikipedia's page on 'mat'[1], which seems to be pretty similar to Carlin's list of Seven Words You Can't Say On TV[2] rather than an entire language of vulgarity.

1: [https://en.wikipedia.org/wiki/Mat_(Russian_profanity)](https://en.wikipedia.org/wiki/Mat_\(Russian_profanity\)) 2: [https://en.wikipedia.org/wiki/Seven_dirty_words](https://en.wikipedia.org/wiki/Seven_dirty_words)

<comment><USER20> I don't really have a citation, just personal experiences. But your wiki link already states: "David Remnick believes that mat has thousands of variations"

The german version of the wiki article has some examples:

пиздеть (pisdet′) = to (tell a) lie, but also possible: to steal.

пиздец (pisdez) = ruin or catastrophe, fubar, fucked up

As you can see, small variations of the same word have different meanings. And the meaning can vary with context.

Edit:

The russian version of the article has some example verbs (all derived from the same word): I tried to translate as far as I understand.

ебануть = ?,

ебануться = to get stupid ( to say or do something stupid ),

ебаться = to do something - very generic,

ебиздить = ?,

ёбнуть = ?,

ёбнуться = ?,

ебстись = ?,

въебать = to get something into something?,

выебать = to get something out of something?,

выёбываться = to fuck around. To decline something to somebody.

доебать = ?

доебаться = to annoy somebody by asking too many questions or trying to get something out of somebody.

доёбывать = to bring something to an end. to finish something,

заебать = to annoy somebody, to get to somebody

заебаться = to get fed up with something, to get tired of something,

наебать = to betray somebody?,

наебаться = to get saturated by something, to get enough of something

наебнуть = to cheat sombody,

наебнуться = for the fun of it, just do it,

объебать = to get a piece of something, to get familiar with something?,

объебаться = to get into something nasty, to fuck up,

остоебенить = blow you mind (not sure about this one)

остоебеть = ?,

отъебать = to fuck with something/somebody, to fight, to degrade somebody,

отъебаться = get rid of something, for example to pass exams or get rid of obligations,

переебать = (maybe) to understand something,

переебаться = to fuck over something, to get something done,

поебать = do something (depending on context),

поебаться = to do nonsense, try hard to no avail, fool around,

подъебать = ?,

подъебаться = ?,

подъебнуть = to make joke, for example april the 1., even flirt?,

разъебать = to ( accidantly or pusposfully ) break something,

разъебаться = to clear a relationship, to bring everything to order?,

съебать = to get away (maybe in a cool way),

съебаться ( to fuck off, to stop anoing ),

уебать = to run away (maybe after stealing something)

<comment><USER22> This list mostly corresponds to my personal experiences as a native Russian speaker, and I've heard 70% used in the wild. To fill two of the ?'s:

      ёбнуть = to f*ck

      ёбнуться = to stumble or injure oneself

What's interesting is that this whole list only uses one root word, of which there are 3 more

</comment></comment></comment></comment></comment><comment><USER23> .

.

.

.

.

.

ด้้้้้็็็็็้้้้้็็็็็้้้้้้้้็็็็็้้้้้็็็็็้้้้้้้้็็็็็้้้้้็็็็็้้้้้้้้็็็็็้้้้้็็็็ ด้้้้้็็็็็้้้้้็็็็็้้้้้้้้็็็็็้้้้้็็็็็้้้้้้้้็็็็็้้้้้็็็็็้้้้้้้้็็็็็้้้้้็็็็ ด้้้้้็็็็็้้้้้็็็็็้้้้้้้้็็็็็้้้้้็็็็็้้้้้้้้็็็็็้้้้้็็็็็้้้้้้้้็็็็็้้้้้็็็็

Wow, what's this? :)

<comment><USER24> Layers upon layers of combining diacritics.

</comment></comment><comment><USER25> I don't quite understand the purpose of this list. It contains potentially malicious input, but also emoticons based on Unicode characters that are completely harmless and used in every second post on Reddit.

<comment><USER26> It's essentially a test suite for character encoding all throughout your application. If you input all those strings (e.g. send chat message) and they arrive incorrectly at some other end (e.g. other user receiving chat message) then there's a problem somewhere.

<comment><USER25> That makes sense. Thanks a lot! Of course, it's very useful for testing. I erroneously assumed it was for input validation.

</comment></comment></comment><comment><USER27> Strongly advise not using cat on the list, you will get beeped at.

<comment><USER28> would that be considered animal abuse :D

</comment></comment><comment><USER29> Yup, can't view the file using the GitHub app for Android

<comment><USER30> Out of curiosity, what happens when you try to do so?

<comment><USER31> Something went wrong

<button>TRY AGAIN<button>

Edit: as far as I could see it's only opening blns.txt that causes this error the other files are fine in the app.

</comment></comment></comment><comment><USER32> The latest commit message of README is "Merge branch 'master' into master" [1]. As someone who doesn't do git, what does that even mean? Does git allows multiple branches with same names?

[1]: [https://github.com/minimaxir/big-list-of-naughty- strings/com...](https://github.com/minimaxir/big-list-of-naughty- strings/commit/84c8b77529dfdfaf3d28d597681dff5f633aa91b)

<comment><USER33> The GitHub flavor of git essentially has two parts:

Repository (i.e., minimaxir/big-list-of-naughty-strings), and branch (i.e., master)

In this case, someone merged eliabieri/big-list-of-naughty-strings at master to minimaxir/big-list-of-naughty-strings at master via a pull request [1].

[1] [https://github.com/minimaxir/big-list-of-naughty- strings/com...](https://github.com/minimaxir/big-list-of-naughty- strings/commit/eec4732aac0d42107e1cc3701cc4a8be7ff351de)

</comment></comment><comment><USER34> Jimmy Clitheroe - the Clitheroe Kid. That brings back some memories. It's also nice to see that England is suitably represented in the place names, obviously Scunthorpe is the classic. I'll tender Somerset for first amongst equals for daft and downright odd place names.

</comment><comment> bloody-crow We need a similar list of weird unconventional emails to make sure every new registration form won't erroneously reject a valid email.

The number of times I get validation errors or some unexpected crashes when I enter my fairly pedestrian email with + sign in it... Jeez.

</comment><comment><USER35> [https://github.com/minimaxir/big-list-of-naughty- strings/blo...](https://github.com/minimaxir/big-list-of-naughty- strings/blob/master/naughtystrings/internal/resource.go#L1228)

just lovely ;)

<comment><USER36> This one got me:

> "If you're reading this, you've been in a coma for almost 20 years now. > We're trying a new technique. We don't know where this message will end up > in your dream, but we hope it works. Please wake up, we miss you."

Strangely terrifying....

<comment><USER37> The question is, which one of us is the message meant for?

<comment><USER38> Why would there be more than one of you?

<comment><USER39> Why are there so many of me posting in this thread?

<comment><USER40> Username checks out.

</comment></comment></comment></comment></comment></comment><comment><USER41> Neat. I recently found that googling the Japanese Post Office emoji results in a totally borked SERP (cross-browser, desktop, including desktop mode on Android Chrome). I assume there are other characters as well.

</comment><comment><USER42> I just do:

      .matches("[a-zA-Z0-9.\\-]+")

And prepared statements or my own NoSQL.</comment>


Microsoft's war on plain text email in open source - <AUTHOR> https://marc.info/?l=openbsd-misc&m=159843434525592&w=2 <comment><USER2> The Register's original story [1] makes clear that the problem was not Microsoft's mail clients:

 _> "It is a fairly specific workflow that is a challenge for some newer developers to engage with. As an example, my partner submitted a patch to OpenBSD a few weeks ago, and he had to set up an entirely new mail client which didn't mangle his email message to HTML-ise or do other things to it, so he could even make that one patch. That's a barrier to entry that's pretty high for somebody who may want to be a first-time contributor." _

_> We assumed that Outlook was to blame. Could Microsoft fix that instead? "The question always is fix it to whose standards, because we are focused much more on business and enterprise models of clients and customers. For them we fixed it to a more HTML-based model so it really depends on who your audience is and who your target is." _

_> It turned out, though, that this time Outlook was not guilty. "I think it was actually Gmail that was a barrier. And he also couldn't do it from Apple Mail. It is just that the modern mail client has intentionally moved towards HTML," she said. _

[1] [https://www.theregister.com/2020/08/25/linux_kernel_email/](https://www.theregister.com/2020/08/25/linux_kernel_email/)

</comment><comment><USER3> The gist of this is that a new OpenBSD contributor couldn't figure out how to open a text email with their current email client. It was set to use HTML messages only. So this developer had to install an entirely new client just to handle messages. Therefore, open source communications must change to use a new system of some sort, presumably like GitHub.

The rest of the thread is rightfully criticizing the claim for its absurdity.

<comment><USER4> This reminds me of a sysadmin in one of my previous company tried to edit a Solaris config file in Word. He told me he couldn't make it to work after he ftp it to the server. I looked at file and saw those CR characters and found out he used Word to edit it.

<comment><USER5> You can still end up in such situations today without the absurdity of using word, if you're using Windows.

I've wasted an hour or so this year trying to figure out why an instance of Microsoft's sshd port of OpenSSH didn't behave as configured.

It took a while for me to realize that if you create/change the config files with Windows tooling, they will likely end up being encoded in UTF-16, which means that every second byte is a NUL character. If you have a C codebase expecting UTF-8 (like most UNIX-ish tools), that can lead to the config files just being ignored (as the tool likely hits a NUL before the first real character). In the case of sshd, you won't get any warnings or errors, either.

<comment><USER6> PowerShell (before Core version) uses UTF-16 for default encoding, I wonder who is happy by this default encoding.

<comment><USER7> You need to know your history in order to understand this. Very briefly: Development on Windows NT began in 1989, three years before UTF-8 was even invented; and there was a difference of opinion between the ISO people and the Unicode Consortium over Unicode being 16-bit or 32-bit, with Joseph D. Becker famously publishing a paper in 1988 declaring that 16 bits were sufficient for Unicode ("with a safety factor of about four").

</comment></comment></comment></comment></comment><comment><USER8> Since it was not previously mentioned, Sarah Novotny (while at Google) was on the original bootstrap committee that organized the Kubernetes ecosystem. This is by no means "Microsoft attacking Linux" or promoting GitHub; this is a personal critique from someone that has helped to create a more approachable, better organized, large scale project for the Linux Foundation already.

</comment><comment><USER9> I agree. I wanted to make a minor change to the Linux kernel. Nothing more complicated than fixing a typo.

Getting everything configured correctly was a pain. Finding out who to email for an unexpected hurdle. It took a bunch of emails back and forth until I got the format right.

I did it, but it put me off contributing again.

Would I have become a top-notch kernel developer if the process was easier? No. But I'd certainly have done some of the boring busy-work to fix minor issues.

I appreciate the workflow is set in its ways. And it probably (intentionally?) prevents a lot of casual abuse of the process. But surely there's a better way to let new people contribute?

<comment><USER10> 1\. make change in git, commit it with a good message.

2\. call ./scripts/get_maintainer.pl -f on the file you changed

3\. call git send-email -1 --cover-letter --annotate and tell it where to send the patch.

4\. done

This is significantly _easier_ than clicking a bunch of buttons on a slow Github or Gitlab instance.

<comment><USER11> You forgot `0. setup outbound mail`. Which may include figuring out how to setup a smartrelay. Which for some provider leads to explicitly enabling mail submission and generating a separate auth token for your email provider. Which you will need to learn about by searching multiple pages. Ah, and if you're in a corp environment, forget about sending anything out on port 25 - you'll need to figure out a proxy as well.

If you're working with that project every day - sure, that's trivial.

If you're sending a drive-by patch useful to others, it's really in the project's best interest to make it as easy as possible. In at least one case I ended up sending an email like "This is a patch, it's in public domain now, your change process is so annoying I'm not going to spend the next hour to set it up, KTHXBYE".

<comment><USER7> I have a couple such on my WWW site. (-:

But in fairness edent's problems were finding out who to address the mail to and how to format the message, not how to actually use electronic mail in the first place; and one might as well note that the procedure is also missing "learn how to plug in a keyboard" and "learn how to edit C source code", if one is going to take it that far.

</comment></comment></comment></comment><comment><USER12> This should probably just be updated to point to The Register's article: [https://www.theregister.com/2020/08/25/linux_kernel_email/](https://www.theregister.com/2020/08/25/linux_kernel_email/)

<comment><USER13> From the register article:

> It turned out, though, that this time Outlook was not guilty. "I think it > was actually Gmail that was a barrier. And he also couldn't do it from Apple > Mail. It is just that the modern mail client has intentionally moved towards > HTML," she said.

</comment></comment><comment><USER14> It would be nice to use something akin to Github... I don't care if it's SVN/Trac, Gitlab, or whatever -- mailing lists + patches have their limitations and there are better ways to organize OS development. It certainly isn't a 'war' though.

</comment><comment><USER15> There's a certain irony I found that some of the emails in the thread get mojibaked due to the mailing list server apparently not knowing how to indicate the charset properly.

</comment><comment><USER16> This was a nice post explaining git and email [https://drewdevault.com/2018/07/02/Email-driven- git.html](https://drewdevault.com/2018/07/02/Email-driven-git.html) and a website that looks nice. [https://git-send-email.io/](https://git-send- email.io/)

I'm curious if any sourcehut users here could explain more about sourcehut's code review tools. Do they make email-based code reviews easier?

</comment><comment><USER17> This isn't exactly new, in terms of Microsoft email tools being a pain to use for submitting patches without HTML munging. In fact, I'm pretty sure it's been a problem for over a decade. The only new part here is them making suggestions.

That said, from my experiences hosting email lists, that might indirectly be the higher bar to entry. So many providers' anti-spam policies basically treat discussion lists as spam, especially if you are so bold as to have multiple subscribers who use that one list. Gods forbid one of the users manages to delete their address or let it go stale without unsubscribing first and the list software lets a single email get bounced.

</comment><comment><USER18> _That 's a barrier to entry that's pretty high for somebody who may want to be a first-time contributor._

I think that's actually a good thing. The person has to be interested enough to setup and learn about plain text email. Its not discrimination based on race, religion or gender identity but discrimination based on interest.

Even simple barriers to entry have positive effects. I found that the quality of bug reports on Gitlab is much higher than on Github because everyone has a Github account and will happily spam a project with a lazy bug report (me included).

</comment><comment><USER19> Surely if you want to patch the Linux kernel, then you can deal with text based emails?

<comment><USER20> I was just thinking this as well when I read that sentence.

Plain text emails are a pretty low bar in terms of technical challenges if one could submit Linux kernel patches.

</comment></comment><comment><USER21> The verge article paints a different picture, which I kind of agreed with (as someone who has grown as a developer in last 5-10 years). But then again, I haven't contributed to Linux so it might be a bad thing. FWIW, the problem with email clients seem to be with Google and Apple. My Outlook mail client handles the text email very well!

</comment><comment><USER22> That's on Microsoft to come up with an alternative. A replacement would be like if plain Git was entirely replaced with a modern looking front end.

You can have your pretty javascript applets, but please don't mess with the greybeards.

Linux needs as many people as possible. Don't scare any of those people away.

</comment><comment><USER23> Would it help if mail clients can be configured with a list of e-mail addresses or domains that are known to prefer plain text, and then automatically warn the user and suggest to switch to plain text? Arguably such mail clients can come with a preconfigured, default list of well-known addresses or domains.

That way the user doesn't have to go out of their way to find the plain- text/html switch.

Another improvement is that mail clients implement a MarkDown or reStructuredText editing mode that will automatically generate a multipart message with both plain text and html parts. Is there any mail client like that out there?

</comment><comment><USER24> Perhaps the ability to submit a patch in the required format is a fizzbuzz?

<comment><USER25> > _a fizzbuzz_

You mean a ceremonial hurdle that has absolutely no bearing on that person's worthiness to work on a particular project?

Yes, it does sound like a fizzbuzz.

<comment><USER26> More like the absolute minimum one would expect any contributor to be able to produce.

I'm not a fan of coding interviews, but being able to produce a functioning fizzbuzz should not be considered very difficult.

<comment><USER25> Something easy can suddenly be impossible when you're anxious or under pressure.

If someone can't do fizzbuzz, they won't be able to do things that are actual signals of ability (like finish a side project or a take-home project).

</comment></comment></comment></comment><comment><USER27> How is this a "war" on plain text email? It is a true statement and it doesn't really sound like a huge corporate initiative, just a personal comment from one of the executives.

</comment><comment><USER28> [https://web.archive.org/web/20200827033621/https://marc.info...](https://web.archive.org/web/20200827033621/https://marc.info/?l=openbsd- misc&m=159843434525592&w=2)

</comment><comment><USER29> Apparently Microsoft engineers can't hit the button in Outlook to turn the email to plain text.

<comment><USER17> At least in the past, on Outlook, that used to still munge attached patches to Windows line-ends instead of DOS, which was still a problem.

That's Microsoft's fault, but if you are at some smaller shop trying to submit a patch, it can be a royal pain in the neck, especially if the admin is a Microsoft fan and only wants to run Exchange protocol.

</comment></comment><comment><USER30> But how could anyone possibly be productive without using Outlook? /s

</comment><comment><USER31> Is there a GUI mail client that's preconfigured to work with mailing list and patch workflows?

<comment><USER32> mutt in an xterm works quite wonderfully.

</comment></comment><comment><USER33> I can't view that Marc site on mobile. I have to horizontal scroll text...

</comment><comment><USER34> Ha Ha, [https://en.wikipedia.org/wiki/Unix_philosophy](https://en.wikipedia.org/wiki/Unix_philosophy)

"Write programs to handle text streams, because that is a universal interface."

</comment><comment><USER35> Url changed from [https://marc.info/?l=openbsd- misc&m=159843434525592&w=2](https://marc.info/?l=openbsd- misc&m=159843434525592&w=2), which points to this article.

Edit: I decided to revert that switch. The article it points to is [https://www.theregister.com/2020/08/25/linux_kernel_email/?](https://www.theregister.com/2020/08/25/linux_kernel_email/?).

</comment><comment><USER36> Man, they are really getting aggressive with Embrace-Extend-Extinguish these days.

High barrier to entry for Linux kernel patching seems like a feature, not a bug.

</comment><comment><USER37> HTML in email is awful but sadly in most place it has won. In Apple Mail you can't attach pictures to plain text email w/o them being converted to HTML again. Gmail behaves completely different. And on Android you can't really deal with fixed lines of plain text emails.

But what is even worse is, that most clients create extreme awful HTML, Outlook comes to mind. In a few version they put in the font-family "sans- serif" instead of sans-serif. Which meant the fall back would fail.</comment>


Facebook isn't cool - so what? - <AUTHOR> http://www.slate.com/blogs/future_tense/2013/11/14/facebook_s_cool_kids_problem_instagram_snapchat_and_the_anti_facebook_phenomenon.html

<comment><AUTHOR> "You know what else isn't cool? Searching for things on Google. Composing documents in Microsoft Word. Email."</comment>


Beware Prestige-Based Discretion - <AUTHOR> http://www.overcomingbias.com/2016/06/beware-prestige-based-discretion.html#comments <comment><USER2> Letting the most prestigious people make important decisions isn't the best system, but it's probably better than the other options.

<comment><AUTHOR> How would you define "most prestigious"? In the academic context: Best at the grant hamster wheel? GPA maximization at $x university? Best politicking? Number of papers? Actively solving relatively trivial problems for little economic value for people who are `Best at the grant hamster wheel`? :P

<comment><USER2> I don't think it's the sort of thing that yields to calculations, but I think it does yield to consensus. So, generally speaking, the most prestigious people are whoever other people agree are prestigious. Sort of like peer review.

The "who should make decisions" question is a lot like the "is this logic malicious" question. You can't answer it ahead of time. You just have to let people make decisions, or let the logic run, hopefully in a controlled context, and see what happens. If a preponderance of people agree it's good, then it's probably good enough.

<comment><AUTHOR> > _I don 't think it's the sort of thing that yields to calculations…_

"But we often have plenty good metrics that people don't want to use."

"I think the problem is more fundamental; basic human prestige programming."</comment></comment></comment></comment>


Japan's Sacred Island - <AUTHOR> https://www.archaeology.org/issues/364-2001/features/8232 <comment><USER2> In China, the feminine Mahayana Buddhist bodhisattva 觀音 ( _guanyin_ ; a Sinified Buddhist-derived mythical goddess) is the deity most frequently worshipped by fisherpeople. Representations of her in China prior to the Song Dynasty (~1000AD) were masculine, from the male _avalokitesvara_ of Sanskrit Mahayana Buddhist doctrine of India, specifically सद्धर्मपुण्डरीकसूत्र (蓮華經; the _Lotus Sutra_ ), which is thought to date to around 100-200AD.

It would be interesting anthropology to compare (particularly premodern) folk rituals from fishing villages and other frequent seagoing cultures.

</comment><comment><USER3> "The sheer cliffs of the small island of Okinoshima rise abruptly out of the sea some 40 miles off the coast of the Japanese island of Kyushu. Okinoshima's sole resident is a Shinto priest who serves as the caretaker of small wooden shrines built among huge boulders on its southern half."

For an instant, I thought the article was referring to Luke Skywalker.

<comment><USER4> Skywalker's island was shot at Skellig Michael - [https://en.wikipedia.org/wiki/Skellig_Michael](https://en.wikipedia.org/wiki/Skellig_Michael) \- an island off the southwest coast of Ireland. The stone huts on the Ireland are the ancient homes of monks.</comment></comment>


Ask HN: Looking for compression algorithm that took 6GB to 800MB. Anyone know? - <AUTHOR> Looking for a good media compression & archiving algorithm for an app I'm building. More specifically, I'm looking an algorithm I had used 10 years ago but had totally forgotten the name of.  Description: It was a self-extracting archive (about 800MB) that spat out a folder of over 6GB (mainly media: movies & audio). Now, the media files themselves were already in compressed format (MP3, MP4, etc...) so I was really impressed. Only drawback (as expected) was that it took over 1 hour to decompress on an Intel dual core. Does this kind of performance ring any bells for anyone? I just need a name. <comment><USER2> Call me a skeptic, but there is no way there was an algorithm available to humans 10 years ago that _reversibly_ compressed 6GB of unique, already- compressed media files down to 800MB. The _only_ way this could have happened is if there were shared files or shared segments between the files. For example, if a DVD had a bunch of audio tracks but some of the tracks were basically just direct copies of the others, then the compressor could recognise the similarity and capitalise thereon. For lossless compression of general data, 7-zip set on Ultra is probably the best available right now. On the other hand, algorithms such as FLAC or PNG work well for losslessly compressing uncompressed media.

</comment><comment><USER3> I did quick internet search and only app mentioned by people that is able to achieve such compression rate is KGB Archiver[1]. And according to its wikipedia page it uses PAQ[2].

[1] [https://en.wikipedia.org/wiki/KGB_Archiver](https://en.wikipedia.org/wiki/KGB_Archiver) [2] [https://en.wikipedia.org/wiki/PAQ6](https://en.wikipedia.org/wiki/PAQ6)

</comment><comment><USER4> We just to joke at NetApp that the Oil and Gas industry had the best compression algorithm, it could compress 100TB of seismic imaging data into a single bit {oil / no-oil } :-)

I created a theoretical compressor which I haven't yet been able to implement which uses the fact that every sequence of bits appears in pi somewhere, so my compressor would just return the digit offset and the length of data. I keep looking for a source for all the digits of pi though, have yet to find it.

<comment><USER5> After having experimented a bit in high school with ideas for compression algorithms like this, I can tell you that one of the problems you'd run into is that you may have to look so deep into pi that the offset may actually take the same number of bits (or even more bits) than the sequence you're trying to find.

<comment><USER4> If you're up for it there is some great number theoretic work to be had in that statement :-). Given a sequence of digits X, P{X} E {Offset n .. Offset p} ? len{Offset n} > len{X}

</comment></comment></comment><comment><USER6> I don't believe there is any known lossless algorithm which can achieve 7.5:1 compression on already-lossy-compressed media files.

If you want further lossy compression for existing media files, look at the newest algorithms supported by ffmpeg.

If you want generic lossless compression, you can do a bit better than the usual suspects (gzip et al), but only if you're willing to put up with very slow compression times.

If you have some other type of specific data (e.g. sparse files) then you could do something custom, but I guess this is unlikely to be your situation.

</comment><comment><USER7> You can look through the software here: [http://www.maximumcompression.com/benchmarks/benchmarks.php](http://www.maximumcompression.com/benchmarks/benchmarks.php)

Or the software linked via here: [http://prize.hutter1.net/](http://prize.hutter1.net/)

As other people say, what you're asking for probably isn't possible.

</comment><comment><AUTHOR> I wanted to add: The self-extractor had no interface -- it was all in the windows command prompt. A message in the prompt said something about "Media compressor" and below it was the percentage extracted.

</comment><comment><USER8> Pied Piper?

<comment><USER9> inside out compression algorithm</comment></comment>