The Hostile Email Landscape - <AUTHOR> http://liminality.xyz/the-hostile-email-landscape/ <comment><USER2> I've managed my own mail server since 1993, and my email address has been the same that entire time. Here are some tips for maintaining sanity:

Greylisting still works amazingly well. With a long, long whitelist and greylisting plus DNSBL, I don't even bother running a spam filter, since the little bit of spam and emails from new senders ends up in its own directory as it came from a non-whitelisted sender.

Comcast finally started blocking residential mail server ports inbound a few years ago, so I had to migrate to a smarthost environment using a VPS as email server for $15/yr.[1]

Last year for a few months, Gmail was dropping everything I sent into the spam folder, even after recipients were marking it not spam. I eventually discovered the "Authentication-Results:" header that Gmail adds to every inbound message. It is under the "Show Original" dropdown menu. That showed that I "hadn't changed anything"(!) on my mail server, but suddenly Gmail was connecting to my mail server over an IPv6 interface, and I had never bothered to put the IPv6 block into the SPF record. Gmail was nice enough to explain exactly what it didn't like about those emails.

[1] [http://lowendbox.com/blog/top-provider-poll-2014-q3-the- resu...](http://lowendbox.com/blog/top-provider-poll-2014-q3-the-results/)

<comment><USER3> > Greylisting still works amazingly well. With a long, long whitelist and > greylisting plus DNSBL, I don't even bother running a spam filter, since the > little bit of spam and emails from new senders ends up in its own directory > as it came from a non-whitelisted sender.

Any good tips on this section in particular? If I'm running my own mail server, how would I get started making sure this is in order?

<comment><USER2> I have been running exim4 for years, but I'm in the process of moving to postfix, as postfix is considerably easier to set up all the DKIM, etc., machinery that is now required. Inbound email comes through procmail and is mainly read in emacs (mh-e), which is kind of old fashioned. I have a small script that makes a new email address within my domain for each new use. I sign up for a lot of mailing lists and groups, and my /etc/aliases is more than 5300 lines. I can track if domainA's address starts coming from domainB and disable that address, but that doesn't happen very often, which is a pleasant surprise.

I also have a small script that puts a new sender on my whitelist of sender email addresses. My whitelist is 12000+ lines right now, collected over many years. Procmail sorts to mailing lists and vendor folders, and finally puts things that are not on the whitelist into a "possible spam" folder. From the five or ten items a day, it is easy to spot legitimate emails and I add those to the whitelist. The majority of spam is blocked by the combination of greylisting and DNSBL lists, as the delay of greylisting (ten minutes for me) is enough for them to make the blackhole list, if they happen to ever attempt delivery again.

I was thinking recently that I should be collecting statistics on the use of a lot of those aliases and whitelisted emails, and maybe start garbage collecting my lists.

There are various reputation reports and services that can tell you how your mail is doing in the major ISPs, but a lot of those require higher traffic than a personal or small business generates. There is one service, DMARC[1], that is free and can give you some visibility into how email from your domain is being processed. I put the txt record in my DNS, and Google, Facebook, Comcast, Yahoo, Fastmail, and a few others send me reports about email they have processed from my domain. It's not that interesting at the moment because things are working, but it might help to debug issues if your email was being rejected. At least I see a few spammers are trying to use my domain from their servers.

[1] [https://dmarc.org/](https://dmarc.org/)

<comment><USER4> If you are using emacs, you might want to try the emacs UI for notmuch:

[http://notmuchmail.org/](http://notmuchmail.org/)

</comment></comment></comment></comment><comment><USER5> I've run into similar issues with a similar setup. It's frustrating. You can convince gmail user A to whitelist your messages, and so they'll get through to user A, but gmail user B probably still won't see messages from you unless you tell him to dig them out of the spam trap. And your messages to A might still be classified as spam if they have attachments or hyperlinks in them. (Even if you've been corresponding with A for several years!)

Once upon a time, to send email, you needed to use SMTP. Now, you must use SMTP, from a IP block that isn't categorized as residential, and which has never before had any association with outgoing spam, and you also must implement several ad-hoc identification protocols like SPF and reverse DNS. You should also use a domain name that you've owned for some time and which is not expiring soon. Every system to which you want to send mail will give different weights to all these signals. If they don't like you, their behavior is to report successful delivery and then silently hide messages from their intended recipients.

Spam is no fun, but the present situation is pretty weak too.

<comment><USER6> [http://cr.yp.to/im2000.html](http://cr.yp.to/im2000.html)

Internet Mail 2000 IM2000 is a project to design a new Internet mail infrastructure around the following concept: Mail storage is the sender's responsibility.

<comment><USER7> Bernstein is an incredibly smart guy. But he's also an example of how the social stuff matters. His software would run the internet if he put a little more effort into the social/political side of things.

The big problem with IM2000 is that it doesn't solve the _real_ problems. It's focused on the economics of storage of mail, which, yeah, are a thing for a few mail administrators, but generally is considered less important than the spam-filtering problem in general.

I mean, I'm not willing to pay for infinite storage, but I certainly would be willing to pay the cost of receiving 100x my current monthly spam load if I didn't have to read any of it. Maybe even 1000x.

The major benefit of the IM2000 standard is that if you can squash the spammer's mail servers, you squash the mail in question, which actually could change things, because it would mean that spammers would need to control the servers they use to send for longer than they do now.

<comment><USER8> That his software does not "run the internet" does not make it any less good. It's there for whoever chooses to use it.

That he focuses on the software instead of pandering probably makes his software better than the alternatives that aim to please even the most foolish of users. At least I think so.

Maybe I interpreted the proposal incorrectly, but I always saw IM2000 (minus the "notifications") as a "pull" solution.

By contrast, conventional email relies on "pushing" spam to the recipient (in practice, a middleman called an "email provider").

A smart IM2000 recipient perhaps would not pull spam from the sender's server.

As such, the spam would never enter the network. It would just sit on the sender's server.

Therefore, IM2000 not only conserves storage but also conserves bandwidth.

<comment> mike-cardwell "A smart IM2000 recipient perhaps would not pull spam from the sender's server."

Based on what criteria? You wouldn't have the message headers/content, so you couldn't filter based on that. All you can filter on is the IP address. But you can already do that with SMTP by just responding with a 5xx code as soon as the sending server connects anyway... Or you could actually fetch the message and then spam filter it. But then, you may as well use SMTP...

What difference does it make which side initiates the TCP connection? If I was a spammer, IM2000 wouldn't concern me one little bit.

</comment></comment></comment></comment></comment><comment><USER9> The problem is not so much the attitude of the big guys. It is that smtp is fundamentally broken. we need a better mail protocol that ensures: 1\. Traffic always encrypted and content always signed 2\. Guarantee that the sender is who it claims he is 3\. Decorrelating the email from the domain, a lot of users are prisoners of their current provider just because the address they gave everyone ends with the provider's domain name, very much like it is very hard to switch bank accounts 4\. Ability to provide disposable adresses which can be deleted when spammed

3 and 4 would require a sort of token system

<comment><USER10> > Decorrelating the email from the domain, a lot of users are prisoners of > their current provider just because the address they gave everyone ends with > the provider's domain name, very much like it is very hard to switch bank > accounts

This one seems completely uninteresting in a world in which $15/year can get you your own domain name, complete with reliable email servers, IMAP, and as many email aliases as you like at that domain. Given that, why create an entire infrastructure to "decouple" email addresses from domains?

<comment><USER11> That's $15 a year that some people can't, won't or haven't paid before. If someone decides later on that they wish to do so, they have to set up forwarding and try and convince contacts to use the new e-mail address instead.

</comment></comment></comment><comment><USER12> Many of the issues we're running into with online systems, particularly those relating to quality and reputation (spam, collaborative filtering or content rating as on HN or Reddit, etc.) have strong analogs in real-world social spaces. And there were real-world mechanisms for dealing with these.

For a new businessman or professional setting out in the world, pre-Internet, "establishing your name" was a requirement. These days the concept's often referred to as "creating a personal brand", but the reality was pretty straightforward: how does an unknown quantity become a known quantity?

A common method was the professional or social introduction. This is still practiced, where a third-party _matchmaker_ will introduce two parties. The matchmaker usually knows both, and can vouch for the newcomer and smooth the path for introductions with the established party. Essentially, the matchmaker stakes _their_ reputation by speaking for another.

Lawrence Lessig describes a similar concept in his book _Code and Other Laws of Cypberspace_ , in a passage describing a physical messaging system, the Yale Wall. This was a board onto which messages could be posted, with the proviso that that they be signed. Unsigned messages would be posted from time to time, effectively presenting an anonymous viewpoint. Removal wasn't instantaneous or automatic, but rather, at some point _prior_ to garbage collection, another individual could review the piece, and, if they felt it warranted posting, sign for it. They weren't registering themselves as _author_ , but as _vouching for the merits of the viewpoint_ \-- not necessarily agreement.

A messaging system in which a new peer might be able to indicate that, hey, peers X, Y, and Z, with established reputations can vouch for me, and for which those peers could confirm their endorsement, might address the "how to build reputation" issues of new mailservers.

I've also found that even large mail systems will frequently have _some_ procedure for getting at least provisionally vetted, effectively a workfactor cost to coming online. Though the process absolutely could be improved.

</comment><comment><USER13> It bothers me that this entire thread is about technology. This is anticompetitive behavior by a small group of oligopolists, plain and simple.

The big providers need to build an accessible system for entering the email market or face tremendous fines from worldwide governments. There are plenty ways imaginable to do this, especially if you allow the process to involve real people and paperwork.

It shouldn't matter whether this is simple negligence or conscious work to keep out competition. The end result is that it's made nigh on impossible by a few market leaders to enter said market.

</comment><comment><USER14> I feel like this is a solvable problem without making any changes to email whatsoever. The problem is the email recipient hosts are suspicious of the sender (as opposed to the message itself being suspicious). So the solution is to have a standardized way for senders to acquire an instantaneous reputation by tying their real-world identity to it (which lets them be held accountable if they do spam), and perhaps by throwing some money at it too. If there was some company that did identity checks, similar to how EV certificates are given out, then that ties your real-world identity to it (this could in fact be done by literally requiring an EV certificate for the hostname of the sender). This company could also take a decent-sized deposit (so you're staking money on not being a spammer) and hold it in trust for a set amount of time. Once the time has passed, and you've sent enough emails for recipients to draw meaningful conclusions, if you have in fact not spammed, then you get your deposit back (minus a service fee). Then all the big email hosts would pay this company to query it about senders the host doesn't already trust, and similarly they'd report any spam from these hosts back to the service.

Heck, this doesn't even have to be a new company. A big host like Google could just start offering this service anyway, as a way to simplify their own handling of unknown senders, although I'd feel more comfortable if this was done by someone else.

<comment><USER15> The core problem to this is that reputation is valuable. Google or Twitter or Facebook or any of the other big web presences _could_ offer a method to determine "reputation" of any of their users, and a way to canonically tie your identity to that reputation, but they'd be undermining their own business model. They want you to stay in their network, because the safety of interacting with only reputable users is the big selling point of their walled-garden social networks over open networks like email

<comment><USER14> What do you mean by "stay in their network", when it comes to sending emails? Google is not in the business of being a paid mass email sender for businesses. What business email they handle is specifically for being the email host for a business's internal email, not sending their newsletter / marketing / account notifications / whatever else.

</comment></comment></comment><comment><USER16> Here is an almost related story: I am in italy and was waiting for an email from the local Apple store to tell me my macbook's repair was complete. After 6 days waiting, I called the store. They said the computer had been ready for a few days.

I checked the spam folder. Gmail had plopped the apple email into the spam folder. Gmail's reason was that the email was in a foreign language, Italian. Didn't matter that the previous 2 weeks all my google.com searches were done from Italy.

<comment><USER12> Google's killed access to various accounts of mine at different times for:

1\. Accessing an account _one time_ over Tor. Subsequent reconnect attempts from the same device I'd previously used, same IP, etc., failed. Took a couple of weeks to get back.

2\. Accessing accounts from different IPs while travelling. Ultimately ending up at the same IP as one of the email accounts I'd previously corresponded heavily with.

3\. Accessing accounts from new device(s). Old (Android) devices could access email but not Web-based account recovery tools.

"Who are you" is the most expensive question in technology. No matter how you get it wrong, you're fucked.

[https://www.reddit.com/r/dredmorbius/comments/3mo7l6/that_go...](https://www.reddit.com/r/dredmorbius/comments/3mo7l6/that_google_identity_thing_again_who_are_you_is/)

<comment><USER17> I was once disappointed not to hear back from a Google recruiter after an interview. Turns out it was in my Gmail spam folder. Along with any other @google.com address.

</comment></comment></comment><comment><USER18> _> This isn't how the internet is supposed to work. _

The email architecture was started back when it was a smaller network of researchers at universities, governments, etc. Everybody basically _trusted_ each other.

Once the "internet" is available to the general public and commercial interests, it becomes vulnerable to the "bad actors" problem (e.g. spam abuse). That's why we have the inevitable situation today of a few entities (e.g. gmail, hotmail) being "trusted", and random residential SMTP servers run by homeowners being "untrusted".

I haven't seen a realistic de-centralized trust proposal for email. Even if a proposal is theoretically sound, what incentive is there for other big players to adopt it?

<comment><USER19> How do you solve the problem of "cold call" with a trusting system ? How does one build up trust with a new account ? Spammers could create million accounts at once and fake trust build up between them. That's why I'm not convinced that trust will solve the abuse of messaging. A messaging application must allow "cold calls". Relying on trust built up by others is not a reliable system because it can be abused.

<comment><USER18> _> How do you solve the problem of "cold call" with a trusting system ? [...] That's why I'm not convinced that trust will solve the abuse of messaging._

You can't remove "trust" or "reputation" from a viable messaging system. The concept of trust _always exists_ whether informally or formally.

When the internet was a small private network between universities and government, how did a new unknown scientist "cold-call" an email? It was not an issue. The scientist just sent the cold email. There was already _implicit_ trust within the system to allow that scenario. Everybody trusted that email@example.com didn't send a million emails about Nigerian money transfers to firstname.lastname@example.com.

Back then, email users didn't need "spam filters". The trust was informally created because the institutional "system" _outside_ of the email system vetted email users. (The "system" being the hiring procedures of SomeUniversity, DepartmentOfDefense, etc.)

All those gates for reputation are gone when we expose that simplistic email architecture to the general public. Trust doesn't go away. You now _must recreate trust_ at a massive scale.

 _> Spammers could create million accounts at once and fake trust build up between them. _

That's not the type of "trust" people are talking about in this thread. That type of incestuous "trust" between bad actors won't work because it doesn't have an initial authority (good actor) to "bless" them which starts an acceptable "chain of trust". The concept is similar to Certificate Authorities. We trust Chrome because we trust Google Inc; and in turn, Google Inc trusts Verisign; and in turn, Verisign EV-certificate processing trusts government offices that register businesses.

<comment><USER19> Ok my assumption of trust credit working model was inaccurate. Could you give an example how trust could be used to control abuse ? If a user is given initial trust credit, how can one prevent spammers to create million of such "trusted" accounts and let them be banned after the hit and run spamming ? That model us used in messaging systems like Twitter but has its limitation. But is known to have its limit. Or did I still got it wrong ?

<comment><USER18> _> , how can one prevent spammers to create million of such "trusted" accounts _

Notice what you did there? You characterized spam accounts as "trusted" even though you didn't specify why or how they'd get trusted in the first place. There's still a gap in your thinking of how the origination of _real_ (not fake) trust happens.

It's like a 1970s email user asking, _" how does the system prevent a 'trusted' Nigerian money scammer from spamming everybody?"_ Well, why would Department of Defense hire a Nigerian scammer? Would a UniversityOfWhatever hire a Nigerian scammer? No. In that scenario, the trust _begins_ with the institution. For the Nigerian scammer to be trusted, _somebody_ authoritative has to "bless" him.

 _> Could you give an example how trust could be used to control abuse ? _

Well, we're still using _trust_ right now to help control abuse. Today, reputable corporations and also millions of honest users (that's us) "trust" Google Inc, Microsoft, Yahoo, FastMail, etc. Spam on a massive scale does not originate from gmail/hotmail/etc because those corporations shut them down.

The issue is that it's _centralized_ trust concentrated in a few big players instead of _de-centralized_ trust. That's the thorny problem that's very hard to solve. All the clever decentralized proposals (sender pays for storage, outbound payment bonds, cpu-proof-of-work, etc) are basically roundabout proxies for recreating "trust" on a bigger scale. I don't know what the final answer will be. I just know that trust/reputation is an unavoidable part of a new solution.

</comment></comment></comment></comment></comment><comment><USER20> This is a true story:

I rent some servers from the Rackspace cloud for personal use. I have my own sites on these machines, and my own email servers.

Meanwhile, I have a day job, and lately it has been consuming 12 hours a day. We missed a deadline and we have all been working like crazy to catch up. I have fallen behind reading my personal email.

Roughly a month ago, my friends who use Gmail stopped getting my email. Or rather, they did not know I was sending them email, because all of my email to them was going to spam.

After a few weeks, I finally had a free weekend to catch up on my personal life, so I did some investigations. Turns Rackspace had switched over to IP6 in a way that impacted my email. I did not have a Sender Policy Framework for IP6, only IP4.

It's likely that Rackspace sent me an email about this, though I never read it because I was busy.

This was easy to fix: I added a SPF for IP6.

However, these kinds of issues do make it harder to maintain a personal email server. Its tough for us to keep up with the changes.

</comment><comment><USER21> Drinking game for this thread. Start at the top of the comments:

* every time someone implies spam filtering is a massive conspiracy against "the little guy", drink

* every time someone suggests Bitcoin be used for something, drink

* every time a solution utterly fails to account for compromised end-user machines sending spam, drink

* every time a comment can be summed up as "assuming a PKI exists, this problem is trivially solved", CHUG THE REST OF THE BOTTLE

<comment><USER22> LOL

That reminds me of an old classic: [http://craphound.com/spamsolutions.txt](http://craphound.com/spamsolutions.txt)

There are a lot of obvious problems with email.

* It's hard to run your own mail server, as described in the original post

* Your address is tied to your provider

* The standards involved in delivering mail (SMTP, POP, IMAP, etc) are complex, layered with hacks (SPF, STARTTLS, etc), hard to implement correctly, hard to secure.

* The standards that govern the actual body of email (multipart MIME, a 90s-vintage subset of "HTML" and "CSS") are even worse

But despite all those flaws, email is built on ubiquitous open standards running on a federated infrastructure.

It's the best thing we have.

Proprietary standards and centralized designs like AIM, ICQ, MSN, FB Messenger, Slack, and Twitter DMs come and go.

Email is forever, and is more useful than all of those combined. The flipside is that it's very hard to "fix".

</comment></comment><comment><USER23> > This isn't how the internet is supposed to work. As we continue to > consolidate on a few big mail services, it's only going to become more > difficult to start new servers.

And this is _exactly_ the reason I setup my own mail server. I'm only 1 man, but I hope more people will do so with time, thus requiring the "big ones" to work on better algorithms for filtering and not base it on reputation.

</comment><comment><USER24> Create an email network where is would cost a penny to send email. It would be payed into bitcoin wallet of folks maintaining infrastructure.

Every email would be digitally signed and encrypted. Certificate with keys would connected to email address (and bitcoin wallet).

Spam would die.

Go build it please.

<comment><USER25> The minute you say bitcoin is the moment you cut out 99% of the population; unless it is in the background and one does not have to interact with it directly.

Other than that, I agree paying to send will reduce spam.

But also look at your postal mail box. There is arguably more spam there than in your digital inbox, and that one costs (stamps).

<comment><USER24> Yes, but if the email is digitally signed I can be certain who sent it (at least the address) and, unlike postal mail, I can electronically sort it. Email from folks I know would receive the highest priority.

Are folks afraid of bitcoin? What can be worse than just having a 16 digit number printed on a card -- that is a 1000x more vulnerable.

<comment><USER25> _Are folks afraid of bitcoin?_

You are part of the tiny percentage of the tiny percentage of the population that understands bitcoin.

</comment></comment></comment></comment><comment><USER26> I have a bit of experience with running email servers. I can't really say that I had similar encounters.

In my experience if you get blocked by big mail providers it's almost always due to some reason. What's tricky is that it may be hard to tell what exactly is wrong, because they won't necessarily tell you (or not in an easy way).

Some advice what I'd do to try to find out what's going on:

1\. Take a sent example mail that is like the blocked one (but obviously one that reached its target destination) with all headers and run it through spamassassin. Don't just look if it hit the spam score (then you did something terribly wrong), look at each individual rule that spamassassin hit. They might give you a clue. A proper mail usually shouldn't hit any or very few positive spamassassin rules.

2\. Check your IP at a service like valli where you can query multiple DNS black lists. If it is on any blacklist try to find out how you can be delisted. There are some rogue blacklists that make it impossible to be delisted at all, you may ignore them (google for them, their behavior is well documented), but these shouldn't be more than 1 or 2. As already said by other commenters, don't forget IPv6.

3\. Read whatever error message you can get your hands on. If you're blocked on the SMTP level read the error message. If your message got sorted into a spam folder look at all the headers. If the provider blocking you has some online docs about their spam filtering read that. If they have some sort of service for mail ISPs where you can sign up to get warnings sign up there.

Of course also the obvious stuff. If you do anything that is mass mailing you are in extra danger. Make sure that you allow people to unsubscribe easily, don't ignore manual attempts by them to unsubscribe ("I want to get off this mailing list") and delete invalid mail addresses.

</comment><comment><USER27> I was thinking about this a while ago and have been meaning to write it up and post it somewhere, so I guess this is as good a time as any.

Hashcash (also known as the precursor to Bitcoin) was proposed to solve this problem in 1997:

[https://en.wikipedia.org/wiki/Hashcash](https://en.wikipedia.org/wiki/Hashcash)

The trouble with it is that it requires computation for each sent message, which is bad for senders with low resource devices or legitimate mailing lists. I want to propose a variant.

Instead of creating an expensive hash against the message, create an even more expensive hash against the (sender TLS certificate, receiver domain name) pair. This implies using TLS but it works just as well even if the certificate is self-signed. Then each mail server only has to generate a hash once per recipient domain, ever. Every message that mail server sends to that domain is tagged with that hash. A legitimate mail server will have already computed hashes for all the domains its users regularly correspond with and rarely if ever need to do any more expensive computations.

If spammers do the same thing then the receiving server can mark all messages sent with that hash as spam. So there is a highly disproportionate cost to spammers (even if they have more computing power) because to avoid that they have to continuously generate expensive new hashes. Which can be made arbitrarily expensive because legitimate servers only need to do it once. And a new hash is much less valuable to a spammer than a domain name or IP address is today because each hash can only be used against one recipient domain. The required amount of computation can be set by the receiving server so domains with more users can require more computation.

If a legitimate mail server is compromised by a spammer then it will have to generate all new hashes (because the spammer will presumably immediately ruin the reputation of the compromised ones), but the reputation of the legitimate sender's email domains is unharmed because the reputation is tied to the hash computation, not the sender's domain name(s).

And adding support to mail servers would require no configuration whatsoever. You install server version N+1 and it starts tagging outgoing messages with hashes that receiving servers can verify.

So here's the traditional form: [http://craphound.com/spamsolutions.txt](http://craphound.com/spamsolutions.txt)

How'd I do?

<comment><USER28> Sounds like a plan!

With a memory-bound proof-of-work system like my Cuckoo Cycle, computing the hash could require the use of more than 4GB of memory for over 5 minutes on a 20-thread server, thus preventing the use of botnets for avoiding the expense.

<comment><USER19> Not sure Internet providers or google will be enthusiast about this proposal. This is equivalent as asking to pay to register a new destination address. In this case the mail relay owner has to pay. It would be more fair to ask users who send mail to pay to register a new destination address. Google and mail relay provider would be more happy with this variant.

<comment><USER27> It's very friendly to large providers. The amount of computation needed to send to a domain is the same regardless of how many emails you send so large providers can amortize it over more emails. And large providers generally operate their own infrastructure, which implies they already have a lot of idle computing capacity during off peak hours, and "off peak" is staggered across regions.

But even if some large providers refuse to hash outgoing mail, they can still verify incoming mail. The interesting benefit is that a _small_ email provider can burn some dollars in computation once to avoid having their mail discarded by a large provider's spam filter indefinitely.

</comment></comment></comment></comment><comment><USER29> I sometimes see similar tales of woe, and I can only say that this does _not_ match my experience. I've done this many times, you set up the mail server, configure DNS correctly (including reverse lookup), and that's it. Never had problems being blacklisted or mail getting classified as spam.

I _suspect_ that people having trouble are _sending a lot of mail_ , like "newletters", etc. But I can't prove this hypothesis.

<comment><USER30> Mail reputations are very real and pose an issue with mail servers. I had a gaming server for years and had many issues with Gmail, Microsoft and Yahoo, to name a few, filtering or blocking emails. Just last week, I setup a mail server using mail in a box on a new server I spun up on Digital Ocean using an IP that was on no blacklists and still had issues with sending emails to various Gmail subscribers.

Even when I used to work at HostGator and handled many of their abuse issues, they had many issues with being blacklisted just because RBLs didn't recognize new HostGator IPs or the rate of email being sent from their new gateways.

So, at least with my personal and professional experience, I can attest to the issues with using self-hosted mail servers.

<comment><USER31> New IPs are often old IPs that have been used for other purposes before, and often blacklisted. Especially IPs on easily spun-up and spun-down hosts like DO. On DO, after every VPS I've removed, and then started up another a week later, it has a different IP. It would be easy for a spammer/phisher/whoever to abuse this quick address re-allocation, but hurts regular users.

</comment></comment></comment><comment><USER32> OP, I don't know if you are reading the comments here but in case you do: Don't get discouraged so quickly.

The reason this is happening is as the blurb from the MS postmaster help page: Your IP doesn't have a reputation yet.

The reason these rules are in place aren't about email monopoly, it's about spam. If anybody could setup a SMTP server and start firing off large amounts of mail, spam would be even more endemic than today.

You can configure your server perfectly, but that doesn't mean much, since it's your IP that's the problem.

If you have legit objectives, it's a pain in the ass for sure. But you are not the only one having this problem, and there's a solution for it.

All the big email service providers (ESP's) like Neolane, Exact Target, Mailchimp, Campaign monitor etc share this problem when they onboard a new client, who requires their own IP.

Deliverability is a surprisingly deep, technical topic, and all major ESP's have entire teams of specialists working on this.

If you want to make such a service as Fastmail, you need to get really into deliverability. It's not a walk in the park, but it's not impossible either.

I'm not a specialist in this particular area myself, so I can't give you that much specific advice. I've just worked elbow to elbow with a lot of these guys, so I know what kind of challenges they work with.

One thing I know for sure is really important, is the "warming up" of IP's. Basically the IP you are sending from needs to accumulate some reputation over a period of time, typically a month or two.

If you send out reasonably small amounts of mail to email addresses that exists and the recipients does not explicitly report you for junk mail, your IP get whitelisted and you will get a much higher delivery rate.

There's no quick fix unfortunately, and email reputation is hard to gain and fast to lose.

But it certainly can be done. You sound very competent on the server side of things, so to get your fastmail-like service up, I think it's just a matter of a bit more persistence and studying deliverability as a technical subject.

Hope this helps.

<comment><AUTHOR> Thanks for the encouragement! I might take another shot at it sometime. I had this server running for about 3 months with just my personal mail, sending probably 1 email per day on average. It's possible slightly higher volume would do a better job of "warming up" the IP.

I self-hosted from about 2007-2008 and 2011-2013, and had nowhere near as much trouble with deliverability. It came as a bit of a surprise how much more difficult it is these days.

<comment><USER32> If you try again, try to see if you can get a client with a decent sized mailing list. Warming up is usually done with progressively larger sendouts. So you'd start with maybe 500 sendouts, then a thousand, then 3000 etc.

But anyway yeah the barrier to entry has risen really much. It's much harder these days than just a couple of years ago, and it probably won't get easier either!

</comment></comment></comment><comment><USER33> Email has been the my last hold out from switch away from gapps completely. I don't want to have to deal with any of this, especially as I do business communication with clients via it. Email wasn't suppose to be like this, and there has to be a better way to enable non-giants to successfully deliver email.

<comment><USER34> Joke answer: the blockchain!

Real answer: It comes down to trust (duh). But, how do we manage trust online? How do we manage trust in real life? Real life trust is through association of groups. But, groups online are meaningless. A "gmail user" doesn't belong to a community, they belong to The Nation of Google.

How do we break down online identities into manageable, trustable communities? How do we bootstrap new communities into the system so we don't end up with a pre-selected list of blessed communities while others languish in obscurity?

Open questions, no answers, and if you say Facebook you have to run 20 laps and give me 1000 pushups.

<comment><USER33> You joke about the blockchain, but HashCash (a similar technique to Bitcoin's Blockchain-difficulty (not the metric itself, but how it's used)) was originally conceived as a means of making email computationally costly to send. The main issue was that spammers would use botnets while normal people would be stuck taking a while sending emails (iirc).

Personally, I would like everyone to have their own rsa or ecc key. While not as expensive as HashCash, it does require more computational effort to send an encrypted email. (This precludes web portals from sending mail, but user convince seems to always trump security! :( ). Still not a great solution.

Web of Trusts are an interesting idea, but fails for the same reason not everyone has a public encryption key: bad ui, complex idea, and no one cares. I honestly think this is the way forward. If someone isn't a few hops from me, they probably don't have any business contacting me anyway -- if they really want to, a.k.a. read my blog and want to ask me a question, they can use HashCash or something else expensive to make the initial contact and slowly build up trust/reputation as we exchange emails.

In the end, everyone is always told they need to check their spam folder for false positives anyway. Maybe we should all just say "screw it" and read all our email. If some combination of the sender and subject doesn't make us think it's legit, then we can delete it.

Maybe we keep spam filters, but stop calling it a "Spam Box" and treat it like trash. Maybe we start calling it "Unknown Box" and treat it with a little skepticism instead of a dumpster.

<comment><USER35> With regards to HashCash, the explanations I've seen didn't persuade me (the counter arguments most often cited have been made by some researchers in some IMHO very weak papers). To me it seems that the real reason why it's not being used is that ISPs would have to allocate resources, which would make them less competitive (who wants to offer free email then pay for a lot of infrastructure, or deal with the mess explaining that users have to pay for better delivery). The CPU cost could be moved to the clients, but somehow no IMAP based mail clients seem to ever have implemented it (chicken and egg problem?), and JavaScript has not been an efficient way to do it.

Also, I guess the big ISPs have an easy enough time (in their view?) analyzing trust, which is probably a better solution in general if applicable. The problem is that it's easier to judge (newly created) internal users (accounts) than external ones (random IPs).

I would think improving the mail system would be a research topic, but perhaps not interesting enough from the point of view of researchers (would it be best done in social sciences or computer science?, and there are many real-world variables, taking time to collect and verify, and working out simulations to predict possible remedies).

</comment></comment></comment></comment><comment><USER36> This largely true and hugely disappointing. Our startup ([https://portal.cloud](https://portal.cloud)) is making it possible for non- hackers to self-host their own email servers. It works very well for the most part, but we have had to explain to a number of them why their email sometimes gets bounced by the big proprietary cloud services.

All of our users have their own domain names, IP addresses, SPF records, and correctly configured (and up to date) Postfix SMTP servers. There is absolutely no excuse for not always delivering their email, and yet this is what the big companies do.

<comment><USER37> There is no obligation to accept email, from anybody.

<comment><USER36> There's no good reason a company should prevent their users from receiving email from their self-hosted friends.

</comment></comment></comment><comment><USER38> Edit: They said they checked the spam lists but the ones that I've listed below may be helpful to others.

OP might have had an IP previously used by spammers or in a spam block. (See edit).

Important to check the IP before using it if it's been given to you by, say, the VPS Provider or the upstream. To see what, if any, reputation it has.

Here are two tools to use:

[http://multirbl.valli.org/](http://multirbl.valli.org/)

[http://mxtoolbox.com/SuperTool.aspx](http://mxtoolbox.com/SuperTool.aspx)

And there are a bunch of similar tools.

</comment><comment><USER39> Outlook.com is possibly the worst offender at present:

1\. They set SPF policies of "-all" on customer domains. These customers have no idea about SPF and often use 3rd party software to send account statements that never get delivered. This breaks email for their customers and for everyone else. They should set "~all" by default as Google Apps does.

2\. They claim not to, but they definitely block entire IP ranges rather than a single IP address. A noisy neighbour in the same datacenter as you can get you blacklisted. Use [http://www.senderbase.org/lookup/ip](http://www.senderbase.org/lookup/ip) to monitor your neighbours and report them to your hosting company. If you spot a bad neighbour, you will no doubt see your own IP blacklisted at Outlook.com's [https://postmaster.live.com/snds/ipStatus.aspx](https://postmaster.live.com/snds/ipStatus.aspx) within a few hours.

3\. They are well known for accepting mail and then silently dropping it, without bouncing sender or receiver and without putting it into a spam folder, even when the sender is a longtime sender to that receiver and when the sender is sending personal anticipated email. They should either reject at the SMTP transaction (sending a bounce later is bad practice and leads to backscatter) or else deliver to a spam folder. Silently dropping email breaks email for everyone.

The sad thing is that many businesses are switching to Office 365 and Outlook.com.

</comment><comment><USER40> This is indeed worrying. I've been running my own E-mail servers for the last 20 years or so and even though my problems weren't as severe, I did run into a few cases where the "big ones" were at least delaying my E-mail.

But this kind of problem invariably arises when we go from a fragmented Internet with lots of small hosts/providers to an Internet of several walled gardens, run by the big guys.

The real answer is to offer a reasonable self-hosting competitor to GMail.

</comment><comment><USER41> I have self-hosted my mailserver for a long time and started to get problems a couple of years ago. The main issue is corporate networks running McAfee's "MxLogic" product that claim to bounce my mail and tell me so, but then go on to deliver it almost all the time.

The difficulty in getting feedback is extremely frustrating, particularly compared with getting feedback from, eg, google's webcrawlers.

</comment><comment><USER42> What has completely decimated mail management in my experience had been the rise of cryptolocker related emails.

You could always deal with traditional viruses by blocking certain types of attachments. A false negative on spam just annoyed someone with some Viagra sale they probably didn't want.

Nowadays, you fail to block an email and suddenly a user loses his entire department's file share for the day.

</comment><comment><USER43> It is simple: the more fear uncertainty and doubt the "big email providers" can cast on no using one of the big email providers the more they chase everyone into their business (when they can read it). You can go on and on how it is "technically hard to fix email" but that is a second order effect to not even trying.

<comment><USER43> "no using" -> "not using". sorry.

</comment></comment><comment><USER44> Having run my own email server about a decade ago I can kind of understand why they are just rejecting things. Dealing with spam for even just a handful of domains was a nightmare.

</comment><comment><USER45> Gmail has Postmaster tools that let domains with moderately large mail flow be able to monitor their spam reputation scores, email authentication e.g DKIM / SPF, DMARC rejection rates, etc. This tool can help address a number issues folks here have mentioned wrt Gmail. In particular please look at the "Delivery Errors Dashboard" description in the help center article.

help center post: [https://goo.gl/7QHoqc](https://goo.gl/7QHoqc) Postmaster tool: [https://gmail.com/postmaster/](https://gmail.com/postmaster/)

(disclosure I work in Gmail)

<comment><USER46> We are using Gmail Postmaster tools (I work at a small ESP), and while this tool is certainly useful for diagnosing rejections, delivery to Gmail inbox / promotions is still a black box.

In our case, a number of IP addresses we are using started showing up as "bad" in Postmaster tools, despite having no increase in complaint rates (actually, in Spam rate dashboard it shows 0.0% for the past 90 days). We have implemented Gmail feedback loop headers into outgoing emails to diagnose the issue, but the Feedback Loop dashboard also does not show any data.

All of these IPs have a Senderscore reputation in the high 90s and deliver perfectly fine to all ISPs, except Gmail. Contacting Gmail team through their form also yields no response.

My takeaway from this situation is that once Gmail starts not liking your emails for some obscure reason, you have practically no way to fix that, apart from getting a new IP / sending domain and starting to build sending reputation from scratch.

</comment></comment><comment><USER47> Did you try to acquire reputation by having real people with accounts in "known" services send emails to your server?

<comment><USER48> If this sort of shit was required to get a basic www-server up and running, do you think anyone would have cared about the internet at all?

The internet was supposed to be for everyone and anyone. Current internet email is terribly flawed. There's just no way around it and the current status cannot be defended.

</comment></comment><comment><USER49> It may be infeasible to run a new SMTP-based mail service from "residential IP's" that can interact with the existing email empire, dominated by store and forward middlemen who expect to make money from the "free" email service they provide.

That empire amounts to a junk email delivery service and later a way to gather information about email users. The later purpose is probably why you want to run a new email service?

However it is certainly feasible to run a new SMTP-based email service from residential IP's that does NOT interact with the existing email empire. One with no middlemen. The sender's SMTP server talks directly to the recipient's SMTP server. You decide what port you want to use. There are thousands to choose from.

There are multiple ways to do this, but I rarely if ever see this option discussed. I suspect it's because like DNS most users are not comfortable configuring mail servers nor with NAT traversal.

If indeed the motivation for running your own mail service is because you do not want your mail stored on third part servers (whether in the sender's mail folders or the recipient's), then the ability to interact with the existing store and forward email providers seems a counterproductive requirement.

</comment><comment><USER50> OP might still have some configuration issue (check the headers of the mail that has been placed in the spam folder. Gmail usually tells you what the problem is).

Last August, I have added IPv6 to our mail server, so its address for sure didn't have any prior reputation. Once the PTR and SPF records were correct, sending mail over v6 to gmail was no problem at all.

I have yet to see any delivery issue over v6 that's caused by a lack of reputation

</comment><comment><USER51> Surprised he didn't mention third party reputation providers such at Return Path.

<comment><USER52> I'm surprised as well, since these services reinforce his point even more. In other words, good reputation = $$$.

Also, I currently use mailgun (and dabbling with mandrill on some new projects) but I'm intrigued by postmarkapp.com take on dedicated IP addresses. Apparently they won't sell dedicated IP addresses because of the time needed to warm them up, so you get a shared IP and their TOS is stricter anti-spam than other ESP's: [http://blog.postmarkapp.com/post/14127210172/the-false- promi...](http://blog.postmarkapp.com/post/14127210172/the-false-promises-of- dedicated-ips)

<comment> old-gregg Mmm... not really. Dedicated IPs are hard to beat if you have a steady traffic of high quality and low-latency delivery is a priority (think of PagerDuty). Most legit businesses/customers have that. Most likely they're not offering dedicated IPs because they're tough to get, especially if you are a small company in email business.

The reason Mailgun and Mandrill can do this is because they belong to much larger companies with better access to IPv4 stockpiles: Rackspace and Mailchimp.

Source: I worked at Mailgun.

</comment></comment></comment><comment> andrew-lucker It's only a matter of time before we start seeing peering issues between major email providers. This is not a system for sharing, it is competition.

</comment><comment><USER53> Email is such a broken system. It's unacceptably hard to get a message from one machine to another and have it not marked as spam.

Furthermore I can't stand when I have to setup all of my accounts across the multitude of devices that I own. Way too many settings to configure regarding ports, domain names, encryption techniques, folder mappings, etc.

Don't get me wrong, I'm super impressed that email has managed to continue to be an essential tool in many people's day to day workflow despite being old enough to be found in a museum. But it's time for a change, and for us to put all we've learned about encryption, data serialization, and user workflow out on the table to design a better future for the universal method of digital asynchronous threaded communication.

And if we've learned anything from Google Wave and friends, these changes have to make sense, and not stray too far from what we're already familiar with and know that works.

Anyone want to weigh in?

</comment><comment><USER54> My experience has been that sending from my server works fine if it's in DNS and RDNS. Sending in that server's name works fine if the SPF records are present. But I don't bulk send; everything I send is something I typed, other than some messages the server sends to me periodically.

</comment><comment><USER55> You can talk about HTTPS and forcing encryption all over the web however much you want but as long as you're stuck with SMTP and the few large coorps as the only viable mail solutions you can forget individual citizen data privacy... at least thats my 2cents.

</comment><comment> grey-area Perhaps the problem here is that there is no verified identity for email servers?

<comment><USER33> What's that even mean?

<comment><USER56> something like EV certificates for smtp servers?

<comment><USER33> That proves I own the domain to some extent. Don't DKIM records do the same? For the purposes of anti-spam, a cert doesn't do anything except prove that you showed some CA that you own the domain (which can be done and is done in other ways currently).

<comment> grey-area At present SMTP is so broken that arbitrary servers are allowed to send mail for any domain, with fake headers, there is no proper verification of sending server or sending identity, the mail is accepted (along with legitimate mail), and put in spam folders (sometimes, as is real mail sometimes). SPF/DKIM are an attempt to solve that of course, but they are not enforced rigorously or universally, and tactics like IP reputation and email content sniffing _still_ seem to be widely used too (the problem reported in the article). Compared to the web, email seems pretty behind on verifying server identity.

<comment><USER33> Yes, that's my point. It doesn't really solve any issues that SPF/DKIM don't already solve -- pointing to certs isn't a new solution, it's a rehash of the same solution.

And yes, the concept of relays should disappear along with much of the other cruft SMTP brings. (And while we're at it, can we fix/replace IMAP, or at least make the spec say that message ids are eternal and can't be invalidated?)

<comment> grey-area IMO Message ids should be defined as a sha512 of headers and content to ensure they are unique. A new protocol is required.

<comment><USER33> Currently headers get modified in-flight, so a content-based ID wouldn't work.

As for a new protocol, there are some out there, but all fail in some way or another. I figured I'd throw my ideas out there and see how much they fail [https://github.com/jimktrains/email_ng](https://github.com/jimktrains/email_ng)

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER57> The problem goes both ways.

Spam needs to stop and so too does the convergence of Internet services in general (not just email).

Frankly, the solution already exists, and just like how billion-dollar companies start with the tech community, techies need to embrace the idea of decentralization by adopting some trust-based model.

Frankly, I do not see why encrypted emails cannot be accepted by users through some peer-sharing agreement where the public-key is online.

There's tons of solutions that are easy for tech people and if this community embraces it, it trickles down, like how a lot of other things do in tech.

<comment><USER58> I don't think we will see a trickle down effect unless there are substantial benefits for the average user to outweigh the cost of switching and learning something new.

If there is a single abstract advantages, like 'more security', that might not be good enough.

We might as well throw in the kitchen sink and come up with a detailed list of every possible advantage an ideal replacement protocol should offer:

\- no domain names required to create addresses

\- 1 terabyte+ attachment file sizes

\- ability to update and delete messages which recipients have not yet downloaded

\- mail transfer and storage agent which don't have access to unencrypted message contents

\- mail agents which can be distributed across multiple personal and home devices

\- client apps which always and non-optionally use local encryption\decryption

\- ability to purchase mail serving and storage resources from a competitive commercial market directly from client application.

\- ability to switch to a different commercial serving\storage resource provider without loosing your previous identity\address.

</comment></comment><comment> grey-area Everyone seems to agree that email is broken (and yet incredibly useful and almost universal in reach).

So moving on from there, how do we fix it? Who is currently working on fixing it? What would a new protocol look like?

<comment><USER59> Chat services such as Slack, and social media like Twitter, are excellent ways to communicate. It's pretty simple. If you implement a request system so that both parties have to agree to let the other send messages, all of the problems are solved. There's no good reason that we need a system that allows anyone to send an arbitrary number of messages to anyone else.

</comment></comment><comment><USER60> > this server was configured perfectly: (...) SPF, DKIM and DMARC policies in > place

SPF, DKIM and DMARC are no indicators of spamminess of a source. These systems have a completely different purpose.

<comment><USER61> They should shift IP reputation to domain reputation though.

</comment></comment><comment><USER62> I suspect many of my messages are getting flagged as spam as well. Is there an easy way of checking whether emails are going through?

</comment><comment><USER63> I've looked into this a bunch as we've been building Nylas. Our backend is essentially a cloud mail user agent (MUA), but can have similar issues to a MTA or mailbox provider.

It turns out that creating a successful product in the email space requires you to build relationships and partnerships with the existing vendors/providers. It takes a lot of work, and is part of what you pay for when using Mailgun/Mandrill/etc.

These "greylists" and systems that drop mail from unknown IPs have been pretty carefully designed and tuned to combat the insane amount of spam out there. They work remarkably well, and for most of today's email users, spam is no longer an issue. (The current state of email abuse innovation is promotions/marketing/etc. which is a more subtle challenge.)

At the end of this article, the author writes, "This isn't how the internet is supposed to work." Ironically this system of obscure reputation-based email is a _direct result of how the email system was actually designed to work_ , with a total lack of permissions or feedback loop. Many SMTP servers used to not even require a password.

Stuff like DKIM/SPF and DMARC is a step in the right direction. But the RFCs upon which our email system is based were written decades ago, and in many cases have fundamental flaws, like SMTP leaking metadata no matter what. I could go on and on about issues with email, and why I care about getting them fixed, but let's just say it was designed in a different era of Internet with different constraints and opportunities.

So how do you build a new email service and not get blocked? Well, you spend a few weeks or months emailing, calling, Skyping, and meeting with folks in the current space. You work your way up through marketing support and random protocol discussion lists until you are talking with the folks who can influence which IPs are blocked/unblocked. Then you convince them you are (1) building a legit venture, (2) are worthy of their trust, and (3) don't directly compete with them. Then you'll get a small number of clean IPs and you _must not screw up!_

There are a few hacks, like sometimes a single partner/vendor will sell you a block of clean IPs and help manage the spam reputation. But usually it's just a lot of sweat and annoying phone calls. It takes way, way longer than setting up SPF/DKIM. The challenge is more relationships than technical.

Once you have a new email sending provider, the burden shifts to _you_ for doing abuse/spam prevention. And you find yourself implementing many of the strategies and systems you cursed when getting started. But that's the circle of rfc2822 life I guess.

Oh, and never use EC2 IPs for sending mail. Most of them have been burned by spammers.

</comment><comment><USER64> I guess big providers has to deal with "newbies" all the time, that don't know how to configure their server and run open relays. And don't know how to add extra headers etc.

That said, silently dropping messages without a notification is probably illegal! And pretty serious! So if you know what you are doing (and you are not a spammer) you should send a cease and desist!

Just make sure you use double opt-in and that providing an e-mail address in sign-ups/etc is optional!

Some will however put your mail in a spam-folder and it's not much you can do about it, just hope your readers complain to their provider.

So basically: setup your smtp relay correctly! Make sure you are not on any black-lists. Add extra headers like dkmi / precedence, add spf (don't froget ipv6) and ptr. Add your relays to white-lists. Publicly publish a privacy and e-mail policy (important! with opt in and optional clause); Link to them and fill out some "email provider" forms at gmail/microsoft. Send out a bunch of test mails. This will take a whole day, but if you do this, you will have no problems unless your IP or domain is perma-banned.

<comment><USER65> There is absolutely no law that says you have to accept an email, nor that you have to send a notification if you don't. A cease and desist?! Are you kidding?

</comment></comment><comment><USER66> Nobody else has mentioned it, so i will; This is a great situation for the NSA. Get all 400M accounts in one fell swoop by using a gag ordered warrant on Google (or microsoft) Ez peasy. Much harder than to contact a sysadmin about their 1 account that isn't being fed into the behemoth..

<comment><USER67> > Nobody else has mentioned it, so i will

Do we really need a comment against the NSA in every single thread mentioning emails or hosting in general? Not that I'm a NSA supporter or anything.

</comment></comment><comment><USER68> I use a private email provider (privateemail.com, via Namecheap), which doesn't seem to stop others receiving my emails, but sites sometimes have trouble emailing me, oddly enough.

Though that might just be because I have a 6-character domain (ajf.me).

</comment><comment><USER19> SMTP Mail is broken. The best justification is that error messages can't even be sent back to avoid abuse by spammers. People are constrained to use services like gmail to avoid troubles. This is a serious privacy threat.

</comment><comment><USER69> The anti-competitive consequences of this are really interesting. Does anyone know if there are historical statistics on email provider market shares? It would be interesting to see how things have changed over time.

</comment><comment><USER70> I did not realize that I am running my private SMTPd (+ imaps) for nearly 8 years. I never had issues. My score on test-mail is 9/10 because of lack of DKIM. I will implement DKIM, see if I can get 10/10.

</comment><comment><USER71> The irony is that at least half the spam I get comes from @gmail.com addresses.

</comment><comment><USER72> Charge the sender one cent per email through a combination of legislation and technology. The problem will instantly go away. The transfer of mailing lists to online forums will be a very small price to pay.

</comment><comment><USER73> McAfee Mail Gateway also uses a reputation, called GTI. Appliance hashes the message, looks up it on this online service and get back a score.

There is a KB explaining that new senders get a high score.

</comment><comment><USER74> This guys story is a sad one but I'm sure we're missing some important information here. New mail servers get set up all the time, it's not impossible for such servers to be accepted by the big players.

If none of these major services ever learned that his server was OK, it's likely that users weren't unmarking the mail he sent as spam. And that leads to the question of why not.

<comment><USER75> In general, everybody uses a service like MailChimp or SendGrid to do this. That allows them to send email, but keeps them from building up a good reputation for no spam on their IP addresses -- that reputation is built by the email provider, not by you as the customer.

For awhile I wondered why so many SaaS apps bothered to pay external (not cheap!) mail services when it's so easy to set up an email server.

This is why. My "easy to set up an email server" info was years out of date.

<comment><USER76> sadly, it's even hard to get some email across when using those ESPs (Email Service Providers).

Our domain has all the correct SPF records, DKIM etc and we're using a reputable email provider (Mandrill). All is sparking clean. Validated etc. Still we see some activation emails to google get delivered, but severely delayed on the Google side (after we see an OK from google for the email being delivered). Our users can't reset their password or activate their account until a certain time passes...

Emails from my mum (who I've emailed on the same account for nearly a decade) suddenly get flagged as spam by gmail. Go figure.

</comment></comment></comment><comment><USER77> It's a lost cause.

The blacklisting ugliness is one more sign email was badly designed. One more nail in the coffin.

</comment><comment><USER61> So, why they look at the IP address at all when there is DKIM in place?

</comment><comment><USER78> I see lots of threads shitting on the guy for doing it wrong vis-a-vis his configuration whilst ignoring his actual problem: An IP address without a reputation score. I've had the same problem and reached the same conclusion. The address can't just be clean, as in not on a blacklist, but has to essentially already be whitelisted via a "known good" reputation score or mail automatically gets blackholed. How do I get my VPS provider of choice to give me an IP address with a good reputation score?

</comment><comment><USER4> Another post lamenting this situation:

[https://sfconservancy.org/blog/2015/sep/15/email/](https://sfconservancy.org/blog/2015/sep/15/email/)

</comment><comment><USER79> Just goes to show reputation isn't the end all, be all, solution to everything. It's so often championed by the Linux kernel team as a reason why the distributed model works, and it's evident that in that case it certainly does. Here though, much outcry about how you can't setup a box and instantly be as respected as established boxes that have earned rep over time. This post just comes off way too butt hurt for my taste.

</comment><comment><USER80> Mail spammers are truly vile people. They have ruined it for everyone.

</comment><comment><USER81> I think it's exaggerated. I am happily running a private mail server on a tiny vserver. My emails get through to whomever I might mail and I do so a lot.

I once heard that my email got flagged as spam.

Of course, if you don't set up your mail server correctly it might be that it's flagged as spam, but it's not really harder than setting up various other things correctly.

It sounds like it was set up correctly by the author. Most people use SpamAssassin, also big companies do. So it should be good.

Maybe the network itself wasn't considered to be good by the mentioned big companies.</comment>


Caddy – The HTTP/2 Web Server with Automatic HTTPS - <AUTHOR> https://caddyserver.com/ <comment><USER2> Has there been a shift in the recent times regarding developer teams paying for software? Yes, ease-of-use is great benefit. It takes a while to become accustomed to nginx configuration. But, $25/instance/month for a web server when nginx can do almost everything (and arguably much better than a server that was only launched two years ago)? It doesn't sit with me well.

<comment><USER3> It's open source, right? Just compile it yourself?

To your lager point, I think paying for software is and has been the norm in many places. So nothing new.

<comment><USER2> I didn't notice that you can compile the source code yourself to avoid paying for the license. That seems to be a nice alternative.

<comment><USER4> Only if you're not using it for commercial stuff. If you're using it for commercial stuff you have to pay the $25/month

<comment><USER5> My understanding of their license page is different.

"If your company uses official Caddy binaries internally, in production, or distributes Caddy, a commercial license is required."

"If I build Caddy from source, which license applies?" "The source code is Apache 2.0 licensed."

[https://caddyserver.com/products/licenses](https://caddyserver.com/products/licenses)

<comment><USER6> Would be interested to see how reproducible builds would interact in such a situation.

I.e. you get a byte-identical binary file if you build it yourself, but you don't have to pay the $25/mo. Even though you end up at exactly the same place.

Something like [http://ansuz.sooke.bc.ca/entry/23](http://ansuz.sooke.bc.ca/entry/23) "What colour are your bits?" I suppose.

</comment></comment></comment></comment></comment></comment><comment><USER7> And it still doesn't obey the DNS specs.

Specifically, the DNS RFCs define that, given no search domain, a relative hostname (e.g. google.com) is equivalent to its absolute hostname form (e.g. google.com.).

This is used in SSL validation as well, a certificate valid for one is valid for the other, and in reverse.

Every webserver SHOULD respond to both names identically, or redirect from one to the other.

Let's try that. [https://news.ycombinator.com./](https://news.ycombinator.com./) yup, works. [https://www.nginx.com./](https://www.nginx.com./) as well. In fact, nginx, Apache2, IIS, the Google Cloud load balancers, AWS's load balancers, every major site supports this.

The only http servers breaking the RFC for absolutely no reason are Caddy and Traefik.

And the issue has been closed as WONTFIX months ago.

Do you really want to use an http server that doesn't even consider following the RFCs? And pay for that?

<comment><USER8> You make it sound like the Caddy folks completely ignored the issue without providing a reason ("absolutely no reason", "doesn't even consider"). After looking into it, I find your post misleading and unfair, to say the _least_.

Link to the issue for other users who, like me, were concerned by your post:

[https://github.com/mholt/caddy/issues/1632](https://github.com/mholt/caddy/issues/1632)

According to the author, the issue is that (a) there are two RFCs that contradict each other on this distinction, and (b) most browsers do not treat them as such for the purpose of their same-origin policies. So he thinks it's best not to enable this alternate URL by default since it's trivial to add it as another route if you want.

I don't know if he's right or wrong, nevertheless, this clearly isn't a dude that doesn't give a shit.

<comment><USER9> I don't see two conflicting rfcs. Can you point them out? I saw a link to a Mozilla thread discussing how they should do normalization, but nothing about not supporting the fully qualified format. Cross origin rules don't apply for looking up a virtual host.

Also, the concerns over security the caddy developer make aren't clarified and probably don't exist.

No dot has 2 possible meanins and with dot has one _to the resolving application_. To the server resolving a virtual host there is only one for both.

<comment><USER8> > I don't see two conflicting rfcs. Can you point them out?

The quote is:

"RFC 1034 says the two domains are the same, but RFC 3986 says they are not. I can't tell from the http spec whether the two Host values are equivalent or not."

<comment><USER9> From rfc 3986:

> The rightmost domain label of a fully qualified domain name in DNS may be > followed by a single "." and should be if it is necessary to distinguish > between the complete domain name and some local domain.

When there is no ambiguity then they mean the same thing.

Anyway, as a sibling comment said, it's not a good rfc to use for this purpose.

</comment></comment></comment></comment></comment><comment><USER10> What's interesting is that it's the first web server not focused on speed or performance but on the user experience. I love nginx, but I had huge headaches configuring it when migrating php apps with huge htaccess rules to nginx. The terms of the license do not seem super clear though.

<comment><USER11> > I had huge headaches configuring it when migrating php apps with huge > htaccess rules to nginx

I also had this issue whenever I did migrations between different web applications that used different URL structures. To preserve my sanity, I now do the redirects in the application level.

I predict nginx will get a lot of competition from servers and applications written in Rust. Until then, it's viable as a reverse proxy and serves static files really great.

<comment><USER12> nginx has a huge headstart though, it's stable, it's fast, has lots of modules, etc.

I'm open to competition in the field, I'm not sure the competition will emerge as fast as people think (just like competition for other well-established software bricks in general).

<comment><USER13> and already supports HTTP/2 and HTTPS.

</comment></comment></comment></comment><comment><USER14> As someone who is very very new to hosting their own services, this was an absolute piece of cake to set up. My first time dealing with reverse proxies and it took less than an hour to get going.

<comment><USER15> Good to know! As someone who's been called a senior SRE (I hate titles) and has been building platforms for years, it's great to see high quality software enabling new comers like your self to the field. Welcome! :-)

<comment><USER16> For someone that hates titles, you both managed to call yourself an 'SRE', and drop the fact that you've been 'building platforms for years'.

Just sayin'.

<comment><USER17> I think it's always valid to point out in threads like this that everyone on the internet appears to be a domain expert. Everyone is also an entrepreneur, and anyone who is unemployed is still a consultant. It can make it more difficult to evaluate the merits of a piece of software like Caddy because you have to wade through all the "this is awesome!"s which are somehow considered valid contributions to discussion.

</comment></comment></comment></comment><comment><USER18> $25 a month is quite pricey for the edge case of someone starting up sites that will be commercial but unsure as to profitability yet. I guess I will build and donate $25 for the year. probably building is the smart thing to do anyway for a server you're getting for its security capabilities.

<comment><USER19> Agree. The $25/month stops me using it for side projects that generate small but non-zero revenue. Love the concept but not sure when I'll get to play time on something genuinely personal.

<comment><USER20> Can't you build it from source?

<comment><USER4> I'm sure he can build it from source, but the "personal" license forbids using it for commercial purposes, so the point is moot.

<comment><USER21> Those licences are only for the binaries. The source is Apache licenced and can be used for commercial purposes for free.

</comment></comment></comment></comment></comment><comment><USER6> They make it quite hard to find out, but here it is:

> Caddy obtains certificates for you automatically using Let's Encrypt.

Not sure why that is not stated front and centre. It's a good idea.

<comment><USER22> That's literally the first selling point in 128px font. [1]

It might not mention the implementation details but the concept is what matters.

[1] [https://i.imgur.com/hdEaKpG.jpg](https://i.imgur.com/hdEaKpG.jpg)

<comment><USER6> No, it's not. It doesn't explain how it works, which makes it come across as an empty marketing promise.

</comment></comment></comment><comment><USER23> A great alternative that we use in production for thousands of domains is [https://github.com/GUI/lua-resty-auto-ssl](https://github.com/GUI/lua-resty- auto-ssl)

<comment><USER24> Caddy can handle tens of thousands of domains. I know a couple of instances which do.

</comment></comment><comment><USER25> Previously (not comprehensive):

[https://news.ycombinator.com/item?id=12719563](https://news.ycombinator.com/item?id=12719563)

[https://news.ycombinator.com/item?id=9452606](https://news.ycombinator.com/item?id=9452606)

[https://news.ycombinator.com/item?id=11152761](https://news.ycombinator.com/item?id=11152761)

Related: [https://caddyserver.com/blog/caddy-0_11-telemetry.html](https://caddyserver.com/blog/caddy-0_11-telemetry.html)

</comment><comment><USER26> When Caddy was first released I tried, and failed, to get Caddy to serve an https site _locally_. Is that possible now? The docs[1] hint that it could be if I add an entry to my hosts file pointing an IP address of, say, a Docker container, as that wouldn't technically be localhost or an IP address. It doesn't explicitly say it's possible though. Adding something to the tutorials would be immensely helpful if it does work.

[1] [https://caddyserver.com/docs/automatic- https](https://caddyserver.com/docs/automatic-https)

<comment><USER27> What do you mean by locally? If you want HTTPS automatically then the site must be publically available so that LetsEncrypt can verify the domain and grant the certificate. If that's not possible then you'll have to use the DNS challenge and setup a provider plugin.

It doesn't matter where the backend points and you can use it to serve a docker container if you want, but that's different from the the host/frontend address you use.

<comment><USER26> Locally in the sense of a local development server. The issue is that there wouldn't be a real DNS record pointing at the machine (well, unless you added one to point at your external IP address, but that's a pain for teams). I guess if Let's Encrypt needs to verify the domain it won't be possible...

<comment><USER21> Let's encrypt can issue certs without needing a public route to your machine if you use the dns challenge. Here's what I do:

1\. Add a public A record (or host file) local.mydomain.tld to 127.0.0.1.

2\. Host my dns with cloudflare (other providers have plugins too), and install the caddy plugin to do the dns challenge for certs.

3\. Caddy can then get certs for local.mydomain.tld and serve them locally.

</comment></comment></comment></comment><comment><USER28> That's interesting! I wonder how it scales relative to, say, nginx.

<comment><USER29> It scales fine… if you don't already know the answer to that question, none of the web servers you choose are the bottle-neck to your site. ;-)

<comment><USER30> That's a bizarre statement. I could be incredibly familiar with nginx, apache, varnish, etc., and have a website that scales to a huge amount of users and still have no idea how well Caddy scales compared to nginx due to having no prior knowledge of Caddy's performance. Indeed, questions about scaling and performance are going to be some of the very first questions asked by anyone running such a site - that's going to be one of the single most important characteristics about it. (Hopefully after security....)

</comment></comment></comment><comment><USER13> I am immediately suspicious of this because HTTP/2 only operates over HTTPS. So for them to market this webserver as being special because it defaults to HTTP/2 over HTTPS is the sort of thing a snake oil vendor would do.

Also completely unfond of the license. "Caddy is amazing because it has 3 line config files!" So what? That's only appealing to people who are afraid of editing config files. Here's a harsh reality for the developers (who probably won't see this, ah well), but "config files" are not worth $25/mo or whatever the full scale commercial costs of this are. Do the developers think that their target audience are incapable of configuring traditional webservers?

Just because you pour your blood sweat and tears into a thing doesn't mean that thing is worth any money.

<comment><USER31> The idea was brilliant before the web changed and made him a bit obsolete.</comment></comment>


Debugging Through WebAssembly Is Impossible - <AUTHOR> https://erikmcclure.com/blog/debugging-through-webassembly-is-impossible/ <comment><USER2> I experienced similar frustration recently. I couldn't even generate debug symbols, as doing so will freeze computer (I was compiling llvm into wasm). I had to rely on logs.</comment>


Ask HN: When have you coded yourself out of a real-life problem?  - <AUTHOR>

What are some of the ways that you coded yourself out of a real-life problem?
For me, I wanted to go to Momofuku Ko, except they have an annoying .NET reservation system for their coveted 12 seats and 2 seatings, which you can only book 6 days in advance for 24 seats. You have seconds to enter your credit card number to confirm.
But it is a good restaurant, and reservations disappear fast; all are gone by 10:00 AM ET. So like any self-respecting hacker, I got TextMate and Ruby out, performed some sudo gem install mechanize magic, did some serious Regex/XQuery work, wrote a function to perform exponential smoothing on calculating the time difference from my clock to the Momofuku clock, and slept soundly at 7 AM Arizona time, knowing that my reservation would be booked (it's unfair I have to get up at 6:55 AM to check if my resy went through, just because I live in a state like Arizona where food is bad except for Pizzeria Bianco and some others!)
Hope to hear some great things from you guys!
Cheers, Jason
Post scriptum: Frank Bruni, former NYT food editor, hates the resy system:
NYT, Frank Bruni, "Going Ko-Ko," http://goo.gl/NYfj; NYT, Frank Bruni, "More Fun with Ko," http://goo.gl/gKGZ; NYT, Frank Bruni, "Ko-da," http://goo.gl/6rNc; Departures, "Eating Small in NY," http://departures.com/articles/eating-small-in-new-york <comment><USER2> I was going to Paris for the first time and I had been learning french, but my pronunciation was still terrible. I use a spaced repetition flashcard system [1] to learn new things and improve my recall, and I wanted to use it to learn some french pronunciation. About.com has a 2500-word french audio dictionary [2], so I spent a few hours building a scraper to pull down all the pages, parse out the words (their markup is crazy), download the mp3 files, and build an xml file so I could import it all into my SRS system.

Not a terribly challenging problem, but it's always nice to be able to quickly hack up some code to solve something.

1\. <http://www.mnemosyne-proj.org/>

2\. <http://french.about.com/od/vocabulary/a/audiodictionary.htm>

<comment><USER3> I use Mnemosyne quite a bit too.

I also have enjoyed writing scrapers for data I wanted but wasn't being provided in a downloadable format - most commonly stuff on forums.

I wrote a scraper that scraped the entire textfiles.org website for their texts (mostly for fun), I wrote a scraper that logged in as me into my favorite phpBB forum and scraped every post in every forum in every thread - the script would create a directory for the forum, a directory within that named by a number to denote which page it was on, a directory within that named after the thread, a directory within that named by a number to denote which page it was on, and a text file containing only the content div (to retain the formatting of the original post).

I used Python and Mechanize for most of this - the phpbb forum scraper was the most fun and rewarding.

I've also written page scrapers for websites that have things like "1324 of the best side boobs!" I didn't want to Save As for all of them! So I wrote a script that downloaded them for me (this one was easy though compared to the phpbb script).

</comment></comment><comment><USER4> This is more of a hack than coding, but I owed the school library money for a lost book in grade school. I didn't have the money and didn't want to to tell my parents about it, so I studied the library system.

It was a self-check in, check out system with a scanning wand. You take your book, log your student number, scan the barcode, and it was added to your account inventory. Returning the book was the same way; but if you didn't have the book, you obviously couldn't scan it back in. I figured out you could look up the book's ISBN through your account inventory, write it down, and manually enter it into the return form.

BAM! Lost book was magically returned :D. I told a few friends about it, I'm sure the school lost out on a few hundred dollars worth of books that year.

</comment><comment><USER5> At College they used 386s and 486s running DOS and Windows 3.1. At home I used an Amiga 1200. The two are not exactly compatible. I would routinely be given assignments, have to write them in QuickBasic for class and would have to spend half a day at college instead of at home. I didn't have a Microsoft Basic variant (other than GW-BASIC which wasn't quite compatible) and used AMOS instead to test the algorithm, then rewrite in QuickBasic. I tried submitting work in C but my lecturer would refuse to accept them as we hadn't covered that module yet.

In the end I wrote an AMOS -> QuickBasic converter in C, first for DICE C on Amiga, then tweaked for Turbo C on the PC. I could then do the rest of my coursework at home on my Amiga, instead of the grubby old slow 386s.

</comment><comment><USER6> I've written my own custom data recovery tools to recover lost files. The first was a hack of ddrescue to only recover sectors containing a certain string (because the drive had so many bad sectors that a full ddrescue would have taken weeks to complete). Recently I wrote a tool to recover deleted directory contents from ext file systems that works even if all the inodes are gone. I didn't need the file data, only the names and folder structure, and I realized that if you don't need the data you never need to dereference the inodes. Instead you can treat the inode _number_ like a database key to cross reference directory blocks.

</comment><comment><USER7> I realized I had lost an important file (a letter of recommendation from my university co-op placement). I no longer had the e-mail it was attached to (this is prior to me using webmail), nor the e-mail program. But what's this? I still have the mailbox files. Well, I know python has some multipart utilities... maybe I can copy and paste the relevant parts out of this message and into a script and have it write out the attachment sub-message. After a bit of fiddling, it worked!

I was elated. It was by no means a very challenging thing to do, but I would have been totally fucked if I didn't know how to do software development.

</comment><comment><USER8> There is a particular database system used by newspapers that encodes the date in base64 in a query parameter. You could search and find the dates, but you needed a subscription to access the links. It was setup so if you had the link you could link others to it free of charge.

Well, the guy at the local library with subscription started to get irritated with me after the third or fourth link so I built my own search engine to use their search engine. I was a research assistant at the time.

</comment><comment><USER9> In 10th grade, we had some security software on our computers in high school called 'Fortress' (circa 1998). I had a multi-media project due for a class that I never did. I compiled eofspresentation.exe in VB that did one thing:

Popped up a dialog, "Cannot run due to security"

<comment><USER10> Reminds me of grabbing a random .dll, renaming it to .doc, putting it on a floppy disk and handing it in, all to gain myself a few extra days to finish my assignment.

<comment><USER11> I did a smililar thing for my final year mid year thesis review - took a pdf, opened it up in wordpad (!), cut out chunks of it, saved it, and submitted it. Got 2 weeks extra, nabbed an A myself.

... not sure if I should have said this in public, but I remember being strangely proud for having bent the system a bit :)

</comment></comment></comment><comment><USER12> I hated powerpoint (and still mostly do), so I ended up coding a presentation in BlitzMax (a version of Basic designed for cross platform game development) in order to make a presentation for my geometry class sophomore year. I had all different shapes displayed on screen and transformed them around and such to show off different proofs.

</comment><comment><USER13> Problem: born in hungary. Solution: during high school, I've complemented my english courses using the Internet (reading RFCs, and white papers). After university, I've literally hacked myself out of the country, by doing consulting on the side, and saving every penny I could.

Currently in London, and content, if not happy.

</comment><comment><USER14> Back in 7th grade, I wanted to know everything about my high school crush, so I phished her email password via email link baits (followed by fake Hotmail error pages). Worked so well that I started phishing the entire school's passwords.</comment>


Billionaire investor Peter Thiel: 'Always aim for a monopoly' - <AUTHOR> http://uk.businessinsider.com/peter-thiel-talk-in-london-on-business-and-politics-2015-4

<comment><USER2> Spoken from a position of power. Not practical (or indeed useful) advice for most people.</comment>


Tokyo Doubles Estimate for Total Radiation Release In 1st Week After Quak - <AUTHOR> http://online.wsj.com/article/BT-CO-20110606-701246.html

<comment><USER2> There is way too much questionable information at this point and conspiracy theories abound.

I have no clue what to believe other than "Holy shit - it is bad"

What I find interesting is that there have been higher than normal storm activities between .JP and CA --- I personally like the conspiracy that some are promoting that HAARP is being used to create precipitation to drop the radiation into the ocean rather than let it get to the US coast.

Sure, far fetched - but sounds awesome. Although, it goes against the other conspiracy theories that US used HAARP created the quake in the first place in order to shore up the strength of the fiat US dollar as the double dip 2011 comes to hit hard....

If anything, I like living in science fiction.</comment>


Ask HN: Do you like the new HackerNews Design? - <AUTHOR>

What do you think of the new HackerNews bar? With green for up-trending articles and red for down-trending? <comment><USER2> Remarkable how evenly distributed the up and down trending articles are...

<comment><USER3> There must be some kind of psychological correlation to attract up-votes when in an even ranked position...

<comment><USER4> It is not there anymore, maybe there was a bug?

</comment></comment></comment><comment><USER5> Oops I thought it was a xmas theme ;)

</comment><comment><USER6> I see a red bar near the footer but nothing else looks different

</comment><comment><USER7> I noticed it on my tablet and thought it had something to do with Christmas. It isn't loading on my laptop, so I assumed it is a simple CSS change for Christmas.

</comment><comment><USER8> I thought they were simply a choice of colors! And I like the new design!</comment>


Twitter Gives Developers 6 Months To "Properly" Display Tweet - <AUTHOR> http://techcrunch.com/2012/08/16/twitter-gives-developers-6-months-to-properly-display-tweet-use-new-authentication-and-rate-limts/

<comment><USER2> It looks like Twitter won't be happy until the only Twitter developers are the ones that work at Twitter.

</comment><comment><USER3> To play a bit of Devil's advocate... Most of the popular apps fall under the free API access, don't they? If so, isn't it unfair for everyone to say how users need to jump ship and find an alternative? I mean, Twitter is giving you API access for one of the most utilized services on the Internet... for free. You just have to make sure your app follows some guidelines.

Unless I've misinterpreted this?

<comment><USER2> People aren't building apps for fun. They're trying to build a business. No one in their right mind is going to invest in a business that's built in an area that can be bulldozed at any moment, and that's what twitter's platform is.

For developers, twitter is now officially a burning platform.

<comment><USER3> That's true of any API, though. You don't build your entire business around another company's API without a contract.

And yes, people are building apps for fun. Programming isn't just a job.

</comment></comment></comment><comment><USER4> Can you imagine if any other content provider or medium had this restriction? Or even worse, imagine how Twitter's growth would not have occured if the developers were intially subject to these restrictions...

<comment><USER5> Craig's List comes to mind. They're doing fine.

</comment></comment><comment><USER6> Twitter seems to be taking themselves too seriously. Guidelines for displaying a tweet? Come on.

<comment><USER7> How are they expecting to enforce that? Are they really going to download and test every third-party app?

<comment><USER8> Require screenshot submissions, hire a bunch of temps, and they're all set.

<comment><USER9> ... or use their recently open-sourced mechanical turk library.

</comment></comment></comment></comment><comment><USER10> It certainly is time for an alternative, isn't it.

<comment><USER11> Hence Dalton Caldwell's "Audacious Proposal" and the birth of App.net.

<https://join.app.net/>

<comment><USER12> So we have a walled garden network fucking over its users, and you think the solution is to replace it with another walled garden because its rulers are supposedly more benevolent? That's not very ambitious. Think bigger. StatusNet/Identica is free, open source, and (most importantly) federated.

<comment><USER13> A walled garden lets anyone in, but not out. You're thinking of something more along the lines of a gated community.

<comment><USER12> No, I'm thinking of a walled garden. <https://www.google.com/search?q=walled+garden+twitter>

<comment><USER13> Ah, appears we're both correct: <http://en.wikipedia.org/wiki/Walled_garden_(technology)>

"... refers to a carrier or service provider's control over applications, content, and media on platforms..."

"More generally ... refers to a closed or exclusive set of information services provided for users"

<comment><USER14> No, i actually think just he's correct. It's OK to be wrong though, so no big deal.

Though I do disagree with his sentiment. I think App.Net is a perfectly good alternative to Twitter and every attempt at open, federated services since the original one (eg www) have failed to catch-on.

<comment><USER15> www was not the original federated service. E-mail is still around and quite successful.

<comment><USER16> Usenet is dead now, but worked for quite some time.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER17> How will they handle console twitter apps (ttyter and bitlbee for example) where formatting can't happen to match their requirements?

<comment><USER18> The authors of these requirements don't seem to know or care that the World- Wide Web was never intended to mandate one visual-only rendering of your content, and in fact doesn't even do that very effectively. Most of them are pretty reasonable semantics (e.g., @user links to their profile) but as soon as you mention "lines" or "icons" or "top right" you're Doing It Wrong, and no affordances using other services is just blatantly anti-competitive.

<comment><USER19> … not to mention the issue of accessibility. Is Twitter going to block access for visually impaired users by enforcing the new rules?

Tweetie (now Twitter app) at least is a horrible app from an accessibility point of view and could only improve …</comment></comment></comment>


NVIDIA Introduces dual Cortex A9 based Tegra 2  - <AUTHOR> http://www.anandtech.com/gadgets/showdoc.aspx?i=3714

<comment><AUTHOR> I can't help but be very excited about these Cortex A9 boards coming out. The ARM processors are getting better really fast. The power / performance ratio compared to the Atoms is just so good, it's almost ridiculous.

A few years ago we had the iPhone, then the 3GS came with the Cortex A8 processor and the speed improvement is enormous. The Cortex A9 has an even better performance / Hertz ratio. Anand expects a similar (iPhone -> 3GS) speedup is possible with the Cortex A9.

The Tegra has 2 cores, but even more cores are possible. According to Anand, the chips can also scale to beyond 2GHz on a 28nm design. There are many chip producers creating chips on the Cortex A9 design, there is enough choice and difference between chips to create decent competition. With the way the ARM designs are licensed, silicon bakers can add whatever functionality they want, targetting smartphones, tablets, netbooks, whatever.

Most companies have chip samples and development boards ready by now. I can't wait to see what the A9 will bring us in the next 2 years.</comment>