Google Is Eating Our Mail - <AUTHOR> https://www.tablix.org/~avian/blog/archives/2019/04/google_is_eating_our_mail/ <comment><USER2> Google has no incentive to fix these kinds of problems.

It's big enough that when someone complains that a message sent wasn't received, the intended recipient will say, "I never have problems with my Gmail account. It must be you." And the sender has to switch to Gmail to reliably communicate with the outside world.

I wish this was just paranoia, but we've seen multiple discussions on HN about Google programs and policies that alter the internet in ways that only benefit Big G. It's like we're heading back to the days when people didn't know the difference between AOL and "the internet."

<comment><USER3> I'm one of the PMs for Gmail and hang around HN quite a bit. This is my personal take, not an official reply.

It's simply not true we have no incentive to fix this. Here are a few:

Firstly, Gmail's success is entirely predicated on the health of the global email ecosystem. Gmail does not, inherently, have any network effects (unlike FB, messengers, any other comms tool of Gmail's scale). Email itself, of course has a huge network effect, and _that_ is because you can email anyone in the world, regardless of what email system they use. It's because email is open. If we lose an open, healthy ecosystem with many providers, we'll destroy the base we stand on.

Secondly, we care deeply about having positive relationships with developers and all our users. I can tell you it definitely makes me sad to see articles like this. There are going to be false positives, we will make mistakes, but we certainly care a lot about fixing issues like this when we hear about them.

I agree Postmaster tools has been underinvested in and we could do much better there.

<comment><USER4> I appreciate the response. From someone who has operated a mail server since 2002, it comes across that Gmail does not care about cooperating with small but legitimate mail servers.

SPF, DKIM, reverse DNS, no blacklists, no open relay, longtime ownership of IPs, etc etc. Using various mail testers returns a 10/10 deliverability score.

And yet, messages sent to Gmail always go into the spam folder, or are never delivered at all. These are everyday regular messages, I have never used mailing lists or sent bulk automated messages.

The issue is, there is no recourse, no fix, no acknowledgement of the problem with false positives. There is no tool available to me to understand or correct the "problem". Hint: this comes across as Gmail not giving a shit.

Gmail has a responsibility to be more accountable, even if these problems are unintentional, because Gmail is such an enormous node in a federated network.

> If we lose an open, healthy ecosystem with many providers, we'll destroy the > base we stand on.

Correct. Gmail is contributing to the erosion of email reliability. Please course correct.

<comment><USER5> I've had gmail send the following to the spam folder:

\- Legitimate class action notices related to Amazon purchases.

\- Email coming from addresses to which I had already sent email. (!)

\- Email from my landlord.

\- Email coming from Google itself.

Based on the contents of my spam folder, which I have to check fairly often because of the extreme overaggressiveness, I would be vastly better off if nothing ever got filtered at all. [1]

>> There are going to be false positives, we will make mistakes, but we certainly care a lot about fixing issues like this when we hear about them.

This doesn't sound honest, or at least not complete. People have been complaining about this for years. I have personally been complaining about this for years. The loss of obviously legitimate email is completely outrageous.

It doesn't look intentional (look at that fourth category!), but it certainly doesn't look like anyone is trying to address the problem.

[1] Yes, if spam filtering was disabled, more spam might get sent.

<comment><USER6> Your comment motivated me to check my Gmail spam folder and 10% (3 out of 30) were false positives, two of which were pretty important. :(

<comment><USER7> I've recently started checking quite regularly my spam folder as I've noticed more and more legit emails ending up there. One of them being support emails from TradeMe (one of New Zealand's biggest sites), keep getting put into spam, even after multiple "mark as not spam", along with some other kinda important emails from TradeMe. I've had to put in manual filters to force an email from TM to skip the spam.

<comment><USER8> And yet, in my own none Gmail hosted email (fastmail), I currently have 2/55 false positive spam emails. I very rarely ever check it. To note I usually get one actual spam (non newsletter blog spam) to my actual inbox, a month.

</comment></comment></comment></comment></comment></comment></comment><comment><USER9> > I'm sure there is no malicious intent behind this

How can anyone be sure of this? This is only one of Google's practices that seems to follow a pervasive pattern of eroding open internet standards while presenting Google's own proprietary implementation as somehow superior. Eventually, the open standard loses all meaning because the most popular implementation does not actually adhere to it. Meanwhile, Google reaps enormous benefits in the form of additional signals for its advertising business. How can this not be grounded in malicious intent?

> and that there are some very smart people working on spam prevention at > Google.

There are some very smart people working on advertising at Google. The rush to forget the primary nature of Google - it's an adtech firm - is why they've been allowed to skate for so long. Gmail's spam filtering is just a pretext for passing all email through a machine learning system. Sure, one possible signal emitted by that system is whether a message is spam or not. Perhaps this determination is conflated with wether the message is useful for ad targeting: after all, when viewed from Google's own perspective certain e-mail messages contain no information which can be used for ad targeting, so they must be spam. The user's interests are clearly secondary to this.

So, back to the "smart people" working on this: at what point do we begin judging engineers for working at Google? There's a lot of highly vitriolic criticism that emanates from Google's workforce on a variety of subjects, but how many of them would actually pull the pin and leave their employer? I don't have any statistics to offer, but it seems to me that we still have a ways to go before Google has become completely drained of engineering mindshare.

<comment><USER10> Email in Gmail isn't used for ad targeting, so this line of thinking doesn't follow well.

<comment><USER11> Up until recently it was used for ad targeting.

</comment></comment></comment><comment><USER12> I'm also one of those oldfarts that insists on hosting my private emails for me and a few friends. I've owned the ip since early 2000's and have had zero "spam"/blacklist incidents. One trend I've been noticing is that Microsoft/Outlook have regularly started to block me. I have to contact their abuse which takes a day or so to get unblocked. To date, I've done so >10 times:

       hotmail-com.olc.protection.outlook.com[126.96.36.199] said: 550 5.7.1
       Unfortunately, messages from [<redacted>] weren't sent. Please contact
       your Internet service provider since part of their network is on our block
       list (S3150). You can also refer your provider to
       http://mail.live.com/mail/troubleshooting.aspx#errors.
       [VE1EUR01FT028.eop-EUR01.prod.protection.outlook.com] (in reply to MAIL
       FROM command)
       Reporting-MTA: dns; <redacted>
       X-Postfix-Queue-ID: 14A41FEB66
       X-Postfix-Sender: rfc822; <redacted>
       Arrival-Date: Thu,  14 Mar 2019 14:07:42 +0200 (CEST)

<comment><USER13> At >10 times I'd automate that process.

<comment><USER12> Form has captcha: [https://support.microsoft.com/en- us/supportrequestform/8ad56...](https://support.microsoft.com/en- us/supportrequestform/8ad563e3-288e-2a61-8122-3ba03d6b8d75)

<comment><USER14> Is it illegal to pay for one of the captcha solvers? Ones like this are the easiest, captcha solves cost something like 4 cents each, and all you do is send the image to an API, which responds with the solved captcha.

</comment></comment></comment></comment><comment><USER15> As someone who runs their own personal mail server, this annoys me to no end.

I set up DKIM, SPF, and reverse-DNS records and resented every moment of it. Even after all that, there's some chance that an email from my server will be marked as insecure/spam or otherwise just not be delivered because Google has come up with some new brilliant mail security/auth/permission scheme that the world has to adopt tomorrow or be cut off from all Gmail users.

<comment><USER16> >Even after all that, there's some chance that an email from my server will be marked as insecure/spam

Don't I know it. This happened to us but not with Google. We use Exchange Online (Office365) and have DKIM, SPF headers configured. At some point our emails stopped being received by bell and cox emails addresses ... sporadically but sometimes for days at a time. It turned out that the spam filter provider (used by bell and maybe cox) correlated one innocuous phrase in our confidentiality notice (which is appended to all outgoing email signatures) with spam and therefore marked the entire email as spam as well.

<comment><USER17> To be honest I don't like this confidentiality notes and from what I've read (not a lawyer) they don't have any practical effect either.

<comment><USER2> I always giggle when I see those.

Especially the part about how "if you are not the intended recipient, you are ordered to destroy all copies of this message."

It's my computer. You sent the bad message. You're in no position to "order" me to do anything.

I suspect they're mandated by self-important Lower Middle Manager of the Year Award nominees, not actual legal departments. The sort of people who never do anything but push papers around, yet put a bunch of meaningless initials after their name in e-mail signatures because they went to a conference.

<comment><USER18> > Especially the part about how "if you are not the intended recipient, you > are ordered to destroy all copies of this message." It's my computer. You > sent the bad message. You're in no position to "order" me to do anything.

Best case scenario, someone listens and destroys the message.

Worst case scenario, the company is no worse off than if they hadn't included the disclaimer.

Makes sense from my perspective.

</comment></comment></comment></comment></comment><comment><USER19> _> I've been running a small SMTP and IMAP mail server for [...] around 15 years [...] I have SPF records and DKIM message signing setup on the domains I use. The server is hosted on commercial static IP space (with the very same IP it first went on-line) and I've made sure with the ISP that correct reverse DNS records are in place. [...] Being a good administrator and a well-behaved player on the network is no longer enough [...] Now every time I write a mail I wonder whether Google's AI will let it through or not. [...] So far Google has blocked personal messages to friends and family in multiple languages, as well as business mail. I stopped guessing what text their algorithms deem suspicious._

Yep, I know the author's frustration very well. I made a previous comment[0] trying to warn others of personal email servers' _outgoing email_ being spam- holed -- and yet some of the replies still argued I was overstating the difficulties.

Everybody's risk tolerance is different. Personally, I just don't have the bandwidth to administer my own private email server and constantly worry if recipients are receiving my emails.

[0] [https://news.ycombinator.com/item?id=15526089](https://news.ycombinator.com/item?id=15526089)

</comment><comment><USER2> The last time a company screwed with e-mail at scale was Earthlink.

Remember for a couple of years, every time you'd send a message to someone at Earthlink, you'd get an automated rely demanding that you verify yourself before the message would go through?

Now I can't remember the last time I saw an @earthlink.* e-mail address.

Google has apparently learned from that and put the "error" in the 550 messages, where they can't be seen by the end user, and lead to non-helpful resolutions for sysadmins.

The result is that the blame for missing messages goes to the sender, not to the recipient's email service.

If Gmail at least notified the sender that there was a problem, then a pattern of responsibility could be established. But this is just another dark pattern.

<comment><USER20> If enough expected messages don't show up, users are going to start thinking gmail is broken. The problem for Google is that they are not the only trusted brand consumers interact with. I've seen a lot of stuff start ending up in spam that shouldn't be there. If enough password resets, customer service responses, and order confirmations aren't delivered, it looks like Google has a problem. If it doesn't even make it to the spam box it makes Google look really bad.

Google has definitely been changing things, everyone presumes it's some rogue ML system. My company had solid Gmail delivery for almost 10 years, and at a fairly large volume. Now we can't get it delivered any more. Very close to showing users a message not to use Gmail.

<comment><USER21> > If enough expected messages don't show up, users are going to start thinking > gmail is broken.

I doubt that. They will rather consider that the people (or their IT) that send them emails made a mistake. It's Google after all.

<comment><USER22> If several different IT departments all start sending that message though, people start to think maybe google has lost it...

<comment><USER2> _If several different IT departments all start sending that message though, people start to think maybe google has lost it..._

Many IT departments still build their networks tailored to Windows machines, to the exclusion of others. Inertia is hard to overcome.

</comment></comment></comment></comment></comment><comment><USER23> I run a mail monitoring service [0], and we hear this complaint every now and then.

A couple of things that regularly seems to trigger false positives in spam algorithms:

\- no or misconfigured SPF and/or DKIM

\- no or misconfigured reverse-DNS

\- automatically included footer texts (confidentiality, copyright, safe a tree don't print, etc)

\- regular automatic replies from the domain (such as out of office notifications)

\- the use of embedded images (logos, human signatures, etc)

We sometimes joke that these triggers were built in by the algorithm developers as a means of punishing those who litter their email with pointless texts and images.

[0] [https://www.mailhardener.com](https://www.mailhardener.com)

<comment><USER24> You can satisfy all of this, have an IP with an absolutely spotless reputation that you have personally controlled for years that has never sent spam, and still be arbitrarily shitlisted with zero recourse whatsoever.

<comment><USER23> IP addresses are really insignificant in spam detection because spammers can easily switch to a new IP address.

Like with Google's search algorithm, the spam detection algorithms are secret and evolve constantly. So I'm not claiming I know how the algorithm works, but it wouldn't surprise me at all if IP addresses aren't even used anymore in Google's spam detection.

So no, you are not 'shitlisted' as you called it. There are no lists, there are only algorithms.

<comment><USER24> Whatever you care to call it is irrelevant, you jump through all the hoops, and your mail dosen't get delivered.

</comment></comment></comment></comment><comment><USER25> Same here. I love email for a variety of reasons, one of them for being an open standard and protocol.

So, naturally, as many of you, I went the mail/postfix, DKIM, SPF, etc way. And all is fine until you start receiving random hard bounces with no real debugable answer for Google.

It got me deeply sad and questioning my decisions: since you can't really ignore Gmail, email isn't in practice "open" anymore. So I might as well sign up for Facebook, WhatsApp and the likes. It's been years and I haven't yet, but it's getting harder and harder.

</comment><comment><USER26> The real issue here is that if you tell some gmail user to contact their e-mail provider to resolve delivery issues on their side, like this, they'll just stare at you, not knowing what you're even talking about.

It's like there's an assumption that gmail is perfect, and the problem is with the sender. Even if that was true, a normal mail hosting company would at least tell its customer why the mail is not being delivered, so that the customer can tell the sender what to fix.

The gmail recipient is never exposed to this side of google. So they don't know what a nightmare comapny it is to communicate with.

Why should everyone and their dog be solving gmail users's problems with receiving messages? It's such a demented system. It should be the other way round. Recipients, via their provider should be solving their issues with spam filtering and blocking.

If the gmail user would be blocked by my mail server, I would not tell them to go guess what's wrong, fix gmail, and to have fun. It should not be acceptable the other way round either.

</comment><comment><USER27> I'm in the same situation as many people here. Started as a small personal email projects many years ago just to learn. I really enjoyed it, reading all the specs and making it always better year after year. Dmarc felt like a total achievement.

I started to host emails to many friends, small businesses and even a SaaS I developed. The subscription needs an email validation and I'm aware that the activation email ends up in the Spam folder for the new customers using Google emails. This activation email has everything from dkim, spf, dmarc, to unsubscribe link, full physical address of the business, etc and still I can't hit a good enough score.

I was thinking to start using Google service to send the activation link and hosting my personal domains, but seeing that I am not alone, I will continue to improve my little email projects.

Thanks all for cheering me up on this. I'm sure we can come up with a solution and I would be happy to help. When do we start?

</comment><comment><USER28> I have logwatch logs that I email to my gmail address via Google's own SMTP servers using my own Google credentials for authentication.

Hilariously, Google will flag these emails sent to myself using my own credentials and their infrastructure as spam. I have no faith in them ever getting this right.

</comment><comment><USER29> Google seems to not agree DKIM is setup properly?

[https://toolbox.googleapps.com/apps/checkmx/check?domain=tab...](https://toolbox.googleapps.com/apps/checkmx/check?domain=tablix.org)

<comment><USER30> Thanks for the link. I think the tool you linked specifically checks whether DKIM is set to allow Google's servers to send mail on the behalf of your domain.

These checkers for example don't find any errors with the DNS record:

[https://dmarcian.com/dkim- inspector/?domain=tablix.org&selec...](https://dmarcian.com/dkim- inspector/?domain=tablix.org&selector=alpha)

[https://mxtoolbox.com/SuperTool.aspx?action=dkim%3atablix.or...](https://mxtoolbox.com/SuperTool.aspx?action=dkim%3atablix.org%3aalpha&run=toolpage)

One thing that this second one does find is the lack of a version number in the record. I see that the RFC [1] got updated since the last time I checked and having that is now recommended instead of optional. I'll add that. Thanks!

EDIT: after adding version record, Google's tool now shows green checkmark for DKIM as well.

In any case, it's impossible to test a DKIM setup just by looking at the DNS records. The true test is to verify the actual signature in the message. Last time I checked, the mail on the receiving end (Gmail), did have "dkim=pass" in the Authentication-Results headers.

[1]: [https://tools.ietf.org/html/rfc6376#section-3.6.1](https://tools.ietf.org/html/rfc6376#section-3.6.1)

<comment><USER31> You may also want to add a DMARC policy. A dmarc record generator can be found at [https://mxtoolbox.com/DMARCRecordGenerator.aspx](https://mxtoolbox.com/DMARCRecordGenerator.aspx)

</comment></comment></comment><comment><USER32> Recently Google has bounced _replies_ to emails from a gmail.com address to us, with a message about it being "suspected unsolicted email" (for a reply???). That's just egregious (we're on a dedicated server at a reputable supplier, same IP for years, proper email processess, not on any blacklist and only low-volume usage - certainly no spam).

<comment><USER33> Yup, I literally lost jobs because of responses to enquiries hitting GMail's spam trap, before finally giving up and moving from self-hosting to Fastmail.

</comment></comment><comment><USER34> This also mirrors my experience, as I host my own email and it's solely for me. At one point, I couldn't successfully send any email to gmail, but then it changed for some reason and simply gets sent to spam folders instead. Perhaps it's entirely related to some people emailing me first that it gets through at all.

I don't send much mail to gmail, though. Sans that, my only issue has been a mail server that uses Reverse DNS, which I don't have set up, and entirely ignores my email without it.

I suppose I can understand this if some people get a great deal of spam, but requiring so much of this on an unencrypted message seems more like useless reassurances than anything. I'm not criticizing email for being unencrypted, but this seems more like another hoop to jump through than anything.

Also of note, almost all of the spam I receive is from gmail addresses and I wouldn't be surprised if the invalid addresses that send messages demanding bitcoin are also from gmail, but with fake From fields.

</comment><comment><USER35> I'll just chip in with everyone else who is also self-hosting:

I'm in a similar boat; been running email servers since before GMail existed. My personal one I've been running out of a home server closet since 2001. I've also done everything I can to guarantee I'm not running an open relay and not sending email unsolicited. Have been mostly lucky so far, but occasionally I will have people on mailing lists I manage (people I have met IRL and put them on the list to organize group meetings IRL) not get email. Used to be other stupid mail providers (AOL comes to mind), but these days it appears to be Google, sometimes.

I've had this domain nearly twenty years and run email on it for that same amount of time. I'm not going to "just switch", especially to a privacy invading ad-spewing "alternative" that doesn't give me as much control. Fix your damn servers, Google.

<comment><USER36> Google seems to have entered terminal senility. In my circle of friends googles search engine is pretty much useless, they have killed all the useful applications we used and now this.

I can't help but think that we are seeing the google transition to late 90s microsoft now.

<comment><USER37> What do you mean by "In my circle of friends googles search engine is pretty much useless"

<comment><USER36> I mean that artists are unironically using bing as a better alternative for image searches, academics have gone back to tracking citations + libgen/scihub to find information and programmers are using duck duck go to debug their code.

This is a huge problem since these people used to be the core of googles search user base.

</comment></comment></comment></comment><comment><USER38> I have run a private mailserver for over 20 years, and I have the same problem. There's no telling what Google will let through. I have a second account at a medium-sized email provider, basically paying them for access to the email deliverability cartel.

<comment><USER39> We need a new message protocol

<comment><USER40> I fear this will lead to the same problem: If it's interesting enough for Google, they will offer a free service, gobble up a large number of users, at which point they can hold the new protocol hostage, add their own extensions, and everyone else must follow.

<comment><USER38> It's called XMPP, and Google ran a public gateway federating for Google Talk. Basically nobody used it, and what little use it did see was mostly third parties sending spam to Google users over the gateway, so they shut it down.

Centralized systems seem to give users what they want. This is why iMessage, FB Messenger, WhatsApp, Snapchat, and Instagram are huge, and email is dying (and XMPP failed to get any real traction in the first place).

</comment></comment></comment></comment><comment><USER41> Add another perspective to this. Once I updated my phone, and google would not accept new google Authenticator codes after set up. Locked out of 15 years of email.

<comment><USER42> I had a fright last time I changed phones. After restoring my backup onto the new phone, I was about to wipe the old one; I did a final pass through my apps just in case... and saw that Google Authenticator configuration did not carry over to the new phone (which makes sens from a security perspective but is a PITA to manage). I had to re-enroll my device with all the services on which I use MFA.

After this episode I made damn sure I had recovery codes stored in a safe place.

<comment><USER43> When a family member had a phone stolen not long ago, the only major functionality on it that they couldn't quickly and securely disable was the Google stuff.

The procedures for doing so seemed to be unnecessarily complicated and difficult to find when starting, ironically, from a Google search on another device.

Worse, the security policies seemed to be fundamentally flawed, because they kept insisting on some form of authentication based on a trusted device _when the purpose of the transaction was to notify them that the trusted device had been stolen_.

There has been an unhealthy trend recently of assuming that everyone has a mobile phone and that communications to that phone/number are a good method of authentication, without adequate thought to what happens if the physical device and/or the associated phone number are compromised, or to whether protocols like SMS are really suitable for this sort of application. And some of the really important things, like banks and government services and email providers (which are in practice a gateway to everything else you do online) are often among the worst offenders. I don't know what to do about this, but certainly raising awareness of this kind of problem would be a good start.

<comment><USER44> > Worse, the security policies seemed to be fundamentally flawed, because they > kept insisting on some form of authentication based on a trusted device when > the purpose of the transaction was to notify them that the trusted device > had been stolen.

Are you suggesting any random person without any authentication proof to be able to just sign people out of their devices ? That would be a broken security.

<comment><USER43> I'm suggesting that a security policy should actually be practical. There are any number of viable ways to handle this. Requiring someone to possess the device they are reporting stolen is not one of them.

</comment></comment></comment></comment></comment><comment><USER45> Add me to the victim list. I've been running my own mail server for 13 years and have to tell everyone I'm intending to email to check their spam as that's probably where my mail will be. I send a handful of mails a week probably. Almost always goes to the bin if it goes to gmail.

<comment><USER46> Same here: I have to text my friends to tell them I sent them an email and check their spam. Nonsense.

</comment></comment><comment><USER47> Lest anyone think Gmail is less susceptible to problems because of the sheer volume of mail they handle (so any problem gets a lot of 'eyes' on it), they had an amusing and long-standing bug that meant any email containing a link to any domain name starting with "0x" automatically went to spam.

You could literally have two long standing, legitimately used accounts send an email to each other containing a link to a URL like [http://0xANYTHINGHERE.com/](http://0xANYTHINGHERE.com/) and it would be insta-spammed. I suspect it was a hard coded rule to avoid people using "long IP" URLs to circumvent other filters.. except there are lots of legitimate 0x domains that _aren 't_ long IPs.

It was fixed sometime in the past year but I got a lot of use out of it in talks I've given about email deliverability over the years.

<comment><USER48> What is your favorite public 0x domain?

<comment><USER47> [https://0x46.net/](https://0x46.net/)

</comment></comment></comment><comment><USER49> I run a highly customised PHPBB forum.

Without these customisations that forum would be overrun with all sorts of spam.

However, these customisations only stop spam postings but can't stop actual registrations.

Based on the users that I see who are registering I see a great majority of these spammers love using Gmail accounts.

So while it is good that Google Gmail is trying to fix these spam issues, from where I stand Gmail users seem to be a big part of the spamming problem.

Spammers love Gmail only because they can easily create spamming e-mail accounts.

</comment><comment><USER50> Google? Any large organization. I help a small non-profit run their site in my free time, and Google's actually been the nicest about our oldschool forwarders and Mailman lists. Despite being on no public blacklists, AT&T domains drop us at the border, with the only appeal process being via email, and I've never heard back. AOL and Yahoo? Flip a coin.

Of course, email forwarding turns out to suck, but we're just going to suck it up and move to G Suite for organizational email addresses and let folks forward from there. E-lists, OTOH, I haven't found a good integration to automate membership in the organization vs. G-Suite; perhaps it's time to just move to a forum.

<comment><USER51> FYI Google offers free GSuite basic for nonprofits:

[https://www.google.com/nonprofits/offerings/apps-for- nonprof...](https://www.google.com/nonprofits/offerings/apps-for- nonprofits.html)

<comment><USER50> Indeed, the free GSuite is the plan.

</comment></comment></comment><comment><USER52> A while back I'd explored trying to own my data, especially with email, and found that the efforts involved in hosting your own email server were tantamount to a full-time job.

The amount of fighting you have to do to stay on everyone's whitelists is absurd.

<comment><USER53> There's a good middlespace here, where you pay someone for email service who isn't an adtech company. Sure, you don't host it yourself, but someone who's not selling you is and you're their actual customer. (I pay FastMail, personally.)

<comment><USER54> I think you are still being sold even if you pay. Nothing really stops that. Even if they give away or even pay the bigger providers they are ultimately selling a white list. Since if they just accepted whoever gave them money with zero policing of abuse, even a warning and take down system for spam then nobody would accept their whitelist as it eventually becomes a blacklist of "the one spammers use" and their business model essentially becomes a scam.

</comment></comment></comment><comment><USER55> Reading this thread made me realize the magnitude of this problem. It seems very sensible that small email server administrators should unite into a single effort in order to better publicize this issue, raise awareness and shame Google into compliance.

Google seems to be suffocating the internet bit by bit on all fronts and it needs to be stopped.

</comment><comment><USER56> Imagine a phone carrier dropping your calls before the phone even rang on the other end. This is how I feel most of the time when I email people.

Every new recipient I email, if I don't hear from them within 2 days, I have to contact them out of band to ask them to check their spam folder. The problem is usually Gmail's heavy filtering.

</comment><comment><USER57> Can confirm, my server is also not accepted by google for years now when mailing someone with a business account. I'm not going to bother fixing Google's problems and support the behaviour, though, I'll just contact them through other means and let them know their email provider is blocking mail addressed to them.

At work we use sendgrid because of this. Have to trust a centralised third party to send out api keys. It's frustrating.

<comment><USER19> _> Can confirm, my server is also not accepted by google for years now when mailing someone with a business account. _

Hey, I remember you were one of the replies[1] in my previous thread that said I was exaggerating the "send emails" issue. Maybe your difficulty with Gmail's mystery filtering algorithm will warn others that depending on personal email servers for reliable outgoing email is a non-trivial endeavor.

Perhaps personal email servers are not impossible to set up but they're also not as easy to debug as some make it out to be.

[1] [https://news.ycombinator.com/item?id=15526127](https://news.ycombinator.com/item?id=15526127)

<comment><USER57> Good memory! A few things:

\- My comment in this thread was mainly about gmail for businesses, hence the contrast between "I'm not going to fix their problems and just tell the recipient their setup is broken" and the previous "it's doable". Other mail providers are usually no problem at all, and personal Google accounts are also better.

\- For email to personal gmail accounts it seemed to be enough to send a few mails and get a good reputation. Since then (and since my previous comment), I think I've started ending up in spamboxes again, but the number of messages I send to Google is very low. It's not outright rejected, though, since I would notice that.

\- I still think this part of my previous message is very true: "I don't think we should dissuade people from doing it, especially if the fact that more people doing it means that it'll be easier next time because it'll be slightly more common. Many of us are in tech and the field is a small subset of the population. Even if it's a small amount of servers setup by us, that could make a noticeable on those working at bigcorps who write the hostile receivers."

> Maybe your difficulty with Gmail's mystery filtering algorithm will warn > others that depending on personal email servers for reliable outgoing email > is a non-trivial endeavor.

I hope not. For what it's worth, I'm still at it and have no plans to stop hosting my own mail! I'm very happy not to share my data with a third party (the ideological part of it), but on a practical level, I also don't have a spam problem and I can be sure that email arrives. No message has ever not reached me due to spam filtering. It's also much faster than, say, Runbox, which we use at work and takes at least 30 seconds for any sign up email to arrive (too short to get coffee, too long not to be slightly annoyed). As another data point, my girlfriend switched from 1und1 to my mail server because it has some conveniences, without most of the downsides (since I manage it all for her). She hasn't had delivery issues so far.

As a related tidbit, I've also recently started blocking Google from my website. They're being a dick on the internet (not to me, it's more of a solidarity move, see [https://lucb1e.com/!130](https://lucb1e.com/!130)) and like with email, I'm not dependent on Google so I am again in the rare position to say "no, I'm not going to play your game" to Google. I'd rather encourage others to do the same than to give up and treat both email and search results as a walled garden.

<comment><USER19> _> I hope not. For what it's worth, I'm still at it and have no plans to stop hosting my own mail!_

You misunderstand the point I'm trying to emphasize. I'm _not_ trying to dissuade _you_ from running your own email. There are real benefits to controlling your own email server and it's great you've configured something that works for you.

I'm saying I disagree with how some proponents _communicate_ the drawbacks. They discount (or are blissfully unaware of) the real difficulties of running a personal email server for _critical outgoing emails_.

In the previous thread and this one, there are several of us with _decades_ of email admin experience saying it is tricky to debug "sender reputation" while the opposing side insists Gmail/MSOutlook spamholes are easy to fix and not a big deal. Therefore, one of the sides misunderstands the realities. With these conflicting accounts, readers contemplating running their own email server will have to decide who is more accurately reporting the state of the email ecosystem for personal servers.

The author (Tomaž Šolc) of the article discussed in this thread had ~15 years experience administering his own email servers and tried to do all the "correct" things as an upstanding citizen of the email ecosystem and yet _his emails still suddenly got rejected by Gmail_. Readers will have to conclude if he got bit by forces out of his control -- or \-- he's incompetent and doesn't know what he's doing. (From my point of view, the author wasn't incompetent and his frustrations with Gmail's filter is a common reality. Maybe this time, he can update DMARC to fix Gmail rejections... for now. But eventually, a new Google AI spam algorithm will mysteriously reject his server's emails again.)

<comment><USER57> Fair points. I don't have much to add to your comment, so I'll just say that I was nodding along with every paragraph. Thanks for clarifying what you meant to emphasize, I think I understand now!

</comment></comment></comment></comment></comment><comment><USER58> In my opinion the Gmail web interface should expose a way to generate something that is similar to a password, whose delivery is then controlled via the web portal.

This is similar to the +whatevertag trick that gmail pioneered for tagging, however that can be removed by malicious parties (spammers) via a simple regex. So Google have almost all of the infrastructure but should just add a bit more to get the rest of the way there.

What I mean in specific :

1\. I want to sign up to and receive your newsletter (you[re Ted) but I don't trust you yet. so I should navigate to gmail.com, click something like "generate another inbox", leave it set it to "For now deliver this mail to my inbox", add the description "for Ted's Possibly Spammy Newsletter", and then click "generate". It should give me email@example.com - similar to a phone number but a bit longer and personalized to one recipient - and then I should give that to the recipient to use, in this case the possibly spammy newsletter. It should always be delivered to my inbox, as I've set. Once one of the spammers sells my email address (for example I start getting advance payment scams) I'll be able to disable further spam from there by sending it to the trash but also know that Ted's newsletter is the one that got compromised or sold it. You can do this today by going through the steps of registering a new gmail address and turning on forwarding, but it takes like 10 minutes to do so. it should be like 10 seconds.

This should be possible because people always have easy access to the gmail web interface. There's no reason it can't be a bit more like a social network where you confirm it from the web interface as well.

that's my idea anyway.

</comment><comment><USER59> I've also run a mail server for a long time. Yahoo and Hotmail are probably the most annoying, Yahoo because you can't do anything about it and Hotmail for blocking entire IP-ranges. I don't blame them though, preventing spam is very hard. Spammers always have spf,dkim,dmarc,reverse dns,signing, etc meanwhile many legitimate senders has none. I'ts a hard problem to solve. I've tried things like increasing the response time so the sender have to sit and wait, or used spam lists, but there are too many false positives. One easy solution is to make it illegal to send spam. Since my country made sending unsolicited e-mail an offense, spam decreased a lot. The second problem is hacked servers. Unless you are running a mail server, always block port 25 in the firewall. So at least the hackers wont be able to send spam on your behalf. Many ISP's already block port 25.

</comment><comment><USER60> I had my personal domain hosted with Google years ago. One of the reasons I moved away is that emails that I was sending to other Google Apps domains were bouncing as spam when the destination address was an alias. The bounce came from Google Groups, which is what Google used for handling aliases within Apps domains.

e.g. [https://pastebin.com/u48DAaLP](https://pastebin.com/u48DAaLP)

That particular example was sent via SMTP, but I had the same problem when sending via the Gmail web interface, and it occurred sending to at least three different Google Apps domains.

After I moved my domain off Google Apps (I switched to Fastmail for a variety of reasons, but that issue was the kicker), I was able to send to those same addresses without issue. In fairness to Google, I was on the Google Apps free tier at the time, so there was nowhere to go for support.

</comment><comment><USER61> I've always thought that, I also used to operate my own email server and I've encountered the same problem with other email providers, always the big ones, yahoo (there was a time when yahoo was big in email, at least here), microsoft and now yes, google. What is even worse is that sometimes google just drops the email messages, replies with 250 ok, but the email never gets delivered to inbox or even spam. Definetly the big 4 or 5 or whatever are taking over on all aspects of our life. Actually if I think better about it, they have already done so, there was an article here on hn about blocking google, amazon and ms network blocks, nothing really works as expected without them.

</comment><comment><USER31> Regarding the issue with mailing lists for open source projects this isn't just an issue regarding Gmail. DKIM and DMARC cause no end of problems for mailing lists and it effects mail delivery to all major providers.

Mailman, which most open source projects use for mailing lists, have developed work arounds to address some of the issues. Unfortunately my experience is that many projects run older versions that don't have these work arounds or if running newer versions they have not been enabled. Most likely because no one has revisited the configuration since initial deployment on an older version. After all they didn't start the project to spend their time being mailing list admins.

</comment><comment><USER62> Am I the only one here experiencing that GMail does reasonably good job in filtering out the spam, while I have to visit the Spam folder approximately once a week, just to randomly correct a mail or 2 a month? (not a Google employee)

<comment><USER63> That's not the issue, though. The issue is that Gmail are too zealous in blocking and wind up blocking legitimate email with no way for the sender to debug what happened. From your perspective, nothing happened because how do you prove you never got an email you never knew was sent to you?

The people having problems here are non-Gmail senders sending email to Gmail recipients.

</comment></comment><comment><USER64> If ProtonMail would standardize a client protocol that other MUAs implement (not their "bridge" kludge), I'd start encouraging people to move there.

I'm not interested in ProtonMail's encryption (and it's potentially a liability, attracting aggressive state action). I'm mainly interested in their apparent respect for the privacy of users' private communications. And also hoping that ProtonMail has a bit more reliable delivery than GMail.

In any case, rising competition lifts all performance boats, or something like that.

</comment><comment><USER65> I've also hosted my own email for several years now, on several domains. The one thing I have that does not appear to be listed is a DMARC policy. As the comments on the blog post suggest, configuring DMARC works. I have no problem delivering to either Gmail or Outlook.

I don't think it's unreasonable to be strict regarding DMARC delivery. My MTA has a fairly strict SPF configuration - any email with an invalid spf result is rejected. This can come about because a legitimate company has misconfigured their spf records (happened twice in all the years I have hosted, discussions via postmaster@ helped them configure their dns correctly), but 99.999% of the time it is a spammer. What is worse is that rejecting email for domains without any SPF records can still result in valid email being lost, in 2019.

In this specific case, I don't think Google are "being evil". They're trying to reduce spam in the email ecosystem and they're doing it by using standards they themselves adhere to (Gmail send me reports of dmarc statistics each day google domains receive email from my box).

On the other hand, I do of course support either self hosting, or using another provider so as to ensure we do not end up with a Gmail monopoly. If I did not self host, I would find another provider like (but may not) Fastmail, Posteo etc (I would have to seriously review the options, which I haven't done).

</comment><comment><USER66> I'm glad I'm not the only one seeing these problems. I send mail from a personal server, and recently discovered that my messages to people @gmail were frequently sent to spam or simply dropped. I briefly thought friends were ghosting me until I figured it out. I set up SPF, DKIM, and DMARC, but it seems like the only way to get reliable email these days is to use Gmail, Hotmail, or Yahoo (at least for a little while...).

</comment><comment><USER67> As far as I can tell, Gmail isn't actually "real email" anymore, and you shouldn't use it for anything important. It's verging in being a walled garden.

</comment><comment><USER68> Does AI compares whom data to define which message is important or No? It doesn't track any specific way on evolution of mentality when it goes to define data correlated to humans usage. Like others products on market. Therefore, if you apply this thought to AI I guess they just made with basics conglomerated informations gathering. That CAN´T move a barrier when satisfying humans usage. It is just how you confront which message is correct to say is relevant or no. AI can be freely a way companies uses their needs on market influence because they're , on a very simple association, doing by themselves, then people get a notion of arbitrariety on those machines that create discussion to the whole universe you see about robot revolution etc, but not the case. I guess they try to hard to make then appear as human to be soon accepted by costumers, its just the way it is, it is not the time to require maximum levels of perfection on how easy they're to addapt as a user

</comment><comment><USER69> Gmail can be pretty aggressive with spam filtering, but I'm very happy with the bias toward flagging things as spam. Spam was such an infuriating problem, and lots of marketers would happily push the line if they didn't live in terror of being blacklisted by Google. If only Google could the legacy telephone networks like they run Gmail.

<comment><USER70> Flagging as spam is different. The OP is talking about rejecting it before it even gets to your inbox.

If gmail is improperly flagging a message I get as spam, I can create a filter to never send it to spam. If they're rejecting it like the OP talks about, if I want to stay with gmail I have no way to get that email.

</comment></comment><comment><USER71> The black box is more mysterious to me. I had a client for whom I managed a G-Suite account with a custom domain name. The domain was not on a blacklist, and things generally worked well.

One day a colleague and I discovered that he had not received some of my emails (intra-domain - email@example.com to email@example.com).

This is all within the confines of Google. Google had flagged some messages as spam, and by what determination I could not fathom. The content seemed perfectly typical.

I have had really pleasant experiences with G-Suite human support, at least in terms of the quality of interaction. But they could not answer why some intra- domain emails were being flagged as spam. I have suspicions that it would take a whole team of G engineers to maybe identify what bit of logic in their systems (incorrectly) marked some of the emails as spam.

It seems the beast (automation) is just almost not under their control anymore.

</comment><comment><USER72> Isn't charging people to use other carriers one of the big things that got Bell broken up? Google seems to tread in a of very dangerous waters. I wonder if they're just not aware of how close they are to being shutdown entirely, simply because so far they've always withstanded most legal challenges.

</comment><comment><USER73> There may be a workaround here: configure your own mailserver to relay outgoing messages through a service with a known good reputation. For example relaying via sparkpost or mandrill should be trivial to do. You might be able to do that under a free developer account if you don't send a lot of email.

</comment><comment><USER74> Thank you for sharing your experiences, insights.

#1 - Does anyone send test emails and measure delivery rates? As in send yourself a bunch of emails and see what happens.

USPS and its major customers and vendors do this with physical mail. They measure stuff like UAA (undeliverable as addressed). FWIW, their Inspector General estimates 4.3% of mail was UAA in 2013. Report Number: MS-AR-14-006 [https://www.uspsoig.gov/document/undeliverable-addressed- mai...](https://www.uspsoig.gov/document/undeliverable-addressed-mail)

#2 - What is the responsibility, liability for email relays to treat everyone equally? For comparison, a US retailer has to accept US currency, but can (sometimes) turn away problematic clients. Is there anything like that for electronic exchanges, transactions?

</comment><comment><USER75> If people stopped using G in such high percentage, then they would most likely need to adapt back to a more open internet, instead of trying to wall it off around their services.

Reminds me of jabber.ccc.de that stopped providing new accounts because they felt they were ruining a federated system.

</comment><comment><USER76> This is the reason I sadly don't recommend to customers that they run their own email servers, and instead outsource it to GSuite or O365. Being on the hook when emails aren't delivered for one or more of a dozen reasons that you can't control is no fun at all.

</comment><comment><USER77> "Only when something like this happens you realize just how impossible it is to talk to someone on the modern internet without having Google somewhere in the middle." This says it all and is likely become a strategic imperative for Google along the way.

</comment><comment><USER78> >I wonder. Google as a company is famously focused on machine learning through automated analytics and bare minimum of human contact

It should be noted that ultimately these efforts to "learn" result in ML, AI whatever pointed AT you, not working for you.

<comment><USER54> Well of course - don't trust the client is a rule for a reason. I get the spirit and how it can suck but it seems to be for a reason. Anything which blindly works for "you" can be abused by other "you"s like spammers and other bad actors.

<comment><USER78> I think the catch with AI and ML there is "reason" but it's just math or something that happens because.

If it is "reasonable" (trying to find a definition here) can only be determined by humans.... provided they care or understand it.

I always quote this one but I really like it:

"Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them."

― Frank Herbert, Dune

I would argue that it is clear that to some extent, Google does not care about the end results.

</comment></comment></comment><comment><USER79> Interesting...

Like the author, I've been running my own mailserver for over a decade and am very conscientious about ensuring that no attackers use it as a spam relay.

While the vast majority of the people I exchange email with don't use GMail at all, so it can take a while before I notice any issues with it, I did happen to notice that GMail was rejecting my outgoing email a couple of weeks ago.

This week, I finally got around to trying to address the problem (it's not high priority because having GMail reject my emails isn't really a huge deal).

...and I found that it is working again without my changing anything. Weirdness abounds.

</comment><comment><USER80> Running your own e-mail server for receiving SMTP makes a lot of sense in many ways. I've been doing it for close to a decade myself.

 _Sending_ SMTP yourself (directly, without an SMTP relay service) sets you up for trouble.

</comment><comment><USER81> What boggles the mind is that they've a very string disincentive to fix it: I had to move away from my privately run email to Gmail for that very reason, and I'd gather I'm far from being alone.

</comment><comment><USER82> Better that what I've been seeing, which is Google just sending emails sent from my private domain to spam, with no bounce back or notification of any kind. This is despite the fact that I send the emails FROM gmail using their 'send as' feature. Nor does it seem to matter if I'm writing to a contact whom I've exchanged many emails with in both directions for years. If I send from my @gmail address they always go through. Send the same content from my domain and it's a crapshoot.

Edit: And of course, I do have DKIM and SPF configured.

</comment><comment><USER83> I tried to set up a Discourse discussion forum for our neighborhood this year, and got stuck on setting up an email server that wouldn't get blocked automatically by all major providers. Email is not a healthy ecosystem. You have to pay to send from one of the big trusted providers or else you get blocked just because your email ip address is untrusted by default. Yes fighting spam is important, but it's shocking how much email providers rely on simple ip filters and trust levels rather than AI analysis of email content.

</comment><comment><USER84> This article didn't mention whether or not their servers are attempting delivery using TLS/StartTLS, which is a good thing to check. I think Google penalizes email delivered insecurely.

</comment><comment><USER85> In the article:

> I can't tell other people to go off Gmail

I disagree. There are reasons to switch off gmail. Not just Google eating mail but also for privacy reasons. Google knows all about your banking, eCommerce orders, your media subscriptions, health issues and many other dependencies.

A good alternative is protonmail. It is private, has a mobile app, is a free but you can also pay to support the service. I also consider protonmail much more secure than gmail.

<comment><USER86> Or, if you don't want to commit yourself to yet another walled garden, there are quite a few other email providers that to choose from.

<comment><USER87> Please list a few. I need 3 personal inboxes, 2 home business inboxes (bills, etc), and the ability to make new inboxes for stupid startup prototypes you buy a domain for, hack on for a few days with dreams of huge sales, and then forget about for a year until the domain expires. You know, normal hacker stuff.

<comment><USER88> You may want to check out Helm - [https://thehelm.com](https://thehelm.com) (I'm a co-founder)

</comment></comment></comment></comment><comment><USER89> I'm quite happy with the ease of setting up and running a Mail-in-a-Box [1] instance. After seeing this article I did an experiment and while my hosted mail sent to a @gmail.com address did not bounce it went straight to the Spam folder. At least I can add other users to my email and enjoy private conversations.

1\. [https://mailinabox.email/](https://mailinabox.email/)

</comment><comment><USER90> Yet one more example of this. See [1], [2].

Definitely Google has done something to mess up their spam filter algorithms in the last year.

[1] [https://news.ycombinator.com/item?id=19536465](https://news.ycombinator.com/item?id=19536465) [2] [https://news.ycombinator.com/item?id=19500357](https://news.ycombinator.com/item?id=19500357)

</comment><comment><USER91> If I recall my time at a main-sleaze email sender, some ISPs will send SMTP retry requests on random incoming email to see if there is a functioning SMTP server on the other side of the originator's email. Often times, spammers won't care about the email that's being sent, so won't respond, or in cases where they are hiding the origin, won't be able to respond.

</comment><comment><USER92> I don't know that this is purposeful in all cases. I've had issues on a mail host where sending an email to a different box on the _same domain_ would reject the message as spam, refusing to even deliver it. With respect to google, however, the fact that they offer a service of their own that _conveniently_ can avoid this issue is more than a little shady.

</comment><comment><USER93> Its like the deep learning has identified a feature it can use to determine spam: is this from the top 10,000 domains? No? It's spam.

Spam from small domains might be pretty high as a category, but of course we don't want statistical judgements about categories to outweigh the merits of the individual. Maybe Google's algorithms have been watching too much Fox News.

</comment><comment><USER94> I often see and hear about this issue a lot and I wonder how other email companies cracked it.

I use FastMail for instance and never have this issue, but I know so many people who gave up on running their own mail servers at the small enterprise level because of stuff like this I often wonder how FastMail does not have these issues but others do. Is it a headers thing I wonder?

<comment><USER95> Fastmail is big compared to the operator in this article. Fastmail is probably big enough that some Gmail engineers will know how Fastmail's SMTP chatter looks. If you don't have volume you don't get that.

</comment></comment><comment><USER96> Someone posted on reddit the other day about (seemingly) the exact same issue [https://www.reddit.com/r/selfhosted/comments/bbrvlt/any_reco...](https://www.reddit.com/r/selfhosted/comments/bbrvlt/any_recommendations_for_a_reliable_smtp_service/)

</comment><comment><USER97> I use my own SMTP server for receiving, but for sending I use the ISP's server (I think I have seem to have read somewhere that not only Google but some other services as well, will not accept mail from dynamic IP addresses). I have not had problems that I know of, so far. Please tell me of whatever mistake I may have made of such thing.

</comment><comment><USER98> Gmail is rejecting valid content as spam and it never hits my spam folder. If I was not expecting an email as I was in this case, I would be totally oblivious. Something is up with the spam filter recently. They might have decrease the tolerance for spam and as a result a lot of legitimate traffic is being hit in the cross fire.

</comment><comment><USER99> That's the price you're paying for a) self hosting and b) relying on decentralized protocols that don't have sensible abuse protections.

People hosting their own servers enabled wide spread abuse due to misconfigurations. Because everyone could do it and because defaults were shit for decades, stuff like open relays were common. People defaulted to the wrong ports. Almost no one bothered to offer STARTTLS/Transport Encryption. Spam would have killed mail by now if it hadn't been for major players like Google, GMX, Hotmail/Outlook/etc.

Back in the day, greylisting was commonly regarded as a best practice, leading to the impression that email is unreliable and prone to latencies.

I'm sorry it's this difficult to host mail by yourself nowadays, but I'm happy to have a spam-free inbox every day and if this is the price for that, I'm sure about 1-2 billion people are willing to pay it.

I'm quite astounded that there have been no updates to mail protocols in the last couple of years to at least mitigate the most common issues, but all I see are band-aids that are complex to setup and horrible to debug in case of issues.

</comment><comment><USER100> I've had Google blackhole me several times. Each time without any apparent reason. It's impossible to talk to a human being at Google who is able or willing to help. I've had similar problems with MS, but at least I was able to talk to a human being and get the situation resolved.

</comment><comment><USER101> To me this is more evidence that we definitely need public decentralized platforms that can replace Google and once we have them we need to eliminate the company. Private companies should not be allowed to monopolize platforms and control our lives.

</comment><comment><USER102> Thanks for writing this up, this mirrors my experience setting up a modern email server for personal & business use.

I currently self-host for non-mission-critical email, use FastMail for business, and continue to use google apps for personal/mission-critical.

</comment><comment><USER103> If you run your own email server check out Postmark's DMARC monitor, it's a free service:

[http://dmarc.postmarkapp.com/](http://dmarc.postmarkapp.com/)

( _tablix.org_ doesn't have a DMARC record)

</comment><comment><USER104> He's been using the same IP address for his mail server for 15 years. It's possible his IP is on a subnet range that regularly gets blacklisted by Gmail due to other actors on the subnet sending malicious email/spam to Gmail.

<comment><USER26> Presumably, Google's smart enough to identify individual IPs in the range that don't send spam. I have a real shitty neigborhood IP wise, having my mail server on a $1 VPS. Public Outlook.com drops all my mail regardless of content for example, citing spammy IP range. Yet gmail somehow learned after a while to pass most of my messages through.

The real problem is lack of provided reasons for blocking, so people waste a lot of time trying to figure it out by guessing and trying random shit.

</comment></comment><comment><USER105> Does anyone here use encrypted and signed messages on a regular basis? How does gmail handle that on either sending or receiving ends? Is delivery rate better or worse if it is fully encrypted?

</comment><comment><USER106> Reminds me of Spamhaus. Are they still around? When I worked for an ISP we had to deal with them regularly. They'd happily hold you hostage over a single e-mail.

<comment><USER107> yes, they are, and they've been really really good. haven't seen any of the behavior you refer to. ever. now SORBS on the other hand... doesn't really exist anymore (fortunately) but spamhaus has been great.

especially when used as a filter in Postscreen.

</comment></comment><comment><USER108> I use non-Google email with SpamArrest, gives me complete control over whitelist/blacklist. Legitimate users always have a way of getting through.

</comment><comment><USER109> Ignore it and let people be angry (at Google for not receiving their mail from various places). Meanwhile move to a decent service like ProtonMail.

</comment><comment><USER110> Self hosting email is a pain in the butt.

Try fastmail infomaniak.com (can also buy email for external domain) gandi.net (email included in domain)

<comment><USER110> PS: From the comments of the link

"We run small hosting company, if any domain blocks our clients mails we block theirs and problem gets solved magically." :-)

</comment></comment><comment><USER111> I had exactly the same kind of problems when I tried to set up a new mail server.

In the end I just gave up and started using mailgun as a relay.

<comment><USER87> Can you tell us a little more about your setup? I have a home email server, incoming only because of ISP restrictions, and I would love to make it into a real email server for my family and business.

<comment><USER111> It's just smtp relay really, nothing fancy. I still run my own mail server but I relay outbound over to mailgun

</comment></comment></comment><comment><USER112> I am in the process to detach Google products, somehow gmail is the only product I cannot find a working alternative.

</comment><comment><USER113> I wonder if originally this technically constitutes an illegal form of interception of mail in some jurisdictions.

Perhaps there really would be a lot more spam without such filtering, but it points to the actual problem being elsewhere. Perhaps we need some kind of cheap and userriendly (uniform but decentralized) email court system, and fine / ban email accounts that misbehave?

</comment><comment><USER114> I have a similar problem and it sounds like some here do as well.

What can we do about Google's email monopoly?

</comment><comment><USER115> I for one welcome our new overlords

</comment><comment><USER116> Does google have legal obligation to receive mails from 3rd party smtp servers ?

</comment><comment><USER117> I have a lot of success by having put my mailservers in the dnswl.org system.

</comment><comment><USER118> Can someone explain in simple terms why, in 2019, anyone is running their own mail server? Unless you are the sysadmin, server guy/gal, whatever, this just sounds like signing up for a world of hurt doing it yourself.

<comment><USER119> Contrary to popular opinion, I don't think the technical aspects of operating a mail server are all that hard. Perhaps I'm underestimating the learning curve since I've been doing it for 27 years. (If you're running your own mail server, you _are_ the sysadmin.)

The frustration comes with scenarios such as the one outlined in this blog post, where small mail server operators get bullied even though they are doing everything right. I can completely understand not wanting to operate a mail server due to this situation, or not having interest in leveling-up server administration skills.

For those of us who do have expertise in running mail servers, it's a shame we have to deal with these obstacles.

<comment><USER120> The biggest obstacle of course is dealing with deliverability problems. As most posters have commented here, getting people to accept your mail, and accepting mail from reputable sources isn't an easy problem to solve. At FastMail, a considerable amount of time is spent ensuring customers don't need to worry about deliverability problems or spammy inboxes.

To me, that's the biggest reason not to spend time running your own infrastructure for mail. Getting people to accept your mail reliably, is much harder than adjusting some postfix configuration.

</comment></comment></comment><comment><USER121> Ironically Google's spam always seems to go through...

</comment><comment><USER122> They want your users to use gmail, that's all.

</comment><comment><USER123> Cue the *"but it's so easy to host your own email! Mine has been working fine for years!"

Until this happens. Which it does very often, for lots of reasons that are out of your control.

</comment><comment><USER124> One more reason to de-Googlify my life. AWS has WorkMail I think it is time to try it out.

</comment><comment><USER125> Fair play.

I like my emails with pesto sauce, tuna, and cheese ... wicked combination.

</comment><comment> 5874-4b22-a4e0 Op should sign up for gsuite.

</comment><comment><USER126> Only 3 people I know using Gmail any longer. I stopped back in 2010 and "didn't feel a thing".

<comment><USER127> How do you really know? G-Suite has become extremely popular with business in the last few years.

<comment><USER26> You'll see something like this in the logs:

... status=sent (250 2.0.0 OK 1556280611 c1###########x.57 - gsmtp)

even if the e-mail address doesn't end with @gmail.com

</comment></comment></comment><comment><USER128> Have also run into this. Google is the quintessential evil tech corporation. We should all start avoiding their services when possible.</comment>


CSS Road Guide - <AUTHOR> https://lyty.dev/css/index.html <comment><AUTHOR> You can learn from free @ [https://lyty.dev/css/index.html](https://lyty.dev/css/index.html)

</comment><comment><USER2> what's the best resource to learn CSS in 2020? Any suggestions for either paid and/or unpaid books/online courses

<comment><USER3> I noted quite a few here: [https://wiki.nikitavoloboev.xyz/front- end/css#links](https://wiki.nikitavoloboev.xyz/front-end/css#links)

I liked [https://absolutely.every-layout.dev/](https://absolutely.every- layout.dev/) for its focus on thinking in layouts</comment></comment>


Ask HN: So, Bitcoins - would you still buy them? - <AUTHOR>

I am fascinated by Bitcoins. I set up my wallet last year, but never bought them (yeah, I know. Idiot). But I am still fascinated by the whole concept and I'm thinking of buying one or two. The only thing is - I have no experience in trading stocks and this rapid growth looks to me like some kind of bubble. So, HN, what do you think - is the growth somehow sustainable or will this thing burst?
(edit, somehow only half of my entry was posted. Sorry for that.) <comment><USER2> As long as you have the funds to spare. Just don't invest anything you can't afford to loose.

That's what I do. I agree, it's very fascinating to watch a digital currency come into use.

I've bought two cell phones, and a video card using bitcoins. And I've hired a developer for a day with bitcoins. So I see some value besides just investing in them, which also was a bit more incentive.

<comment><AUTHOR> Yeah, I wanted buy them and then just see what I can do with it. It just looks fun to have them and play with a digital currency.

</comment></comment><comment><USER3> This is my whole problem with Bitcoin. Too many people see it through an "investment lens". Bitcoin is supposed to be a currency not an investment at it's heart:

"Used as a medium of exchange for goods and services, currency is the basis for trade."

People's perception coupled with the deflationary nature of Bitcoin makes me not want to participate.

<comment><AUTHOR> I think I phrased myself wrong. For me it is not an investment (as in: Buy now get rich), but more a experiment or plaything. I'm just not sure if I shouldn't wait another month or to for the BTCs to get drop again. On the other hand, they are just money, so if I have BTC worth 50 USD i should be able to buy stuff worth 50 USD (using more or less BTCs). I just need to wrap my mind around it.

</comment></comment><comment><USER4> If you were planning on buying $50 worth of BTC last year, you can buy $50 worth today. Don't think about buying "one or two". There's no benefit to having whole numbers.

<comment><AUTHOR> Ah, good idea.</comment></comment>


The sorry state of Visual Studio syntax highlighting (2012) - <AUTHOR> http://lolengine.net/blog/2012/11/27/visual-studio-syntax-highlighting

<comment><USER2> What the hell makes you think your dislike of the Visual Studio syntax highlighting system makes it okay to be so bloody rude? Would you be happy if everybody who disliked your work said "fuck you, ulam2"?

I'm not saying that sort of language is never appropriate - it's a fair response to patent lawsuits, for example. But let's keep feedback on user interface issues a bit more civilized.

<comment><USER3> Because it helps fishing views, obviously

</comment></comment><comment><USER4> I know, pointless comment, but some syntax highlighting issues hardly seem worth a "fuck you". Article title would be much better without that.

</comment><comment><USER5> Post is from 2012, but still it is weird to be so inflammatory about the inabilities of customizing a specific part of syntax highlighting.

Unless you have actually written syntax highlighting code parsers yourself better as what is in Visual Studio I am of the opinion you do not have any rights to be that rude.

<comment><USER6> I've written more than a few syntax highlighters (as well as semantic highlighters and entire IDEs based on incremental parsing and type checking). This is one of those features where eclipse is quite ahead of visual studio, but its quite a minor feature it makes sense that isn't a high priority.

<comment><USER5> You're the author of the post? Can't seem to see who author "Sam" is at that blog, but I did notice that not only did the author write the complaint, but also wrote code to address his issues in a later post.

[http://lolengine.net/blog/2012/12/01/announce- vslol-220.127.116.11-...](http://lolengine.net/blog/2012/12/01/announce- vslol-18.104.22.168-release)

FWIW, I've worked on syntax highlighters myself and IDEs as well which is what prompted my reaction. The particular problem didn't seem to be a big enough problem for me to warrant the post. But hey.. what do I know, I'm not Sam, at least kudos to him that he actually did something about the issue he had besides writing the inflammatory post.

<comment><USER6> Not the author of the post but have been frustrated by this before.

</comment></comment></comment></comment><comment><USER7> So one person does not like syntax highlighting of Microsoft and he/she uses expletives. Bugger you, sir, for writing this article.

<comment><USER8> The article discusses a few workarounds. I didn't like the tone, but it's interesting, not only a long rant.

</comment></comment><comment><USER9> I keep forgetting that is a worse swear word in american english. 2 years later though the situation has not improved.

</comment><comment><USER10> why do you need colors at all?</comment>


Ask HN: Are AWS Multi AZ ELB IPs Virtual? - <AUTHOR> Setup:
Multi AZ ELB hostname: myapp.elb.amazonaws.com Hostname resolves to IPs (Round Robin): 220.127.116.11 and 126.96.36.199
In the event that the LB service behind IP 220.127.116.11 fails, does AWS remove the IP 188.8.131.52 from the hostname RR rotation or does it simply keep the same IP (i.e. virtual) but points to a healthier LB service?
Documentation doesn't seem to discuss this detail. Hopefully someone here knows :) <comment><USER2> Wait.. are you using the IP of the ELB instead of the A or AAA record and then using Route 53's round robin DNS? You should absolutely not be doing that if so but if you _must_ you should enable health check on Route 53.

Edit: Here is the mention from the ELB panel.

> Because the set of IP addresses associated with a LoadBalancer can change > over time, you should never create an "A" record with any specific IP > address.

<comment><AUTHOR> No I'm not using ELB IPs directly. My concern is this scenario:

1) ELB with IP 220.127.116.11 develops an issue (AWS internal outage)

2) DNS resolver for an end user of my app has the pair myapp.elb.amazonaws.com/220.127.116.11 cached previously for longer than the TTL (resolvers are notorious for this)

3) When end user resumes work within the app, they're referencing the LB at 126.96.36.199, which is down.

Hence my question. Is the mapping of 220.127.116.11 to its respective LB static or virtual? Meaning, does AWS keep moving the 18.104.22.168 around to a healthy LB service seamlessly?

Thanks</comment></comment>


Show HN: Manage passwords with GPG - <AUTHOR> https://github.com/drduh/pwd.sh <comment><USER2> [http://www.passwordstore.org/](http://www.passwordstore.org/) is a similar solution, and I prefer it to this solution for a few reasons:

* git integration.

* Separate file for each secret, so that I can store the password on the first line and then other sensitive account details on subsequent lines.

* -c flag for copying passwords to clipboard (but only copies the first line of the file, so it doesn't interfere with the usecase above)

* tab completion for user account names. However, this comes at a slight cost to security -- anyone with access to your machine (or git repo) can see all of the websites / accounts for which you have a password.

* Everything happens via the file system and secrets are just gpg encrypted text files. So it's really easy to implement new utilities on top of your password store. This is true for this solution as well, but somehow having separate files for each account makes it safer to implement utilities that do account management.

<comment><USER3> These tools really need a browser plugin for them to be useful to me.

<comment><USER4> Has anyone tried using these?

\- [https://github.com/gustaebel/passext](https://github.com/gustaebel/passext) (Chrome)

or

\- [https://github.com/jvenant/passff](https://github.com/jvenant/passff) (firefox)

<comment><USER5> From the chrome extension github:

      "This is pre-alpha quality software, the result of a three
      day project. It will crash your browser, leak your
      passwords and destroy your home. This is actually my first
      Chrome extension and I am no expert javascript programmer."

YMMV

</comment></comment></comment></comment><comment><USER6> I'm using Emacs for something similar.

This works because Emacs can open .gpg files. It will decrypt them on opening (asking for your password) and encrypt on saving. This is very powerful in combination with Orgmode (.org), or any other module that provides auto- folding.

So I open my .org.gpg file and everything is folded. Then I search for what I need, and only that part (containing some secrets) is unfolded.

Of course, this is no substitute for a proper password manager, but proved to be useful a lot more often than I initially thought.

<comment><USER7> I'm not sure this is a good solution for this use case (password management).

The obvious way of getting the password out of the Emacs buffer is copy and paste, which leaves the password on the clipboard where it is very easy to find. Manually removing it (by copying something else) can't be relied upon. (If you use Klipper you have an even bigger problem.)

I know all bets are off with any kind of password manager if the host is compromised, but password managers and browser plugins presumably at least try to scrub passwords from memory, which will save you if you forget to lock your screen or your window manager has a screen lock bypass bug (very common).

The Emacs solution will protect you if you don't use full-disk encryption and your disk falls into the wrong hands, but that applies (or should apply) to all password managers.

What am I missing?

Edit: The same applies to the solution in the article.

</comment></comment><comment><USER8> Why bother with the fixed functionality?

        (defun cc () "Secrets File" (interactive) (find-file "/home/ajross/.cc.gpg"))

Launch with "M-x cc" (which is simple enough) or bind to a keystroke. Emacs will prompt you for the decryption cleanly, your distro will surely cache them with gpg-agent, and you can then just edit it and cut and paste as you like.

I'm pretty sure it's "cc" because it was originally a list of credit card numbers, but quite frankly I've been doing this so long I've forgotten.

</comment><comment><USER9> Pass ([http://www.passwordstore.org/](http://www.passwordstore.org/)) is password manager based on GPG that's been around for longer. It has some more tools available that are built around it and helps organize passwords as well.

</comment><comment><USER10> I'm definitely all for multiple options, but a tool that does exactly this exists already, and uses git so you never lose history and can easily pass the store around.

[http://www.passwordstore.org/](http://www.passwordstore.org/)

</comment><comment><USER11> A bit late to the party, but for anyone who loves pass/password-store etc, but does not like the entry names themselves being stored in the clear, I have a port of pass that fixes exactly that problem:

[https://github.com/abgoyal/password- store2](https://github.com/abgoyal/password-store2)

Yes, it even has bash completion. Its fallen a bit behind the upstream as I have not had the time to port in the new features (nor felt the need :-/). Comments/patches welcome.

</comment><comment><USER12> This and passwordstore.org are neat. I used to do one-password-per-gnupg-file with plain GnuPG commands, and liked it quite a bit. Lately though, I've been using passwords in plaintext in a AES256 disk image with a shared long passphrase. The main advantage so far is that other family members can use the disk image work flow easily. The main disadvantage is that the disk image format is OS-platform specific. I know that KeePass probably works best for family, so we may switch to that in the future.

</comment><comment><USER13> I just use a GPG-encrypted org-mode file... It's neatly organized as a table, and always easily accesible from my recent files buffer.

<comment><USER14> So far 3 person mentioned this, tempting.

</comment></comment><comment><USER15> Does anyone know if there is a way to make something equivalent to this but using GPG's symmetric crypto instead of public crypto? This way you don't need to carry a key file around with you, just the encrypted secrets.

<comment><AUTHOR> This is a great point. The advantage I see to using pubkey is integration with, for example, a cryptocard ([https://trmm.net/Yubikey](https://trmm.net/Yubikey)). Still, I wonder if the default should be to use _gpg --symmetric_ instead.

<comment><USER15> The reason I asked that is because last time I messed around in this problem space I had some trouble making gpg-agent remember the password for symmetrically-encrypted stuff.

</comment></comment></comment><comment><USER16> Reading in and storing the passphrase in a shell script seems like a bad idea, particularly since GPG already has a tool for reading passphrases that will cache your credentials.

[edit] I see that the passphrase is passed as a command-line argument. Aren't those viewable to other users on linux?

</comment><comment><USER17> An other solution to manage password in one file using GPG encryption: [https://github.com/boussouira/bash-pass](https://github.com/boussouira/bash- pass)

</comment><comment><USER18> [https://github.com/jimktrains/polygonus](https://github.com/jimktrains/polygonus) was a little tool I wrote a while back.

</comment><comment><USER19> I was going to try use one of these solutions, when I read on the EFF that they all provide a single point of failure.

If anyone gets hold of your master key/pwd, they would have access to all your usernames & pwds.

[https://ssd.eff.org/en/module/how-use- keepassx](https://ssd.eff.org/en/module/how-use-keepassx)

Best to keep them separate - in your brain!

<comment><USER20> Thats true, but you do forget someone. If you use four-five different password spread among different sites, they will be easier to break then any generated password from keepass, pass or passwordstore.

>If anyone gets hold of your master key/pwd, they would have access to all your usernames & pwds. You would still need access to the physical storage medium. This is either a threat or not depending on your threat model, and for most people this is simply not a threat. And tbh, if someone got a hold of your unencrypted computer you got another problem.

</comment></comment><comment><USER21> We created our password file with ccrypt, and modern vim can open it right up with no fuss.</comment>