Improvisational theater can reduce anxiety in teens who struggle socially - <AUTHOR> https://news.umich.edu/anxious-teens-gain-confidence-by-performing-off-script/ <comment><USER2> It didn't really give any indications as to how it works in the article, but the people I've known with the worst social phobia have been those who've been most scared of potential (or imagined) social punishment, usually because they feel there is a "script" that they will have trouble following in a social setting (e.g. responding to something "the wrong way", or not having the "right" answer ready for a question, etc.) Seems like it makes sense that if you're constantly places in scenarios with no script and no right or wrong answer, it would help alleviate the anxiety over not knowing "what everyone knows" you're supposed to do in [situation], and that being confident in what you say is like 50% of being seen as a socially capable person.

<comment><USER3> The study notes a high attrition rate (~45% of those who started this dropped out), and this appears to be a voluntary endeavor.

This is significant to me, as I suspect many of the people with anxiety that is significant enough to be diagnosable would not be willing to do it or would pretty much shut down if placed in the setting.

To delve into personal anecdotes:

I had a theater class in school that regularly included improv exercises like described. In 11 years of it (K-10 or so), I don't think it ever went well.

The actual manifestation of the problem with "no script" was drawing a complete blank as soon as I was asked, not having a bunch of ideas I was scared to respond with.

It's certainly possible that these people have a better method, but I had multiple theater teachers and I was socially fine in normal settings. All these activities tended to do was give me more anxiety issues because everyone else could come up with things and I'd just be stuck there no matter how people tried to help me get going.

Those are not really common situations in adult life, but I'm still bad with "icebreaker" games and other similar things that "put me on the spot" to come up with something outside of normal social interactions.

<comment><USER4> Gosh I can relate to the blank feeling. Any idea why this might be the case?

<comment><USER5> I used to have this pretty bad. I believed that it showed that I was autistic, or lacking some kind of fundamental creative spark, or something like that. It's gotten a lot better since I got treated for depression and anxiety. I think it was really just me shutting down ideas at a pre-conscious level. I don't know much about how the brain works. But this is my (pseudoscientific BS) mental model of it:

1\. Our ideas and memories are composed of connections between neurons.

2\. "Creativity" is the expression of random, new connections between those neurons.

3\. Creative expression requires focusing our attention on those new neural pathways, filling them in and adding detail.

4\. Depression and anxiety caused me to focus my attention on the idea of failing to come up with something "on the spot" (and the many minor traumatic memories of failing in this way), instead of the infinite weak, undeveloped ideas that could be explored with a moment's focused attention. It's hard to be fun when you are having a small anxiety attack.

5\. The antidepressant I'm taking has gotten me out of this rut. Whether it's by turning off my anxiety, or helping my nascent creative neural pathways reach further by preventing the reuptake of some neurotransmitter, I don't know.

<comment><USER6> I think it's related to the exploration exploitation tradeoff in reinforcement learning. Creativity is related to exploration and randomness (and a high learning rate) while exploitation to using past knowledge to extract the maximum benefit. The brain switches off the creative/learning part in order to maximise the latter. In reality we need a mix of both, exploration is necessary for improvement and exploiting past knowledge is necessary for achieving high rewards. In neural nets, a too high or too low learning rate would not lead you to the optimum, you need to find the exact sweet spot. Anxiety would turn the balance too much into exploitation mode when we need to be creative.

</comment></comment></comment></comment></comment><comment><USER7> This is a bit far out, but if anyone here lives in London and wants to try their hand at this: I run improv workshops. Usually for actors , but I'd love to do a HN session or two :) if there's interest.

I will definitely make sure to tailor it 100% to be approachable by even the most awkward introverted IT'ers among us; I've been in the business for long enough to know how. No pressure, guaranteed. Sitting and watching for the entire workshop is also fine.

No charge beyond a few quid pp to cover a room. beyond that we just need an evening and some snacks.

My contact is in my profile; reply here or drop me a line—would be a lot of fun to make some magic happen!

</comment><comment><USER8> I often have social anxiety and I can confirm these findings. Some years ago I did an improv class. First I was really scared but we did a lot of exercises where you really had no pressure to be funny but funny situations just developed without anyone trying. I learned to say "yes" to new things and go into the unknown without being scared.

It was super low pressure and very supportive and I would rate it in the top 5 of life transforming experiences for me. Others are travelling without preparation, joining Toastmasters and picking up martial arts.

</comment><comment><USER9> This was basically folk knowledge when I was growing up in Brazil. Theater classes for teenagers were like 40% attractive people who want to make it as actors/60% awkward teens.

From a basic trust in the scientific process (but also noting the limits of scientific psychology) I figure that social anxiety and the like are serious mental illnesses. But I immigrated as a three-year old and didn't know the language/make any friends for a while, so my social skills always lagged. Theater lessons (some of which was improv, but we wrote our own sketches for a capstone play for our own families) sort of helped me balance that. If there was any anxiety it was secondary/symptomatic of some underdeveloped stuff, much like people develop math anxiety.

In retrospect, I attribute it to the "growth mindset" phenomenon: beginners make fools of themselves onstage whether they are awkward or easygoing, so you realize that being-comfortable-in-your-skin is something you can learn rather than something you were born without. Maybe this is what people are going to "hackathons" for - in a pressured time-boxed situation people are less likely to look down on the less math/programming-literate and help them instead.

</comment><comment><USER10> "Make someone do something they're afraid of at a level they can handle and they get better at it and less afraid".

Not surprising. This is how I conquered my deep fear of flying. I took some flying lessons and even had the instructor do stalls and other scary stuff and I was 95% better afterward.

It's so much easier to take a drug or blame biochemistry than to face your fears. I think it's underrated in today's world where there's a handy diagnosis for everything.

Note: not casting aspersions on people with serious issues. But when 20% of the US population is supposed to have anxiety disorders, something doesn't make sense.

<comment><USER11> Not so fast. Facing fears can be beneficial or lead to deep traumas depending on many other factors.

<comment><USER10> Notice it says "at a level they can handle".

And if it all goes terribly wrong there's still powerful drugs for that too. So what is there to lose I say.

</comment></comment></comment><comment><USER12> I'm somewhat skeptical of these results. I'm a member of an (amateur) improv group. When I improvise, I often feel more nervous because I'm thinking about several things at once: acting, supporting my partner, finding the "game" of a scene. In this way, I can get "stuck" in my head much more than I would in everyday life. While it certainly feels great to have a good show or practice, it can feel pretty crappy to have a show that goes poorly. If I had problems with anxiety, I'm not sure that improv would help.

<comment><USER13> I've been taking improv classes for most of this year and I disagree with your conclusion.

You don't heal from anxiety by hiding from anything scary. You get better by doing things you feel anxious about (and then not dying). And as for presence - if you're getting stuck in your head while on stage, it becomes much harder to listen and be present to your scene partner and the needs of the scene. In my experience there's a limit to how good your scenes can be if you're in your head about them.

Improv for me has helped my mental health in lots of tiny ways - I've gotten better at visibly failing. I've gotten better at expressing negative emotion and I've gotten a lot more present. I feel like improv theatre is a microcosm of normal struggles; a lot of the long term issues that we face come up in improv too. And on stage you get the chance to face those demons in a safe, supportive environment surrounded by friends.

Being in your head is exactly this sort of developmental challenge. Your improv will (long term) get better if you get out of your head with it and instead be more present and intuitive. And that's really important in regular life too. For people who struggle with that, what a perfect place to practice.

Personally I highly recommend doing some improv classes to anyone smart and creative who wants to push themselves and grow. It's hard in the best way.

</comment></comment><comment><USER14> My take is that theater, or taking the stage in general, helps tremendously.

I remember being such a shy child that in school I couldn't even get myself to stand in line to the sink in which all the others were washing their hands before lunch.

My mother sent me to acting classes when I was eight or so and this experience changed me forever.

It's the sheer adrenaline of being at the center of attention with your group that does this - just about any social situation becomes manageable after that.

</comment><comment><USER15> I'm a believer in these findings. I do wish that the model for learning improv in the private sector was different. As it is, adults have to move to cities where theaters are offering classes, then pay out for the classes (which can run up to $300-400 per 8-week class in the bigger schools). It's expensive, is the point, and this creates a barrier for people of a certain class. I'd love to see it offered in more high schools, or as a free-or-cheap activity akin to soccer or community theatre.

Don't even get me started on the level of privilege needed to pay through enough class levels to let one of the theaters audition you for their non- paying staff performer spots.

<comment><USER16> It's not as though improv teachers are walking around buying luxury vehicles, though. The classes are as expensive as they need to be for the instructors to eat.

There's nothing stopping people from starting up a community improv troupe without formal training from a teacher, just as you don't need someone with an MFA in Performance to start a community theater group. There are enough books out there on the art form that a dedicated group could learn the necessary techniques on their own.

And high-level youth soccer can be _way_ more expensive than even UCB-level improv classes.

<comment><USER17> I've looked at acting classes, and even beginning classes are much more expensive (2-5x) than any other type of classes I've taken, like martial arts, music, or dance -- even private lessons.

Do dancers not need to eat? Do musicians not study as long to become competent teachers? Why is acting priced so different from every other art?

<comment><USER16> Supply, demand, and upside.

Movie stars get paid orders of magnitude more than top martial artists/musicians/dancers, so more people try to be actors, so there's a larger market, meaning instructors can charge more.

Also depends where you're looking. The professional acting scenes in LA/NYC tend to choke out the hobbyist scenes (why see a community theater show in Manhattan when you can get sub-$50 tickets to a Broadway show from TDF?), so if you're looking for acting classes in those areas, they're directed at and priced for professionals.

</comment></comment></comment></comment><comment><USER18> Improv is one the most fun and useful thing I've done. I don't think I had any diagnosable social issues, but it's made me better socially. It's also made me better at presenting and pitching. And it's given me something to spend time on that's not work or tech related.

</comment><comment><AUTHOR> The original study is available here: [https://www.sciencedirect.com/science/article/abs/pii/S01974...](https://www.sciencedirect.com/science/article/abs/pii/S0197455618301928)

Here is the abstract:

Adolescents who have Social Anxiety Disorder do not receive the support they need. Improvisational theater involves regular exposure to social performance situations, and is recognized as a potential psycho-social support to enhance well-being and symptom reduction. The current study examines whether participating in a school-based improvisational theater program predicts reductions in symptoms of social anxiety. A total of 268 middle and high school students who participated in a ten-week school-based improvisational theater program completed surveys in a single group pre/post design. Adolescents who screened positive for social phobia at the beginning of class reported reduced symptoms of social anxiety at post-test. This change predicts increases in social skills, hope, creative self-efficacy, comfort performing for others, and willingness to make mistakes, along with marginal decreases in symptoms of depression. Given that no prior study has examined school-based improvisational theater training and its relationship to social anxiety, this work offers an important early contribution to the empirical literature on improvisation and mental health. School-based improvisational theater training offers an accessible, non-clinical alternative for addressing social anxiety problems among adolescents.

<comment><USER19> Note that this study doesn't differentiate improve from other performance activities like musical performance, drama/theater, or speech and debate.

</comment></comment><comment><USER20> Improv is absolutely wonderful for social anxiety - the point is to say "yes, and" and every time someone just gives a hard "no" the entire scene stops and everyone can feel it. Social anxiety for many is the fear of social/phsycial consequences for expression - its a perfect "safe" environment by which to learn to express yourself.

<comment><USER21> I'm an active participant in improv and I'd like to disagree right here.

If you have someone who is really being off the rails and is JUST expressing themselves and not really listening/collaborating... It's essentially the same as a `no` to everyone else's idea.

I have a hard time reconciling these things. Where it "should" be safe for "everyone" but not if you're too crazy!

I think some people really get off on the expressive/generative part of improv and not so much on the listening/collaborating.

<comment><USER20> I suppose that there are quite a few people who express their social anxiety by having pressure of speech, which it'd be a downside for, but for myself and most others who go "quiet" out of deference or fear its been very helpful.

If the person doesn't do well at passing the ball in a conversation, often times they can fix that pretty easily by having it explained to them, and given what would normally seem way overly direct/aggressive advice usually works well for those people.

</comment></comment></comment><comment><USER22> The thought of even trying that is giving me anxiety.

<comment><USER23> You too? My whole body tightened up more than a max-effort deadlift just _reading_ the article.

</comment></comment><comment><USER24> I keep seeing anecdotes about the unreasonable effectiveness of improv. Many people say it's in the top 2 or 3 subjects they've studied and learned, regardless of their background/profession. Maybe it's time to look into studying it myself, I'd be happy if people shared any recs on how to get started.

<comment><USER25> You really should. I got schnookered into joining a drama club in high school and it was probably one of the best school experiences that I ever had.

The ability to stand in front of a big room and not be anxious was both liberating to me and profoundly useful. I cannot even imagine the opportunity that I would have missed had I not had that confidence.

As a side benefit, it's alot of fun, I wish I had kept it up!

<comment><USER19> snookered [https://en.wiktionary.org/wiki/snookered](https://en.wiktionary.org/wiki/snookered)

vs

schnookered [https://en.wiktionary.org/wiki/schnookered](https://en.wiktionary.org/wiki/schnookered)

I hope you weren't drunk in high school. Would a drama club allow alochol?

<comment><USER25> Try urban dictionary.

<comment><USER26> Try "slewfooted" =]

</comment></comment></comment></comment></comment><comment><USER27> This basically confirms my experiences, the way I improved my social anxiety and public speaking was with (forced) immersion therapy from the public speaking and communication I had to do in college, and at work to a lesser degree.

<comment><USER28> On a smaller scale, I know many introverts who worked retail or food customer service which brought them out of their shell.

<comment><USER29> It would be nice if Software Engineering jobs would involve more human interaction. With all this Scrum I have the impression the Engineer is supposed to only work on tickets, have no meetings ideally because it's more efficient when Product people etc. do that.

Kind of a bad trend IMHO...

<comment><USER30> Many flavors of Scrum have daily standups where you must (briefly) talk to the rest of the team, and also involve doing pair programming where you are constantly discussing something with another team member sitting next to you. (This last part actually saps my energy and is something I don't enjoy about agile methodologies...)

The problem is that highly technical and focused conversation seems nothing like aimless conversation in social situations.

<comment><USER29> I also think that for most mundane tasks pair programming is more of a burden than an improvement. For really difficult tasks that take hours to just imagine the solution, it can be actually productive to sit together and solve it that way.

</comment></comment></comment></comment></comment><comment><USER31> I read the headline and basically said, "You have checked in on the local High School's Drama club right?"

At least in the Bay Area, Drama Club seems to be a safe place for socially awkward teens.

<comment><USER32> That's how it was a decade ago on the east coast as well, when I went through it (tangentially, admittedly).

5-7 well adjusted folks who are looking to act for attention (nothing wrong with that, to be clear), 25-30 folks who feel weird in their own skin and see drama club as a way to bond with people like them.

</comment></comment><comment><USER33> I never did improv, but in high school, the way that I got over a lot of social anxiety was learning to be funny; making my peers laugh was a good enough way to break the ice and has really helped me in my career. Granted, in high school the only trick to making a joke is to imply something sexual or say a swear word, but the principles still basically holds.

Picking up a few joke books was really helpful to me, so I suspect that taking a class might be even better.

<comment><USER34> If I learned one thing from my 201 class at Upright Citizens Brigade theater, it was that I'm not as funny as I think I am and that I should stop those "funny-man" impulses from high school. Humor has a way of controlling the conversation and tone. Improv is about sharing and cooperating over that control. You share a stage with 15 other folks who want to relish in that limelight, and you quickly learn what happens if you don't cooperate.

I only bring this up since improv helped me think about social situations in a very different lens. It's like taking up a recreational sport - you'll think you're fit until actually pushed into a challenging environment.

</comment></comment><comment><USER35> When I was in middle school in the late 70's I was exposed to "Interp" \- dramatic interpretation, a form of debate that is dramatic and improvisational. My school never developed its own team, but I envied the schools that did. Looks like it is still going strong: [https://www.youtube.com/watch?v=LNDZ9yblPpI](https://www.youtube.com/watch?v=LNDZ9yblPpI)

<comment><USER16> When I was in high school, that superset of debate and Interp events was called forensics, and joining my high school's team was one of the best decisions I ever made.

Memories for life and public speaking skills to go with it.

<comment><USER35> I can imagine the revolution if this ever went mainstream. Talk about getting kids out of their social anxiety shells.

</comment></comment></comment><comment><USER36> I know this is a bit tangential, but my 2nd grader has had a string of "public speaking" assignments and it struck me as being odd (ie 7yo is kinda young for something like that). But apparently it is a bit of a trend in schools and they very much do it that early on purpose. It builds up confidence slowly before social awareness really kicks in. I thought it was a brilliant idea.

<comment><USER37> I'm told that most people fear public speaking more than they fear death. I'm not wired that way[1], but it seems like a logical thing to help people overcome at a young age.

[1] Which isn't me saying I'm a badass or anything; I just don't especially fear public speaking in particular.

</comment></comment><comment><USER38> In public speaking, I used to turn red, sweat and babble... In business school, a three day seminar with theater actors made me realize it didn't have to be like that - it unlocked my attitude and set me on a self-improvement path... I am now at ease speaking anywhere anytime - it took a few years but improvisation with actors has been key to start heading there.

</comment><comment><USER39> I wonder if this is limited to improvisational theater or if something like a roleplaying game, where the person has to act out a role with no script, can help as well. I would think that they get to perform the same sort of social interactions and while still being "low cost".

<comment><USER40> I'm guessing that the best script for people with some kind of anxiety is: acting that you simply don't care. That way, you don't look stupid, just a little uncooperative or disinterested. From there you can build on confidence, thus keeping your head from overflowing with negative thoughts, and actually making a more interesting performance.

</comment></comment><comment><USER41> I've found improv to be frustrating (but not anxiety inducing) in the sense that you can "yes and" a topic to a place that's difficult for someone else to pick up.

Almost like you're playing catch, but you tossed the ball to a place that makes it too hard for your friend to catch. Another example would be writing a comment (even this one) and consciously making sure it's easy to reply to, rather than just expressing myself.

I suppose it's like having a binary tree, and trying to pick a node that's not too far down the tree.

I find myself having to take a step back and consider what I'm going to say and if it makes sense (in general) for someone else to continue off of rather than just talking.

</comment><comment><USER42> I'd take an improv class as an adult if it weren't so nerve wracking.

<comment><USER43> You should do it! I took a year of improv as an adult (30's) and found it an incredibly valuable experience. It benefitted my communication skills tremendously and reduced the anxiety I had talking to strangers.

"yes, and" helps with the communication protocol. You're more constructive and work harder to clearly communicate your points.

"you don't have to say anything clever" takes you off the spot and reduces the pressure. If you're less comedically included, you can still heighten the stakes by interpreting what's going on and going along with it. The audience just wants to see human drama, and the most truthful thing is how you feel about what's happening. Just to do it -- it's that easy. There are no expectations.

Improv teaches a _meta language_. Your intent, what you're going to do with the scenario, how you feel about each other and the world around you -- none of this has to be spoken. You're having a dialog with your scene partner(s) and communicating to them in realtime outside of the actual words that you're speaking. This is especially important, because so much of real life communication is non-verbal. Improv opens you up to that whole world.

There are no wrong choices. Yes, there are more optimal choices, but there are no bad ones as long as you follow the simple rules of improv. You don't even need to worry about that until you become serious about improv as a hobby.

You'll find analogs to all of these points in your own communication with friends, peers, superiors, et al.

Improv is one of the best courses on human communication. It's also cheap, fun, and highly effective.

<comment><USER44> > Improv teaches a meta language. Your intent, what you're going to do with > the scenario, how you feel about each other and the world around you -- none > of this has to be spoken.

Thanks for articulating that insight. I've looked into improv off-and-on for years but have not found the clinching argument on its value-add in my situation. I'm fortunate enough to not be prone to social anxiety (I'm a people-connector type and am actually drawn to strangers), and public speaking doesn't faze me, so I haven't really been able to see how improv would fit into my life. But improving non-verbal communication is a meta skill that I could get behind.

I'm still not sure how "yes and" would apply in improving daily conversation though. Do you have any positive experiences to share?

</comment></comment></comment><comment><USER34> I work remote, and I am certain that improv can help a virtual team read eachother on calls and engage in "groupthink".

I don't know how I can alter a class so it's Zoom-ready, but maybe this research will help me convince some Guinea Pigs at my company.

</comment><comment><USER45> If anyone's interested in checking out improv musicals, check out the excellent and hilarious Off Book podcast: [https://youtu.be/Ivf58k53nu0?t=132](https://youtu.be/Ivf58k53nu0?t=132)

</comment><comment><USER46> Wet blanket researcher: No randomization, purposive sampling, high attrition, this is not great science. Doing randomized trials for this sort of thing is difficult but necessary. A correlational pre-post study is a good start!

</comment><comment><USER47> I've never encountered with "improvisional theater". How does it work? What practically is "performing without a script"? You stay on stage and say anything that comes to your mind?

<comment><USER48> You know when kids play make-believe? Where one kid is a knight, and the other is a dragon? That's improv.

Improv can take a lot of forms -- there's one style that's basically "make up a movie, complete with narrated camera moves". There's another that's "a series of unconnected scenes". My team is doing one now that is basically an improvised podcast or radio play.

Simply "standing on stage and talking" is bad form, because it's not especially interesting. How many tv shows are just people talking? None; people are doing _something_. Sometimes those people are very active, like if they're riding a dragon into combat. Sometimes those people are less active, like if they're a radio host talking to callers. But never are people just standing: they're adjusting the radio microphone, or flipping a page to read off of.

<comment><USER49> > You know when kids play make-believe? Where one kid is a knight, and the > other is a dragon? That's improv.

So, role playing games?

<comment><USER21> It's best not distilled that way. But some forms of improv can resemble things like role playing games. But not all.

<comment><USER48> And some forms of role playing games resemble improv, but also not all. :)

There's definitely overlap!

</comment></comment></comment></comment></comment><comment><USER50> It can also make them really annoying at parties. But hey, I guess they got over their anxiety, so that's good.

</comment><comment><USER51> I think VR can be used for this also. Emotional and social intelligence training is the best use of this tech.

</comment><comment><USER52> There's an entire self-help book on this: Improv Wisdom, by a Stanford instructor who teaches improv.

</comment><comment><USER53> I wanna do improv since past 2 years, but there's nothing in Pune city.

</comment><comment><USER54> I feel like I got a lot out of my college LARP group for similar reasons.

</comment><comment><USER55> Worked for me and it was my favorite class in middle school.

</comment><comment><USER56> Works for adults, too.

</comment><comment><USER57> You mean, interacting with people in 'safe' real life helps with anxiety about interacting with people? Shocker!

</comment><comment><USER58> Teens? How about adults!?

</comment><comment><USER59> Worked for 25-year-old me.</comment>


Gmail marking email from me as spam - <AUTHOR> https://email@example.com/msg08806.html <comment><USER4> The email landscape is substantially more complex now than it was three decades ago. The poster seems to be frustrated that he is mostly getting technical explanations to his rather emotionally fueled question. He is literally phishing for an argument on the morals of a big company deciding he is a potential spammer without absolute proof. He made it clear that his opinion is that there should be a system in place at Google that he can petition for his reputation, and that emails that look like spam should say 'potentially spam' rather than send it to a spam folder. All this while sending emails from an origin that is very well known for spammers and whose actions depict a typical bot account warming up. (Poster claims to on occasion initiate personal conversations with people who have their emails publicly listed, such as an author from a news site he might disagree with). There is just no reliable system today that allows for someone to host their email servers in a spam haven, not use relays, act like a spammer, and expect not to get your emails sent to spam folders.

It's insulting to the folks that are pouring their time replying to what they think is a user in need of technical assistance, only to eventually stop replying when it becomes obvious that he only replies with questions of morality and treats most answers as if they were platitudes specifically attempting to dismiss his efforts to communicate.

<comment><USER5> I don't think OP stop replying, he seems pretty active on another thread of the same mailing list concerning another big player (MSFT) filtering out some messages for some reasons. [https://www.mail- firstname.lastname@example.com/msg09066.html](https://www.mail- email@example.com/msg09066.html)

As a hobbit* , I'm actually pretty happy this whole conversation is starting to gain more traction.

My domain is always green on every test I could run (mailtester.com & all) yet some of my mails go to the junk without any warning. At least warn postmaster when you decide to block a domain because of bad neighbour/hosting!

* This specific reply of the chain is a must read and is where learned that I'm actually a mail hobbit. [https://email@example.com/msg08958.html](https://email@example.com/msg08958.html)

<comment><USER6> Even after reading that, I'm not sure what an email hobbit is. Someone who just wants to live in a cozy hole in the ground with a few nice email amenities?

<comment><USER7> It was an extended metaphor categorizing an ISP's (or a cloud provider) customers into multiple buckets: Kobolds, Lizardmen, Hobbits, and Princesses.

Princeses were the customres which sometimes required a lot of customer support, but paid really well because they purchased a lot of services/VM's. Hobbits were customers who basically minded their own business, and didn't guy a lot of stuff, but didn't cause any problems.

Lizardmen were customers which had possibly clueless marketing departments that would occasionally send e-mail marketing campaigns which crossed the line, but which also sent plenty of legitimate e-mails, and which were mostly trainable after the ISP/cloud provider smacked them on the nose with a rolled- up newspaper --- and Kobolds were spamvertising companies that did nothing but send spam, and were constantly switching ip addresses / VM's, and knew d*mn well that they were doing something evil, and were not trainable.

The problem is that if the ISP / cloud provider has the vast majority of their customers being Kobolds and Lizardmen, it becomes economically hard to just cut off the bad actors, because the vast majority of their customer base are bad actors. If the provider originally had a good mix of all of these customer mixes, but wasn't proactively cutting off the Kobolds and educating the Lizardmen, then their network block would start getting a bad reputation, and all of the major e-mail providers (not just GMail) would start blacklisting the netblock, or at the very least, treating any e-mail sent from that network block as a strong SPAM signal, and this would cause the other customer types to find other providers.

So if you happen to be a customer of such a network/cloud provider, you could try complaining to the provider, but they are really in between a rock and a hard place, because if you're the 1 good hobbit and all of the otheer customers for that provider are Kobolds and Lizardmen, what is the provider supposed to do? You're probably better off switching to a better provider. Unfortunately, the better provider may also cost more, but you get what you pay for.

</comment></comment></comment></comment><comment><USER8> I would argue in this narrow case - I would not point to nefarious undercurrents.

Google's ML mail spam models are that way because there's too many bad actors. Email, even in 2019, still extremely valuable.

If you can get into people's email inbox, there's still quite high conversion rate - better than clicks/videos on a pure percentage basis. Even the author wants to reach out to people in a cold fashion.

As a gmail user, I 100% appreciate keeping my inbox extremely clean. Google's spam filter is a welcome addition.

As a business owner, I'm continuously having to do things so I don't end up in spam. Less images, specific wording and the list goes on.

I can't have it both ways. I would not be on Gmail if it was like yahoo. My goodness... my parent's inbox, on yahoo, makes me anxious.

<comment><USER9> > Google's ML mail spam models are that way because there's too many bad > actors. Email, even in 2019, still extremely valuable.

Google's spam models freaking suck donkey dick. I use Fastmail but my university uses Google for email. A service we use (Piazza, class Q&A) sends notifications via email. Over the years I have gotten probably about 1k email notifications from them, and replied to a few of them too. Google has recently started marking about 5% of their notifications as spam. 95% of the notifications come through; 5% get rejected. There's no discernible logic to this.

There are organizations and people within the university which have me on their mailing lists; with them too 90% of the emails get through but 5% get randomly marked as spam. And they're perfectly normal emails that Google just decides to move to spam. Even with people that I have had email conversations with, it just spams random stuff.

Every couple of months I check my Fastmail spam folder just for the lolz of seeing what's in there. With Google I _have_ to check spam every day or I'll miss important stuff.

There are tons of people who have posted very similar experiences on HN. On this thread I just saw another person comment that Google's own emails go to spam - I've observed this too on my personal Gmail account which I occasionally check.

The spam filters have a tough job, but other companies are succeeding at keeping all the spam out without a gigantic false positive rate. Google's filter is hilariously bad.

<comment><USER10> I'm not defending gmail, but I will say that some users simply suck (unintentionally). I bet some of your peers are just hitting the spam button instead of simply deleting the email, generating a bad reputation for the sender.

I've seen this with my SaaS. We ONLY send transactional emails like purchase receipts and login links (using postmark). We're very intentional about not sending anything remotely resembling marketing or spam, and sending as little as possible. Yet I still get the occasional user who will flag their purchase receipt as spam when all they really mean to do is delete it. They don't get the difference and don't realize that it matters which button they click.

I've noticed that in the past year or so gmail has made hitting the spam button more explicit and now have a confirmation message. Obviously they did this because some people were simply mashing that spam button instead of the delete or archive button.

I don't even blame the users. Most don't know the impact it has. But bad actors sending tons of spam have trained them to be cynical of any generic form email.

<comment><USER11> May be you should not send purchase receipt by default? I don't know why everyone does that as I don't care about it at all. If it's not required by law, add button to your UI which will mail receipt (or better just let user download it) and that's about it. While I'm not marking those receipts as spam, I'm sure that some users would do that as it's unwanted mail for them.

<comment><USER10> We actually have users contact our help desk asking for the receipt not realizing it's already in their inbox or spam folder.

I've considered adding an opt-out checkbox during checkout but I honestly doubt it would ever be unchecked and if it was opt-in we'd have users contacting support asking for it because they didn't realize they had to check the box. Either way it's one more toggle on the UI which I'm not a fan of unless absolutely necessary.

I'm curious if you've run a business and experimented with not sending receipts, and if so how it played out?

</comment></comment></comment></comment></comment><comment><USER12> I run an email security/monitoring business [0], and we hear this issue with Gmail a lot, in fact: we have been bitten by it as well during the early days.

The problem at hand is that it is really hard to debug situations like this. Google won't tell you why a particular email is blocked, because that information will be immediately exploited by spammers.

We build our software suite because we found that mistakes in email configuration are easy to make and really hard to identify.

I did a quick scan of rafa.eu.org and found the following:

\- SPF is setup [1] with a neutral 'all' mechanism, which basically disables SPF for this domain. Hence, DMARC will set SPF as 'pass' even though SPF did nothing here to help the spam algorithm to assess the sender.

\- A DMARC record exists and has a valid syntax [2], but the proposition (p value) is set to 'none', which basically disables DMARC altogether. It will enable reporting, but nothing more than that.

\- I don't know you DKIM selector, so I can't assess that, but make sure the DKIM signature address is aligned (using the same domain name).

\- The TLS configuration of your email server uses a self-signed certificate and is very much outdated (it offers SSLv3 and RC4 ciphers). I used testlssh.sh [3] to check this.

So yes, you technically did setup TLS, SPF and DMARC, but in all 3 you have configuration errors. Running a mail server in 2019 requires a bit more work and maintenance than it used to be 10 years ago, but the reason is spammers, not Google. Remember: it is in Google's best interest to have as few false positives, as it ultimately benefits the users. It's just really frustrating that it is hard to figure out why Google is marking your email as spam.

[0] [https://www.mailhardener.com](https://www.mailhardener.com)

[1] [https://www.mailhardener.com/tools/spf- validator?domain=rafa...](https://www.mailhardener.com/tools/spf- validator?domain=rafa.eu.org)

[2] [https://www.mailhardener.com/tools/dmarc- validator?domain=ra...](https://www.mailhardener.com/tools/dmarc- validator?domain=rafa.eu.org)

[3] [https://testssl.sh/](https://testssl.sh/)

<comment><USER13> No, it's incorrect to say that running an email server requires just "a bit more work". Google basically no longer allows low volume servers to send mail.

I wrote more about this here:

[https://www.attejuvonen.fi/dont-send-email-from-your-own- ser...](https://www.attejuvonen.fi/dont-send-email-from-your-own-server/)

<comment><USER14> My low volume server can send mail very nicely. You can't however do your job half-assed, lack of strong SPF/TLS/DKIM isn't okay from both Google's and anti-spam perspective.

<comment><USER13> If you would have actually read the link in the post you're replying to, you would know that Gmail is not accepting mail from me despite that I've spent a huge amount of time in verifying that everything is configured correctly.

Also: please stop spreading unsubstantiated claims about your magical server magically sending mail past Gmail's spam filters. There's virtually no chance of that being true, and if it somehow was true, extraordinary claims require extraordinary evidence.

<comment><USER15> I run a personal mail server for just myself and I've been getting full delivery to Gmail so far as well. Same with Outlook.

I'm waiting for the day they try to shut me down.

<comment><USER13> Can you show any evidence for these claims?

</comment></comment></comment></comment></comment></comment><comment><USER16> Google's making us play by their rules:

1\. Our ML models see that your emails spam? You're spam.

2\. You're using Tor or a VPN? Here's 10 reCaptcha's.

3\. Don't use Google Webmaster or use AMPs? Good luck finding your page on the 1st page.

4\. Youtube video includes education hacking tutorials? Demonetized.

Obviously some alternatives are moving to Fastmail and using Duck Duck Go, but we need Google to stop this "my way or you don't exist" attitude before they become a huge conglomerate which controls major aspects of many people's lives (which they arguably already do) and leaves everyone (like small companies that don't use Google or just regular people who don't want a Google account) separated from the world they're forming.

<comment><USER17> "Moving to Fastmail" is such a popular alternative that if they ever get to any kind of reasonable size we'll be in the same position all over again.

That should not be the suggestion. Many smaller providers encourages the competition necessary for an open network, not a small number of huge providers.

<comment><USER18> I've been using Fastmail for about 5 or 6 years now. My only problem is the 1GB mailbox size limit I have. Maybe I should upgrade. Otherwise I love it. Their webmail client is phenomenal and works with mobile and desktop. I don't like their Android app but I haven't tried it in a year. Their web client is so good I don't need an app for them.

I do see mail going to spam though. It's never bugged me that much.

<comment><USER19> You can also just use the native imap client provided by your operating system. The beauty of using a provider and service that adheres to open email standards.

</comment></comment></comment></comment><comment><AUTHOR> I'm actually having a similar problem recently, although my messages are declined at the SMTP stage, "due to the very low reputation of the sending domain", which I attribute to the output of certain cron jobs being sent towards my own Gmail account — it seems like my whole primary domain name has subsequently been blacklisted a few weeks ago:

* [https://news.ycombinator.com/item?id=21340460](https://news.ycombinator.com/item?id=21340460)

* [https://mailman.nanog.org/pipermail/nanog/2019-October/10381...](https://mailman.nanog.org/pipermail/nanog/2019-October/103817.html)

It also seems that more and more of the mailing list posts end up in the Spam folder, too. For example, I'm subscribed to the nginx list, and recently found that all the mails from one of the core developers are now in the Spam folder in Gmail.

Gmail is becoming less and less useful as days go by. If it cannot be used for the mailing lists anymore, and inbound forwarding into the account is so unreliable as well, leaves fewer and fewer reasons to continue using it, especially as it's no longer free as they've stopped their infinite storage growth, so, I now need to pay 1,99 USD/mo because I bought into their infinite storage claims back in the day, and subscribed to way too many mailing lists to fit in 15GB of space.

<comment><USER20> I have a hotmail throwaway account (it has since been migrated to outlook with the microsoft buyout) and its spam filter is completely backward. There is currently 373 e-mails in my inbox (100% spam, expected since it is a throwaway...) and 4 e-mails in my spam folder--100% of them the account registration/confirmation e-mails that I was wanted.

I have migrated to fastmail for all of my personal e-mail, which I have been pretty happy with. So far I haven't caught any legit email which has been categorized as spam and negligible spam that has made it through into my inbox (mailing lists notwithstanding--I sort them using automation rules). My one complaint is that there is no way to turn off their "thou shall not receive executable attachments" filter so I still need people to play the "set windows to show the file extension, change it to .txt, and then attach it" game when people want to send compiled code via e-mail.

In any case, it is certainly a much better situation than the gmail spam filter I used while in school, which had about a 10% chance of marking outside messages as spam--in fact it even started marking e-mails sent by my professor (via the university gmail) as spam...

<comment><USER17> > it has since been migrated to outlook with the microsoft buyout

Aside: Microsoft bought Hotmail in 1997! How old is that account?!

<comment><USER20> Honestly not sure, but certainly sometime in the 20th century. It was my main e-mail back then, I migrated to Yahoo! shortly after the Microsoft buyout and kept the HoTMaiL as a throwaway. When Google announced Gmail and the 1GB (with projection to infinite) storage I switched to them as soon as I got an invite, but when they changed their slogan from "Don't be evil" to "Do the right thing" with the Alphabet restructure I ditched it and moved to my own domain with Fastmail. If Fastmail starts to show signs of evil it will be as easy as updating my domain records to point toward OurMailIsActuallyGoodIPromiseThisTime.com and click through the certificate warnings on my mail clients.

</comment></comment></comment></comment><comment><USER21> I thought this had been getting better. At least more e-mail from my personal server has been getting through to its destinations. I've written about this too a few years back:

[https://battlepenguin.com/tech/how-google-and-microsoft- made...](https://battlepenguin.com/tech/how-google-and-microsoft-made-email- unreliable/)

..and I reference the Hostile E-mail Landscape blog post by Ribton (really good btw).. which is gone now and redirects to some bullshit software site (there are archives of it).

I've got DKIM, SPF, DMARC, reverse DNS for IPv4 and v6 and I still don't have the volume to show up in the postmaster tools Google has.

My own e-mail server occasionally has spam come through. I really don't care. I sometimes mark it to be learned, but honestly I'd rather have some SPAM get through than get HAM marked and placed in the big fucking SPAM bucket I never check.

But for big providers and dangers like ransomware, this isn't an option. So we have shitty unreliable e-mail (even on gmail back in 2012; I had delivery issues to other people on gmail).

Where does the blame go? The 1% of people who give scammers money, get ransomware infections and click on everything? Over aggressive spam filters? The lack of real e-mail server diversity? Who the fuck knows anymore.

<comment><USER22> > _but honestly I 'd rather have some SPAM get through than get HAM marked and > placed in the big fucking SPAM bucket I never check._

This is why I like that my email provider ([https://mailbox.org/en/](https://mailbox.org/en/)) simply rejects all messages detected as spam: False positives don't get lost in a spam folder I never check, instead the sender gets a mail delivery failure notice by their own mail server and at least get a notification that they couldn't reach me.

<comment> em-bee but what if email is the only way to reach you?

so i got a notification that you didn't get the email. fine. i still have no way to talk to you unless i sign up to some other email service and try it again.

depending on the reason to email you, i may not care enough. (maybe i wanted to buy your service)

if a customer emails me and i don't respond because i didn't get the email, i have a problem, not the customer.

the better solution would be both. respond with a delivery notice, AND deliver the email to my spam folder.

<comment><USER23> > but what if email is the only way to reach you?

I have no idea if it actually is, but it _might_ be a legal issue too in some jurisdictions. German law for example requires commercial sites to have an option for quick and electronic communication - Email is generally a valid choice. If your mail server rejects incoming mail (be it because it's overly sensitive, offline, or you never bothered setting it up etc), that could very well be considered as non-compliant.

<comment><USER24> Wouldn't a simple contact form on a website meet the definition of quick and electronic communication?

<comment><USER23> Yes, but that has it's own challenges with spam protection, privacy laws, users not spelling their email correctly etc, and you still need an email address to receive those messages (or check for new form entries regularly), so an email address is often used as a shortcut.

</comment></comment></comment></comment></comment></comment><comment><USER25> I share the sentiment and the frustration.

That said, if you read far down enough in the thread, it's pretty clear that the OP has very, very unrealistic expectations about others' obligations to him, and very incorrect notions about how statistics work. The argument eventually turns to "well google has a lot of processing power at its disposal so I don't see why it can't use this in order to turn incomplete data into more complete data" and that's when I checked out :\

Edit: also this Brandon Long has the patience of a saint.

</comment><comment><USER17> The attitude of the replies he gets lower down in the thread is shocking. I hope whatever mailing list that is doesn't have any kind of good reputation, because it seems like the consensus opinion there is shockingly anti- cooperative, anti-internet, "me before the network", even going as far to say "if Google thinks you're a spammer then you probably are".

How things have changed.

<comment><USER26> That's a meeting of technical and moral view. I'd love for corps to have no impact on people, for distributed knowledge and resources, for everyone being in control of their data, and general socialist direction.

But the technical answer to "why is my email landing in spam" is still "follow best practices, build up reputation, and you'll be fine... unless Google decides otherwise, which they can". Posting about how unfair that is is not going to change anything.

<comment><USER27> > Posting about how unfair that is is not going to change anything.

It won't change anything, unless enough people complain. And even just recognizing that things are not right is the first step required to make a change. Of course, I won't hold my breath waiting for Google to change their ways, but I think it is in fact important to make a clear note their is not the right way. As the guy writes:

"Maybe it is just so, that big companies ignore small external senders who are sending mail to them, simply because they can, and we can't do anything about this (however, it's always worth trying to check whether we really can't - <USER2> that's what I'm doing by posting my issue on this list). But even if we can't do anything about this, we should not pretend that everything is OK, they are correct, and it's the sender's fault. No, on the contrary: we should state it explicitly and clearly that this behaviour is not OK. Maybe we have to live with it; but it doesn't make it less bad."

</comment></comment></comment><comment><USER28> Might be a good time to mention a new development with Yahoo Mail. They now appear to use a form of shadow ban.

They will accept an email, with all outward symptoms being normal, but it will not appear _anywhere_ in the user's mailbox. Not even in the Spam folder.

This started few months ago when we were sending out our (rather infrequent) newsletter. 5th or so message to ...@yahoo.com address failed with

        421 4.7.0 [TSS04] Messages from xx.xx.xx.xx temporarily
              deferred due to user complaints - 22.214.171.124; see
              https://help.yahoo.com/kb/postmaster/SLN3434.html
              (in reply to MAIL FROM command)

That's with just a couple of seconds in. They kept replying with 421 for a while, but later switched to just dropping connections. Then, in the course of the next few days, they would accept 1-2 emails and then go back to denying them again.

As I mentioned above, the "best" part is that none of these emails would actually make it into recipient's mailboxes. They disappear.

Better yet, all subsequent (non-bulk) emails have the same fate as do _replies_ to emails from Ymail users. This last bit is what makes this look like a shadow ban.

All attempts to contact Yahoo went unanswered, including those made through Gmail. Ditto for whitelisting requests through their web form.

If anyone has any ideas what the hell's going on, I'd be curious to hear them.

<comment><USER29> I don't think this is new. Many email providers will throttle / blackhole you if they decide you're sending too fast or see too many bounces.

Slow down your send rate, make sure your server is configured correctly, and prune your lists.

<comment><USER28> "Many providers" \- any concrete examples? For an SPF/DKIM'd server that's been at the same IP for nearly a decade and had not a single spamming incident.

PS. The list is immaculately maintained, the server is configured as properly as it gets and very conservatively rate-throttled. There's nothing obvious.

<comment><USER13> Gmail also occasionally blackholes mail.

</comment></comment></comment></comment><comment><USER30> Gmail's ML models categorize plenty of Google's own emails as spam.

I honestly don't think they are nefarious, just overly insistent.

I remember email before Gmail's spam filters. Everyday was a task of deleting spam first then responding to emails.

<comment><USER31> This is the consequence of the spam filters doing a good job. I too remember when consumers would clamour for better spam detections and different companies would hawk products that purported to do so.

But now that spam detection mostly works well for most people, it's a forgotten issue. Now we have people on this thread questioning Google's authority to filter spam because from their PoV, Google isn't adding much value to begin with.

<comment><USER32> The problem is that Google errs too much on the it-is-spam side. Having to delete a few spam mails isn't a big issue if the alternative is losing legitimate mails because of overzealous filters.

<comment><USER33> That may be true for us, tech savvy users.

Imagine your grandma getting phishing emails from banks. If only 1% goes through, because Google is a bit more lenient on the detection, I can see the phishing success rate going through the roof.

It's a very hard problem to solve.

</comment></comment></comment></comment><comment><USER34> Sandstorm should have really caught on. We need to make self hosting drop dead simple so that even non-technical people can do it. The power of these mega- tech companies can only be whittled down if we take collective, well thought out action. And it is not just about tackling power imbalances. Owning your own data is a fundamental aspect of our digital lives which ought to have become simple and natural, given Moore's Law and the phenomenal device capabilities in recent years, yet somehow we are going the other way. Even as our devices become ever more capable (hardware wise), the software is moving to ever more to the Cloud and we're being locked out of the full capacities of our devices except in the narrow realm of media consumption.

<comment><USER35> Since people specialize, we have organized societies and legal systems so that people can use services with reasonable trust.

With new technology, the laws of course lag. You see a wild west, then monopolies etc before it settles.

</comment></comment><comment><USER36> Gonna second whoever said: Use DKIM.

I actually check Spam daily now, after finding support emails whose logs got snagged in the dragnet & personal stuff — even email from my _Dad_ got shunted to Spam.

I still tip the hat, though, to Google, whose algorithms are wise enough to assiduously stow both the "jump on a call" and "donate to my campaign" emails to spam.

<comment><USER37> The _vast majority_ of messages in my spam folder are not spam on Gmail at the moment. Most of them are from providers I've semi-regularly tried to get it to recognize as not spam for months or years. Of course Gmail also outright rejects a lot of stuff at the SMTP stage, so their filters are better than they look, but the amount of misclassification is extremely frustrating.

<comment><USER36> I have this issue with OSSEC messages (automated messages from an Open Source IDS). Same thing, too, have been marking them as "not spam" for years. I think it's because I don't always review them...:/ But just because you don't always open a message class doesn't mean it's spam, Google!

<comment><USER37> At one place we even had them start bulk-rejecting messages for our Google Apps domain from a whitelisted source.

The entire point of the whitelisting was to allow our internal mail server to forward transactional messages to _addresses on our own domain_ , so it was not even that it was ever reported as spam by anyone, or ever hitting external addresses where I'd understand if Google was more trigger happy.

It was ridiculous realising that they basically had no way for even paying customers to tell them that "that e-mail server we told you is ours and we always want to receive _all_ e-mail from no matter how spammy you think it is, using the mechanism you say it there to do exactly that, well, we _really mean it_ ". Instead it'd randomly just stop working. [this is a few years ago; who knows, maybe they've fixed it]

</comment></comment></comment></comment><comment><USER38> Email is not the same it has been for years/decades. These days you need to use all the security tools available, most notably TLS and DKIM, set a strict DMARC policy, and don't do anything unexpected such as sending lots of emails out of the blue.

Email is all about reputation: it can take a lot of effort to build trust, but just one bad move and it's all gone.

<comment><USER23> > Email is all about reputation: it can take a lot of effort to build trust, > but just one bad move and it's all gone.

Is it though? That may well be true for small individuals, but the Spam I get these days comes via the large mail providers, Amazon SES, sendgrid etc, and nobody in their right mind is going to block them. "Too big to block" is a thing now, "too big to have to care about abuse reports" is too.

<comment><USER38> Yes, the big mail providers are an issue. But it's also an issue the other way around. We use Postmark at my place of work and unfortunately, that doesn't prevent all of our mail from being marked as spam. Some mail gets blocked because it comes from the same IP address as some spam mail. I know that Postmark puts effort in preventing spammers to use their service but Amazon probably doesn't care a lot about people misusing SES.

<comment><USER23> Wouldn't that imply that they have a lot of spammers on their network? As far as I understand, the value proposition is both "you don't need to worry about email" and "we have so many customers (sending legitimate emails that are very similar in content) and so little spam, we have great reputation and you're inheriting that reputation".

If that doesn't work out, is their reputation not good enough? Is the receiving server too strict? Are those blocks occurring on major providers or for individual servers where an annoyed person might just blacklist an IP if they get spam from them more than once?

<comment><USER39> Note that "a lot of spammers" isn't really a lot in laymen terms. It varies by provider, but last I worked on this if anywhere from 2-5% of all emails you send are clicked as spam the provider will start blocking you.

We got blocked once because we got a single spam click: the ISP said they have a zero tolerance policy. We had years of sending hundreds of email per week without a spam click.

</comment></comment></comment></comment></comment><comment><USER40> OP already received a decent reply,

> A quick look at Talos Intelligence reports a 1600% increase in email from > your domain over the last day; email reputation os 'neutral' at this time. > Not good, not bad, but neutral.

> Also you're on OVH, about which a quick look through the list's archives > will possibly prove instructive. It's reasonably likely (as likely as not) > that you're running on an IP in a neighbourhood with some poor neighbours.

> As Mathieu pointed out in his second email, building a good reputation takes > time. Losing a good reputation is a matter of rather less time and can be > influenced by factors outside of your control.

The rest of that reply chain is the usual debate:

OP. My domain's emails are marked as spam by Gmail

REPLY. Their system, their rules

OP. I don't accept that

Repeat ad nauseam.

<comment><USER17> Ah yes. There are lies, damn lies, and statistics.

A "1600%" increase in email could be explained by 1 email yesterday, and 16 emails today.

Which is in fact what he claims lower down in the thread.

<comment><USER40> The final exchange between OP and someone who works in detecting spam:

[https://www.mail- email@example.com/msg09020.html](https://www.mail- email@example.com/msg09020.html)

This is where HN discussion should pick up, not at the beginning where we need to rehash everything.

</comment></comment></comment><comment><USER5> This finally get me to the point of understanding the "bad neighbor" signal that probably get my personal email (and some of my jobs) rejected from Gmail.

Reading the thread was very instructive, but the most irritating point is that this rejection is mostly silent even if you send only 20 mail a week from your email address and all DKIM & all are in place. Lots of pedantic comment in this thread however, it seems that nasty provider are well known by experts, but for the love of internet, please spread your mighty feedback to small-folk so that we can act accordingly.

Gonna open a ticket at my provider OVH (now I know).

<comment><USER41> Ok, so the way OVH can solve the problem is the same way e.g. Amazon SES solves the problem.

1\. They block all outgoing mail (outgoing 25, 465 and 587) and only open it up once you have submitted a ticket where you decribe for what purpose and how often you want to send mail (if you start off wanting to send a million mails per week they won't open outgoing mail at all). Manually review of those tickets is required so that bad actors can't get through. Difficulty of getting through the review needs to be adjusted upwards till it is too much work for bad actors (and they switch to another provider).

2\. Monitor bounce-backs and have a spam reporting form for OVH IP ranges. Start blocking outgoing mail pretty soon if a small number of those arrive. Of course this is a cut-throat business ("email marketing"/spam), so competitors may submit false spam reports...

3\. Therefore, for (2) you'll have to keep some records of the mail sent, i.e. you'll require mail to be send via relay... The review and the mail relay costs something, so you make it a monthly/per mail cost. Voilà, you have Amazon SES.

Currently they probably block individual instances if they get a lot of complaints about one sending a lot of spam. They can't distinguish legitimate mail from spam because it is all SSL encrypted.

<comment><USER5> Until now I trusted them so I use their own email servers. They are perfectly aware of my usage since while sending they even implement a trick so that one can't send bulk email from their SMPTP.

And that's why I now fear they are utterly incompetent...

</comment></comment></comment><comment><USER42> I feel like there is no way to do this. I was once very happy when I finally got my own basement email server to work (I tried it off and on for years before often giving up), running on Arch. For some time I was the happiest person in the world, I could send emails of >100 mb to my wife, have unlimited storage, unlimited aliases (which can even go to multiple people). I could email my server and make it do things. I was using TLS (whereas me.com and hotmail.com at the time were not!), I was using the internet the way it was intended!

But then Outlook.com stopped accepting my emails, they were not even in any spam folders. Several emails later I got a response from someone at MS: We don't know how the filters work, they are black boxes to us. Sorry, try to write emails that don't look like spam...

Soon after I gave up on my server. Maybe you can use a third party (paid) smtp forwarding service?

</comment><comment><USER43>
        I repeated this for three different accounts

Registering multiple Gmail accounts from the same machine in close succession almost certainly isn't helping you _not_ look like a malicious user.

</comment><comment><USER44> "It is always better to let the possible spam go through than to filter out something that is not spam."

Spoken like an email sender, not an email receiver.

I'm _deeply_ sympathetic with the sender, and not inclined to defend Google here, but I'm also aware that Google is a very large target with a lot of bad actors specifically targeting them, and I just checked my spam folder in gmail: wow!

The problem here is spammers, and a communication system that assumes good faith on the part of senders. Gmail is choosing one way of trying to counteract that problem which is causing problems for people in certain IP blocks, and that sucks. But the problem is still spammers, not Google.

</comment><comment><USER45> This guy has an odd (seemingly nonsensical) quote in his email signature.

To a machine learning algorithm, it might look kind of like the random nonsense some spammers put in spam to corrupt Bayesian filters. He might want to try temporarily taking out that signature if he's using that as part of his emails.

<comment><USER46> He tests this particular hypothesis here: [https://www.mail- email@example.com/msg08825.html](https://www.mail- firstname.lastname@example.com/msg08825.html)

<comment><USER47> He could have ruined his reputation with that signature?

</comment></comment></comment><comment><USER48> I also have my own mail server, and I have the same problem.

I'm guessing this is machine learning algorithms confusing correlation with causation once again. Being a small private mail server is strongly correlated with being a spammer. So even if you are not relying any spam, you certainly look like someone that would. And even if you have not done it yet, you will probably in the near future, so they might as well treat you as a spammer right away to avoid potential problems. (analogy with racism intended).

</comment><comment> S-E-P Later down the thread it shows that OVH Hosting's reputation is probably the reason why it's marked as spam.

[https://talosintelligence.com/reputation_center/lookup?searc...](https://talosintelligence.com/reputation_center/lookup?search=OVH%20Hosting&org_lookup=1#ip- addresses) ^ pretty nifty look at how your neighbors screw you sometimes

<comment><USER49> That's most likely the reason: OVH is well known for their lack of attention on spammers using their VPSs, so most operators will just filter their IP blocks.

I run a low volume email server and 100% of the connections from OVH blocks are spam/phishing/vuln-scanning/etc. It's true that Gmail _could_ do more to help low volume operators understand deliverability problems, but in my opinion OVH _should_ be doing something about their spam issue.

My advice when looking for a VPS provider for running an email server is to read their AUP and check their blocks on Talos.

</comment></comment><comment><USER50> Subbing to this, I'm having this exact issue. I've got dkim, dmarc, spf, the TLS flay and rdns. No other tested mail server blocks me, just gmail.

I've googled and cannot find a legitimate way to determine why I'm marked as spam or how to unmark it. I'm coming from a fresh domain and have no intention of ever spamming.

Googles support, or lack thereof, will hopefully be their downfall.

</comment><comment><USER51> Because of this post I checked my gmail spam folder for the first time in months. Of the 429 messages, 8 were not spam, which honestly is kind of a high false positive rate.

Incidentally, two messages from Google themselves were correctly marked as spam, in that they were advertising for services that I didn't care about.

</comment><comment><USER17> Is this problem unique to Gmail? Do owners of self-hosted mail servers also have problems sending mail to Yahoo, Outlook, AOL, iCloud?

Do these providers have a better mechanism for getting yourself un-blocked?

<comment><USER52> In my experience struggling to deliver email from my self-hosted server, Gmail's by far the worst. Not that I haven't had the occasional issue with Yahoo, Outlook and AOL. But here's the key difference: Yahoo, Outlook and AOL actually offer tools to resolve the issue! For small senders like me who fall below Gmail's postmaster tools limits, we have no recourse other than trying to find some sympathetic Google employee and DMing them. The experience really blows.

I desperately wish Gmail would offer some sort of registration program that would let me attest to my identity and thereby give my server the benefit of the doubt. I'd happily cough up my private medical records if that meant my family's emails reached Gmail's inboxes most of the time.

<comment><USER53> If you actually are looking for advice, then in descending importance order:

* Enforce a DMARC policy instead of explicitly disabling it.

* Enforce a strict SPF policy instead of ~all.

* Send over IPv6. Usually stricter rules are applied and not-up-to-snuff emails are rejected at the SMTP level.

* Your provider hosts lots of spammers for their size. Consider switching.

* Enable MTA-STS.

The first two I'd consider obvious glaring mistakes. The next two are less important, but would help. MTA-STS is a nice-to-have.

I can't analyse your DKIM setup, since I don't know your selector. Feel free to send me a sample email if interested.

<comment><USER52> Thanks for giving it a look! Enforcing DMARC and strict SPF tend to screw up mailing lists and forwarding. I don't see how I can enable them without causing a lot of other problems for me and my users.

No arguments about my provider. Any tips on how to select a better one? I'm thinking someone who adds some friction to unblocking outbound port 25, e.g. Vultr, might be better.

<comment><USER53> DMARC shouldn't be an issue for mailing lists, as long as one of DKIM/SPF passes. Are you seeing reports of both DKIM and SPF failing for mailing lists you use?

Properly configured mailing lists shouldn't modify contents of emails causing DKIM to fail.

I personally use Linode and would recommend them. But basically any tier 1/2 provider, i.e. too big/popular to block, will do. However, I would avoid DigitalOcean per my experience due to them completely blocking SMTP on IPv6 in addition to badly implemented IPv6 support and seemingly subpar support.

</comment></comment></comment></comment></comment><comment><USER54> Wild guess but I'm wondering if the problem isn't domain name. *.eu.org looks like a perfect domain to send phishing campaigns from (i.e. perhaps some other subdomains of it were caught sending spam and his domain got a penalty as a side-effect?)

</comment><comment><USER55> It's a bit surprising that he got anything through without SPF and an MX record. That seems to be the minimum to get mail into the inbox for a new domain at least, so his domain probably had some positive reputation at one point.

<comment><USER56> A while back I had someone setup a new domain with office 365 email and a completely broken spf record that only allowed his web server. Gmail accepted everything without a problem.

I agree an SPF record is an expected minimum but "use a big provider" is more relevant ime.

</comment></comment><comment> em-bee we can argue all day long about why this happens. more interesting for me would be, what can i do about it.

i have noticed the same problem. since a few weeks, i can no longer reach gmail from my mail-server.

my mail volume is to small that marking mails non-spam would help. besides i nowadays mainly use email for situations where i don't have an alternative way to communicate, so when i email someone, and it gets rejected, there is no backchannel to let the recipient know about that.

reading through that thread, one solution suggests itself: don't host your email server with a provider that tolerates spammers.

does anyone have recommendations?

where can i host my own mailserver, and not suffer from being stuck in an IP block that is on any blacklists?

are there other solutions?

can i use a reputable mailserver as a proxy? so that mail gets forwarded to my own server, and mail i send out gets sent through them, but i still do all the filtering, routing, management of email addresses, storage, etc, of emails on my own server?

i know this is technically possible but which servers out there support that?

</comment><comment><USER40> The final exchange between OP and someone who works in detecting spam:

[https://www.mail- email@example.com/msg09020.html](https://www.mail- email@example.com/msg09020.html)

This is where HN discussion should pick up, not at the beginning where we need to rehash everything.

</comment><comment><USER57> Although the post is about self-hosted mail, I wouldn't be surprised to learn something fairly major changed in Google's spam filter a couple of weeks ago.

We use Google Apps (i.e. Gmail) and two weeks ago one of my co-workers noticed that another co-worker's emails were going to spam. Even though we're sending/receiving within the one GApps domain.

Upon investigation I discovered that one co-workers emails were indeed going to spam for everyone, myself included _unless_ he sent the email from his Windows machine. In which case it passed Google's spam filter no problem. His mail signature is more complicated on Apple Mail, 2 URL links (to our own domain) and an embedded image (NOT a URL).

He's been using that same signature structure for yonks; yet just in the last two weeks it's a no go.

<comment><USER58> It was two weeks ago when I started suddenly having delivery problems (ie consistently delivered to the spam folder) for emails sent from SES from AWS.

It's very obvious to me now when reading these replies that Google changed their spam filtering and it's affecting many more than just me

</comment></comment><comment><USER59> The issue affects only outbound mail, not the ability to operate your mail domain as such.

The fix/workaround is to find an SMTP relay which has reputation, and send your SMTP through that. At least temporarily, until the domain you want to be gets better reputation.

This is what people do who operate little domains that are physically hooked up via residential subscriber IP addresses. You will never be able to send SMTP anywhere from a subscriber IP. Rather, you typically use your ISP's mail forwarding hosts (usually via an authenticated connection: your SMTP server logs in to theirs to do its business). (Otherwise they would be operating an "open relay".)

</comment><comment><USER60> The legal naivete and indifference of otherwise technological savvy people is mind boggling.

It is not the task of mail providers to block or hide email considered to be spam. It is the job of the end users, ie the email providers cusstomers and their software.

Consider physical junk mail, ie the type that comes through the letter box. The post office does not decide what is junk mail or not, their job is to deliver it and it is the customer's job to discard it.

The proper thing is for the mail provider to tag the emails their systems consider to be junk and deliver it the customers inbox and let the customers decide.

Google's customers are the problem, not Google itself.

<comment><USER61> >Consider physical junk mail...

Where I live it takes a sticker 'no spam' glued to the mail box and no more junk happens

<comment><USER62> I've lived in several places where this practice is common and none where it remotely works - it's totally ignored. Where are you based?

<comment><USER63> I will just second that this works in my country (Lithuania).

</comment></comment></comment></comment><comment><USER64> I remember when Gmail was new. You got an e-mail that said "hey, we think you are sending spam, please do this". I replied and said that I always send from this domain, so if they think I'm spamming, just block the domain, then they white-listed it instead. I always see DMARC as a solution in these kind of threads, but the problem is that it's trivial for spammers to also setup DMARC. In fact almost 100% of the spam I receive follow the e-mail guidelines better then most legitimate e-mails.

We need a new protocol! Where e-mails are opt-in, like with messaging apps, you first need to be introduced.

<comment><USER11> It might be extremely weird but not impossible, if Google algorithms classify properly setup domain as a suspicious one.

<comment><AUTHOR> No, not at all. There are reports that even if you use G Suite, and whitelist your own domain to send yourself logs through crontab, your may still have problems with the email deliverability.

My own domain was blocked for nothing more than sending myself a list of a few dozen domain names through cron, which Google deems as containing malicious links, even though there are no links present.

</comment></comment></comment><comment><USER65> For some reason, Google also marks "Google Alerts" notifications as spam. They contain spammy search results, but the email comes from Google. At least they are consequent in not white-listing their own services :)

<comment><USER66> Gmail also marks messages from gmail or other google services as spam. Gmail also marks all legitimate mails from Paypal as spam (phishing). I use several email addresses and google accounts, but in the end I forward most mails into the same gmail box. Their spam filters cannot handle the concept of forwarded mail correctly.

</comment></comment><comment><USER67> I'm 100% okay with this aspect of Google controlling mails, etc.

The level of spam I have received has dropped precipitously because of it. For that I'm grateful. Whatever they have done has worked, and people don't seem to remember how bad it was 10 years ago. Now my email is relatively clear of spam and that's a great thing for safety and security as well.

</comment><comment><USER68> OT: Outreach via email and I am not talking about doing some crappy newsletters has the highest sophistication level in our company. And I am not talking about setting up some lame DMARC/SPF, we (must) do 10x more. DB adminstration is easy compared to getting emails through on a large scale.

</comment><comment><USER69> Just to add another experience, our company also started getting a lot of legitimate email marked as spam as of about two weeks ago. This included emails coming from the same domain (with all the proper SPF and DKIM setup in place).

</comment><comment><USER70> I see most people saying Google errs too much on the side of its spam," but it strikes me as better for everyone that spam never get through so as to incentivize people not to send spam, thereby destroying most spam before it even makes it to your spam box. You can always check your spam box, so its just a mild inconvenience mbst of the time. If you're expecting an email, you can whitelist the sender. If you're sending email to someone who isn't suspecting it, then maybe you need to take a moment out to think about whether you're spamming people's inboxes with emails they don't want.

</comment><comment><USER71> I run into a similar situation, and I have done every thing you are supposed to. DMARC, SPF, MTA-STS, DKIM, all of it. I have only one IP address that ever sends legitimate email from my domain (I proxy all my personal messages through that IP), on AWS, and it's been that way for years.

Despite that, the reputation systems give my system a neutral reputation. Sure, I know that there are lots of people who spam through AWS, but AWS IPs are not allocated in bunches, and anyone can get one. There should be some way to get my reputation up to good, but I haven't yet figured it out.

<comment><USER49> Neutral, in your case, probably means that there is not enough volume coming from your server, and as far as I know the reputation of your /32 does not depend on your block (of course, providers may still decide to filter your block). Did you set up the PTR record and SMTP banner?

</comment></comment><comment><USER72> I had similar problems with Gmail rejecting mail from my perfectly configured domain. I eventually gave up after finding, sometimes months afterwards, that an email was not delivered (and when found within 30 days, it was always in the recipient's spam folder).

I eventually switched to fastmail (from a reputable, but too tiny for google to care about provider), and all the problems were gone.

But ... I am almost sure the issue was that I had a catchall address cc:ing a Gmail address; not sure why that would trigger spaminess detection, but that's the best I could reason with my data.

</comment><comment><USER73> I think the major problem here is IP reputation. Especially if you are hosting on public VPS or worse your home network. The root of the problem is spam abuse. Spammers have no problem to spin up random VPS to send spam. They can set rDNS, SPF, DMARC too. There is enormous amount of spam. Of course it should be handled on message-by-message basis, using proper antispam and ML. But it looks like some providers simply block various IP prefixes. :( It makes their work easier and it is marketing to use their services "which work".

</comment><comment><USER74> The domain didn't even have an MX record defined until recently, let alone DKIM and SPIF. I'm surprised Gmail _ever_ delivered emails from that domain to anywhere other than spam.

</comment><comment><USER75> I have the same problem with my email server. I have valid and properly configured DKIM, SPF and PRT, so I don't know what else I'm expected to do...?

</comment><comment><USER76> I have the problem of spammers sending emails faking they are from my domain but through hacked computers. So I get thousands of rejects. DMARC shows you where it is happening, spf can cause some of it to get blocked, but you are stuck with anyone can make up email addresses and send them out through other mail servers and in-turn your domain is certainly going to start going to spam due to this.

</comment><comment><USER77> A few weeks ago, we organized a competition and we asked the top 3 participants to send us their solution to our gmail. Two were using their own domains (.io and .su) and one was using Gmail. The two with their own domains just got in the spam folder, and without surprise that Gmail one survived.

"If you want to set up your own email domain rather than using Gmail, good luck."

</comment><comment><USER66> I sent a text format mail containing a couple of innocent words and a youtube link of a completely innocent clip to my wife. Gmail in their infinite wisdom classified it as spam. Obviously users need to be protected from watching youtube clips.

I used the smtp server of my ISP. I'm know that it is tightly monitored and not the source of any significant amount of real spam.

</comment><comment><USER78> This is truly a severe problem, and I've given up trying to fix it. I don't run my own mail servers anymore. I have an Outlook business account. Microsoft is big enough that google doesn't mess with their email.

I sort of understand why Google does this--probably 95% of email from random mail servers are spam! But that doesn't make it right.

</comment><comment><USER79> I like infomaniak. You can buy separate Email hosting. No EU or US jurisdiction. [https://www.infomaniak.com/en/hosting/web-and-mail/mail- host...](https://www.infomaniak.com/en/hosting/web-and-mail/mail-hosting)

Gandi offers free email with domain registration.

<comment><USER80> > No EU or US jurisdiction.

As far as I can tell, they're based in Switzerland. The political pressure if either of them EU cough a bit will probably cause Swiss authorities to fall over immediately. I wouldn't consider myself safe in your position.

<comment><USER79> My email is not valuable enough for the US or the EU to put pressure on them. If you are so worried you have to encrypt your email anyway. A Russian hoster likely would not bend. Be advised that a Russian IP address does not help you to pass Spam filters...

</comment></comment></comment><comment><USER81> This was the single reason I couldn't operate my own mail server. My mails were just always marked as spam by Google, no matter how much I made sure the whole TLS/DKIM/DMARC/DNSSEC/etc setup was correct. We tend to think about E-mail as something that is still distributed, but Google really killed that.

</comment><comment><USER82> I ran into this issue many years ago, and I eventually gave up trying to increase the reputation of my mail server. For all my projects that require mailing, I've now succumbed to spoofing headers using Gmail's SMTP, which surprisingly doesn't go to spam, or using a third party mailing service like AWS.

</comment><comment><USER83> I've had multiple legitimate emails from Google employees and their sales teams all end up in my Gmail spam folder, even from senior members of various divisions.

I still get spam in my inbox.

They definitely know this is a problem and have people working on it - <USER3> spam filtering at that scale is much harder than it may appear at first.

<comment><USER84> Personally, I find Gmail's spam filtering to be _excellent_.

In the past year or so,. I've had a single false positive, and zero false negatives.

</comment></comment><comment><USER85> If you have an email provider that classifies ham as spam, complain.

If complaining doesn't help, consider switching away.

Remember at Google you're not the customer. The reason Google want to keep you using their product is so that they can learn more about you and then ask their customers more money to show ads to you.

</comment><comment><USER86> Google Cloud Platform <email@example.com> \- just found in my Gmail Spam folder! =D

</comment><comment><USER87> Gmail always marks the emails from my veterinarian, who has {firstname.lastname@example.com, as spam. It's slightly irritating. However, seeing the mails that come in to our sales endpoints, gmail addresses are responsible for a huge percentage of the obvious spam.

</comment><comment><USER44> This is a pretty fantastic summary of the current state of email: [https://www.mail- email@example.com/msg08958.html](https://www.mail- firstname.lastname@example.com/msg08958.html)

</comment><comment><USER88> It's not just gmail. When my university went to office365 mail for faculty, pretty much all of my email from the upper administration (e.g. the president of the university) was marked as junk. I complained, but there were no solutions.

</comment><comment><USER89> Plain text email with no attachments is almost never marked as spam from what I have seen (unless there are several spammy keywords). I wonder why people don't use plain text email to ensure more reliable delivery of important messages?

</comment><comment><USER90> I just remember the post not long ago ""The "mail is hard" myth" [https://news.ycombinator.com/item?id=20851880](https://news.ycombinator.com/item?id=20851880)

</comment><comment><USER91> Blame spammers, not Google. Google isn't trying to block innocuous emails.

<comment><USER63> Why not both? Google can be evil too.

<comment><USER91> Yeah, but in this case Google isn't being evil.

Do you really thing Google wants to intentionally stop your innocuous emails from getting to your mail box? Why in the world would they want that?

<comment><USER92> Possibly to get you to use Gmail instead of self-hosting. (Note: I have no proof they actively want that, I'm just speculating)

<comment><USER91> The fact that Google emails get caught in Gmail spam should make it clear that such speculation is completely unwarranted.

</comment></comment></comment></comment></comment><comment><USER93> Unfortunately, OVH doesn't care about spam from their networks, so people really need to move. Too bad we don't have other ways to properly punish network administrators who ignore complaints.

</comment><comment><USER94> This is just Google's way of getting you to switch to their service.

</comment><comment><USER95> Google will of course argue it's for it's users sake. What are the odds that if legislation passes, Google simply lets unnecessary spam through and pretends they can't do better?

</comment><comment><USER96> I would hazard a guess that this mostly happens when many people mark your messages as spam from your domain. Or if you send enough email to gmail users with certain keywords/phrases.

</comment><comment><USER97> This issue aside, I wasn't aware of [https://talosintelligence.com/](https://talosintelligence.com/)

The information there is very interesting.

</comment><comment><USER98> This is why I disabled Google spam capture entirely on my account. They were capturing things that I had marked as non-spam many times over, so to me, the feature became worthless.

</comment><comment><USER99> Check your Gmail message and look in the headers. I'm willing to bet you have some fails and softfails there. That's why you're been labeled as spam.

<comment><USER50> Trust me it's not, I've read theM countless times, google just decided I'm spam, and don't tell me why. I run a small business and have a business name registered in the same name which cost $500 AUD, I don't appear to have an option apart from paying AWS or Microsoft to host me.

</comment></comment><comment><USER100> One could just use a Gmail account for contacting people with Gmail addresses. If there's extended conversation, then migrate to ones preferred email account.

</comment><comment><USER101> The do that to me all the time and also tell me there is suspicious activity on my account when I login and there is no way to turn off these "features"

</comment><comment><USER102> I think there's a gmail filter where lets you prevent some mail from being market as spam...

</comment><comment><USER103> it sucks that even though i haven't actively used gmail in at least 5 years, google is still able to hoover up my personal, and what should be private, info. and this kind of anti-competitiveness, which seems to come to light occasionally and only dimly at that, stinks just as bad.

i can't wait until everyone else realizes how insidious a company google is. it's starting to happen. i'm hearing the low rumbles among (non-technical) folks i know. thank goodness!

</comment><comment><USER104> outlook.com spam rules are much harder.... twitter,edx,coursera, even MS own mails go to the junk folder .

</comment><comment><USER105> I'm so sick of technology.

</comment><comment><USER106> google is trying to become a large monopoly for email services tbh. Not only that but more and more businesses start using gsuite to avoid getting marked as spam for most clients and customers

</comment><comment><USER107> Yep the person in charge of this at Google should be fired.

And I say this as someone who has no reliance on or use of outbound email marketing whatsoever.

In other words, I'm not an embittered emailer who thinks their email is being unfairly treated as spam. I'm just a user who sometimes looks in the spam folder and is aghast at some of the false positives Google has. And yes some of the false negatives in the inbox as well.

<comment><USER108> You don't understand the scale of this problem. It's such a fad to "fire" someone because you think "it's an easy problem".

<comment><USER109> I think it should suffice to know that after one repeatedly trains the google spam filter with "report not spam" and "mark as spam", it should learn that well enough, no?

As an example: Some dumb ass Verizon customer gave them my email address as their own. I got a ton of communications from Verizon that I could not turn off (bill past due, thank you for you promise to pay, and all sorts of really annoying messages). I called Verizon, I was told that neither them nor I could do anything about it because I did not own the Verizon account, and I did not have the account PIN. Their official solution for this problem: Mark all the Verizon emails as spam and don't call them again (I kid you not).

The result: I marked about 100 Verizon emails as spam and Gmail learned that really well; now I do not see Verizon emails in my inbox. Why TF can't gmail do this with all the other emails I mark as spam and not spam?

I just got in the habit of firing up the hideous gmail app on my phone to go through my spam folder every day and vet out the non-spam out of there. I usually go though about 250 emails per day. I end up looking at all the spam, but it's far faster to fish out the good messages from spam than if I turned spam detection off altogether.

No matter how complex the problem to solve is, Google is failing at a level where even I could program it better, which is not a high bar at all.

<comment><USER110> > Google is failing at a level where even I could program it better

You should do so. You can set up filters to hook into keywords, domain names, etc., and route to spam. If the stuff you're getting is that obvious in structure, you should be able to kill it with a handful of such filters.

<comment><USER109> Do you know by chance know if user-0defined filters take precedence over Google's spam detection? If so, I might go for it.

I say "might" because no matter what I program into it, I'll never be sure there's not an important false positive that got trapped, and so I'll forever be checking my spam anyway. It might be all for naught.

</comment></comment></comment></comment></comment><comment><USER111> Did you try changing your name to Netflix, Etrade or American Express? I swear, no matter how many times you mark their emails as spam, they eventually wind up back in your inbox ( admittedly in the Promotions section rather than the Primary inbox ).

</comment><comment><USER112> Prior to GMail, we used to check our "Junk" folder occasionally in whatever email service we used.

When GMail arrived with its nice spam classification, it was great. Worked well, better than any competitors out there. This gradually conditioned us to ignore the "Spam" directory completely. Now things have moved to the opposite extreme. It is quite difficult to host independent email servers and send emails from it without GMail classifying it as spam. The high false-positive rate has become the norm. Instead of GMail trying to reduce the false-positive rate, it has now become the responsibility of those hosting email servers to jump through the hoops to ensure that their emails are not classified as spam.

Makes one think: If we knew what we know today about spam and abuse of Internet services, would SMTP have been designed the way it is designed today? What fundamental design change could have made it immune to spam? Perhaps a pull-based design instead of push-based? Perhaps something even cleverer?

<comment><USER113> My conspiracy theory is that google intentionally has degraded the ability of spam to be detected.

Why? Google Inbox did it absolutely perfectly. They wrapped up every email, from every major source, into nearly perfect bundles.

You could receive hundreds of unsolicited ("sOlIcItEd") marketing emails a.k.a. spam and Inbox would perfectly roll them into one bundle with a 1 swipe gesture to archive.

Then they killed Inbox and Gmail has functionally no ability to collate spam into a bundle and hide it.

That's intentional. Email marketing is huge, and google has a massive vested interest in ensuring that gmail users can see advertisements.

</comment></comment><comment><USER114> Ive tried thanking users for supporting my website, but I rarely get a response.

I'm not sure if they are "star struck" or are not receiving the email.

Any suggestions to confirm?

</comment><comment><USER115> I'm someone who uses the Cockli email service and I too often have my emails marked as spam. Makes things a little annoying.

</comment><comment><USER116> I moved my private server email to gsuite and this problem went away entirely.

Running your own mail server is very dangerous. I don't recommend it.

<comment><USER73> Thanks, that means you support these practices and you like that only way to prevent your mail to go to the spam folder is to actually pay for their services and send email only through them. Soon we will have to confirm our identity and pay google in order to use the internet as a whole, otherwise all recaptchas will fail. It will soon become survailance like in China just controlled by monopolies (tight to the government anyway). :)</comment></comment>