A quick look at MySQL 5.4 - <AUTHOR> http://dev.mysql.com/tech-resources/articles/mysql-54.html

<comment><USER2> [EDIT: My comment refers to the original title of the link. This has since been changed]

> A quick look at MySQL 5.4 - 60% performance gain over MySQL 5.1

Sorry, but this is just plain wrong. The actual article does a better time explaining and disclosing this small, specific speedup. MySQL InnoDB in 5.4 is faster on >4 multicore servers because it properly accounts for the extra CPUs. Calling it a "60% performance gain" is really misleading.

> One of the primary fixes in MySQL 5.4 is that the InnoDB storage engine can > now address more than 4 CPU's/cores

<comment><AUTHOR> You are correct, it's mostly targeted at multicore servers and the preview only holds Google's InnoDB patches. Generally thought, I hope they keep looking at performance issues beyond the InnoDB mutex patches - - and set some of the Sun optimization wizards to optimize MySQL. Sun has some _very_ bright people (those that have made ZFS, DTrace, Solaris, Java's VM etc.) that could optimize MySQL to new levels. Let's just hope they don't drown in bureaucracy.

</comment></comment><comment><USER3> It is good to see that MySQL is getting some improvements. It is a decent database, but they really need to fix replication and a number of other critical issues. Hopefully the different release cycle will help them target the issues better.

<comment><USER4> _It is a decent database_

It's approaching the same feature-set as Microsoft SQL Server had over a decade ago, but still severely lacking in optimizations and performance for anything besides basic queries.

If you were to install an OS today, would you call Windows 98 "decent" and go for that, or ditch it for something modern?

<comment><USER5> > but still severely lacking in optimizations and performance for anything > besides basic queries.

And yet it runs several of the world's most popular websites.

<comment><USER6> Right. You don't need a particularly sophisticated database to serve the workloads required by a typical website, even a popular one.</comment></comment></comment></comment>


Executive Order: Trump Wants Silicon Valley to Hire Americans Again - <AUTHOR> https://techvibes.com/2017/01/31/trump-wants-silicon-valley-to-hire-americans-again <comment><USER2> As someone who's thinking of moving to the United States it's interesting to me how this will play out. On the one hand I can imagine that some companies are misusing the H1B quotas to hire not-particularly-skilled workers willing to work for lower than market wages. But on the other hand it is not at all clear to me that companies engaging in this would be your typical SV corporation.

Perhaps someone more tuned in to this situation can chime in with some stats whether this is actually an issue for the American labor force i.e. a serious problem that needs to be addressed.

<comment><USER3> Big companies will grow their offices in Canada and Europe. H1B prospects will go there instead, and these growing offices will attract many Americans as well. The tax revenue from these people, not to mention more abstract benefits like "brain capital", will go to other nations.

People, especially those with fond memories of manufacturing booms, need to start realizing that you can't have capitalism and avoid globalization (not unless you shut _all_ the doors, which is not a reasonable suggestion). The sooner people realize it the smoother things will go.

<comment><USER2> On a certain level I agree with the fluidity of the labor market and companies may eventually successfully grow their companies in Europe for example as you suggest, in the face of increasing visa hurdles and anti-immmigrant rhetoric from Trump. However, it must also be noted that major barriers exist to free movement of labor, not so much in terms of legislation today but simply individual preferences.

I can think of a few reasons why the US is great on several dimensions people often compare countries they would wish to move to (vast expanse of nature, friendly culture, relatively low taxes, no language barriers aka english speaking populace). In comparison many countries in Europe would simply be a no-go for me simply from the language point of view. I would merely exist as a second class citizen unable to decipher my electricity bills or read billboards etc. To me little things like this cause a deterioration of the quality of life over a long period of time. I would never feel a sense of understanding of my environment in such a situation and to me that seems fundamental in the long run.

They way I see it, there should however be legitimate fear of losing out on human capital to other english speaking nations like Canada, Australia, UK, etc.

<comment><USER3> _To me little things like this cause a deterioration of the quality of life over a long period of time_

It's impossible to interpret your post in any way other than that you seem to not realize languages can be learned. I realize this is very unlikely, but that's how it reads.

Also, realize that for a huge percentage of the people we are talking about English is not their first language.

<comment><USER2> I have learned a foreign language as an adult (french) to some degree of fluency after 3 years but it's a lot of hard work and not something I would take on lightly on top of other all the other things I need to focus on as an adult. Also, I notice that different people are happy with widely varying degrees of fluency. Personally I couldn't see myself achieve the kind of easy fluency I enjoy in english in french unless I spent 10 more years living in France, and there is a huge opportunity cost behind such a decision.

About your second point, I'm merely talking about people who would consider moving from the US to some other country because of the incumbent president's actions/policies. To me it seems that the vast majority of such people would not consider moving to Germany or Sweden for example, just because of the language constraints (unless they are german or swedish, which would constitute a small percentage of employees).

</comment></comment></comment></comment></comment><comment><USER4> As the great philosopher jagger said: You cant always get what you want- (The very best for the job on the planet) but if you try sometimes- (you get someone who is just very good at learning) - you get what you need.

<comment><USER5> Yeah, americans are so exceptional. We can teach the high school drop outs and English majors to run the tech giants. And all the inferior foreign competitors will still be bowing over and witnessing the mighty angels dancing glamorously for their final judgement. USA! USA! USA!

<comment><USER6> Right. I am a liberal getting sick of people getting picked over me.

I have lived on both coasts and know for a fact that H1-b holders under cut me and then hotrack in Redmond while sending home remittances.

I have also went up for work, gotten approval for the job from a local just to get the thumbs down by a visa holder.

I am also aware the we need foriegn workers to help tanslate and sell products over seas.

I travel, a lot. And I have many stories.

<comment><USER7> I think that a lot of people feel the same.

I've been turned down for jobs because I don't speak a Chinese language.

I've been turned down for a job because I wouldn't be a culture fit for the majority Indian development staff.

I've been told "No, you can't work from home.. if you can work remotely, we can eliminate your position and have someone in India do it for 1/4 the price."

I'd imagine that a lot of Americans are sick of being told that they _need_ to have immigrants coming to do the jobs, and are sick of being told that "there are jobs Americans won't do." I know that I am.</comment></comment></comment></comment>


Myth of the the extraordinary teacher - <AUTHOR> http://www.latimes.com/news/opinion/commentary/la-oe-herman-class-size-20110731,0,3910343.story

<comment><USER2> Her big question is "Do students really learn best this way?". Nobody knows. Because education researchers don't care about empiricism.

The English-speaking education research community abandoned empiricism when they started getting "bad" results.

There were a series of studies that proved a right-wing education fad (Direct Instruction) was empirically better than all the left-wing education fads. So the left-wingers in education academia decided it was better to throw out empiricism than lose the battle.

I'm not saying DI is great. It's a fad, like all the other fads. But using empiricism to iterativly improve it, or search for alternatives would be the way to go.

Instead, education research is now often based on Sophistic arguments, appeals to authority, appeasing policy czars (yay funding!), appeasing parents, and other methods which simply don't work.

I know this is, itself, not empirical. A more solid argument would be a pointer to what teachers are taught these days, and so we can see how much empiricism vs ex-cathedra-style argument they are exposed to.

</comment><comment><USER3> A lot of the discussion here is focused on teachers because we think that they are the keystone to a quality education. I would argue that teachers are a vital part but they lack the power that family and culture have. In many Asian cultures, the power of the family is crushing and the values that the parents have translate to intense pressure to do well in the education environment. It doesn't matter if your teacher is good; the message is that you will force yourself to succeed in education.

Even though Western values are quickly taking root here, (via the movies, music, etc.), there exists a tremendous amount of cultural pressure to "excel academically" (for some definition of this idea). I speak mainly for China because that is the education system I'm most familiar with, but I assume that we can find similar patterns in other Asian cultures: mom and dad work 5 am to 8 pm selling egg pancakes for 50 cents a pop on the side of the street while little Johnny (or Jenny) goes to school from 7 am to 4 pm with a two hour break between 12 pm and 2 pm. When they get home, they get to look forward to 6 hours of homework. No going to soccer, maybe some piano lessons if you're rich enough. Certainly no time for video games, making out with your girlfriend, creating a gang, doing drugs (although there are exceptions, I'm speaking in broad generalizations here…).

But there does not exist this idea that you can be who you want to be, that somehow you can have a good life by developing other talents. There's also not this idea that you have a choice. I've encountered college students that tell me that they /have to/ graduate. This means that if they don't pass their tests, they will find some other way to get their diploma.

Compare this to your typical attitudes towards education in America. I'm not saying that one is better than the other, I'm just saying that there is so much pressure engrained in certain cultures "to do well" (i.e. score high on a standardized test) that makes it hard for American culture to compete.

This explanation does not explain why European cultures fare much better. I'd love to hear what some of the cultural attitudes towards education are from there as well as the strength of family influence.

<comment><USER4> _This explanation does not explain why European cultures fare much better._

Actually, this explanation does work pretty well for that also. Assume for the moment that Americans of European descent and Europeans have similar cultures.

Americans of European descent achieve test scores which are comparable to the top European nations + Australia.

[http://super-economy.blogspot.com/2010/12/amazing-truth- abou...](http://super-economy.blogspot.com/2010/12/amazing-truth-about-pisa- scores-usa.html)

The only reason the US as a whole does not fare as well as most European nations is because it's large proportion of non-Europeans drag the average down.

You get similar results when comparing Americans with Asian culture to Asians. Your culture explanation has a lot of explanatory power, and almost certainly more explanatory power than any explanation based on school system.

[http://super-economy.blogspot.com/2011/01/how-well-do- above-...](http://super-economy.blogspot.com/2011/01/how-well-do-above- average-american.html)

<comment><USER5> International comparative test scores seem kind of irrelevant.

What seems to be important are the contributions of a small number of members of a society, who are either beaten into submission of forced conformity, or allowed to flourish.

</comment></comment></comment><comment><USER6> Here's a notion so crazy it could work: since we are clearly unwilling to pay what it takes to pay for classrooms smaller than 30, what if we instead had some classes with 20, and others with 40+? Any material which is pure lecture wouldn't lose much (see the college lecture halls of 100+), while other material gets more benefit from individual attention. It also allows teachers to specialize: Be an engaging public speaker, or focus on bonding with students and their unique learning styles.

It might only be feasible in schools without widespread behavior problems, but hey, it's a thought.

<comment><USER7> _Our children [snip] deserve to have a teacher who has time to read their work, to listen, to understand why they're crying or sleeping or not doing homework.

To teach each child in my classroom, I have to know each child in my classroom. _

I wonder how much of what we think makes a great teacher just comes from "Stand and Deliver", or "Dangerous Minds". I always thrived more from the "lecture" style of teaching, which I got from very few teachers until I reached college. Maybe that's a personal affinity, but so must be the need for one-on-one interaction. Maybe the tasks at hand do have a lot to do with it.

Interesting idea though, lukifer. I'd be interested in what empirical studies show. I'm sure there is plenty, as biased as most might be in one way or the other.

<comment><USER5> Good point. I am skeptical that listening to why they are crying is a major contributor to a high quality education.

</comment></comment></comment><comment><USER8> The biggest problem we have is grouping by age instead of ability. Every core subject should be taught to people who are at the same level, even if it means 9 and 14 year olds in the same class.

Once you don't have to try teaching five different grade levels the same math (or other subject), 30 students become much more tolerable (not ideal, but manageable).

<comment><USER9> we recently moved three of our children to a Montessori school (one initially, then the other two six months later).

I'm quite astonished, still, at how different it is to traditional schooling. Classes are mixed age, usually spanning about 3 years, and children move to the next cycle when ready, not at a set date. Within a class, there's no mention whatsoever of what grade kids are. I don't think my kids even know.

Walk into a classroom, and you'll see the children spread out in groups of 2 or three, each group working quietly on an activity. The walls are lined with bookshelves full of activities; everything (almost) is hands on tactile experience. The older kids help the younger ones to understand things and the teacher, because she is not "teaching" (in the traditional sense) is free to work one-on-one with the children and make sure they really understand what they're doing.

Best of all, the kids love it. They really do. My daughter, who was previously hated school with a passion and was getting further and further behind (school didn't care) and was barely reading, is now an avid reader, progressing well in maths and writing and _loves_ school.

Edit: few more details for anyone who doesn't have much experience with Montessori stuff, and I had none until only a year ago.

Each child has a journal in which the teacher writes a number of activities for that student at the end of each week (the teachers, because they spend a large amount of 1-on-1 time, know _exactly_ where each child is in each subject). During the week, a child needs to complete those activities, but they're free to choose when and how. The remainder of the time is there's to work on/explore what ever they like.

There's no time slots, so kids are free to spend as much or as little time as they like on activities. The school actively encourages them to work towards spending the entire afternoon (3 hours straight) focused on a single activity if possible. The kids respond well to this too. Like science? then you've got three hours uninterrupted to work on some experiments with a few friends (group work is very much encouraged). Of course you can stop any time for a break and a snack. Heck, want to spend the entire _day_ working on the one thing? Sure, so long as the teacher's confident you'll complete your journal tasks by the end of the week

There's also no exams. none. no tests or quizzes either. Here in Australia all primary school children are required to sit the NAPLAN[0] tests, and this school is no exception. Kids do no preparation and it's barely mentioned except for the days of the exams. Surprisingly (to me) the school ranked very well.

I'm extremely envious; I'd have loved the opportunity to enjoy school. If my kids have to spend 13 years of their life at school they may as well enjoy it

[0] <http://en.wikipedia.org/wiki/NAPLAN>

</comment></comment><comment><USER10> "we can't demand that teachers be excellent in conditions that preclude excellence"

We can if we keep judging "excellence" by standardized tests which we water down until they give an answer we like and institute policies which push educators into doctoring the answer sheets.

Or did you want the students to actually learn stuff?

<comment><USER11> All the hand wringing over American teachers is entirely misplaced. American teachers are very good. Possibly the best in the world. Once you disaggregate national educational outcomes by ethnicity this becomes clear. White Americans outperform all white nations except Finland. Asian Americans of all extractions outperform their home nations. Black Americans outperform all black nations. And so on.

Judging teachers without accounting for the students they happen to teach is rather unfair. But because race and ethnicity are totally essential to understanding this it's unmentionable.

<comment><USER12> that's actually quite an interesting finding. do you have a citation? not calling you out, I'm genuinely interested, as I could totally see that happening with simpson's paradox and all.

<comment><USER13> Take a good look at that guys posting history. You're not likely to find a reputable source.

<comment><USER14> Actually I remember some well liked and upvoted blog on HN posting stats that separated out affluent ethnicities from poor ethnicities and making US education look good.

<comment><USER14> Yea this [http://super-economy.blogspot.com/2010/12/amazing-truth- abou...](http://super-economy.blogspot.com/2010/12/amazing-truth-about-pisa- scores-usa.html) is the blog, mentioned by yummyfajitas

</comment></comment></comment></comment></comment></comment><comment><USER15> "How to Improve the Supply of High Quality Teachers"

[http://edpro.stanford.edu/Hanushek/admin/pages/files/uploads...](http://edpro.stanford.edu/Hanushek/admin/pages/files/uploads/Teacher%20quality.Brookings.pdf)</comment>


Scala IDE 2.0 Released - <AUTHOR> http://scala-ide.org/blog/

<comment><USER2> Looks great and tooling might be a reason for me to try to pick up Scala again.

But part of the post (supporting 3 different versions Scala versions, with different IDE branches for each and the combinatorial result) resonated with me and reminded me of the recent (?) article about the inability of mixing libraries compiled with different versions of Scala.

If someone from the team is here: How hard is this problem affecting the IDE project?

<comment><USER3> The problem affects us not at all, as we build from source.

The compatibility problem is a different one: Unlike almost everybody else, Scala IDE uses the standard Scala compiler in presentation mode. That compiler has undergone changes from 2.8 to 2.9 to trunk. The IDE needs to work with all three versions.

<comment><USER4> What's "presentation mode"?

<comment><USER5> Using various parts of the original compiler as a asynchronous, interruptible, focused service for use in IDE.

This fixes the problems usually occurring when people have to re-implement type checkers/compilers, see for example the problems when Eclipse's Java stuff disagrees with javac or ecj creates different bytecode and breaks assumptions of other tools.

</comment></comment></comment></comment><comment><AUTHOR> Congrats to the scala community and the team that worked on this.

If there's a feature you'd like to see in a future release, you can suggest it here: <https://groups.google.com/forum/?hl=en#!forum/scala-ide-dev>

<comment><USER5> Great, thanks for the info!

Finally IntelliJ Scala got a serious competitor.

The features in the video look great, hopefully they also work like that in practice!

</comment></comment><comment><USER6> Congratulations to everyone involved...

</comment><comment><USER7> IntelliJ IDEA FTW!

<comment><USER8> I think it's great they're making the kinds of improvements the community has been asking for. Then again, I too read that list and thought "isn't that what I take for granted in IntelliJ?"

<comment><USER9> IntelliJ is very good for Scala, more responsive too. One of the intellij plugin updates did get very confused and was falsely highlighting errors in my project, haven't checked for a more recent update, but it was fairly recent. Glad to see the eclipse plugin getting some tlc, since most java devs love using eclipse even in cases where it's not the best choice.</comment></comment></comment>


Git rebase in depth - <AUTHOR> https://git-rebase.io <comment><USER2> About 99.9% of the time when people talk about rebase they talk about 'editing' history or 'rewriting' history as in the first sentence of the article.

I find that terminology terribly misleading and when I was learning git and rebase it confused the heck out of me.

No commits are harmed in the operation of `git rebase`. All the commits you had in the repo before the rebase are _still_ in the repo. Git rebase creates a new sequence of commits and after doing its work relocates the branch name to the tip of the new sequence but you can easily access the previous commits if need be:

        $ git co feature-branch
        $ git rebase develop
        $ git co -b before-rebase-feature-branch feature-branch@{1}

<comment><USER3> It's not 'editing' the raw history files, but you're still presenting a false history to your coworkers. To me there are essentially two kinds of rebases:

\- Summarizing history: squashing "implemented subfeature A.A" and "implemented subfeature A.B" into "implemented feature A"

\- Rewriting history: moving commits around, changing the base commit, and so on

In my opinion summarizing history is acceptable, you're making a creative decision that certain information will not be useful in the review/when trying to understand the code in the future.

Rewriting, on the other hand, is essentially lying. You're creating repository states that never existed, and which you have never tested. In the worst case, consider the following history:

        *     F: (master) Merge branch 'component2'
        |\
        | *   E: (component2) Fixed component 2's integration with 1
        | *   D: (component2) Merge branch 'master' into component2
        | |\
        | |/
        |/|
        * |   C: (master) Refactored component 1's API
        | *   B: (component2) Implemented component 2 that depends on 1
        |/
        *     A: (master) Base

Yes, it could probably be completely linearized, but that would be a horrible idea. Commit B will leave the repository in a completely nonsensical state. Sure, you could squash in E to mitigate it (since, luckily, nothing else happened in component 2 in the meantime), but then you're still stuck explaining what will likely look like a bunch of really weird design decisions compared to if you had designed against component 1's new API immediately.

Historical context matters. If in doubt, don't rebase. _Never_ `git pull --rebase` blindly.

<comment><USER2> It isn't lying. Advocates of rebase are always talking about a work flow where you are curating a set of proposed changes before merging into your "public" branches (.e.g, development or master).

No one is advocating that you use rebase on your public branches or basically any branch that has been "published". We are talking about feature branches or spikes or branches that exist just on one developer's machine.

<comment><USER3> You're lying about the path that you took to get to that point, and you're creating a lot of (public) nonsense commits on the way there (unless you're very careful, and/or overly squash-happy).

Whether you've published the true history earlier is irrelevant to that discussion.

<comment><USER2> Let me try again. I'm advocating that you use rebase to improve the quality of your changes that will be reviewed before merging or even before being reviewed at all.

If I make three commits and then realize that I should have included something in the first commit, I use rebase to create a new sequence of three commits that has the corrected version of the first commit. I haven't shared those commits with anyone, this is just work that I've done locally.

Are you seriously advocating that creating a pull request with: (A, B, C, A-fixup) is better than using rebase and then creating a pull request with: (better-A, B, C)?

You think that second case is "lying" because I didn't show the intermediate step that included the mistake?

<comment><USER3> Yes, it is lying.

You can mitigate most of the damage if it is convincing enough (for example, go through B' and C' and make sure everything still makes sense at each point), but realistically nobody is going to do that, because it's pretty inefficient way to spend your time. And even then, you're still removing context (unless you're just fixing a typo).

> I'm advocating that you use rebase to improve the quality of your changes > that will be reviewed before merging or even before being reviewed at all.

That was clear from the start. But the fact that X breaks Y doesn't imply that Y is a good idea when X doesn't apply.

<comment><USER2> I really don't understand why you are choosing to use the word "lying".

Us mere humans make mistakes all the time. Typos, omissions, false starts, and so on. What is the value of throwing that raw set of events at a reviewer or complicating the understanding of the changes when viewed in retrospect from the future? What is the reason you call curating the work into a more polished form "lying"? Why do you think the time spent being intentional about changes isn't valuable when compared to the time spent by a reviewer (or your future self) to sort through the flotsam and jetsam of your intermediate work?

<comment><USER4> But reviewers in most Git workflows mainly look at PRs. Then if as a reviewer you want to see how the sausage was made, you can zoom in on the commits, including all the messy reality of how the work was done. In some circumstances you might of course want to hide this, but in an open and safe collegial environment this lets the reviewer understand your thought and work process.

<comment><USER5> The reviewer also won't see all the things the author tried without ever committing these states. They don't need to see all the messy steps, or they would have needed to look over the authors shoulder all the time.

I rather review the final patch series with changes in logical order and not necessarily in the order the code was written or with intermediate work that was later reverted or changed again. I do look at commits, because also every commit message counts and is supposed to explain the individual change.

<comment><USER4> Sure, it's not a perfect record. But as in most things, perfect is the enemy of good.

The messy reality is valuable, when talking with your teammates about how the work was done and what kind of bumps were along the way. It's not about looking over their shoulders, it's about using data to develop together as a team, eliminating hinderances, etc - if you have the mutual trust to do that. And of course you yourself can go back and look for patterns of mistakes or problematic areas in code based on your history.

Like I said, to judge the change its, the whole PR diff is usually the most useful unit of inspection when you just want to see what happens. And if it's a big pr, you can of course always merge child PR's or branches against the big PR/branch, and look at the merge diffs.

<comment><USER4> Another formulation of the "learning as a team" idea-

The science principle of publishing your experiments, including failed ones, has the same benefits in sw engineering: others can build on your failed attempts, or save time by not replicating them.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER6> My eyes were opened on git-rebase when I read [https://matthew- brett.github.io/pydagogue/rebase_without_tea...](https://matthew- brett.github.io/pydagogue/rebase_without_tears.html)

The full version of the command as

        $ git rebase --onto new-base start end

takes the commit range (start,end] and re-commits them on top of the new-base commit. The commit range doesn't have to be a full branch and you don't even need to be on the branch to run the command this way. It's very intuitive and I nearly always use the full version now.

I've also gotten into the habit of "pinning" my branch before I rebase so that I have it in its original form. If the branch name is my-branch, then the command

        $ git branch my-branch{-hold,}

which is a handy (bash-specific?) shortcut of

        $ git branch my-branch-hold my-branch

leaves you on my-branch and creates a new branch label called my-branch-hold that points to the same place.

EDIT: clarification of pre-rebase branching

<comment><USER7> I'll sometimes do the same for a rebase that looks like it will be hairy, and I like to call the bookmarks, for example,

        git branch my-branch-mulligan

</comment></comment><comment><USER8> Git rebase is great. Honestly I think the argument that "if you have to push -f that means rebase is wrong" is making a huge assumption about how people use branches and why people are force pushing branches.

Force pushing branches is what you do when you have pushed a branch that you expect to modify. Why would you do that? Because that's how Github and Bitbucket have taught people to conduct PR's.

If your immediate reaction is that "rebase is bad UX", ask yourself whether or not pull requests are good UX. I honestly think rebase is great, but that pull requests are extremely bad UX, and the UX blame is misplaced on rebase when where it really belongs is on pull requests.

<comment><USER9> > Force pushing branches is what you do when you have pushed a branch that you > expect to modify. Why would you do that? Because that's how Github and > Bitbucket have taught people to conduct PR's.

I thought that Github and Bitbucket encouraged people to push up additional commits to fix issues in their PR. So, a typical PR will end up with a commit history like:

      Implement a feature method
      Add calls to new feature method
      Update to version 1.2.3
      fixing missing semi-colon
      addressed comments
      one more thing
      now its working

People who force-push are the ones who are trying to keep a clean commit history (meaning you don't have those extra 4 commits). So, your point about a PR being bad UX versus a rebase is correct, but not for the reason you state.

<comment><USER10> Almost no projects I've worked on that use GitHub ask people to push fixup commits. In fact, maintainers (like me) often have to ask people to squash their commits into reasonable chunks.

<comment><USER11> It's not the project that asks people to push fixups, it's that GitHub, as opposed to e.g. Gerrit, makes it hard to see what changed (how your comments were addressed) after a force push, so it's best that people only add commits.

Ideally they'd use git commit --fixup=<sha> -p, and then git rebase --autosquash when the maintainer approves the merge, but few people care.

(And what I'm saying isn't really accurate any more since GitHub does show force-pushes in the pull request UI these days and one can run git range-diff on that. But this wasn't possible last year.)

</comment></comment></comment></comment><comment><USER12> When you have a command so confusing that you need an entire website dedicated to a single command, and still need to warn against using it, then _perhaps_ you've got the UX wrong.

<comment><USER13> git is a version control framework more so than a version control system. It starts from simple primitives, exposes them to the user, then builds complex and powerful tools on top of them. Because git rebase gives you primitives to accomplish high-level tasks (e.g. "reorder these commits"), the learning experience is different because you have to learn the low-level details to accomplish your high-level task. However, because those low-level details are accessible to you, you are afforded a greater flexibility in inventing new high-level tasks.

<comment><USER12> But those high level tasks could also be exposed directly. There's nothing to stop there being more commands which more directly accomplish the desired tasks.

The idea that git is good _because_ it is difficult to use is just "git snobbery", as is the idea that it _must_ be difficult because it's a DVCS.

There's nothing to stop git having two levels of the API, one exposed for tools to build off of with the full complexity and another for every day use.

<comment><USER14> Mercurial's interface is just fine, and these days it's just as powerful as Git.

Things could be better. There's an existence proof. It just lost the mindshare war and so now we're stuck with Git, which I still have to look up basic syntax for because its command set is contradictory and makes no sense. (Is it git <x>? git <y> \--x? git <z> <a-b>? Something else entirely? Who knows!)

<comment><USER15> I used to agree with this, now I've stopped worrying. Because if git is the worst part of your workflow, that's a great problem to have. But at many places, git is the _best_ part.

(I've also had to work with various IBM CVS, and they are universally garbage. When I get frustrated at git, all I have to do is think back to those.)

So yes, Mercurial is better, but is it worth the effort? Not in my experience.

<comment><USER16> I know it's popular to shit on anything that isn't git these days, but you mentioned IBM CVS. I've used a couple of them, but primarily RTC (Rational Team Concert). I know that was an IBM acquisition and not a home-grown solution (what wasn't?). I personally prefer some features of RTC over how to do the equivalent in git. Namely, being able to move change sets (think commits) around freely, not having to deal with rebasing/merging into whatever branch you want to put it in/on. I also think there's something to be said for a CVS system that is built for teams that work together daily, compared to a system that's built for a "remote contributor" model.

That being said, I use git daily and find that I'm able to do everything I want and more, so I'm not looking to make a switch. Unfortunately, most people don't care to learn how to use git beyond "checkout, commit, push, call for help".

<comment><USER15> There was one before RTC, called CMVC, which was truly awful, especially using after 2010 felt like an insult to developer productivity.

I forget all the reasons why RTC isn't great, but the main one: if the server goes down, you're screwed. This happened several times, and we basically went to the pub instead of working. Slow to check out. Streams sucked compared to branches (especially when the server admin restricted creation of streams, meaning you simply could not branch at all if I'm remembering), and the capability to stash changes/switch branches to work on different work items if one was blocked was also more cumbersome. Code review was terrible.

A centralised paradigm does simplify things a lot mentally, but the workflow suffers IMO.

<comment><USER16> You certainly make some good points about the downsides of RTC. RTC's streams are often compared to git's branches because they're the closest construct, but they are definitely very different and have pretty minimal overlap, considering they're basically the parallel construct. IMO stashing changes was not bad (suspending change sets, I believe it was called), but perhaps I was mostly doing that within 1 stream and not between streams. I agree that code review was not great, though I'm not a _huge_ fan of GitHub's comment/PR review mechanism either. I'm not aware of code reviewing built in to git itself, though I could be totally missing it.

<comment><USER15> > I'm not aware of code reviewing built in to git itself

I guess you can pull a branch or email a patch and diff it with the diff tool of your choice.

There's a few options for gir review UIs, e.g. Gerrit or GitLab. Kind of unix-y, just have the VCS be a good VCS.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER17> The warning regarding "public, shared, or stable branches" is always warranted, but I think those warnings end up reverberating where they needn't. Before interacting with anything public or shared — when working solo or locally — `rebase` can be hugely helpful. When starting out with something complicated, I often make separate commits for different files or steps; using rebase to reorder commits or amend can make turning your first steps into a viable change much easier, and can help with merge conflicts down the line. I also find it helpful for large commit messages; rather than needing to write everything at once, making liberal use of `fixup` or `squash` can keep disparate thoughts or bug fixes manageable.

I'd never use it on a public or shared branch, but `rebase --interactive` and `--exec` are some of my most-mused git aliases.

<comment> V-2 I'd use rebase --interactive quite aggressively on a public branch when it's a _feature branch_ that's not yet been merged. As I'm the only owner of it, the way I see it I owe no guarantees to anyone. You're welcome to watch it, but it's work in progress in every aspect.

<comment><USER17> Github and refined-github[1] make it easier, by keeping track of changes from a `push -f` and having merge/squash/rebase options on merging, but I know of large projects with an explicit "no rebasing" rule as it can get confusing for reviewers. I'd say it depends on the project, maintainers, and general workflow. Which is good!

1: [https://github.com/sindresorhus/refined- github](https://github.com/sindresorhus/refined-github)

</comment></comment></comment><comment><USER18> The 'push -f' suggestion could instead recommend 'push --force-with-lease' to be less error prone in case one is accidentally pushing to a concurrently modified branch.

Also, unless the diagram is confusing with the alignment, the "rebase to rebase" example seems to be implicitly assuming --onto, because the last common ancestor of 'master' and 'feature-2' includes 2 commits on that branch: the first of 'feature-1' and then the one that is only on 'feature-2'.

<comment><USER13> I've heard this before, but I felt that `--force-with-lease` requires a lengthier explanation in an an already intimidating article, is harder to type, and generally isn't useful for users of the Sourcehut workflow. It's definitely a useful tool, though. Maybe I should add a footnote.

<comment><USER19> I don't get the "extra letters" argument as a Computer Scientist. Most of my time is spent figuring out what I'm going to do before I type anything, whether that's research, staring at code for hours to see how it works, or something else. I could type 3x as many characters each day and would probably only work an extra 10 minutes per day. Maybe I'm the exception, but I don't like brevity for brevity sake.

<comment><USER20> You could also consider it "harder to remember".

<comment><USER19> I normally use "git push --fo<TAB>" or "git push --fo<UP ARROW>" in zsh, but after seeing the switch option "\--force-with-lease" pop up each time I've memorized it now. I let my CLI do the work for me most of the time, but when I can't there's always "man COMMAND". After doing anything a bunch of times, you remember it.

</comment></comment></comment></comment></comment><comment><USER21> I've you use rebase to rewrite history a lot I have found/developed a couple helpful commands (tested & working on Linux bash).

Put those in your ~/.gitconfig:

      [alias]
       fu = "!f() { local msg=\"fixup! $(git log --oneline -n1 | cut -d ' ' -f2-)\"; git commit -am \"${msg}\" && git rebase -i --autosquash HEAD~2; }; f"
       fuc = "!f() { local msg=\"$(git log --oneline -n1 | cut -d ' ' -f2-)\"; if [[ \"${msg}\" != "fixup!"* ]]; then msg=\"fixup! ${msg}\"; fi; git commit -am \"${msg}\"; }; f"
       xx = "!f() { git reset --hard && git clean -f -d; }; f"

From now on:

git fu = (git fix up) combines last commit and all staged changes into one commit with the last commit message

git fuc = (git fix up comment) commits all staged changes as a new commit with the last commit message, but prefixed with "fixup!". Except if the last commit message is already prefixed with "fixup!". Now you can work on the same thing but commit incremental steps. In the end you just do: git rebase -i master (or similar) and they will be all in one commit.

git xx = get rid off all unstaged/uncommited changes in current directory. Very destructive, very useful.

Last but not least. If you have a lot of "fixup!" or "squash!" commits and need to interactively rebase without autosquashing them do:

git rebase --no-autosquash -i master

</comment><comment><USER22> Weird they don't talk of the `git commit --fixup=` command. And then `--autosquash` when rebasing.

<comment><USER23> Is --fixup the same as --amend?

<comment><USER11> Conceptually, it's similar, and after git rebase --autosquash, the result is the same. But you can amend/fixup (call it whatever you want) any commit in your branch, not just the last one.

</comment></comment></comment><comment><USER24> FWIW i've never really needed rebase. i am pretty happy with seeing all the commits that ever happened.

<comment> rich-tea What do you use your git history for? History is either worth keeping, in which case you should maintain it like any other artifact, or it's not, in which case you should squash down master to a single commit every time you merge.

But maybe you use your history for something else that I haven't considered.

<comment><USER25> While I like the idea of rearranging commits to convey a nice (but "not how it originally happened") development sequence, I think in practice this matters less than (say) good commit messages, or the difference between merging and rebasing.

(--fixup type commits aside).

Practical benefits from not squashing history:

\- Can bisect to find bug introduction.

\- Can annotate/praise/blame to find who/when some change was made.

\- Adam Tornhill's "Code as a Crime Scene" argues that it'd be beneficial to consume VCS history to provide health metrics on the codebase. (e.g. use VCS to check which sources have many contributors (thus potentially high defects), or check for "lost knowledge" from developers who have left).

\- Can build/run an older version of the software.

But is there really a big advantage from putting time into maintaining a sequence of commits? EDIT: Ah, I see another comment point out that "maintaining a nice history" tends to mean fixing very borked commits. That makes sense. :-)

<comment> rich-tea All of these advantages don't make sense if half your commits are broken versions of the software. Rebasing helps ensure that each commit is valid. That's important for the reasons you mention. Having a log of what you actually did is not important.

</comment></comment></comment></comment><comment><USER26> The reflog has saved my bacon more times than I care to admit.

</comment><comment><USER27> I just recently(last month) broke my habit of `git checkout feature && git merge master`, replacing it with `git checkout feature && git rebase master`. Don't know why I spent so long doing it with merge, and just mentally trying to ignore the useless commit messages that resulted.

EDIT: I meant to include this link, which is a pretty brief yet thorough explanation of when/how/why to rebase: [https://www.atlassian.com/git/tutorials/merging-vs- rebasing](https://www.atlassian.com/git/tutorials/merging-vs-rebasing)

</comment><comment><USER28> I've a script I need to publish that does a bisect-like thing to rebase across thousands of commits quickly and ensuring that when there are conflicts you are asked to resolve them at the commit that caused them. It was original written by @vdukhovni (GitHub).

Also, I've this gist on how to think about git: [https://gist.github.com/nicowilliams/a6e5c9131767364ce2f4b39...](https://gist.github.com/nicowilliams/a6e5c9131767364ce2f4b3996549748d)

</comment><comment><USER29> A few markup errors:

• In a number of code blocks, angle brackets are not escaped, and so text doesn't appear in the end result.

• `<span style="color: maroon">#include</span> &ltstdio.h>` lacks the semicolon

• The paragraph immediately after the #conflicts heading lacks its <p>.

I also think it would be worthwhile mentioning in the first footnote a hazard of empty commits: that they will be dropped by default when rebasing.

</comment><comment><USER30> I'm missing something obvious but all of the history you show is in the reverse order to default behaviour?

Am I missing a config setting you explain somewhere?

<comment><USER5> This looks normal. The order of the 'git rebase' plan shows the sequence in which commits will be applied from top to bottom.

You are probably thinking of 'git log', which uses the reverse order with the newest commit at the top.

<comment><USER30> Huh... how long have I been using git...?!

Not sure why I was convinced otherwise, must be log's behaviour as you say. Had to test to believe it!

</comment></comment></comment><comment><USER31> Regarding the rewriting of history: can't the git history itself be stored in git somehow, so its change over time (and e.g. by tools such as git-rebase) is properly tracked? Of course you'd need a meta tool to change the history of the history, but perhaps it's not needed?

<comment><USER13> It is: man git-reflog.

</comment></comment><comment><USER32> I fear that any process that allows for a _git push -f_ at any time is eventually going to result in someone hammering a shared/public repo when they meant to hammer their personal repo.

<comment><USER33> Just block force pushes to shared branches.

<comment><USER34> one of us owes the other a coke :-)

<comment><USER32> The author should add "block force push to master" to the comprehensive guide :)

</comment></comment></comment></comment><comment><USER35> Is there a benefit to rebasing vs merging, specifically if your flow is to squash commits before merging back into base?

<comment><USER8> Most people's aversion to rebase is an aversion to altering history. Since squashing alters history anyway, I'd argue that you might as well just use rebase when you do it.

In a rebase-oriented workflow, every commit has exactly one parent, and the commit history is entirely linearized. Merge commits have more than one parent, which means going backwards through the history means that you have to essentially navigate the branching structure to navigate the history. That makes it pretty hard to do!

A linear history is comparatively easier to reason about than a history with all of its branches. It's also, in a sense, less "true"! So it depends what you care about. Do you care about telling the exact story of everything that happened to everyone on the project, or do you care about the history of changes to the one shared copy?

Linear histories also make it easy to step backwards through the history one commit at a time. I personally just do `get checkout HEAD^` over and over, walking through the history in reverse, since most breakages are noticed within a few commits of their occurrence. I really like being able to do that. A lot of people think that's useless!

If you have one copy of the project that is considered authoritative, and all developers are synchronizing via that single authoritative copy, creating a clean and linear history is possible.

</comment></comment><comment><USER36> > _One of Git 's core value-adds is the ability to edit history_

I didn't make it past this line. git is extremely good at keeping history immutable; it's also good at creating new, alternative histories and moving between them.

<comment><USER37> Editing history in git is also well supported and _very_ commonly used, and git does it well. It's just not for absolute beginners, and the UI is occasionally not great. If git did not want you to ever change history, filter-branch, rebase, rebase -i, fixups, squashes and a bunch of other inclusions are odd, to say the least.

<comment><USER36> My point is, git doesn't let you edit history; it lets you make branches easily & the original history remains unaltered. It's trivially easy to detect altered history, and talking about 'editing history' gives the impression that repos are vulnerable to attack.

<comment><USER37> What wording would you suggest to refer to these kinds of operations? Replacing history?

You mentioned "altered" history, which to me isn't much different from "edited" or "modified", though maybe that was just a slip and not your preference?

There does need to be some term for it. If I push a reordered branch, I clearly did more than nothing, even if we don't want to call that modifying history.

I do question if any term will work perfectly, or if there is any full solution other than one learning git to an intermediate level before discussing it. The problem, as I see it, is that there are (at least) two concepts of history, there is the DAG of commits, where ancestors are older, and there are branch-heads where you can change what they point to in unrestricted ways (pretty much the reflog). I'm not sure English has a really good metaphor that's going to capture everything.

</comment></comment></comment></comment><comment><USER38> All the git-rebase-fu I have ever needed while working with pull requests, I have found in this well written guide: [https://github.com/susam/gitpr](https://github.com/susam/gitpr)

Quoting from this document below.

      # Rebase topic branch on the main development branch (optional).
      git checkout TOPIC-BRANCH
      git rebase master

      # Edit commits, e.g., last 3 commits in topic branch (optional).
      git checkout TOPIC-BRANCH
      git rebase -i HEAD~3

      # Force push rebased/edited commits to the pull request (optional).
      git push -f origin TOPIC-BRANCH

<comment><USER39> Thank you for sharing this link. Until now, I never understood properly how rebase and fast-forward merges work. This simple note in this document made it all clear.

"Beginners to this workflow should always remember that a Git branch is not a container of commits, but rather a lightweight moving pointer that points to a commit in the commit history.

        A---B---C
                ↑
             (master)

When a new commit is made in a branch, its branch pointer simply moves to point to the last commit in the branch.

        A---B---C---D
                    ↑
                 (master)

A branch is merely a pointer to the tip of a series of commits. With this little thing in mind, seemingly complex operations like rebase and fast- forward merges become easy to understand and use."

Suddenly everything makes sense now!

</comment></comment><comment> rich-tea What I really like to do is make actual fixup (or squash) commits during a code review and just push these to the branch normally. That way reviewers can easily keep up with the changes. Right at the end, the maintainer requests that the original developer does a rebase --autosquash before the branch is actually merged.

This works really well but I rarely see people talk about it. It means you don't have to use something complicated like github or gitlab to keep up with rebases. It works for everyone.

<comment><USER22> Indeed this is the whole reason I like fixups. Reviewers can actually keep up with the requested changes and know exactly which fixup-commit is related to which change.</comment></comment>


Show HN: SeniorTracer - passive monitoring for elderly people - <AUTHOR> http://seniortracer.com/sensorweb/main?view=chLang&lang=UK

<comment><AUTHOR> This is my friend's weekend project, but I find the idea interesting. It was originally made because of his elderly relative.</comment>


Roll Your Own Analytics - <AUTHOR> https://www.pcmaffey.com/roll-your-own-analytics/ <comment><USER3> (Data engineer here)

Nice article! I did something very similar to this for my blog but used Snowplow's javascript tracker ([https://github.com/snowplow/snowplow- javascript-tracker](https://github.com/snowplow/snowplow-javascript-tracker)), a cloudfront distribution with s3 log forwarding, a couple lambda functions (with s3 "put" triggers), S3 as the post-processed storage layer, and AWS athena as the query layer. The system costs under $1 per month, is very scalable, and is producing amazingly good/structured data with mid-level latency. I've written about it here:

[https://bostata.com/post/client-side-instrumentation-for- und...](https://bostata.com/post/client-side-instrumentation-for-under-one- dollar/)

By using the snowplow javascript tracker, you get a ton of functionality out of the box when it comes to respecting "do not track", structured event formatting, additional browser contexts, etc. If you want to see how the blog site is functionally instrumented, filter network requests by "stm" (sent time) and you'll see what's being collected.

I've found (after setting similar systems for 15+ companies of varying scale) that where a system like this breaks down is when you want to warehouse event data and tie it to other critical business metrics (stripe, salesforce, database tables that underpin the application, etc). Another point it starts to break down is when you need low-latency data access. At that point it makes more and more sense to run data into a stream (kinesis/kafka/etc) and have "low latency" (couple hundred ms or less) and "high latency" (minutes/hours/etc) points of centralization.

Using multi-az/replicated stream-based infrastructure (like snowplow's scala stuff) has been completely transformational to numerous companies I've set it up at. A single source of truth when it comes to both low-latency and med/high-latency client side event data is absolutely massive. Secondly, being able to tie many sources of data together (via warehousing into redshift or snowflake) is eye-opening every single time. I've recently been running ~300k+ requests/minute through snowplow's stream-based infrastructure and it's rock- solid.

Again, nice post! It's awesome to see people doing similar things. :)

<comment><AUTHOR> Thanks! Not a data engineer, but I used to work at a data engineering company, and can attest to the complexity and rawness of the industry. Your setup looks solid!

As a frontend engineer having seen under the hood of data pipelines at scale, I wanted to reverse engineer the parts of it that I care about (product analytics via event logging), and package it up for my little side projects.

It's awesome that this is inspiring to people. If people get anything from what I wrote, it'd be this: While large companies all roll their own data pipelines, it's _not that difficult_ for a startup / smaller co / individual to do product analytics on a level that makes sense for them, without just automatically reaching for GA or whatever.

<comment><USER3> I totally agree

</comment></comment></comment><comment><USER4> Very cool. For those that want to de-Google-ify their website analytics in a more practical sense, consider using a self-hosted Matomo (formerly Piwik) instance: [https://matomo.org/](https://matomo.org/)

I find it has 80% of the features GA has, it's GPL v3+ and only takes a few minutes to setup.

<comment><USER5> Depending on what kind of analytics you want, Fathom is quite nice too. I recently set it up on my servers (about a month ago) and its been working well. It's open source as well, with a public Trello board that speaks to future dev goals. [https://github.com/usefathom/fathom](https://github.com/usefathom/fathom)

<comment><USER6> And here's my write up of how you can quickly host your own instance of Fathom using dokku: [https://www.kn8.lt/blog/hosting-your-own-analytics-with- fath...](https://www.kn8.lt/blog/hosting-your-own-analytics-with-fathom/)

</comment></comment></comment><comment><USER7> It looks as if in the next several months Firefox will become all but invisible from most 3rd party trackers as Firefox "will strip cookies and block storage access from third-party tracking content, based on lists of tracking domains by Disconnect." [1]

The full list of trackers that will be blocked by default is substantial. [2]

1 [https://blog.mozilla.org/futurereleases/2019/02/20/enhanced-...](https://blog.mozilla.org/futurereleases/2019/02/20/enhanced- tracking-protection-testing-update/)

2 [https://github.com/disconnectme/disconnect-tracking- protecti...](https://github.com/disconnectme/disconnect-tracking-protection)

<comment><USER8> Hopefully this actually works.

I have a very limited set of domains (manually) allowed to save cookie data in Firefox pas when a session ends. With everything else _supposed_ to be auto deleted after the session ends ("Keep until: 'I close Firefox'").

Invariably though, after a week or so there are quite a few cookies saved for other domains anyway, which Firefox has decided for some unknown reason it's going to keep regardless. Without any explanation of why it's done so. Grrrr.

While manually clearing those out works, it doesn't seem like the current code base is working as intended.

</comment></comment><comment><USER9> I like the idea.... but this doesn't make sense.

I'm all for not using Google Analytics- I don't use it on my site. But why would you then use Google Sheets to hold the info? That's ridiculous. That's like telling people how to make an apple pie without sugar, and the last step before baking is dumping in a pound of sugar!

If you want an easy data store, use something like SQLite. There are plenty of options that are easy to self host, and a lot of libraries have been written to make it easier for you.

<comment><AUTHOR> For sure, there are a bunch of reasons not to use Google Sheets. For this, it's just an easy, free, and temp way to get a new site up without needing to setup a backend.

It's not really a hard requirement for me to get off Google though, just to avoid using their ad-tech. For privacy concerns, using Google Sheets isn't all that different from using MySQL on Google Cloud, etc.

<comment><USER10> When looking at solutions, did you consider using a private git repo?

On that thought, you could potentially hook it up to Google Sheets or editors that integrate with remote git repos. Skipping 3rd party hosts, you could host your own git server for cheap too.

Awesome project nonetheless! I'm a hypocrite still using GA but want to move off of it this year, so reading your solution has me thinking about my own.

<comment><AUTHOR> Lots of ways to skin this cat ;). Figuring out the best way that works for your setup is half the fun. Thanks and good luck!

</comment></comment></comment></comment><comment><USER11> At the past 3 places I've worked, we've setup Snowplow Analytics ([https://snowplowanalytics.com/](https://snowplowanalytics.com/)) and would strongly recommend it over GA, Segment, and other third-party systems.

If you're looking for a first-party system, Snowplow is an amazing setup.

<comment><USER12> I looked into Snowplow but was intimidated by their Trackers/Collectors/Enrich/Storage/Data Modeling/Analytics flow.

How hard is it to setup? Does it come with a UI to easily view, sort and filter these events into graphs like Mixpanel or Amplitude?

<comment><USER3> Snowplow is awesome - it doesn't come with a UI but here's a sample of what data is included:

[https://github.com/snowplow/snowplow/wiki/canonical-event- mo...](https://github.com/snowplow/snowplow/wiki/canonical-event-model)

A pretty common move is to drop this data into redshift/snowflake and query it with Mode/Looker/Tableau/whatever. Athena is a viable option as well, until you get into higher data volumes and don't want to pay for each scan.

Context: I'm a tech lead (data engineering) @ a public company, have set this system up 15+ times @ numerous other companies, and could not live without it at this point. Current co's snowplow systems process 250M+ events per day peaking @ 300k+ reqs/min, on very cost-efficient infra.

</comment></comment></comment><comment><USER13> This is like a marketer suggesting an engineer roll his own security.

If its a fun side project, do it! If you actually want it to work, don't do it. If you want to /really/ use it (checking more than sessions), don't do it.

<comment><USER14> Many places I have worked end up using N solutions plus atleast one from scratch BI platform.

I don't think rolling your own auth or encryption is an apples to apples comparison.

One of the new hotness in marketing is micro-funnels. You have to have customized metrics built into the product for that to work.

</comment></comment><comment><USER15> Awesome work! I had the same issues with GA, but decided to go with Matomo (Piwik). It's an Open Source, self-hosted alternative to Analytics. So far I couldn't be more impressed!

</comment><comment><USER16> There's also GoAccess [1] which can monitor and parse your nginx or apache logs to show you realtime stats in a html report or even directly in your terminal. The numbers do not fully match google analytics numbers, but I love how it looks.

[1] [https://goaccess.io/](https://goaccess.io/)

</comment><comment><USER17> _Working backwards, I start by defining what data I care about tracking. This is actually one of the biggest ancillary benefits of rolling your own analytics. I get to perfectly fit my analytics to my application._

This could easily make the whole thing worthwhile. Ultimately, the number one ingredient for useful analytics is figuring out what you want to know, and how you are going to use that knowledge. It's usually a missing ingredient and GA encourages a much more passive approach.

<comment><USER18> Its not very scientific though. You're starting from a point of assuming you already know what you want instead of collecting data and analyzing it to see what correlates to what.

<comment><USER17> I would argue it's more scientific (even though the goal generally isn't science). Hypothesize, design measurements, test, conclude.

Modern science often has the opposite issue. Economists (for example) have a lot of data. They can automate regression analysis to find correlations, then fit the theory (in economics, this is often an intricate model of an economic system) to the "result."

The problem is, 99% certainty doesn't mean anything if effectively test 1000 hypotheses. 10 correlations will occur randomly.

<comment><USER18> More data does not make you a worse data scientist. You can run a bad study at any amount of data. The only difference is a new hypothesis will have historical data to look at. More data not require you create some over fitted model.

More data is strictly better.

<comment><USER19> I disagree with "More data is strictly better". if you said that everything else equal, more data is better, I would be closer to agreement.

But when you have more data, biases become more of a problem. p-hacking becomes easier, and the easier something is to do, the more likely it is to happen.

I would frame it this way: the signal/noise ratio of data decreases as the size increases.* The overall value increases, but at a slower rate over time.

* The caveat - once you get above a certain volume of data, new processing techniques become available that aren't available at smaller volumes.

</comment></comment></comment></comment></comment><comment><USER20> If you want to decommission Google all the way, you can have a look at Countly ([https://github.com/countly/countly- server](https://github.com/countly/countly-server)). It is also available on Digital Marketplace so it is quite straightforward to deploy on a cheap instance. It has an API so you can query your data, too.

</comment><comment><USER21> We removed all tracking scripts from the front-end framework we use (vuepress) and are only checking nginx logs with geoip and org extensions enabled. This is more than enough for us as we're in the non-consumer software business with a relatively low volume of page views by human visitors. We see the org, the country, the city, and the page flow. Good enough.

</comment><comment><USER22> I hate all these user spying tools, whether home grown or bought in. Please, if you write software, don't do it.

Just let us buy the software and run it. Like in the good old days.

<comment><USER23> I don't see anything wrong with keeping track of how users interact with your site in aggregate. It helps you improve it.

<comment><USER22> Where is the distinction between spying on a user using a piece of traditional software and a web site. At the end of the day, you are spying on users to better yourself and sometimes at the cost of the user, who mostly is not aware of the manipulation that goes on. Sorry for the cynicism, but that is the reality.

<comment><USER24> First-party analytics is like blocking people in face masks entering a store, using security cameras or counting daily visitors. Your "cynicism" is more Luddide-ism.

As with physical venues that employ analytics, you can easily just __not __visit those sites that want to know a bit more about how people consume content than seeing "GET /page.html" in HTTP logs.

I'm writing this text to you as a huge free-software proponent so I'm not a "corporate shill" when I say analytics are really useful even to the most privacy-respecting pieces of software, it allows to spend resources much more effectively than making blind decisions about what users want - the vocal users opening GH issues about things are the 0.01% and those people shouldn't be the ones people building webpages, -services make decisions upon.

<comment><USER22> When any normal person visits a store they can see the security cameras watching, and they can likely image that they may be counted.

It is not accurate to say that it is like that the average person who goes to a web site is even in the slightest aware of all the different ways in which they are being used.

<comment><USER24> Just like when entering a store visiting a webpage implies already that the thing your visiting does not belong to you, that the content there is not yours, the door might be counting visitors and to dissuade thieves from stealing there most likely are CCTV cameras. If you do not agree with that, don't enter the store or visit the web page, it's that simple. It is not accurate to say that someone else's ignorance is a reason alone not to do something that in the end will benefit both you and maybe other customers.

</comment></comment></comment></comment></comment></comment><comment><USER25> There are of course other benefits to rolling your own analytics:

1\. you can build the analyzers how you want, to relate different disparate events to each other and get a report of that easier than you can with a 3rd part solution (if the 3rd party solution can even do what you want)

2\. You can do real time querying of the data relating to what the user is interacting with at the moment.

downsides, stuff like gender/age analysis will be outside of your grasp - <USER2> least at the beginning.

</comment><comment><USER26> Another option is to use a cloudfront hosted pixel, and have a cloudwatch job to schedule processing of that data every X minutes. This gives you ultra fast edge response time on your tracker, instead of slow lambda + api gateway, but you have to back-process the data. This should also cost less if you have a lot of traffic.

Instead of Google sheets, throw it to bigquery. Or reprocess hourly/daily into parquet then scan with athena.

<comment><AUTHOR> I can't POST with a pixel though, and I care about event logging.

For sure, Google Sheets is just a free, easy and temp stand-in for a real data store.

</comment></comment><comment><AUTHOR> Hi HN, if anyone's wondering how this is performing with the 1st page crush... tldr; trial by fire, but it works :)

Since I posted it 2 hrs ago:

* logged 1200 sessions (this doesn't include folks who bounce before the page loads.)

* 8 "lock acquisition" errors from Google App script, which basically means it timed out trying to get a slot from too many concurrents

* 10 minutes of lambda runtime (Netlify gives 100hrs / month free)

* 590 ms average network latency

I'll update the post with full details on performance once the dust settles.

<comment><USER27> I think you're more likely to hit Netlify's limit of 125k requests per month vs the 100hr limit.

</comment></comment><comment><USER28> I wish pages like this one had RSS feed. I would like to follow future articles, but I do not want it to end up in my email inbox.

<comment><USER29> OT, what do you use to follow RSS feeds with? I've tried a few different feed readers since google reader was shut down, but never settled on one for more than a couple days.

I sadly rely on reddit/hn to get updates/news now.

<comment><USER30> I have recently switched to feedbro after trying feedly and inoreader. Recommended.

[https://nodetics.com/feedbro/](https://nodetics.com/feedbro/)

</comment></comment></comment><comment><USER31> This is quite timely! I'm in the middle of transitioning away from google analytics for my apps/sites (to de-google-ify as much as possible). However, my plan was to implement self-hosted matomo/piwik. In fact, i was considering only implementing the server log-reading feature (and not the javascript tracker script) primarily to lessen webpage bloat but also because my analytics needs are quite basic and minimal. (I'd like to say that i'm some awesome digital marketer, and "need all the data things", but honestly, i really can't justify all that for my small apps/sites. ;-)

That being said, i do love seeing "roll your own" examples - which help foster creativity, remind us that the big silos are not the only ones that can actually produce helpful utilities and platforms, and to further de-centralize stuff on the web (indieweb baby!) Kudos @pcmaffey!

<comment><USER3> Agreed!

</comment></comment><comment><USER32> I encourage efforts like this. Far too many sites pour their data into Google Analytics, giving Google far too much information about our browsing. In my opinion, even logging the IP addresses of visitors is risky with laws like the GDPR coming into force.

I decided to practice what I preach by rolling my own basic analytics for my site. I had a different set of requirements[0] than this person but I am comfortable with the results.

This only flaw I have found is that it has revealed that most of my blog posts get pathetically few hits[1].

[0] [https://sheep.horse/2019/1/the_world%27s_worst_web_analytics...](https://sheep.horse/2019/1/the_world%27s_worst_web_analytics.html)

[1] [https://sheep.horse/visitor_statistics.html](https://sheep.horse/visitor_statistics.html)

</comment><comment><USER33> This is great! I'm actually building my own analytics site for similar reasons[1] and to learn Rails. It's still in early development, but I invite anyone to join and test it out in the meantime.

[1] [https://analytics.zjm.me](https://analytics.zjm.me)

</comment><comment><USER34> Nice! Good job! A bit too long to read everything, but liked the intro.

One issue though. I've reached the end and had no idea what to do from there (aka you don't have a scrollbar OMG). First time I've used Page Up to go back up on a site :)))

</comment><comment><USER35> I love the idea of rolling your own solutions, it's fun and challenging and you'll learn a lot especially when it comes to scaling. For actual deployment for a startup though, I struggle to see any benefit whatsoever.

</comment><comment><USER36> Same DIY analytics with Google Storage + BigQuery: [https://github.com/fedia/bigpapa](https://github.com/fedia/bigpapa)

Still googly, but less scalability issues.

</comment><comment><USER37> I did something similar a couple years ago for a game, where I simply wanted to track active player count over time: [https://devlog.disco.zone/2016/09/02/google-sheets- analytics...](https://devlog.disco.zone/2016/09/02/google-sheets-analytics/)

Roll-your-own is really nice when you have small data and when you have simple numbers to query (such as "get number of active players").

</comment><comment><USER38> This is awesome! I'm still an analytics noob with a tiny bit of GA experience. I have my own site I'm launching but it's totally DIY, for the DIY punk/hardcore scene. Some of the first party suggestions in this thread have been great!

</comment><comment><USER39> nice. heads up though; your link to netlify goes to neltify which doesnt exist.

<comment><AUTHOR> Cool, thanks

</comment></comment><comment><USER40> I'd like to do the same for a mobile app I'm building, but I find Firebase has a lot of features that would be hard to build out myself. i.e User Session length, install attribution

</comment><comment><USER41> I really like the in line links on this page. Not having to open a new tab for a one line explanation is great. I wonder if this should be a browser level implementation one day.

</comment><comment><USER42> It seems like losing sessions because people closed their browser or unchecked exit events is potentially the weekest point of the setup. Still, very nice and user friendly!

</comment><comment><USER43> Can anyone post a good guide as to what analytics I should be collecting? I want to implement a system of my own like this, but would like a quick primer on the subject.

<comment><USER3> I've found the documentation here to be very comprehensive if you want to start learning why/what/how:

[https://github.com/snowplow/snowplow/wiki/javascript- tracker](https://github.com/snowplow/snowplow/wiki/javascript-tracker) [https://github.com/snowplow/snowplow/wiki/canonical-event- mo...](https://github.com/snowplow/snowplow/wiki/canonical-event-model) [https://developer.matomo.org/api-reference/tracking- api](https://developer.matomo.org/api-reference/tracking-api)

</comment></comment><comment><USER44> All the effort to avoid Google Analytics, only to turn around and use Google Sheets?

</comment><comment><USER45> When I read this title I imagined a person getting out a chest of D&D dice. Change my mind.

</comment><comment><USER46> I've been checking the server logs for over 13 years now with awstats and that has always covered my needs. javascript based logging makes no sense.

</comment><comment><USER47> Your site has no visible scroll bar. What was your design reason for hiding it? It's probably one of the most useful controls to use on a site. You should track its usage with your own analytics.

Or for now, track how many people moved their mouse to where they expected to see and use a scroll bar, noticed there was none and then closed the page.

<comment><USER48> >You should track its usage with your own analytics.

I appreciate the joke, but don't give them ideas. Since one can't normally track interactions with the scrollbar, I wouldn't be surprised if they were to implement their own "scrollbar" from DIVs to be able to track if the user had dragged it.

Just put back the built-in scrollbar. Period.

<comment><USER13> You don't track the scrollbar itself, just scroll depth by quartile

[https://www.simoahava.com/analytics/scroll-depth-trigger- goo...](https://www.simoahava.com/analytics/scroll-depth-trigger-google-tag- manager/)

<comment><USER48> This can't differentiate between scrolling via the scrollbar and other means (keyboard, trackpad, mouse wheel).</comment></comment></comment></comment>


Apply Unix Principles to Write Testable Microservices - <AUTHOR> https://microservices-on-my-mind.blogspot.com/2019/04/break-functional-and-orchestration.html?m=1 <comment><USER3> In 1997 at OOPSLA Alan Kay said "every object on the internet should have a URL". The response at that time (and many years afterward) could be generally described as smugnorant mockery. A bunch of smart-asses decided that "it doesn't work", mostly by sharing gossip about CORBA failures, rather than looking into relevant research and experiments.

20+ years later we get the same idea presented as some groundbreaking revelation, except it's made orders of magnitude more complicated by the virtue of being an ad-hoc accretion of "hip" technologies. All created with zero overarching vision by people who clearly aren't even aware of past research or ideas. The crowds applaud - themselves.

This is depressing.

<comment><USER3> Let me just point out some ideas that seem to have been forgotten:

\- The idea that machines can exchange live objects, which can bring their "friends" along. (Before anyone says something about this not working - <USER2> is very similar to how we load JavaScript in the browser on nearly every page on the web. Except way more formalized and universal.)

\- The idea that concurrency can be achieved by versioning object changes and reconciling different version of reality when you need to. Pseudo-time, etc.

\- The idea that objects that respond the same to the same messages are equivalent. Which extends to the notion that a node on the network can be modeled as an object and conversely a local object can be made a node on the network.

...

Joe Armstrong had a lot of good intuitions when designing Erlang. I's a pity that after a brief surge in popularity Erlang/Elixir have been downgraded to "not hip enough" status.

...

Another thing. If you like Unix pipes, you have to love late-bound dynamically typed objects that communicate through message passing, because Unix pipes can be generalized as such. Yet I routinely see people who claim to like the former and hate the latter.

It's especially depressing how many people don't see the point of "message passing" as the fundamental building block of programming.

Just a couple examples why it matters:

\- This progression. Object interface -> inter-object protocol (extends to sequences and state) -> object communication language (allows objects/system to _derive_ protocols).

\- If objects communicate through messages then object interactions with its environment are also messages. Which means you can create virtualized environments for an object by restricting or manipulating those messages.

<comment><USER4> Yeah, with all this micorservices, serverless, lambda hype I can't help but feel this all was addressed, at least in the Java world, with EJB.

<comment><USER5> Yep, the complexity of deploying docker and k8s versus uploading an WAR or EAR file into a JEE application server.

One of the nice things of being in Java and .NET lands since day one, and C++ since its kindergarten years, is to see all the cool kids coming up with "totally new stuff". watch them re-learn what we already did, and eventually collect some improvements and those stacks adopt what was actually new since the last reboot.

In the long run the turtle still seems a better option.

</comment></comment></comment></comment><comment><USER6> Sounds like a nightmare to me. Something that could've been a module in a regular application is split across multiple services introducing multiple network calls that can fail (not to mention the response time once the workflow grows). If people who don't know how to write modular code start using microservices as a solution, the only thing that changes is that you end up with distributed big ball of mud instead of the regular big ball of mud.

<comment><USER7> It's worth clarifying that workflows are generally used to coordinate long running processes that can last up to days - think coordinating receiving an order, picking, dispatch etc. It's not intended to fit within the space of an http request/response.

As for should everything be in a module? Idk, but it's a much nicer place to start than microservices

</comment></comment><comment><USER8> Microservices are an organizational pattern: "we have 500 programmers working on one application, how the hell do we do it."

Solution: break up application into independent projects, each team of developers works on one of them, have extra team that builds tooling to make everyone happy.

Using microservices with a small teams is a terrible idea, usually, because you're switching to a distributed system when you don't need to. And distributed systems are hard to get right. And you're not even following the organizational pattern, because you have one team with N microservices.

<comment><USER5> And even then, those 500 programmers can be distributed across application modules, delivering libraries, no need for microservices.

<comment><USER9> Kind of. If you need to scale compute across multiple nodes, starting out as some kind of service abstraction over a method call will help. But most people don't need that scale.

<comment><USER5> We already learned back in the days of SUN RPC how scaling across method calls works in practice across the network.

<comment><USER9> By over, I meant over and above, fwiw.

I think RPC is a dreadful network programming abstraction. Anything that makes you think it has the same QoS as local calls is just a trap, stuff that works in development and fails horribly in production. Errors, latency, timeouts, retries etc are first class concerns once you bring in the network, far more so than local.

</comment></comment></comment></comment></comment><comment><USER7> The workflow/orchestrator pattern is key to keeping services decoupled and simplifying your code. I'm working on a Typescript library to encapsulate all the technical detail in [https://github.com/node- ts/bus/tree/master/packages/bus-work...](https://github.com/node- ts/bus/tree/master/packages/bus-workflow) so that you only worry about message inputs and outputs keeping it easy to see the logical flows of your system.

It's Inspired from the EIP pattern ([https://www.enterpriseintegrationpatterns.com/patterns/messa...](https://www.enterpriseintegrationpatterns.com/patterns/messaging/ProcessManager.html)) and frameworks from other languages that have had this type of thing for years.

The author also hints about an alternative, commonly known as "routing slips" ([https://www.enterpriseintegrationpatterns.com/patterns/messa...](https://www.enterpriseintegrationpatterns.com/patterns/messaging/RoutingTable.html)), where the "next step" is loaded into the message header and gets passed down to the next handler as part of a broader process. This removes the need for a centralised orchestrator, but has a huge downside of being tough to handle failure paths, compensating logic or switching logic.

</comment><comment><USER10> Funny, I just read man pages for the ip command and I am no longer sure about "do one thing..." tningy

<comment><USER11> `ip` and `openssl` are both mega-guilty of this! At least the man pages for the `ip` sub-commands are sanely named though..

example:

`man openssl` to find how to use `rand` sub-command, instructs you to reference `man rand`.

`man ip` to find how to use the `address` sub-command, instructs you to reference `man ip-address`.

</comment></comment><comment><USER12> There's some irony in the example.

> ls -a | grep "gear"

Using 2 services to do the job of 1 (find)

<comment><USER13> I feel like you missed the point of the article. Programs like find where taking the functions of two (and more) smaller more finite subsystems and rolling them into one, more complicated system. That added complication assumes all the functionality you would ever need is baked in -- in this case it might be, but this was just a example to start out with.

$ man find | wc -l 1020

$ man ls | wc -l 207

$ man grep | wc -l 400</comment></comment>


Quiet Subdivision Is Home to a Booming N.F.L. Blog - <AUTHOR> http://www.nytimes.com/2009/11/21/sports/football/21florio.html?_r=1&hpw

<comment><AUTHOR> Interesting in how a one-man blog, originally a sideline activity, became an important media property associated with NBC Sports.</comment>