text:
Pomax — New Entry
Pomax — RSS description testing
My RSS generator wasn't adding article bodyies to the RSS, which caused some problems for certain RSS readers. Let's see if this fixes it.
Patrick McManus — Proxy Connections over TLS - Firefox 33
This feature is from bug 378637 and is sometimes known as HTTPS proxying. I find that naming a bit ambigous - the feature is about connecting to your proxy server over HTTPS but it supports proxying for both http:// and https:// resources (as well as ftp://, ws://, and ws:/// for that matter). https:// transactions are tunneled via end to end TLS through the proxy via the CONNECT method in addition to the connection to the proxy being made over a separate TLS session.. For https:// and wss:// that means you actually have end to end TLS wrapped inside a second TLS connection between the client and the proxy.
There are some obvious and non obvious advantages here - but proxying over TLS is strictly better than traditional plaintext proxying. One obvious reason is that it provides authentication of your proxy choice - if you have defined a proxy then you're placing an extreme amount of trust in that intermediary. Its nice to know via TLS authentication that you're really talking to the right device.
Also, of course the communication between you and the proxy is also kept confidential which is helpful to your privacy with respect to observers of the link between client and proxy though this is not end to end if you're not accessing a https:// resource. Proxying over TLS connections also keep any proxy specific credentials strictly confidential. There is an advantage even when accessing https:// resources through a proxy tunnel - encrypting the client to proxy hop conceals some information (at least for that hop) that https:// normally leaks such as a hostname through SNI and the server IP address.
Somewhat less obviously, HTTPS proxying is a pre-requisite to proxying via SPDY or HTTP/2. These multiplexed protocols are extremely well suited for use in connecting to a proxy because a large fraction (often 100%) of a clients transactions are funneled through the same proxy and therefore only 1 TCP session is required when using a prioritized multiplexing protocol. When using HTTP/1 a large number of connections are required to avoid head of line blocking and it is difficult to meaningfully manage them to reflect prioritization. When connecting to remote proxies (i.e. those with a high latency such as those in the cloud) this becomes an even more important advantage as the handshakes that are avoided are especially slow in that environment.
This multiplexing can really warp the old noodle to think about after a while - especially if you have multiple spdy/h2 sessions tunneled inside a spdy/h2 connection to the proxy. That can result in the top level multiplexing several streams with http:// transactions served by the proxy as well as connect streams to multiple origins that each contain their own end to end spdy sessions carrying multiple https:// transactions.
To utilize HTTPS proxying just return the HTTPS proxy type from your FindProxyForURL() PAC function (instead of the traditional HTTP type). This is compatible with Google's Chrome, which has a similar feature.
function FindProxyForURL(url, host) {
if (url.substring(0,7) == "http://") {
return "HTTPS proxy.mydomain.net:443;"
return "DIRECT;"
Squid supports HTTP/1 HTTPS proxying. Spdy proxying can be done via Ilya's node.js based spdy-proxy. nghttp can be used for building HTTP/2 proxying solutions (H2 is not yet enabled by default on firefox release channels - see about:config network.http.spdy.enabled.http2 and network.http.spdy.enabled.http2draft to enable some version of it early). There are no doubt other proxies with appropriate support too.
If you need to add a TOFU exception for use of your proxy it cannot be done in proxy mode. Disable proxying, connect to the proxy host and port directly from the location bar and add the exception. Then enable proxying and the certificate exception will be honored. Obviously, your authentication guarantee will be better if you use a normal WebPKI validated certificate.
Soledad Penades — "Invest in the future, build for the web!", take 2, at OSOM
I am right now in Cluj-Napoca, in Romania, for OSOM.ro, an small totally non profit volunteer-organised conference. I gave an updated, shorter revised version of the talk I gave at Amsterdam past June. As usual here are the slides and the source for the slides.
It is more or less the same, but better, and I also omitted some sections and spoke a bit about Firefox Developer Edition.
Also I was wearing this Fox-themed sweater which was imbuing me with special powers for sure:
(I found it at H & M past Saturday, there are more animals if foxes aren't your thing).
There were some good discussions about open source per se, community building and growing. And no, talks were not recorded.
I feel a sort of strange emptiness now, as this has been my last talk for the year, but it won't be long until other commitments fill that vacuum. Like MozLandia—by this time next week I'll be travelling to, or already in, Portland, for our work week. And when I'm back I plan to gradually slide into a downward spiral into the idleness. At least until 2015.
Looking forward to meeting some mozillians I haven't met yet, and also visiting Ground Kontrol again and exploring new coffee shops when we have a break in Portland, though
Kevin Ngo — 'Card Not Formatted' Error on Pentax Cameras with Mac OSX Card Reader
With some 64GB SDHC and SDXC cards on Pentax (and possibly other) cameras, you might get a 'Card Not Formatted' error. It may happen if you take some shots, plug the SD card into your Mac's card reader, upload the shots, and then unplug it. I've seen the error on my K30 and K3. Though, it's not an issue with the camera or the card.
The issue is with unplugging it. With some SD cards on OSX, the SD card has to be properly ejected rather than straight-up unplugging it. Or else it'll be in some sort of weirdly formatted state. That may be obvious, but I never ran into issues unplugging cards before.
If you hit the error, you don't have to reformat the card. Simply plug it back into your machine, eject it, and then everything will have properly torn down for the card to be usable.
Gregory Szorc — Test Drive the New Headless Try Repository
Mercurial and Git both experience scaling pains as the number of heads in a repository approaches infinity. Operations like push and pull slow to a crawl and everyone gets frustrated.
This is the problem Mozilla's Try repository has been dealing with for years. We know the solution doesn't scale. But we've been content kicking the can by resetting the repository (blowing away data) to make the symptoms temporarily go away.
One of my official goals is to ship a scalable Try solution by the end of 2014.
Today, I believe I finally have enough code cobbled together to produce a working concept. And I could use your help testing it.
I would like people to push their Try, code review, and other miscellaneous heads to a special repository. To do this:
$ hg push -r . -f ssh://firstname.lastname@example.com/gecko-headless
That is:
- Consider the changeset belonging to the working copy
- Allow the creation of new heads
- Send it to the gecko-headless repo on hg.gregoryszorc.com using SSH
Here's what happening.
I have deployed a special repository to my personal server that I believe will behave very similarly to the final solution.
When you push to this repository, instead of your changesets being applied directly to the repository, it siphons them off to a Mercurial bundle. It then saves this bundle somewhere along with some metadata describing what is inside.
When you run hg pull -r on that repository and ask for a changeset that exists in the bundle, the server does some magic and returns data from the bundle file.
Things this repository doesn't do:
- This repository will not actually send changesets to Try for you.
- You cannot
hg pullor
hg clonethe repository and get all of the commits from bundles. This isn't a goal. It will likely never be supported.
- We do not yet record a pushlog entry for pushes to the repository.
- The hgweb HTML interface does not yet handle commits that only exist in bundles. People want this to work. It will eventually work.
- Pulling from the repository over HTTP with a vanilla Mercurial install may not preserve phase data.
The purpose of this experiment is to expose the repository to some actual traffic patterns so I can see what's going on and get a feel for real-world performance, variability, bugs, etc. I plan to do all of this in the testing environment. But I'd like some real-world use on the actual Firefox repository to give me peace of mind.
Please report any issues directly to me. Leave a comment here. Ping me on IRC. Send me an email. etc.
Update 2014-11-21: People discovered a bug with pushed changesets accidentally being advanced to the public phase, despite the repository being non-publishing. I have fixed the issue. But you must now push to the repository over SSH, not HTTP.
Asa Dotzler — Flame Distribution Update
About three weeks ago, I ran out of Flame inventory for Mozilla employees and key volunteer contributors. The new order of Flames is arriving in Mountain View late today (Friday) and I'll be working some over the weekend, but mostly Monday to deliver on the various orders you all have placed with me through email and other arrangements.
If you contacted me for a Flame or a batch of Flames, expect an email update in the next few days with information about shipping or pick-up locations and times. Thanks for your patience these last few weeks. We should not face any more Flame shortages like this going forward.
Jennie Rose Halperin — Townhall, not Shopping Mall! Community, making, and the future of the Internet
I presented a version of this talk at the 2014 Futurebook Conference in London, England. They also kindly featured me in the program. Thank you to The Bookseller for a wonderful conference filled with innovation and intelligent people!
A few days ago, I was in the Bodleian Library at Oxford University, often considered the most beautiful library in the world. My enthusiastic guide told the following story:
After the Reformation (when all the books in Oxford were burned), Sir Thomas Bodley decided to create a place where people could go and access all the world's information at their fingertips, for free.
"What does that sound like?" she asked. "…the Internet?"
While this is a lovely conceit, the part of the story that resonated with me for this talk is the other big change that Bodley made, which was to work with publishers, who were largely a monopoly at that point, to fill his library for free by turning the library into a copyright library. While this seemed antithetical to the ways that publishers worked, in giving a copy of their very expensive books away, they left an indelible and permanent mark on the face of human knowledge. It was not only preservation, but self-preservation.
Bodley was what people nowadays would probably call "an innovator" and maybe even in the parlance of my field, a "community manager."
By thinking outside of the scheme of how publishing works, he joined together with a group of skeptics and created one of the greatest knowledge repositories in the world, one that still exists 700 years later. This speaks to a few issues:
Sharing economies, community, and publishing should and do go hand in hand and have since the birth of libraries. By stepping outside of traditional models, you are creating a world filled with limitless knowledge and crafting it in new and unexpected ways.
The bound manuscript is one of the most enduring technologies. This story remains relevant because books are still books and people are still reading them.
As the same time, things are definitely changing. For the most part, books and manuscripts were pretty much identifiable as books and manuscripts for the past 1000 years.
But what if I were to give Google Maps to a 16th Century Map Maker? Or what if I were to show Joseph Pulitzer Medium? Or what if I were to hand Gutenberg a Kindle? Or Project Gutenberg for that matter? What if I were to explain to Thomas Bodley how I shared the new Lena Dunham book with a friend by sending her the file instead of actually handing her the physical book? What if I were to try to explain Lena Dunham?
These innovations have all taken place within the last twenty years, and I would argue that we haven't even scratched the surface in terms of the innovations that are to come.
We need to accept that the future of the printed word may vary from words on paper to an ereader or computer in 500 years, but I want to emphasize that in the 500 years to come, it will more likely vary from the ereader to a giant question mark.
International literacy rates have risen rapidly over the past 100 years and companies are scrambling to be the first to reach what they call "developing markets" in terms of connectivity. In the vein of Mark Surman's talk at the Mozilla Festival this year, I will instead call these economies post-colonial economies.
Because we (as people of the book) are fundamentally idealists who believe that the printed word can change lives, we need to be engaged with rethinking the printed word in a way that recognizes power structures and does not settle for the limited choices that the corporate Internet provides (think Facebook vs WhatsApp). This is not as a panacea to fix the world's ills.
In the Atlantic last year, Phil Nichols wrote an excellent piece that paralleled Web literacy and early 20th century literacy movements. The dualities between "connected" and "non-connected," he writes, impose the same kinds of binaries and blind cure-all for social ills that the "literacy" movement imposed in the early 20th century. In equating "connectedness" with opportunity, we are "hiding an ideology that is rooted in social control."
Surman, who is director of the Mozilla Foundation, claims that the Web, which had so much potential to become a free and open virtual meeting place for communities, has started to resemble a shopping mall. While I can go there and meet with my friends, it's still controlled by cameras that are watching my every move and its sole motive is to get me to buy things.
85 percent of North America is connected to the Internet and 40 percent of the world is connected. Connectivity increased at a rate of 676% in the past 13 years. Studies show that literacy and connectivity go hand in hand.
How do you envision a fully connected world? How do you envision a fully literate world? How can we empower a new generation of connected communities to become learners rather than consumers?
I'm not one of these technology nuts who's going to argue that books are going to somehow leave their containers and become networked floating apparatuses, and I'm not going to argue that the ereader is a significantly different vessel than the physical book.
I'm also not going to argue that we're going to have a world of people who are only Web literate and not reading books in twenty years. To make any kind of future prediction would be a false prophesy, elitist, and perhaps dangerous.
Although I don't know what the printed word will look like in the next 500 years,
I want to take a moment to think outside the book,
to think outside traditional publishing models, and to embrace the instantaneousness, randomness, and spontaneity of the Internet as it could be, not as it is now.
One way I want you to embrace the wonderful wide Web is to try to at least partially decouple your social media followers from your community.
Twitter and other forms of social media are certainly a delightful and fun way for communities to communicate and get involved, but your viral campaign, if you have it, is not your community.
True communities of practice are groups of people who come together to think beyond traditional models and innovate within a domain. For a touchstone, a community of practice is something like the Penguin Labs internal innovation center that Tom Weldon spoke about this morning and not like Penguin's 600,000 followers on Twitter. How can we bring people together to allow for innovation, communication, and creation?
The Internet provides new and unlimited opportunities for community and innovation, but we have to start managing communities and embracing the people we touch as makers rather than simply followers or consumers.
The maker economy is here— participatory content creation has become the norm rather than the exception. You have the potential to reach and mobilize 2.1 billion people and let them tell you what they want, but you have to identify leaders and early adopters and you have to empower them.
How do you recognize the people who create content for you? I don't mean authors, but instead the ambassadors who want to get involved and stay involved with your brand.
I want to ask you, in the spirit of innovation from the edges
What is your next platform for radical participation? How are you enabling your community to bring you to the next level? How can you differentiate your brand and make every single person you touch psyched to read your content, together? How can you create a community of practice?
Community is conversation. Your users are not your community.
Ask yourself the question Rachel Fershleiser asked when building a community on Tumblr: Are you reaching out to the people who want to hear from you and encouraging them or are you just letting your community be unplanned and organic?
There reaches a point where we reach the limit of unplanned organic growth. Know when you reach this limit.
Target, plan, be upbeat, and encourage people to talk to one another without your help and stretch the creativity of your work to the upper limit.
Does this model look different from when you started working in publishing? Good.
As the story of the Bodelian Library illustrated, sometimes a totally crazy idea can be the beginning of an enduring institution.
To repeat, the book is one of the most durable technologies and publishing is one of the most durable industries in history. Its durability has been put to the test more than once, and it will surely be put to the test again. Think of your current concerns as a minor stumbling block in a history filled with success, a history that has documented and shaped the world.
Don't be afraid of the person who calls you up and says, "I have this crazy idea that may just change the way you work…" While the industry may shift, the printed word will always prevail.
Publishing has been around in some shape or form for 1000 years. Here's hoping that it's around for another 1000 more.
Mozilla Reps Community — Reps Weekly Call – November 20th 2014
Last Thursday we had our regular weekly call about the Reps program, where we talk about what's going on in the program and what Reps have been doing during the last week.
- FOSDEM update.
- Post-event metrics and receipts (Important reminder)
- Firefox Tiles Reps FAQ.
- 10 days of Mozillians.
- Yahoo agreement.
- Community newsletter.
- #fx10 Jakarta
- Pending Reps applicants.
AirMozilla video
Don't forget to comment about this call on Discourse and we hope to see you next week!
Pascal Finette — Introduction to Exponential Thinking and Technology
A few weeks ago I had the privilege to deliver the closing keynote at GroupM's What's Next Illuminate conference in New York City. I gave a short introduction to exponential thinking (the stuff we teach at Singularity University) and then walked the audience through a whole bunch of examples (focussed on media).
The talk was a shortened and more media-related version of my "Technology Trends" talk I give here at SU to groups from all over the world quite often.
Here's the video:
Mike Taylor — document.body.scrollTop vs document.documentElement.scrollTop
Here's a track from Web Compatibility's Greatest Hits Album (Volume I) that just doesn't want to go away—with the latest club remix titled "scrolling to sections from the menu in the mobile Google News site doesn't work due to setting
scrollTop position on
document.body in Firefox for Android".
Here's some background for those with less refined musical tastes.
(Why yes I can do this bad metaphor stuff all day long, why do you ask?)
If you want to get or set the vertical scroll position of a document, you can use
element.scrollTop. According to the CSSOM View Module spec, if you're in standards mode you need to operate on the document's root element (the
<html> element—or
document.documentElement in DOM land). In quirks mode you would use the
<body> element, via
This works in IE and Firefox and the late Presto Opera.
In Blink and WebKit browsers, it's the exact opposite. Both have attempted to implement the standard (safari, chrome), but both have had to back out their patches due to sites breaking (some Google properties and webkit.org among them, as luck would have it).
The bug that was filed against WebKit for Facebook breaking as a result of changing to the standard is especially interesting because it shows the tension between following standards (and other browsers) and breaking sites for their own users.
It's also a good example of how user-agent-string-based development can sometimes make it hard, if not impossible, to remove some of the crappier stuff from the web platform.
Here's some excerpts, but the whole bug is a good read.
It really doesn't matter how faithfully you implemented the spec. If it causes a major backward compatibility with the Web, we can't have it.
Yes, the regression doesn't reproduce if we fake the UA string as I mentioned in the comment #31.
Maybe sites will update one day and let other browsers do the right thing™. (Not that I'm holding my breath over here.)
Until then I guess we get to have fun writing stuff like this (found on apple.com a few weeks back):
(document.documentElement ||
document.body.parentNode ||
Yunier José Sosa Vázquez — Nueva estrategia de búsqueda para Firefox promueve la elección y la innovación
Google dejará de ser el buscador por defecto en Firefox a partir de diciembre para Estados Unidos, según el anuncio oficial publicado por Chris Beard en el blog de Mozilla. En otras regiones del planeta también Google será reemplazado por otros "competidores" en aras de promover la elección en la Web.
Las búsquedas son una parte esencial en la experiencia en Internet para todos, solamente los usuarios de Firefox realizan más de 100 millones de búsquedas por año.
Con Firefox, Mozilla popularizó la integración del buscador en el navegador aliándose con compañías de Internet como Google, Yahoo y otras más para generar remuneración y avanzar en su misión. Google ha sido el buscador por defecto globalmente en Firefox desde el 2004 y al vencerse este año el contrato, Mozilla ha tomado esto como una oportunidad para revisar su estrategia y explorar otras opciones.
Según Beard, al evaluar los socios, para Mozilla la primera consideración fue asegurar una estrategia alineada a los valores de elección e independencia, capaz de posicionarlos y avanzar en su misión para brindar un mejor servicio a los usuarios y a la Web. Al final, cada opción disponible por los socios era fuerte, mejorando los términos económicos y reflejando los valores que Firefox brinda al ecosistema. Pero una opción sobresalió por encima de las demás.
Mozilla ha finalizado su práctica de tener un único buscador global en Firefox y en su lugar han adoptado una forma flexible que permitirá tener buscadores por país:
Estados Unidos
- Yahoo será el buscador por defecto durante los próximos 5 años.
- Iniciando en diciembre, los usuarios de Firefox serán introducidos en una nueva y mejorada experiencia de búsqueda con Yahoo en la cual resalta una moderna interfaz.
- Con esta asociación, Yahoo soportará Do Not Track (DNT) en Firefox
- Google, Bing, DuckDuckGo, eBay, Amazon, Twitter y Wikipedia continuarán siendo opciones de búsqueda alternativas.
- Yandex será el buscador por defecto.
- Google, DuckDuckGo, OZON.ru, Price.ru, Mail.ru, y Wikipedia continuarán siendo opciones de búsqueda alternativas.
- Baidu continuará siendo el buscador por defecto.
- Google, Bing, Youdao, Taobao y otras opciones locales continuarán siendo opciones de búsqueda alternativas.
Resto de los países
- Firefox es un navegador para todo el mundo, indiferente de la preferencia de búsqueda.
- Firefox ahora tiene más opciones en cuanto a proveedores de búsqueda que cualquier otro navegador, con 61 proveedores preinstalados por 88 versiones en lenguas diferentes.
- A pesar que Mozilla decidió no renovar el contrato, Google continuará siendo una opción de búsqueda preinstalada.
- Google continuará proveyendo la Geolocalización y Navegación segura en Firefox.
- Mozilla se enfocará en expandir su trabajo con socios motivados a explorar innovadoras y nuevas interfaces de búsqueda, experiencias de contenidos, y mejoras de privacidad en escritorio y móvil.
Por eso la independencia importa. Al no perseguir el lucro, nos permite crear opciones diferentes. Opciones que mantienen la Web abierta, por todos lados e independiente. Pensamos que hoy se da un gran paso hacia esa dirección.
Cabe destacar que Google se mantendrá disponible como buscador por defecto para los demás países pero la oferta está abierta para que otros socios interesados se sumen a esta estrategia.
Fuente: The Mozilla Blog
Fuente: Google System
Mozilla Fundraising — Mozilla Now Accepts Bitcoin
Pascal Finette — Community Building (Lessons from Mozilla)
Today I had the great honor and pleasure to teach a class on Building (Online) Communities at Electronic Arts' internal leadership development program. For my presentation I took a trip down memory lane and pulled out the key insights and learnings from my time at Mozilla.
Here's the deck:
Lukas Blakk — Artisanal Contributors
Part 1: Start In Person
Ascend had very few 'rules' but there was one which was non-negotiable: it's an in-person program. We didn't do distance learning, online coursework, or video-based classes. We did bring in a couple of speakers virtually to speak to the room of 20 participants but the opposite was never true.
This was super important in how we were going to build a strong cohort. Don't get me wrong, I'm a fan of remote work and global contribution as well as with people working from wherever they are. This was a 6 week intensive program though and in order to build the inter-dependent cohort I was hoping to1, it had to be in person at first. Those cruicial early stages where someone is more likely to 'disappear' if things were hard, confusing, or if they couldn't get someone's attention to ask a question.
It's been over 5 years since I graduated from my software development program and over 8 years since I started lurking in IRC channels2 and getting to know Mozillians in digital space first. I wouldn't have stuck with it, or gotten so deeply involved without my coursework with Dave Humphrey though. That was a once a week class, but it meant the world to be in the same room as other people who were learning and struggling with the same or similar problems. It was an all-important thread connecting what I was trying to do in my self-directed time with actual people who could show more caring about me and my ability to participate.
Even as an experienced open source contributor I can jump into IRC channels for projects I'm trying to work on – most recently dd-wrt for my home server setup – and when I ask a question (with lots of evidence for what I've already tried and an awareness of what the manual has to say) I get no response, aka: Crickets. There are a host of reasons, and I know more than a beginner might about what those could be: timezones, family comitments, no one with the expertise currently in the channel, and more. None of that matters when you're new to this type of environment. Silence is interpreted as a big "GO AWAY YOU DON'T BELONG HERE" despite the best intentions of any community.
In person learning is the best way to counter that. Being able to turn to a colleague or a mentor and say what's happening helps get you both reassurance that it's not you, but also someone who can help you get unstuck on what to do next. While you wait for a response, check out this other topic we're studying. Perhaps you can try other methods of communication too, like in a bug or an email.
Over the course of our first pilot I also discovered that removing myself from the primary workroom the Ascend participants were in helped the cohort to rapidly built up strengths in helping each other first3. The workflow looked more like: have a question/problem, ask a cohort member (or several), if you still can't figure it out ask on IRC, and if then if you're still stuck find your course leader. This put me at the end of the escalation path4 and meant that people were learning to rely both on in-person communications as well as IRC but more importantly were building up the muscle of "don't stop asking for help until you get it" which is really where open source becomes such a great space to work in.
Back to my recent dd-wrt experience, I didn't hear anything back in IRC and I felt I had exhausted the forums & wikis their community provided. I started asking in other IRC channels where tech-minded people hung out (thanks womenwhohack!) and then I tried yet another search with slightly different terms. In the end I found what I needed in a YouTube tutorial. I hope that sufficiently demonstrates that a combination of tactics are what culminate in an ability to be persistent when learning in open source projects.
Never underestimate the importance of removing isolation for new contributors to a project. In person help, even just at first, can be huge.
- Because the ultimate goal of Ascend was to give people skills for long-term contribution and participation and a local cohort of support and fellow learners seemed like a good bet for that to be possible once the barrier-removing help of the 6 week intensive was no longer in place. ↩
- By the way, I'm such a huge fan of IRC that I wrote the tutorial for it at Mozilla in order to help get more non-engineering folks using it, in my perfect world everyone is in IRC all the time with scrollback options and logging. ↩
- Only after the first three weeks when we moved to the more independent work, working on bugs, stage. ↩
- Which is awesome because I was always struggling to keep up with the course creation as we were running it, I didn't realize that teaching 9-5 was asking for disaster and next time we'll do 10-4 for the participants to give the mentors pre and post prep time. ↩
Lukas Blakk — Release Management Tooling: Past, Present, and Future
Release Management Tooling: Past, Present, and Future
As I was interviewing a potential intern for the summer of 2015 I realized I had outlined all our major tools and what the next enhancement for each could be but that this wasn't well documented anywhere else yet.
By coming to Release Management from my beginnings as a Release Engineer, I've been a part of seeing our overall release automation improve across the whole spectrum of what it takes to put out packaged software for multiple platforms and we've come a long way so this post is also intended to capture how the main tools we use have gotten to their current state as well as share where they are heading.
Past: Release Manager on point for a release sent an email to the Release-Drivers mailing list with an hg changeset, a version, build number, and this was the "go" to build for Release Engineering to take over and execute a combination of automated/manual steps (there was even a time when it was only said in IRC, email became the constant when Joduinn pushed for consistency and a traceable trail of events). Release Engineers would update a config files & locale changes, get them attached to a bug, approved, uplifted, then go reconfigure the build machines so they could kick off the release build automation.
Present: Ship-It is an app developed by Release Engineering (bhearsum) that allows a Release Manager to input the configurations needed (changeset, version, build number, partials to be created, l10n changesets) all in one place, and on submit the build automation picks up this change from a db, reconfigures the build machine, and triggers builds. When all goes well, there are zero human hands between the "go" and the availability of builds to QA.
Future: In two parts:
1. To have a simple app that can take a list of bug numbers and check them for landing to {branch} (where branch is Beta, Release, or ESR), once all the bug numbers listed have landed, check tree herder for green status on that last changeset, submit to Ship-It if builds are successful. Benefits: hands off even sooner, knowing that all the important fixes are on the branch in question, and that the tree is totally green prior to build (sometimes we "go" without all the results because of human timing needs).
2. Complete End-To-End Release Checklist, dynamically updated to show what stage a release job is at and who's got the ball in their court. This should track from buglist added (for the final landings a RM is waiting on) all the way until the release notes are live and QA signs off on updates for the general release being in the wild.
Nucleus (aka Release Note App)
Past: Oh dear, you probably don't even want to know how our release notes used to be made. It's worse than sausage. There was a sqlite db file, a script that pulled from that db and generated html based on templates and then the Release Manager had to manually re-order the html to get the desired appearance on final pages, all this was then committed to SVN and with that comes the power to completely break mozilla.org properties. Fun stuff. Really. Also once Release Management was more than just one person we shared this sqlite db over Dropbox which had some fun quirks, like clobbering your changes if two people had the file open at the same time. Nowhere to go but up from here!
Present: Thanks to the web production team (jgmize, hoosteeno, craigcook, jbertsch) we got a new Django app in place that gives us a proper databse that's redundant, production quality, and not in our hands. We add in release notes as well as releases and can publish notes to both staging and production without any more commits to SVN. There's also an API that can be scripted to.
Future: The future's so bright in this area, let me get my shades. We have a flag in Bugzilla for relnote-firefox where it can get set to ? when something is nominated and then when we decide to take on that bug as a release note we can set it to {versionNum}+. With a little tweaking on the Bugzilla side of things we could either have a dedicated field for "release-note text" or we could parse it out of a syntax in a comment (though that's gonna be more prone to user error, so I prefer the former) and then automatically grab all the release notes for a version, create the release in Nucleus, add the notes, publish to staging, and email the link around for feedback without any manual interference. This also means we can dynamically adjust release notes using Bugzilla (and yes, this will need to be really cautiously done), and it makes sure that our recent convention of having every release note connect to a bug persist and become the standard.
Release Dash
Past: Our only way to visualize the work we were doing was a spreadsheet, and graphs generated from it, of how many crasher bugs were tracked for a version, how many bugs tracked/fixed over the course of 18 weeks for a version, and not much else. We also pay attention to the crash rate at ship time, whether we had to do a dot release or chemspill, and any other release-version-specific issues are sort of lost in the fray after we're a couple of weeks out from a release. This means we don't have a great sense of our own history, what we're doing that works in generating a more stable/successful release, and whether a release is in fact ready to go out the door. It's a gamble, and we take it every 6 weeks.
Present: We have in place a dashboard that is supposed to allow us to view the current crash data, select Talos (performance) data, custom bug queries, and be able to compare a current release coming down the pipe to previous releases. We do not use this dashboard yet because it's been a side project for the past year and a half, primarily being created and improved upon by fabulous – yet short-term – interns at Mozilla. The dashboard relies on Elastic Search for Bugzilla data and the cluster it points to is not always up. The dash is written in php and that's no one's strong suit on our current team, our last intern did his work by creating a Python Flask app that would work into the current dash. The present situation is basically: we need to work on this.
Future: In the future, this dashboard will be robust, reliable, production-quality (and supported), and it will be able to go up on Mozilla office screens in the dashboard rotation where it will make clear to any viewer:
* Where we are in the current release cycle
* What blockers remain for releas
* How our stability is (over/under acceptable rates)
* If we're meeting performance expectations
And hopefully more. We have to find more ways to get visibility into issues a release might hit once it's with the larger population. I'd love to see us get more of our Beta user's feedback by asking for it on specific features/fixes, get a broader Beta audience that is more reflective of our overall release population (by hardware, location, language, user types) and then grow their ability to report issues well. Then we can find ways to get that front and center too – including to developers because they are great at confirming if something unusual is happening.
What Else?
Well, we used to have an automated script that reminded teams of their open & tracked bugs on Beta/Aurora/Nightly in order to provide a priority order that was visible to devs & their managers. It's a finicky script that breaks often. I'd like to see that replaced with something that's not just a cronjob on my personal VPS. We're also this close to not needed to update product-details (still in SVN) on every release. The fact that the Release Management team has the ability to accidentally take down all mozilla.org properties when a mistake is made submitting svn propedits is not desireable or necessary. We should get the heck away from that asap.
We'll have more discussions of this in Portland, especially with the teams we work closely with and Sylvestre and I will be talking up our process & future goals at FOSDEM in 2015 as well as following it with a work week in Paris where we can put our heads down and code. Next summer we get an intern again and so we'll have another set of skilled hands to put on tooling & web service improvements.
Always improving. Always automating. These are the things that make me excited for the next year of Release Management.
Mozilla Open Policy & Advocacy Blog — Spotlight on the Open Technology Institute: A Ford-Mozilla Open Web Fellow Host
{This is the third installment in our series highlighting the 2015 Host Organizations for the Ford-Mozilla Open Web Fellows program. We are now accepting applications to be a 2015 fellow. We are thrilled to feature the New America Foundation's Open Technology Institute as a host. Over the years, OTI has been a meaningful change agent, helping to protect the free and open Web. Working at OTI, the Open Web Fellow will be developing tools that lead to greater transparency, enabling all stakeholders to better understand how public policy and business practices impact the Web experience.}
Spotlight on the Open Technology Institute: A Ford-Mozilla Open Web Fellow Host Organization
By Kevin Bankston, Policy Director, and Georgia Bullen, Senior Data Analyst; Open Technology Institute
Last month's MozFest 2014 provided us a welcome opportunity to think about what we at New America's Open Technology Institute hope to do over the next year as one of the few organizations lucky enough to host a Ford-Mozilla Open Web Fellow during that fellowship program's inaugural year. At OTI, we are committed to freedom and social justice in the digital age. To achieve these goals, we engage in policy debates, build technology, and work with communities to understand needs, test tools and build alternative models of infrastructure. And we are looking for a passionate maker to help us with our work in 2015. In particular, to help make more transparent the workings of the Internet and the companies that offer services over it.
So much of what impacts our online experience happens without us seeing it, making it easy to overlook.
For example, look at the Net Neutrality debate, where decisions made at interconnection points deep in the network have both business and policy implications. At OTI, we have tools that allow us to dig into the technical depths of the issue through our Measurement Lab platform, and we recently published a major report laying out much of that data. But we need help figuring out how to make this information more available and more clear so that policy experts, advocates, industry professionals and everyday Internet users can understand what interconnection is, how it works, and how it affects the online experience. We've started on one of these efforts by working on a visualization tool that we're calling the Measurement Lab Observatory, but there's so much more we can do with the Measurement Lab data, as well as the platform and tools to make it more accessible to everyone–if only we can find the right fellow.
With the help of the participants at our MozFest usability workshop, we thought about other ways to get people involved in Internet measurement, such as building a network troubleshooting tool that could generate new M-Lab data while also testing your connection. We also talked about developing out our Firefox Browser extension to have different themes depending on a user's needs, such as a journalist or advocate dashboard which includes recent news about Internet policy issues, or a "notebook" app with which Internet citizen scientists can run and annotate tests as part of the M-Lab research team.
These are just the types of ideas that we're hoping our incoming Ford-Mozilla Fellow can run with.
On the policy and governance side, there's also a lot more that we could be doing to reveal what happens behind the scenes between governments and Internet companies. Many companies now publish "Transparency Reports" that include information about how and when governments ask for user's data. However, there's no standardization in how companies report, making it hard to meaningfully combine or compare the data from different companies — and hard for new companies to get into the reporting game. Building on some of our previous research and education efforts around transparency reporting, in 2015 we will be launching a project called the Transparency Reporting Toolkit. We're going to build a Web portal filled with best practices information and tools to help companies create and upload reports in a standardized way, and tools for others to mash up and visualize the data from multiple companies' reports. OTI's technologists and data visualization experts are gearing up to build those tools, but it's a big project and we could use some help — possibly yours.
Ultimately, we can only make good policy with good information, and we can only get good information – and, crucially, understand that information – with good tools. We're ready to move forward on all of these projects in 2015, full steam ahead. All we need now is the right technologist to help us make those tools. If that sounds exciting to you, apply to be a 2015 Ford-Mozilla Open Web Fellow and work with us and the Mozilla community to help build new windows into the technical and political depths of the Internet.
Apply to be a 2015 Ford-Mozilla Open Web Fellow. Visit www.mozilla.org/advocacy.
Soledad Penades — Using the Firefox Developer Edition dark theme with Nightly
With a recent version of Nightly, go to about:config and set browser.devedition.theme.enabled to true.
Open DevTools (I use alt + cmd + i, or you can also go to the Tools → Web Developer → Toggle tools menu). Then open DevTools preferences by clicking on the gear icon, and select "Dark Theme" on the top right, underneath the Themes.
Screenshot for clarification:
Note: you might not get the full effect if there is "legacy stuff" in your profile. If it doesn't look as you expect… your best option might be to just create a new profile when you start the browser.
Note 2: for some reason the tabs weren't rendering correctly on my normal nightly profile because the about:config browser.tabs.drawInTitlebar entry was set to false instead of true—I set it to true and now everything looks fine for me.
Or just use the standard Firefox Developer Edition if you're not an impatient person like me
David Rajchenbach Teller — RFC: We deserve better than runtime warnings
Consider the following scenario:
- Module A prints warnings when it's used incorrectly;
- Module B uses module A correctly;
- Some future refactoring of module B starts using module A incorrectly, hence displaying the warnings;
- Nobody realises for months, because we have too many warnings;
- Eventually, something breaks.
How often has this happened to everyone of us?
This scenario has many variants (e.g. module A changed and nobody realized that module B is now in a situation it misuses module A), but they all boil down to the same thing: runtime warnings are designed to be lost, not fixed. To make things worse, many of our warnings are not actionable, simply because we have no way of knowing where they come from – I'm looking at you, Cu.reportError.
So how do we fix this?
We would certainly save considerable amounts of time if warnings caused immediate assertion failures, or alternatively test failures (i.e. fail, but only when running the unit tests). Unfortunately, we can do neither, as we have a number of tests that trigger the warnings either
- by design (e.g. to check that we can recover from such misuses of A, or because we still need a now-considered-incorrect use of an API to keep working until we have ported all the clients to the better API);
- or for historical reasons (e.g. the now incorrect use of A used to be correct, but we haven't fixed all tests that depend on it yet).
However, I believe that causing test failures is still the solution. We just need a mechanism that supports a form of whitelisting to cope with the aforementioned cases.
Introducing RuntimeAssert
RuntimeAssert is an experiment at provoding a standard mechanism to replace warnings. I have a prototype implemented as part of bug 1080457. Feedback would be appreciated.
The key features are the following:
- when a test suite is running, a call to `RuntimeAssert` causes the test suite to fail;
- when a test suite is running, a call to `RuntimeAssert` contains at least the filename/line number of where it was triggered, preferably a stack wherever available;
- individual tests can whitelist families of calls to `RuntimeAssert` and mark them either as expected;
- individual tests can whitelist families of calls to `RuntimeAssert` and mark them as pending fix;
- when a test suite is not running, a call to `RuntimeAssert` does nothing costly (it may default to PR_LOG or Cu.reportError).
Possible API:
- in JS, we trigger a test failure by calling
RuntimeAssert.fail(keyword, string or Error)from production code;
- in C++, we likewise trigger a test failure by calling
MOZ_RUNTIME_ASSERT(keyword, string);
- in the testsuite, we may whitelist errors by calling
Assert.whitelist.expected(keyword, regexp)or
Assert.whitelist.FIXME(keyword, regexp).
// // Module // let MyModule = { oldAPI: function(foo) { RuntimeAssert.fail("Deprecation", "Please use MyModule.newAPI instead of MyModule.oldAPI"); // ... }, newAPI: function(foo) { // ... }, }; let MyModule2 = { api: function() { return somePromise().then(null, error => { RuntimeAssert.fail("MyModule2.api", error); // Rather than leaving this error uncaught, let's make it actionable. }); }, api2: function(date) { if (typeof date == "number") { RuntimeAssert.fail("MyModule2.api2", "Passing a number has been deprecated, please pass a Date"); date = new Date(date); } // ... } } // // Whitelisting a RuntimeAssert in a test. // // This entire test is about MyModule.oldAPI, warnings are normal. Assert.whitelist.expected("Deprecation", /Please use MyModule.newAPI/); // We haven't fixed all calls to MyModule2.api2, so they should still warn, but not cause an orange. Assert.whitelist.FIXME("MyModule2.api2", /please pass a Date/); Assert.whitelist.expected("MyModule2.api", /TypeError/, function() { // In this test, we will trigger a TypeError in MyModule2.api, that's entirely expected. // Ignore such errors within the (async) scope of this function. });
In the long-term, I believe that RuntimeAssert (or some other mechanism) should replace almost all our calls to Cu.reportError.
In the short-term, I plan to use this for reporting
- uncaught Promise rejections, which currently require a bit too much hacking for my tastes;
- errors in XPCOM.lazyModuleGetter & co;
- failures during AsyncShutdown;
- deprecation warnings as part of Deprecated.jsm.
Doug Belshaw — Firefox Interest Dashboard: privacy-respecting analytics for your web browsing history
On a recent Mozilla project call I heard about the new Firefox Interest Dashboard. As someone who loves self-tracking, but stopped using my Fitbit due to privacy concerns, this is awesome.
Some of the numbers may be a bit off, and the categorisation certainly is in some cases, but it's a promising start! The great thing is that if you use Firefox Sync it uses your data from other installations you use, too!
From the Content Services team:
This is an early version of interest categorization we're working on. We invite you to test out this experimental beta add-on and help us out with the misclassified results. We would love to hear from you on suggestions on improvement or any feedback through the flag icon on the interest timeline.
Unlike other analytics services, the FAQ assures users that "all of the interest analysis and categorization is done on the client-side of your browser. No personal data is stored on Mozilla's servers."
Download the add-on (Firefox only)
Questions? Comments? Direct them to email@example.com or discuss in the #TeachTheWeb discussion forum.
Julien Vehent — SSL/TLS for the Pragmatic
Tonight I had the pleasure to present "SSL/TLS for the Pragmatic" to the fine folks of Bucks County Devops. It was a fun evening, and I want to thank the organizers, Mike Smalley & Ben Krein, for the invitation.
It was a great opportunity to summarize 18 months of work at Mozilla on building the Server Side TLS Guidelines. By the feedback I received tonight, and on several other occasions, I think we've achieved the goal of building a document that is useful to operations people, and made TLS just a little easier to understand.
We are not, however, anywhere done with the process of teaching TLS to the Internet. Stats speak for themselves, with 70% of sites still supporting SSLv3, 86% enabling RC4, and about 32% still not preferring PFS over RSA handshakes. But things are getting better every day, and ongoing efforts may bring safe defaults in Linux servers as soon as Fedora 21. We live in exciting times!
The slides from my talk are below, and on github as well. I hope you enjoy them. Feel free to share your comments at julien[at]linuxwall.info.
Giorgio Maone — s/http(:\/\/(?:noscript|flashgot|hackademix)\.net)/https\1/
I'm glad to announce noscript.net, flashgot.net and hackademix.net have been finally switched to full, permanent TLS with HSTS
Please do expect a smörgåsbord of bugs and
bunny funny stuff :)
Andreas Gal — Yahoo and Mozilla Form Strategic Partnership
SUNNYVALE, Calif. and MOUNTAIN VIEW, Calif., Wednesday, November 19, 2014 – Yahoo Inc. (NASDAQ: YHOO) and Mozilla Corporation today announced a strategic five-year partnership that makes Yahoo the default search experience for Firefox in the United States on mobile and desktop. The agreement also provides a framework for exploring future product integrations and distribution opportunities to other markets.
The deal represents the most significant partnership for Yahoo in five years. As part of this partnership, Yahoo will introduce an enhanced search experience for U.S. Firefox users which is scheduled to launch in December 2014. It features a clean, modern and immersive design that reflects input from the Mozilla team.
"We're thrilled to partner with Mozilla. Mozilla is an inspirational industry leader who puts users first and focuses on building forward-leaning, compelling experiences. We're so proud that they've chosen us as their long-term partner in search, and I can't wait to see what innovations we build together," said Marissa Mayer, Yahoo CEO. "At Yahoo, we believe deeply in search – it's an area of investment, opportunity and growth for us. This partnership helps to expand our reach in search and also gives us an opportunity to work closely with Mozilla to find ways to innovate more broadly in search, communications, and digital content."
"Search is a core part of the online experience for everyone, with Firefox users alone searching the Web more than 100 billion times per year globally," said Chris Beard, Mozilla CEO. "Our new search strategy doubles down on our commitment to make Firefox a browser for everyone, with more choice and opportunity for innovation. We are excited to partner with Yahoo to bring a new, re-imagined Yahoo search experience to Firefox users in the U.S. featuring the best of the Web, and to explore new innovative search and content experiences together."
To learn more about this, please visit the Yahoo Corporate Tumblr and the Mozilla blog.
About Yahoo
Yahoo is focused on making the world's daily habits inspiring and entertaining. By creating highly personalized experiences for our users, we keep people connected to what matters most to them, across devices and around the world. In turn, we create value for advertisers by connecting them with the audiences that build their businesses. Yahoo is headquartered in Sunnyvale, California, and has offices located throughout the Americas, Asia Pacific (APAC) and the Europe, Middle East and Africa (EMEA) regions. For more information, visit the pressroom (pressroom.yahoo.net) or the Company's blog (yahoo.tumblr.com).
About Mozilla
Mozilla has been a pioneer and advocate for the Web for more than a decade. We create and promote open standards that enable innovation and advance the Web as a platform for all. Today, hundreds of millions of people worldwide use Mozilla Firefox to discover, experience and connect to the Web on computers, tablets and mobile phones. For more information please visit https://www.mozilla.com/press
Yahoo is registered trademark of Yahoo! Inc. All other names are trademarks and/or registered trademarks of their respective owners.
Filed under: Mozilla
Monty Montgomery — Daala Demo 6: Perceptual Vector Quantization (by J.M. Valin)
Jean-Marc has finished the sixth Daala demo page, this one about PVQ, the foundation of our encoding scheme in both Daala and Opus.
(I suppose this also means we've finally settled on what the acronym 'PVQ' stands for: Perceptual Vector Quantization. It's based on, and expanded from, an older technique called Pyramid Vector Quantization, and we'd kept using 'PVQ' for years even though our encoding space was actually spherical. I'd suggested we call it 'Pspherical Vector Quantization' with a silent P so that we could keep the acronym, and that name appears in some of my slide decks. Don't get confused, it's all the same thing!)
David Dahl — Encryptr: 'zero knowledge' essential information storage
Encryptr is one of the first "in production" applications built on top of Crypton. Encryptr can store short pieces of text like passwords, credit card numbers and other random pieces of information privately, in the cloud. Since it uses Crypton, all data that is saved to the server is encrypted first, making even a server compromise an exercise in futility for the attacker.
A key feature is that you can run Encryptr on your phone as well as your desktop and all data is available in each place immediately. Have a look:
The author of Encryptr, my colleague Tommy @therealdevgeeks, has recently blogged about building Encryptr. I hope you give it a try and send him feedback through the Github project page.
Christian Heilmann — Simple things: styling ordered lists
This blog started as a scratch pad of simple solutions to problems I encountered. So why not go back to basics?
It is pretty easy to get an ordered list into a document. All you have to do is add an OL element with LI child elements:
<ol> <li>Collect underpants</li> <li>???</li> <li>Profit</li> </ol>
But what if you want to style the text differently from the numbers? What if you don't like that they end with a full stop? The generated numbers of the OL are somewhat of that dark magic browsers do for us (something we work on dragging into the sunlight with ShadowDOM).
In order to make those more style-able in the past you had to add another element to get a hook:
<ol class="oldschool"> <li><span>Collect underpants</span></li> <li><span>???</span></li> <li><span>Profit</span></li> </ol>
.oldschool li { color: green; } .oldschool span { color: lime; }
Which is kind of a terrible hack and doesn't quite scale as you may never know who edits your list. With newer browsers we have a better way of doing that using CSS counters. The browser support is ridiculously good, so there should be no excuse for us not to use them:
Using counter, you keep the HTML structure:
<ol class="counter"> <li>Collect underpants</li> <li>???</li> <li>Profit</li> </ol>
You then reset the counter for each of the lists with this class:
.counter { counter-reset: list; }
This means each list will start at 1 and not go on through the document tree. You then get rid of the list style and style the list item like you want to. In this case we give it a colour and we position it relative. This allows us to position other, new content in there and contain it to the list item:
.counter li { list-style: none; position: relative; color: lime; }
Once you hid the normal numbering with the list-style: none; you can create your own numbers using counter and generated CSS content:
.counter li::before { counter-increment: list; content: counter(list) '.'; position: absolute; top: 0px; left: -1.2em; color: green; }
If you wanted to remove the full stop, all you need to do is remove it in the CSS. You have now full styling control over these numbers, for example you can animate them slightly moving from one colour to another and zoom out a bit:
.animated li::before { transition: 0.5s; color: green; } .animated li:hover::before { color: white; transform: scale(1.5); }
Counters allow you for a lot of different times of numbering. You can for example add a leading zero by using counter(list,decimal-leading-zero), you can use Roman numerals with counter(list,lower-roman) or even Greek ones with counter(list,lower-greek).
If you want to see all of that in action, check out this Fiddle:
Pretty simple, and quite powerful. Here are some more places to read up on this:
Jonathan Watt — Converting Mozilla's SVG implementation to Moz2D - part 2
This is part 2 of a pair of posts describing my work to convert Mozilla's SVG implementation to directly use Moz2D. Part 1 provided some background information and details about the process. This post will discuss the performance benefits of the conversion of the SVG code and future work.
For the most part the performance improvements from the conversion to Moz2D were gradual; as code was incrementally converted, little by little gfxContext overhead was avoided. On doing an audit of our open SVG performance bugs it seems that painting performance is no longer one of the reasons that we perform poorly, except for when we us cairo backed DrawTargets (Linux, Windows XP and other Windows versions with blacklisted drivers), and with the exception of one bug that needs further investigation. (See below for the issues that still do causes SVG performance problems.)
Besides the incremental improvements, there have been a couple of interesting perf bumps that are worth mentioning.
The biggest perf bump by far came when I converted the code that does the actual filling and stroking of SVG geometry to directly use a DrawTarget. The time taken to render this map then dropped from about 15s to 1-2s on my Mac. On the same machine Chrome Canary shows a blank window for about 5s, and then takes a further 20s to render. Now, to be honest, this improvement will be down to something pathological that has been removed rather than being down to avoiding Thebes overhead. (I haven't got to the bottom of exactly what that was yet.) The DrawTarget object being drawn to is ultimately the same object, and Thebes overhead isn't likely to be more than a few percent of any time spent in this code. Nevertheless, it's still a welcome win.
Another perf bump that came from the Moz2D conversion was that it enabled us to cache path objects. When using Thebes, paths are built up using gfxContext API calls and the consumer never gets to touch the resulting path. This prevents the consumer from keeping hold of the path and reusing it in future. This can be a disadvantage when the path is reused frequently, especially when D2D is being used where path creation is relatively expensive. Converting to Moz2D has allowed the SVG code to hold on to the path objects that it creates and reuse them. (For example, in addition to their obvious use during rasterization, paths might be reused for bounds calculations (think invalidation areas, objectBoundingBox content, getBBox() calls) and hit-testing.) Caching paths made us noticeably more responsive on this cool data visualization (temporarily mirrored here while the site is down) when mousing over the table rows, and gave us a +25% boost on this NYT article, for example.
For those of you that are interested in Talos, I did take a look at the SVG test data, but the unfortunately frequent up-and-down of unrelated regressions and wins makes it impossible to use that to show any overall impact of Moz2D conversion on the Talos tests. (Since the beginning of the year the times on Windows have improved slightly while on Mac they have regressed slightly.) The incremental nature of most of the work also unfortunately meant that the impact of individual patches couldn't usually be distinguished from the noise in Talos' results. One notable exception was the change to make SVG geometry use a path object directly which resulted in an improvement in the region of 6% for the svg_opacity suite on Windows 7 and 8.
Other than the performance benefits, several parts of the SVG implementation that were pretty messy and hard to get into and debug have become a lot more approachable. This has already allowed me to fix various SVG bugs that would otherwise have taken a lot longer to work out, and I hope it makes the code easier to approach for devs who aren't so familiar with it.
One final note on performance for any of you that will do your own testing to compare build - note that the enabling of e10s and tiled layers has caused significant changes in performance characteristics. You might want to turn those off.
Future SVG work
As I noted above there are still SVG performance issues unrelated to graphics speed. There are three sources of significant SVG performance issues that can make Mozilla perform poorly on SVG relative to other implementations. There is our lack of hardware acceleration of SVG filters; there's the issue of display list overhead dwarfing painting on SVGs that contain huge numbers of elements (display lists being an implementation detail, and one that gave us very large wins in many other cases); and there are a whole bunch of "strange" bugs that I expect are related to our layers infrastructure that are causing us to over invalidate (and thus do work painting when we shouldn't need to).
Currently these three issues are not on a schedule, but as other higher priority Mozilla work gets ticked of I expect we'll add them.
Future Moz2D work
The performance benefits from the Moz2D conversion on the SVG code do seem to have been positive enough that I expect that we will continue converting the rest of layout in the future. As usual, it will all depend on relative priorities though.
One thing that we should do is audit all the code that creates DrawTargets to check for backend type compatibility. Mixing hardware and software backed DrawTargets when we don't need to can cause us to unwittingly be taking big performance hits due to readback from and/or upload to the GPU. I fixed several instances of mismatch that I happened to notice during the conversion work, and in one case accidentally introduced one which fortunately was caught because it caused a 10-25% regression in a specific Talos test. We know that we still have outstanding bugs on this (such as bug 944571) and I'm sure there are a bunch of cases that we're unaware of.
I mentioned above that painting performance is still a significant issue on machines that fall back to using cairo backed DrawTargets. I believe that the Graphics team's plan to solve this is to finish the Skia backend for Moz2D and use that on the platforms that don't support D2D.
There are a few things that need to be added to Moz2D before we can completely get rid of gfxContext. The main thing we're missing is push-group API on DrawTarget. This is the main reason that gfxContexts actually wraps a stack of DrawTargets, which has all sorts of irritating fallout. Most annoying it makes it hazardous to set clip paths or transforms directly on DrawTargets that may be accessed via a wrapping gfxContext before the DrawTarget's clip stack and transform has been restored, and why I had to continue passing gfxContexts to a lot of code that now only paints directly via the DrawTarget.
The only Moz2D design decision that I've found myself to be somewhat unhappy with is the decision to make patterns relative to user-space. This is what most other hardware accelerated libraries do, but I don't think it's a good fit for 2D browser rendering. Typically crisp rendering is very important to web content, so we render patterns assuming a specific user-space to device-space transform and device space pixel alignment. To maintain crisp rendering we have to make sure that patterns are used with the device-space transform that they were created for, and having to do this manually can be irksome. Anyway, it's a small detail, but something I'll be discussing with the Graphics guys when I see them face-to-face in a couple of weeks.
Modulo the two issues above (and all the changes that I and others had made to it over the last year) I've found the Moz2D API to be a pleasure to work with and I feel the SVG code is better performing and a lot cleaner for converting to it. Well done Graphics team!
Jonathan Watt — Converting Mozilla's SVG implementation to Moz2D - part 1
One of my main work items this year was the conversion of the graphics portions of Mozilla's SVG implementation to directly use Moz2D APIs instead of using the old gfxContext/gfxASurface Thebes APIs. This pair of posts will provide some information on that work. This post will give some background and information on the conversion process, while part 2 will provide some discussion about the benefits of the work and what steps we might want to carry out next.
For background on why Mozilla is building Moz2D (formerly called Azure) and how it can improve Mozilla's performance see some of the earlier posts by Joe, Bas and Robert.
Early Moz2D development
When Moz2D was first being put together it was initially developed and tested as an alternative rendering backend for Mozilla's implementation of HTML <canvas>. Canvas was chosen as the initial testbed because its drawing is largely self contained, it requires a relatively small number of features from any rendering backend, and because we knew from profiling that it was being particularly impacted by Thebes/cairo overhead.
As Moz2D started to become more stable, Thebes' gfxContext class was extended to allow it to wrap a Moz2D DrawTarget (prior to that it was backed only by an instance of a Thebes gfxASurface subclass, in turn backed by a cairo_surface_t). This might seem a bit strange since, after all, Moz2D is supposed to replace Thebes, not be wrapped by it adding yet another layer of abstraction and overhead. However, it was an important step to allow the Graphics team to start testing Moz2D on Mozilla's more complicated, non-canvas, rendering scenarios. It allowed many classes of Moz2D bugs and missing Moz2D features to be worked on/out before beginning a larger effort to convert the masses of non-canvas rendering code to Moz2D.
In order to switch any of the large number of instances of gfxContext to be backed by a DrawTarget, any code that might encounter that gfxContext and try to get a gfxASurface from it had to be updated to handle DrawTargets too. For example, lots of forks in the code had to be added to BasicLayerManager, and gfxFont required a new GlyphBufferAzure class to be written. As this work progressed some instances of Thebes gfxContexts were permanently flipped to being backed by a Moz2D DrawTarget, helping keep working Moz2D code paths from regressing.
SVG, the next Guinea pig
Towards the end of 2013 it was felt that Moz2D was sufficiently ready to start thinking about converting Mozilla's layout code to use Moz2D directly and eliminate its use of gfxContext API. (The layout code being the code that decides where and how most things are placed on the screen, and by far the biggest consumer of the graphics code.) Before committing a lot of engineering time and resources to a large scale conversion, Jet wanted to convert a specific part of the layout code to ensure that Moz2D could meet its needs and determine what performance benefits it could provide to layout. The SVG code was chosen for this purpose since it was considered to be the most complicated to convert (if Moz2D could work for SVG, it could work for the rest of layout).
Stage 1 - Converting all gfxContexts to wrap a DrawTarget
After drawing up a rough list of the work to convert the SVG code to Moz2D I got stuck in. The initial plan was to add code paths to the SVG code to check for and extract DrawTargets from gfxContexts that were passed in (if the gfxContext was backed by one) and operate directly on the DrawTarget in that case. (At some future point the Thebes forks could then be removed.) It soon became apparent that these forks were often not how we would want the code to be structured on completion of Moz2D conversion though. To leverage Moz2D more effectively I frequently found myself wanting to refactor the code quite substantially, and in ways that were not compatible with the existing Thebes code paths. Rather than spending months writing suboptimal Moz2D code paths only to have to rewrite things again when we got rid of the Thebes paths I decided to save time in the long run and first make sure that any gfxContexts that were passed into SVG code would be wrapping a DrawTarget. That way maintaining Thebes forks would be unnecessary.
It wasn't trivial to determine which gfxContexts might end up being passed to SVG code. The complexity of the code paths and the virtually limitless permutations in which Web content can be combined meant that I only identified about a dozen gfxContexts that could not end up in SVG code. As a result I ended up working to convert all gfxContexts in the Mozilla code. (The small amount of additional work to convert the instances that couldn't end up in SVG code allowed us to reduce a whole bunch of code complexity (and remove a lot of then dead code) and simplified things for other devs working with Thebes/Moz2D.)
Ensuring that all the gfxContexts that might be passed to SVG code would be backed by a DrawTarget turned out to be quite a task. I started this work when relatively few gfxContexts had been converted to wrap a DrawTarget so unsurprisingly things were a bit rough. I tripped over several Moz2D bugs at this point. Mostly though the headaches were caused by the amount of code that assumed gfxContexts wrapped and could provide them with a gfxASurface/cairo_surface_t/platform library object, possibly getting or then passing those objects from/to seemingly far corners of the Mozilla code. Particularly challenging was converting the image code where the sources and destinations of gfxASurfaces turned out to be particularly far reaching requiring the code to be converted incrementally in 34 separate bugs. Doing this without temporary performance regressions was tricky.
Besides preparing the ground for the SVG conversion, this work resulted in a decent number of performance improvements in its own right.
Stage 2 - Converting the SVG code to Moz2D
Converting the SVG code to Moz2D was a lot more than a simple case of switching calls from one graphics API to another. The stateful context provided by a retained mode API like Thebes or cairo allows consumer code to set context state (for example, fill pattern, or anti-alias mode) in points of the code that can seem far removed from other code that takes an action (for example, filling a path) that relies on that state having been set. The SVG code made use of this a lot since in many cases (for example, when passing things through for callbacks) it simplified the code to only pass a context rather than a context and some state to set.
This wouldn't have been all that bad if it wasn't for another fundamental difference between Thebes/cairo and Moz2D -- in Moz2D paths and patterns are relative to user-space, whereas in Thebes/cairo they are relative to device-space. Whereas with Thebes we could set a path/pattern and then change the transform before drawing (perhaps, say, to apply a clip in a different space) and the position of the path/pattern would be unaffected, with Moz2D such a transform change would change (and thus break) the rendering. This, incidentally, was why the SVG code was expected to be the hardest area to switch to Moz2D. Partly for historic reasons, and partly because some of the features that SVG supports lead it to, the SVG code did a lot of setting state, changing transforms, setting some more state and then drawing. Often the complexity of the code made it difficult to figure out which code could be setting relevant state before a transform change, requiring more involved refactoring. On the plus side, sorting this out has made parts of the code significantly easier to understand, and has been something I've wanted to find the time to do for years.
Benefits and next steps
To continue reading about the performance benefits of the conversion of the SVG code and some possible next steps continue to part 2.
Giorgio Maone — Avast, you're kidd... killing me - said NoScript >:(
If NoScript keeps disappearing from your Firefox, Avast! Antivirus is likely the culprit.
It's gone Berserk and mass-deleting add-ons without a warning.
I'm currently receiving tons of reports by confused and angry users.
If the antivirus is dead (as I've been preaching for 7 years), looks like it's not dead enough, yet.
Christian Heilmann — What I am looking for in a guest writer on this blog
Simple: go try guest writing someplace else. This is my personal blog and if I am interested in something, I come to you and do it interview style in order to point to your work or showcase something amazingly cool that you have done.
Please, please, please with cherry on top, stop sending me emails like this one:
I'm {NAME}, a freelance writer/education consultant. I found "Christian Heilmann" on a Google search and thought I would contact you to see if you would like to work with me. I own a website on Job Application Service that I'm currently promoting for myself. I thought we could benefit each other somehow? If you are interested, I'd be happy to write a very high-quality article for your site and get a couple permanent links from it? While your website is benefiting from my high-quality article, I'm getting links from your site, making this proposition mutually beneficial.
Shall I write an article that matches your niche and send it across for your review or do you need me to write on a particular topic that interests you and your readers, I'm open to any topic, thoughts please?
If this does not interest you, I am sorry to have bothered you. Have a good day! If this does great I hope we can build a long-term business relationship together! If you wish to have a chat on the phone please let me know your phone number and when a good time to call is :) If you'd like, I can share samples with you.
I am very happy you know how to enter a name in Google and find the blog of that person. That's a good start. Nobody got hurt, you didn't overdo it with the research or spent too much effort before asking for my phone number and pointing out just how much you would get out of this "mutually beneficial relationship". Seriously, I would love to be a fly on the wall when you try dating.
I've worked hard on this blog, that's why it has some success or is at least found. Go work on yours yourself. That's how it should be. A blog is you. Just like this one is mine.
Gervase Markham — BMO show_bug Load Times 2x Faster Since January
The load time for viewing bugs on bugzilla.mozilla.org has got 2x faster since January. See this tweet for graphical evidence.
If you are looking for a direction in which to send your bouquets, glob is your man.
David Rajchenbach Teller — The Future of Promise
Doug Belshaw — Native apps, the open web, and web literacy
In a recent blog post, John Gruber argues that native apps are part of the web. This was in response to a WSJ article in which Christopher Mims stated his belief that the web is dying; apps are killing it. In this post, I want to explore the relationship between native apps and web literacy. This is important as we work towards a new version of Mozilla's Web Literacy Map. It's something that I explored preliminarily in a post earlier this year entitled What exactly is 'the mobile web'? (and what does it mean for web literacy?). This, again, was in response to Gruber.
This blog focuses on new literacies, so I'll not be diving too much into technical specifications, etc. I'm defining web literacy in the same way as we do with the Web Literacy Map v1.1: 'the skills and competencies required to read, write and participate on the web'. If the main question we're considering is are native apps part of the web? then the follow-up question is and what does this mean for web literacy?
Defining our terms
First of all, let's make sure we're clear about what we're talking about here. It's worth saying right away that 'app' is almost always used as a shorthand for 'mobile app'. These apps are usually divided into three categories:
- Native app
- Hybrid app
- Web app
From this list, it's probably easiest to describe a web app:
It's trickier to define a native app, but the essence can be seen most concretely through Apple's ecosystem that include iOS and the App Store. Developers use a specific programming language and work within constraints set by the owner of the ecosystem. By doing so, native apps get privileged access to all of the features of the mobile device.
A hybrid app is a native app that serves as a 'shell' or 'wrapper' for a web app. This is usually done for the sake of convenience and, in some cases, speed.
The boundary between a native app and a web app used to be much more clear and distinct. However, the lines are increasingly blurred. For example:
- APK files (i.e. native apps) can be downloaded from the web and installed on Android devices.
- Developments as part of Firefox OS mean that web technologies can securely access low-level functionality of mobile devices (e.g. camera, GPS, accelerometer).
- The specifications for HTML5 and CSS3 allow beautiful and engaging web apps to be used offline.
Web literacy and native apps
As a result of all this, it's probably easier these days to differentiate between a native app and a web app by talking about ecosystems and silos. Understanding it this way, a native app is one that is built specifically using the technologies and is subject to the constraints of a particular ecosystem. So a developer creating an app for Apple's App Store would have to go through a different process and use a different programming language than if they were creating one for Google's Play Store. And so on.
Does this mean that we need to talk of a separate 'literacy' for each ecosystem? Should we define 'Google literacy' as the skills and competencies required to read, write and participate in Google's ecosystem? I don't think so. While there may be variations in the way things are done within the different ecosystems, these procedural elements do not constitute 'literacy'.
What we're aiming for with the Web Literacy Map is a holistic overview of the skills and competencies people require when using the web. I think at this juncture we've got a couple of options. The first would be define 'the web' more loosely to really mean 'the internet'.
This is John Gruber's preferred option. He thinks we should focus less on web browsers (i.e. HTML) and more on the connections (i.e. HTTP). For example, in a 2010 talk he pointed out a difference between 'web apps' and 'browser apps'. His argument rested on a technical point, which he illustrated with an example. When a user scrolls through their timeline using the Twitter app for iPhone, they're not using a web browser, but they are using HTTP technologies. This, said Gruber, means that ecosystems such as Apple's and the web are not in opposition to one another.
While this is technically correct, it's a red herring. HTML does matter because the important thing here is the open web. Check out Gruber's sleight of hand in this closing paragraph:
Arguments about "open" and "closed" often devolve into unresolvable cross-talk where the two sides have different definitions of what open and closed really mean. But the weird thing about a truly open platform is that its openness allows closed things to be built on top of it. In broad strokes, that's why GNU/GPL software isn't "open" in the way that BSD software is (and why Richard Stallman outright rejects the term "open source"). If you expand your view of "the web" from merely that which renders inside the confines of a web browser to instead encompass all network traffic sent over HTTP/S, the explosive growth of native mobile apps is just another stage in the growth of the web. Far from killing it, native apps have made the open web even stronger.
I think Gruber needs to read up on enclosure and the Commons. To use a 16th-century English agricultural metaphor, the important thing isn't that the grass is growing in the field, it's that it's been fenced off and people are excluded.
A way forward
A second approach is to double-down on what makes the web different and unique. Mozilla's mission is to promote openness, innovation & opportunity on the web and the Web Literacy Map is a platform for doing this. Even if we don't tie development of the Web Literacy Map explicitly to the Mozilla manifesto it's still a key consideration. Therefore, when we're talking about 'web literacy' it's probably more accurate to define it as 'the skills and competencies required to read, write and participate on the open web.
What do we mean by the 'open web'? While Tantek Çelik approaches it from a technical standpoint, I like Brad Neuberg's (2008) focus on the open web as a series of philosophies:
Decentralization - Rather than controlled by one entity or centralized, the web is decentralized – anyone can create a web site or web service. Browsers can work with millions of entities, rather than tying into one location. It's not the Google or Microsoft Web, but rather simply the web, an open system that anyone can plug into and create information at the end-points.
Transparency - An Open Web should have transparency at all levels. This includes being able to view the source of web pages; having human-readable network identifiers, such as URLs; and having clear network entry points, such as HTTP and REST exposes.
Hackability - It should be easy to lash together and script the different portions of this web. MySpace, for example, allows users to embed components from all over the web; Google's AdSense, another example, allows ads to be integrated onto arbitrary web pages. What would you like to hack together, using the web as a base?
Openness - Whether the protocols used are de facto or de-jure, they should either be documented with open specifications or open code. Any entity should be able to implement these standards or use this code to hook into the system, without penalty of patents, copyright of standards, etc.
From Gift Economies to Free Markets - The Open Web should support extreme gift economies, such as open source and Wikis, all the way to traditional free market entities, such as Amazon.com and Google. I call this Freedom of Social Forms; the tent is big enough to support many forms of social and economic organization, including ones we haven't imagined yet. Third-Party Integration - At all layers of the system third-parties should be able to hook into the system, whether creating web browsers, web servers, web services, etc.
Third-Party Innovation - Parties should be able to innovate and create without asking the powers-that-be for permission.
Civil Society and Discourse - An open web promotes both many-to-many and one-to-many communication, allowing for millions of conversations by millions of people, across a range of conversation modalities.
Two-Way Communication - An Open Web should allow anyone to assume three different roles: Readers, Writers, and Code Hackers. Readers read content, Writers write content, and Code Hackers hack new network services that empower the first two roles.
End-User Usability and Integration - One of the original insights of the web was to bind all of this together with an easy to use web browser that was integrated for ease of use, despite the highly decentralized nature of the web. The Open Web should continue to empower the mainstream rather than the tech elite with easy to use next generation browsers that are highly usable and integrated despite having an open infrastructure. Open should not mean hard to use. Why can't we have the design brilliance of Steve Jobs coupled with the geek openness of Steve Wozniak? Making them an either/or is a false dichotomy.
The Web Literacy Map describes the skills and competencies required to read, write and participate on the open web. But it's also prescriptive. It's a way to develop an open attitude towards the world:
Open is a willingness to share, not only resources, but processes, ideas, thoughts, ways of thinking and operating. Open means working in spaces and places that are transparent and allow others to see what you are doing and how you are doing it, giving rise to opportunities for people who could help you to connect with you, jump in and offer that help. And where you can reciprocate and do the same.
Native apps can mitigate against the kind of reciprocity required for an open web. In many ways, it's the 21st century enclosure of the commons. I believe that web literacy, as defined and promoted through the Web Literacy Map, should not consider native apps part of the open web. Such apps may be built on top of web technologies, they may link to the open web, but native apps are something qualitatively different. Those who want to explore what reading, writing and participating means in closed ecosystems have other vocabularies – provided by media literacy, information literacy, and digital literacy – with which to do so.
Comments? Questions? Direct them here: firstname.lastname@example.com or discuss this post in the #TeachTheWeb discussion forum
Wladimir Palant — "Unloading" frame scripts in restartless extensions
The big news is: e10s is coming to desktop Firefox after all, and it was even enabled in the nightly builds already. And while most of the times the add-ons continue working without any changes, this doesn't always work correctly. Plus, using the compatibility shims faking a single-process environment might not be the most efficient approach. So reason enough for add-on authors to look into the dreaded and underdocumented message manager and start working with frame scripts again.
I tried porting a simple add-on to this API. The good news: the API hasn't changed since Firefox 17, so the changes will be backwards-compatible. And the bad news? Well, there are several.
- Bug 1051238 means that frame scripts are cached — so when a restartless add-on updates the old frame script code will still be used. You can work around that by randomizing the URL of your frame script (e.g. add
"?" + Math.random()to it).
- Bug 673569 means that all frame scripts run in the same shared scope prior to Firefox 29, so you should make sure there are no conflicting global variables. This can be worked around by wrapping your frame script in an anonymous function.
- Duplicating the same script for each tab (originally there was only a single instance of that code) makes me wonder about the memory usage here. Sadly, I don't see a way to figure that out. I assume that
about:memoryshows frame scripts under the
outOfProcessTabChildGlobalentry. But due to the shared scope there is no way to see individual frame scripts there.
- Finally, you cannot unload frame scripts if your restartless extension is uninstalled or disabled.
messageManager.removeDelayedFrameScript()will merely make sure that the frame script won't be injected into any new tabs. But what about tabs that are already open?
Interestingly, it seems that Mark Finkle was the only one to ask himself that question so far. The solution is: if you cannot unload the frame script, you should at least make sure it doesn't have any effect. So when the extension unloads it should send a
"email@example.com:disable" message to the frame scripts and the frame scripts should stop doing anything.
So far so good. But isn't there a race condition? Consider the following scenario:
- An update is triggered for a restartless extension.
- The old version is disabled and broadcasts "disable" message to the frame scripts.
- The new version is installed and starts its frame scripts.
- The "disable" message arrives and disabled all frame scripts (including the ones belonging to the new extension version).
The feedback I got from Dave Townsend says that this race condition doesn't actually happen and that
loadFrameScript and
broadcastAsyncMessage are guaranteed to affect frame scripts in the order called. It would be nice to see this documented somewhere, until then it is an implementation detail that cannot be relied on. The work-around I found here: since the frame script URL is randomized anyway (due to bug 1051238), I can send it along with the "disable" message:
messageManager.broadcastAsyncMessage("firstname.lastname@example.com:disable", frameScriptURL);
The frame script then processes the message only if the URL matches its own URL:
addMessageListener("firstname.lastname@example.com:disable", function(message)
if (message.data == Components.stack.filename)
Raniere Silva — MathML November Meeting
MathML November Meeting
Sorry for the delay to write this.
This is a report about the Mozilla MathML November IRC Meeting (see the announcement here). The topics of the meeting can be found in this PAD (local copy of the PAD) and the IRC log (local copy of the IRC log) is also available.
The next meeting will be in January 7th at 8pm UTC (check the time at your location here). Please add topics in the PAD.
Yes. Our December meeting was cancelled. =(
Christie Koehler — Happy 10th Birthday, MozillaWiki!
Last Monday, Firefox turned 10 years old. Thunderbird turns 10 on 7 December.
This week we celebrate another birthday: MozillaWiki turns 10 on Wednesday, 18 November!
I'm immensely proud of our wiki, its ten year history, and of all the work Mozillians do to make MozillaWiki a hub of collaboration and a living memory for the Mozilla Project.
To show our appreciation for your efforts over the last decade, the MozillaWiki team has created a 10th Birthday badge.
All you need to do to join in the celebration and claim the badge is log in to MozillaWiki. Once you've done that, you'll see a link to claim the badge at the top of the page. Don't have a MozillaWiki account? No worries! Create one during this Birthday celebration and you can claim the badge too.
A bit of MozillaWiki history
Before I talk about all the good work we've done, and what we have planned for the remainder of this year and beyond, let's take a quick stroll through the last 10 years. Thank you Internet Archive for hosting these snapshots of the wiki!
July 2004
The earliest snapshot I could find of the domain wiki.mozilla.org was from July 2004. It looks like we were hosting separate wiki installations, which may or may not have been Mediawiki.
November-December 2004
According to WikiApiary, the current installation of MozillaWiki was created on 18 November 2004. The closest snapshot to this date in the Internet Archive is 11 December 2004:
April 2005
By April 2005, the wiki had been upgraded, had a new theme (Cavendish), and had started using Apache rewrite rules to make the url pretty (e.g. no index.php).
August 2008
Three years later, in April 2008, we were still rockin' the Cavendish theme and the Main Page had some more content, including links to the weekly project call that continues to this day.
December 2010
We started tracking releases in December 2007 (see version). Here's what the Releases page looked like in December 2010.
May 2011
In May 2011, after 6 years of service, Cavendish was retired as the default skin and replaced with GMO.
July 2012
A year later, July 2012, MozillaWiki looked much the same.
July 2013
By July 2013, the Main Page was edited to include a few recent changes, but otherwise looked very similar.
August 2014
By August 2014, the revitalization of the MozillaWiki was in full swing and we were preparing for a major update to both the skin (GMO to Vector) as well as the underlying software (Mediawiki 1.19 to 1.23). We also had made significant changes to the content of the Main Page based on results of our recent user survey.
November 2014
Here's what the wiki looks like today, 17 November, the day before it's birthday. We're running a slightly modified Vector skin and Mediawiki 1.23.x branch.
MozillaWiki today
Pages, visitors and accounts
As of 16 November, MozillaWiki has 115,912 pages, all public, and nearly 10k uploaded files. About 630 people per month, on average, log in and make contributions to the wiki. These include both staff and volunteers. Want to track these stats yourself? Visit Special:Statistics.
The number of daily visitors ranges from 9k-30k, with an average likely around 13-14k. Who are these visitors? According our analytics software we get visitors from all over the world, with the greatest concentration being from the US, Canada and UK.
The wiki has over 330,000 registered user accounts. I estimate that about 300k of these are inactive spam accounts, so the real number for user accounts is probably closer to 30,000.
What kinda of content is hosted on MozillaWiki?
All kinds of project activity is coordinated and recorded on the wiki. This includes activity related to our products: Firefox, Firefox OS, WebMaker, etc. It also includes community activities such as Reps, Firefox Student Ambassadors, etc. Most project activities have some representation on MozillaWiki. People also use the wiki to track projects and goals on an individual level. In this regard, it served as a place for Mozillians' profiles long before we had mozillians.org.
The MozillaWiki isn't setup for localized content now, but this hasn't stopped our localized community from translating content. Every day a significant portion of account requests come from volunteers from regional communities and are often in a language other than English. In 2015, depending on resources available, we plan to significantly improve support for localized content on MozillaWiki.
2014 Accomplishments
This year we've made significant progress towards revitalizing MozillaWiki.
Accomplishments include:
- Forming a team of dedicated volunteers to lead a revitalization effort.
- Creating an About page for MozillaWiki that clarifies its scope and role in the project, including what is appropriate content and how to report issues.
- Fixing years-old bugs that cause significant usability problems (table sorting, unavailability of Wikieditor, etc.).
- Identifying a product owner for MozillaWiki and creating a Module for it, lead by a mix of staff and contributors.
- Halting the creation of new spam and cleaning up significant amounts of spam content.
- Upgrading Mediawiki from 1.19.x branch to 1.23.x branch AND changing the default theme without any significant downtime or disruptions to users.
- Organizing a user survey and using those results to guide much of our roadmap, including the redesign of the Main Page and sidebar navigation.
Thank you everyone who has been a part of this work!
There's still plenty to do, and many ways to contribute
We've made so much progress on the technical and infrastructure debt of MozillaWiki that we're now ready to focus on improving content and collaboration mechnisms.
How can I help?
The are many ways you can help, and we have contribution opportunities for all kinds of skill levels and time commitments.
We're working on documenting and organizing these contribution opportunities here: https://wiki.mozilla.org/MozillaWiki:Contribute so check that page often.
Join our mailing-list or community call
If you'd like to help us organize those opportunities, or have other ideas for improving the wiki, join one of our MozillaWiki Team communication channels or one of our community meetings. These meetings are held twice a month on Tuesday at 8:30 PST / 15:30 UTC. Our next meeting is 16 December. All who are interested in contributing to the wiki are welcome.
In the meantime, log in to MozillaWiki and celebrate its birthday with us by claiming the birthday badge!
Christian Heilmann — We have a massive recruitment problem
A few months ago, I flew over to see my parents for their 50th wedding anniversary. As some of you may know, I have a humble background. My dad was a coal miner and then factory worker and my mother has always been a home maker / housewife. I am the only one in my family that went to college and I skipped university as the thing to do was to make money in a job as soon as you are 18.
It was humbling and almost embarassing to have conversations with my family. Half of them are either unemployed or worried about their jobs. The rest are unhappy in their jobs but see no way to change that as they need the security. Finding joy in family life and leisure time is more important than enjoying the work. A job is a job, you got to do what you got to do and all that.
That's why it feels surreal to come back into "our world" and get offers for jobs I don't want. A lot of them. Some with ridiculous amounts of money offered and most with perks that would make my family blush and sense a trap.
Why are we not recruiter compatible and vice versa?
We're lucky to be that sought after and yet it seems there is no happy symbiosis between us and recruiters. On the contrary, as soon as you even mention recruiting most of us techies start ranting.
I feel uneasy doing that. I feel like an arrogant ass and I feel that we should be more grateful about the opportunities we get. The relationship of recruiter and job seeker should be high-fives and unicorns. Instead there is a massive sense of dread: "Oh god, another job offer, how tiring".
There are reasons for our dismay:
- A lot of headhunters/recruiters work on a commission basis and are measured by how many contacts they had that day. This leads to a scattergun approach and you get offers that are not "you" at all.
- Many recruiters seem to just look for keywords and then send the offer out to you. That's why you get Java positions when you have a JavaScript background. Just like you would send car mechanic jobs to a carpet expert.
- Others go for company names. A great example was this recruiter trying to hire someone's dog as a Java/Python developer.
- Many recruiting sites are very pushy to get you into their database to show potential hiring companies just how many job searchers they have. This leads to very old and outdated profiles and you get offers for jobs you've done years ago. Basically they don't want to find you a job, they want you as an ad.
- People write ridiculous job descriptions and send them to us. In the past I wrote what kind of people I try to hire and once it went through HR and recruitment review something completely ridiculous ended up online. You've seen those. Asking people for two degrees, but not older than 20. Seven years experience in a half year old technology, and similar confusing points.
- There is probably nothing more intrusive to someone who feels at home online to be called by somebody. Recruiters, however, seem to see the "personal touch" as the most important thing.
On the other side of this issue, we are not innocent either:
- Instead of telling people why we didn't want their offer, we just ignore them. There is no learning on either side.
- We love our own tools and are not too interested in changing that. Every recruitment department I worked with needed a CV in a document format for filing and keeping. Instead of having one of those at hand we love to create online CVs and portfolios or point people to our GitHub account as "real people who would hire me find all they need there". This is navel gazing and arrogant. If I want to go on a bus, I need a bus ticket. A macaroni picture with glitter on it saying "most amazing responsive bus ticket" will not get me on there. Have the tool for the task.
- We don't keep our presence up-to-date. If you're not seeking, say so on LinkedIn. Have a template to send back to recruiters telling them "thanks, but no."
- We also shouldn't create profiles of our dog on LinkedIn. This is a professional tool, if we don't use these in a professional manner we shouldn't be surprised that they go to the dogs.
- Keep your skills up-to-date. If you never ever want to work with a certain product any longer, remove it from your online presence. That way keyword searchers don't find you.
We need to communicate better
I feel there is a massive waste going on and an accumulation of frustration on both sides. We need to get better in helping another to make this the natural partnership it should be. I feel terrible hearing about friends not in our world who send out hundreds of applications and don't get answers whilst we complain about people trying to offer us jobs. It feels almost unreal.
There are a few good ideas around and there is a start to clean this mess up. Joblint is a tool that comes to mind. It is an analysing tool that takes job descriptions and allows you to
"Test tech job specs for issues with sexism, culture, expectations, and recruiter fails".
A lot of miscommunication could be avoided simply by using that.
Considering giving a helping hand
Maybe I should do something about this and use my time off to reach out and try to change something. I wonder if a workshop for recruiters about issues to avoid would be of interest? In any case, let's try to be more understanding. Recruiters do their jobs the same way we do ours. By understanding their drives and goals, we can make both of our lives easier. By being arrogant and come across as divas we shouldn't be surprised if job descriptions start calling out for rockstars, ninjas, gurus and mavens.
Let's highlight the great experiences we had, and share what worked. Maybe that could be the lever we need to crack this nut open.
Josh Aas — Let's Encrypt
Today we announced a project that I've been working on for a while now – Let's Encrypt. This is a new Certificate Authority (CA) that is intended to be free, fully automated, and transparent. We want to help make the dream of TLS everywhere a reality. See the official announcement blog post I wrote for more information.
Eric Rescorla and I decided to try to make this happen during the summer of 2012. We were trying to figure out how to increase SSL/TLS deployment, and felt that an innovative new CA would likely be the best way to do so. Mozilla agreed to help us out as our first major sponsor, and by May of 2013 we had incorporated Internet Security Research Group (ISRG). By September 2013 we had merged a similar project started by EFF and researchers from the University of Michigan into ISRG, and submitted our 501(c)(3) application. Since then we've put a lot of work into ISRG's governance, found the right sponsors, and put together the plans for our CA, Let's Encrypt.
I'll be serving as ISRG's Executive Director while we search for more permanent leadership. During this time I'll remain with Mozilla.
Too many people to thank for their help here, many of whom work for our sponsors, but I want to call out Eric Rescorla (Mozilla) and Kevin Dick (Right Side Capital Management) in particular. Eric was my original co-conspirator, and Kevin has spent innumerable hours with me helping to create partnerships and the necessary legal infrastructure for ISRG. Both are incredible at what they do, and I've learned a lot from working with them.
Now it's time to finish building the CA – lots of software to write, hardware to install, and auditing to complete. If you have relevant skills, we hope you'll join us.
Jean-Marc Valin — Daala: Perceptual Vector Quantization (PVQ)
Here's my new contribution to the Daala demo effort. Perceptual Vector Quantization has been one of the core ideas in Daala, so it was time for me to explain how it works. The details involve lots of maths, but hopefully this demo will make the general idea clear enough. I promise that the equations in the top banner are the only ones you will see!
Andreas Gal — Firefox and Cisco's Project Squared
Yesterday I was at Cisco's Collaboration Summit where Cisco's CTO for Collaboration Jonathan Rosenberg and I showed Cisco's new WebRTC-based Project Squared collaboration service running in Firefox, talking to a Cisco Collaboration Desktop endpoint without requiring transcoding.
This demo is the culmination of a year long collaboration between Cisco and Mozilla in the WebRTC space. WebRTC enables voice and video communication directly from within the browser. This means that anyone can build a video conferencing service just using WebRTC and HTML5 standards, without the need for the user to download a plugin or a native application.
Cisco is not only developing WebRTC-based services that run on the Web. They have also joined a growing number of organizations and companies helping Mozilla to build a better Web. Over the last year Cisco has contributed numerous technical improvements to Mozilla's WebRTC implementation, including support for screen sharing and the H.264 video codec. These features are now shipping in Firefox. We intend to use them in the future in Mozilla's own Hello communication service that we are bringing to Firefox.
Cisco's contributions to the Web go beyond just advancing Firefox. For the last three years the IETF, the standards body defining the networking protocols for WebRTC, has been unable to agree on a mandatory video codec for WebRTC, putting ubiquitous interoperability in doubt.
One of the major blockers to coming to a consensus was that H.264 is subject to royalty-bearing patents, which made it problematic for open source projects such as Firefox to deploy it. To break this logjam, Cisco open-sourced its H.264 code base and made it available in plugin form. Any product — not just Firefox — can download the plugin and use it to enable H.264 without paying any royalties.
This collaboration between Mozilla and Cisco enabled Firefox to add support for H.264 in WebRTC, and also played a significant role in the compromise reached at the last IETF meeting to adopt both H.264 and VP8 as mandatory video codecs for WebRTC in browsers. As a result of this compromise, in the future all browsers should match the capabilities already available in Firefox.
Mozilla will continue to work on advancing Firefox and the Web, and we are excited to have strong partners like Cisco who share our commitment to the open Web as a shared technology platform.
Filed under: Mozilla Tagged: Mozilla, WebRTC
Adam Lofting — Will our latest donation form help us raise more money this year?
Alexander Surkov — Accessibility goes into DOM
I have bunch of issues with the approach I wanted to outline here. Just to keep things in one place.
The purpose
I've been told that primary reason is a testing propose, but having role and name only is not enough to run UAIG tests or any accessibility automation tool since it would require other accessibility properties.
Also they say that it might be used for non accessibility proposes. I realize that semantics, the ARIA adds, can be used by non assistive technologies. In Firefox we have a large number of non AT consumers but we don't have a good idea in most of cases what they are for. So I don't really have the use case, and thus it's hard to say whether accessible role and name only works well for non a11y proposes.
Concerning to assistive technologies I think they also need a much larger API.
Blowing the DOM
Anything useful should require extra accessible properties as I said above. These are accessible description, states, relations, ability to navigate the hierarchy etc. That means sooner or later the Element interface has to be changed to a great extent. Check out AtkObject to get an idea about possible changes.
In beginning of times accessibility interfaces was built on top of DOM and later they were turned into full APIs. Now we are faced to backward process, accessibility APIs are getting back to DOM. I'm not sure if that's a good idea because accessibility tasks are something very specific, and accessibility API might be not suitable for common needs of web apps.
Not every semantically meaningful piece on the screen has a DOM node, for example, list bullets don't necessary have DOM elements associated with them. So Element based accessibility API is too restrictive to fit the requirements of the assistive technologies.
Last but not least is performance issue. In most of browsers the accessibility engine is kept separately and it gets running on demand. If accessibility is merged with the DOM then nothing tells the user this method may trigger heavy accessibility computations and make his app slower. Surely the browsers will learn how to get smarter but the approach will have a perf hit either way.
What's it going to be then, eh?
The idea is to provide a separate accessibility interface. If you like it then it can be done by parts, for example, introduce role and name only for the first round same as the original proposal says. Later you can think of adding all other properties.
This idea was welcomed initially, then later it was rejected as being too complex and accessibility centric. But - and that's most important thing - it doesn't have disadvantages the Element approach has.
Andreas Gal — Let's Encrypt: One more step on the road to TLS Everywhere
Principle 4 of the Mozilla Manifesto states: Individuals' security and privacy on the Internet are fundamental and must not be treated as optional.
Unfortunately treating user security as optional is exactly what happens when sites let users connect over insecure HTTP rather than HTTP over TLS (HTTPS). What insecure means here is that your network traffic is totally unprotected and can be read and/or modified by anyone who shares a network with you, including random people sharing Starbucks or airport WiFi.
One of the biggest reasons that web sites don't deploy TLS is the requirement to get a digital certificate — a cryptographic credential which allows a user's browser to know it's talking to the right site and not to an attacker. Certificates are issued by Certificate Authorities (CAs) often using a clumsy and error-prone manual process. A further disincentive to deployment is that most CAs charge a fee for their certificates, which not only prices some people out of the market but also interferes with automatic issuance and renewal.
Mozilla, along with our partners Akamai, Cisco, EFF, and Identrust decided to do something about this situation. Together, we've formed a new consortium, the Internet Security Research Group, which is starting Let's Encrypt, a new certificate authority designed to bring security to everyone. Let's Encrypt is built around a few key principles:
- Free: Certificates will be offered at no cost.
- Automatic: Certificates will be issued via a public and published API, allowing Web server software to automatically obtain new certificates at installation time and without manual intervention.
- Independent: No piece of infrastructure this important should be controlled by a single company. ISRG, the parent entity of Let's Encrypt, is governed by a board drawn from industry, academia, and nonprofits, ensuring that it will be operated in the public interest.
- Open: Let's Encrypt will be publishing its source code and protocols, as well as submitting the protocols for standardization so that server software as well as other CAs can take advantage of them.
Let's Encrypt will be issuing its first real certificates in Q2 2015. In the meantime, we have published some initial protocol drafts along with a demonstration client and server at: https://github.com/letsencrypt/node-acme and https://github.com/letsencrypt/heroku-acme. These are functional today and can be used to issue test certificates.
It's been a long road getting here and we're not done yet, but this is an important step towards a world with TLS Everywhere.
Filed under: Mozilla
Soledad Penades — Tools for the 21st century musician—super abridged dotJS edition
I attended dotJS yesterday where I gave a very short version of past past week's talk at Full Frontal (18 minutes versus 40).
The conference happened in a theatre and we were asked not to use bright background so I changed my slides to be darker and classier.
It didn't really go as smoothly as I expected (a kernel panic a bit before the start of the talk, and I got nervous and distracted so I got more nervous and…), but I guess I can't always WIN! It was fun to speak in French if only one line, though: Je suis très contente d'être parmi vous!–thanks to Thomas for the assistance in coming up with the perfect presentation line, and Guillaume and Sasha for listening to me repeat it until it resembled passable French!
While the video is edited and released, here's a sample in the form of slides, online and their source code in GitHub.
It was fun to use CSS filters to invert the images so they would not be a big white block on top of a dark background. Yay CSS filters!
.filter-invert {
filter: invert(100%) brightness(2);
Also, using them in transitions between slides. I discovered that I could blur between slides. Cinematic effects! (sorta, as I cannot get vertical/horizontal blur). But:
.bespoke-active.emphatic-text {
filter: none;
.bespoke-inactive.emphatic-text {
filter: blur(10px);
I use my custom plugin presentation-fullscreen for getting real fullscreen in my slides. It's on npm:
npm install presentation-fullscreen --save
then just
will add a new option to the contextual menu for making the whole body go fullscreen.
I shall write about this tip and how I use bespoke.js in general, and a couple thoughts and ideas I had during the conference soon. Topics including (so I don't forget): why a mandatory lack of anonymity is not the solution to doxxing, and the ideal talk length.
Mozilla Release Management Team — Firefox 34 release date moving to Dec 1/2
The Firefox 34 release date will move out one week from Nov 25 to Dec 1/2. This change impacts Firefox Desktop, Firefox for Android, Firefox ESR, and Thunderbird.
The purpose of this change is to allow for an additional week of stabilization during the 34 cycle.
Details of the change:
- Release date change from Nov 25 to Dec 1/2 (need to determine the date that works best given the work week)
- Merge date change from Tue, Nov 24 to Fri, Nov 28
- Two additional desktop betas (10 and 11) will be added to the calendar this week on our usual beta build schedule (build Mon and Thu, release Tue and Fri)
- One additional mobile beta (beta 11) will be added to the schedule.
Note that mobile beta 10 will gtb on schedule on Mon.
Mobile beta 11 will gtb on Thu with desktop in order to be ready early the following week.
- RC builds will happen on Mon, Nov 24
Patrick Finch — Is the Web dying?
This article may or may not be pay-walled, depending on how you arrive at it. It is an exploration of the shift to apps.
The history of computing is companies trying to use their market power to shut out rivals, even when it's bad for innovation and the consumer….That doesn't mean the Web will disappear. Facebook and Google still rely on it to furnish a stream of content that can be accessed from within their apps. But even the Web of documents and news items could go away. Facebook has announced plans to host publishers' work within Facebook itself, leaving the Web nothing but a curiosity, a relic haunted by hobbyists.
This is something I was getting at with my post yesterday: that advertising remains one of the Web's unique selling points. It is much more effective as an advertising platform than mobile apps are. At the moment, the Internet giants extract an enormous amount of value from the content on the Web, using it to drive engagement with their services. The Web has very low barriers to entry, but economic sustainability is difficult and the only proven revenue model appears to be advertising at scale. The model needs liberating.
(Note: The source of this article, the Wall Street Journal, may appear to refute that, (given it has a paywall), but I believe that their model is essentially freemium and it isn't clear to me what revenues they derive from subscription customers.)
Wil Clouser — Altering large tables without bringing down your service
When we run ALTER statements on our big tables we have to plan ahead to keep from breaking whatever service is using the database. In MySQL, many times* a simple change to a column (say, from being a short varchar to being a text field) can read-lock the entire table for however long it takes to make the change. If you have a service using the table when you begin the query you'll start eating into your downtime budget.
If you have a large enough site to have database slaves you'll have a double-whammy - all reads will block on the master altering the table, and then, by default, the change will be replicated out to your slaves and not only will they read-lock the table while they alter it, but they will pause any further replication until the change is done, potentially adding many more hours of outdated data being returned to your service as the replication catches up.
The good news is, in some situations, we can take advantage of having database slaves to keep the site at 100% uptime while we make time consuming changes to the table structure. The notes below assume a single master with multiple independent slaves (meaning, the slaves aren't replicating to each other).
Firstly, it should go without saying, but the client application needs to gracefully handle both the existing structure and the anticipated structure.
When you're ready to begin, pull a slave out of rotation and run your alter statement on it. When it completes, put the slave back into the cluster and let it catch up on replication. Repeat those steps for each slave. Then failover one of the slaves as a new master and pull the old master out of rotation and run the alter statement on it. Once it has finished put it back in the cluster as a slave. When the replication catches up you can promote it back to the master and switch the temporary master back to a slave.
At this point you should have the modified table structure everywhere and be back to your original cluster configuration.
Special thanks to Sheeri who explained how to do all the above and saved us from temporarily incapacitating our service.
*What changes will lock a table vary depending on the version of MySQL. Look for "Allows concurrent DML?" in the table on this manual page.
Byron Jones — happy bmo push day!
the following changes have been pushed to bugzilla.mozilla.org:
- [1096565] GET REST calls should allow arbitrary URL parameters to be passed in addition the values in the path
- [1097813] Bug.search causes error when using simple token auth and specifying 'token' instead of 'Bugzilla_token'
- [1036802] Requests to the native rest/bzapi endpoints with gzip encoding always result in HTTP/200 responses
- [1097382] OS sniffing should detect Windows 10 from "Windows NT 6.4″ instead of detecting Windows NT
- [1098956] remove autoland support
- [1100368] css concatenation breaks data: urls
discuss these changes on mozilla.tools.bmo.
Filed under: bmo, mozilla
Alex Vincent — Another 100K bug reports… and nobody noticed
Bugzilla bug report #1,100,000
We used to have a little cheering for every 100,000 bug reports filing. Bugzilla hasn't keeled over yet!
But somehow, in the aftermath of the megabug, I think we forgot to plan for this one. Oops!
It looks like it took us about 19 months to get here from the previous one. I'll leave it to Gervase Markham to dig up the appropriate statistics.
Jennifer Boriss — Redesign of reddit's Login/Account Creation Window and reddit.com/login
Tim Taubert — The sad state of server-side TLS Session Resumption implementations
The probably oldest complaint about TLS is that its handshake is slow and together with the transport encryption has a lot of CPU overhead. This certainly is not true anymore if configured correctly (even if some companies choose to ignore that).
One of the most important features to improve user experience for visitors accessing your site via TLS is session resumption. Session resumption is the general idea of avoiding a full TLS handshake by storing the secret information of previous sessions and reusing those when connecting to a host the next time. This drastically reduces latency and CPU usage.
Enabling session resumption in web servers and proxies can however easily compromise forward secrecy. To find out why having a de-factor standard TLS library (i.e. OpenSSL) can be a bad thing and how to avoid botching PFS let us take a closer look at forward secrecy, and the current state of server-side implementation of session resumption features.
What is (Perfect) Forward Secrecy?
(Perfect) Forward Secrecy is an important part of modern TLS setups. The core of it is to use ephemeral (short-lived) keys for key exchange so that an attacker gaining access to a server cannot use any of the keys found there to decrypt past TLS sessions they may have recorded previously.
We must not use a server's RSA key pair, whose public key is contained in the certificate, for key exchanges if we want PFS. This key pair is long-lived and will most likely outlive certificate expiration dates as you would just use the same key pair to generate a new certificate after the current expired. In case the server is compromised it would be far too easy to determine the location of the private key on disk or in memory and use it to decrypt recorded TLS sessions from the past.
Using Diffie-Hellman key exchanges where key generation is a lot cheaper we can use a key pair exactly once and discard it afterwards. An attacker with access to the server can still compromise the authentication part as shown above and {M,W}ITM everything from here on using the certificate's private key, but past TLS sessions stay protected.
How can Session Resumption botch PFS?
TLS provides two session resumption features: Session IDs and Session Tickets. To better understand how those can be attacked it is worth looking at them in more detail.
Session IDs
In a full handshake the server sends a Session ID as part of the "hello" message. On a subsequent connection the client can use this session ID and pass it to the server when connecting. Because both server and client have saved the last session's "secret state" under the session ID they can simply resume the TLS session where they left off.
To support session resumption via session IDs the server must maintain a cache that maps past session IDs to those sessions' secret states. The cache itself is the main weak spot, stealing the cache contents allows to decrypt all sessions whose session IDs are contained in it.
The forward secrecy of a connection is thus bounded by how long the session information is retained on the server. Ideally, your server would use a medium-sized cache that is purged daily. Purging your cache might however not help if the cache itself lives on a persistent storage as it might be feasible to restore deleted data from it. An in-memory storage should be more resistant to these kind of attacks if it turns over about once a day and ensures old data is overridden properly.
Session Tickets
The second mechanism to resume a TLS session are Session Tickets. This extension transmits the server's secret state to the client, encrypted with a key only known to the server. That ticket key is protecting the TLS connection now and in the future and is the weak spot an attacker will target.
The client will store its secret information for a TLS session along with the ticket received from the server. By transmitting that ticket back to the server at the beginning of the next TLS connection both parties can resume their previous session, given that the server can still access the secret key that was used to encrypt.
We ideally want the same secrecy bounds for Session Tickets as for Session IDs. To achieve this we need to ensure that the key used to encrypt tickets is rotated about daily. It should just as the session cache not live on a persistent storage to not leave any trace.
Apache configuration
Now that we determined how we ideally want session resumption features to be configured we should take a look at a popular web servers and load balancers to see whether that is supported, starting with Apache.
Configuring the Session Cache
The Apache HTTP Server offers the
SSLSessionCache directive
to configure the cache that contains the session IDs of previous TLS sessions
along with their secret state. You should use
shmcb as the storage type, that is
a high-performance cyclic buffer inside a shared memory segment in RAM. It will
be shared between all threads or processes and allow session resumption no
matter which of those handles the visitor's request.
The example shown above establishes an in-memory cache via the path
/path/to/ssl_gcache_data with a size of 512 KiB. Depending on
the amount of daily visitors the cache size might be too small (i.e. have a
high turnover rate) or too big (i.e. have a low turnover rate).
We ideally want a cache that turns over daily and there is no really good way to determine the right session cache size. What we really need is a way to tell Apache the maximum time an entry is allowed to stay in the cache before it gets overridden. This must happen regardless of whether the cyclic buffer has actually cycled around yet and must be a periodic background job to ensure the cache is purged even when there have not been any requests in a while.
You might wonder whether the
SSLSessionCacheTimeoutdirective can be of any help here - unfortunately no. The timeout is only checked when a session ID is given at the start of a TLS connection. It does not cause entries to be purged from the session cache.
Configuring Session Tickets
While Apache offers the SSLSessionTicketKeyFile directive to specify a key file that should contain 48 random bytes, it is recommended to not specify one at all. Apache will simply generate a random key on startup and use that to encrypt session tickets for as long as it is running.
The good thing about this is that the session ticket key will not touch persistent storage, the bad thing is that it will never be rotated. Generated once on startup it is only discarded when Apache restarts. For most of the servers out there that means they use the same key for months, if not years.
To provide forward secrecy we need to rotate the session ticket key about daily and current Apache versions provide no way of doing that. The only way to achieve that might be use a cron job to gracefully restart Apache daily to ensure a new key is generated. That does not sound like a real solution though and nothing ensures the old key is properly overridden.
Changing the key file while Apache is running does not do it either, you would
still need to gracefully restart the service to apply the new key. An do not
forget that if you use a key file it should be stored on a temporary file
system like
Disabling Session Tickets
Although disabling session tickets will undoubtedly have a negative performance impact, for the moment being you will need to do that in order to provide forward secrecy:
Ivan Ristic adds that to disable session tickets for Apache using
SSLOpenSSLConfCmd, you have to be running OpenSSL 1.0.2 which has not been released yet. If you want to disable session tickets with earlier OpenSSL versions, Ivan has a few patches for the Apache 2.2.x and Apache 2.4.x branches.
To securely support session resumption via tickets Apache should provide a configuration directive to specify the maximum lifetime for session ticket keys, at least if auto-generated on startup. That would allow us to simply generate a new random key and override the old one daily.
Nginx configuration
Another very popular web server is Nginx. Let us see how that compares to Apache when it comes to setting up session resumption.
Configuring the Session Cache
Nginx offers the ssl_session_cache directive
to configure the TLS session cache. The type of the cache should be
shared to
share it between multiple workers:
The above line establishes an in-memory cache with a size of 10 MB. We again have no real idea whether 10 MB is the right size for the cache to turn over daily. Just as Apache, Nginx should provide a configuration directive to allow cache entries to be purged automatically after a certain time. Any entries not purged properly could simply be read from memory by an attacker with full access to the server.
You guessed right, the
ssl_session_timeoutdirective again only applies when trying to resume a session at the beginning of a connection. Stale entries will not be removed automatically after they time out.
Configuring Session Tickets
Nginx allows to specify a session ticket file using the ssl_session_ticket_key directive, and again you are probably better off by not specifying one and having the service generate a random key on startup. The session ticket key will never be rotated and might be used to encrypt session tickets for months, if not years.
Nginx, too, provides no way to automatically rotate keys. Reloading its configuration daily using a cron job might work but does not come close to a real solution either.
Disabling Session Tickets
The best you can do to provide forward secrecy to visitors is thus again switch off session ticket support until a proper solution is available.
HAproxy configuration
HAproxy, a popular load balancer, suffers from basically the same problems as Apache and Nginx. All of them rely on OpenSSL's TLS implementation.
Configuring the Session Cache
The size of the session cache can be set using the tune.ssl.cachesize directive that accepts a number of "blocks". The HAproxy documentation tries to be helpful and explain how many blocks would be needed per stored session but we again cannot ensure an at least daily turnover. We would need a directive to automatically purge entries just as for Apache and Nginx.
And yes, the
tune.ssl.lifetimedirective does not affect how long entries are persisted in the cache.
Configuring Session Tickets
HAproxy does not allow configuring session ticket parameters. It implicitly supports this feature because OpenSSL enables it by default. HAproxy will thus always generate a session ticket key on startup and use it to encrypt tickets for the whole lifetime of the process.
A graceful daily restart of HAproxy might be the only way to trigger key rotation. This is a pure assumption though, please do your own testing before using that in production.
Disabling Session Tickets
You can disable session ticket support in HAproxy using the no-tls-tickets directive:
A previous version of the post said it would be impossible to deactivate session tickets. Thanks to the HAproxy team for correcting me!
Session Resumption with multiple servers
If you have multiple web servers that act as front-ends for a fleet of back-end servers you will unfortunately not get away with not specifying a session ticket key file and a dirty hack that reloads the service configuration at midnight.
Sharing a session cache between multiple machines using memcached is possible but using session tickets you "only" have to share one or more session ticket keys, not the whole cache. Clients would take care of storing and discarding tickets for you.
Twitter wrote a great post about how they manage multiple web front-ends and distribute session ticket keys securely to each of their machines. I suggest reading that if you are planning to have a similar setup and support session tickets to improve response times.
Keep in mind though that Twitter had to write their own web server to handle forward secrecy in combination with session tickets properly and this might not be something you want to do yourselves.
It would be great if either OpenSSL or all of the popular web servers and load balancers would start working towards helping to provide forward secrecy by default and server admins could get rid of custom front-ends or dirty hacks to rotate keys.
Mozilla Release Management Team — Firefox 34 beta8 to beta9
- 31 changesets
- 56 files changed
- 920 insertions
- 250 deletions
List of changesets:
|Matthew Gregan
|Bug 1085175. r=roc, a=dveditz - 9cd882996cbe
|Jordan Santell
|Bug 1078539 - Add a doorhanger widget for the developer edition notification to browser console, developer tools, webide and responsive design mode. r=jryans, a=lmandel - e7f8aa528841
|James Willcox
|Bug 1097126 - Restrict MediaCodec backend to Android 5.0 and higher r=blassey a=lmandel - 7dfbe52d1a2b
|James Willcox
|Backed out changeset 7dfbe52d1a2b a=lmandel - b0fea8a116aa
|James Willcox
|Bug 1097276 - Disable MediaCodec backend r=edwin a=lmandel - cd70fa61662a
|Chenxia Liu
|Bug 1093619 - Don't display onboarding screen for Webapp profiles. r=margaret, a=lmandel - 5cbc59a67d8c
|James Willcox
|Bug 1097276 - Disable fragmented MP4 support on Android r=rillian a=lmandel - f7dd649eb2f6
|Richard Newman
|Bug 1095298 - Ignore 'C' locale when initializing nsLocaleService on Android. r=emk, a=lmandel - 43fd2720be09
|Olli Pettay
|Bug 1096263 - XMLHttpRequest.send({}) should not throw. r=bz, a=lmandel - 9e57cec588a9
|Olli Pettay
|Bug 1096263 - XMLHttpRequest.send({}) should not throw, tests. r=bz, a=lmandel - 0197e9eb324f
|Botond Ballo
|Bug 1068961 - Reset clip rect for color layers. r=roc, a=lmandel - 9f14f2af8bf7
|Robert O'Callahan
|Bug 1084672 - Call NotifyDidPaint from the refresh driver to ensure it gets called regardless of whether OMTC is used or not. r=mattwoodrow, a=lmandel - 2b08e1cb3c6f
|Matthew Gregan
|Bug 1096716 - Delay buffer frame calculation in WMF audio decoder until after UpdateOutputType to avoid using obsolete rate/channel values. r=cpearce, a=lmandel - 49f10dbc7d69
|Boris Zbarsky
|Bug 1090616 - Don't assume that the nodes we just pasted are still in the DOM, because mutation listeners suck like that. r=smaug, a=lmandel - 609915862295
|Steven Michaud
|Bug 1092855 - Work around bad interaction between jemalloc and Apple uninitialized memory bug. r=spohl, a=lsblakk - 4bdf71e69d10
|Matthew Gregan
|Bug 1092859 - Always use soft volume in WinMM cubeb backend. r=padenot, a=lmandel - f2dd9f2a084a
|Steven Michaud
|Bug 1086977 - Facebook's old "Facebook Photo Uploader" (fbplugin) crashes on load on OS X Yosemite. r=bsmedberg a=lmandel - e588ff4e326e
|Neil Rashbrook
|Bug 1070768 - Fix up more references to XPFE's autocomplete.css. r=Ratty, a=lmandel - aa474c125c53
|Gijs Kruitbosch
|Bug 1096787 - Copy new logins.json database when using fx reset. r=MattN, a=lsblakk - b718e8c0d423
|Randell Jesup
|Bug 1080312 - Update iteration code from upstream. r=jesup, a=abillings - 4bb1c6116c39
|Gijs Kruitbosch
|Bug 1096695 - hardcode strings for beta to give more info about sslv3 being dead, r=bz,gavin, a=lmandel - d585e4e50468
|Gijs Kruitbosch
|Bug 1096695 - fix test failures in test_aboutCrashed.xul, rs=bustage,a=bustage - 117eb4e49c72
|Bas Schouten
|Bug 1096913 - Remove the simple cache entry when replacing the cache entry. r=mwu, a=lmandel - e1ee2331bd12
|Kannan Vijayan
|Bug 1081850 - Check for resolve hooks when optimizing no-such-prop operations in baseline. r=jandem, a=lmandel - e685be9bd4d6
|Mike Hommey
|Bug 1096651 - Avoid using random bits when determining SSE3/SSE4 availability for the JIT. r=luke, a=lmandel - 3e5cb63660bd
|L. David Baron
|Bug 1086937 - Patch 0: Add missing null check of root element so this patch series doesn't expose a crash in layout/style/crashtests/472237-1.html. r=birtles, a=lmandel - 4ccd3e117f5d
|Robert O'Callahan
|Bug 1092842 - When setting cliprects on background color display items, don't shrink them to exclude opaque borders (unless there's nonzero border-radius). r=mattwoodrow, a=lmandel - 19296c34b1ca
|Robert O'Callahan
|Bug 1097437 - Work around Quartz bug where corners of stroked rects don't get a solid color when they should. r=jrmuizel, a=lmandel - 9e4c3c78fe01
|Steven Michaud
|Bug 1017595 - Links from external applications sometimes fail to open when Firefox is hidden. r=spohl, a=lmandel - 89c3e0133233
|Robert O'Callahan
|Reftest manifest changes that were incorrectly landed as part of Bug 1096181, but should have been part of Bug 1097437 or Bug 1092842. a=orange - c486cd17bebb
|Jordan Santell
|Bug 1078539 - Disable dev edition promo banner in beta temporarily. r=jryans, a=lmandel - 1242fc159d04
Yunier José Sosa Vázquez — Jan de Mooij: "Seguiremos haciendo un Firefox más rápido"
Hace poco días tuvimos el placer de compartir con ustedes una importante noticia para el futuro de Firefox. Me refiero a las mejoras en velocidad al ejecutar Java Script del motor de nuestro navegador favorito. Para ese artículo estaba esperando la respuesta de uno de sus desarrolladores, y dada a la importancia de la buena nueva se publicó.
Pregunta (P). La última pregunta de nuestra conversación anterior estuvo relacionada con la puntuación en Octane y hoy han alcanzado su objetivo, ya somos los primeros. ¿Qué aspectos de Ion Monkey han mejorado para alcanzar esta increíble velocidad al ejecutar Java Script (JS)?
Respuesta (R)./ Hemos realizado muchas mejoras a Ion Monkey pero también a otras partes del motor JS. Habilitar la Recolección Generacional de Basura nos ha ayudado mucho con respecto a Octane. En este artículo publicado por Nicholas Nethercote se explica con más detalle esta característica. También se han aplicado optimizaciones menores que ayudan a obtener mejores puntuaciones.
P. ¿Cómo impactan estos cambios en los usuarios finales de Firefox ?
R./ Estos cambios harán que muchas cosas vayan muy rápido, principalmente cuando se ejecuten aplicaciones escritas en JS que requieran alto procesamiento de CPU. Ejemplo de esto son los juegos, emuladores y pdf.js, donde el rendimiento JS es sumamente importante.
P. ¿Podrían añadir pruebas en AWFY sobre Linux? Porque las plataformas actuales solo incluyen Windows, Mac, Android y Firefox OS.
R./ Sí podríamos añadir Linux!, pienso que las puntuaciones en Linux deben parecerse mucho a las alcanzadas en Mac. En un futuro añadiremos esta plataforma.
P. ¿Qué significan Firefox (Shell) y Firefox (PGO) en AWFY?
R./ Firefox (Shell) es el shell de JS, una pequeña aplicación de línea de comandos independiente que los desarrolladores de SpiderMonkey usan. Es muy útil pues no tenemos que compilar el navegador completo cuando trabajamos en el motor JS. Puedes encontrar más información al respecto en la página del proyecto.
Firefox (PGO) es una versión compilada para Windows basada en perfil de optimización guiado. Todas nuestras versiones oficiales usan PGO. Firefox compilado con PGO es más rápido que sin PGO en muchos casos.
P. Es maravilloso que Firefox ahora vaya de primero en la velocidad como hace varios años. Esto representa un regalo por sus 10 años, dime cómo se sienten en Mozilla con este logro.
R./ Nosotros estamos muy emocionados con todas nuestras mejoras en rendimiento porque nos tomó mucho esfuerzo conseguirlo. Pero no pararemos aquí, seguiremos encontrando nuevas vías para hacer Firefox más rápido.
Jan de Mooij: Muchas gracias por escribir acerca de lo que hacemos en Mozilla.
Por ahora sólo nos queda seguir la evolución del motor como el año anterior y esperar que avance aún más. Siempre con la confianza puesta en Mozilla y sus desarrolladores que nunca nos han defraudado.
Patrick Finch — Priceless
Many people don't know/didn't realise/don't care/already know/already know and didn't care, but I am back full time with Mozilla after a hiatus of a number of months. In fact, I jumped at the chance to join Mozilla's Content Services team and after a fairly short conversation with Darren, Mozilla VP of Content Services, I knew it was what I wanted to do.
I feel very happy to be back, and I wanted to put a few thoughts on paper about my motivation. I think back to a conversation I had earlier this year in Barcelona with Alina, someone who always expands my thinking. I believe in the web. I believe in Mozilla's efforts to maintain its importance in how the Internet is developed and consumed. I believe it is entirely preferable to the reemergence of technological islands, walled gardens, empires, (call them what you will). And yet, from a different perspective, isn't the web evidently facilitating such empires anyway?
Taking it further: the web, this incredible creation that has enriched our lives in ways we could not have imagined for ourselves previously, is also an agent of economic and cultural imperialism, in the same way that free trade and liberalised capital markets arguably have been in the 1980s and 1990s? I realise that many Mozillians will have an inherent faith in market solutions and I certainly believe in free trade…up to a point.
People will identify Content Services with advertising. And how do I feel about advertising in general? About the same way as when I first read J.K. Galbraith on the subject, over 20 years ago. Advertising troubled Galbraith, even in 1958 when he first published The Affluent Society. Advertising gives cultural force to the means of production. Or as he put it, "wants are increasingly created by the process by which they are satisfied". That is, advertising is the means by which supply creates demand. It allows capital to influence our psyche and creates new cultural barriers to market entry. In 2014, it's hard to imagine an economy without advertising. And while I do not entirely share Naomi Klein's wholly negative views on what brands mean, I do find that idea that so many of our cultural signifiers are created with the purpose of persuading us to consume x in preference to y to be more than a little uncomfortable.
Given that perspective, why Content Services? Well, Content Services is that it is not all about advertising. Content Services will help deliver Mozilla a new voice for its community and with its users. But saying that, the most important thing for Content Services is advertising. That is because advertising is the most important economic activity on the web – by a long way.
Look at what advertising has enabled on the web. How much content is free to consume as a consequence of federated advertising? Many Firefox user choose to block adverts, and other people round on those users for failing to honour the implicit contract between the publisher and the reader. I am not sure I subscribe to that point of view entirely, but to fail to view advertising as an exchange of value between the user and the website is to be disingenuous. And it is something that the web does extraordinarily well – and at scale, and on an aggregated basis. It has empowered the user, and the search engine, the social network, and it has wreaked havoc on publishers.
Almost every user of the web enjoys a huge consumer surplus. That is, they would pay far more for the web than it costs them (once you've paid for access to the network, you "pay" almost nothing to use the web). And some consumers enjoy a much larger surplus than others. Typically, richer consumers, who have a higher propensity to pay, transmit an effective subsidy to poorer consumers who would have a lower propensity to pay. And this aggregated arbitrage is a good thing. Generally.
Except that it has given rise to incalculably powerful and valuable empires. These empires might be defined as the ones who own the user's identity. The ones you log in to, who know the most about you, about what you do, who you know and what you read. Empires which extract value from all of the industries they touch, from online publishers to mobile network operators and ISPs. I must stress none of this is a criticism of a Google or a Facebook: they have delivered huge utility and have successfully exploited their market positions. But it is notable that a company with no significant patents or copyrights, nor indeed revenues, and which employed a reported 55 people could be valued at $19Bn. It is reasonable to suppose, under such circumstances, that there are major externalities generated by this business, or that this business is a significant free rider, which almost all internet business are: something those of use who support net neutrality implicitly agree is a good thing (I do not intend the term pejoratively).
What are the externalities? As we're fond of telling each other, if you're not paying, you're the product. The price we pay is our attention and exposure to adverts, and knowledge about ourselves. We are being broken down, analysed, reassembled as segments, profiles, tracked as individuals and as sub-markets and, yes, being spied upon. Some people are relaxed about this, perhaps feeling that they have nothing to hide, and besides, they haven't even come for the Socialists yet…
What's more, the cultural impact is abysmal. In the old world, advertising inventory was finite, confined to the space for adverts on billboards, in newspapers and so on. When the Mozilla community created its iconic New York Times advert, it was an incredible demonstration of the power of a community – placing such an advert cost real money. But in the online world, inventory is flexible, and theoretically infinite. You can grow your inventory by retaining users, get getting more clicks. And you do this by writing clickbait headlines, by instrumenting your users, by taking your publication downmarket, by designing your site so that one article extends over multiple webpages, and so on and so forth. The effects on our culture are obvious. The effects on our psyche, we're only just starting to understand.
And then, there is a battle over online privacy. Or perhaps, more aptly, an arms race. Privacy measures are met with counter-measures. Tracking cookies, which still seem important today, may seem positively anodyne in years to come. The more intimacy we gain with the internet, and the more capabilities it assumes, the deeper and deeper this problem becomes.
So, where does all this leave us? Well, there is another Mozillian I would mention who has frequently inspired me: former CEO John Lilly. Almost exactly four years ago, John gave a talk for the House of Commons, in which he presciently suggested that just as troubling as the Orwellian aspects of the internet are, so to should be aware of the dangers of a culture that is amusing us into bovine submission. John is a man who reads books, and as he points out, the internet is as much Brave New World as 1984. And John also spelled out the importance for mass participation in the creation of counter-measures to this. Actually, just go and read his post again, if you haven't already.
Advertising on the web is a problem, it risks trivialising our culture, creating a mass surveillance system and is supporting new forms of digital empires. And yet, it is better than the alternatives: all this economic value being pushed to proprietary technology platforms. And it is in danger: it is in danger of being unpalatable in a modern democracy, and of being superseded by proprietary technologies with even worse consequences. That is why Mozilla has to act, and why it is entirely appropriate that we involve ourselves in this industry. It is why we should conceive and build solutions to these problems, and look to empower all parts of the internet ecosystem that generate value for the consumer. This problem is our problem. We must not just try to wish it out of existence.
Our first duty is clear: it is to the Firefox user, and the trust they have in Mozilla. It would not be right that we would send our users to whichever service on the internet and rule out-of-scope the consequences for them (and nor do we). We build Firefox users the tools to be in charge of their experience. But we must help instantiate the rest of the world we want to see, bringing advertisers and publishers who share these values into the Mozilla community. We will understand their needs, and where they are transparent, where they scale and support heterogeneity, where they offer a reasonable, knowable and workable exchange of value, we should finds ways to facilitate them. Until that happens, the concentration of power on the internet will only continue. And honestly, who else but Mozilla is going to address this problem?
And more important to me than any of this, is to be working side-by-side again with my many wonderful friends at Mozilla.
Daniel Stenberg — Why curl defaults to stdout
(Recap: I founded the curl project, I am still the lead developer and maintainer)
When asking curl to get a URL it'll send the output to stdout by default. You can of course easily change this behavior with options or just using your shell's redirect feature, but without any option it'll spew it out to stdout. If you're invoking the command line on a shell prompt you'll immediately get to see the response as soon as it arrives.
I decided curl should work like this, and it was a natural decision I made already when I worked on the predecessors during 1997 or so that later would turn into curl.
On Unix systems there's a common mantra that "everything is a file" but also in fact that "everything is a pipe". You accomplish things on Unix by piping the output of one program into the input of another program. Of course I wanted curl to work as good as the other components and I wanted it to blend in with the rest. I wanted curl to feel like cat but for a network resource. And cat is certainly not the only pre-curl command that writes to stdout by default; they are plentiful.
And then: once I had made that decision and I released curl for the first time on March 20, 1998: the call was made. The default was set. I will not change a default and hurt millions of users. I rather continue to be questioned by newcomers, but now at least I can point to this blog post! :-)
About the wget rivalry
As I mention in my curl vs wget document, a very common comment to me about curl as compared to wget is that wget is "easier to use" because it needs no extra argument in order to download a single URL to a file on disk. I get that, if you type the full commands by hand you'll use about three keys less to write "wget" instead of "curl -O", but on the other hand if this is an operation you do often and you care so much about saving key presses I would suggest you make an alias anyway that is even shorter and then the amount of options for the command really doesn't matter at all anymore.
I put that argument in the same category as the people who argue that wget is easier to use because you can type it with your left hand only on a qwerty keyboard. Sure, that is indeed true but I read it more like someone trying to come up with a reason when in reality there's actually another one underneath. Sometimes that other reason is a philosophical one about preferring GNU software (which curl isn't) or one that is licensed under the GPL (which wget is) or simply that wget is what they're used to and they know its options and recognize or like its progress meter better.
I enjoy our friendly competition with wget and I seriously and honestly think it has made both our projects better and I like that users can throw arguments in our face like "but X can do Y"and X can alter between curl and wget depending on which camp you talk to. I also really like wget as a tool and I am the occasional user of it, just like most Unix users. I contribute to the wget project well, both with code and with general feedback. I consider myself a friend of the current wget maintainer as well as former ones.
Andreas Gal — VP8 and H.264 to both become mandatory for WebRTC
WebRTC is one of the most exciting things to happen to the Web in years: it has the potential to bring instant voice and video calling to anyone with a browser, finally unshackling us from proprietary plugins and installed apps. Firefox, Chrome, and Opera already support WebRTC, and Microsoft recently announced future support.
Unfortunately, the full potential of the WebRTC ecosystem has been held back by a long-running disagreement about which video codec should be mandatory to implement. The mandatory to implement audio codecs were chosen over two years ago with relatively little contention: the legacy codec G.711 and Opus, an advanced codec co-designed by Mozilla engineers. The IETF RTCWEB Working Group has been deadlocked for years over whether to pick VP8 or H.264 for the video side.
Both codecs have merits. On the one hand, VP8 can be deployed without having to pay patent royalties. On the other hand, H.264 has a huge installed base in existing systems and hardware. That is why we worked with Cisco to develop their free OpenH264 plugin and as of October this year, Firefox supports both H.264 and VP8 for WebRTC.
At the last IETF meeting in Hawaii the RTCWEB working group reached strong consensus to follow in our footsteps and make support for both H.264 and VP8 mandatory for browsers. This compromises was put forward by Mozilla, Cisco and Google. The details are a little bit complicated, but here's the executive summary:
- Browsers will be required to support both H.264 and VP8 for WebRTC.
- Non-browser WebRTC endpoints will be required to support both H.264 and VP8. However, if either codec becomes definitely royalty free (with no outstanding credible non-RF patent claims) then endpoints will only have to do that codec.
- "WebRTC-compatible" endpoints will be allowed to do either codec, both, or neither.
See the complete proposal by Mozilla Principal Engineer Adam Roach here. There are still a few procedural issues to resolve, but given the level of support in the room, things are looking good.
We believe that this compromise is the best thing for the Web at this time: It lets us move forward with confidence in WebRTC interoperability and allows people who for some reason or another really can't do one of these two codecs to be "WebRTC-compatible" and know they can interoperate with any WebRTC endpoint. This is an unmitigated win for users and Web application developers, as it provides broad interoperability within the WebRTC ecosystem.
It also puts a stake in the ground that what the community really needs is a codec that everyone agrees is royalty-free, and provides a continuing incentive for proponents of each codec to work towards this target.
Mozilla has been working for some time on such a new video codec which tries to avoid the patent thickets around current codec designs while surpassing the quality of the latest royalty-bearing codecs. We hope to contribute this technology to an IETF standardization effort following the same successful pattern as with Opus.
Filed under: Mozilla Tagged: Mozilla, VP8, WebRTC
Kartikaya Gupta — Killing the office suite
Have you ever had the experience of trying to write a document in MS Word (or Open/LibreOffice) and it keeps "correcting" your formatting to something you don't want? The last time I experienced that was about a year ago, and that was when I decided "screw it, I'll just write this in HTML instead". That was a good decision.
Pretty much anything you might want to use a word processor for, you can do in HTML - and oftentimes it's simpler. Sure, there's a bit of a learning curve if you don't know HTML, but that's true for anything. Now anytime I need to create "a document" (a letter, random notes or signs to print, etc.) I always do it in HTML rather than LibreOffice, and I'm the happier for it. I keep all my data in git repositories, and so it's a bonus that these documents are now in a plaintext format rather than a binary blob.
I realized that this is probably part of a trend - a lot of people I know nowadays to "powerpoint" presentations using web technologies such as reveal.js. I haven't seen many people comment on using web tech to do word processing, but I know I do it. The only big "office suite" thing left is the spreadsheet. It would be awesome if somebody wrote a drop-in JS spreadsheet library that you could include into a HTML page and instantly turn a table element into a spreadsheet.
I'm reminded of this old blog post by Joel Spolsky: How Trello is different. He talks about most of the people who use Excel really just use it because it provides a table format for entering things, rather than it's computational ability. HTML already provides that, but whenever I've tried doing that I find the markup/content ratio too high, so it always seemed like a pain. It would be nice to have a WSYIWYG tool that let you build a table (or spreadsheet) and import/export it as raw HTML that you can publish, print, share, etc.
As an addendum, that blog post by Joel also introduced me to the concept of unshipped code as "inventory", which is one of the reasons I really hate finding old bugs sitting around in Bugzilla with perfectly good patches that never landed!
Niko Matsakis — Allocators in Rust
There has been a lot of discussion lately about Rust's allocator story, and in particular our relationship to jemalloc. I've been trying to catch up, and I wanted to try and summarize my understanding and explain for others what is going on. I am trying to be as factually precise in this post as possible. If you see a factual error, please do not hesitate to let me know.
The core tradeoff
The story begins, like all interesting design questions, with a trade-off. The problem with trade-offs is that neither side is 100% right. In this case, the trade-off has to do with two partial truths:
- It is better to have one global allocator than two. Allocators like jemalloc, dlmalloc, and so forth are all designed to be the only allocator in the system. Of necessity they permit a certain amount of "slop", allocating more memory than they need so that they can respond to requests faster, or amortizing the cost of metadata over many allocations. If you use two different allocators, you are paying those costs twice. Moreover, the allocator tends to be a hot path, and you wind up with two copies of it, which leaves less room in the instruction cache for your actual code.
- Some allocators are more efficient than others. In particular, the default allocators shipped with libc on most systems tend not to be very good, though there are exceptions. One particularly good allocator is jemalloc. In comparison to the default glibc or windows allocator, jemalloc can be noticeably more efficient both in performance and memory use. Moreover, jemalloc offers an extended interface that Rust can take advantage of to gain even more efficiency (for example, by specifying the sizes of a memory block when it is freed, or by asking to reallocate memory in place when possible).
Clearly, the best thing is to use just one allocator that is also efficient. So, to be concrete, whenever we produce a Rust executable, everyone would prefer if that Rust executable – along with any C code that it uses – would just use jemalloc everywhere (or whatever allocator we decide is 'efficient' tomorrow).
However, in some cases we can't control what allocator other code will use. For example, if a Rust library is linked into a larger C program. In this case, we can opt to continue using jemalloc from within that Rust code, but the C program may simply use the normal allocator. And then we wind up with two allocators in use. This is where the trade-off comes into play. Is it better to have Rust use jemalloc even when the C program within which Rust is embedded does not? In that case, the Rust allocations are more efficient, but at the cost of having more than one global allocator, with the associated inefficiencies. I think this is the core question.
Two extreme designs
Depending on whether you want to prioritize using a single allocator or using an efficient allocator, there are two extreme designs one might advocate for the Rust standard library:
- When Rust needs to allocate memory, just call
mallocand friends.
- Compile Rust code to invoke jemalloc directly. This is what we currently do. There are many variations on how to do this. Regardless of which approach you take, this has the downside that when Rust code is linked into C code, there is the possibility that the C code will use one allocator, and Rust code another.
It's important to clarify that what we're discussing here is really the default behavior, to some extent. The Rust standard library already isolates the definition of the global allocator into a particular crate. End users can opt to change the definition of that crate. However, it would require recompiling Rust itself to do so, which is at least a mild pain.
Calling malloc
If we opted to default to just calling
malloc, this does not mean
that end users are locked into the libc allocator or anything like
that. There are existing mechanisms for changing what allocator is
used at a global level (though I understand this is relatively hard on
Windows). Presumably when we produce an actual Rust executables, we
would default to using jemalloc.
Calling malloc has the advantage that if a Rust library is linked into
a C program, both of them will be using the same global allocator,
whatever it is (unless of course that C program itself doesn't call
However, one downside of this is that we are not able to take advantage of the more advanced jemalloc APIs for sized deallocation and reallocation. This has a measureable effect in micro-benchmarks. I am not aware of any measurements on larger scale Rust applications, but there are definitely scenarios where the advanced APIs are useful.
Another potential downside of this approach is that
malloc is called
via indirection (because it is part of libc; I'm a bit hazy on the
details of this point, and would appreciate clarification). This
implies a somewhat higher overhead for calls to malloc/free than if we
fixed the allocator ahead of time. It's worth noting that this is the
normal setup that all C programs use by default, so relative to a
typical C program, this setup carries no overhead.
(When compiling a statically linked executables, rustc could opt to
malloc and friends to jemalloc at this point, which would
eliminate the indirection overhead but not take advantage of the
specialized jemalloc APIs. This would be a simplified variant of the
hybrid scheme I eventually describe below.)
Calling jemalloc directly
If we opt to hardcode Rust's default allocator to be jemalloc, we gain
several advantages. The performance of Rust code, at least, is not
subject to the whims of whatever global allocator the platform or
end-user provides. We are able to take full advantage of the
specialized jemalloc APIs. Finally, as the allocator is fixed to
jemalloc ahead of time, static linking scenarios do not carry the
additional overhead that calling
malloc implies (though, as I noted,
one can remove that overhead also when using
malloc via a simple
hybrid scheme).
Having Rust code unilatelly call jemalloc also carries downsides. For example, if Rust code is embedded as a library, it will not adopt the global allocator of the code that it is embedded within. This carries the performance downsides of multiple allocators but also a certain amount of risk, because a pointer allocated on one side cannot be freed on the other (some argue this is bad practice; this is certainly true if you do not know that the two sides are using the same allocator, but is otherwise legitimate, see the section below for more details).
The same problem can also occur in reverse, when C code is used from within Rust. This happens today with rustc: due to the specifics of our setup, LLVM uses the system allocator, not the jemalloc allocator that Rust is using. This causes extra fragmentation and memory consumption. It's also not great because jemalloc is better than the system allocator in many cases.
To prefix or not to prefix
One specific aspect of calling jemalloc directly concerns how it is built. Today, we build jemalloc using name prefixes, effectively "namespacing" it so that it does not interfere with the system allocator. This is what causes LLVM to use a different allocator in rustc. This has the advantage of clarity and side-stepping certain footguns around dynamic linking that could otherwise occur, but at the cost of forking the allocators.
A recent PR aimed to remove the prefix. It was rejected
because in a dynamic linking scenario, this creates a fragile
situation. Basically, the dynamic library ("client") defines
to be jemalloc. The host process also has a definition for
(the system allocator). The precise result will depend on the flags
and platform that you're running on, but there are basically two
possible outcomes, and both can cause perfectly legitimate code to
- The host process wins,
mallocmeans the same thing everywhere (this occurs on linux by default).
mallocmeans different things in the host and the client (this occurs on mac by default, and on linux with the
In the first case, crashes can arise if the client code should try to
intermingle usage of the nonstandard jemalloc API (which maps to
jemalloc) with the standard malloc API (which the client believes to
also be jemalloc, but which has been remapped to the system allocator
by the host). The jemalloc documentation isn't 100% explicit
on the matter, but I believe it is legal for code to (e.g.) call
mallocx and then call
free on the result. Hence if Rust should
link some C code that did that, it would crash under the first
In the second case, crashes can arise if the host/client attempt to
transfer ownership of memory. Some claim that this is not a legitimate
thing to do, but that is untrue: it is (usually) perfectly legal for
client code to (e.g.) call
strdup and then pass the result back to
the host, expecting the host to free it. (Granted, it is best to be
cautious when transfering ownership across boundaries like this, and
one should never call
free on a pointer unless you can be sure of
the allocator that was used to allocate that pointer in the first
place. But if you are sure, then it should be possible.)
UPDATE: I've been told that on Windows, freeing across DLL boundaries is something you can never do. On Reddit, Mr_Alert writes: "In Windows, allocating memory in one DLL and freeing it in another is very much illegitimate. Different compiler versions have different C runtimes and therefore different allocators. Even with the same compiler version, if the EXE or DLLs have the C runtime statically linked, they'll have different copies of the allocator. So, it would probably be best to link rust_alloc to jemalloc unconditionally on Windows." Given the number of differences between platforms, it seems likely that the best behavior will ultimately be platform dependent.
Fundamentally, the problems here are due to the fact that the client is attempting to redefine the allocator on behalf of the host. Forcing this kind of name conflict to occur intentionally seems like a bad idea if we can avoid it.
A hybrid scheme
There is also the possibility of various hybrid schemes. One such
option that Alex Crichton and I put together, summarized in
this gist, would be to have Rust call neither the standard
malloc nor the jemalloc symbols, but rather an intermediate set of
APIs (let's call them
rust_alloc). When compiling Rust libraries
("rlibs"), these APIs would be unresolved. These rust allocator APIs
would take all the information they need to take full advantage of
extended jemalloc APIs, if they are available, but could also be
"polyfilled" using the standard system malloc interface.
So long as Rust libraries are being compiled into "rlibs", these
rust_alloc dependencies would remain unresolved. An
rlib is
basically a statically linked library that can be linked into another
Rust program. At some point, however, a final artifact is produced, at
which point the
rust_alloc dependency must be fulfilled. The way we
fulfill this dependency will ultimately depend on what kind of
artifact is produced:
- Static library for use in a C program: link
- Dynamic library (for use in C or Rust): link
- Executable: resolve
rust_allocto jemalloc, and override the system malloc with jemalloc as well.
This seems to offer the best of both worlds. Standalone, statically linked Rust executables (the recommended, default route) get the full benefit of jemalloc. Code that is linked into C or dynamically loaded uses the standard allocator by default. Any C code used from within Rust executables will also call into jemalloc as well.
However, there is one major caveat. While it seems that this scheme would work well on linux, the behavior on other platforms is different, and it's not yet clear if the same scheme can be made to work as well on Mac and Windows.
Naturally, even if we sort out the cross-platform challenges, this
hybrid approach too is not without its downsides. It means that Rust
code built for libraries will not take full advantage of what jemalloc
has to offer, and in the case of dynamic libraries there may be more
overhead per
malloc invocation than if jemalloc were statically
linked. However, by the same token, Rust libraries will avoid the
overhead of using two allocators and they will also be acting more
like normal C code. And of course the embedding program may opt, in
its linking phase, to redirect
malloc (globally) to jemalloc.
So what should we do?
The decision about what to do has a couple of facets. In the immediate term, however, we need to take steps to improve rustc's memory usage. It seems to me that, at minimum, we ought to accept strcat's PR #18915, which ensures that Rust executables can use jemalloc for everything, at least on linux. Everyone agrees that this is a desirable goal.
Longer term, it is somewhat less clear. The reason that this decision is difficult is that there is no choice that is "correct" for all cases. The most performant choice will depend on the specifics of the case:
- Is the Rust code embedded?
- How much allocation takes place in Rust vs in the other language?
- What allocator is the other language using?
(As an example, the performance and memory use of
rustc improved
when we adopted jemalloc, even partially, but other applications will
fare differently.)
At this point I favor the general principle that Rust code, when
compiled as a library for use within C code, should more-or-less
behave like C code would behave. This seems to suggest that, when
building libraries for C consumption, Rust should just call
and people can use the normal mechanisms to inject jemalloc if they so
choose. However, when compiling Rust executables, it seems
advantageous for us to default to a better allocator and to get the
maximum efficiency we can from that allocator. The hybrid scheme aims
to achieve both of these goals but there may be a better way to go
about it, particularly around the area of dynamic linking.
I'd like to see more measurement regarding the performance impact of foregoing the specialized jemalloc APIs and using weak linking. I've seen plenty of numbers suggesting jemalloc is better than other allocators on the whole, and plenty of numbers saying that using specialized APIs helps in microbenchmarks. But it is unclear what the impact of such APIs (or weak linking) is on the performance of larger applications.
I'd also like to get the input from more people who have experience in this area. I've talked things over with strcat a fair amount, who generally favors using jemalloc even if it means two allocators. We've also reached out to Jason Evans, the author of jemalloc, who stressed the fact that multiple global allocators is generally a poor choice. I've tried to reflect their points in this post.
Note though that whatever we decide we can evolve it as we go. There is time to experiment and measure. One thing that is clear to me is that we do not want Rust to "depend on" jemalloc in any hard sense. That is, it should always be possible to switch from jemalloc to another allocator. This is both because jemalloc, good as it is, can't meet everyone's needs all the time, and because it's just not a necessary dependency for Rust to take. Establishing an abstraction boundary around the "Rust global allocator" seems clearly like the right thing to do, however we choose to fulfill it.
Patrick Cloke — Thunderbird Summit 2014
Last month (Oct. 15th to Oct. 18th, to be precise), twenty volunteers descended on Mozilla's Toronto office to discuss Mozilla Thunderbird. This included Mozilla employees, Thunderbird contributors of all sorts (developers, user interface designers, add-on reviewers), Lightning contributors, and chat/Instantbird contributors.
It was great to spend some quality hacking time with Florian and to meet Nihanth, both Instantbird guys who I talk to most days on IRC! I also had the pleasure of re-meeting a few people from the Mozilla Summit last year (I attended in Toronto) and to meet some brand new people!
A few pictures of the chat contributors: Nihanth; me, Florian and Nihanth; and Daniel (dialing in!)
It was really nice to actually sit down for a few days and work on Instantbird/Thunderbird without the distractions of "real life". I, unfortunately, spent the first day fixing an Instantbird bustage (from a mozilla-central change that removed some NSS symbols…why, I have no idea). But after that, we got some really exciting work done! We started cleaning up and finalizing some patches from Google Summer of Code 2014 to add WebRTC support to XMPP! You can check out the progress in bug 1018060.
First working call over Instantbird WebRTC.
Other highlights of the trip include eating the "Canadian delicacy" of poutine (with pulled pork on it)!
Mozilla Reps Community — Reps Weekly Call – November 13rd 2014
Last Thursday we had our regular weekly call about the Reps program, where we talk about what's going on in the program and what Reps have been doing during the last week.
- New Reps dashboard
- Portal UX initiative
- WoMoz update
- FX10 celebrations
AirMozilla video
Don't forget to comment about this call on Discourse and we hope to see you next week!
Mozilla Release Management Team — Firefox 34 beta7 to beta8
- 28 changesets
- 109 files changed
- 4554 insertions
- 219 deletions
List of changesets:
|Phil Ringnalda
|Bug 978211 followup, make compare-mozconfig work on Win64 again, r=glandium a=NPOTB - 8201c42832ef
|Mike Shal
|Bug 1072073 - pretty-l10n-check should also be -j1; r=glandium a=NPOTB - 6ab542eb236d
|Mike Shal
|Bug 1013730 - Have mach ignore broken disk io stats; r=gps a=NPOTB - daabf4d8995f
|Mike Shal
|Bug 1077597 - force -j1 for {pretty-}package-tests; r=glandium a=NPOTB - 8a6160e2ef98
|Mike Shal
|Bug 1084163 - Remove 'make check' from automation/build; r=glandium a=NPOTB - 5591e0a83c4d
|Mike Shal
|Bug 1085026 - Use sha512 hashes for mar files; r=glandium a=NPOTB - 72d8ba95b2db
|Mike Shal
|Bug 1087104 - Implement partial mar generation in make for 'mach build'; r=glandium a=NPOTB - ee2c3cfb4a7b
|Mike Shal
|Bug 1087104 - Set the partialInfo property for Balrog; r=glandium a=NPOTB - dc18ad2b4816
|David Keeler
|Bug 1085509 - Add telemetry for how many permanent certificate overrides users have. r=mmc, r=jcj, a=lsblakk - 1f1e5b70a075
|Christoph Kerschbaumer
|Bug 1069762 - Make CSP violation reports match the spec for redirects. r=sstamm, a=dveditz - b77384b124a4
|Gavin Sharp
|Bug 1061736: add DuckDuckGo as a search engine option in Firefox, r=dolske, a=gavin - 2231ed05a1b8
|Paul Kerr [:pkerr]
|Bug 1023539: Fix occasional timeouts of TURN webrtc transports with one-way connections r=bwc a=lmandel - d73c4671a18f
|Jan Keromnes
|Bug 1011562 - Ship Firefox OS fonts in Mulet. r=mshal, a=NPOTB - 00f9c65b2f83
|Gijs Kruitbosch
|Bug 1062096 - browser_aboutHome should use a fake search engine instead of google to test FHR reporting. r=adw, a=test-only - e32540fb1289
|Gijs Kruitbosch
|Bug 1094421 - Prepend www. to the search suggestion URL to avoid intermittent timeouts. rs=Mossop,me, a=test-only - db6d19e2b8e6
|Mark Finkle
|Bug 883254 - Add the duckduckgo searchplugin. r=margaret, a=sledru - f1c1658280cd
|Nick Alexander
|Bug 883254 - Add the duckduckgo searchplugin to certain locales. f=glandium, r=mfinkle, a=sledru - 32c3529eb076
|Markus Stange
|Bug 1061327 - Don't stop searching for scrolled layers when encountering a ScrollInfoLayer. r=botond, a=lmandel - 0174d3047d1a
|Markus Stange
|Bug 1061327 - When the scrolled layer is not an ancestor of the scrollbar layer, search the whole layer tree. r=botond, a=lmandel - eed413466305
|Panos Astithas
|Bug 1090967 - Don't use the Aurora-specific profile by default if this is not Aurora. r=bsmedberg, a=lmandel - 46829698a2b9
|Gijs Kruitbosch
|Bug 690307 - Make trimURL not generate URLs that parse back into search queries. r=mak, a=lmandel - ffb4891a237d
|Gijs Kruitbosch
|Bug 690307 - Add more tests for the localhost + spaces case. r=mak, a=test-only - 9ebc7ee50a9c
|Margaret Leibovic
|Bug 1093871 - Telemetry probe for number of items in reading list. r=rnewman, a=lmandel - f85f63d11f68
|Dão Gottwald
|Bug 1093368 - Customize mode theme picker shouldn't pass the default theme object to LightweightThemeManager.previewTheme. r=jaws, a=lmandel - fa1706ebf845
|Jeff Muizelaar
|Bug 1021265. Fix DisplayLink version expansion code. r=Bas,a=lawrence - 252c3ab238d0
|Benoit Girard
|Bug 1089380 - Remove ClipRectInLayersCoordinates. r=mattwoodrow, a=lmandel - 1e8f0a8c4474
|David Keeler
|Bug 1083118 - backout removal of unsafe, non-standardized legacy window.crypto functions r=bz a=lmandel ba=lmandel - fea4ac1165f9
|David Keeler
|Bug 1083118 - re-enable unsafe legacy window.crypto functions by default r=bz a=lmandel ba=lmandel - 87fd4f56cfed
Doug Belshaw — MozFest Web Literacy Map roundup
Because late is better than never, right?
Web literacy was, of course, a theme that ran through all of the Mozilla Festival this year. However, in this post I want to focus on a couple of sessions that focused specifically on the Web Literacy Map.
Prototypes and Pathways for Web Literacy
Session details (from schedule)
This session was led by Karen Smith, with me supporting. It was a practical, hands-on session where participants were able to chart learning pathways around the Privacy competency of the Web Literacy Map. This was based on a deliverable from the Badge Alliance working group on Digital & Web Literacies. We also used the recent UX Personas work to help frame the discussion.
Participants were asked to choose a persona and stick it to a large sheet of paper. They then explored what things that person was likely to want around privacy, and which things they'd like to avoid.
We then went through the kinds of badges from the Badge Alliance deliverable, and participants were asked to construct pathways that would make sense in their persona's context.
- Etherpad: https://bit.ly/PrivPathways
- Resources: the resources from this session can be found here.
Towards v2 of Mozilla's Web Literacy Map
Session details (from schedule)
I ran this session with community members Ibrahima Saar, Alvar Maciel, and Kim Wilkens. We led three groups of participants through the current Web Literacy Map and encouraged through what's changed in the year since v1 was launched. What's missing? How could it be better represented?
We ended up with some interesting results. As you can see with the group above, they imagined the Mozilla Manifesto as being the roots of a 'tree' that also included additional competencies.
Another group though three-dimensionally almost in terms of 'synapses'. The also experimented with a 'layered' approach.
The third group used 'refurbishing a house' as an organising metaphor. The journey started by looking underneath the carpets, then exploring further…
- MozFest 2014 photo set: https://flic.kr/s/aHsk29E96p
Remotee challenge
My super-talented colleague Jess Klein unfortunately couldn't be at MozFest, so she put together a number of 'remotee challenges'. One of these was how might we visualize the information in the Web Literacy Map?
It's worth reading the whole thread, as there's a lot of back-and-forth. I'd like to highlight a few things in particular:
- Greg McVerry geeked out on the etymology of the current strand names (Exploring / Building / Connecting) and a suggestion of what we could change them to (Reading / Writing / Participating).
- Chad Sansing suggested separating out the 'back-end' and 'front-end' aspects of the Web Literacy Map. Jess riffed on this with a chore chart approach.
- Caleb got involved by imagining and then drawing and interactive 'filter' for web literacies.
We'll be discussing these approaches in next Monday's community call. The focus of that call is on looking at responses to Proposal 3 of the community survey: 'I believe the Web Literacy Map should look more like a 'map'.
The canonical URL for all development relating to v2.0 of the Web Literacy Map is: https://bit.ly/weblitmapv2
Comments? Questions? Direct them to firstname.lastname@example.com or add them to the #TeachTheWeb discussion forum
Jeff Walden — Ten years
It was summer 2002, and I was on a bike tour across Michigan. For downtime reading I'd brought a bunch of unread magazines. One of the magazines I brought was a semi-recent PC Magazine with a review of the various browsers of the time. The major browsers were the focus, but a sidebar mentioned the Mozilla Suite and noted its being open source and open to downloads and contributions from anyone. It sounded unique and piqued my interest, so I filed the information away for future reference.
Sometime after I got home I downloaded a version of the Mozilla Suite: good, certainly better than Internet Explorer, but ponderous in its UI. I recommended it to a few people, but because of the UI somewhat half-heartedly, in an "if you can tolerate the UI, it's better" sort of sense. Somehow I stumbled into downloading betas and later nightlies, and I began reading and triaging bugs, even reporting a few bugs (duplicates!) of my own.
Sometime later, probably through the MozillaZine default bookmark, I learned about Phoenix, the then-current name of the browser whose ultimate name would be Firefox. (Very shortly after this it acquired the Firebird name, which stuck for only a couple releases until the ultimate rename.) It seemed to do everything the Suite did (or at least everything I cared about) without the horrible UI. I began recommending it unreservedly to people, surreptitiously installing it on high school computers, and so on.
One notable lack in Firebird of the time was its lack of help documentation. Firebird was good stuff. I wanted to see it succeed. I could fix this. So I began contributing to the Firebird Help project that wrote built-in help documentation for the browser. At the time this was an external project whose contents were occasionally imported into the main tree. (I believe it later moved directly into the tree, although I'm not certain. Ultimately the entire system was replaced with fully-online documentation, which fixed a whole bunch of problems around ease of contribution — not least that I'd lost time to contribute to that particular aspect, mostly having moved onto other things in the project.) Thus began the start of several years of work writing help documentation describing various parts of the UI, including a late-breaking October 2004 weekend spent documenting the new preferences UI in 1.0 — in just before the buzzer!
I observed release day from a distance in my dorm room, but Air Mozilla made that experience more immediate than it might have been. (218 users on the IRC channel! How times have changed. Our main developer channel as I write this contains 448 people, and that seems pretty typical.) Air Mozilla wasn't nearly as polished or streamlined as it is now. Varying-quality Creative Commons music as interludes between interviews, good times. But it was a start.
Ten years (and two internships, one proto-summit and two summits, and fulltime employment) later, I think it's both surprising and unsurprising just how far Mozilla and Firefox have come. Surprising, in that entering a market where the main competitor has 95% of the market usually isn't a winning strategy. (But you can't win if you don't try.) Yet also unsurprising, as Internet Explorer was really bad compared to its competition. (When released, IE6 was a really good browser, just as Tinderbox [now doubly-replaced] was once a really good continuous-integration system. But it's not enough to be good at one instant.) And a serendipitously-timed wave of IE security vulnerabilities over summer 2004 helped, too.
Here's to another ten years and a new round of challenges.
Yunier José Sosa Vázquez — Mozilla planea llevar la realidad virtual a Firefox
A principios de este año Mozilla inició un proyecto para explorar las posibilidades de llevar la realidad virtual a la Web, y en junio lanzó una versión experimental de Firefox con soporte para Oculus Rift, uno de los más populares receptores de cabeza de realidad virtual.
MozVR tiene como objetivo compartir todo tipo de experiencias virtuales en la Web, proporcionar recursos y mostrar el trabajo de los desarrolladores en la creciente comunidad web relacionada con la realidad virtual. Permitiendo que los usuarios experimenten la potencia de las posibilidades de la RV en Internet. Hasta el momento han sido creados varios demos demostrando hasta donde puede llegar la creatividad humana, entre estos demos podemos encontrar: Sechelt (un vuelo en 3D basado en WebGL a través de la Columbia Británica) y The Polar Sea (un documental que llevará a los usuarios a través del ártico mediante videos de 360º).
Al visitar MozVR.com, los usuarios podrán seguir una serie de indicaciones para configurar y usar Oculus Rift sin problemas con Firefox. Para disfrutar de esta nueva experiencia, hay que utilizar una versión de Firefox que debe descargarse desde aquí, en estos momentos no tenemos disponible su descarga desde Firefoxmanía. En Mozilla también están trabajando para llevar la realidad virtual a Chromuim y su código fuente está publicado en GitHub.
Fuente: MozVR
Mozilla WebDev Community — Webdev Extravaganza – November 2014
Once a month, web developers from across Mozilla get together to work on a sequel to The Pragmatic Programmer titled The Unhinged Technical Architect. While we argue over the use of oxford commas, we find time to talk about the work that we've shipped, share the libraries we're working on, meet new folks, and talk about whatever else is on our minds. It's the Webdev Extravaganza! The meeting is open to the public; you should stop by!
You can check out the wiki page that we use to organize the meeting, view a recording of the meeting in Air Mozilla, or attempt to decipher the aimless scrawls that are the meeting notes. Or just read on for a summary!
Shipping Celebration
The shipping celebration is for anything we finished and deployed in the past month, whether it be a brand new site, an upgrade to an existing one, or even a release of a library.
New Mozilla.org Pages and The Open Standard
craigcook stopped by to share a bunch of new things that launched from the Web Productions team, including a new mozilla.org homepage and a new contribute page. He also mentioned The Open Standard, which was launched with support from the Web Productions team.
Sites using contribute.json
We heard from peterbe about a new listing of sites with a contribute.json file. The listing pulls info hourly from the contribute.json files for each site in the list. Pull requests are welcome to add more Mozilla sites to the list.
Humble Mozilla Bundle and Voxatron Snippet
Yours truly mentioned the Humble Mozilla Bundle, a promotion with Humble Bundle where we offered several popular games for purchase that can run within a web browser.
To promote the bundle, jgruen and other Mozillians worked with Joseph White to make a minimal port of the Voxatron for use in an about:home snippet. All told, the snippet was about 200kb large and still managed to cram in a full 3d voxel engine that Firefox users were able to play with on their home page.
Open-source Citizenship
Here we talk about libraries we're maintaining and what, if anything, we need help with for them. Except this week there was nothing shared. Never mind!
New Hires / Interns / Volunteers / Contributors
Here we introduce any newcomers to the Webdev group, including new employees, interns, volunteers, or any other form of contributor.
|IRC Nick
|Kristján Oddsson
|careers.mozilla.org and snippets.mozilla.com
The Roundtable is the home for discussions that don't fit anywhere else.
ErikRose wanted to use configobj and asked for opinions on the library. peterbe gave a positive recommendation based on his experience using it in configman.
Tabzilla Update Bar
mythmon wanted to let people know about a new feature in Tabzilla. You can now trigger a feature called the Update Bar, which notifies users on old versions of Firefox that they should update their browser. pmac also called out the Translation Bar, which offers localized versions of the current page to users viewing your site in a language that doesn't match their preferred locale.
Workweek at Bernie's
I also gave a reminder about the Webdev meetup happening at the Portland Coincidental Workweek, an event now known as the Workweek at Bernie's. Follow that link for more details, and if you're going to be at the workweek and want to attend, contact me to RSVP.
After skimming the back cover of The Pragmatic Programmer, we came up with an outline describing how our book can teach you how to:
- Fight software;
- Not just duplicate knowledge, but infinitely copy it for massive gains;
- Write code so solid and enduring that it will run until AWS randomly kills your box;
- Encourage programming by fate;
- Nuke-proof your code using aspect-oriented programming and a few pounds of refrigerator-grade steel;
- Capture real, living requirements for sale as folk medicine in foreign countries;
- Test ruthlessly and physically punish any code that misbehaves;
- Delight your users with micro-transactions;
- Build teams of slouching young programmers wearing hoodies and jeans to attract investors; and
- Automate yourself out of a job.
If you're interested in web development at Mozilla, or want to attend next month's Extravaganza, subscribe to the email@example.com mailing list to be notified of the next meeting, and maybe send a message introducing yourself. We'd love to meet you!
See you next month!
David Dahl — Crypton: The Privacy Framework
Since I left Mozilla last year, I have been working on Crypton ( https://crypton.io ), an HTML5 application framework that places privacy above all else. Last month, my team was invited to the 'Hack In The Box', Malaysia security conference to lead a Lab Session on Crypton.
We were required to write a whitepaper for HiTB to publish, which was a great exercise, as my team has been meaning to write a paper for some time. It was a long trip, but worth it. We led a group of engineers through most of the Crypton API in about 2 hours.
I lived-coded the 'skeleton' of a messaging application in 74 lines of JavaScript. The coolest thing about this session was using Firefox's Scratchpad for all of the live-coding. It worked so well, we plan on doing more sessions like this.
Crypton is intended for use inside of mobile and desktop applications (FirefoxOS, too). Our initial target for development is via Cordova and node-webkit. The API hides all of the complexity of cryptography from the developer. Developers use APIs that look like any other hosted API, for instance, account creation looks something like this:
var myAccount; crypton.generateAccount('alice', 'password', function callback(error, successResult){ if (error) { console.error(err); return;} myAccount = successResult; });
Beneath this elegant, every-day-looking API call, a set of encryption keys are generated for encryption, signing and HMAC as well as a stretched key via the password that wraps all other keys. This keyring is then stored on the server making multiple-device operations easy.
As we move forward with the Crypton framework, we are building a "private backend service" which will make using Crypton trivially easy to use and require no system administration. More on this in a future post.
Mozilla Open Policy & Advocacy Blog — Designing Tiles for Trust
In August, I wrote about why we believe that trust is the most important currency of the Web. As I explained then, putting the user first, through transparency, choice and control is the only way to bring about the Web we want. In that post, I described several of our efforts designed to help us positively influence the ecosystem to garner more trust from users. One of those efforts was the Tiles feature. To influence the ecosystem, we have to participate in it.
As we move forward with Tiles, we wanted to share more details on our approach and invite your feedback. On November 10, we announced the release of a 10th anniversary edition of Firefox and firmly took our stand as an independent voice on the Web. With the anniversary edition, we made the Tiles experiment a part of Firefox.
We developed Tiles as an engaging and useful experience for our users. We designed the feature with a core focus on our Privacy Principles. Here are a few examples of how those principles influenced the feature:
- We ensure that no data is sent to us until you interact with the feature.
- You control the feature and can turn it off easily if you don't find it useful.
- You can audit us – all of our code is open and auditable by you. In particular, you can learn more about the code that powers this feature here.
- If a user has previously opted into Do Not Track, we assume this means the user does not want to see Tiles so we pref Tiles off for those users. (Note: If a user subsequently opts in to DNT, the user will need to switch Tiles off).
- The data we collect is transmitted over HTTPS/TLS.
We'd love your feedback on these principles, and any ideas or suggestions you might have to make Tiles more valuable to users. Leave a comment, or better yet, use this form to submit feedback directly to the Tiles team.
We're excited to move forward with Tiles and will continue to innovate with ways we can create positive impacts through this feature. Simultaneously, we will use our experiments through our Polaris initiative to test additional ways we can help create transparency, choice and control for our users.
Advancing Content — Announcing Firefox Tiles Going Live Today
With the 10th anniversary update to Firefox, there was an important update to the new tab experience, promoting Tiles to the Firefox stable build, and making them available to hundreds of millions of users around the world. Today we are excited to announce our first two sponsored Tiles partners: CVS Health and their media agency Mindshare North America, and Booking.com.
What are Tiles for?
For years, the new tab page in Firefox was unique in being intentionally blank – but by 2012, we learned that we could facilitate many users' workflow through the new tab page. We added thumbnails based on a calculation of "frecency" (frequency and recency of a user's browsing history, essentially the same way that the Awesome bar calculates relevance). We learned that many users find these history thumbnails useful; but we were not entirely satisfied with the feature. Thumbnails might be broken, and the experience could be much more dynamic.
We need to be able to use our voice with our users, for example to raise awareness around issues that affect the future of the Internet, and to promote those causes that we believe are important to that future.
We have been exploring the content discovery space. There are many aspects of digital advertising that concern us: from the overall integrity of the advertising system on the Web, to the user having control over what happens to their data, and then to what happens to the data once the user has given their consent. I have been writing for a while on this blog about the principles we follow and the ideas we have to improve digital advertising.
Lastly, we wanted to explore ways to contribute to the sustainability of the project in a way that we felt could align with Mozilla's values.
Tiles are our first iteration on starting to solve these problems. They create a more useful, attractive and dynamic new tab page. Tiles also represent an important part of our efforts to create new communications, content and advertising experiences over which Firefox users maintain control.
Partnering with Mozilla
We're very excited to have partnered with CVS Health (and Mindshare/GroupM) in the United States and Booking.com globally as our first two Firefox sponsored Tiles partners. We are live in 8 languages and 25 different countries*, and will continue to iterate with Mindshare/GroupM and Booking.com, as well as with our community, as we continue to improve on the experience.
We have been delighted to work with Mindshare/GroupM and Booking.com. When we collaborate, we need to understand the vision and objectives of the partner, and to understand if that partner is able to work within the framework of Mozilla's principles. Running sponsored content in Tiles is results-based, not surveillance-based. We do not allow tracking beacons or code in Tiles. We are not collecting, or providing them with, the data about you that most digital ad networks do. There are certain categories that require screening or what's commonly known as age-gating, or have other sensitivities, that we will stay away from, such as alcohol and pharmaceuticals.
The user's experience
For users with no browsing history (typically a new installation), they will see Directory Tiles offering an updated, interactive design and suggesting useful sites. A separate feature, Enhanced Tiles, will improve upon the existing new tab page experience for users who already have a history in their browser.
Tiles provides Mozilla (including our local communities) new ways to interact with and communicate with our users. (If you've been using a pre-release Firefox build, you might have seen promotions for Citizenfour, a documentary about Edward Snowden and the NSA, appearing in your new tab in the past few weeks.)
Tiles also offers Mozilla new partnership opportunities with advertisers and publishers all while respecting and protecting our users. These sponsorships serve several important goals simultaneously by balancing the benefits to users of improved experience, control and choice, with sustainability for Mozilla.
What users currently see in the New:Tab page on Firefox desktop will continue to evolve, just like any digital product would. And it will evolve along the lines I discussed earlier here. Above all, we need to earn and maintain users' trust.
You can learn more about Tiles here.
Looking ahead
User control and transparency are embedded in all of our design and architecture, and principles that we seek to deliver our users throughout their online life: trust is something that you earn every day. The Tiles-related user data we collect is anonymized after we receive it – as it is for other parts of Firefox that we instrument to ensure a good experience. And of course, a user can simply switch the new tab page Tiles feature off. One thing I must note: users of ad blocking add-ons such as Ad Block Plus will see adverts by default and will need to switch Tiles off in Firefox if they wish to see no ads in their New Tab page. You can read more about how we design for trust here. (Note: AdBlock Plus is not a Mozilla product and their content blocking mechanism is under their control. We expect that they may add support for the Tiles in future releases)
With the testing we've done, we're satisfied that users will find this an experience that they understand and trust – but we will always have that as a development objective. You can expect us to iterate frequently, but we will never assume trust – we will always work to earn it. And if we do have and maintain that trust, we can create potentially the best digital advertising medium on the planet.
We believe that we can do this, and offer a better way to deliver and to receive adverts that users find useful and relevant. And we also believe that this is a great opportunity for advertisers who share our vision, and who wish to reach their audience in a way that respects them and their trust. If that's you, we want to hear from you. Feel free to reach out to email@example.com.
And a big thank you to our initial launch partners, CVS Health, Booking.com, and Citizenfour who see our vision and are supporting Mozilla to have greater impact in the world.
* that list in full:
Argentina, Australia, Austria, Belarus, Belgium, Brazil, Canada, Chile, Colombia, Ecuador, France, Germany, Hong Kong, Japan, Kazakhstan, Mexico, New Zealand, Peru, Russia, Saudi Arabia, Spain, Switzerland, United Kingdom, United States and Venezuela.
Advancing Content — Announcing Firefox Tiles Going Live
With the 10th anniversary update to Firefox, there was an important update to the new tab experience, promoting Tiles to the Firefox stable build, and making them available to hundreds of millions of users around the world. Today we are excited to announce our first two sponsored Tiles partners: CVS Health and their media agency Mindshare North America, and Booking.com.
What are Tiles for?
For years, the new tab page in Firefox was unique in being intentionally blank – but by 2012, we learned that we could facilitate many users' workflow through the new tab page. We added thumbnails based on a calculation of "freceny" (frequency and recency of a user's browsing history, essentially the same way that the Awesome bar calculates relevance). We learned that many users find these history thumbnails useful; but we were not entirely satisfied with the feature. Thumbnails might be broken, and the experience could be much more dynamic.
We need to be able to use our voice with our users, for example to raise awareness around issues that affect the future of the Internet, and to promote those causes that we believe are important to that future.
We have been exploring the content discovery space. There are many aspects of digital advertising that concern us: from the overall integrity of the advertising system on the Web, to the user having control over what happens to their data, and then to what happens to the data once the user has given their consent. I have been writing for a while on this blog about the principles we follow and the ideas we have to improve digital advertising.
Lastly, we wanted to explore ways to contribute to the sustainability of the project in a way that we felt could align with Mozilla's values.
Tiles are our first iteration on starting to solve these problems. They create a more useful, attractive and dynamic new tab page. Tiles also represent an important part of our efforts to create new communications, content and advertising experiences over which Firefox users maintain control.
Partnering with Mozilla
We're very excited to have partnered with CVS Health (and Mindshare/GroupM) in the United States and Booking.com globally as our first two Firefox sponsored Tiles partners. We are live in 8 languages and 25 different countries*, and will continue to iterate with Mindshare/GroupM and Booking.com, as well as with our community, as we continue to improve on the experience.
We have been delighted to work with Mindshare/GroupM and Booking.com. When we collaborate, we need to understand the vision and objectives of the partner, and to understand if that partner is able to work within the framework of Mozilla's principles. Running sponsored content in Tiles is results-based, not surveillance-based. We do not allow tracking beacons or code in Tiles. We are not collecting, or providing you with, the data about our audience that most digital ad networks do. There are certain categories that require screening, or have other sensitivities, that we will stay away from, such as alcohol and pharmaceuticals.
The user's experience
For users with no browsing history (typically a new installation), they will see Directory Tiles offering an updated, interactive design and suggesting useful sites. A separate feature, Enhanced Tiles, will improve upon the existing new tab page experience for users who already have a history in their browser.
Tiles provides Mozilla (including our local communities) new ways to interact with and communicate with our users. (If you've been using a pre-release Firefox build, you might have seen promotions for Citizenfour, a documentary about Edward Snowden and the NSA, appearing in your new tab in the past few weeks.)
Tiles also offers Mozilla new partnership opportunities with advertisers and publishers all while respecting and protecting our users. These sponsorships serve several important goals simultaneously by balancing the benefits to users of improved experience, control and choice, with sustainability for Mozilla.
What users currently see in the New:Tab page on Firefox desktop will continue to evolve, just like any digital product would. And it will evolve along the lines I discussed earlier here. Above all, we need to earn and maintain users' trust.
Looking ahead
User control and transparency are embedded in all of our design and architecture, and principles that we seek to deliver our users throughout their online life: trust is something that you earn every day. The Tiles-related user data we collect is anonymized after we receive it – as it is for other parts of Firefox that we instrument to ensure a good experience. And of course, a user can simply switch the new tab page Tiles feature off. One thing I must note: users of ad blocking add-ons such as Ad Block Plus will see adverts by default and will need to switch Tiles off in Firefox if they wish to see no ads in their New Tab page. You can read more about how we design for trust here.
With the testing we've done, we're satisfied that users will find this an experience that they understand and trust – but we will always have that as a development objective. You can expect us to iterate frequently, but we will never assume trust – we will always work to earn it. And if we do have and maintain that trust, we can create potentially the best digital advertising medium on the planet.
We believe that we can do this, and offer a better way to deliver and to receive adverts that users find useful and relevant. And we also believe that this is a great opportunity for advertisers who share our vision, and who wish to reach their audience in a way that respects them and their trust. If that's you, we want to hear from you. Feel free to reach out to firstname.lastname@example.com.
And a big thank you to our initial launch partners, CVS Health, Booking.com, and Citizenfour who see our vision and are supporting Mozilla to have greater impact in the world.
* that list in full:
Argentina, Australia, Austria, Belarus, Belgium, Brazil, Canada, Chile, Colombia, Ecuador, France, Germany, Hong Kong, Japan, Kazakhstan, Mexico, New Zealand, Peru, Russia, Saudi Arabia, Spain, Switzerland, United Kingdom, United States and Venezuela.
Robert O'Callahan — Relax, Scaling User Interfaces By Non-Integer Scale Factors Is Okay
About seven years ago we implemented "full zoom" in Firefox 3. An old blog post gives some technical background to the architectural changes enabling that feature. When we first implemented it, I expected non-integer scale factors would never work as well as scaling by integer multiples. Apple apparently thinks the same, since (I have been told) on Mac and iOS, application rendering is always to a buffer whose size is an integer multiple of the "logical window size". GNOME developers apparently also agree since their org.gnome.desktop.interface scaling-factor setting only accepts integers. Note that here I'm conflating user-initiated "full zoom" with application-initiated "high-DPI rendering"; technically, they're the same problem.
Several years of experience shows I was wrong, at least for the Web. Non-integer scale factors work just as well as integer scale factors. For implementation reasons we restrict scale factors to 60/N for positive integers N, but in practice this gives you a good range of useful values. There are some subtleties to implementing scaling well, some of which are Web/CSS specific.
For example, normally we snap absolute scaled logical coordinates to screen pixels at rendering time, to ensure rounding error does not accumulate; if the distance between two logical points is N logical units, then the distance between the rendered points stays within one screen pixel of the ideal distance S*N (where S is the scale factor). The downside is that a visual distance of N logical units may be rendered in some places as ceil(S*N) screen pixels and elsewhere as floor(S*N) pixels. Such inconsistency usually doesn't matter much but for CSS borders (and usually not other CSS drawing!), such inconsistent widths are jarring. So for CSS borders (and only CSS borders!) we round each border width to screen pixels at layout time, ensuring borders with the same logical width always get the same screen pixel width.
I'm willing to declare victory in this area. Bug reports about unsatisfactory scaling of Web page layouts are very rare and aren't specific to non-integral scale factors. Image scaling remains hard; the performance vs quality tradeoffs are difficult --- but integral scale factors are no easier to handle than non-integral.
It may be that non-Web contexts are somehow more inimical to non-integral scaling, though I can't think why that would be.
Mic Berman — An exercise for creating space, saying 'no' and redefining your next chapter: The Mason Jar Exercise for Spaciousness, Choice and Resonance
How much space do you have in your life? How much of it is filled with obligations you've accepted for others? What percentage do you give to yourself for pursuits that are just for you? Could you stand to make a little more space in your life just for you? Could less commitments mean more quality time to focus on your Vision for your life?
I asked myself these questions recently and discovered that my life was almost entirely filled with obligations, guilt and hardly any room to pursue my own passions, or time to rejuvenate. One of my core values is to be of service to people - I love being helpful, creating value, offering support and transformation as a coach. Because of this, I felt obligated to serve my clients whenever they needed, whether it cut in to personal commitments or not . I would feel guilty if I wasn't able to be available for them. It became that I was always putting everyone else first, and I was way at the bottom of the list —or sometimes I never made it on to the list at all.
When I noticed how busyness had taken over, I devised what I call "the Mason Jar Exercise". I needed to create space and reconnect with Vision and Purpose and, have time to contemplate what's next so that I can choose actions that support my next big chapter.
Many of the folks I coach have similar challenges. They feel overwhelmed, have no time for themselves and feel so busy in their lives they've lost their way, lost their deep connection to their vision and purpose.Do you find that any of what I'm talking about applies to you?? Can you check in with how your life is resonating with you, how much you feel in tune with your values in all of your pursuits, supporting roles and actions?
If you find, like so many of us, you could use more space and time in your life, try out the Mason Jar exercise for yourself. Here's how:
The Mason Jar Exercise for Spaciousness, Choice and Resonance
1. Take a Mason Jar and fill it with gravel (or any other small rocks) in proportion to how full you feel your life is out of integrity with your values. It's a great way to visualize the things taking up space in your life, I was at about 85%.
2. Fill the remaining space with some of your favourite things e.g. momentoes, shells, etc.
3. As you stare at this now full, or even over full jar, contemplate what you want for your life and write out your 'stake'—the mantra or statement that propels you forward in integrity. For me it was "with laughter and steadfast focus I open the door to my new way".
4. Each day choose a stone to 'throw back into nature'. In my case, I would declare my stake, open my front door and toss it into our forest. Then you must challenge yourself in the day to let go of or get rid of an obligation or choice you've made that's out of line with your values until the jar is empty save for your few favourite things. For example, 'Today I am ridding myself of the obligation to ___ and this is making room for me to embrace ___". Now you can begin a study of the space, the singular beauty and significance of the items that remain. Notice what you want more of and where the draw is to fill it back up again.
This was a powerful and transformative exercise for me. It took me about a month to complete. I noticed how full my life was in activity and distraction and that I felt out of integrity with my values and that I wasn't making resonant choices. The result was a beautiful appreciation for my life and a deep acknowledgement of how I want to live. Less has become more. Stillness has expanded my perspectives. Now I feel like I can take on my next mountain.
What are you noticing?
Roberto A. Vitillo — Recommending Firefox add-ons with Spark
We are currently evaluating possible replacements for our Telemetry map-reduce infrastructure. As our current data munging machinery isn't distributed, analyzing days worth of data can be quite a pain. Also, many algorithms can't easily be expressed with a simple map/reduce interface.
So I decided to give Spark another try. "Another" because I have played with it in the past but I didn't feel it was mature enough to be run in production. And I wasn't the only one to think that apparently. I feel like things have changed though with the latest 1.1 release and I want to share my joy with you.
What is Spark?
In a nutshell, "Spark is a fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala and Python, and an optimized engine that supports general execution graphs. It also supports a rich set of higher-level tools including Spark SQL for SQL and structured data processing, MLlib for machine learning, GraphX for graph processing, and Spark Streaming."
Spark primary abstraction is the Resilient Distributed Dataset (RDD), which one can imagine as a distributed pandas or R data frame. The RDD API comes with all kinds of distributed operations, among which also our dear map and reduce. Many RDD operations accept user-defined Scala or Python functions as input which allow average Joe to write distributed applications like a pro.
A RDD can also be converted to a local Scala/Python data structure, assuming the dataset is small enough to fit in memory. The idea is that once you chopped the data you are not interested in, what you are left with fits comfortably on a single machine. Oh and did I mention that you can issue Spark queries directly from a Scala REPL? That's great for performing exploratory data analyses.
The greatest strength of Spark though is the ability to cache RDDs in memory. This allows you to run iterative algorithms up to 100x faster than using the typical Hadoop based map-reduce framework! It has to be remarked though that this feature is purely optional. Spark works flawlessly without caching, albeit slower. In fact in a recent benchmark Spark was able to sort 1PB of data 3X faster using 10X fewer machines than Hadoop, without using the in-memory cache.
A Spark cluster can run in standalone mode or on top of YARN or Mesos. To the very least for a cluster you will need some sort of distributed filesystem, e.g. HDFS or NFS. But the easiest way to play with it though is to run Spark locally, i.e. on OSX:
brew install spark spark-shell --master "local[*]"
The above commands start a Scala shell with a local Spark context. If you are more inclined to run a real cluster, the easiest way to get you going is to launch an EMR cluster on AWS:
aws emr create-cluster --name SparkCluster --ami-version 3.3 --instance-type m3.xlarge \ --instance-count 5 --ec2-attributes KeyName=vitillo --applications Name=Hive \ --bootstrap-actions Path=s3://support.elasticmapreduce/spark/install-spark
Then, once connected to the master node, launch Spark on YARN:
yarn-client /home/hadoop/spark/bin/spark-shell --num-executors 4 --executor-cores 8 \ --executor-memory 8g —driver-memory 8g
The parameters of the executors (aka worker nodes) should obviously be tailored to the kind of instances you launched. It's imperative to spend some time understanding and tuning the configuration options as Spark doesn't automagically do it for you.
Now what?
Time for some real code. Since Spark makes it so easy to write distributed analyses, the bar for a Hello World application should be consequently be much higher. Let's write then a simple, albeit functional, Recommender Engine for Firefox add-ons.
In order to do that, let's first go over quickly the math involved. It turns out that given a matrix of the rankings of each user for each add-on, the problem of finding a good recommendation can be reduced to matrix factorization problem:
The model maps both users and add-ons to a joint latent factor space of dimensionality . Both users and add-ons are thus seen as vectors in that space. The factors express latent characteristics of add-ons, e.g. if a an add-on is related to security or to UI customization. The ratings are then modeled as inner products in that space, which is proportional to the angle of the two vectors. The closer the characteristics of an add-on align to the preferences of the user in the latent factor space, the higher the rating.
But wait, Firefox users don't really rate add-ons. In fact the only information we have in Telemetry is binary: either a user has a certain add-on installed or he hasn't. Let's assume that if someone has a certain add-on installed, he probably likes that add-on. That's not true in all cases and a more significant metric like "usage time" or similar should be used.
I am not going to delve into the details, but having binary ratings changes the underlying model slightly from the conceptual one we have just seen. The interested reader should read this paper. Mllib, a machine learning library for Spark, comes out of the box with a distributed implementation of ALS which implements the factorization.
Now that we have an idea of the theory, let's have a look at how the implementation looks like in practice. Let's start by initializing Spark:
val conf = new SparkConf().setAppName("AddonRecommender") val sc = new SparkContext(conf)
As the ALS algorithm requires tuples of (user, addon, rating), let's munge the data into place:
val ratings = sc.textFile("s3://mreid-test-src/split/").map(raw => { val parsedPing = parse(raw.substring(37)) (parsedPing \ "clientID", parsedPing \ "addonDetails" \ "XPI") }).filter{ // Remove sessions with missing id or add-on list case (JNothing, _) => false case (_, JNothing) => false case (_, JObject(List())) => false case _ => true }.map{ case (id, xpi) => { val addonList = xpi.children. map(addon => addon \ "name"). filter(addon => addon != JNothing && addon != JString("Default")) (id, addonList) }}.filter{ case (id, addonList) => { // Remove sessions with empty add-on lists !addonList.isEmpty }}.flatMap{ case (id, addonList) => { // Create add-on ratings for each user addonList.map(addon => (id.extract[String], addon.extract[String], 1.0)) }}
Here we extract the add-on related data from our json Telemetry pings and filter out missing or invalid data. The ratings variable is a RDD and as you can see we used the distributed map, filter and flatMap operations on it. In fact it's hard to tell apart vanilla Scala code from the distributed one.
As the current ALS implementation doesn't accept strings for the user and add-on representations, we will have to convert them to numeric ones. A quick and dirty way of doing that is to hash the strings:
// Positive hash function def hash(x: String) = x.hashCode & 0x7FFFFF val hashedRatings = ratings.map{ case(u, a, r) => (hash(u), hash(a), r) }.cache val addonIDs = ratings.map(_._2).distinct.map(addon => (hash(addon), addon)).cache
We are nearly there. To avoid overfitting, ALS uses regularization, the strength of which is determined by a parameter . As we don't know beforehand the optimal value of the parameter, we can try to find it by minimizing the mean squared error over a pre-defined grid of values using k-fold cross-validation.
// Use cross validation to find the optimal number of latent factors val folds = MLUtils.kFold(hashedRatings, 10, 42) val lambdas = List(0.1, 0.2, 0.3, 0.4, 0.5) val iterations = 10 val factors = 100 // use as many factors as computationally possible val factorErrors = lambdas.flatMap(lambda => { folds.map{ case(train, test) => val model = ALS.trainImplicit(train.map{ case(u, a, r) => Rating(u, a, r) }, factors, iterations, lambda, 1.0) val usersAddons = test.map{ case (u, a, r) => (u, a) } val predictions = model.predict(usersAddons).map{ case Rating(u, a, r) => ((u, a), r) } val ratesAndPreds = test.map{ case (u, a, r) => ((u, a), r) }.join(predictions) val rmse = sqrt(ratesAndPreds.map { case ((u, a), (r1, r2)) => val err = (r1 - r2) err * err }.mean) (model, lambda, rmse) } }).groupBy(_._2) .map{ case(k, v) => (k, v.map(_._3).reduce(_ + _) / v.length) }
Finally, it's just a matter of training ALS on the whole dataset with the optimal value and we are good to go to use the recommender:
// Train model with optimal number of factors on all available data val model = ALS.trainImplicit(hashedRatings.map{case(u, a, r) => Rating(u, a, r)}, factors, iterations, optimalLambda._1, 1.0) def recommend(userID: Int) = { val predictions = model.predict(addonIDs.map(addonID => (userID, addonID._1))) val top = predictions.top(10)(Ordering.by[Rating,Double](_.rating)) top.map(r => (addonIDs.lookup(r.product)(0), r.rating)) } recommend(hash("UUID..."))
I omitted some details but you can find the complete source on my github repository.
To submit the packaged job to YARN run:
spark-submit --class AddonRecommender --master yarn-client --num-executors 4 \ --executor-cores 8 --executor-memory 8g addon-recommender_2.10-1.0.jar
So what?
Question is, how well does it perform? The mean squared error isn't really telling us much so let's take some fictional user session and see what the recommender spits out.
For user A that has only the add-on Ghostery installed, the top recommendations are, in order:
- NoScript
- Web of Trust
- Symantec Vulnerability Protection
- Better Privacy
- LastPass
- DuckDuckGo Plus
- HTTPS-Everywhere
- Lightbeam
- Google Translator for Firefox
One could argue that 1 out of 10 recommendations isn't appropriate for a security aficionado. Now it's the turn of user B who has only the Firebug add-on installed:
- Web Developer
- FiddlerHook
- Greasemonkey
- ColorZilla
- User Agent Switcher
- McAfee
- RealPlayer Browser Record Plugin
- FirePHP
- Session Manager
There are just a couple of add-ons that don't look that great but the rest could fit the profile of a developer. Now, considering that the recommender was trained only on a couple of days of data for Nightly, I feel like the result could easily be improved with more data and tuning, like filtering out known Antivirus, malware and bloatware.
Kim Moir — Scaling capacity while saving cash
|Scale in the Market ©Clint Mickel, Creative Commons by-nc-sa 2.0
Although I didn't have the opportunity to give the talk in person, the slides for it are available on slideshare and my mozilla people account The talk describes how we scaled our continuous integration infrastructure on AWS to handle double the amount of pushes it handled in early 2013, all while reducing our AWS monthly bill by 2/3.
|Cost per push from Oct 2012 until Oct 2014. This does not include costs for on premise equipment. It reflects our monthly AWS bill divided by the number of monthly pushes (commits). The chart reflects costs from October 2012-2014.
Thank you to Dinah McNutt and the other program committee members for organizing this summit. I look forward to watching the talks once they are online.
Kim Moir — Mozilla pushes - October 2014
We didn't have a record breaking month in terms of the number of pushes, however we did have a daily record on October 18 with 715 pushes.
12821 pushes, up slightly from the previous month
414 pushes/day (average)
Highest number of pushes/day: 715 pushes on October 8
22.5 pushes/hour (average)
General Remarks
Try keeps had around 39% of all the pushes, and gaia-try has about 31%. The three integration repositories (fx-team, mozilla-inbound and b2g-inbound) account around 21% of all the pushes
August 2014 was the month with most pushes (13,090 pushes)
August 2014 has the highest pushes/day average with 422 pushes/day
July 2014 has the highest average of "pushes-per-hour" with 23.51 pushes/hour
October 8, 2014 had the highest number of pushes in one day with 715 pushes
Mozilla Reps Community — New dashboard and a first VIP action item: documenting impact
We're very proud to announce that we have a new Reps Dashboard that lists your action items! You will be able to find most of the action items that you have in the Reps program, it will help you organize your activities and plan better your time. We're also hoping that this will help mentors and council members manage the work load, be able to prioritize and ultimately keep the program running smoothly.
Check out the dashboard and let us know your thoughts. We know there might be some improvements to be done, so your feedback will help us figure this out.
This dashboard comes at the perfect time, we have a first mission for ALL Reps and the dashboard will allow you to do this in no time. We want to understand the impact of the Reps program in 2014 so we are asking all of you to please update ALL the post event metrics for this year. It won't take much time and at the end you'll help us articulate better the impact that we're having with the Reps program. We are introducing text fields, so that you can add important links to your post events metrics. So any links to the makes created, the press articles generated, the social media impact would be of great help!
Help us understand how much reach your event had. How many people attended, how many people did you reach on social media, how many through press articles or blog posts? Let's work together on making the impact of our work understandable. We have so much to be proud of, let's document it!