text:
The D-Bus protocol is frozen (only compatible extensions are allowed) as of November 8, 2006. However, this specification could still use a fair bit of work to make interoperable reimplementation possible without reference to the D-Bus reference implementation. Thus, this specification is not marked 1.0Ahem. After over a decade (first release in Sept. 2003!!) people still haven't documented what it actually does.
"D-Bus is low-overhead because it uses a binary protocol"then
"Immediately after connecting to the server, the client must send a single nul byte."followed by
"A nul byte in any context other than the initial byte is an error; the protocol is ASCII-only."Mmmh. Hmmm. Whut?
The string-like types are basic types with a variable length. The value of any string-like type is conceptually 0 or more Unicode codepoints encoded in UTF-8, none of which may be U+0000. The UTF-8 text must be validated strictly: in particular, it must not contain overlong sequences or codepoints above U+10FFFF. Since D-Bus Specification version 0.21, in accordance with Unicode Corrigendum #9, the "noncharacters" U+FDD0..U+FDEF, U+nFFFE and U+nFFFF are allowed in UTF-8 strings (but note that older versions of D-Bus rejected these noncharacters).So there seems to be some confusion what things like "binary" mean, and UTF-8 seems to be quite challenging too, but no worry: At least we are endian-proof!
A block of bytes has an associated byte order. The byte order has to be discovered in some way; for D-Bus messages, the byte order is part of the message header as described in the section called "Message Format". For now, assume that the byte order is known to be either little endian or big endian.Hmm? Why not just define network byte order and be happy? Well ... we're even smarterer:
The signature of the header is: "yyyyuua(yv)"Ok, so ...
1st BYTE Endianness flag; ASCII 'l' for little-endian or ASCII 'B' for big-endian. Both header and body are in this endianness.We actually waste a BYTE on each message to encode endianness, because ... uhm ... we run on ... have been ... I don't get it. And why a full byte (with a random ASCII mnemonic) instead of a bit? The whole 32bit bytemash at the beginning of the header could be collapsed into 8 bit if it were designed. Of course performance will look silly if you use an ASCII protocol over the wire with generous padding ... so let's kdbus because performance. What teh? Just reading this "spec" makes me want to get more drunk.
@org.freedesktop.systemd1.Privileged("true") UnlockSessions();Looking through the man page there's no documentation what a line beginning with "@" means. Because it's obvious!@113###
Method, interface, property, and signal elements may have "annotations", which are generic key/value pairs of metadata. They are similar conceptually to Java's annotations and C# attributes.I have no idea what that means, but I guess as the name implies it's just a textual hint for the developer. Or not? Great to see a specification not define its own terms.
@org.freedesktop.systemd1.Privileged("true") CreateSession(in u arg_0, in u arg_1, in s arg_2, in s arg_3, in s arg_4, in s arg_5, in u arg_6, in s arg_7, in s arg_8, in b arg_9, in s arg_10, in s arg_11, in a(sv) arg_12, out s arg_13, out o arg_14, out s arg_15, out h arg_16, out u arg_17, out s arg_18, out u arg_19, out b arg_20);with the details being:
CreateSession() and ReleaseSession() may be used to open or close login sessions. These calls should never be invoked directly by clients. Creating/closing sessions is exclusively the job of PAM and its pam_systemd module.*cough*
LockSession() asks the session with the specified ID to activate the screen lock.And then we look at the sourcecode to learn that it actually just calls:
session_send_lock(session, streq(sd_bus_message_get_member(message), "LockSession"));Which calls a function somewhere else which then does:
return sd_bus_emit_signal( s->manager->bus, p, "org.freedesktop.login1.Session", lock ? "Lock" : "Unlock", NULL);So in the end it internally sends a dbus message to a different part of itself, and that sends a dbus signal that "everyone" is supposed to listen to.
PowerOff(in b arg_0); Reboot(in b arg_0); Suspend(in b arg_0); Hibernate(in b arg_0); HybridSleep(in b arg_0);Here we have a new API call for each flavour of power management. And there's this stuff:
The main purpose of these calls is that they enforce PolicyKit policyAnd I have absolutely no idea about the mechanism(s) involved. Do I need to query PK myself? How does the dbus API know? Oh well, just read more code, and interpret it how you think it might work. Must be good enough.
About a year ago, I was writing about my Round the World trip winding down and returning to the workforce, my career. I've gone through a whole bunch of 'things' in the past year which mostly remind me that 1) life is short and random, and 2) I can do anything I want to.
On the first topic, I did a number to my spine and compressed L1, L2 vertebrae. About a 3 month resting period and I'm still recovering from that one, probably will be for the rest of my life. Oh, I broke my wrist too. All this from a little skydiving accident, which I'll spare the details. I'm back at the gym, eating well, and really inspired to build myself better than I was. I count my lucky stars that I'm able to make a full recovery. Ergo, life is short and random. However, it really opened up my viewpoints on many of life's topics and made me realize all the calculated risks that humans take everyday.
Speaking of risks, enter new job…
A year ago, I was writing about starting a new job, getting a new apartment, and new car all within two weeks. Now I'm able to say that I'm at it again. While it may not be the same as traveling to a new country every few weeks, it is still very exciting. In December, I'll be starting a new role at a new company, SPS Commerce. It was great working at Reeher, and I have nothing but good things to say about the company and the people. I'm also moving, but only 15 minutes away.
I'm thrilled to accelerate my career and position myself where I was prior to my career break started. It hasn't been exactly what I envisioned, but does anything work out like we think? Now, for all the naysayers that say a career-break on your mid-20s resume is career suicide… I challenge you to go for your dreams because life is short and you can do anything you want to.
Mike coined the term multimedia fiefdoms recently. He points to a number of different streaming, purchase and rental services for video content (movies, TV series) as the new battleground for users (consumers in this case). There are of course a few more sides in this battle, including music and books, but the idea is still perfectly valid.
What he didn't get into the details of is what happens one of those fiefdoms capitulates, declaring itself won over, and goes away. It's not a fun situation to be in, but we actually have plenty of examples of it, and these, more than anything else, should drive the discourse around and against DRM, in my opinion.
For some reasons, the main example of failed fiefdoms is to be found in books, and I lived through (and recounted) a few of those instances. For me personally, it all started four years ago, when I discovered Sony gave up on their LRF format and decided to adopt the "industry standard" ePub by supporting Adobe Digital Editions (ADEPT) DRM scheme on their devices. I was slow on the uptake, the announcement came two years earlier. For Sony, this meant tearing down their walled garden, even though they kept supporting the LRF format and their store for a while – they may even do still, I stopped following two years ago when I moved onto a Kindle – for the user it meant they were now free to buy books from a number of stores, including some publishers, bookstores with online presence and dedicated ebookstores.
But things didn't always go smoothly: two years later, WHSmith partnered with Kobo, and essentially handed the latter all their online ebook market. When I read the announcement I was actually happy, especially since I could not buy books off WHSmith any more as they started looking for UK billing addresses. Unfortunately it also meant that only a third of the books that I bought from WHSmith were going to be ported over to Kobo due to an extreme cock-up with global rights even to digital books. If I did not go and break the DRM off all my ebooks for the sake of it, I would have lost four books, having to buy them anew again. Given this was not for the seller going bankrupt but for a sell-out of their customers, it was not understandable that they refused to compensate people. Luckily, it did port The Gone-Away World which is one of my favourite books.
Fast forward another year, and the Italian bookstore LaFeltrinelli decided to go the same way, with a major exception: they decided they would keep users on both platforms — that way if you want to buy a digital version of a book you'll still buy it on the same website, but it'll be provided by Kobo and in your Kobo library. And it seems like they at least have a better deal regarding books' rights, as they seemed to have ported over most books anyway. But of course it did not work out as well as it should have been, throwing an error in my face and forcing me to call up Kobo (Italy) to have my accounts connected and the books ported.
The same year, I end up buying a Samsung Galaxy Note 10.1 2014 Edition, which is a pretty good tablet and has a great digitizer. Samsung ships Google Play in full (Store, Movies, Music, Books) but at the same time install its own App, Video, Music and Book store apps, it's not surprising. But it does not take six months for them to decide that it's not their greatest idea, in May this year, Samsung announced the turn down of their Music and Books stores — outside of South Korea at least. In this case there is no handover of the content to other providers, so any content bought on those platforms is just gone.
Not completely in vain; if you still have access to a Samsung device (and if you don't, well, you had no access to the content anyway), a different kind of almost-compensation kicks in: the Korean company partnered with Amazon of all bookstores — surprising given that they are behind the new "Nook Tablet" by Barnes & Noble. Beside a branded «Kindle for Samsung» app, they provide one out of a choice of four books every month — the books are taken from Amazon's KDP Select pool as far as I can tell, which is the same pool used as a base for the Kindle Owners' Lending Library and the Kindle Unlimited offerings; they are not great but some of them are enjoyable enough. Amazon is also keeping honest and does not force you to read the books on your Samsung device — I indeed prefer reading from my Kindle.
Now the question is: how do you loop back all this to multimedia? Sure books are entertaining but they are by definition a single media, unless you refer to the Kindle Edition of American Gods. Well, for me it's still the same problem of fiefdoms that Mike referred to; indeed every store used to be a walled garden for a long while, then Adobe came and conquered most with ePub and ADEPT — but then between Apple and their iBooks (which uses its own, incompatible DRM), and Amazon with the Kindle, the walls started crumbling down. Nowadays plenty of publishers allow you to buy the book, in ePub and usually many other formats at the same time, without DRM, because the publishers don't care which device you want to read your book on (a Kindle, a Kobo, a Nook, an iPad, a Sony Reader, an Android tablet …), they only want for you to read the book, and get hooked, and buy more books.
Somehow the same does not seem to work for video content, although it did work to an extent, for a while at least, with music. But this is a different topic.
The reason why I'm posting this right now is that just today I got an email from Samsung that they are turning down their video store too — now their "Samsung Hub" platform gets to only push you games and apps, unless you happen to live in South Korea. It's interesting to see how the battles between giants is causing small players to just get off the playing fields… but at the same time they bring their toys with them.
Once again, there is no compensation; if you rented something, watch it by the end of the year, if you bought something, sorry, you won't be able to access it after new year. It's a tough world. There is a lesson, somewhere, to be learnt about this.
So I was perusing Reddit—an activity that can be nothing more than a way to pass time, or, on occasion, can be rewarding—this evening, and found a picture of a tombstone that a father designed for his differently abled child who passed away untimely.
The picture certainly will resonant with anyone who has a child with a "disability." The image, though, was not the part of the post that really stuck out to me. No, there was a comment about it that really put the concept of death into perspective:
When your parents or elders die, you feel like you've lost a connection to the past. I've been told that losing a child is like living through the process of losing the future.
I agree with person who responded by saying that it is a "crushingly profound statement." The death of a child is not only untimely, but it is a chronological anomaly that simply shouldn't occur. We as humans recognise items in space and time that are out of place on a regular basis—they catch our attention. For instance, have you ever been watching a film about a time period of long ago and noticed something that wasn't available at that time (known as an anachronism, by the way)? The loss of a child is arguably the epitome of disturbances in the natural order of time.
For good measure, here is the full thread on Reddit, a link to the particular comment that I referenced, and the image hosted on imgur.
As a side note, the wonderful comment came from a user named Turkeybuzzard, which should be an indication to not pre-judge.
Today we did shutdown our now oldest running Gentoo Linux production server : ns2.
Obviously this machine was happily spreading our DNS records around the world but what's remarkable about it is that it has been doing so for 2717 straight days !
$ uptime 13:00:45 up 2717 days, 2:20, 1 user, load average: 0.13, 0.04, 0.01
As I mentioned when we did shutdown stabber, our beloved firewall, our company has been running Gentoo Linux servers in production for a long time now and we're always a bit sad when we have to power off one of them.
As usual, I want to take this chance to thank everyone contributing to Gentoo Linux ! Without our collective work, none of this would have been possible.
People who follow me probably know already that I'm not a native English speaker. Those who don't but will read this whole post will probably notice it by the end of it, just by my style, even if I were not to say it right at the start as I did.
It's not easy for me and it's often not easy for my teammates, especially so when they are native speakers. It's too easy for the both of us to underestimate or overestimate, at the same time sometimes, how much information we're conveying with a given phrase.
Something might sounds absolutely rude in English to a native speaker, but when I was forming my thought in my head it was intended to be much softer, even kind. Or the other way around: it might be actually quite polite in English, and my interpretation of it would be, to me, much ruder. And this is neither an easy or quick problem to solve, I have been working within English-based communities for a long while – this weblog is almost ten years old! – and still to this day the confusion is not completely gone.
It's interestingly sometimes easier to interact with other non-native speakers because we realize the disconnect, but other times is even harder because one or the other is not making the right amount of effort. I find it interestingly easier to talk with speakers of other Latin languages (French, Spanish, Portuguese), as the words and expressions are close enough that it can be easy to port them over — with a colleague and friend who's a native French speaker, we got to the point where it's sometimes faster to tell the other a word in our own language, rather than trying to go to English and back again; I promised him and other friends that I'll try to learn proper French.
It is not limited to language, culture is also connected: I found that there are many connections between Italian culture and Balkan, sometimes in niches that nobody would have expected it to creep up, such as rude gestures — the "umbrella gesture" seems to work just as good for Serbs as it does for Italians. This is less obvious when interacting with people exclusively online, but it is something useful when meeting people face to face.
I can only expect that newcomers – whether they are English speakers who have never worked closely with foreigners, or people whose main language is not English and who are doing their best to communicate in this language for the first time – to have a hard time.
This is not just a matter of lacking grammar or dictionary: Languages and societal customs are often interleaved and shape each other, so not understanding very well someone else's language may also mean not understanding their society and thus their point of view, and I would argue that points of view are everything.
I will make an example, but please remember I'm clearly not a philologist so I may be misspeaking, please be gentle with me. Some months ago, I've been told that English is a sexist language. While there wasn't a formal definition or reasoning for why stating that, I've been pointed at the fact you have to use "he" or "she" when talking about a third party.
I found this funny: not only in Italian you have to do so when talking about a third party, but you have to do so when talking about a second party (you) and even about a first party (me) — indeed, most adjectives and verbs require a gender. And while English can cop-out with the singular "they", this does not apply to Italian as easily. You can use a generic, plural "you", but the words still need a gender — it usually become feminine to match "your persons".
Because of the need for a gender in words, it is common to assume the male gender as a "default" in Italian; some documentation, especially paperwork from the public administration, will use the equivalent of "he/she" in the form of "signore/a", but it becomes cumbersome if you're writing something longer than a bank form, as every single word needs a different suffix.
I'm not trying to defend the unfortunate common mistake to assume a male gender when talking about "the user" or any other actor in a discussion, but I think it's a generally bad idea to assume that people have a perfect understanding of the language and thus assign maliciousness when there is simple naïve ignorance, as was the case with Lennart, systemd and the male pronouns. I know I try hard to use the singular "they", and I know I fall short of it too many times.
But the main point I'm trying to get across here is that yes, it's not easy, in this world that becomes more and more small, to avoid the shocking contrast of different languages and cultures. And it can't be just one side accommodating to this, we all have to make an effort, by understanding the other side's limit, and by brokering among sides that would be talking past each other anyway.
It's not easy, and it takes time, and effort. But I think it's all worth it.
It is surprisingly hard to find this fuse diagram online. I actually had the diagram in the glove box of my car but it is cold out and I didn't want to sit outside reading the manual. I went in trying to find the source of my rear window defroster failure and found the fuse blown and "melted" to the plastic. I broke the fuse when I removed it and then replaced it with a spare fuse. It looks like the previous owner used a 30A when it should have been 25A. Anyway, works like a charm now – ready for winter.
So, I've kind of taken over Request Tracker (bestpractical.com).
Initially I took it because I'm interested in using RT at work to take track customer service emails. All I did at the time was bump the version and remove old, insecure versions from the tree.
However, as I've finally gotten around to working on getting it setup, I've discovered there were a lot of issues that had gone unreported.
The intention is for RT to run out of its virtual host root, like /var/www/localhost/rt-4.2.9/bin/rt and configured by /var/www/localhost/rt-4.2.9/etc/RT_SiteConfig.pm, and for it to reference any supplementary packages with ${VHOST_ROOT} as its root. However, because of a broken install process and a broken hook script used by webapp-config that didn't happen. Further, the rt_apache.conf included by us was outdated by a few years, too, which in itself isn't a bad thing, except that it was wrong for RT 4+.
I spent much longer than I care to admit trying to figure out why my settings weren't sticking when I edited RT_SiteConfig.pm. I was trying to run RT under its own path rather than on a subdomain, but Set($WebPath, '/rt') wasn't doing what it should.
It also complained about not being able to write to /usr/share/webapps/rt/rt-4.2.9/data/mason_data/obj, which clearly wasn't right.
Once I tried moving RT_SiteConfig.pm to /usr/share/webapps/rt/rt-4.2.9/etc/, and chmod and chown on ../data/mason_data/obj, everything worked as it should.
Knowing this was wrong and that it would prevent anyone using our package from having multiple installation, aka vhosts, I set out to fix it.
It was a descent into madness. Things I expected to happen did not. Things that shouldn't have been a problem were. Much of the trouble I had circled around webapp-config and webapp.eclass.
But, I prevailed, and now you can really have multiple RT installations side-by-side. Also, I've added an article (wiki.gentoo.org) to our wiki with updated instructions on getting RT up and running.
Caveat: I didn't use FastCGI, so that part may be wrong still, but mod_perl is good to go.
Nach mehreren Wochen downtime — primär durch mich verschuldet — ist rsync1.de.gentoo.org nun wieder online.
Wie vorher wird das komplette Repository aus einer RAM disk ausgeliefert, daher ist der Mirror relativ flott.
# rsync --list-only rsync://rsync1.de.gentoo.org/gentoo-portage/ drwxr-xr-x 3,480 2014/11/16 16:01:19 . -rw-r--r-- 121 2014/01/01 01:31:01 header.txt -rw-r--r-- 3,658 2014/08/18 21:01:02 skel.ChangeLog -rw-r--r-- 8,119 2014/08/30 12:01:02 skel.ebuild -rw-r--r-- 1,231 2014/08/18 21:01:02 skel.metadata.xml drwxr-xr-x 860 2014/11/16 16:01:02 app-accessibility drwxr-xr-x 4,800 2014/11/16 16:01:03 app-admin drwxr-xr-x 100 2014/11/16 16:01:03 app-antivirus [..] drwxr-xr-x 1,240 2014/11/16 16:01:21 x11-wm drwxr-xr-x 340 2014/11/16 16:01:21 xfce-base drwxr-xr-x 1,340 2014/11/16 16:01:21 xfce-extra
Die Hardware darunter ist gesponsort von Manitu.
I would like to introduce you to Gambit, a rather young Qt-based chess UI with excellent usability and its very own engine.
emerge -av games-board/gambit
as usual.
|1) or 2) true
|1) false, 2) false
|"dev-lang/perl:=" needed in RDEPEND and most likely also DEPEND
|"dev-lang/perl" needed in RDEPEND, maybe also in DEPEND
|no need to do anything
|GENTOO_DEPEND_ON_PERL_SUBSLOT=no possible before inherit
|GENTOO_DEPEND_ON_PERL_SUBSLOT=yes needed before inherit
|no need to do anything
Maxim asked me to to check a stream from a security camera that he could not decode with avconv without forcing the format to
Since it is served as http the first step had been checking the mime type. Time to use
curl -I.
# curl -I "http://host/some.cgi?user=admin&pwd=pwd" | grep Content-Type
Interesting enough it is a
Content-Type: multipart/x-mixed-replace;boundary=object-ipcamera
Basically the cgi sends a jpeg images one after the other, we even have a (old and ugly) muxer for it!
Time to write a demuxer.
We already have some documentation on how to write a demuxer, but it is not complete so this blogpost will provide an example.
Libav code is quite object oriented: every component is a C structure containing a description of it and pointers to a set of functions and there are fixed pattern to make easier to make new code fit in.
Every major library has an
all${components}.c in which the components are registered to be used. In our case we talk about libavformat so we have
The components are built according to
CONFIG_${name}_${component} variables generated by configure. The actual code reside in the
${component} directory with a pattern such as
${name}.c or
${name}enc.c if both demuxer and muxer are available.
The code can be split in multiple files if it starts growing to an excess of 500-1000 LOCs.
We have some
REGISTER_ macros that abstract some logic to make every component selectable at configure time since in Libav you can enable/disable every muxer, demuxer, codec, IO/protocol from configure.
We had already have a muxer for the format.
REGISTER_MUXER (MPJPEG, mpjpeg);
Now we register both in a single line:
REGISTER_MUXDEMUX(MPJPEG, mpjpeg);
all${components} files are parsed by configure to generate the appropriate Makefile and C definitions. The next run we'll get a new
CONFIG_MPJPEG_DEMUXER variable in
config.mak and
Now we can add to
libavformat/Makefile a line like
OBJS-$(CONFIG_MPJPEG_DEMUXER) += mpjpegdec.o
and put our
mpjpegdec.c in
libavformat and we are ready to write some code!
Usually I start putting down a skeleton file with the bare minimum:
The AVInputFormat and the core
_read_header and
_read_packet callbacks.
#include "avformat.h"
static int ${name}_read_probe(AVProbeData *p)
return 0;
static int ${name}_read_header(AVFormatContext *s)
return AVERROR(ENOSYS);
static int ${name}_read_packet(AVFormatContext *s, AVPacket *pkt)
return AVERROR(ENOSYS);
AVInputFormat ff_${name}_demuxer = {
.name = "${name}",
.long_name = NULL_IF_CONFIG_SMALL("Longer ${name} description"),
.read_probe = ${name}_read_probe,
.read_header = ${name}_read_header,
.read_packet = ${name}_read_packet,
I make so that all the functions return a no-op value.
This function will be called by the av_probe_input functions, it receives some probe information in the form of a buffer. The function return a score between 0 and 100;
AVPROBE_SCORE_MIME and
AVPROBE_SCORE_EXTENSION are provided to make more evident what is the expected confidence. 0 means that we are sure that the probed stream is not parsable by this demuxer.
This function will be called by avformat_open_input. It reads the initial format information (e.g. number and kind of streams) when available, in this function the initial set of streams should be mapped with avformat_new_stream. Must return
0 on success. The skeleton is made to return
ENOSYS so it can be run and just exit cleanly.
This function will be called by av_read_frame. It should return an AVPacket containing demuxed data as contained in the bytestream. It will be parsed and collated (or splitted) to a frame-worth amount of data by the optional parsers. Must return
0 on success. The skeleton again returns
Now let's implement the mpjpeg support! The format in itself is quite simple:
- a boundary line starting with
- a
Content-Type line stating
- a
Content-Length line with the actual buffer length.
- the jpeg data
We just want to check if the
Content-Type is what we expect basically, so we just
go over the lines (
\n\r-separated) and check if there is a tag
Content-Type with a value
static int get_line(AVIOContext *pb, char *line, int line_size)
int i, ch;
char *q = line;
for (i = 0; !pb->eof_reached; i++) {
ch = avio_r8(pb);
if (ch == 'n') {
if (q > line && q[-1] == 'r')
*q = '';
return 0;
} else {
if ((q - line) < line_size - 1)
*q++ = ch;
if (pb->error)
return pb->error;
return AVERROR_EOF;
static int split_tag_value(char **tag, char **value, char *line)
char *p = line;
while (*p != '' && *p != ':')
if (*p != ':')
return AVERROR_INVALIDDATA;
*p = '';
*tag = line;
while (av_isspace(*p))
*value = p;
return 0;
static int check_content_type(char *line)
char *tag, *value;
int ret = split_tag_value(&tag, &value, line);
if (ret < 0)
return ret;
if (av_strcasecmp(tag, "Content-type") ||
av_strcasecmp(value, "image/jpeg"))
return AVERROR_INVALIDDATA;
return 0;
static int mpjpeg_read_probe(AVProbeData *p)
AVIOContext *pb;
char line[128] = { 0 };
int ret;
pb = avio_alloc_context(p->buf, p->buf_size, 0, NULL, NULL, NULL, NULL);
if (!pb)
return AVERROR(ENOMEM);
while (!pb->eof_reached) {
ret = get_line(pb, line, sizeof(line));
if (ret < 0)
ret = check_content_type(line);
if (!ret)
return AVPROBE_SCORE_MAX;
return 0;
Here we are using avio to be able to reuse
get_line later.
The format is pretty much header-less, we just check for the boundary for now and
set up the minimum amount of information regarding the stream: media type, codec id and frame rate. The boundary by specification is less than 70 characters with
-- as initial marker.
static int mpjpeg_read_header(AVFormatContext *s)
MPJpegContext *mp = s->priv_data;
AVStream *st;
char boundary[70 + 2 + 1];
int ret;
ret = get_line(s->pb, boundary, sizeof(boundary));
if (ret < 0)
return ret;
if (strncmp(boundary, "--", 2))
return AVERROR_INVALIDDATA;
st = avformat_new_stream(s, NULL);
st->codec->codec_type = AVMEDIA_TYPE_VIDEO;
st->codec->codec_id = AV_CODEC_ID_MJPEG;
avpriv_set_pts_info(st, 60, 1, 25);
return 0;
Even this function is quite simple, please note that
AVFormatContext provides an
AVIOContext. The bulk of the function boils down to reading the size of the frame,
allocate a packet using
av_new_packet and write down if using
static int parse_content_length(char *line)
char *tag, *value;
int ret = split_tag_value(&tag, &value, line);
long int val;
if (ret < 0)
return ret;
if (av_strcasecmp(tag, "Content-Length"))
return AVERROR_INVALIDDATA;
val = strtol(value, NULL, 10);
if (val == LONG_MIN || val == LONG_MAX)
return AVERROR(errno);
if (val > INT_MAX)
return AVERROR(ERANGE);
return val;
static int mpjpeg_read_packet(AVFormatContext *s, AVPacket *pkt)
char line[128];
int ret, size;
ret = get_line(s->pb, line, sizeof(line));
if (ret < 0)
return ret;
ret = check_content_type(line);
if (ret < 0)
return ret;
ret = get_line(s->pb, line, sizeof(line));
if (ret < 0)
return ret;
size = parse_content_length(line);
if (size < 0)
return size;
ret = get_line(s->pb, line, sizeof(line));
if (ret < 0)
goto fail;
ret = av_new_packet(pkt, size);
if (ret < 0)
return ret;
ret = avio_read(s->pb, pkt->data, size);
if (ret < 0)
goto fail;
ret = get_line(s->pb, line, sizeof(line));
if (ret < 0)
goto fail;
// Consume the boundary marker
ret = get_line(s->pb, line, sizeof(line));
if (ret < 0)
goto fail;
return ret;
return ret;
For now I walked you through on the fundamentals, hopefully next week I'll show you some additional features I'll need to implement in this simple demuxer to make it land in Libav: AVOptions to make possible overriding the framerate and some additional code to be able to do without
Content-Length and just use the boundary line.
PS: wordpress support for syntax highlight is quite subpar, if somebody has a blog engine that can use pygments or equivalent please tell me and I'd switch to it.
After my challenge with the fused multiply-add instructions I managed to find some time to write a new test utility. It's written ad hoc for unpaper but it can probably be used for other things too. It's trivial and stupid but it got the job done.
What it does is simple: it loads both a golden and a result image files, compares the size and format, and then goes through all the bytes to identify how many differences are there between them. If less than 0.1% of the image surface changed, it consider the test a pass.
It's not a particularly nice system, especially as it requires me to bundle some 180MB of golden files (they compress to just about 10 MB so it's not a big deal), but it's a strict improvement compared to what I had before, which is good.
This change actually allowed me to explore one change that I abandoned before because it resulted in non-pixel-perfect results. In particular, unpaper now uses single-precision floating points all over, rather than doubles. This is because the slight imperfection caused by this change are not relevant enough to warrant the ever-so-slight loss in performance due to the bigger variables.
But even up to here, there is very little gain in performance. Sure some calculation can be faster this way, but we're still using the same set of AVX/FMA instructions. This is unfortunate, unless you start rewriting the algorithms used for searching for edges or rotations, there is no gain to be made by changing the size of the code. When I converted unpaper to use libavcodec, I decided to make the code simple and as stupid as I could make it, as that meant I could have a baseline to improve from, but I'm not sure what the best way to improve it is, now.
I still have a branch that uses OpenMP for the processing, but since most of the filters applied are dependent on each other it does not work very well. Per-row processing gets slightly better results but they are really minimal as well. I think the most interesting parallel processing low-hanging fruit would be to execute processing in parallel on the two pages after splitting them from a single sheet of paper. Unfortunately, the loops used to do that processing right now are so complicated that I'm not looking forward to touch them for a long while.
I tried some basic profile-guided optimization execution, just to figure out what needs to be improved, and compared with
codiff a proper release and a PGO version trained after the tests. Unfortunately the results are a bit vague and it means I'll probably have to profile it properly if I want to get data out of it. If you're curious here is the output when using
rbelf-size -D on the
unpaper binary when built normally, with profile-guided optimisation, with link-time optimisation, and with both profile-guided and link-time optimisation:
% rbelf-size -D ../release/unpaper ../release-pgo/unpaper ../release-lto/unpaper ../release-lto-pgo/unpaper
exec data rodata relro bss overhead allocated filename
34951 1396 22284 0 11072 3196 72899 ../release/unpaper
+5648 +312 -192 +0 +160 -6 +5922 ../release-pgo/unpaper
-272 +0 -1364 +0 +144 -55 -1547 ../release-lto/unpaper
+7424 +448 -1596 +0 +304 -61 +6519 ../release-lto-pgo/unpaper
It's unfortunate that GCC does not give you any diagnostic on what it's trying to do achieve when doing LTO, it would be interesting to see if you could steer the compiler to produce better code without it as well.
Anyway, enough with the microptimisations for now. If you want to make unpaper faster, feel free to send me pull requests for it, I'll be glad to take a look at them!
If the hostname is currently blank, (null) or localhost, or force_hostname is YES or TRUE or 1 then dhcpcd sets the hostname to the one supplied by the DHCP server.Guess what. Now my hostname is 192.168.0.7, I mean 192.168.0.192.168.0.7, err...
The council addressed a number of issues this month. The change with the biggest long-term significance was clearing the way to proceed with the git migration once infra is ready. This included removing changelogs from future git commits, removing cvs headers, and simplifying our news repository format. The infra and git migration projects will coordinate the actual migration hopefully in the not-so-distant future.
The council also endorsed getting rid of herds, but acknowledged that there are some details that need to be worked out before pulling the plug. The bikeshedding was moved back to the lists so all could share in the fun.
There are still some concerns with the games team. The council decided to give the team more time to sort things out internally before interfering. It was acknowledged that most of the serious issues were already resolved with the decision to allow anybody to elect to make their packages a part of the games herd or not. Some QA concerns with some games were brought up, but it was felt that this is best dealt with on a per-package basis with QA/treecleaners and that games shouldn't receive any special treatment one way or the other.
Other decisions include removing einstall from EAPI6, and approving GLEP64 (VDB caching / API). There was also a status update on multilib (nearly done), and migrating project pages to the wiki (sadly we can't just get rid of unmigrated projects like the x86 and amd64 arches).
(by Ian Stakenvicius)
On November 7th, packages inheriting python-single-r1 got a whole lot easier for end-users to manage.
It used to be that any package supporting just one Python required it to have a python_single_target_* USE-flag set to choose it, even if the package was only compatible with one Python in the first place. Since November 7th, if a package is only compatible with a single supported Python version (say, python-2.7), then it no longer uses python_single_target_* use flags and relies instead on that implementation being enabled in PYTHON_TARGETS.
The most visible change seen from this is package rebuilds from removal of a lot of PYTHON_SINGLE_TARGET flags, especially on python-2.7-only packages. However, the removal of these flags also means that setting PYTHON_SINGLE_TARGET to something other than python2_7 no longer needs all of those packages to be listed in package.use.
Portage users are also likely to notice that exceptions to PYTHON_SINGLE_TARGET that would require package.use changes are now also be calculated properly by –autounmask, instead of solely being reported as an illegible REQUIRED_USE error.
Gentoo is made up of 243 active developers, of which 39 are currently away.
Gentoo has recruited a total of 804 developers since its inception.
This section summarizes the current state of the Gentoo ebuild tree.
|% of Packages
The following GLSAs have been released by the Security Team
|perl-core/Locale-Maketext (and 1 more)
|Perl, Perl Locale-Maketext module: Multiple vulnerabilities
|Bash: Multiple vulnerabilities
|06 Oct 2014
|08 Oct 2014
|08 Oct 2014
|11 Oct 2014
|13 Oct 2014
|14 Oct 2014
|15 Oct 2014
|18 Oct 2014
|29 Oct 2014
|30 Oct 2014
|30 Oct 2014
|01 Oct 2014
|02 Oct 2014
|02 Oct 2014
|04 Oct 2014
|05 Oct 2014
|05 Oct 2014
|05 Oct 2014
|05 Oct 2014
|06 Oct 2014
|06 Oct 2014
|07 Oct 2014
|08 Oct 2014
|09 Oct 2014
|10 Oct 2014
|10 Oct 2014
|10 Oct 2014
|11 Oct 2014
|11 Oct 2014
|11 Oct 2014
|11 Oct 2014
|11 Oct 2014
|11 Oct 2014
|11 Oct 2014
|11 Oct 2014
|11 Oct 2014
|11 Oct 2014
|12 Oct 2014
|12 Oct 2014
|12 Oct 2014
|13 Oct 2014
|13 Oct 2014
|13 Oct 2014
|14 Oct 2014
|14 Oct 2014
|14 Oct 2014
|14 Oct 2014
|14 Oct 2014
|14 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|15 Oct 2014
|16 Oct 2014
|16 Oct 2014
|16 Oct 2014
|16 Oct 2014
|16 Oct 2014
|16 Oct 2014
|16 Oct 2014
|17 Oct 2014
|18 Oct 2014
|18 Oct 2014
|18 Oct 2014
|18 Oct 2014
|18 Oct 2014
|19 Oct 2014
|19 Oct 2014
|19 Oct 2014
|19 Oct 2014
|19 Oct 2014
|19 Oct 2014
|19 Oct 2014
|20 Oct 2014
|21 Oct 2014
|21 Oct 2014
|21 Oct 2014
|21 Oct 2014
|22 Oct 2014
|22 Oct 2014
|22 Oct 2014
|22 Oct 2014
|23 Oct 2014
|23 Oct 2014
|23 Oct 2014
|24 Oct 2014
|24 Oct 2014
|24 Oct 2014
|24 Oct 2014
|24 Oct 2014
|25 Oct 2014
|25 Oct 2014
|25 Oct 2014
|25 Oct 2014
|25 Oct 2014
|25 Oct 2014
|26 Oct 2014
|26 Oct 2014
|27 Oct 2014
|27 Oct 2014
|27 Oct 2014
|27 Oct 2014
|27 Oct 2014
|27 Oct 2014
|27 Oct 2014
|28 Oct 2014
|28 Oct 2014
|29 Oct 2014
|29 Oct 2014
|29 Oct 2014
|29 Oct 2014
|29 Oct 2014
|29 Oct 2014
|29 Oct 2014
|30 Oct 2014
|30 Oct 2014
|31 Oct 2014
|31 Oct 2014
|31 Oct 2014
|31 Oct 2014
The Gentoo community uses Bugzilla to record and track bugs, notifications, suggestions and other interactions with the development team.
The following tables and charts summarize the activity on Bugzilla between 01 October 2014 and 01 November 2014. Not fixed means bugs that were resolved as NEEDINFO, WONTFIX, CANTFIX, INVALID or UPSTREAM.
|Bug Activity
|Not fixed
The following table outlines the teams and developers with the most bugs resolved during this period
|Bug Count
|Gentoo Linux Gnome Desktop Team
|Gentoo Perl team
|Gentoo Games
|Gentoo KDE team
|Gentoo's Team for Core System packages
|Netmon Herd
|Python Gentoo Team
|PHP Bugs
|Gentoo Toolchain Maintainers
The developers and teams who have been assigned the most bugs during this period are as follows.
|Bug Count
|Gentoo Linux bug wranglers
|Gentoo Linux Gnome Desktop Team
|Gentoo's Team for Core System packages
|Gentoo Security
|Gentoo KDE team
|Python Gentoo Team
|Gentoo Games
|Gentoo Perl team
|Default Assignee for New Packages
Send us your favorite Gentoo script or tip at firstname.lastname@example.com
Interested in helping out? The GMN relies on volunteers and members of the community for content every month. If you are interested in writing for the GMN or thinking of another way to contribute, please send an e-mail to email@example.com.
Please head over to this forum post.
Today, 11 November, is an interesting holiday in the United States. It is the day in which we honour those individuals who have served in the armed forces and have defended their country. I say that it is an interesting holiday because I am torn on how I feel about the entire concept. On one hand, I am incredibly grateful for those people that have fought to defend the principles and freedoms on which the United States was founded. However, the fight itself is one that I cannot condone.
Threats to freedom in any nation are brought about by political groups, and should be handled in a political manner. I understand that my viewpoint here is one of pseudoutopian cosmography, but it is one that I hope will become more and more realistic as both time and humanity march onward. The "wars" should be fought by national leaders, and done so via discussion and debate; not by citizens (military or civilian) via guns, bombs, or other weaponry.
I also understand that there will be many people who disagree (in degrees that result in emotions ranging from mild irritation to infuriated hostility) with my viewpoint, and that is completely fine. Again, my dilemma comes from being simultaneously thankful for those individuals who have given their all to defend "freedom" (whatever concept that word may represent) and sorrowful that they were the ones that had to give anything at all. These men and women had to leave their families knowing that they may never return to them; knowing that they may die trying to defend something that shouldn't be challenged in the first place—human freedoms.
Let us not forget a quote by former President of the United States, John F. Kennedy who stated that "mankind must put an end to war before war puts an end to mankind."
As you may recall, I was looking for a dedicated PyPy maintainer for quite some time. Sadly, all the people who helped (and who I'd like to thank a lot) ended up lacking time soon enough. So finally I've decided to look into the hacks reducing build-time memory use and take care of the necessary ebuild and packaging work myself.
So first of all, you may notice that the new PyPy (source-code) ebuilds have a new USE flag called low-memory. When this flag is enabled, the translation process is done using PyPy with some memory-reducing adjustments suggested by upstream. The net result is that it finally is possible to build PyPy with 3.5G RAM (on amd64) and 1G of swap (the latter being used the compiler is spawned and memory used during translation is no longer necessary), at a cost of slightly increased build time.
As noted above, the low-memory option requires using PyPy to perform the translation. So while having to enforce that, I went a bit further and made the ebuild default to using PyPy whenever available. In fact, even for a first PyPy build you are recommended to install dev-python/pypy-bin first and let the ebuild use it to bootstrap your own PyPy.
Next, I have cleaned up the ebuilds a bit and enforced more consistency. Changing maintainers and binary package builders have resulted in the ebuilds being a bit inconsistent. Now you can finally expect pypy-bin to install exactly the same set of files as source-built pypy.
I have also cleaned up the remaining libpypy-c symlinks. The library is not packaged upstream currently, and therefore has no proper public name. Using libpypy-c.so is just wrong, and packages can't reliably refer to that. I'd rather wait with installing it till there's some precedence in renaming. The shared library is still built but it's kept inside the PyPy home directory.
All those changes were followed by a proper version bump to 2.4.0. While you still may have issues upgrading PyPy, Zac already committed a patch to Portage and the next release should be able to handle PyPy upgrades seamlessly. I have also built all the supported binary package variants, so you can choose those if you don't want to spend time building PyPy.
Finally, I have added the ebuilds for PyPy 3. They are a little bit more complex than regular PyPy, especially because the build process and some of the internal modules still require Python 2. Sadly, PyPy 3 is based on Python 3.2 with small backports, so I don't expect package compatibility much greater than CPython 3.2 had.
If you want to try building some packages with PyPy 3, you can use the convenience PYTHON_COMPAT_OVERRIDE hack:
PYTHON_COMPAT_OVERRIDE='pypy3' emerge -1v mypackage
Please note that it is only a hack, and as such it doesn't set proper USE flags (PYTHON_TARGETS are simply ignored) or enforce dependencies.
If someone wants to help PyPy on Gentoo a bit, there are still unsolved issues needing a lot of specialist work. More specifically:
The third week of October was quite action-packed, with a whole bunch of conferences happening in Düsseldorf. The Linux audio developer community as well as the PulseAudio developers each had a whole day of discussions related to a wide range of topics. I'll be summarising the events of the PulseAudio mini summit day here. The discussion was split into two parts, the first half of the day with just the current core developers and the latter half with members of the community participating as well.
I'd like to thank the Linux Foundation for sparing us a room to carry out these discussions — it's fantastic that we are able to colocate such meetings with a bunch of other conferences, making it much easier than it would otherwise be for all of us to converge to a single place, hash out ideas, and generally have a good time in real life as well!
With a whole day of discussions, this is clearly going to be a long post, so you might want to grab a coffee now. :)
We have a few blockers for 6.0, and some pending patches to merge (mainly HSP support). Once this is done, we can proceed to our standard freeze → release candidate → stable process.
For simplifying packaging, it would be nice to be able to build all the available BlueZ module backends in one shot. There wasn't much opposition to this idea, and David (Henningsson) said he might look at this. (as I update this before posting, he already has)
We briefly discussed plans around the recently introduced shared ringbuffer channel code for communication between PulseAudio clients and the server. We talked about the performance benefits, and future plans such as direct communication between the client and server-side I/O threads.
Tanu (Kaskinen) has a long-standing set of patches to add a generic routing framework to PulseAudio, developed by notably Jaska Uimonen, Janos Kovacs, and other members of the Tizen IVI team. This work adds a set of new concepts that we've not been entirely comfortable merging into the core. To unblock these patches, it was agreed that doing this work in a module and using a protocol extension API would be more beneficial. (Tanu later did a demo of the CLI extensions that have been made for the new routing concepts)
As a consequence of the discussion around the routing framework, David mentioned that he'd like to take forward Colin's priority list work in the mean time. Based on our discussions, it looked like it would be possible to extend module-device-manager to make it port aware and get the kind functionality we want (the ability to have a priority-order list of devices). David was to look into this.
Relatedly, we discussed the need to export the PA internal headers to allow externally built modules. We agreed that this would be okay to have if it was made abundantly clear that this API would have absolutely no stability guarantees, and is mostly meant to simplify packaging for specialised distributions.
Which led us to the other bit of infrastructure required to write modules more easily — making our protocol extension mechanism more generic. Currently, we have a static list of protocol extensions in our core. Changing this requires exposing our pa_tagstruct structure as public API, which we haven't done. If we don't want to do that, then we would expose a generic "throw this blob across the protocol" mechanism and leave it to the module/library to take care of marshalling/unmarshalling.
Alexander shared a number of his findings about resampler quality on PulseAudio, vs. those found on Windows and Mac OS. Some questions were asked about other parameters, such as relative CPU consumption, etc. There was also some discussion on how to try to carry this work to a conclusion, but no clear answer emerged.
It was also agreed on the basis of this work that support for libsamplerate and ffmpeg could be phased out after deprecation.
The discussion came around to the possibility of having a mode where (if the hardware supports it), PulseAudio just plays out samples without resampling, conversion, etc. This has been brought up in the past for "audiophile" use cases where the card supports 88.2/96 kHZ and higher sample rates.
No objections were raised to having such a mode — I'd like to take this up at some point of time.
Alexander has some code for filtering low frequencies for the LFE channel, currently as a virtual sink, that could eventually be integrated into the core.
David raised a question about the current status of rtkit and whether it needs to exist, and if so, where. Lennart brought up the fact that rtkit currently does not work on systemd+cgroups based setups (I don't seem to have why in my notes, and I don't recall off the top of my head).
The conclusion of the discussion was that some alternate policy method for deciding RT privileges, possibly within systemd, would be needed, but for now rtkit should be used (and fixed!)
Discussions came up about the possibility of using kdbus and/or memfd for the PulseAudio transport. This is interesting to me, there doesn't seem to be an immediately clear benefit over our SHM mechanism in terms of performance, and some work to evaluate how this could be used, and what the benefit would be, needs to be done.
David has now submitted patches for controls that affect multiple outputs (such as "Headphone+LO"). These are currently being discussed.
Tanu would like to add code to support collecting audio streams into "audio groups" to apply collective policy to them. I am supposed to help review this, and Colin mentioned that module-stream-restore already uses similar concepts.
Tanu proposed the addition of new objects to represent streams and objects. There didn't seem to be consensus on adding these, but there was agreement of a clear need to consolidate common code from sink-input/source-output and sink/source implementations. The idea was that having a common parent object for each pair might be one way to do this. I volunteered to help with this if someone's taking it up.
Alexander brough up the need for a filter API in PulseAudio, and this is something I really would like to have. I am supposed to sketch out an API (though implementing this is non-trivial and will likely take time).
David plans to see if we can use profile availability to help determine when an HDMI device is actually available.
The usability of flat-volumes for browser use cases (where the volume of streams can be controlled programmatically) was discussed, and my patch to allow optional opt-out by a stream from participating in flat volumes came up. Tanu and I are to continue the discussion already on the mailing list to come up with a solution for this.
Alexander raised concerns about the quality of rewinding code in some of our filter modules. The agreement was that we needed better documentation on handling rewinds, including how to explicitly not allow rewinds in a sink. The example virtual sink/source code also needs to be adjusted accordingly.
Wim Taymans' work on adding back HSP support to PulseAudio came up. Since the meeting, I've reviewed and merged this code with the change we want. Speaking to Luiz Augusto von Dentz from the BlueZ side, something we should also be able to add back is for PulseAudio to act as an HSP headset (using the same approach as for HSP gateway support).
Takashi Iwai raised a question about what a good way to run PA in a container was. The suggestion was that a tunnel sink would likely be the best approach.
Based on discussion from the previous day at the Linux Audio mini-summit, I'm supposed to look at the possibility of consolidating the various mixer configuration formats we currently have to deal with (primarily UCM and its implementations, and Android's XML format).
(thanks to Tanu, David and Peter for reviewing this)
I just finished updating 102 packages. The change? Removing the following from the ebuilds:
DEPEND="selinux? ( sec-policy/selinux-${packagename} )"
In the past, we needed this construction in both DEPEND and RDEPEND. Recently however, the SELinux eclass got updated with some logic to relabel files after the policy package is deployed. As a result, the DEPEND variable no longer needs to refer to the SELinux policy package.
This change also means that for those moving from a regular Gentoo installation to an SELinux installation will have much less packages to rebuild. In the past, getting
USE="selinux" (through the SELinux profiles) would rebuild all packages that have a DEPEND dependency to the SELinux policy package. No more – only packages that depend on the SELinux libraries (like
libselinux) or utilities rebuild. The rest will just pull in the proper policy package.
Good news, everyone! I'm finally rid of Windows.
A couple weeks ago my Windows installation corrupted itself on the 5 minute trip home from the community theatre. I didn't command it to go to sleep, I just unplugged it and closed the lid. Somehow, it managed to screw up its startup files, and the restore process didn't do what it was supposed to so I was greeted with a blank screen. No errors. Just staring into the void.
I've been using Windows as the sole OS on this machine with Gentoo running in VirtualBox for various reasons related to minor annoyances of unsupported hardware, but as I needed a working machine sooner rather than later and the only tools I could find to solve my Windows problem appeared to be old, defunct, and/or suspicious, I downloaded an ISO of SystemRescueCd (www.sysresccd.org) and installed Gentoo in the sliver of space left on the drive.
There were only two real reasons why I was intent on keeping Windows: Netflix (netflix.com) and EVE Online (eveonline.com). I intended to get Windows up and running once the show was over at the theatre, but then I read about Netflix being supported in Linux (www.mpagano.com). That left me with just one reason to keep Windows: EVE. I turned to Wine (www.winehq.org) and discovered reports of it running EVE quite well (appdb.winehq.org). I also learned that the official Mac OS release of EVE runs on Cider (www.transgaming.com), which is based on Wine.
I had another hitch: I chose the no-multilib stage3 for that original sliver thinking I wouldn't be running anything other than 64 bit software, and drive space was at a premium. EVE Online is 32 bit.
So I had to begin my adventure with switching to multilib. This didn't involve me reinstalling Gentoo thanks to a handy, but unsupported and unofficial, guide (jkroon.blogs.uls.co.za) by Jaco Kroon.
As explained on Multilib System without emul-linux Packages (wiki.gentoo.org), I decided it's better to build my own 32 bit library. So, the next step is to mask the emulation packages:
# /etc/portage/package.mask app-emulation/emul-linux-x86-*
Because I didn't want to build a 32 bit variant for everything on my system, I iterated through what Portage wanted and marked several packages to build their 32 bit variant via use flags. This is what I wound up with:
# /etc/portage/package.use app-arch/bzip2 abi_x86_32 app-emulation/wine mono abi_x86_32 dev-libs/elfutils static-libs abi_x86_32 dev-libs/expat abi_x86_32 dev-libs/glib abi_x86_32 dev-libs/gmp abi_x86_32 dev-libs/icu abi_x86_32 dev-libs/libffi abi_x86_32 dev-libs/libgcrypt abi_x86_32 dev-libs/libgpg-error abi_x86_32 dev-libs/libpthread-stubs abi_x86_32 dev-libs/libtasn1 abi_x86_32 dev-libs/libxml2 abi_x86_32 dev-libs/libxslt abi_x86_32 dev-libs/nettle abi_x86_32 dev-util/pkgconfig abi_x86_32 media-libs/alsa-lib abi_x86_32 media-libs/fontconfig abi_x86_32 media-libs/freetype abi_x86_32 media-libs/glu abi_x86_32 media-libs/libjpeg-turbo abi_x86_32 media-libs/libpng abi_x86_32 media-libs/libtxc_dxtn abi_x86_32 media-libs/mesa abi_x86_32 media-libs/openal abi_x86_32 media-sound/mpg123 abi_x86_32 net-dns/avahi abi_x86_32 net-libs/gnutls abi_x86_32 net-print/cups abi_x86_32 sys-apps/dbus abi_x86_32 sys-devel/llvm abi_x86_32 sys-fs/udev gudev abi_x86_32 sys-libs/gdbm abi_x86_32 sys-libs/ncurses abi_x86_32 sys-libs/zlib abi_x86_32 virtual/glu abi_x86_32 virtual/jpeg abi_x86_32 virtual/libffi abi_x86_32 virtual/libiconv abi_x86_32 virtual/libudev abi_x86_32 virtual/opengl abi_x86_32 virtual/pkgconfig abi_x86_32 x11-libs/libX11 abi_x86_32 x11-libs/libXau abi_x86_32 x11-libs/libXcursor abi_x86_32 x11-libs/libXdamage abi_x86_32 x11-libs/libXdmcp abi_x86_32 x11-libs/libXext abi_x86_32 x11-libs/libXfixes abi_x86_32 x11-libs/libXi abi_x86_32 x11-libs/libXinerama abi_x86_32 x11-libs/libXrandr abi_x86_32 x11-libs/libXrender abi_x86_32 x11-libs/libXxf86vm abi_x86_32 x11-libs/libdrm abi_x86_32 x11-libs/libvdpau abi_x86_32 x11-libs/libxcb abi_x86_32 x11-libs/libxshmfence abi_x86_32 x11-proto/damageproto abi_x86_32 x11-proto/dri2proto abi_x86_32 x11-proto/dri3proto abi_x86_32 x11-proto/fixesproto abi_x86_32 x11-proto/glproto abi_x86_32 x11-proto/inputproto abi_x86_32 x11-proto/kbproto abi_x86_32 x11-proto/presentproto abi_x86_32 x11-proto/randrproto abi_x86_32 x11-proto/renderproto abi_x86_32 x11-proto/xcb-proto abi_x86_32 python_targets_python3_4 x11-proto/xextproto abi_x86_32 x11-proto/xf86bigfontproto abi_x86_32 x11-proto/xf86driproto abi_x86_32 x11-proto/xf86vidmodeproto abi_x86_32 x11-proto/xineramaproto abi_x86_32 x11-proto/xproto abi_x86_32
Now emerge both Wine — the latest and greatest of course — and the questionable library so textures will be rendered:
emerge -av media-libs/libtxc_dxtn =app-emulation/wine-1.7.29
You may get some messages along the lines of:
emerge: there are no ebuilds to satisfy ">=sys-libs/zlib-1.2.8-r1".
This was a bit of a head scratcher for me. I have syslibs/zlib-1.2.8-r1 installed. I didn't have to accept its keyword. It's already stable! I haven't really looked into why, but you have to accept its keyword to press forward:
# echo '=sys-libs/zlib-1.2.8-r1' >> /etc/portage/package.accept_keywords
You'll have to do the above several times for other packages when you try to emerge Wine. Most of the time the particular version it wants is something you already have installed. Check what you do have installed with eix or other favorite tool so you don't downgrade anything. Once wine is installed, as your user run:
$ winecfg
Download the EVE Online Windows installer and run it using Wine:
$ wine EVE_Online_Installer_*.exe
Once that's done, invoke the launcher as:
$ force_s3tc_enable=true wine 'C:\Program Files (x86)\CCP\EVE\eve.exe'
force_s3tc_enable=true is needed to enable texture rendering. Without it, EVE will freeze during start up. (If you didn't emerge media-libs/libtxc_dxtn, EVE will start, but none of the textures will load, and you'll have a lot of black on black objects.) I didn't have to do any of the other things I've found, such as disabling DirectX 11.
As for my Linux setup: I have a Radeon HD6480G (SUMO/r600) in my ThinkPad Edge E525, and I'm using the open source radeon (www.x.org) drivers with graphics on high and medium anti-aliasing with Mesa and OpenGL. For the most part, I find the game play to be smooth and indistinguishable from my experience on Windows.
There are a few things that don't work well. Psychedelic, rendering artifacts galore when I open the in-game browser (IGB) or switch to another application, but that's resolve without logging out of EVE by changing the graphics quality to something else. It may be related to resource caching, but I need to do more testing. I haven't tried going into the Captain's Quarters (other users have reported crashes entering there) as back on Windows that brings my system to a crawl, and there isn't anything particularly interesting about going in there…yet.
Overall, I'm quite happy with the EVE/Wine experience on Gentoo. It was quite easy and there wasn't any real troubleshooting for me to do.
If you're a fellow Gentoo-er in EVE, drop me a line. If you want to give EVE a go, have an extra week on me.
Update: I've been informed by Aatos Taavi that running EVE in windowed mode works quite well. I've also been informed that we need to declare stable packages in portage.accept_keywords because abi_x86_32 is use masked.
One of the new features of the 2.4 SELinux userspace is support for module priorities. The idea is that distributions and administrators can override a (pre)loaded SELinux policy module with another module without removing the previous module. This lower-version module will remain in the store, but will not be active until the higher-priority module is disabled or removed again.
The "old" modules (pre-2.4) are loaded with priority 100. When policy modules with the 2.4 SELinux userspace series are loaded, they get loaded with priority 400. As a result, the following message occurs:
~# semodule -i screen.pp libsemanage.semanage_direct_install_info: Overriding screen module at lower priority 100 with module at priority 400
So unlike the previous situation, where the older module is substituted with the new one, we now have two "screen" modules loaded; the last one gets priority 400 and is active. To see all installed modules and priorities, use the
--list-modules option:
~# semodule --list-modules=all | grep screen 100 screen pp 400 screen pp
Older versions of modules can be removed by specifying the priority:
~# semodule -X 100 -r screen
I have been trying my best not to comment on systemd one way or another for a while. For the most part because I don't want to have a trollfest on my blog, because moderating it is something I hate and I'm sure would be needed. On the other hand it seems like people start to bring me in the conversation now from time to time.
What I would like to point out at this point is that both extreme sides of the vision are, in my opinion, behaving childishly and being totally unprofessional. Whether it is name-calling of the people or the software, death threats, insults, satirical websites, labeling of 300 people for a handful of them, etc.
I don't think I have been as happy to have a job that allows me not to care about open source as much as I did before as in the past few weeks as things keep escalating and escalating. You guys are the worst. And again I refer to both supporters and detractors, devs of systemd, devs of eudev, Debian devs and Gentoo devs, and so on so forth.
And the reason why I say this is because you both want to bring this to extremes that I think are totally uncalled for. I don't see the world in black and white and I think I said that before. Gray is nuanced and interesting, and needs skills to navigate, so I understand it's easier to just take a stand and never revise your opinion, but the easy way is not what I care about.
Myself, I decided to migrate my non-server systems to systemd a few months ago. It works fine. I've considered migrating my servers, and I decided for the moment to wait. The reason is technical for the most part: I don't think I trust the stability promises for the moment and I don't reboot servers that often anyway.
There are good things to the systemd design. And I'm sure that very few people will really miss sysvinit as is. Most people, especially in Gentoo, have not been using sysvinit properly, but rather through OpenRC, which shares more spirit with systemd than sysv, either by coincidence or because they are just the right approach to things (declarativeness to begin with).
At the same time, I don't like Lennart's approach on this to begin with, and I don't think it's uncalled for to criticize the product based on the person in this case, as the two are tightly coupled. I don't like moderating people away from a discussion, because it just ends up making the discussion even more confrontational on the next forum you stumble across them — this is why I never blacklisted Ciaran and friends from my blog even after a group of them started pasting my face on pictures of nazi soldiers from WW2. Yes I agree that Gentoo has a good chunk of toxic supporters, I wish we got rid of them a long while ago.
At the same time, if somebody were to try to categorize me the same way as the people who decided to fork udev without even thinking of what they were doing, I would want to point out that I was reproaching them from day one for their absolutely insane (and inane) starting announcement and first few commits. And I have not been using it ever, since for the moment they seem to have made good on the promise of not making it impossible to run udev without systemd.
I don't agree with the complete direction right now, and especially with the one-size-fit-all approach (on either side!) that tries to reduce the "software biodiversity". At the same time there are a few designs that would be difficult for me to attack given that they were ideas of mine as well, at some point. Such as the runtime binary approach to hardware IDs (that Greg disagreed with at the time and then was implemented by systemd/udev), or the usage of tmpfs ACLs to allow users at the console to access devices — which was essentially my original proposal to get rid of pam_console (that played with owners instead, making it messy when having more than one user at console), when consolekit and its groups-fiddling was introduced (groups can be used for setgid, not a good idea).
So why am I posting this? Mostly to tell everybody out there that if you plan on using me for either side point to be brought home, you can forget about it. I'll probably get pissed off enough to try to prove the exact opposite, and then back again.
Neither of you is perfectly right. You both make mistake. And you both are unprofessional. Try to grow up.
Edit: I mistyped eudev in the original article and it read euscan. Sorry Corentin, was thinking one thing and typing another.
In a few moments, SELinux users which have the ~arch KEYWORDS set (either globally or for the SELinux utilities in particular) will notice that the SELinux userspace will upgrade to version 2.4 (release candidate 5 for now). This upgrade comes with a manual step that needs to be performed after upgrade. The information is mentioned as post-installation message of the
policycoreutils package, and basically sais that you need to execute:
~# /usr/libexec/selinux/semanage_migrate_store
The reason is that the SELinux utilities expect the SELinux policy module store (and the semanage related files) to be in
/var/lib/selinux and no longer in
/etc/selinux. Note that this does not mean that the SELinux policy itself is moved outside of that location, nor is the basic configuration file (
/etc/selinux/config). It is what tools such as semanage manage that is moved outside that location.
I tried to automate the migration as part of the packages themselves, but this would require the
portage_t domain to be able to move, rebuild and load policies, which it can't (and to be honest, shouldn't). Instead of augmenting the policy or making updates to the migration script as delivered by the upstream project, we currently decided to have the migration done manually. It is a one-time migration anyway.
If for some reason end users forget to do the migration, then that does not mean that the system breaks or becomes unusable. SELinux still works, SELinux aware applications still work; the only thing that will fail are updates on the SELinux configuration through tools like semanage or setsebool – the latter when you want to persist boolean changes.
~# semanage fcontext -l ValueError: SELinux policy is not managed or store cannot be accessed.
~# setsebool -P allow_ptrace on Cannot set persistent booleans without managed policy.
If you get those errors or warnings, all that is left to do is to do the migration. Note in the following that there is a warning about 'else' blocks that are no longer supported: that's okay, as far as I know (and it was mentioned on the upstream mailinglist as well as not something to worry about) it does not have any impact.
~# /usr/libexec/selinux/semanage_migrate_store Migrating from /etc/selinux/mcs/modules/active to /var/lib/selinux/mcs/active Attempting to rebuild policy from /var/lib/selinux sysnetwork: Warning: 'else' blocks in optional statements are unsupported in CIL. Dropping from output.
You can also add in
-c so that the old policy module store is cleaned up. You can also rerun the command multiple times:
~# /usr/libexec/selinux/semanage_migrate_store -c warning: Policy type mcs has already been migrated, but modules still exist in the old store. Skipping store. Attempting to rebuild policy from /var/lib/selinux
You can manually clean up the old policy module store like so:
~# rm -rf /etc/selinux/mcs/modules
So… don't worry – the change is small and does not break stuff. And for those wondering about CIL I'll talk about it in one of my next posts.
Just wanted to wish you a Happy 17th Birthday, Noah. I hope that it is a great day for you, and that the upcoming year is even better than this past one! My wish for you this year is that you are able to take time to enjoy the truly important things in life: family, friends, your health, and the events that don't require anything more than your attention. Take the time—MAKE the time—to stop and appreciate the world around you.
Yesterday I have released a new version of unpaper which is now in Portage, even though is dependencies are not exactly straightforward after making it use libav. But when I packaged it, I realized that the tests were failing — but I have been sure to run the tests all the time while making changes to make sure not to break the algorithms which (as you may remember) I have not designed or written — I don't really have enough math to figure out what's going on with them. I was able to simplify a few things but I needed Luca's help for the most part.
Turned out that the problem only happened when building with
-O2 -march=native so I decided to restrict tests and look into it in the morning again. Indeed, on Excelsior, using
-march=native would cause it to fail, but on my laptop (where I have been running the test after every single commit), it would not fail. Why? Furthermore, Luca was also reporting test failures on his laptop with OSX and clang, but I had not tested there to begin with.
A quick inspection of one of the failing tests' outputs with
vbindiff showed that the diffs would be quite minimal, one bit off at some non-obvious interval. It smelled like a very minimal change. After complaining on G+, Måns pushed me to the right direction: some instruction set that differs between the two.
My laptop uses the
core-avx-i arch, while the server uses
bdver1. They have different levels of SSE4 support – AMD having their own SSE4a implementation – and different extensions. I should probably have paid more attention here and noticed how the Bulldozer has FMA4 instructions, but I did not, it'll show important later.
I decided to start disabling extensions in alphabetical order, mostly expecting the problem to be in AMD's implementation of some instructions pending some microcode update. When I disabled AVX, the problem went away — AVX has essentially a new encoding of instructions, so enabling AVX causes all the instructions otherwise present in SSE to be re-encoded, and is a dependency for FMA4 instructions to be usable.
The problem was reducing the code enough to be able to figure out if the problem was a bug in the code, in the compiler, in the CPU or just in the assumptions. Given that unpaper is over five thousands lines of code and comments, I needed to reduce it a lot. Luckily, there are ways around it.
The first step is to look in which part of the code the problem appears. Luckily unpaper is designed with a bunch of functions that run one after the other. I started disabling filters and masks and I was able to limit the problem to the deskewing code — which is when most of the problems happened before.
But even the deskewing code is a lot — and it depends on at least some part of the general processing to be run, including loading the file and converting it to an
AVFrame structure. I decided to try to reduce the code to a standalone unit calling into the full deskewing code. But when I copied over and looked at how much code was involved, between the skew detection and the actual rotation, it was still a lot. I decided to start looking with
gdb to figure out which of the two halves was misbehaving.
The interface between the two halves is well-defined: the first return the detected skew, and the latter takes the rotation to apply (the negative value to what the first returned) and the image to apply it to. It's easy. A quick look through gdb on the call to
rotate() in both a working and failing setup told me that the returned value from the first half matched perfectly, this is great because it meant that the surface to inspect was heavily reduced.
Since I did not want to have to test all the code to load the file from disk and decode it into a RAW representation, I looked into the
gdb manual and found the
dump commands that allows you to dump part of the process's memory into a file. I dumped the
AVFrame::data content, and decided to use that as an input. At first I decided to just compile it into the binary (you only need to use
xxd -i to generate C code that declares the whole binary file as a byte array) but it turns out that GCC is not designed to compile efficiently a 17MB binary blob passed in as a byte array. I then opted in for just opening the raw binary file and
fread() it into the
AVFrame object.
My original plan involved using creduce to find the minimal set of code needed to trigger the problem, but it was tricky, especially when trying to match a complete file output to the md5. I decided to proceed with the reduction manually, starting from all the conditional for pixel formats that were not exercised… and then I realized that I could split again the code in two operations. Indeed while the main interface is only
rotate(), there were two logical parts of the code in use, one translating the coordinates before-and-after the rotation, and the interpolation code that would read the old pixels and write the new ones. This latter part also depended on all the code to set the pixel in place starting from its components.
By writing as output the calls to the interpolation function, I was able to restrict the issue to the coordinate translation code, rather than the interpolation one, which made it much better: the reduced test case went down to a handful of lines:
void rotate(const float radians, AVFrame *source, AVFrame *target) {
const int w = source->width;
const int h = source->height;
// create 2D rotation matrix
const float sinval = sinf(radians);
const float cosval = cosf(radians);
const float midX = w / 2.0f;
const float midY = h / 2.0f;
for (int y = 0; y < h; y++) {
for (int x = 0; x < w; x++) {
const float srcX = midX + (x - midX) * cosval + (y - midY) * sinval;
const float srcY = midY + (y - midY) * cosval - (x - midX) * sinval;
externalCall(srcX, srcY);
externalCall being a simple function to extrapolate the values, the only thing it does is printing them on the standard error stream. In this version there is still reference to the input and output
AVFrame objects, but as you can notice there is no usage of them, which means that now the testcase is self-contained and does not require any input or output file.
Much better but still too much code to go through. The inner loop over
x was simple to remove, just hardwire it to zero and the compiler still was able to reproduce the problem, but if I hardwired
y to zero, then the compiler would trigger constant propagation and just pre-calculate the right value, whether or not AVX was in use.
At this point I was able to execute
creduce; I only needed to check for the first line of the output to match the "incorrect" version, and no input was requested (the
radians value was fixed). Unfortunately it turns out that using
creduce with loops is not a great idea, because it is well possible for it to reduce away the
y++ statement or the
y < h comparison for exit, and then you're in trouble. Indeed it got stuck multiple times in infinite loops on my code.
But it did help a little bit to simplify the calculation. And with again a lot of help by Måns on making sure that the
cosf() functions would not return different values – they don't, also they are actually collapsed by the compiler to a single call to
sincosf(), so you don't have to write ugly code to leverage it! – I brought down the code to
extern void externCall(float);
extern float sinrotation();
extern float cosrotation();
static const float midX = 850.5f;
static const float midY = 1753.5f;
void main() {
const float srcX = midX * cosrotation() - midY * sinrotation();
No external libraries, not even libm. The external functions are in a separate source file, and beside providing fixed values for sine and cosine, the
externCall() function only calls
printf() with the provided value. Oh if you're curious, the
radians parameter became
0.6f, because 0, 1 and 0.5 would not trigger the behaviour, but 0.6 which is the truncated version of the actual parameter coming from the test file, would.
Checking the generated assembly code for the function then pointed out the problem, at least to Måns who actually knows Intel assembly. Here follows a diff of the code above, built with
-march=bdver1 and with
-march=bdver1 -mno-fma4 — because turns out the instruction causing the problem is not an AVX one but an FMA4, more on that after the diff.
movq -8(%rbp), %rax
xorq %fs:40, %rax
jne .L6
- vmovss -20(%rbp), %xmm2
- vmulss .LC1(%rip), %xmm0, %xmm0
- vmulss .LC0(%rip), %xmm2, %xmm1
+ vmulss .LC1(%rip), %xmm0, %xmm0
+ vmovss -20(%rbp), %xmm1
+ vfmsubss %xmm0, .LC0(%rip), %xmm1, %xmm0
.cfi_def_cfa 7, 8
- vsubss %xmm0, %xmm1, %xmm0
jmp externCall@PLT
It's interesting that it's changing the order of the instructions as well, as well as the constants — for this diff I have manually swapped
.LC0 and
.LC1 on one side of the diff, as they would just end up with different names due to instruction ordering.
As you can see, the FMA4 version has one instruction less:
vfmsubss replaces both one of the
vmulss and the one
vsubss instruction.
vfmsubss is a FMA4 instruction that performs a Fused Multiply and Subtract operation —
midX * cosrotation() - midY * sinrotation() indeed has a multiply and subtract!
Originally, since I was disabling the whole AVX instruction set, all the
vmulss instructions would end up replaced by
mulss which is the SSE version of the same instruction. But when I realized that the missing correspondence was
vfmsubss and I googled for it, it was obvious that FMA4 was the culprit, not the whole AVX.
Great, but how does that explain the failure on Luca's laptop? He's not so crazy so use an AMD laptop — nobody would be! Well, turns out that Intel also have their Fused Multiply-Add instruction set, just only with three operands rather than four, starting from Haswell CPUs, which include… Luca's laptop. A quick check on my NUC which also has a Haswell CPU confirms that the problem exists also for the
core-avx2 architecture, even though the code diff is slightly less obvious:
movq -24(%rbp), %rax
xorq %fs:40, %rax
jne .L6
- vmulss .LC1(%rip), %xmm0, %xmm0
- vmovd %ebx, %xmm2
- vmulss .LC0(%rip), %xmm2, %xmm1
+ vmulss .LC1(%rip), %xmm0, %xmm0
+ vmovd %ebx, %xmm1
+ vfmsub132ss .LC0(%rip), %xmm0, %xmm1
addq $24, %rsp
+ vmovaps %xmm1, %xmm0
popq %rbx
- vsubss %xmm0, %xmm1, %xmm0
popq %rbp
.cfi_def_cfa 7, 8
Once again I swapped
.LC0 and
.LC1 afterwards for consistency.
The main difference here is that the instruction for fused multiply-subtract is
vfmsub132ss and a
vmovaps is involved as well. If I read the documentation correctly this is because it stores the result in
%xmm1 but needs to move it to
%xmm0 to pass it to the external function. I'm not enough of an expert to tell whether gcc is doing extra work here.
So why is this instruction causing problems? Well, Måns knew and pointed out that the result is now more precise, thus I should not work it around. Wikipedia, as linked before, points also out why this happens:
A fused multiply–add is a floating-point multiply–add operation performed in one step, with a single rounding. That is, where an unfused multiply–add would compute the product b×c, round it to N significant bits, add the result to a, and round back to N significant bits, a fused multiply–add would compute the entire sum a+b×c to its full precision before rounding the final result down to N significant bits.
Unfortunately this does mean that we can't have bitexactness of images for CPUs that implement fused operations. Which means my current test harness is not good, as it compares the MD5 of the output with the golden output from the original test. My probable next move is to use
cmp to count how many bytes differ from the "golden" output (the version without optimisations in use), and if the number is low, like less than 1‰, accept it as valid. It's probably not ideal and could lead to further variation in output, but it might be a good start.
Optimally, as I said a long time ago I would like to use a tool like
pdiff to tell whether there is actual changes in the pixels, and identify things like 1-pixel translation to any direction, which would be harmless… but until I can figure something out, it'll be an imperfect testsuite anyway.
A huge thanks to Måns for the immense help, without him I wouldn't have figured it out so quickly.
It is not really obvious to many people how U2F is better than OTP for two-factor authentication; in particular I've seen it compared with full-blown smartcard-based authentication, and I think that's a bad comparison to do.
Indeed, since the Security Key is not protected by a PIN, and the NEO-n is designed to be semi-permanently attached to a laptop or desktop. At first this seems pretty insecure, as secure as storing the authorization straight into the computer, but it's not the case.
But let's start from the target users: the Security Key is not designed to replace the pure-paranoia security devices such as 16Kibit-per-key smartcards, but rather the on-phone or by-sms OTPs two-factor authenticators, those that use the Google Authenticator or other opensource implementations or that are configured to receive SMS.
Why replacing those? At first sight they all sound like perfectly good idea, what's to be gained to replace them? Well, there are plenty of things, the first of being the user friendliness of this concept. I know it's an overuse metaphor, but I do actually consider features on whether my mother would be able to use them or not — she's not a stupid person and can use a computer mostly just fine, but adding any on more procedures is something that would frustrate her quite a bit.
So either having to open an application and figure out which of many codes to use at one time, or having to receive an SMS and then re-type the code would be not something she'd be happy with. Even more so because she does not have a smartphone, and she does not keep her phone on all the time, as she does not want to be bothered by people. Which makes both the Authenticator and SMS ways not a good choice — and let's not try to suggests that there are way to not be available on the phone without turning it off, it would be more to learn that she does not care about.
Similar to the "phone-is-not-connected" problem, but for me rather than my mother, is the "wrong-country-for-the-phone" problem: I travel a lot, this year aiming for over a hundred days on the road, and there are very few countries in which I keep my Irish phone number available – namely Italy and the UK, where Three is available and I don't pay roaming, when the roaming system works… last time I've been to London the roaming system was not working – in the others, including the US which is obviously my main destination, I have a local SIM card so I can use data and calls. This means that if my 2FA setup sends an SMS on the Irish number, I won't receive it easily.
Admittedly, an alternative way to do this would be for me to buy a cheap featurephone, so that instead of losing access to that SIM, I can at least receive calls/SMS.
This is not only a theoretical. I have been at two conferences already (USENIX LISA 13, and Percona MySQL Conference 2014) and realized I cut myself out of my LinkedIn account: the connection comes from a completely different country than usual (US rather than Ireland) and it requires reauthentication… but it was configured to send the SMS to my Irish phone, which I had no access to. Given that at conferences is when you meet people you may want to look up on LinkedIn, it's quite inconvenient — luckily the authentication on the phone persists.
The authenticator apps are definitely more reliable than that when you travel, but they also come with their set of problems. Beside the not complete coverage of services (LinkedIn noted above for instance does not support authenticator apps), which is going to be a problem for U2F as well, at least at the beginning, neither Google's or Fedora's authenticator app allow you to take a backup of the private keys used for OTP authentication, which means that when you change your phone you'll have to replace, one by one, the OTP generation parameters. For some services such as Gandi, there is also no way to have a backup code, so if you happen to lose, break, or reset your phone without disabling the second factor auth, you're now in trouble.
Then there are a few more technical problems; HOTP, similarly to other OTP implementations, relies on shared state between the generator and the validator: a counter of how many times the code was generated. The client will increase it with every generation, the server should only increase it after a successful authentication. Even discounting bugs on the server side, a malicious actor whose intent is to lock you out can just make sure to generate enough codes on your device that the server will not look ahead enough to find the valid code.
TOTP instead relies on synchronization of time between server and generator which is a much safer assumption. Unfortunately, this also means you have a limited amount of time to type your code, which is tricky for many people who're not used to type quickly — Luca, for instance.
There is one more problem with both implementations: they rely on the user to choose the right entry and in the list and copy the right OTP value. This means you can still phish an user to type in an OTP and use it to authenticate against the service: 2FA is a protection against third parties gaining access to your account by having your password posted online rather than a protection against phishing.
U2F helps for this, as it lets the browser to handshake with the service before providing the current token to authenticate the access. Sure there might still be gaps on is implementation and since I have not studied it in depth I'm not going to vouch for it to be untouchable, but I trust the people who worked on it and I feel safer with it than I would be with a simple OTP.
I have already posted a howto on how to set up the YubiKey NEO and YubiKey NEO-n for U2F, and I promised I would write a bit more on the adventure to get the software packaged in Gentoo.
You have to realize at first that my relationship with Yubico has not always being straightforward. I have at least once decided against working on the Yubico set of libraries in Gentoo because I could not get a hold of a device as I wanted to use it. But luckily now I was able to place an order with them (for some two thousands euro) and I have my devices.
But Yubico's code is usually quite well written, and designed to be packaged much more easily than most other device-specific middleware, so I cannot complain too much. Indeed, they split and release separately different libraries with different goals, so that you don't need to wait for enough magnitude to be pulled for them to make a new release. They also actively maintain their code in GitHub, and then push proper
make dist releases on their website. They are in many ways a packager's dream company.
But let's get back to the devices themselves. The NEO and NEO-n come with three different interfaces: OTP (old-style YubiKey, just much longer keys), CCID (Smartcard interface) and U2F. By default the devices are configured as OTP only, which I find a bit strange to be honest. It is also the case that at the moment you cannot enable both U2F and OTP modes, I assume because there is a conflict on how the "touch" interaction behaves, indeed there is a touch-based interaction on the CCID mode that gets entirely disabled once enabling either of U2F or OTP, but the two can't share.
What is not obvious from the website is that to enable U2F (or CCID) modes, you need to use yubikey-neo-manager, an open-source app that can reconfigure the basics of the Yubico device. So I had to package the app for Gentoo of course, together with its dependencies, which turned out to be two libraries (okay actually three, but the third one
sys-auth/ykpers was already packaged in Gentoo — and actually originally committed by me with Brant proxy-maintaining it, the world is small, sometimes). It was not too bad but there were a few things that might be worth noting down.
First of all, I had to deal with
dev-libs/hidapi that allows programmatic access to raw HID USB devices: the ebuild failed for me, both because it was not depending on udev, and because it was unable to find the libusb headers — turned out to be caused by bashisms in the
configure.ac file, which became obvious as I moved to
dash. I have now fixed the ebuild and sent a pull request upstream.
This was the only real hard part at first, since the rest of the ebuilds, for
app-crypt/libykneomgr and
app-crypt/yubikey-neo-manager were mostly straightforward — only I had to figure out how to install a Python package as I never did so before. It's actually fun how distutils will error out with a violation of install paths if
easy_install tries to bring in a non-installed package such as
nose, way before the Portage sandbox triggers.
The problems started when trying to use the programs, doubly so because I don't keep a copy of the Gentoo tree on the laptop, so I wrote the ebuilds on the headless server and then tried to run them on the actual hardware. First of all, you need to have access to the devices to be able to set them up; the
libu2f-host package will install udev rules to allow the
plugdev group access to the
hidraw devices — but it also needed a pull request to fix them. I also added an alternative version of the rules for systemd users that does not rely on the group but rather uses the ACL support (I was surprised, I essentially suggested the same approach to replace
pam_console years ago!)
Unfortunately that only works once the device is already set in U2F mode, which does not work when you're setting up the NEO for the first time, so I originally set it up using
kdesu. I have since decided that the better way is to use the udev rules I posted in my howto post.
After this, I switched off OTP, and enabled U2F and CCID interfaces on the device — and I couldn't make it stick, the manager would keep telling me that the CCID interface was disabled, even though the USB descriptor properly called it "Yubikey NEO U2F+CCID". It took me a while to figure out that the problem was in the
app-crypt/ccid driver, and indeed the change log for the latest version points out support for specifically the U2F+CCID device.
I have updated the ebuilds afterwards, not only to depend on the right version of the CCID driver – the README for
libykneomgr does tell you to install pcsc-lite but not about the CCID driver you need – but also to check for the HIDRAW kernel driver, as otherwise you won't be able to either configure or use the U2F device for non-Google domains.
Now there is one more part of the story that needs to be told, but in a different post: getting GnuPG to work with the OpenPGP applet on the NEO-n. It was not as straightforward as it could have been and it did lead to disappointment. I'll be a good post for next week.
When the Google Online Security blog announced earlier this week the general availability of Security Key, everybody at the office was thrilled, as we've been waiting for the day for a while. I've been using this for a while already, and my hope is for it to be easy enough for my mother and my sister, as well as my friends, to start using it.
While the promise is for a hassle-free second factor authenticator, it turns out it might not be as simple as originally intended, at least on Linux, at least right now.
Let's start with the hardware, as there are four different options of hardware that you can choose from:
I got the NEO, but mostly to be used with LastPass – the NFC support allows you to have 2FA on the phone without having to type it back from a computer – and a NEO-n to leave installed on one of my computers. I already had a NEO from work to use as well. The NEO requires configuration, so I'll get back at it in a moment.
The U2F devices are accessible via
hidraw, a driverless access protocol for USB devices, originally intended for devices such as keyboards and mice but also leveraged by UPSes. What happen though is that you need access to the device, that the Linux kernel will make by default accessible only by root, for good reasons.
To make the device accessible to you, the user actually at the keyboard of the computer, you have to use udev rules, and those are, as always, not straightforward. My personal hacky choice is to make all the Yubico devices accessible — the main reason being that I don't know all of the compatible USB Product IDs, as some of them are not really available to buy but come from instance from developer mode devices that I may or may not end up using.
If you're using systemd with device ACLs (in Gentoo, that would be
sys-apps/systemd with
acl USE flag enabled), you can do it with a file as follows:
# /etc/udev/rules.d/90-u2f-securitykey.rules
ATTRS{idVendor}=="1050", TAG+="uaccess"
ATTRS{idVendor}=="2581", ATTRS{idProduct}=="f1d0", TAG+="uaccess"
If you're not using systemd or ACLs, you can use the
plugdev group and instead do it this way:
# /etc/udev/rules.d/90-u2f-securitykey.rules
ATTRS{idVendor}=="1050", GROUP="plugdev", MODE="0660"
ATTRS{idVendor}=="2581", ATTRS{idProduct}=="f1d0", GROUP="plugdev", MODE="0660"
-These rules do not include support for the Plug-up because I have no idea what their VID/PID pairs are, I asked Janne who got one so I can amend this later.- Edit: added the rules for the Plug-up device. Cute their use of f1d0 as device id.
Also note that there are properly less hacky solutions to get the ownership of the devices right, but I'll leave it to the systemd devs to figure out how to include in the default ruleset.
These rules will not only allow your user to access
/dev/hidraw0 but also to the
/dev/bus/usb/* devices. This is intentional: Chrome (and Chromium, the open-source version works as well) use the U2F devices in two different modes: one is through a built-in extension that works with Google assets, and it accesses the low-level device as
/dev/bus/usb/*, the other is through a Chrome extension which uses
/dev/hidraw* and is meant to be used by all websites. The latter is the actually standardized specification and how you're supposed to use it right now. I don't know if the former workflow is going to be deprecated at some point, but I wouldn't be surprised.
For those like me who bought the NEO devices, you'll have to enable the U2F mode — while Yubico provides the linked step-by-step guide, it was not really completely correct for me on Gentoo, but it should be less complicated now: I packaged the
app-crypt/yubikey-neo-manager app, which already brings in all the necessary software, including the latest version of
app-crypt/ccid required to use the CCID interface on U2F-enabled NEOs. And if you already created the udev rules file as I noted above, it'll work without you using root privileges. Just remember that if you are interested in the OpenPGP support you'll need the
pcscd service (it should auto-start with both OpenRC and systemd anyway).
I'll recount separately the issues with packaging the software. In the mean time make sure you keep your accounts safe, and let's all hope that more sites will start protecting your accounts with U2F — I'll also write a separate opinion piece on why U2F is important and why it is better than OTP, this is just meant as documentation, howto set up the U2F devices on your Linux systems.
The september council meeting was quite uneventful. The only outcome of note was that the dohtml function for ebuilds will be deprecated now and banned in a later EAPI, with some internal consequences for, e.g., einstalldocs.
New LiveDVD - Iron Penguin Edition thanks to the Gentoo Infrastructure team and Fernando Reyes. If you haven't yet checked it out, what are you waiting for? Go get it on your closest mirror.
(shameless copy of Tomas Chvatal's report on the gentoo-project mailing list)
Hello guys,
First I would like to say big thank you to Amy (amynka) for prodding and nudging people and working on the booth. Next in line is Christopher (chithead) whom also handled our booth and even brought with him fancy MIPS machine and monitor all the way from Berlin. Kudos for that. And last I want to commend all the people giving the talks during the day. In the end we did bit Q&A with users, which was short so rest I spent asking how we should do the miniconf and what would be desired. So first lets take look on what we had and what we can do there to make it even cooler for next time:
Place where we share/sell SWAG chat with community. People stopped by, took some stickers here and there and watched the MIPS boxie we had there. I have to admit that I screwed up with our materials a bit and we didn't have much on the stand. I thought we have more leftover stickers/brochures, but we had just few and super plan to get Gentoo t-shirts failed me miserably…
Someone from Gentoo ev. could arrive too and actually sell some stuff like cups/tshirts as we seem unable to get something working here in Czech republic. With that we would have really pretty booth. People were quite interested in our merchandise and are even willing to buy it.
We had one day of talks, and basically everything went smoothly and videos will be available in near future on youtube. I will try to remember to post link here as reply when it is done (if it is not here in a week, prod me on irc because that means I forgot).
We should make the thing 2 days, so it is worth for people to go to Prague, for one day I guess it is not that motivating. We should start looking for talks sooner than couple of months in advance so people can plan for it.
First here are photos:
Linuxdays people are more than happy to provide us with the room if we have the content. Most of the people attending to the conference speak english, so even tho quite parts of the tracks are czech, we can talk with the people around. We could do it yearly/bi-yearly, my take would be to create 2 days miniconf each two year, so next one could be done 2016 unless of course you want it next year again and tell me right now
Gentoo is made up of 242 active developers, of which 43 are currently away.
Gentoo has recruited a total of 803 developers since its inception.
This section summarizes the current state of the portage tree.
|% of Packages
The following GLSAs have been released by the Security Team
|Bash: Code Injection (Updated fix for GLSA 201409-09)
|Bash: Code Injection
|libxml2: Denial of Service
|c-icap: Denial of Service
|Chromium: Multiple vulnerabilities
|Adobe Flash Player: Multiple vulnerabilities
|MySQL: Multiple vulnerabilities
|dhcpcd: Denial of service
|Net-SNMP: Denial of Service
|Wireshark: Multiple vulnerabilities
|07 Sep 2014
|07 Sep 2014
|07 Sep 2014
|07 Sep 2014
|07 Sep 2014
|07 Sep 2014
|07 Sep 2014
|07 Sep 2014
|07 Sep 2014
|07 Sep 2014
|07 Sep 2014
|07 Sep 2014
|07 Sep 2014
|07 Sep 2014
|07 Sep 2014
|07 Sep 2014
|07 Sep 2014
|07 Sep 2014
|07 Sep 2014
|09 Sep 2014
|09 Sep 2014
|09 Sep 2014
|15 Sep 2014
|19 Sep 2014
|29 Sep 2014
|01 Sep 2014
|02 Sep 2014
|02 Sep 2014
|02 Sep 2014
|02 Sep 2014
|02 Sep 2014
|02 Sep 2014
|02 Sep 2014
|04 Sep 2014
|04 Sep 2014
|04 Sep 2014
|04 Sep 2014
|04 Sep 2014
|04 Sep 2014
|05 Sep 2014
|06 Sep 2014
|09 Sep 2014
|10 Sep 2014
|11 Sep 2014
|11 Sep 2014
|11 Sep 2014
|11 Sep 2014
|11 Sep 2014
|11 Sep 2014
|11 Sep 2014
|11 Sep 2014
|11 Sep 2014
|11 Sep 2014
|11 Sep 2014
|11 Sep 2014
|11 Sep 2014
|11 Sep 2014
|11 Sep 2014
|11 Sep 2014
|11 Sep 2014
|11 Sep 2014
|13 Sep 2014
|14 Sep 2014
|14 Sep 2014
|14 Sep 2014
|15 Sep 2014
|15 Sep 2014
|16 Sep 2014
|16 Sep 2014
|17 Sep 2014
|18 Sep 2014
|20 Sep 2014
|20 Sep 2014
|21 Sep 2014
|21 Sep 2014
|21 Sep 2014
|21 Sep 2014
|21 Sep 2014
|21 Sep 2014
|21 Sep 2014
|21 Sep 2014
|21 Sep 2014
|21 Sep 2014
|21 Sep 2014
|21 Sep 2014
|21 Sep 2014
|21 Sep 2014
|22 Sep 2014
|22 Sep 2014
|22 Sep 2014
|23 Sep 2014
|25 Sep 2014
|25 Sep 2014
|25 Sep 2014
|25 Sep 2014
|25 Sep 2014
|25 Sep 2014
|25 Sep 2014
|25 Sep 2014
|26 Sep 2014
|26 Sep 2014
|27 Sep 2014
|27 Sep 2014
|27 Sep 2014
|27 Sep 2014
|28 Sep 2014
|28 Sep 2014
|28 Sep 2014
|28 Sep 2014
|29 Sep 2014
|30 Sep 2014
|30 Sep 2014
|30 Sep 2014
|30 Sep 2014
|30 Sep 2014
The Gentoo community uses Bugzilla to record and track bugs, notifications, suggestions and other interactions with the development team.
The following tables and charts summarize the activity on Bugzilla between 01 September 2014 and 01 October 2014. Not fixed means bugs that were resolved as NEEDINFO, WONTFIX, CANTFIX, INVALID or UPSTREAM.
|Bug Activity
|Not fixed
The following table outlines the teams and developers with the most bugs resolved during this period
|Bug Count
|Gentoo Security
|Gentoo Linux Gnome Desktop Team
|Python Gentoo Team
|Qt Bug Alias
|Perl Devs @ Gentoo
|Gentoo KDE team
|Portage team
|Gentoo Games
|Netmon Herd
The developers and teams who have been assigned the most bugs during this period are as follows.
|Bug Count
|Gentoo Linux bug wranglers
|Gentoo Security
|Gentoo Linux Gnome Desktop Team
|Gentoo's Team for Core System packages
|Gentoo Games
|Portage team
|Python Gentoo Team
|Gentoo KDE team
|Perl Devs @ Gentoo
(thanks to Thomas D. for the link to the blog post)
In case you like messing with your kernel Kconfig options to tweak the kernel image for your Gentoo boxes, you may want to know that menuconfig accepts regular expressions for searching symbols. You can start the search by typing '/'. For example, if you want to find all symbols ending with PCI do something like this after pressing '/'.
You get a bunch of results, and then you can press the number listed on the left to jump directly to that symbol.
Related references:
Send us your favorite Gentoo script or tip at firstname.lastname@example.com
Interested in helping out? The GMN relies on volunteers and members of the community for content every month. If you are interested in writing for the GMN or thinking of another way to contribute, please send an e-mail to firstname.lastname@example.com.
Please head over to this forum post.
dev-lang/perlOf course, as always, bugs may be present; what you get as Perl installation is called unstable or testing for a reason. We're looking forward to your reports on our bugzilla.
I've been pretty busy lately, albeit behind the corners, which leads to a lower activity within the free software communities that I'm active in. Still, I'm not planning any exit, on the contrary. Lots of ideas are just waiting for some free time to engage. So what are the challenges that have been taking up my time?
One of them is that I recently moved. And with moving comes a lot of work in getting the place into a good shape and getting settled. Today I finished the last job that I wanted to finish in my appartment in a short amount of time, so that's one thing off my TODO list.
Another one is that I started an intensive master-after-master programme with the subject of Enterprise Architecture. This not only takes up quite some ex-cathedra time, but also additional hours of studying (and for the moment also exams). But I'm really satisfied that I can take up this course, as I've been wandering around in the world of enterprise architecture for some time now and want to grow even further in this field.
But that's not all. One of my side activities has been blooming a lot, and I recently reached the 200th server that I'm administering (although I think this number will reduce to about 120 as I'm helping one organization with handing over management of their 80+ systems to their own IT staff). Together with some friends (who also have non-profit customers' IT infrastructure management as their side-business) we're now looking at consolidating our approach to system administration (and engineering).
I'm also looking at investing time and resources in a start-up, depending on the business plan and required efforts. But more information on this later when things are more clear :-)
Vittorio started (with some help from me) to fix all the issues pointed by Coverity.
Coverity (and scan-build) are quite useful to spot mistakes even if their false-positive ratio tend to be quite high. Even the false-positives are usually interesting since the spot code unnecessarily convoluted. The code should be as simple as possible but not simpler.
The basic idea behind those tools is to try to follow the code-paths while compiling them and spot what could go wrong (e.g. you are feeding a NULL to a function that would deference it).
The problems with this approach are usually two: false positive due to the limited scope of the analyzer and false negatives due shadowing.
Coverity might assume certain inputs are valid even if they are made impossible by some initial checks up in the codeflow.
In those case you should spend enough time to make sure Coverity is not right and those faulty inputs aren't slipping somewhere. NEVER try to just add some checks to the code pointed as first move, you might either hide issues (e.g. if Coverity complains about uninitialized variable do not just initialize it to nothing, check why it happens and if the logic behind is wrong).
While fixing issues using those tools you might notice to your surprise that every time you fix something, something new appears out of thin air.
This is not magic but simply that the static analyzers usually keep some limit on how deep they go depending on the issues already present and how much time had been spent already.
That surprise had been fun since apparently some of the time limit is per compilation unit so splitting large files in smaller chunks gets us more results (while speeding up the building process thanks to better parallelism).
Usually fixing some high-impact issue gets us 3 or 5 new small impact issues.
I like solving puzzles so I do not mind having more fun, sadly I did not have much spare time to play this game lately.
So far about 100+ patches got piled up over the past weeks and now they are sent in small batches to ease the work of review. (I have something brewing to make reviewing simpler, as you might know)
During the review what probably about 1/10 of the patches will be rejected and the relative coverity report updated with enough information to explain why it is a false positive or the dangerous or strange behaviour pointed is intentional.
You need good tools to do a good job.
Even the best tool in the hand of a novice is a club.
I'm quite fond in improving the tools I use. And that's why I started getting involved in Gentoo, Libav, VLC and plenty of other projects.
I already discussed about lldb and asan/valgrind, now my current focus is about patch trackers. In part it is due to the current effort to improve the libav one,
Before talking about patches and their tracking I'd digress a little on who produces them. The mythical Contributor: without contributions an opensource project would not exist.
You might have recurring contributions and unique/seldom contributions. Both are quite important.
In general you should make so seldom contributors become recurring contributors.
A recurring contributor can accept to spend some additional time to setup the environment to actually provide its contribution back to the community, a sporadic contributor could be easily put off if the effort required to send his patch is larger than writing the patch itself.
Th project maintainers should make so the life of contributors is as simple as possible.
Lately most opensource projects saw the light and started to use decentralized source revision control system and thanks to github and many other is the concept of issue pull requests is getting part of our culture and with it comes hopefully a wider acceptance to the fact that the code should be reviewed before it is merged.
In a decentralized development scenario new code is usually developed in topic branches, routinely rebased against the master until the set is ready and then the set of changes (called series or patchset) is reviewed and after some round of fixes eventually merged. Thanks to bitbucket now we have forking, spooning and knifing as part of the jargon.
The review (and merge) step, quite properly, is called knifing (or stabbing): you have to dice, slice and polish the code before merging it.
During a review bugs are usually spotted as well way to improve are suggested. Patches might be split or merged together and the series reworked and improved a lot.
The process is usually time consuming, even more for an organization made of volunteer: writing code is fun, address issues spotted is not so much, review someone else code is much less even.
The old fashioned way to issue a pull request is either poke somebody telling that your branch is ready for merge or just make a set of patches and mail them to whoever is in charge of integrating code to the main branch.
git provides a nifty tool to do that called
git send-email and is quite common to send sets of patches (called usually series) to a mailing list. You get feedback by email and you can update the set using the
--in-reply-to option and the message id.
Platforms such as github and similar are more web centric and require you to use the web interface to issue and review the request. No additional tools are required beside your git and a browser.
gerrit and reviewboard provide custom scripts to setup ephemeral branches in some staging area then the review process requires a browser again. Every commit gets some tool-specific metadata to ease tracking changes across series revisions. This approach the more setup intensive.
Testing patches from the mailing list is quite simple thanks to
git am. And if the reply-to field is used properly updates appear sorted in a good way.
This method is the simplest for the people used to have the email client always open and a console (if they are using a well configured emacs or vim they literally do not move away from the editor).
On the other hand, people using a webmail or using a basic email client might find the approach more cumbersome than a web based one.
If your only method to track contribution is just a mailing list, gets quite easy to forget which is the status of a set. Patches could be neglected and even who wrote them might forget for a long time.
Patchwork tracks which patches hit a mailing list and tries to figure out if they are eventually merged automatically.
It is quite basic: it provides an web interface to check the status and provides a mean to just update the patch status. The review must happen in the mailing list and there is no concept of series.
As basic as it is works as a reminder about pending patches but tends to get cluttered easily and keeping it clean requires some effort.
The web interface makes much easier spot what is pending and what's its status, people used to have everything in the browser (chrome and mozilla could be made to work as a decent IDE lately) might like it much better.
Reviewing small series or single patches is usually nicer but the current UIs do not scale for larger (5+) patchsets.
People not living in a browser find quite annoying switch context and it requires additional effort to contribute since you have to register to a website and the process of issuing a patch requires many additional steps while in the email approach just require to type
git send-email -1.
The gerrit interfaces tend to be richer than the Github counterparts. That can be good or bad since they aren't as immediate and tend to overwhelm new contributors.
You need to make an additional effort to setup your environment since you need some custom script.
The series are tracked with additional precision, but for all the practical usage is the same as github with the additional bourden for the contributor.
Plaid is my attempt to tackle the problem. It is currently unfinished and in dire need of more hands working on it.
It's basic concept is to be non-intrusive as much as possible, retaining all the pros of the simple git+email workflow like patchwork does.
It provides already additional features such as the ability to manage series of patches and to track updates to it. It sports a view to get a break out of which series require a review and which are pending for a long time waiting for an update.
What's pending is adding the ability to review it directly in the browser, send the review email for the web to the mailing list and a some more.
Probably I might complete it within the year or next spring, if you like Flask or python contributions are warmly welcome!
Some of you probably read me ranting on G+ and Twitter about blog post editors. I have been complaining about that since at least last year when Typo decided to start eating my drafts. After that almost meltdown I decided to look for alternatives on writing blog posts, first with Evernote – until they decided to reset everybody's password and required you to type some content from one of your notes to be able to get the new one – and then with Google Docs.
I have indeed kept using Google Docs until recently, when it started having some issues with dead keys. Because I have been using US International layout for years, and I'm too used to it when I write English too. If I am to use a non-deadkeys keyboard, I end up adding spaces where they shouldn't be. So even if it solves it by just switching the layout, I wouldn't want to write a long text with it that way.
Then I decided to give another try to Evernote, especially as the Samsung Galaxy Note 10.1 I bought last year came with a yet-to-activate 12 months subscription to the Pro version. Not that I find anything extremely useful in it, but…
It all worked well for a while until they decided to throw me into the new Beta editor, which follows all the newest trends in blog editors. Yes because there are trends in editors now! Away goes the full-width editing, instead you have a limited-width editing space in a mostly-white canvas with disappearing interface, like node.js's Ghost and Medium and now Publify (the new name of what used to be Typo).
And here's my problem: while I understand that they try to make things that look neat and that supposedly are there to help you "focus on writing" they miss the point quite a bit with me. Indeed, rather than having a fancy editor, I think Typo needs a better drafting mechanism that does not puke on itself when you start playing with dates and other similar details.
And Evernote's new editor is not much better; indeed last week, while I was in Paris, I decided to take half an afternoon to write about libtool – mostly because J-B has been facing some issues and I wanted to document the root causes I encountered – and after two hours of heavy writing, I got to Evernote, and the note is gone. Indeed it asked me to log back in. And I logged in that same morning.
When I complained about that on Twitter, the amount of snark and backward thinking I got surprised me. I was expecting some trolling, but I had people seriously suggesting me that you should not edit things online. What? In 2014? You've got to be kidding me.
But just to make that clear, yes I have used offline editing for a while back, as Typo's editor has been overly sensible to changes too many times. But it does not scale. I'm not always on the same device, not only I have three computers in my own apartment, but I have two more at work, and then I have tablets. It is not uncommon for me to start writing on a post on one laptop, then switch to the other – for instance because I need access to the smartcard reader to read some data – or to start writing a blog post while at a conference with my work laptop, and then finish it in my room on the personal one, and so on so forth.
Yes I could use Dropbox for out-of-band synchronization, but its handling of conflicts is not great, if you end up having one of the devices offline by mistake — better than the effets of it on password syncs but not so much better. Indeed I have bad experiences with that, because it makes it too easy to start working on something completely offline, and then forget to resync it before editing it from a different service.
Other suggestions included (again) the use of statically generated blogs. I have said before that I don't care for them and I don't want to hear them as suggestions. First they suffer from the same problems stated above with working offline, and secondly they don't really support comments as first class citizens: they require services such as Disqus, Google+ or Facebook to store the comments, including it in the page as an external iframe. I not only don't like the idea of farming out the comments to a different service in general, but I would be losing too many features: search within the blog, fine-grained control over commenting (all my blog posts are open to comment, but it's filtered down with my ModSecurity rules), and I'm not even sure they would allow me to import the current set of comments.
I wonder why, instead of playing with all the CSS and JavaScript to make the interface disappear, the editors' developers don't invest time to make the drafts bulletproof. Client-side offline storage should allow for preserving data even in case of being logged out or losing network connection. I know it's not easy (or I would be writing it myself) but it shouldn't be impossible, either. Right now it seems the bling is what everybody wants to work on, rather than functionality — it probably is easier to put in your portfolio, and that could be a good explanation as any.
I ran into this documentary today…
Some of the recent releases of Trojitá, a fast Qt e-mail client, mentioned an ongoing work towards bringing the application to the Ubuntu Touch platform. It turns out that this won't be happening.
The developers who were working on the Ubuntu Touch UI decided that they would prefer to end working with upstream and instead focus on a standalone long-term fork of Trojitá called Dekko. The fork lives within the Launchpad ecosystem and we agreed that there's no point in keeping unmaintained and dead code in our repository anymore -- hence it's being removed.
Our latest roadtrip was as amazing as it was challenging because we decided that we'd spend an entire month in Turkey and use our own motorbike to get there from Paris.
Our main idea was to spare ourselves from the long hours of road riding to Turkey so we decided from the start to use ferries to get there. Turns out that it's pretty easy as you have to go through Italy and Greece before you set foot in Bodrum, Turkey.
Turkish customs are very friendly and polite, it's really easy to get in with your own vehicle.
This roadtrip added 6000 kms to our brave and astonishing Harley-Davidson Nightster. We encountered no problem at all with the bike even though we clearly didn't go easy on her. We rode on gravels, dirt and mud without her complaining, not to mention the weight of our luggages and the passengers
That's why this post will be dedicated to our bike and I'll share some of the photos I took of it during the trip. The real photos will come in some other posts.
I can't describe well enough the pleasure and freedom feeling you get when travelling in motorbike so I hope those first photos will give you an idea.
I have to admit that it's really impressive to leave your bike alone between the numerous trucks parking, loading/unloading their stuff a few centimeters from it.
We arrived in Piraeus easily, time to buy tickets for the next boat to Kos.
Kos is quite a big island that you can discover best by … riding around !
After Bodrum, where we only spent the night, you quickly discover the true nature of Turkish roads and scenery. Animals are everywhere and sometimes on the road such as those donkeys below.
This is a view from the Bozburun bay. Two photos for two bike layouts : beach version and fully loaded version
On the way to Cappadocia, near Karapinar :
The amazing landscapes of Cappadocia, after two weeks by the sea it felt cold up there.
Our last picture from the bike next to the trail leading to our favorite and lonely "private" beach on the Datça peninsula.
Hi all,
One of the projects I had last year that I ended up suspending due to lack of time was S390 documentation and installation materials. For some reason there wasn't any materials available to install Gentoo on a S390 system without having to rely in an already installed distribution.
Thanks to Marist College, IBM and Linux Foundation we were able to get two VMs for building the release materials, and thanks to Dave Jones @ V/Soft Software I was able to document the installation in a z/VM environment. Also thanks to the Debian project, since I based the materials in their procedure.
So most of the part of last year and the last few weeks I've been polishing and finishing the documentation I had around. So what I've documented: Gentoo S390 on the Hercules emulator and Gentoo S390 on z/VM. Both are based in the same pattern, since
Gentoo S390 on the Hercules emulator
This is probably the guide that will be more interesting because everyone can run the Hercules emulator, while not everyone has access to a z/VM instance. Hercules emulates an S390 system, it's like QEMU. However QEMU, from what I can tell, is unable to emulate an S390 system in a non-S390 system, while Hercules does.
So if you want to have some fun and emulate a S390 machine in your computer, and install and use Gentoo in it, then follow the guide: https://wiki.gentoo.org/wiki/S390/Hercules
Gentoo S390 on z/VM
For those that have access to z/VM and want to install Gentoo, the guide explains all the steps needed to get a Gentoo System working. Thanks to Dave Jones I was able to create the guide and test the release materials, he even did a presentation in the 2013 VM Workshop! Link to the PDF . Keep in mind that some of the instructions given there are now outdated, mainly the links.
The link to the documentation is: https://wiki.gentoo.org/wiki/S390/Install
I have also written some tips and tricks for z/VM: https://wiki.gentoo.org/wiki/S390/z/VM_tips_and_tricks They're really basic and were the ones I needed for creating the guide.
Installation materials
Lastly, we already had the autobuilds stage3 for s390, but we lacked the boot environment for installing Gentoo. This boot environment/release material is simply a kernel and a initramfs built with Gentoo's genkernel based in busybox. It builds an environment using busybox like the livecd in amd64/x86 or other architectures. I've integrated the build of these boot environment with the autobuilds, so each week there should be an updated installation environment.
Have fun!
In multimedia, video is basically crunching numbers and get pixels or crunching pixels and getting numbers. Most of the operation are quite time consuming on a general purpose CPU and orders of magnitude faster if done using DSP or hardware designed for that purpose.
Most of the commonly used system have video decoding and encoding capabilities either embedded in the GPU or in separated hardware. Leveraging it spares lots of cpu cycles and lots of battery if we are thinking about mobile.
The usually specialized hardware has the issue of being inflexible and that does clash with the fact most codec evolve quite quickly with additional profiles to extend its capabilities, support different color spaces, use additional encoding strategies and such. Software decoders and encoders are still needed and need badly.
The hardware acceleration support in Libav grew (like other eldritch-horror tentacular code we have lurking from our dark past) without much direction addressing short term problems and not really documenting how to use it.
As result all the people that dared to use it had to guess, usually used internal symbols that they wouldn't have to use and all in all had to spend lots of time and
had enough grief when such internals changed.
Every backend required a quite large deal of boilerplate code to initialize the backend-specific context and to render the hardware surface wrapped in the AVFrame.
The Libav backend interface was quite vague in itself, requiring to override
get_format and
get_buffer in some ways.
Overall to get the whole thing working the library user was supposed to do about 75% of the work. Not really nice considering people uses libraries to abstract complexity and avoid repetition…
As that support was written with just slice-based decoder in mind, it expects that all the backend would require the software decoder to parse the bitstream, prepare slices of the frame and feed the backend with them.
Sadly new backends appeared and they take directly either bitstream or full frames, the approach had been just to take the slice, add back the bitstream markers the backend library expects and be done with that.
Last year since the number of backends I wanted to support were all bitstream-oriented and not fitting the mode at all I started thinking about it and the topic got discussed a bit during VDD 13. Some people that spent their dear time getting hwaccel1 working with their software were quite wary of radical changes so a path of incremental improvements got more or less put down.
The new VDA backend and the update VDPAU backend are examples of it.
The Intel QSV backend is used as a testbed for hwaccel 1.3.
Another year, another meeting. We sat down again to figure out how to get further closer to the end result of not having the casual users write boilerplate code to use hwaccel to get at least some performance boost and yet let the power users have the full access to the underpinnings so they can get most of it without having to write everything from scratch.
The user just needs to use AVOption to set specific keys such as hwaccel and optionally hwaccel-device and the library will take care of everything. The frames returned by
avcodec_decode_video2 will contain normal system memory and commonly used pixel formats. No further special code will be needed.
All the default initialization, memory/surface allocation and such will remain overridable, with the difference that an additional callback called
get_hw_surface will be introduced to separate completely the hwaccel path from the software path and specific functions to hand over the ownership of backend contexts and surfaces will be provided.
The software fallback won't be anymore automagic in this case, but a specific
AVERROR_INPUT_CHANGED will be returned so would be cleaner for the user reset the decoder without losing the display that maybe was sharing the same context. This leads the way to a simpler mean to support multiple hwaccel backends and fall back from one to the other to eventually the software decoding.
We try our best to help people move to the new APIs.
Moving from HWAccel1 to HWAccel2 in general would result in less lines of code in the application, the people wanting to keep their callback need to just set them after
avcodec_open2 and move the pixel specific
get_buffer to
get_hw_surface. The presence of
av_hwaccel_hand_over_frame and
av_hwaccel_hand_over_context will make much simpler managing the backend specific resources.
Right now the review is on the HWaccel1.3, I hope to complete this step and add few new backends to test how good/bad that API is before adding the other steps. Probably HWAccel2 will take at least other 6 months.
Help in form of code or just moral support is always welcome!
Contrary to some articles you may read on the internet, NetFlix is working great on Gentoo.
Here's a snap shot of my system running 3.12.30-gentoo sources and google chrome version 39.0.2171.19_p1.
$ equery l google-chrome-beta
* Searching for google-chrome-beta …
[IP-] [ ] www-client/google-chrome-beta-39.0.2171.19_p1:0
Back from holidays, this new version of py3status was due for a long time now as it features a lot of great contributions !
This version is dedicated to the amazing @ShadowPrince who contributed 6 new modules
Huge thanks to this release's contributors :
The next 1.7 release of py3status will bring a neat and cool feature which I'm sure you'll love, stay tuned !
I recently ran into the Q&A with Linus Torvalds @ Debconf 2014 video.
During the session, Torvalds is being criticized for lack of respect and replies that ~"respect is to be earned". Technically, he confuses respect as in admiration with respect as in dignity. Simplified, he is saying that human dignity does not matter to him. Linus, I'm fairly disappointed.
It has been four months since my last major build and release of Lilblue Linux, a pet project of mine [1]. The name is a bit pretentious, I admit, since Lilblue is not some other Linux distro. It is Gentoo, but Gentoo with a twist. It's a fully featured amd64, hardened, XFCE4 desktop that uses uClibc instead of glibc as its standard C library. I use it on some of my workstations at the College and at home, like any other desktop, and I know other people that use it too, but the main reason for its existence is that I wanted to push uClibc to its limits and see where things break. Back in 2011, I got bored of working with the usual set of embedded packages. So, while my students where writing their exams in Modern OS, I entertained myself just adding more and more packages to a stage3-amd64-hardened system [2] until I had a decent desktop. After playing with it on and off, I finally polished it where I thought others might enjoy it too and started pushing out releases. Recently, I found out that the folks behind uselessd [3] used Lilblue as their testing ground. uselessd is another response to systemd [4], something like eudev [5], which I maintain, so the irony here is too much not to mention! But that's another story …
There was only one interesting issue about this release. Generally I try to keep all releases about the same. I'm not constantly updating the list of packages in @world. I did remove pulseaudio this time around because it never did work right and I don't use it. I'll fix it in the future, but not yet! Instead, I concentrated on a much more interesting problem with a new release of e2fsprogs [6]. The problem started when upstream's commit 58229aaf removed a broken fallback syscall for fallocate64() on systems where the latter is unavailable [7]. There was nothing wrong with this commit, in fact, it was the correct thing to do. e4defrag.c used to have the following code:
#ifndef HAVE_FALLOCATE64 #warning Using locally defined fallocate syscall interface. #ifndef __NR_fallocate #error Your kernel headers dont define __NR_fallocate #endif /* * fallocate64() - Manipulate file space. * * @fd: defrag target file's descriptor. * @mode: process flag. * @offset: file offset. * @len: file size. */ static int fallocate64(int fd, int mode, loff_t offset, loff_t len) { return syscall(__NR_fallocate, fd, mode, offset, len); } #endif /* ! HAVE_FALLOCATE */
The idea was that, if a configure test for fallocate64() failed because it isn't available in your libc, but there is a system call for it in the kernel, then e4defrag would just make the syscall via your libc's indirect syscall() function. Seems simple enough, except that how system calls are dispatched is architecture and ABI dependant and the above is broken on 32-bit systems [8]. Of course, uClibc didn't have fallocate() so e4defrag failed to build after that commit. To my surprise, musl does have fallocate() so this wasn't a problem there, even though it is a Linux specific function and not in any standard.
My first approach was to patch e2fsprogs to use posix_fallocate() which is supposed to be equivalent to fallocate() when invoked with mode = 0. e4defrag calls fallocate() in mode = 0, so this seemed like a simple fix. However, this was not acceptable to Ts'o since he was worried that some libc might implement posix_fallocate() by brute force writing 0′s. That could be horribly slow for large allocations! This wasn't the case for uClibc's implementation but that didn't seem to make much difference upstream. Meh.
Rather than fight e2fsprogs, I sat down and hacked fallocate() into uClibc. Since both fallocate() and posix_fallocate(), and their LFS counterparts fallocate64() and posix_fallocate64(), make the same syscall, it was sufficient to isolate that in an internal function which both could make use of. That, plus a test suite, and Bernhard was kind enough to commit it to master [10]. Then a couple of backports, and uClibc's 0.9.33 branch now has the fix as well. Because there hasn't been a release of uClibc in about two years, I'm using the 0.9.33 branch HEAD for Lilblue, so the problem there was solved — I know its a little problematic, but it was either that or try to juggle dozens of patches.
The only thing that remains is to backport those fixes to vapier's patchset that he maintains for the uClibc ebuilds. Since my uClibc stage3′s don't use the 0.9.33 branch head, but the stable tree ebuilds which use the vanilla 0.9.33.2 release plus Mike's patchset, upgrading e2fsprogs is blocked for those stages.
This whole process may seem like a real pita, but this is exactly the sort of issues I like uncovering and cleaning up. So far, the feedback on the latest release is good. If you want to play with Lilblue and you don't have a free box, fire up VirtualBox or your emulator of choice and give it a try. You can download it from the experimental/amd64/uclibc off any mirror [11].
One of the interesting thing that I noticed after shellshock was the amount of probes for vulnerabilities that counted on webapp users to have direct network access. Not only
ping to known addresses to just verify the vulnerability, or
wget or
curl with unique IDs, but even very rough
nc or even
/dev/tcp connections to give remote shells. The fact that probes are there makes it logical to me to expect that for at least some of the systems these actually worked.
The reason why this piqued my interest is because I realized that most people don't do the one obvious step to mitigate this kind of problems by removing (or at least limiting) the access to the network of their web apps. So I decided it might be a worth idea to describe a moment why you should think of that. This is in part because I found out last year at LISA that not all sysadmins have enough training in development to immediately pick up how things work, and in part because I know that even if you're a programmer it might be counterintuitive for you to think that web apps should not have access, well, to the web.
Indeed, if you think of your app in the abstract, it has to have access to the network to serve the response to the users, right? But what happens generally is that you have some division between the web server and the app itself. People who have looked into Java in the early nougthies probably have heard of the term Application Server, which usually is present in form of Apache Tomcat or IBM WebSphere, but here is essentially the same "actor" for Rails app in the form of Passenger, or for PHP with the
php-fpm service. These "servers" are effectively self-contained environments for your app, that talk with the web server to receive user requests and serve them responses. This essentially mean that in the basic web interaction, there is no network access needed for the application service.
Things gets a bit more complicated in the Web 2.0 era though: OAuth2 requires your web app to talk, from the backend, with the authentication or data providers. Similarly even my blog needs to talk with some services, to either ping them to tell them that a new post is out, and to check with Akismet for blog comments that might or might not be spam. WordPress plugins that create thumbnails are known to exist and to have a bad history of security and they fetch external content, such as videos from YouTube and Vimeo, or images from Flickr and other hosting websites to process. So there is a good amount of network connectivity needed for web apps too. Which means that rather than just isolating apps from the network, what you need to implement is some sort of filter.
Now, there are plenty of ways to remove access to the network from your webapp: SElinux, GrSec RBAC, AppArmor, … but if you don't want to set up a complex security system, you can do the trick even with the bare minimum of the Linux kernel,
iptables and
CONFIG_NETFILTER_XT_MATCH_OWNER. Essentially what this allows you to do is to match (and thus filter) connections based of the originating (or destination) user. This of course only works if you can isolate your webapps on a separate user, which is definitely what you should do, but not necessarily what people are doing. Especially with things like
mod_perl or
mod_php, separating webapps in users is difficult – they run in-process with the webserver, and negate the split with the application server – but at least
php-fpm and Passenger allow for that quite easily. Running as separate users, by the way, has many more advantages than just network filtering, so start doing that now, no matter what.
Now depending on what webapp you have in front of you, you have different ways to achieve a near-perfect setup. In my case I have a few different applications running across my servers. My blog, a WordPress blog of a customer, phpMyAdmin for that database, and finally a webapp for an old customer which is essentially an ERP. These have different requirements so I'll start from the one that has the lowest.
The ERP app was designed to be as simple as possible: it's a basic Rails app that uses PostgreSQL to store data. The authentication is done by Apache via HTTP Basic Auth over HTTPS (no plaintext), so there is no OAuth2 or other backend interaction. The only expected connection is to the PostgreSQL server. Pretty similar the requirements for phpMyAdmin: it only has to interface with Apache and with the MySQL service it administers, and the authentication is also done on the HTTP side (also encrypted). For both these apps, your network policy is quite obvious: negate any outside connectivity. This becomes a matter of
iptables -A OUTPUT -o eth0 -m owner --uid-owner phpmyadmin -j REJECT — and the same for the other user.
The situation for the other two apps is a bit more complex: my blog wants to at least announce that there are new blog posts, and it needs to reach Akismet; both actions use HTTP and HTTPS. WordPress is a bit more complex because I don't have much control over it (it has a dedicated server, so I don't have to care), but I assume it mostly is also HTTP and HTTPS. The obvious idea would be to allow ports 80, 443 and 53 (for resolution). But you can do something better. You can put a proxy on your localhost, and force the webapp to go through it, either as a transparent proxy or by using the environment variable
http_proxy to convince the webapp to never connect directly to the web. Unfortunately that is not straight forward to implement as neither Passenger not
php-fpm has a clean way to pass environment variables per users.
What I've done is for now is to hack the
environment.rb file to set
ENV['http_proxy'] = 'http://127.0.0.1:3128/' so that Ruby will at least respect it. I'm still out for a solution for PHP unfortunately. In the case of Typo, this actually showed me two things I did not know: when looking at the admin dashboard, it'll make two main HTTP calls: one to Google Blog Search – which was shut down back in May – and one to Typo's version file — which is now a 404 page since the move to the Publify name. I'll be soon shutting down both implementations since I really don't need it. Indeed the Publify development still seems to go toward the "let's add all possible new features that other blogging sites have" without considering the actual scalability of the platform. I don't expect me to go back to it any time soon.
Two years ago, I took on the maintenance of thttpd, a web server written by Jef Poskanzer at ACME Labs [1]. The code hadn't been update in about 10 years and there were dozens of accumulated patches on the Gentoo tree, many of which addressed serious security issues. I emailed upstream and was told the project was "done" whatever that meant, so I was going to tree clean it. I expressed my intentions on the upstream mailing list when I got a bunch of "please don't!" from users. So rather than maintain a ton of patches, I forked the code, rewrote the build system to use autotools, and applied all the patch. I dubbed the fork sthttpd. There was no particular meaning to the "s". Maybe "still kicking"?
I put a git repo up on my server [2], got a mail list going [3], and set up bugzilla [4]. There hasn't been much activity but there was enough because it got noticed by someone who pushed it out in OpenBSD ports [5].
Today, I finally pushed out 2.27.0 after two years. This release takes care of a couple of new security issues: I fixed the world readable log problem, CVE-2013-0348 [6], and Vitezslav Cizek <email@example.com> from OpenSUSE fixed a possible DOS triggered by specially crafted .htpasswd. Bob Tennent added some code to correct headers for .svgz content, and Jean-Philippe Ouellet did some code cleanup. So it was time.
Web servers are not my style, but its tiny size and speed makes it perfect for embedded systems which are near and dear to my heart. I also make sure it compiles on *BSD and Linux with glibc, uClibc or musl. Not bad for a codebase which is over 10 years old! Kudos to Jef.