Massive Number of Vulnerabilities Found in X.Org - <AUTHOR> http://lists.x.org/archives/xorg-devel/2013-May/036276.html <comment><USER2> The fact that one must watch out for integer overflow on a buffer size calculation for malloc, realloc, etc. should be no surprise to a thoughtful C programmer. It sounds like somebody just did an audit of an old code base looking for bad patterns and this is what came out of it. It sounds like great work on the part of the people doing audits/fixes but I'm not sure it's freakout material ("massive" seems a bit sensationalist).

<comment><USER3> The only reason this isn't worth a freakout is that nobody running Linux in a graphical desktop configuration could ever have assumed that the machine was multiuser-safe to begin with. The reality is probably that no Linux (or FreeBSD) system in any configuration is secure from an attacker that can run native code in an unprivileged process. But with X11 running, you never had a chance.

I'm always amused by the dismissive tone programmers take to findings like this --- "oh, any thoughtful C programmer could avoid these problems". No, they can't. How many billions of dollars does it take to prove this point? The very best secure programmers in the world, building C (and C++) codebases from scratch specifically to address these problems, almost invariably fail to do so: memory corruption bugs get found in these systems just like they do in '90s-vintage X11 code.

C is my very favorite language and I'm not saying people shouldn't use it (well... they should avoid it, actually), but people should work in it with their eyes open.

<comment><USER4> At no time has there ever been a Linux kernel release with no local root exploits. There have been periods of uncertainty when we didn't know the exact exploit but we always find out eventually. Keep this in mind whenever you think about Linux security.

<comment><USER5> This is true for virtually all software.

That said, Linux users do have an unwarranted confidence in the security of the operating system. The kernel developers actually don't care that much about security--but because it was designed with some level of sanity, most installations are headless, and the non-headless installations aren't used by very many people, it has been able to tout some apparently obvious superiority to e.g. Windows (which isn't really the case post-Vista, when Microsoft had enough of being a punching bag.) Same thing goes for Mac--a whole lot of confidence, but not a lot to back it up. Curiously, a lot more Mac malware has shown up after they gained considerable market share.

There is no security in this world, only opportunity.

<comment><USER3> Careful. I'm not making a "Linux vs. Windows" argument here. I'm making a _Linux desktop_ vs. _any other computing environment_ argument. Not all computing environments are equally exposed to attacks. It is for instance much more annoying to exploit iOS vulnerabilities than it is to pivot from user "nobody" to root on an Ubuntu desktop system.

That these two environments aren't comparable --- one is a heavily-locked down and simplified computing environment and one is a general purpose desktop operating system --- is exactly my point.

<comment><USER5> I totally agree. It wouldn't be fair to compare Ubuntu to Chrome OS either. (My response was aimed more at thrownaway2424's comment, and I didn't mean to give the impression that everything is equally vulnerable, just that most software has exploitable bugs that may or may not be known.)

</comment></comment></comment></comment></comment></comment><comment><USER6> On a parallel note, I remember several years ago seeing Microsoft spruiking its 'Windows 2008 Server Headless'. Running a windows server without the gui was mentioned right there in the MS-generated marketing material as reducing the attack surface, as 70% of all discovered Windows exploits couldn't be done without the GUI components.

<comment><USER7> spruik [ sprook ]

        promote: to promote goods, services, or a cause by addressing people in a public place

I had no idea that was a word. I thought it was a typo until I couldn't figure out what it was supposed to be.

Thanks for teaching me something new today!

<comment><USER8> You know people who have those speakers and a microphone in stores and stand outside a store and try to sell?

They are known as spruikers.

<comment><USER9> I don't recall ever having seen that in the US.

<comment><USER10> Common enough in AU!

</comment></comment></comment></comment></comment><comment><USER11> He did a presentation on it that's much better reading:

[http://cansecwest.com/slides/2013/Assessing%20the%20Linux%20...](http://cansecwest.com/slides/2013/Assessing%20the%20Linux%20Desktop%20Security%20-%20Ilja%20van%20Sprundel.ppt)

<comment><USER12> PDF version? I would post one myself but <https://bugs.gentoo.org/show_bug.cgi?id=471112>

<comment><USER13> [https://docs.google.com/viewer?url=http%3A%2F%2Fcansecwest.c...](https://docs.google.com/viewer?url=http%3A%2F%2Fcansecwest.com%2Fslides%2F2013%2FAssessing%2520the%2520Linux%2520Desktop%2520Security%2520-%2520Ilja%2520van%2520Sprundel.ppt)

<comment><USER12> Cheers. I will use that URL next time. The comments about package managers ring true; I have a bug open on that right now for Gentoo: <https://bugs.gentoo.org/show_bug.cgi?id=453620>

</comment></comment></comment></comment><comment><USER14> Wow, that's huge! I'm wondering how this has not been noticed before.

I know, X11 is pretty old, but still, with the increasing awareness for security in the past, it's kind of funny nobody has ever checked such a prominent piece of software, not even in a Master's class.

Glad to see they are responding quickly.

<comment><USER15> > Wow, that's huge! I'm wondering how this has not been noticed before.

Most likely it's not huge. If it were, it would have been noticed before.

As to why it's not huge... That's a very interesting question.

<comment><USER16> Because usually you trust the x server.

This is saying that most client libraries are exploitable by a malicious x server.

A vulnerability in the x server itself, allowing a client to take control of it, would be rather more worrying to most people.

<comment><USER15> Yes, but how likely it is for your X server to get compromised? How often do you connect to your machine using an untrusted X server?

This is like saying your car can be stolen by anyone who has its key. Yes, it most likely can. Or that your house is vulnerable to anyone who's inside it.

<comment><USER17> This is likely why it's not been noticed before, I'd imagine the practical consequences of this are sort of moot: If the X Server is untrustworthy, you've likely got bigger problems.

I'll admit, my first thought was X11 Forwarding, as I understand you could have a bogus X server to cause junk to happen on clients on a remote server with X11 Forwarding enabled, which you'd already need credentials for. Again, if this was viable, the attacker having SSH credentials is a bigger problem.

I can't see any way for this to be really damaging in the wild, but I'm not a security guy so I could be totally off the mark.

</comment></comment></comment></comment></comment><comment><USER18> > The X.Org security team would like to take this opportunity to remind

> X client authors that current best practices suggest separating code

> that requires privileges from the GUI, to reduce the attack surface of

> issues like this.

A naïve reading of this statement seems to suggest that no "modern operating system" built on X can be secure should they follow the X.Org team's advice because no privileged clients can exist. Perhaps this is a gross misunderstanding of mine with respect to how distros use X, but is it not possible to display to the user a privileged password input window that can't be tampered with? And, if said window can be made, wouldn't it be privileged and thereby susceptible to this attack unless they displayed it without using an X server?

Or would the X.Org security team suggest that operating systems run dual X servers, or many X servers, one for each privilege level, and then render one on top of another?

This mess has me quite confused as to the security model surrounding X.

<comment><USER19> I think the recommendation is to have a non-privileged X client that invokes a privileged non-X program.

</comment></comment><comment><USER20> Lame attempt to find setuid/setgid files which link to an X library:

      find /bin/ /sbin/ /lib* /usr/ /var/ /root /home -type f \( -perm -4000 -o -perm -2000 \) -exec /bin/sh -c 'ldd {} | grep -q X && echo exists in {}' \;

Tools that have to read the shadow file (like xlock and xscreensaver) are sometimes setuid root. Apparently this is not necessary though: <http://www.shmoo.com/mail/bugtraq/nov98/msg00087.html> (My slack box is set up this way)

kppp is setuid root to perform operations only root can do, but supposedly does it in a thread separate from GUI operations. Perhaps Linux Capabilities could be used instead and setuid can finally be thrown out the window?

</comment><comment><USER21> These are actually vulnerabilities in Xlib, which is already well known to be pretty crufty. X11 security has always been a joke anyway, since basically any client can keylog the session at any point. There is some X security policy stuff, but it was never fully developed.

The interesting part here is that there's revived interest in _fixing_ Xlib.

And for the people talking about Wayland being the future: Let's not forget that client-side window decorations are not exactly going to be a security improvement.

<comment><USER22> > Let's not forget that client-side window decorations are not exactly going > to be a security improvement.

Why would it have anything to do with security?

<comment><USER21> If we care about giving applications _restricted_ access to the display and input devices, then we need to prevent those applications from fooling the user. Otherwise, a rogue application can basically do phishing, clickjacking, etc.

If we don't care about restricting access, then it doesn't make sense to call these bugs security vulnerabilities at all.

</comment></comment></comment><comment><USER23> The headline is inflammatory and probably misleading, I thought it was regarded as poor form to editorialize by rewriting headlines.

Yet again, in threads like this there are lots of "never done anything" armchair quarterbacks pontificating about how to write "massive" (correct use of the word) software libraries without regard to the age of the software in question. X is really old, security concerns were different then. X preceded Linux by a long way and it was the increasing usage of Linux which made X itself achieve widespread adoption. Yes, it is kind of shocking that no one has scoured it for vulnerabilities so far but having looked at the code a long while ago I wouldn't want to have done it, it is _awful_.

</comment><comment><USER24> The original title is clearer, because it's germane that the vulnerabilities are in the X client libraries rather than the X.Org server.

</comment><comment><USER25> It is easy to dismiss these vulnerabilities from a perspective of a common desktop user. "I already have privileges on this box" or "any desktop user already has a way to escalate privileges."

But X11 is also used in process control settings where crappy ancient programs run on decrepit old machines with modern X.org servers used for displays. Would we be so dismissive if someone revealed a host of major vulnerabilities in VNC or RDP clients? (Probably. :))

</comment><comment><USER26> Ilja van Sprundel, the guy who found these vulnerabilities, gave an interesting talk about "unusual bugs" at the CCC conference a couple of years ago. Might be interesting to some of you:

Video [http://media.ccc.de/browse/congress/2006/23C3-1456-en- unusua...](http://media.ccc.de/browse/congress/2006/23C3-1456-en- unusual_bugs.html)

Slides [https://dspace.it.su.se/dspace/bitstream/10102/316/1/Unusual...](https://dspace.it.su.se/dspace/bitstream/10102/316/1/Unusual+bugs+23c3.pdf)

</comment><comment><USER27> Has anyone done a security analysis on Weyland/Weston yet?

</comment><comment><USER28> X.org? Still vulnerable!? I am shocked, shocked, to see these holes still there. There's nothing really new here, what needs to be new is the will for programmers to finally go and fix this stuff.

</comment><comment><USER29> Let's bring Mir and Wayland already.

</comment><comment><USER30> These seem like such basic errors. Not validating responses because you trust the software you interact with to behave is such a basic mistake that you'd think the people that work on the X server from X.Org would have learned not to do that long ago.

<comment><USER31> It sounds like these errors predate X.Org _qua_ X.Org. A lot of people have had some responsibility for this code at some point without noticing this.

Bear in mind X11 is pretty old, and these are some of the oldest libraries in the whole X constellation. That isn't an "excuse", but it's a reason.

<comment><USER32> And when the stuff was originally written, this was probably not considered to be a "security boundary" in the sense that the client will have higher privileges than the server. As the email notes, this happens rather rarely.

<comment><USER33> Actually it was more common back then. Remember, "client" and "server" are backwards in the context of X. A "thin client" actually runs an X Server, and you remotely launch an xterm on the central server as an "x client", exported to your display.

However, as the email states, this only gets you the same access your user already had on the remote system, unless it's a setuid program. The canonical example and only one I can think of off the top of my head is xscreensaver or xlock. There are now GUI versions of su/sudo that would also be targets, but I don't think variants of these were used back when this topology was common.

<comment><USER34> This is a good read about how jwz coded xscreensaver to be secure and the pitfalls of using GUI toolkits:

<http://www.jwz.org/xscreensaver/toolkits.html>

</comment></comment></comment></comment></comment><comment><USER35> Shocker. Protocol issues. Time for another extension! Or to just shoot X.org in the head and quell the FUD surrounding Wayland.

<comment><USER36> > Or to just shoot X.org in the head and quell the FUD surrounding Wayland.

Yes, we should stop being so critical and automatically embrace everything new as much better than what came before it, due to being developed by agile superstar developers.

<comment><USER35> Are you kidding me? Or do you just make assumptions and accusations about technologies you don't know about?

"Super star agile developers"? That's the best FUD you can put in my mouth? By the way, you might look up the "superstar" developers behind Wayland: Ctrl+F "The developers": [http://en.wikipedia.org/wiki/Wayland_(display_server_protoco...](http://en.wikipedia.org/wiki/Wayland_\(display_server_protocol\)) ..... Huh, that's sure interesting.

 _Nowhere_ did I imply that Wayland was better than X simply because it was newer. That's the _exact_ kind of FUD I'm talking about from the X.org crowd that fears Mir and Wayland because they don't understand: the current technologies, the future technologies, the evolution of Wayland, the __DEVELOPERS __behind Wayland AND the HUGE list of flaws in X.

Stop spreading uninformed FUD, do _not_ put words in my mouth and don't accuse me of being a hipster.

<comment><USER37> This is not helping either of you or any of us. Stay clam, take some time out and reflect.

</comment></comment></comment></comment><comment><USER38> Today's HN is sponsored by MIR</comment>


Ask HN: Murphy's Law - <AUTHOR> It seems that as I'm getting done with beta testing and closer to a date where I can finally go public with my web app, it is like everything is going wrong.<p>Just wondering if anyone else has ever experienced a setback or a bunch of setbacks right before going public or right after going public. <comment><USER2> Often Murphy's Law is just poor planning.

1\. Two is one and one is none. Redundancy helps for critical situations.

2\. 95% of the code is usually done by launch, but that remaining 5% is often the hardest part that people procrastinated on. Usually certain kinds of bugs.

3\. People tend to cram in more features on the last minute, adding more bugs.

If anything projects should be planned out with lots of "do nothing" padding right before launch.

<comment><AUTHOR> #2 definitely seems to be where I'm falling in right now.. it's not that things don't work.. its just minor bugs that I'm catching.</comment></comment>


Taking FogBugz Forward - <AUTHOR> https://medium.com/make-better-software/taking-fogbugz-forward-f01f4fed275b <comment><AUTHOR> I'm a current and pretty-long-time (almost ten years?) FogBugz user. Right now I've got both a business and personal account.

Some things I'd like:

\- Markdown – I hate 'rich text'; the _Copy as Markdown..._ Chrome extension makes links so easy in GitHub and GitLab

\- A case description text field shown at the top of each case and with an edit history; with Markdown, task lists are so easily and nicely kept right up at the top

\- 'Cleaner' HTML for Vimium – some of the fancy (JavaScript) page elements don't work with Vimium; I hate using my mouse unless I'm playing a game

I'm planning on (slowly) migrating to GitLab, for both of my accounts. I'm using _gitlab.com_ for myself and we'll host GitLab internally at work. But I'll miss FogBugz's new-case-via-email. In GitLab there's a separate email address for each project and moving an issue between projects is actually closing the original issue and creating a new one (so messier). GitLab also persists in emailing you every comment, which would be fine if I wasn't the person sending the email. FogBugz treats email replies as a separate thing from regular comments.</comment>


Cyberattack exposes information of 15M LifeLabs Canadian customers - <AUTHOR> https://www.cbc.ca/news/canada/british-columbia/lifelabs-cyberattack-15-million-1.5399577 <comment><AUTHOR> The article claims 15M in Ontario and BC. That's about 3 out of 4 residents of those provinces. I guess there's a lot of old records in there. The scope is so large that pretty much every adult can now be intimidated by threats of releasing medical test results.</comment>


Introducing Bootstrap Studio - <AUTHOR> https://bootstrapstudio.io/ <comment><USER3> Great job, it looks really useful and something I would be interested in using in the future. Some feedback...

1 - The webpage could make it more obvious that the paid for version is a NodeJS based application. It took me a little time to work this out because I read the start of the page and immediately went to the online demo.

2 - Provide an option to choose between Bootstrap 3 and 4, even when 4 is fully released there will still be people stuck on 3 for quite some time. The ability to create/edit projects with either will help you a lot.

3 - Raise you price to at least $99. Any professional freelancer or design company will not think twice about this price point and can see it saves a lot more than this in time on just the first project. I think you should be aiming at professionals and not hobbyists. Odd though it sounds, the less people pay the more they complain about a product and the more demanding the support they ask for. Trust me, you could offer this for $10 and someone will still complain that your 100's of hours of hard work is worth only $5.

4 - Always specify a time limit of a special offer. This creates a sense of urgency and is the reason that offers work. Mention on your site when the offer expires, otherwise people don't know and so they will wait. Then they will miss the deadline and are either upset decide not to buy because they wanted the cheaper price.

5 - If you have a roadmap of additions then mention some of the changes to be released in the next year. This shows it is being actively developed and some of the new features might be the ones people are waiting for before buying.

</comment><comment><USER4> I know this is a Bootstrap tool, which means it has baggage.

But if something like this existed for email, I would buy it in a snap. I have yet to find a desktop tool that can make decent responsive html emails, and that is one of the main strengths that Bootstrap should offer. Skip the JQ efects.

Perhaps you can make a email mode that is JS free, strips out the unused CSS, and inlines the remaining rules?

<comment><USER5> You should look into [http://www.stampready.net](http://www.stampready.net) \- along with Campaign Monitor's email builder ([https://www.campaignmonitor.com/features/create-custom- email...](https://www.campaignmonitor.com/features/create-custom-emails/)), it's one of the best drag-and-drop tools for creating really good, responsive emails. The problem with any of these tools is that they still spit out sometimes convoluted code, but that can't really be avoided since they need to cover their asses in literally dozens of different popular email clients.

If you're looking to code, you should check out Litmus Builder ([http://litmus.com/email-builder](http://litmus.com/email-builder)), a code editor specifically built for email design. Has a bunch of templates available, instant previews in a bunch of different clients, email-specific CSS inlining, etc. Full disclosure: I work at Litmus, but even if I didn't, I'd still use Builder for the previews alone.

<comment><USER6> Thanks. Do these support responsive emails on the gmail android client? Unfortunately the gmail client does not support media queries, which makes building a responsive email for it very difficult.

<comment><USER5> That's a good question. I'm not sure about StampReady, but Campaign Monitor has a few of the best email designers in the biz working there (namely Nicole Merlin and Stig Morten Myre) who strive for really robust templates. Nicole in particular has written about her approach (typically called 'hybrid' or 'spongey' development), which works without media queries. It's basically using fluid tables, max-width, and MSO conditional tables to get things working and is the best approach around these days. You can read more about it here: [http://webdesign.tutsplus.com/tutorials/creating-a-future- pr...](http://webdesign.tutsplus.com/tutorials/creating-a-future-proof- responsive-email-without-media-queries--cms-23919)

I wouldn't be surprised if Campaign Monitor's email templates built in their builder follow the same approach. MailChimp's too, for that matter. The team there use similar techniques.

</comment></comment></comment></comment><comment><USER7> [http://pingendo.com/](http://pingendo.com/) : good free alternative.

<comment><USER8> Nice. Bootstrap 4 demo is available at: [http://v4.pingendo.com/playground.html](http://v4.pingendo.com/playground.html)

<comment><USER9> I got welcomed with the "Your browser is not supported, please use either Chrome or Safari".

Sorry guys, but if your little app doesn't work in Firefox, I don't care about it, it's probably not worth time to even look at it.

<comment><USER10> I gave it a 30 seconds test with Firefox and it seemed to work. Probably not a deep enough test to say that it works, but I guess it's not far from working. I tried the Bootstrap 4 version.

</comment></comment></comment></comment><comment><USER11> Looks super cool and very useful. If anyone thinks that $25 or $50 is expensive for 3 installs, they are just being disingenuous. That's less than most devs charge their clients per hour and tools like this easily save a few hours on each project.

And good to see another Bulgarian venture :-)

<comment><USER12> $50 is nothing for such a tool for a one time payment! Even at that price it is very difficult for the tool to make a profitable full time business model. I want to also add that the 50% discount is not a helpful message either.

I always ask my team, how is that as developers we always want free software but yet want a salary hike every year as a software programmer!

<comment><USER13> >I always ask my team, how is that as developers we always want free software but yet want a salary hike every year as a software programmer!

A lot of people are short-sighted and self-centered. It's completely and totally rational to buy this product if you're billing at $75/hr and this tool saves you 10+ hours that you couldn't justify billing.

Some people just don't "get it" \- that you must produce something of value before you can consume, and that every exchange must leave everyone better off. This is what used to upset me about the software pirates, torrenters, etc. It'll all catch up in the end though, so no need to over-analyze people like this.

<comment><USER14> I think a lot of people are not billing 75/hour or billing at all.

When you're at work, you typically just have to use the tools the rest of the team does. Then when you are at home, working on a side project... it's essentially a hobby and saving time in a hobby is just a 'nice to have'.

</comment></comment></comment></comment><comment><USER15> Wow this is gorgeous! Nice work.

I have a product in a similar space, Beaver Builder ([https://www.wpbeaverbuilder.com](https://www.wpbeaverbuilder.com)). A few differences being that it's an in-browser tool and it's a WordPress plugin.

Who is your target customer? Are you hoping to improve workflows for frontend developers or enable non-developers to build web pages? Also, what are your thoughts on maintaining a Bootstrap Studio site?

FWIW, we hit a nice niche with freelance web designers and web agencies. Drag and drop streamlines the development process and it also enables more tech- savvy clients to jump in and make their own edits and updates.

<comment><USER16> Sorry to go off topic, but i've been interested in your product for a couple of weeks. I recently created a site for young people interested in horses. I built it with a theme called KLEO that included Visual Composer. VC is having insane load times, partially because of admin-ajax.php. Does BeaverBuilder also make use of admin-ajax.php? Visual Composer is clearly too heavy and unoptimised for us, so i'd be more than likely to go with BeaverBuilder.

<comment><USER15> You've heard of us!? No kidding!? That's awesome to hear. :)

We use a lot of ajax when the actual page builder is in use, but we do our best to ensure that page load times for published pages are low as possible.

Outputting lean and efficient auto-generated markup (relative to what was out there) was one of our big goals from the start.

We have a free version of the software in the WordPress repo: [https://wordpress.org/plugins/beaver-builder-lite- version/](https://wordpress.org/plugins/beaver-builder-lite-version/)

You can demo the tool, peek at the markup, and see how it well it plays with your WP theme/install with the free version.

<comment><USER16> Hey Robby, thanks for the reply. Yeah, I saw it mentioned a couple of times on /r/webdev and /r/wordpress in threads where people were bashing VC while talking positively about your plugin ;-)

I'll try out the lite theme and hopefully find it fitting to our site. Once again, thanks for replying to my off-topic comment. I just had to take the chance since i've been considering your product lately!

<comment><USER15> My pleasure. Feel free to shoot me an email anytime, too:

robby [at] fastlinemedia [dot] com

</comment></comment></comment></comment></comment><comment><USER17> Is there a big market for WYSIWYG apps for developers? I think this is a great application but might it be better targeted toward non-technical people? Remember iWeb? This reminds me of a more developer-centric, flexible version of that. But no developer would actually build with it. It was for the people who now use Wix, Weebly, and Squarespace for their websites.

Developers should be able to put together a Bootstrap front end just as easily in code and probably prefer working in code.

Maybe there's a huge developer market for this and I just happen to not know anyone who'd be into this.

<comment><USER18> There is a whole spectrum of people who work "on the web" in some form or another: hard-core developers, front-end coders, designers, small business owners, content creators, etc... So any tool from Squarespace to BootstrapStudio to Emacs is potentially useful to a subset of the market.

<comment><USER17> But this is specifically marketed toward developers and designers. So my question is are there really enough of those developers and designers interested in a WYSIWYG took or would it be better to drop the Bootstrap focus and focus on the small business owners and content creators?

<comment><USER18> > But this is specifically marketed toward developers and designers.

"Developers and designers" encompasses quite a range of skills. Many designers don't know how to code or don't want to deal with it, or would rather do it in a visual editor rather than in a text editor.

On the other side, there are developers who aren't proficient in modern HTML, CSS, web technologies, and the obnoxious-to-set-up modern web tool chain. A GUI like this could be a handy way of bypassing some of these problems so they can get that website built quickly without it looking like it belongs on the 1990s web. (I admit this case is more rare.)

> would it be better to drop the Bootstrap focus and focus on the small > business owners and content creators?

I agree with you there, and that's actually what I'm working on. In my application the web developer/designer can create custom "components" with HTML and CSS and some rules on how these components can fit together, and the site owner / small business person / content creator can manipulate their site in a completely visual, drag-and-drop, edit-in-place environment. There are no mandatory tie-ins with any frameworks or libraries, and the system doesn't alter your markup or insert additional junk. Any valid HTML and CSS the developer puts in will work and will come out essentially as entered.

</comment></comment></comment></comment><comment><USER19> IMO this is another example of unnecessary fragmentation of the dev tools ecosystem. Why do we need a whole IDE for Bootstrap? Can't we instead make a plugin for an existing IDE to accomplish the same thing?

<comment><USER20> Are there any widely used WYSIWYG web editors left that this could be a plugin for? I can't remember seeing one of those in ages.

<comment><USER14> Good question. Most designers I know use Sketch and I'm pretty sure Sketch doesn't output HTML or CSS.

</comment></comment></comment><comment><USER21> I salute this initiative. A free 3-7 days trial would be so welcomed though, dispite the online demo which I assume it is not entirely the same as the desktop app. Looks good otherwise, and love the comments from other people as well. Love the price too.

</comment><comment><USER22> There is one more alternative that I tried before. Decent. [http://www.pinegrow.com](http://www.pinegrow.com)

</comment><comment><USER23> I find the existence of this kind tool to be strange, but that may just be because of my preferred way of working. Despite the fact that i'm not hugely keen on Bootstrap, I recognise that people find it useful. But do people really love it so much that they build an ecosystem and actual paid apps around it? It just seems like such a weird thing to focus your energy on.

<comment><USER24> Are you kidding? There are plenty of devs that build bootstrap front ends - <USER2> this makes life easier for them and saves time / effort, it's an easy decision to purchase this. There are other front end frameworks that with IDE's that people pay for too such as vaadin. You're probably not in the target audience for this.

</comment></comment><comment><USER25> Not Bootstrap specific but Macaw 1.6 [1] is now a free alternative since they were bought by InVision.

[1] [http://macaw.co](http://macaw.co)

<comment><USER26> "Will you still be updating Macaw? Version 1.6 will be the last update to Macaw."

</comment></comment><comment><USER27> This looks promising but I would need a free trial before paying for it.

<comment><USER28> I was actually hyped a little bit until I saw your comment. Forgot about the price.

$50 for 1 year free updates and only 3 installs is a little bit steep.

<comment><USER29> How much would you charge - if _you_ made it?

<comment><USER28> Same $50 but lifetime updates and no limited installs.

Charge me again for Bootstrap studio 3 or something but don't end updates for 2 after 1 year.

</comment></comment></comment></comment><comment><USER30> Wow. Very impressed. I'm a big fan of Webflow ([https://webflow.com](https://webflow.com)). I would love if I could use Bootstrap in Webflow's visual designer but this is great.

<comment><USER31> Isn't Webflow based on bootstrap?

</comment></comment><comment><USER32> Love it.

I'm very bad at website design and at 25$ this is a no brainer . Can see myself using this for small websites

<comment><USER33> ^^this

Pretty confident that i'll get my $25 from it...

</comment></comment><comment><USER34> Pretty. I hope it doesn't mangle markup like GUI tools (that tackled the problem) in the distant past did. Frontpage or Netscape Composer anyone?

<comment><USER35> GoLive, Dreamweaver too. It's hard to convey semantic intent and create efficient structures in markup when your tool has to be generic enough to have a flexible GUI.

It still comes down to how skillful the creator is, to not end up with nested divs and classes like "rt-col1-flex-span-head".

<comment><USER36> Adobe Contribute... i.e. Adobe code mangler.

That's been my experience in the past with these kind of tools as well. Its easy to make a mess.

</comment></comment></comment><comment><USER33> This is fantastic. Got more done in 5 minutes than I usually do in an hour of trial and error.

Any keyboard shortcuts available? At least a shortcut for Duplicate would be good.

Edit: Shortcuts here > [https://bootstrapstudio.io/pages/keyboard- shortcuts](https://bootstrapstudio.io/pages/keyboard-shortcuts)

</comment><comment> nevi-me $25! Thank you! I'm buying me a copy right now! Someone was saying you should charge $99 as freelance developers would buy it at that price, but as a hobbyist who $99 is how much I spend on almost monthly groceries because of the exchange rate, I'm glad that the price is at reach.

From looking at the site, this looks awesome, especially since I can import my own Bootstrap themes. I haven't used a visual CSS editor in over half a decade, and after the page refreshes that I spent time on just last night, I hope this will be a great tool for me to use.

Thanks again :)

EDIT: I see it comes bundled with Bootswatch themes, this is awesome as I use some of them! Great tool so far!

</comment><comment><USER37> I'm not a web dev, but I had a plan for some time now for a small sideproject and this seems it has a potential to accelerate things for me. Does it generate fairly vanilla/idiomatic bootstrap? I would take the generated html to the backend, but if I later decided to edit html directly I wouldn't want to deal with weirdness left by your software. What are your payment options? They seems to be hidden behind email form. Any weird drm? I seem to format/reinstall OS more frequently and wouldn't want to deal with problems arising from the software refusing to run when I change my distro.

<comment><USER38> I bought it about an hour ago. The code generation looks fairly standard. You get 3 installs, but you can deactivate them to use elsewhere (it seems)

</comment></comment><comment><USER39> Looks like a very useful product, I've tried and want to use these kinds of tools.

So far I tried Pinegrow and Bootply, I'm certainly going to give this one a try.

[http://pinegrow.com/](http://pinegrow.com/)

[http://www.bootply.com/](http://www.bootply.com/)

</comment><comment><USER40> I just made the purchase.

This tool will hopefully make my life easier. I'll still code everything carefully by hand but I have a hard time imagining how the UI will look like and I find myself spending quite some time coding and F5 repeatedly only to be disappointed by the result.

These thigs come with experience. I believe people that dedicate their time mostly to UI get real good and would not need a tool like this but for me I believe it'll do wonders.

</comment><comment><USER41> This looks really cool. It seems like you have a lot of granular control over the elements and layout. That being said, most of the websites I build these days don't need this level of customization. I've found that Blocs[1] is more my tempo. I just really want to organize structure at a high level, add content, polish, and export.

[1] - [https://www.blocsapp.com](https://www.blocsapp.com)

<comment><USER34> Nice, but Mac only. :/

</comment></comment><comment><USER39> Just a FYI, Chrome is giving me a warning on the HTTPS certificate (vulnerable SHA1 certificate), try to get one from LetsEncrypt or make an update to your StartSSL one.

</comment><comment><USER42> Cool. When I added a new page (untitled.html) and renamed it the page name change wasn't reflected back. Also, There appears to be a page tab for the page editor, but I'm only getting one tab while I have more than one page defined in the design.

</comment><comment><USER43> I have enjoyed layoutit.com for simple free wireframing.

You can define your layout pretty quickly if you have ever made a bootstrap layout before.

I know there are other alternatives as well but does anyone have any significant experience with this?Is this a lot better, is this the best one out there?

<comment><USER44> I'm no great designer and only doing hobby sites but I've found PineGrow ([http://pinegrow.com/](http://pinegrow.com/)) quite nice.

<comment><USER45> I tried using Pinegrow for my hobby stuff. It seems to work great but the price is a little steep for someone who is just playing around with side hobbies.

They hooked me on the $49 price when I started playing around and I was ready to buy until I saw that the price jumps to $99 if I want master page support and some other nice things.

I realize it is well worth it but I just can't justify a $100 editor for my hobby projects. I just purchased this tool for $25 as its pricing aligns with what I'm willing to spend much better.

</comment></comment></comment><comment><USER46> Trial is crucial before buying.

</comment><comment><USER47> Part of me really wants to hand this to the users I program an internal webapp for and let them spec the basic layout of the the pages, another part of me knows that is a no good, very bad, horrible idea. :/

</comment><comment><USER48> I can't believe the negativity in the comments.

This looks great and I'll probably be looking to purchase this. I assume that this would help me out when i'm fiddling around trying to make stuff look good in react.

</comment><comment><USER49> bought it a short while ago

please tell me there's a code you are using for that wonderful checkout. email-->entering postal code --> entering card --> done.

any reason why you are capturing postal code first?

I will post another comment with a review later

is this built using electron? where can you find a boiler template project complete with installation wizard?

edit: just realized you can't even import HTML files or I'm dumb. I clicked open but it only lets you select some proprietary file. This is a HUGE MINUS because I was looking forward to editing existing bootstrap template and you can't!

<comment><USER50> The checkout form is Paddle: [https://www.paddle.com/features/checkout/](https://www.paddle.com/features/checkout/)

Stripe also has the same thing: [https://stripe.com/checkout](https://stripe.com/checkout)

</comment></comment><comment><USER8> i think this can be achieved by a bootstrap snippet set; a decent text editor with live reload/files watch setup.

a dedicated app is not needed.

<comment><USER51> It's $25. How much time would it take someone starting from scratch to gather those snippets, choose a text editor, get it installed and configured, and set it up to use the snippets, watch files, etc.? If their time is worth anything at all I can understand the attraction of paying $25 to skip straight to the part where they're doing the actual work people pay them to do.

<comment><USER8> if you're a web developer, you should probably have those already.

</comment></comment></comment><comment><USER52> We've been using this for about 6 months to bang out prototypes very quickly. Happy with it so far.

</comment><comment><USER53> It looks really interesting

\---

But:

"Bootstrap Studio is a desktop application filled with powerful features."

That line tells me absolutely nothing about it

</comment><comment><USER54> What CSS framework are they using for the app [0]? It doesn't look like Bootstrap.

[0] [https://bootstrapstudio.io/demo/assets/css/styles.css](https://bootstrapstudio.io/demo/assets/css/styles.css)

</comment><comment><USER55> Just bought it after being quite impressed with the online demo, and figured it was worth a shot for $25.

First impression after downloading: an unsigned OS X app? Really? This is commercial software; it's not that expensive to get a dev certificate.

</comment><comment><USER56> Which is the OS they use in the video?

</comment><comment><USER57> Looks like a watered down Webflow

</comment><comment><USER58> great resource for developers to work on bootstrap. It would have more been interesting if free version was available.

</comment><comment><USER59> Only Chrome? Firefox please!

<comment><USER60> "Only Chrome" is just the demo. I assume the limitation is due to BootstrapStudio being built with Electron (I believe).

</comment></comment><comment><USER61> > Sorry, our online demo only works in Google Chrome for now.

Sorry, if you don't bother supporting Firefox or Safari, then I don't bother looking your website either.

(No, I don't ask you to support IE 8)

Edit: Didn't know it was desktop application. Explanation below.

<comment><USER62> In all fairness, it's a desktop application, not a browser application. So I wouldn't expect them to support every browser.

(written in Firefox)

<comment><USER61> I guess big percentage of users who visit that page never get idea that it is desktop app.

First lines (1200px height) don't mention desktop app.

>Introducing Bootstrap Studio

>A powerful web design tool for creating responsive websites using the Bootstrap framework.

Then there is call to action button, which says that we don't support you.

Most users then close tab.

</comment></comment></comment><comment><USER63> Guys you need to get a trial version up ASAP. Few people will risk real money on a tool from an unknown team without being able to try it out.

Now you're just wasting your marketing..

</comment><comment><USER64> It's unfair of me, but I'm always turned off when I see a desktop app that works on multiple platforms. That irrationally soured my first impression, because I think it otherwise looks like a great tool.

<comment><USER65> What? How on earth multiplatform support could be a bad thing?

<comment><USER4> If the developer were to have made a swift version for Mac, a C# version for Windows and a C++ version for Linux, that would be very bad. A startup simply cannot reliably keep all up to date, and the product would suffer.

If they wrote it in Java and in that way claim multiple platforms, I daresay it would be even worse, on all platforms.

If they used a tool that did the conversion for them, it would likely be a security and usability disaster, even though I need sources for that.

So a bias against a multi-platform tool is understandable.

The one exception is a Javascript tool, since things like Chromium have a LOT of effort put into them by companies with thousands of developers on payroll. However the number of tools that should run in such a environment is limitted.

<comment><USER66> Guess what, it is that one exception. Also, now we have lots of apps developed using electron, so welcome to 2016.

[http://electron.atom.io/](http://electron.atom.io/)

</comment></comment></comment></comment><comment><USER67> This looks very nice. But please move the video to the top of the page. Always remember engage first, then offer people a chance to try it out in the browser.

Also I had to reach the bottom of the page to find out that it was a desktop app. Is it built in Electron? One last tip if this works as well as it looks in the demo you could easily double your price.

<comment><USER68> The "Run Browser Demo" button engages me much better than video. I very rarely have patience for linear media.

<comment><USER69> But only in Chrome. So the best would be show the demo button only in Chrome and the video in other browsers.

<comment><USER68> Chrome-only demo is unfortunate in this particular case (I also use another webkit variant). If you ask me, scrollable page with screenshots is still better than video.</comment></comment></comment></comment>


CERN to Start Up the Large Hadron Collider. Now Here's How It Plans to Stop It. - <AUTHOR> http://spectrum.ieee.org/aug08/6558

<comment><USER2> Although the article is certainly interesting the headline is misleading. The article is from August. There is no news here as to when the LHC will be operational again.

</comment><comment><USER3> <http://news.ycombinator.com/item?id=276520></comment>


Use the (really) minimalist Google front page. - <AUTHOR> http://lifehacker.com/5377098/make-the-minimalist-fading-google-homepage-your-default If you think the Google front page is a little ... cluttered, try this cookie hack to see the uber-minimalist, fading front page that Google has been experimenting with. <comment><USER2> I don't like going to the front page or even using the browser's search box. Better to type CTR-L to get to the address bar and then use a shortcut registered "g". "g terms" and you are quickly at the results.

alias "g" to "[http://www.google.com/search?hl=en&q=%s](http://www.google.com/search?hl=en&q=%s)

Putting this in the address bar also means you get incremental history of previous searches. If the term was long, you don't need to type it again. If you need to modify a recent search slightly, it's there as a template.

<comment><USER3> I like using smart bookmarks like this for specialty searches, e.g. only searching Wikipedia

<http://www.google.com/search?q=%s+site%3Aen.wikipedia.org>

But if I'm searching the entire internet, I just hit CTRL+K to jump to the search box. Beatcha by two keystrokes! :-p

<comment><USER2> Agreed. But like I said, you won't get the benefits of the history incremental search. Also, I maybe should have mentioned I have a lot of other one letter shortcuts like that so "g" fits rights in.

<comment><USER4> Actually, you do get history for your searches, _and_ Google's autosuggest, in modern versions of Firefox. What you say used to be true, but from what I can see CTRL-K strictly dominates a bookmark keyword if you're starting from scratch. That said, if you've already got "CTRL-L g space" in your fingers, it's not so much better that you should switch.

<comment><USER2> I'm on Firefox 3.5 and didn't see that behavior.

<comment><USER4> In about:config, do you have browser.search.suggest.enabled set to True? I have a firefox right here and it's working fine.

(since that's probably obscure as the joke is old, that's a reference to the ancient copied at <http://www.ahajokes.com/lig069.html> )

</comment></comment></comment></comment></comment></comment><comment><USER5> We already do this. It's the "Search" box on Firefox, Safari, Opera, Galeon and Konqueror.

The only time I visit the google homepage to search something is when I'm stuck with Internet explorer.

<comment><USER6> Better yet, it's the location box in Chrome. Turns out, it's fairly trivial to tell the difference between a url and a search string. I wish firefox would adopt this design as well, it just makes sense.

</comment></comment><comment><USER7> I don't think I'll ever use this, but I love it, for the simple reason that it consolidates Google's famous blank design. Going to that page and seeing the logo and the blank box feels really comforting, like Google's making an effort to commit to their simplicity.

</comment><comment><USER8> <http://www.google.com/ncr>

</comment><comment><USER9> Love this: it's about remembering why people come to the page. Grandparent- proof.</comment>


Publishing with Apache Kafka at The New York Times - <AUTHOR> https://www.confluent.io/blog/publishing-apache-kafka-new-york-times/ <comment><USER3> The very definition of over-engineered. This is just event-sourcing turned into a marketing article for Kafka.

It doesn't really matter what the "source of truth" system is, although Kafka doesn't seem quite as mature/stable enough for that. With such little data, they can push into a nice graph database instead, run it entirely in memory and meet 10x any demand they'll ever see along with all the query types they'll need. Add in an elasticsearch cluster on the side and problem solved.

Any database can serve as a log replay, as long as you save all the versions then it's just called a query.

<comment><USER4> Unless I'm mistaken, If I were to build out a simple event log represented by a relational DB, I have bottle necks when writing to it, and have lag in terms of processing the events, and if I were also pushing those events to a queue to hydrate aggregate snapshots I would have to have client logic to deal with duplicate events or not acking processed events etc?

Intuitively, I guess kafka is "more realtime" and "more available" when compared to the home-brew event log?

EDIT: obviously those constraints in my home-brew event log are relaxed when my problem domain is amenable to things like associative operators, idempotency, inverses etc.

<comment><USER5> There's no reason not to make good use of Kafka or similar solutions. The issue is that people use it without understanding it. In this article, they say that Kafka is their system of record and their primary long-term storage. That's very silly.

<comment><USER6> Why is it silly to use as system of record or long term storage?

</comment></comment></comment></comment><comment><USER7> FWIW, the article mentions the book "Designing Data-Intensive Applications" by Martin Kleppmann. I wanted to throw out my own endorsement for the book, it's been instrumental in helping me design my own fairly intensive data pipeline.

<comment><USER8> Dear HN reader - if you're not quite ready to buy the book, take a listen to this episode of Software Engineering Daily ([https://softwareengineeringdaily.com/2017/05/02/data- intensi...](https://softwareengineeringdaily.com/2017/05/02/data-intensive- applications-with-martin-kleppmann/)). It will give you a sense of what Martin Kleppmann is all about and how he thinks about problems. I ordered my copy of "Designing Data-Intensive Applications" after listening to this episode.

</comment></comment><comment><USER9> Excellent, well written article. The key take away seems to be that instead of an temporary event stream log, since the number of news articles (and associated assets) is finite and cannot explode, they store all the "logs" forever (I'm using the term log as is defined in the article, as a unit of a time-ordered data structure).

I wonder if NYT can help other news websites by making their code open source? I'm a huge fan of NYT and their jump to digital has just been amazing. However, I would also like my local newspaper (which covers more regional news) to be able to serve quality digital content.

<comment><USER10> > _I wonder if NYT can help other news websites by making their code open > source?_

Hey! I, and a number of other news nerds have been encouraging FOSS for the past decade or so. And in fact a number of major open source projects have come out of news related projects, including Django, Backbone.js/Underscore.js, Rich Harris's work on Svelt.js, and a whole lot more.

Most often the problem with local news organizations are operational constraints. The news biz has seen a huge downturn over this same period of time. Most orgs, both on the reporting side and on the tech side are super strapped for people-time.

It's not enough to have FOSS software, you also have to have folks doing devops and maintaining systems often at below-market salaries.

</comment></comment><comment><USER11> This is a flawed architecture. It will work at release, but it will be difficult to manoeuvre with, and they will grow to hate it.

As your business changes, your data changes. Imagine if on day one, they had one author per article. On day 1000, they change this to be a list of authors.

Kafka messages are immutable. Each of those green boxes on the right hand side of the first diagram will need to have special-case logic to unpack the kafka stream, with knowledge of its changes (up until 17 May 2017, treat the data like this, but between then and 19 May 2017 do x, and after that do y).

Document pipelines is a rare instance of a context where XML is the best choice. They should have defined normalised file formats for each of their data structures. Something like the gateway on the left of the first diagram would write files in that format. (At some future time, they will need to modify the normalised formats. Files are good for that. You can change the gateway and your stored files in coordination.)

Secondly, they should have a gateway coming out of the file store. For each downstream consumer, they should have a distinct API.

These APIs might look the same on the first day of release. But they should be separate APIs so that you are free to refactor them independently.

When you have a one-to-one API relationship, you can negotiate significant refactors in a single phone call. When you have more than one codebase consuming, you need to have endless meetings and project managers. I call this, "The Principle of Two."

Some of the other comments here say that they should have used databases. So far, they have not made the case for it. And databases are easily abused in settings like this one. People connect multiple codebases to them, and use SQL as a chainsaw. Again, you can't negotiate changes.

When you create a system, your data structures are the centre of that system. You need to do everything you can to keep your options open to refactor them at a later time, and to do so in a way that respects APIs that you are offering your partners.

Kafka is a good tool. If used well, your deployment design will stop your system regularly (e.g. every day), nuke the channels, recreate them from scratch, and restart your system against these empty channels. You shouldn't use it as a long-term data store.

<comment><USER12> > Kafka messages are immutable. Each of those green boxes on the right hand > side of the first diagram will need to have special-case logic to unpack the > kafka stream, with knowledge of its changes (up until 17 May 2017, treat the > data like this, but between then and 19 May 2017 do x, and after that do y).

I respectfully disagree. The genius of this approach is that you can make the same transformation on the original Kafka stream to change its schema and prepare the new feed. Once you are satisfied with the results and you have switched all subscribers to the new feed, just turn off the old one. Voila - you only have y.

> This is a rare case where use of XML makes sense.

Sorry, but no. Just no.

<comment><USER13> I still don't understand the hatred around XML. Is it slightly verbose? Yes. Does it support lots of neat functionality that make it great for interoperating between systems, like validations and transformations? Yep. Sure, it's possible to go full architecture astronaut with it, but you can do that with pretty much any programming language.

Meanwhile, I'm just sitting over here wondering whether my YAML file is supposed to have certain indents here or not, "-" or not, and trying to go figure out which magic incantation I need to get it to handle a multi-line string the way I'm expecting.

<comment><USER14> I think parent was answering the usage of XML in this use case, which is not appropriate. XML has many strengths (as you have outlined), but it has also been (mis/ab)used so many times that it gained bad reputation. What my GGP suggests is an example of such. There is nothing to gain from XML here that any proper DB (with schema) wouldn't offer, or in this case, protobuf.

Kafka logs however are solving a different problem. The mental model is different - they do not record state, but the whole history of transactions, which makes it trivial to change the schema if/when need arises. Saying that the schema should be thought in advance and shouldn't change is not realistic IMHO.

</comment></comment></comment></comment><comment><USER15> > Traditionally, databases have been used as the source of truth ... [but] can > be difficult to manage in the long run. First, it's often tricky to change > the schema of a database. Adding and removing fields is not too hard, but > more fundamental schema changes can be difficult to organize without > downtime.

This argument sounds self-contradicting. Kafka doesn't let you change its schema at all! At least postgres gives you the option.

It seems that the author is excited about having a single source of truth that doesn't change, and didn't realize that he could do that with a database, if he just never used the schema-changing features.

Am I missing something? It seems like the author could be totally happy with a bunch of derived postgres databases sitting in front of a "source of truth" database, where he never changes the source of truth database's schema.

Why use kafka?

<comment><USER16> I tend to be the one arguing this, to stick to postgres for most things but even I will admit it does depend on scale.

I'm not sure what the NYT requirements are but from my understanding of Kafka, its persistent redundant distributed queues scale automatically horizontally across machines to support colossal amounts of data. It's possible that they had difficulty fitting everything in a postgres instance.

<comment><USER17> See, that's where I'm confused. I'm no Kafka expert, but they say they use a "single-partition topic" which I believe means the only way they can replicate the data is by replicating the entire log, they can't shard because it's a single partition. The reasoning behind this is because Kafka doesn't support ordering between partitions.

Also I've never thought of Kafka as a persistent data storage solution, it's interesting Confluent is supporting Kafka being used in this way.

<comment><USER18> You are not even a little bit confused! I think you have it perfectly right. And you are not alone in not thinking of Kafka as persistent storage, but when you get down to it there is no reason not to, and people are indeed using it in just that way. And yes, Confluent does give its +1 to this practice. :)

</comment></comment></comment></comment><comment><USER19> I wonder how much of this kind of stuff exists out of necessity and how much of it exists because very smart people are just bored and/or unsatisfied.

Are there any articles that supplement this that explain how much business value is added/lost by the existence/removal of these kind of features? In the case of NYT I suspect its popularity is maintained because of the perception (real or not) of high quality journalism, in spite of any technical failings.

\---

How much would be lost if NYT was just implemented as text articles that are cached and styled with some CSS. "Personalization" could be added by tags each article has and a small component that shows the three most recent articles that share the same tag.

<comment><USER5> >I wonder how much of this kind of stuff exists out of necessity and how much of it exists because very smart people are just bored and/or unsatisfied.

That's a ton of it. Like it or not, publishing a digital newspaper is _not_ a hard or unsolved problem; it's one of the web's core competencies. If you hire people who want to build cool stuff to supervise a CMS, well, you get this kind of outcome.

The raw cost is understated because these experimental setups misinterpret the functionality of the new architectures/formats they're using. It doesn't truly rear its ugly head until there is a major data loss or corruption event. It's not that these _never_ happen with RDBMS, it's just that RDBMS contemplates this possibility and tries to make it pretty hard to do that, whereas message queues just automatically delete stuff ( _by design_ , so they can serve as functional message queues!).

RDBMS have spoiled us and we take its featureset, 40+ years in the making, for granted. We need to be careful and not assume that `GROUP BY` is the only thing we leave on the table when we "adopt" (more accurately _abuse_ ) one of these new-wave solutions as a system of record.

Since no one is going to admit to their boss "this wouldn't have happened if we used Postgres", and since most bosses are not going to know what that means, most of these spectacular failures will never be accurately attributed to their true cause: developers putting their interest in trying new things above their duty to ensure their employer's systems are reliable, stable, and resilient.

<comment><USER20> There are non-negligible problems in the news space like:

      1. Supporting full-text search for a fair number of concurrent users
      2. Availability of the system with minimal downtime
      3. Scalability within the day and year, traffic patterns around e.g., breaking news events will far surpass 2AM traffic
      4. Notifications

I could go on and on but honestly, it's just a tone-deaf response.

Parting pot-shot: "No one is going to admit to their boss that the reason a worldwide news organization can't publish any stories is because their one postgres master node went down, or is waiting on a state transfer to a fallback master"

<comment><USER5> "We can't publish right now because the database had to enter an unplanned maintenance period" is a lot different from "our authoritative archive is gone and we have to try to rebuild it from all these separate 'materialized views', woops."

</comment></comment></comment></comment><comment><USER21> _This is very similar to a normalized model in a relational database, with many-to-many relationships between the assets._

 _In the example we have two articles that reference other assets. For instance, the byline is published separately, and then referenced by the two articles. All assets are identified using URIs of the form nyt: //article/577d0341-9a0a-46df-b454-ea0718026d30. We have a native asset browser that (using an OS-level scheme handler) lets us click on these URIs, see the asset in a JSON form, and follow references. The assets themselves are published to the Monolog as protobuf binaries._

When consuming this data do you have to programatically do relationship fetching on the client side or is eager loading/joins available in some way in Kafka?

Additionally there seems to be a focus on point-in-time specific views of this data, but are you able to construct views using arbitrary values/functions? Let's say each article is annotated with some geo data, can you construct regional versions of these materialized views of articles at the Kafka level? If not it seems like you are pushing a fair amount of existing sophisticated behavior at the RDBMS level up into custom built application servers.

</comment><comment><USER22> > Because the topic is single-partition, it needs to be stored on a single > disk, due to the way Kafka stores partitions. This is not a problem for us > in practice, since all our content is text produced by humans — our total > corpus right now is less than 100GB, and disks are growing bigger faster > than our journalists can write.

Before this line, the author mentions they also store images. There's no way that all their text + images is <100GB right? Something is inconsistent here.

<comment><USER23> More than likely they store references to images in kafka, with the actual image bytes being in a different store.

</comment></comment><comment><USER24> Interesting, we use rabbitmq instead of kafka and we have a re-indexation system... not sure if it's more complex for what I see.

</comment><comment><USER25> I also recommend the talk from Kafka summit. [https://kafka- summit.org/sessions/source-truth-new-york-time...](https://kafka- summit.org/sessions/source-truth-new-york-times-stores-every-piece-content- ever-published-kafka/)

</comment><comment><USER26> If all articles are put into monolog, what is the procedure to fetch all articles, lets say, published in year 1857? Will that be a O(n) operation (assuming all messages published in monolog have a timestamp field).

</comment><comment><USER27> Figure 3: The Monolog, containing all assets every published by The New York Times.

</comment><comment><USER28> Why not just have the 'logs' in s3 in sorted/indexed buckets.

</comment><comment><USER29> Resume driven architecture taken to extreme.

</comment><comment><USER30> Isn't this just event sourcing?

<comment><USER31> Yes it is, but for some reason they called it "Log-based architecture" in the article

<comment><USER32> They call it that because Kafka is a log-based datastore.

</comment></comment></comment><comment><USER5> >We need the log to retain all events forever, otherwise it is not possible to recreate a data store from scratch.

 _SIGH_. Cue the facepalm, head in hands, etc.

I'm not going to get into a big thing here. But if you find yourself saying "I need to keep this thing forever no matter what" and then you try to use something that even entertains the notion of automatic eviction/deletion semantics as the system of record, _you 're doing it wrong_.

Not to burst the bubble of the techno-hipsters, but Kafka is "durable" relative to message brokers like RabbitMQ, not relative to a system _actually designed_ to store decades of mission-critical data. Those systems are called "RDBMS".

Elsewhere in the article he says that they have less than 100GB of data and that it's mostly text. This is massive overarchitecture that isn't even covering the basic flanks that it thinks it is, such as data permanence.

I would really like to read the article that discusses why Postgres or MySQL _couldn 't_ have served this purpose equally well.

<comment><USER33> OOC what makes a RDBMS more durable that a Kafka? Both of them are systems for representing data on disk. I'd love to hear why one representation system is better at disaster recovery than another.

<comment><USER5> In Postgres, I never have to worry that the server will be accidentally loaded with `retention.bytes` or `retention.days` set too low and, as a result, choose to delete everything in the database, generating a wholly artificial "disaster" that can result in long periods of disruption or downtime (at a minimum; worst case is permanent data loss).

It is true that someone could issue `DROP DATABASE`, `rm -rf` the filesystem on the database server, or so forth, so my point is not that other systems are invincible. It's just that a properly-configured RDBMS is designed to take data integrity extremely seriously and provides numerous failsafes and protective mechanisms to try to ensure that any data "loss" is absolutely intentional.

On a RDBMS, things like foreign key constraints prevent deletion of dependent records, mature and well-defined access control systems prevent accidental or malicious record alteration, concurrency models and transactions keep data in a good state, etc. Kafka, on the other hand, is designed to automatically and silently delete/purge data whenever a couple of flags are flipped.

That is not a flaw in Kafka itself; it's designed to do that so that you don't have to interrupt your day and purge expired/old/processed data all the time. It's a flaw in architectures that misinterpret Kafka's log paradigm as a _replacement_ for a real data storage/retrieval/archive system.

I've had this argument countless times with people who've tried to use RabbitMQ as a system of record (if only for a few minutes while the messages sat in queue). There's just some fundamental disconnect for a lot of developers where they don't understand that something accepting the handoff doesn't mean that the data is inherently safe.

<comment><USER34> Kafka is a fine replacement for a RDBMS if it fits your particular use case. It has very strong data consistency guarantees - stronger than most RDBMS - <USER2> you configure it properly (acks=1 et al). It won't even lose data if the leader of a partition crashes during a commit.

It has been explicitly designed for these use cases and even has features like compaction:

[https://kafka.apache.org/documentation/#compaction](https://kafka.apache.org/documentation/#compaction)

Now, I agree with you that _in most cases_ , using Kafka as your primary data store instead of a RDBMS is madness - but that doesn't mean it's a bad idea in general.

<comment><USER35> Isn't that what 'in general' means?</comment></comment></comment></comment></comment>


Eritrea, where ATMs are unknown and SIM cards are like gold dust - <AUTHOR> https://www.bbc.co.uk/news/world-africa-49727573 <comment><AUTHOR> _There are no ATMs in Eritrea. The car-buyer told us that when the border with Ethiopia opened last year following the end of the "state of war" between the two nations, he went to Mekelle city in northern Ethiopia and his ''jaw dropped'' when he saw ''people taking a large amount of money out of a machine''._

It's incredible and harrowing how different their experience is from our own.</comment>