MagicaVoxel – a free 8-bit voxel editor - <AUTHOR> http://ephtracy.github.io/ <comment><USER2> Here are some examples of what talented artist can create with this tool:

[http://imgur.com/gallery/8zEE1/](http://imgur.com/gallery/8zEE1/)

<comment><USER3> The creator also retweets a lot of examples on Twitter: [https://twitter.com/ephtracy](https://twitter.com/ephtracy)

</comment></comment><comment><USER4> Any chance of a Linux version at some point?

Edit: At least it seems to run well under Wine. I suppose a Linux version isn't too strong of a priority.

</comment><comment><USER5> I wish the creator would maintain a blog about the project. It is cool to follow him on twitter and watch him work out the technical details of new features (e.g. marching cubes based mesh extraction). There is some definite interesting technical meat in implementing these tools in the context of a voxel engine.

</comment><comment><USER6> What does the '8-bit' refer to?

<comment><USER7> Aesthetics

<comment><USER8> Which is misused since, if anything, all pictures I've seen resemble 3D projections of games from the 16 bit/SNES era.

<comment><USER9> Most people don't realize how limited the palette actually is on NES. The entire gamut is only 54 colors (nominally 64, but there are duplicates and an "unsafe" entry), and any 16x16 block can only have 4 colors. There are some ways to work around this with clever hacks, but it was mostly the artists being clever rather than the programmers.

<comment><USER8> Huh oddly that explains a lot of the aesthetics of NES games.

</comment></comment></comment></comment></comment><comment><USER10> What is the license for MagicaVoxel?

<comment><USER11> Seems to be closed-source.

<comment><USER12> A bit parasitic to use Github for their hosting.

<comment><USER13> How so? Is that not what the paid tiers are for?

<comment><USER10> If they are using them!

</comment></comment></comment></comment></comment><comment><USER14> This is extremely impressive.

I've tried a wide variety of voxel modeling tools, free and not free, and this one's by far the most intuitive, and prettiest, I've found - in that it's the only one I haven't given up on in intense annoyance after 10 minutes or so.

Might end up using this for an animation project - thanks to the OP!

</comment><comment><USER15> this is very interesting. I'm wondering how the author built the ui for the app. it's cross-platform, but I don't see it links any other ui libraries. my conclusion is that its ui is built directly with opengl.

</comment><comment><USER16> The screenshot looks like a level from the Escher-esque game Monument Valley -- was MagicaVoxel used in developing that?

<comment><USER17> I don't know [1], but!... When you reduce things down to a minimum amount of essential detail, things tend to look similar to other things that share the same constraints. I noticed this when researching Mondrian and the De Stijl (neoplasticism) art movement -- they restricted their palette to mostly primary colors. I could ask if Google's logo or the colors in a bucket of LEGO bricks were inspired by Mondrian directly, but another possibility is that the neoplasticists discovered some kind of minimal, universal design pattern that feels timeless and likely to be "rediscovered" again and again. Are Monument Valley, MagicaVoxel (and Minecraft) inspired by Escher, or is it a natural coincidence that comes out of the minimal use of geometry and perspective they all employ?

[1] Edit: I did some more digging... Looks like they used Unity to create Monument Valley: [https://unity3d.com/showcase/case-stories/monument- valley](https://unity3d.com/showcase/case-stories/monument-valley)

</comment></comment><comment><USER18> Looks great, any reason why CMD-Q doesn't work for quitting the application?

<comment><USER19> No idea, but the Q has been mapped to move the camera (WASDQE). So maybe CMD-Q does something else?

</comment></comment><comment><USER20> I've used this in the past and found it to be great. The UI took a little getting used to, but once you did it was fast to accomplish fairly complicated things. Definitely worth a try.

</comment><comment><USER21> Can you use this to create object things in games for say the Unity3D platform?

<comment><USER22> Yes you can export the models into a standard 3D format used by any number of engines and loaders.

</comment></comment><comment><USER23> Yup, I've used MagicaVoxel on occasion and it's quite a bit of fun.

</comment><comment><USER24> Well Done

</comment><comment><USER25> Well done! — I so tweeted this! This will enrich so many people's lives.

</comment><comment><USER26> Emm, they aren't voxels. Square polygons are not volumetric pixels. Looks nice though!

<comment><USER27> This editor represents data points on a regularly spaced, three-dimensional grid. The data is undoubtedly voxel based.

<comment><USER26> You're mistaken.

<comment><USER28> This is not what I think of as voxels either, but I fear our older definition is becoming obsolete. You could make arguments both for and against whether these textured cubes are technically voxels, but in the end, language changes, and trying to fight these changes is futile.

<comment><USER29> What do you think of as voxels? It seems like a value at a point in a three dimensional grid is the volume equivalent of a pixel.

<comment><USER28> Right, my objection to the textured cubes is that they are not the volume equivalent of a pixel. Pixels are point samples without shape or texture. These cubes have both.

What I call a voxel is much smaller (on the order of a screen pixel), and drawn as a point sprite or screen-facing quad with a single color. Texturing is achieved by having many voxels with different colors/materials.

Here are some videos of what we used to call voxels:

DOS game: [https://www.youtube.com/watch?v=51E_G7NCXVM](https://www.youtube.com/watch?v=51E_G7NCXVM)

Gameboy game: [https://www.youtube.com/watch?v=3zVy-4CEFQU](https://www.youtube.com/watch?v=3zVy-4CEFQU)

Medical imaging: [https://www.youtube.com/watch?v=GrLqFNhVL68](https://www.youtube.com/watch?v=GrLqFNhVL68)

Note that they were not using voxels because they had a certain artistic style to them. They used voxels because with the hardware they had, this was the most realistic rendering they could achieve. Once GPUs came out with hardware- accelerated triangle rasterization, voxel engines died.

For more information about point-sampling, see this classic paper, "A Pixel Is Not A Little Square":

[http://alvyray.com/Memos/CG/Microsoft/6_pixel.pdf](http://alvyray.com/Memos/CG/Microsoft/6_pixel.pdf)

And for a more in-depth discussion of sampling, albeit in one dimension, watch Chris Montgomery's video "Digital Show and Tell":

[http://xiph.org/video/vid2.shtml](http://xiph.org/video/vid2.shtml)

<comment><USER30> I'm a bit confused about what distinction you're trying to make, or how this project represents a change in the meaning of the word 'voxel'.

Voxels are "volumetric cells" just like pixels are "picture cells". The size of them has absolutely nothing to do with it, and never has. Big voxels are still voxels. The data in the file is specified as volumetric samples... Maybe the issue is the texture you're talking about, but I haven't seen any textures in any of the galleries, other than the few examples that render mesh edges in addition to the cubes. Where are you seeing textures?

Six square polygons enclosing a space is surely just as 'voxel' as meshes generated by marching cubes for medical imaging. I've worked in medial imaging, written more than one volume renderer and used used many more than that, for both games and research. As far as I'm concerned, Magica Voxel is both truly voxel and truly awesome, IMO.

Alvy Ray Smith's paper is great, but how to handle sampling is a different subject completely from whether the scenes are volumetric, which they are. These voxels do have a certain artistic style to them, and the aesthetic choice to give them sharp edges was made intentionally and called out explicitly, i.e., "8-bit".</comment></comment></comment></comment></comment></comment></comment>


Dbeaver – Multi-platform database tool - <AUTHOR> https://dbeaver.io/ <comment><USER19> I can't believe how hard it is to find a good database client tool that runs on Linux. I spent hours on Sunday looking for something that worked as well as pgadmin3. I don't prefer these Java tools that force you to manage JDBC drivers and usually share dated and cluttered UIs. Plus, I can't stand having to manage connections to each database instead of to each server. Let me connect to the server and then select my dB.

My notes:

      - <USER2>
      - https://wiki.postgresql.org/wiki/PostgreSQL_Clients
      - https://wiki.postgresql.org/wiki/Community_Guide_to_PostgreSQL_GUI_Tools
      - <USER3>
      - Most clients are Java, which is a pain b/c
        - they require you to list the DB in the connection
        - JDBC drivers can not use a socket connection
        - UI isn't great
      - Valentina Studio
        - Has free & pro ($199) versions
        - Was the easiest to get connected initially, but then found bug that showed it wasn't using socket connection as expected.
      - <USER4>
        - Linux version is a Wine app
      - RazorSQL ($100 / year)
        - Is a Java app
        - Not much different from SQL Workbench, just use that instead?
      - <USER5>
        - Electron, AppImage
        - Wanted me to enter an email address just to use the software, became privacy concerned about how much stuff would get sent to their servers
      - DbVisualizer ($197, $69 renewal)
        - <USER10>
        - Downloaded & tried it, typical java app
      - SQL Workbench/J (OSS)
        - <USER10>
      - <USER8>
        - web based but still Java?
      - <USER9>
        - too crude
      - JetBrains DataGrip ($199 / $159)
        - <USER10>
      - <USER11>
        - has lots of errors when used with PostgreSQL 11
      - <USER12>
        - installed, couldn't get past the splash screen which said it couldn't find the application server
      - <USER13>
        - <USER14>
        - Couldn't get a DB connection to save
      - <USER15>
        - Windows only
      - <USER16>
        - No longer developed
      - <USER17>
        - Windows only
      - <USER18>
        - OSS and enterprise version

<comment><USER20> > JDBC drivers can not use a socket connection

As someone that has done lots of Java development you have lost me there, as JDBC drivers obviously do make use of network connections, usually TCP/IP sockets.

<comment><USER21> they are probably referring to unix sockets. I used dbeaver already and got stuck on this very problem.

<comment><USER20> Ah ok, I seldom see them being used though.

</comment></comment></comment></comment><comment><USER22> Dbeaver is the best FOSS database UI tool hands down. I've done a lot of searching and haven't found anything close. Managed to convert my whole office to it in a matter of months.

Favorite features:

Well integrated ER diagrams that let you quickly see parts of the db related to current table, or the whole thing if you want.

SQL generation from structure or data edits. Great for cranking out DB migrations like a boss.

Ability to maintain many database connections at once, and support for doing stuff before/after connecting which we widely abuse to setup SSH and VPN connections.

Very liberal database support. My original search for a tool like DBeaver was from the need to connect to MySQL, Postgres, and SQL Server and not wanting to use a bunch of different tools. DBeaver supports roughly anything Java supports, which is everything.

Wide OS support. Good luck finding another tool that can connect to MS SQL Server on MacOs that isn't garbage or money.

The icon is also adorable

The only downside is a bit of classic FOSS UI clunk. Not nearly as rediculous as Eclipse or Blender or the like, but it takes a few days to get comfortable.

<comment><USER23> Just installed it for the first time in a few years.

\- Again I wonder why it doesn't show the tables first, I have to go through Schemas -> Public.

\- 480 MB of RES RAM, I'm connected to one smallish Postgres DB.

\- I can't even find "just show me the contents of the table, like 'SELECT * FROM'.

I know, maybe it's tailored for Enterprise Database Architects, but this is my smoke test for a DB GUIs:

"How long and how many clicks does it take so that it provides more value than: 'psql; \dt+; SELECT * FROM X;'"

<comment><USER22> Maybe I've been jaded by SSMS and Mysql Management Studio heh. DBeaver actually has a better UI than most DB tools I've used, even though that's still not saying a lot.

Mouse over the buttons for an excessive amount of time and the tooltips will tell ya what they do.

Double click a table and go to "data" tab to see contents.

Schemas ->public is just a side effect of supporting a lot of different databases. Many support schemas

Ram is because default java heap size, you could probably change launch params to get it much lower though.

On all counts other DB tools I've used are just as obtuse and burn tons of resources. Most are written in Java or another high-level language because it's nigh impossible to get DB drivers for a ton of different databases working together in C. DBeaver is basically leveraging JDBC for what it was designed for

<comment><USER23> I don't know those 2, but I like Sequel Pro and pgadmin.

And yeah, I found "data" and maybe reading the manual would even let me somehow get there quicker. Still, I don't remember ever working with Postgres (or Oracle) in whatever role or project and not needing the "data" view 10 times as often as the schemas or other stuff :)

<comment><USER22> I agree completely. The UI is kinda hokey but I still love it because I've seen so much worse. Try giving it a few weeks, as you need the more advanced stuff you'll learn where the bodies are buried and it becomes semi pleasant to use

</comment></comment></comment></comment></comment><comment><USER24> How is no one talking about [https://www.jetbrains.com/datagrip/](https://www.jetbrains.com/datagrip/)

These guys actually fix their bugs. It works on every database known to humans, and also does nearly everything DBeaver appears to do.

<comment><USER25> The software is great but I would never buy it purely because of their business model.

There is absolutely no reason DataGrip needs to be a subscription service. There aren't any ongoing costs for customers to use it.

It should be a one-off purchase with optional support subscription, maybe requiring re-purchase at major version increments, not this bullshit where you need to maintain a subscription for a license on a binary program you already paid for and installed.

Feels as dodgy as Adobe turning Photoshop into a subscription service

Fuck these business models and the sales idiots who try to apply them to every single product

<comment><USER26> I used to think like this too, but I've since changed my mind and understand why they do this -- and it can even be beneficial for users. There's a few big problems with the old model:

First, you have to define the difference between "major" and "minor" version. Users expect major versions to have some significant improvements or new features. What this means is as a developer, you have incentive not to release minor features regularly, but instead batch them together so you can do a "major" release.

Second, the sales team now dictates the release cycle. More major releases means more money, but do it too frequently and the user base revolts. In many companies sales dictates the releases, but now if development is late it's messing with the company's cash flow and ability to continue to exist.

Third, the software can never be "done", because that means no more major releases. As a result, unless there is a sustainable stream of new users, you get feature and scope creep.

Subscriptions mean predictable revenue, and make the developer's incentive align with users: keeping users happy. This means keeping the product stable, making incremental improvements, and evolving with features that make sense.

Of course the beauty of the market is you should be able to find products sold both ways, and you can choose.

<comment><USER27> I absolutely agree, subscriptions are the way to go for making your software business sustainable and better scalable R&D investments to maintain growth. Gitlab is an example of this in the dev tooling space.

I've bought Mailplane v3 for my Mac a few years back. The app did not get new features, only patches. New features came out with v4, which required another purchase. I didn't buy it because v3 is good enough, buying again is a psychological wall. If it was a subscription I'd be happily paying them and happily using the latest version.

<comment><USER25> Gitlab is a terrible example, they are charging for a SERVICE of providing hosting and support for their higher tiers.

They are not charging a subscription for you to be allowed to download Gitlab and self-host it.

</comment></comment></comment></comment></comment><comment><USER28> I use DBeaver all the time and love it.

MySQL, Postgres, SQLite, etc...all the tools I need in one app.

I generally try to avoid Eclipse-based projects because they can be very bloated, but in this case it's easily justifiable.

<comment><USER29> Does it provide any linting or other code checks? I'm curious because I'm looking for something that can teach me proper SQL coding style and practices as I write it.

<comment><USER30> Yes. At least for Postgres autocomplete is very handy.

</comment></comment></comment><comment><USER31> Been a user of DBeaver for years - across all my platforms. Its does most things well enough that it has not been a problem to use across SQLite, MSSQL, MySQL and various file based data. Easier to use than SQuirreL which used to be my fav. All of these are java tools I think primarily because of starting from JDBC for which there seems to be a driver for just about everything on the planet.

So DBeaver scratches my itch of needing same tool across platforms, across databases.

</comment><comment><USER32> I like to try out new tools likes this, but whilst on my mac, and I generally use Sequel Pro for my databases (I use mysql and mariadb drop-ins). I have my database locally in a docker container, where I forward 3306 to 3300. Entering host as 127.0.0.1 with correct credentials and port, it gives me this error: "Could not connect: Access denied for user 'root'@'172.18.0.1'". This does not happen with Sequel, but occurs with Dbeaver. It actually also happened trying TablePlus which is a piece of software that a fellow commenter was mentioning. I'm sure that it could be solved in my configs by allowing that particular IP, which is the local IP of the virtual machine which is serving docker. However, that is supposed to be completely transparent, thus I should be able to connect through 127.0.0.1. As I haven't looked at the source code, it seems like the host lookup logic is flawed, as it translates my `127.0.0.1` to `172.18.0.1`. Why? Dunno. But this occurred in both Dbeaver and TablePlus. Works in my terminal and in Sequel however.

<comment><USER33> This sounds like a docker network problem more than an issue with dbeaver.

Try binding MySQL to 0.0.0.0 (all interfaces) instead of 127.0.0.1 (loopback). That said, 127.0.0.1 should be fine provided you connect through the service name or the docker container IP), connecting via 127.0.0.1 doesn't make sense. Then again, it's docker on Mac, so I don't really know.

Fwiw, also used sequel pro on Mac, moving to Linux dbeaver was the only thing that came close.

Now I don't really mind either but dbeaver is a bit quirky overall.

<comment><USER32> I know I can solve that by allowing "everytuldhing" to connect, but that is not my point. Hence my ""I'm sure that it could be solved in my configs by allowing that particular IP"" comment. That it's `Docker on mac` is kinda irrelevant, as it's still just a virtual machine with a somewhat static IP. My point is, that when it comes to transparency on that level, this piece of software does not acknowledge that, but Sequel Pro does. I'm not a particular fan of Sequel Pro, but it just works in these types of case. I'm still expecting, that software like this, would be able to handle this type of "proxy mechanism", so from outside the docker container, I could still connect through 127.0.0.1 to it, thus making the default config adequate. Not sure if this is even makes sense, but I feel, that when connecting through a binding docker port, it should still use the origin IP ie. 127.0.0.1

</comment></comment></comment><comment><USER34> Woah, that ER diagram functionality reminds me of a tool that I tried a long time ago but which I had since forgotten. Back then the tool in question had too many issues and shortcomings, but they may have developed it sufficiently now.

[https://pgmodeler.io/](https://pgmodeler.io/)

Being told about Dbeaver and reminded of pgModeler right now is great because I am currently defining the tables for a project I am working on for a client, and as the number of tables grow jumping back and forth between the files that define each of them adds a still small but noticeable overhead. With a ER diagram tool I might be able to speed up a little part of the definition work.

Edit: Hmm, pgModeler might still not be such a great tool [https://news.ycombinator.com/item?id=17693582](https://news.ycombinator.com/item?id=17693582)

</comment><comment><USER35> This is a great tool and it makes my life so much easier, so much better than pgadmin4

</comment><comment><USER36> I stared using Dbeaver a couple months ago, after looking around for an alternative to HeidiSQL on my Windows 10 laptop at work.

Used HeidiSQl for a long time but got fed up with issues with it freezing or crashing, especially when switching DB connections.

Dbeaver has been fantastic! Has some feautures Heidi didn't have, very stable & better looking UI. Thumbs up from me, definitely worth giving a try!

<comment><USER37> Nice to see someone that used HeidiSQL. I'm on the same boat. It feels like an abandoned project. I guess it's finally time to install DBeaver and say goodbye to Heidi.

<comment><USER38> >feels like an abandoned project

HeidiSQL feels like a finished product to me. Really no complaints at all and if you actually take a look at the commits, you'll see that the SINGLE developer puts in a whole lot of effort.

[https://github.com/HeidiSQL/HeidiSQL/commits/master](https://github.com/HeidiSQL/HeidiSQL/commits/master)

The guy is great and calling the project abandoned is really not fair.

</comment></comment></comment><comment><USER38> If only it didn't have such a PTSD-inducing, eclipse-based UI.

HeidiSQL (with WINE) is still by far the best SQL client for me, because the UI is simple yet efficient and very clean.

</comment><comment><USER39> I've had better experiences with DataGrip, but if I wasn't at a place that would pay for proper tools, DBeaver is an OK substitute. Neither, though, have the big thing I want in an analytical SQL workspace:

Simple charts

A whole lot of my work ends up with a three- or four-column table with one column of buckets, and the other being counts of how many things ended up in that bucket. I'd love a view where I could render that into a line or bar chart as a sanity check, and have it update in real-time as I change my query. Instead, I end up having to constantly copy/paste into Excel and do it there.

Yes, I should be using a proper BI or visualization tool for real reports. But sometimes I just need a quick line chart to show that yes, that one count is going in the right direction.

<comment><USER40> Have a look at Metabase. It's open source, web based and runs from a single JAR. Not the same as a full fledged management tool of course, but I like it for simple queries and charts and the price!

DbVisualizer also has built in charts that are simple but... not free/open source.

<comment><USER40> Or if you don't mind the web based approach, Apache Zeppelin too.

</comment></comment></comment><comment><USER41> I've used DBeaver at work on Linux to connect to Microsoft SQL servers and also to some PostgreSQL servers.

It's not _perfect_ but it's not bad. My experience was annoying (at best) to get drivers to work; particular the tsql ones (since Microsoft didn't provide Linux drivers at the time -- I don't know if that's been rectified). Once they're working though, everything else is fairly intuitive.

It's been a couple years since I moved to a different team and no longer need to connect to databases (and can't anymore... yay security). But I've considered using DBeaver at home for personal use.

</comment><comment><USER42> I've used TablePlus before (Mac+Windows); its really good, and pretty affordable at a $60 one time charge.

[https://tableplus.io/](https://tableplus.io/)

<comment><USER43> TablePlus is very nice and it supports NoSQL databases like Cassandra, CockroachDB, and Redis. It's Electron-based, so it is relatively heavyweight, but I like it a lot.

<comment><USER44> It is not Electron, it's native. The reason it's heavy because it contains the database drivers, some of them is hundreds of MBs (Oracle > 200MB).

<comment><USER43> Pardon me - I've checked and you are right. The look&feel really threw me off here, and I could've sworn it's browser-based.

</comment></comment></comment></comment><comment><USER45> I like DBeaver but I've been having some problems with how it handles timeout of connections. I often end up sitting with it unusable for up to 2 minutes because it can't seem to understand that a connection is dead, despite me setting all the available options to use a shorter timeout. I'm sure this is partly due to my specific network conditions but it nearly destroys its usability while other tools (eg: SquirrelSQL) just don't seem to have the same problem.

</comment><comment><USER46> All of my colleagues use postico (I'm the only Linux user at my company), which seems quite nice. Can anyone who knows both tell me how this compares to that?

<comment><USER47> I've used postico, navicat(mac/Linux) and dbeaver.

Postico is very similar to Sequel Pro, simple but effective.

Navicat is the more feature complete option, but using wine on Linux is not cool.

I'm very happy with dbeaver.

</comment></comment><comment><USER48> I prefer TablePlus personally, DBeaver just feels clunky in comparison.

<comment><USER49> No Linux support :(

</comment></comment><comment><USER50> I love DBeaver.

Long time ago I started searching for a DB-client similar to "TOAD", but on Linux and targeting only my following key requirements: 1) write & execute single SQLs on a "page" which has multiple SQLs without having to use any terminator (";") between them, 2) ability to show the execution plan of the SQL in which the cursor is positioned and 3) ability to connect to different kinds of databases.

I ended up with nothing, and I actually even initially excluded DBeaver as running on Java (personal thing - most Java apps I tried always had some kind of bug which made it a no-go for me).

After a while I gave it a try and I ended up being extremely happy with it, especially because I can use it with basically all databases that I am/was using (I'm currently using it mostly with MariaDB & Clickhouse, but used it in the past as well with DB2, Kudu, PostgreSQL).

</comment><comment><USER49> Biggest benefit for me is DBeaver is cross platform so I can use the same tool on Windows (work) and Linux (home).

</comment><comment><USER51> Love DBeaver, been using it for years now. Engineer(s) is very responsive with github bug reports too.

</comment><comment><USER52> Pgcli and PgModeler are usually by go-to these days. (Sometimes pgadmin4 if I need to see stats)

</comment><comment><USER53> I am missing bigger screenshots though, I tried opening them separate but they are quite small.

<comment><USER54> Bigger screenshots, even one very large one, can be seen from the links on the Github page: [https://github.com/dbeaver/dbeaver](https://github.com/dbeaver/dbeaver)

<comment><USER53> Thanks!

</comment></comment></comment><comment><USER55> I use DBeaver to connect to Oracle SQL (among other things) and it works well for this purpose. I tried other alternatives too but DBeaver felt the best among them (layout, options, and general speed).

<comment><USER56> Benthic software makes a really nice sql editor for oracle. "Golden"

[https://www.benthicsoftware.com](https://www.benthicsoftware.com)

Inexpensive and light weight.

<comment><USER55> Looks good, but Windows only :-/

</comment></comment></comment><comment><USER57> Just sharing the database management tool I use and love: [https://github.com/martingms/vipsql](https://github.com/martingms/vipsql)

<comment><USER58> Thanks for the love! Please let me know if there are any features that you feel are missing.

<comment><USER57> I'll open an issue.

</comment></comment></comment><comment><USER59> This looks great. I currently use a few separate applications for working with various databases, looking forward to giving this a shot instead.

</comment><comment><USER60> I use Navicat, and I'm very pleased with it.

<comment><USER61> As we're starting to move away from MS SQL Server, I'm just starting to use postgresql, and Navicat as a tool. But I'm really underwhelmed by Navicat's features and performance. It doesn't seem like a well-thought-out whole, and lacks a lot of obvious features (let's start with freakin' hotkeys for Run and Run Selected).

I just discovered DBeaver today, and while it's not up to the standards of SSMS (which is an amazing tool) with SQL Prompt, it so far beating the pants off Navicat.

<comment><USER60> Run is CMD-R, and Run Selected is SHIFT-CMD-R. [0]

[0][https://www2.navicat.com/manual/online_manual/en/navicat/mac...](https://www2.navicat.com/manual/online_manual/en/navicat/mac_manual/KeyShortcut.html)

</comment></comment></comment><comment><USER56> My company's IT team won't allow install of non-approved tools. I'm stuck with oracle sql developer. fml.

<comment><USER45> Pretty sure if it's java based you can download it and run it entirely in user space if that is OK. Of course, making an end run around IT policy isn't always an advisable solution even if it does work.

<comment><USER40> Yup, I am in the same boat and DBeaver works great just running standalone even without admin rights.

</comment></comment></comment><comment><USER62> It's a little buggy but it does everything the paid tools do. The only thing I really dislike is it requires Java.

</comment><comment><USER63> SSDT and their DACPAC deployments is very good I wish an open source tool will one day try to offer something similar

</comment><comment><USER64> Dbeaver is great.

If you like a cli tool, [http://henplus.sourceforge.net/](http://henplus.sourceforge.net/) is ok and easy to modify if you feel inclined.

It's written in Jython and used jdbc.

There is a version with line-editing around using jline.

</comment><comment><USER65> Doesn't work on the Mac right away. It exits with bunch of Java exceptions ranging from JRE versioning issues to Eclipse exceptions I don't want to care about. Piece of crap in my opinion. Thanks for the suggestion anyways. Next!

</comment><comment><USER66> I've been using in on macOS and it just works.

There are so many free tools for databases and I'm wondering : How do the devs survive with free versions and/or open source?

Don't think donations will amount to any relevant amount.

</comment><comment><USER67> I used Dbeaver for years. Once with MS-SQL, which was quite unstable... But no problem with MariaDB, SQLite and PostgreSQL.

Plus, the built-in support of SSH tunnels is a big plus for my use cases!

</comment><comment><USER68> Cool, can anyone tell how it compares to Datagrip by jetbrains?

<comment><USER69> Does datagrip support nosql?

<comment><USER70> No that I am aware of. But compass for mongo works pretty well. It's fairly simple but to visualize the data and run some queries it's nice.

</comment></comment></comment><comment><USER71> After installing solusOS I was looking for a replacement for MySql workbench and was suggested this and I have to say it works really well.

</comment><comment><USER72> Always used it while ddveloping database connected apps in Eclipse. Never let me down.

</comment><comment><USER73> Anyone using it with exasol? What's your opinion on in memory databases?

</comment><comment><USER74> Sometimes I use it, but data related for sql server, still prefer HeidiSQL.

</comment><comment><USER75> What's the most convenient tool you've found for Postgres?

</comment><comment><USER76> Great tool and it supports msaccess files

</comment><comment><USER77> I use Dbeaver everyday. Thanks for it.

</comment><comment><USER78> while i kinda like dbeaver on windows. the experience of dbeaver on macOS is terrible.

</comment><comment><USER79> I've been using DBeaver for years now, and it just keeps getting better and better. I use SQL every day, and DBeaver is the best client I've found.

It's shocking to me, actually, how absolutely awful most SQL clients are. It truly was an area that was ripe for "disruption".

</comment><comment> tedk-42 Great tool with a great name. Anyone who doesn't like it is an idiot :P

<comment><USER80> No need to insult whoever dislike your preferences.

</comment></comment><comment><USER81> Needs Java. I saw it mentioned in the HeidiSQL thread and thought I would try it. Never completed the installation. Not putting Java on my machine, nope. Yes, I do understand the advantages of Java. Runs everywhere, which is nice. I'm still not installing it.

<comment><USER82> The installer comes with it's own JRE just for DBeaver, so you don't have to install Java systemwide for it to work.

And why the hate for Java? Most other languages have runtime libraries or VMs too, does it really make a difference if that runtime code is baked into the exe vs. stored in a separate folder?

<comment><USER81> Maybe I don't know JRE vs Java well, but I really dislike the constant "update Java now!" messages and the hassle that goes with it.

<comment><USER82> Ah yes, that's different. There's no need to install any of that stuff when you pick the DBeaver installer that ships with a JRE. It basically just adds another folder into the DBeaver directory.

</comment></comment></comment></comment><comment><USER83> I would feel uncomfortable saying the word 'beaver' at work.

<comment><USER84> Why? Is it a bad word in some language?

<comment><USER85> Among some the term beaver is suggestive, or even offensive. Think Thelma & Louise; "Damn! I hate that! I hate being called a beaver! Don't you?"

It's a particularly lowbrow bit of slang and not something that worried me when I recently mentioned the name dbeaver in a professional context, although I am aware of it. It pleases me that I spend my time among people that are either ignorant of this matter or are happy to pretend they are.

<comment><USER70> I'm going to guess the cross over of Thelma and Louise fans and SQL database users is probably fairly low.</comment></comment></comment></comment>


Fixing Getrandom() - <AUTHOR> https://lwn.net/SubscriberLink/800509/c46eba62a7bda958/ <comment><USER4> It's recommended that people check out Theo de Raadt's Hackfest 2014 presentation on OpenBSD's arc4random(3).

[https://www.openbsd.org/papers/hackfest2014-arc4random](https://www.openbsd.org/papers/hackfest2014-arc4random)

Also suggest reading from page 19 on, which covers OpenBSD"s random(4) subsystem, and getentropy(2) syscall which suffers from none of the issue that are plaguing Linux getrandom today.

[https://www.openbsd.org/papers/hackfest2014-arc4random/mgp00...](https://www.openbsd.org/papers/hackfest2014-arc4random/mgp00019.html)

[https://man.openbsd.org/man4/random.4](https://man.openbsd.org/man4/random.4) [https://man.openbsd.org/arc4random.3](https://man.openbsd.org/arc4random.3)

<comment><USER5> There may be some systems where storing entropy in a file across reboots isn't an option, e.g. diskless situations or non-writable filesystems. Though those should be a rare case. Of course, this would actually require cooperation from userspace, and given Linux is just a kernel and userland kind of free to do what they want, it's not as easy as it sounds.

Incidentally, I really wish we could get the arc4random(3) family on GNU/Linux already. It's the only notable *NIX platform that still doesn't provide it; illumos has it, every BSD has it, macOS has it. Given the difficulties with entropy as noted above, however, the whole "non-failing cryptographically secure PRNG" model just won't work with Linux.

<comment><USER6> systemd does store the entropy in a file, but Lennart says this in the comments of the article: "we can only credit a random seed read from disk when we can also update it on disk, so that it is never reused. This means /var needs to be writable, which is really later during boot, long after we already needed entropy".

How does OpenBSD deal with this issue?

<comment><USER7> From _rc(8)_ :

      # Push the old seed into the kernel, create a future seed  and create a seed
      # file for the boot-loader.
      random_seed() {
            dd if=/var/db/host.random of=/dev/random bs=65536 count=1 status=none
            chmod 600 /var/db/host.random
            dd if=/dev/random of=/var/db/host.random bs=65536 count=1 status=none
            dd if=/dev/random of=/etc/random.seed bs=512 count=1 status=none
            chmod 600 /etc/random.seed
      }

* [https://cvsweb.openbsd.org/src/etc/rc?rev=1.537](https://cvsweb.openbsd.org/src/etc/rc?rev=1.537)

FreeBSD:

* [https://svnweb.freebsd.org/base/head/libexec/save-entropy/](https://svnweb.freebsd.org/base/head/libexec/save-entropy/)

* [https://www.freebsd.org/cgi/man.cgi?loader.conf(5)](https://www.freebsd.org/cgi/man.cgi?loader.conf\(5\)) (see "entropy_cache_load")

OpenBSD's boot loader also injects entropy into the kernel.

<comment><USER8> Which is to say, /var/ is writable when they do it.

This business of /var/ not being writable early enough is probably to do with the wonderful complexities of modern Linux, and is just not a problem that OpenBSD has.

<comment><USER6> What if a filesystem error forces /var (or whichever mount point hosts the entropy seed) to remain read-only and seed is never replaced? Seems like unlike systemd the BSDs are fine with crediting entropy even if it has been used already in a previous boot. That's wrong though.

<comment><USER7> > _Seems like unlike systemd the BSDs are fine with crediting entropy even if > it has been used already in a previous boot._

FreeBSD runs a regular cron job so that there are multiple files. If /var goes R-O, you still have a bunch of files from when it wasn't and they are not the same as on initial boot (assuming that (a) /var was mounted R-W at some point, and (b) cron managed to run as well).

Also, on shutdown there is an attempt to write 4096B to both /entropy and /boot/entropy:

* [https://svnweb.freebsd.org/base/head/libexec/rc/rc.d/random?...](https://svnweb.freebsd.org/base/head/libexec/rc/rc.d/random?revision=348122&view=markup)

Further, all of these various seed files are not the only sources of entropy (timers, RDRAND, etc).

<comment><USER6> > (assuming that (a) /var was mounted R-W at some point

Not a good assumption if the initial r/w mount is the one that fails...

<comment><USER7> If the initial /var (or /) is broken, then one has bigger problems and probably has to go in manually to fix things.

But for >99% of cases where the system comes up cleanly, and runs cleanly for a period of time, you'll have a bunch of seed files ready to go for the next boot. This configuration optimizes for the common case.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> Linus merged a jitter entropy fallback that calls the kernel's schedule() function in a loop:

[https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=3f2dc2798b81531fd93a3b9b7c39da47ec689e55)

<comment><USER10> Classic maintainer stuff, love it. Getting adequate fixes in place, when nobody else has been there to submit it, so that the next release can have one bug less.

</comment></comment><comment><USER11> What a mess. The problem here is systemd and/or things like GNOME wanting high quality entropy too soon, often not actually even needing it. Systemd shouldn't need high-quality entropy for itself, and most services it starts shouldn't need any either, though eventually some service is bound to need it (e.g., for TLS), but each service should make its own decisions about early- in-boot entropy. Then instead of a deadly embrace between the kernel and systemd leading to not making progress in boot you'd have some services not quite operational right away -- still a big problem, but a lesser one.

Still, if you're booting a system that will be doing one thing, and that one thing needs entropy, then you have a problem. And the only practical solution then is to accept using things like RDRAND and a previous shutdown's saved seed to seed the system's SRNG, then add better entropy slowly as you can get it.

EDIT: I agree with tytso: if you trust the CPU enough to run on it, then you should trust its RDRAND or equivalent (if it has it).

<comment><USER12> RPis don't necessarily have RDRAND, they especially suffer from increasingly fast boot where waiting for the entropy pool is an actual concern.

Of course, not all applications need that kind of cryptographic entropy but during early boot it's pretty much the best randomness you can get (seeding from datetime isn't available on the RPi, so until you can bring up the network and contact the NTP server, you're stuck with whatever getrandom() returns)

<comment><USER11> RPI 2s have a BMC (binary-only) driver for entropy generation.

<comment><USER12> I have an RPi1 that suffers from this problem and will for a long while still.

</comment></comment></comment></comment><comment><USER13> This is strictly bad. getrandom was fine as it was, it's not the issue with the kernel if people don't read the explicitly documented man page on it. This is now changing behaviour, breaking the god damn userspace which is exactly 100% what the kernel shouldn't be doing.

<comment><USER14> Indeed. I don't see how the solution to this problem isn't just to patch gdm.

<comment><USER15> Because the kernel has a policy of not introducing changes that break userspace, even if userspace is doing something weird.

<comment><USER16> Changing the semantics of getrandom(flags=0) breaks a lot of userspace ABI.

</comment></comment></comment></comment><comment><USER17> > systemd reads a random seed off disk just fine for you, no need to write any > script for that. Problem with the approach is that it's waaaay too late: we > can only credit a random seed read from disk when we can also update it on > disk, so that it is never reused. This means /var needs to be writable, > which is really later during boot, long after we already needed entropy, and > long after the initrd

I understand in general why seed reuse is bad. If an attacker can get access to two things that used the same seed, that attacker can often learn things that the randomness was suppose to make unlearnable.

In this particular case, though, I wonder if that is actually a problem. If the seed is used before you have writable storage, and then the system is rebooted without writable storage ever becoming available (and so before the seed could be updated), and then the system reuses that seed on the next boot --the only way an attacker can get access to two things that used the same seed is if something that used it on the first boot has persisted somewhere.

Since there was no local writable storage the first time, this can only happen if the results of using the seed were communicated off the system, via networking or via a terminal, to someplace that did have working writable storage.

Thus, it would seem, that if you block networking with other machines and terminal access until /var becomes writable and the seed is updated, there is no problem with seed reuse.

<comment><USER18> Blocking doesn't buy you anything because the seed could have already been used to generate data in-memory, which can then later be sent over a network or persisted to permanent storage.

</comment></comment><comment><USER19> > we need to get over people's distrust of Intel and RDRAND.

Am I misreading this or is Ted T'so really suggesting that we should all just stop worrying and love the secret and compeltely unauditable RNG offered by the same company that has literally backdoored every CPU they've sold in the past 12 years?

<comment><USER20> If you don't trust Intel, then don't use Intel CPU's at all. Using Intel CPU's and simultaneously saying, "but we can't trust RDRAND because could be backdoored" is completely insane.

Intel hid an entire x86 core running minix --- with security holes --- and told no one. Between the firmware code which runs in System Management Mode and in UEFI, which persists after the OS is booted and can take over the system and read and write arbitrary memory locations --- if you fear a backdoor in the CPU, RDRAND is not the only place where an attacker can screw you over. The bottom line is the entire CPU can not be audited. So trust it. Or don't trust it. Use another CPU architecture. Or go back to using pen and paper, and don't use any computers or cell phones at all.

We have to trust the bootloader to verify the digital signature on the kernel, so we might as well trust the bootloader to get a secure random seed. But from where? It could call UEFI or try use the RNG from the TPM (if available). But now you have to trust Intel, and/or the motherboard manufacturer, and/or the TPM. The bootloader could read from a seed file from the HDD/SSD. But we know that nation-state intelligence agencies (like the NSA) have the capability of implanting malware into HDD firmware (which is also unauditable), and we also know that we can't trust most consumer grade manufacturers or IOT devices to correctly insert device-specific secure entropy into the seed file at manufacturing time. Otherwise, all of the seed files will likely have the same value, at which point when the device is generating its long-term private key, immediately after it is first plugged in, the key will very likely be weak (see the "Mining your p's and q's"[1] paper).

The bottom line is that unless you propose to personally wire up your CPU from transistors, and then create your own compiler from assembly language (see the "Reflections on Trusting Trust"[2] paper by Ken Thompson about what can be hidden inside an untrustworthy compiler), and then personally audit all of the open source code you propose to compile using that compiler and use on your system, you have to trust _someone_.

What makes this political, and difficult to address, is everyone has different opinions on what they are willing to trust and not trust. But some combinations, such as "we can't trust Intel because their firmware can't be audited", and "but we insist on using Intel CPU's", really don't make much sense.

[1] [https://factorable.net](https://factorable.net)

[2] [https://www.archive.ece.cmu.edu/~ganger/712.fall02/papers/p7...](https://www.archive.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf)

<comment><USER19> RDRAND might not be the only place where an attacker can screw me over, but it's a much easier exploit than many others, and one that is by design impossible to detect.

It would probably not be possible to get away with an XOR instruction that doesn't behave as advertised. There is simply no way to determine if RDRAND is behaving as advertised or not.

> you have to trust someone

Well, yes. Intel specifically has to my mind demonstrated beyond a shadow of a doubt that they are not be trusted (that "entire x86 core running minix --- with security holes" thing you just mentioned, thought it was ARM though). So if the only difference is moving my root of trust from Intel to a random, unknown party that is _not_ Intel, that is already something of an improvement.

<comment><USER21> How would you measure the jitter from your disk? Certainly you can't trust rdtsc either.

</comment></comment></comment></comment><comment><USER22> This feels too obvious to work, but:

The kernel knows it's early in the boot process, and it knows that there isn't enough entropy that it will have to block getrandom() calls. We also know that what caused this issue was some FS efficiency improvements that result in fewer disk accesses during early boot, which results in fewer interrupts, which results in less entropy.

So... when we get into this situation, why doesn't the kernel just start issuing some disk accesses (perhaps with weakly-random offsets and sizes) until enough interrupts are generated to fill the entropy pool to a safe level?

<comment><USER16> Not all Linux systems have disks. The jitter entropy approach described elsewhere is more comprehensive and less dependent on hardware components outside of the CPU.

<comment><USER22> Fair; it does make sense to re-evaluate the whole thing and find the most generic of solutions, but the issue that triggered this discussion was a sort of regression on systems that _do_ have disks.

(Your point also raises the question: on systems that _don 't_ have disks, how are requests for early-boot entropy handled?)

<comment><USER16> > Fair; it does make sense to re-evaluate the whole thing and find the most > generic of solutions, but the issue that triggered this discussion was a > sort of regression on systems that do have disks.

Right. On those kinds of systems, persisting some saved state between boots is your best first option. (As other commenters point out, you really don't need very much! 256 bits or 32 bytes is basically adequate.) Any additional entropy you get from device interrupts or machine-dependent random sources like RDRAND are good to mix in.

(I don't think the adversarial RDRAND thread model is a reasonable one, and that has been adequate addressed elsewhere in this thread by tytso, tptacek, and tedunangst. Any adversarial CPU that can defeat cryptographic digest whitening of entropy sources is so sophisticated/privileged it already fully owns your operating system; game over.)

Linux systems already do this, kind of! But Linux has taken the stance that entropy you can't delete from the boot media _at the time it is fed into the CSPRNG as entropy_ doesn't count ("credit") as far as initial seeding, and that decision is what moots its utility for early CSPRNG availability. It's not an unreasonably paranoid take, but not an especially practical one (IMO). I suspect the net effect on security, holistically, is negative.

> (Your point also raises the question: on systems that don't have disks, how > are requests for early-boot entropy handled?)

Depends which subsystem you ask and how you ask it, right? And whether or not a jitter entropy scheme is enabled in the kernel. (I might be misunderstanding your question, sorry if that's the case.) If you ask /dev/random or getrandom(~GRND_NONBLOCK), they'll block until the CSPRNG is satisfied enough entropy is available. If you ask /dev/urandom, it just spews unseeded and likely predictable garbage at you. If you ask getrandom(GRND_NONBLOCK), it returns -1/EAGAIN.

<comment><USER22> > _(I might be misunderstanding your question, sorry if that 's the case.)_

Not entirely, but I think my question was just more narrow. Considering the case that started this off (machines that used to work, now hanging on boot because of lack of entropy, triggered by too few disk interrupts), I don't get how this generalizes. If you take that same system, and remove the disk (replacing it with netboot or whatever), how would it _ever_ boot, at least not without making some changes to how early entropy is seeded? I guess in the netboot case you end up with the timing of network interrupts that you can feed into the entropy pool, but this whole thing just feels _off_ to me.

If a system was relying on a specific, inefficient set of disk access to give it _just_ enough entropy to get through early boot, and that making the FS code more efficient caused it to fail to have enough entropy, I'd suggest that (likely unbeknownst to the system administrators), this system was _already_ pretty broken. I get why Torvalds decided to bucket this under the "we don't break userspace" umbrella, and appreciate the care he takes to that sort of thing, but suggesting that "FS access patterns during early boot" is sorta a part of the kernel's ABI is... a bit too incredible for me to take seriously.

</comment></comment></comment></comment></comment><comment><USER23> So the discussed sources of entropy are

       - <USER2>
       - <USER3>
       - timer/instruction jitter

Why don't they throw more sources at the problem? LSBs from analog sources such as audio inputs, stray network packets (put the nic in promiscuous mode). Even uninitialized memory might be worth a few bits (not all systems zero it on boot).

<comment><USER6> This is happening so early that audio and network (and /var where the persistent seed is stored) aren't up. The kernel doesn't want to be susceptible to hangs unless userspace brings up all entropy sources early enough.

<comment><USER24> This is about GDM, which is the last thing that gets started.

Hardware devices are initialized before init is started, so you should still be able to gather entropy from the various input devices.

Lack of entropy can't cause the kernel to not boot; it's just userspace (typically services which require encryption) which can have a problem. It would be silly to start a webserver or sshd before the FS is remounted rw.

<comment><USER25> Not if you use modules.

<comment><USER16> Can you elaborate on what you mean?

<comment><USER25> Init is executed before modules start being loaded.

</comment></comment></comment></comment></comment></comment><comment><USER26> I'm surprised nobody in the thread mentioned using a TPM device when available. My 5-year old MB comes with one builtin, and probably all new ones have one. TPM also provides a limited amount of persistent storage.

</comment><comment><USER27> I used to work on a Java application that used a bunch of libraries, and specifically I remember one of the libraries generated some identifier using urandom. I think it was axis2 or something like that. There was unexplainably random hangs when trying to read from urandom. The wikipedia article says that urandom should not block. Anyone smarter than me that can understand this article have any reason why urandom hangs?

<comment><USER28> Java on Linux at used to have this bad behavior where it would short-circuit configuration to use /dev/urandom to use its own internal RNG based on /dev/random. It did not create a single global seed for the internal random number generator, so some systems could overwhelm the entropy pool.

I've solved this for some systems by changing the java security properties file to instead point to /dev/./urandom .

<comment><USER27> I did that and even then it still happened :( one of lifes great mysteries

</comment></comment></comment><comment><USER29> When the issue came up with the OpenSSH hang, I failed to be convinced why this should be a problem at all. Perhaps someone can enlighten me after reading my thoughts in the thread there? [https://news.ycombinator.com/item?id=20463586](https://news.ycombinator.com/item?id=20463586)

<comment><USER30> > Sounds like the real problem is some part of the system can't be bothered to > persist its state.

That's just a special case when you do a reboot. There are many cases where the system starts and there's either no previous state, or the previous state is explicitly killed to prevent sharing.

For example a lot of AWS instances will boot once once in their lifetime, and will do that from a shared AMI. You can't reuse a shared random seed in that case or getting random values from one VM would allow you to predict new values on another.

So persisting the state will help desktop users, but that's addressing only a specific version of this problem, not solving it for everyone.

<comment><USER29> > You can't reuse a shared random seed in that case or getting random values > from one VM would allow you to predict new values on another.

Why can't AWS seed you with a random number before giving you control? In fact does it not do that already? If it doesn't, either I'm surprised, or I don't understand why it's not possible.

<comment><USER30> AWS does not touch your drives itself. (Apart from what you configure) It would be a drama on its own if they tried. And even if they did try, what about encrypted partitions, filesystems which they don't support, systems configured for a different seed location, etc?

For some hypervisors this is solved in a different way: [https://wiki.qemu.org/Features/VirtIORNG](https://wiki.qemu.org/Features/VirtIORNG) but as far as I know AWS does not support it yet.

<comment><USER29> What do you mean they don't touch your drive? Do they just give you a blank disk with no bootable OS (in which case, doesn't that mean you yourself are now responsible for seeding when you install the OS)? I know whenever I've dealt with a cloud VM, it's come with an OS set up by the vendor, but if it's not, then the OS installer needs to seed the image. So my point is, whoever sets up the system seems responsible for seeding it. If AWS gives me a VM, I expect it to come seeded with a unique number, and of course I don't expect them to touch anything _after_ giving me control. Anything I set up beyond that is my responsibility to seed properly. I don't understand why this would be a drama.

<comment><USER30> They give you a VM with an image you choose. It may be an image you prepare or one they provide. But no modifications to the file contents since creation.

Drama would come from the same place as "we shouldn't trust rdrand". That's still avoiding the issue of: sometimes can't write that seed in some configurations.

<comment><USER29> Yeah, so while creating it, they seed it. If you don't trust that, you install another OS yourself, with a kernel your trust, and seed it yourself when installing, from whatever sources of randomness you want. I fail to see why this wouldn't work.

<comment><USER30> They can't seed it while creating the AMI. If they did that, everyone would share the initial seed and I could boot up the published image, run the generator with that seed, and be able to predict (for example) what ssh key your server will have.

(Or if you meant creating the VM, as explained before they can't do that for encrypted volumes for example)

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER10> Why isn't it standard that hypervisors provide a random seed for each of their vms?

<comment><USER16> Virtio-RNG.

</comment></comment><comment> loup-vaillant So, the Linux kernel can't just persist 32 little random bytes? That shouldn't be hard. Even giving a seed for the first boot (or when booting a VM image) shouldn't be hard. Heck, we don't even need to collect entropy. Like, _ever_. Just use Chacha20 with fast key erasure and be done with it.

If your system is too small to be able to persist 32 bytes, it's probably too small to perform any amount of meaningful cryptography anyway.

<comment><USER16> Yes, the availability problem hinges on being able to persist a handful of bytes. The problem isn't the number, which, sure, is small. The problem is the availability of _any_ persistent media in general.

There are absolutely devices with exclusively RO media with plenty of processing power to perform meaningful crypto (e.g., embedded devices with 802.11 wifi). So:

> If your system is too small to be able to persist 32 bytes, it's probably > too small to perform any amount of meaningful cryptography anyway.

This sentence is not based in reality.

<comment><USER31> > There are absolutely devices with exclusively RO media with plenty of > processing power to perform meaningful crypto (e.g., embedded devices with > 802.11 wifi).

Don't most microcontrollers have some EEPROM that could be used for this? I'm assuming even 4 bytes would help a lot. I am pretty sure STM32's with hardware crypto have EEPROM good for a million write cycles.

<comment><USER16> 4 bytes isn't enough, but yes, it's quite possible some SoCs could persist 256-512 bits of entropy in some hardware-specific way. It's hard to abstract that into a general-purpose operating system, though it could be a good solution for specific applications.

<comment> loup-vaillant > _It 's hard to abstract that into a general-purpose operating system_

Linux has lots and lots of hardware specific drivers. This shouldn't be any different.

<comment><USER16> Feel free to start writing device drivers for 1000s of individual SoCs and figure out a way to solicit testing. I agree hardware-specific drivers are a solution for the SoCs with RW media of some kind, but it's not as trivial as you paint it. And (AFAIK) there are still SoCs without RW media. I think jitterentropy is a better and more easily tested use of developer hours than implementing 1000s of SoC drivers, but it's open source; work on what you like.

<comment> loup-vaillant The onus is not on Linux to support devices. The onus is on the _devices_ to support Linux. Linux support is a selling point, that should be leveraged.

The current way of gathering initial entropy should not be the default, it should be a _fallback_. If a device have neither persistent storage (32 bytes for crying out loud) nor a hardware random generator, they should be treated like second class citizen, with only best effort random numbers.

Right now, _everyone_ is a second class citizen. Kind of insane, don't you think?

> _it 's not as trivial as you paint it_

It is, under one (extremely hard to fulfil) condition: _collaboration from hardware vendors_. It's just 32 persistent bytes. Just bury them in a chip and memory map the damn thing! The driver can be reduced to an offset.

> _I think jitterentropy is a better and more easily tested use of developer > hours than implementing 1000s of SoC drivers_

That one? [https://fuchsia.dev/fuchsia- src/zircon/jitterentropy/config-...](https://fuchsia.dev/fuchsia- src/zircon/jitterentropy/config-basic)

It's one of the first links I found, I hope it is obsolete: they say the internal state of Jitterentropy is puny _64 bits_ number. What were they thinking, we need four times as much! That kind of thing is why people are tempted to shed the system's RNG, and ask users to wiggle their mouse instead.

<comment><USER16> I'll pass on most of this rant, just want to correct this misunderstanding:

> That one? [https://fuchsia.dev/fuchsia- > src/zircon/jitterentropy/config-...](https://fuchsia.dev/fuchsia- > src/zircon/jitterentropy/config-..).

> It's one of the first links I found, I hope it is obsolete: they say the > internal state of Jitterentropy is puny 64 bits number

You skimmed too quickly and came to a judgement based on a misreading or misunderstanding of the concept. Obviously 64 bits is not sufficient.

The idea of the jitter entropy mechanism is that most modern CPUs have super high resolution clock or cycle counters available (even embedded boards), and instruction execution speed has mild variance.

You run a series of trials, each of which produces one or more bits of output (classically: 1). You run as many as you want, producing an infinite stream of weak entropy, until you have collected a satisfactory number of output bits. Trials can be run relatively quickly (many nanoseconds or a handful of milliseconds per trial).

For each trial, you perform some minimal workload intended to exacerbate CPU runtime variance (this might be where you saw "64 bits"), and extract some number of output bits (maybe 1) from one or more of: the low bits of the cyclecounter, nanosecond clock, or something similar of that nature.

Caveat: jitter is an even weaker entropy source than most non-HWRNG sources typically consumed by entropy gatherers. Your motto of "just 256 bits" assumes _total independence and 8 bits per byte of entropy_. It isn't met by most real-world entropy sources on server systems (expect 4-5 bits/byte) and especially not by jitter entropy. Empirically, jitter seems to come closer to 1 bit per byte minimum in SP800-90B evaluations on the raw output — it's a pretty weak entropy source.

Anyway: feel free to read more about the concept at any of the sources. The Fuschia writeup you linked is a good one if you read it more closely; there's also:

* From the original (2013) proponent, Stephan Mueller: description: [http://www.chronox.de/jent.html](http://www.chronox.de/jent.html) and sources: [https://github.com/smuellerDD/jitterentropy-library](https://github.com/smuellerDD/jitterentropy-library)

* LWN's 2015 writeup of the concept as a Linux entropy source: [https://lwn.net/Articles/642166/](https://lwn.net/Articles/642166/)

* And this is all suddenly topical due to the latest nonsense from Linus (TFA, 2019), who still does not understand CSPRNGs. Anyway, Linus wrote and merged a version of jitter entropy quite recently: [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=50ee7529ec4500c88f8664560770a7a1b65db72b) This is a relatively happy outcome in that Linus didn't just break the ABI to be completely insecure by default.

<comment> loup-vaillant > _Your motto of "just 256 bits" assumes total independence and 8 bits per > byte of entropy._

Yes of course. This is easily obtained by hashing a much bigger input. The problem is determining how big the input should be. That is, how much entropy it _actually_ holds. You can also hash piecemeal (H is whatever you think is secure):

      H0 = H(I0)
      H1 = H(I1 || H0)
      H2 = H(I2 || H1)
      H3 = H(I3 || H2)

You can stop as soon as you gathered enough input to be happy about its entropy. Then just switch to fast key erasure with Chacha20 and stop wasting cycles on entropy gathering.

> _Anyway, Linus wrote and merged a version of jitter entropy quite recently. > […] This is a relatively happy outcome in that Linus didn 't just break the > ABI to be completely insecure by default._

I'm genuinely relieved. This would have been the _worst_ way to break userspace. Still, tiny embedded systems might need to persist (properly seeded) 32 bytes instead of relying on jitter entropy.</comment></comment></comment></comment></comment></comment></comment></comment></comment>


Duke Nukem Forever exposes crap reviewers - 100 reviews compared - <AUTHOR> http://www.xentax.com/?p=303

<comment><USER2> His methodology?

"We all know that we are right, as the true value of DNF is around 65-70%. Thus, I'd say the 60-90% group of reviewers are The Good."

Srsly?

<comment><USER3> That's only the beginning of the flaws; I saw no mention of the author taking into account the individual scoring systems that the various reviewers used. Some reviewers can be quite harsh, and would use the majority of the 0-100 spectrum for scoring AAA titles, others (i.e. the ones getting the games for free from the publishers) tend to only use between 65-100 (65 being an arbitrary cutoff) on the scale (they'd give a 65 to a game that another reviewer who used the entire scale would give a 0 to). It's especially ironic that the author bashes Metacritic and other aggregation sites for being "shallow," then proceeds to churn out similarly shallow, infographic/graph- laden content. With something as incredibly subjective and contextual as a video game review, distilling a game to a numbered score is pretty ridiculous; then aggregating those scores together into a totally context-lacking mass is pointless, as the result would be totally lacking in almost any meaning whatsoever.

<comment><USER4> The data is what it is. I collected as many scores I could, without the preselection bias that you find with metacritic. Indeed, I do not take into account "individual scoring systems" that various reviewers used, although I did note whether they used a 0-100% scale, or 5 stars. The bottom line is, many a reader will look at the scores to decide, even if you feel that is not the case. Also, reviewers that only use 65-100 are basically incompetent. Use another system if you are not going to use the rest of the scale anyway, or - if you think the actual text of a review matters only - just don't use any scoring system at all.

I present the raw objective scoring data, and these data indicate there are two bell curves possibly, and those bell curves prove the interesting point. I don't care if you project whatever I feel about the game in question on the validity of the data, because that's just nonsense. The data was collected without selection bias, just gathering as much as possible given the timelines. The only thing that matters is that we felt there was something amiss with certain reviews based on our own experience of this game in particular. From then on, it was a tool to expose certain reviewers for showing incompetent behaviour. You can rest assured that I don't care whether DNF sells or not. I do care when reviewers at major sites show questionable conduct. DNF is the tool, from then on we will watch them, also in retrospect. Bad reviews can make or break a game in certain cases, and this kind of power in the wrong hands is not to be tolerated.

<comment><USER2> Your problem is entirely in the line I quoted:

"We all know that we are right, as the true value of DNF is around 65-70%. "

This makes the whole exercise total bullshit. Obviously the data is irrelevant since you assumed your conclusion.

And another thing if you had actually clicked thru to metacritic you'd realize why there are two peaks in your data. The console versions are consistently rated a full star lower then the pc version. I took me less then a minute to verify that at least 2 of the 40-rating reviews were based on console versions.</comment></comment></comment></comment>


Show HN: WWBR – Who Wants to Be Relocated? (July 2019) - <AUTHOR> https://relocate.me/wwbr?hackernews <comment><AUTHOR> Hi there! Andrew here, the founder of Relocate.me ([https://relocate.me](https://relocate.me)), a tech job relocation platform.

As we're working with relocating techies every day, we observe a discouraging trend.

Around 80% of tech job candidates applying from abroad are being rejected on the resume screening phase.

Driven by a desire to change this, our Relocate.me team has launched the new 'Who Wants to Be Relocated?' page.

The aim is to give the relocating tech specialists more chances and better exposure.

Every month, we'll update it with a fresh batch of new applicants.

<comment><USER2> Maybe you could hide full names, email addresses, and LinkedIn profiles to be visible only those who are registered as businesses?

<comment><AUTHOR> Good point! This could be an option, but a bit later

</comment></comment></comment><comment><USER3> Two small suggestions: \- Make it possible to filter based on desired country. If i am hiring from a specific country it would be nice to know who want to relocate here. \- Set a size limit on the "tech skills" column

<comment><AUTHOR> Great suggestions! Thanks

</comment></comment><comment><USER4> So this is just a list publishing peoples' email address?

<comment><AUTHOR> This is pretty similar to the 'Who wants to be hired' list on HN, but with a focus on finding a job abroad.

Moreover, we're promoting this list to tech companies from over 30 countries.

</comment></comment><comment><USER5> I'd also consider who wants to work remotely for a SV company while living somewhere they'd rather be for whatever reason

</comment><comment><USER6> Me, please! Thanks for sharing it</comment>


Add search to your Jekyll blog - <AUTHOR> http://jzhang.io/add-search-to-jekyll <comment><USER2> Here's another approach: Make a templated JSON file that mimics a database- backend. When you generate the site (or just push to a GitHub Pages repo), the JSON file will be populated with the "search index" of your site. Something like this: [https://github.com/swanson/lagom/blob/master/site.json](https://github.com/swanson/lagom/blob/master/site.json)

Then you can use whatever Javascript you'd like to search through your pages by pulling down the JSON file into memory and slicing and dicing to your heart's content.

<comment><USER3> This is really an approach that should get more attention.

People know all about database-backed websites (with an AJAX/REST server), and static-generated sites (with no server at all.)

But there's a middle-ground where you statically generate precomputed AJAX/REST _responses_ to any query you could make, shove them in an S3 bucket and maybe put a CDN (e.g. Cloudflare) in front of it, and then treat _that_ as your "server", building a traditional AJAXy frontend to talk to it. Scales nigh-on infinitely.

The best part is, it's not necessarily read-only! You can write a "real" server, too, that just handles updates. When it receives a request, GET the old version of the object it's modifying from S3 (no need for a database!), patch it up with the AJAXed-in data, and PUT it back. (And follow it up with a CDN single-file-purge API call, if it's relevant.)

</comment></comment><comment><USER4> This got me thinking: The obvious way to do full-text search in the post body would be to build an index of {"word": [post_id, post_id, post_id]} of posts it appears in. However, this could be huge.

Does anyone know if there's a technique that uses a list of posts and a bloom filter that contains all the words in that post? I.e. you iterate over all the posts and check the bloom filter for membership of all the terms.

Since the number of words is (probably) much greater than the number of posts, you just need to loop a few hundred times at most, and you gain _a lot_ in saved space. Also, since a Bloom filter has no false negatives, you are, at least, guaranteed to find all the posts that mention the specified words (with maybe a few "junk" ones in between, but which should be easy for the reader to filter out).

You can't do weighting with this technique, but it should at least be a quick way to figure out which post IDs you want to show.

Does anything like this exist currently?

EDIT: Here's a quick proof of concept: [http://nbviewer.ipython.org/gist/skorokithakis/d115ab734d9ad...](http://nbviewer.ipython.org/gist/skorokithakis/d115ab734d9adbcf306f)

It works fine, but the filters are a bit large (2 KB each), so I'm not sure how much space you save.

EDIT 2: This was so much fun that I wrote it up: [http://www.stavros.io/posts/bloom-filter-search- engine/](http://www.stavros.io/posts/bloom-filter-search-engine/)

<comment><USER5> Very nice. You will save some space and allow for some typos if you stem and soundex before insertion. Also you can save space and improve the run time somewhat if rather than many separate bloom filters you build one large one where each item is post ID + word. If you do that you can also insert each word bare so you get O(1) empty result sets, helpful if you're updating the results with every keystroke in a search box.

<comment><USER4> Huh, very nice idea! That should, indeed save a ton of space and be much simpler when searching! I'll try that now, thank you.

EDIT: Hmm, turns out it's pretty much the same size, which makes sense, I guess: [http://nbviewer.ipython.org/gist/skorokithakis/0abbfebced25f...](http://nbviewer.ipython.org/gist/skorokithakis/0abbfebced25fd4b2ed3)

<comment><USER5> The space savings for the same error rate should be small (I think the likelihood of false positives for a given load goes down slightly with size of the filter) but the benefit in lookup time should be significant for multiword searches. Thinking about it more though if you're doing live search you'll already have computed the results for the first word by the time you are given a second so maybe it doesn't matter.

<comment><USER4> I think it'll be faster to do multiple filters, because the one-filter way requires hashing and comparing N times while the multiple-filter way requires hashing once and comparing N times.

<comment><USER5> Oops, you are correct.

</comment></comment></comment></comment></comment></comment><comment><USER6> We implemented search on our help site ([http://help.simpletax.ca](http://help.simpletax.ca)) using the excellent lunr.js ([http://lunrjs.com](http://lunrjs.com)) + this Jekyll plug-in ([https://github.com/slashdotdash/jekyll-lunr-js- search](https://github.com/slashdotdash/jekyll-lunr-js-search)). The search index is updated when you build your Jekyll site, so it's a piece of cake to maintain and gives you full text search. I don't know how well it would scale, but if you are in the hundreds of posts, you should be ok.

<comment><AUTHOR> That's awesome, especially for a help site. My company does the same thing with our documentations.

</comment></comment><comment><USER7> My Jekyll blog using lunr js for fulltext search with JSON file as backend. Check it out here. [http://dreamand.me/web/fulltext-search-at-jekyll- site/](http://dreamand.me/web/fulltext-search-at-jekyll-site/) and lunr js [https://github.com/olivernn/lunr.js](https://github.com/olivernn/lunr.js)

<comment><AUTHOR> That's neat, but is something wrong? I get this error on your site: [https://cloudup.com/cti8Egu8fQS](https://cloudup.com/cti8Egu8fQS)

</comment></comment><comment><USER8> How about using a lightweight TFIDF engine like whistlepig [[https://github.com/wmorgan/whistlepig](https://github.com/wmorgan/whistlepig)]. It is implemented in ANSI C.

I had no issues writing a racket wrapper using the FFI for it: [http://blog.shriphani.com/2013/08/27/racket-whistlepig- bindi...](http://blog.shriphani.com/2013/08/27/racket-whistlepig-bindings/)

</comment><comment><USER9> Even though its a nice hack, its more of a css based filtering which can be used irrespective of Jekyll. A jekyll specific search should include searching articles. I've been wanting to build something similar using the excellent lunr.js. The idea would be to rebuild the index before each push to the repo, and a special page (search.html) that loads up the index via Ajax and shows the results as you type.

Still, cool hack.

</comment><comment><USER10> Swiftype?</comment>


Google search results in Google Sheets or Excel - <AUTHOR> https://www.aozata.com/how-to-get-google-search-results-in-google-sheets-excel/ <comment><USER2> A more accurate title would be "How to get Google search results in Google Sheets or MS Excel" as per the original article (corrected for capitalization). "inside sheets" sounds dangerously close to "between the sheets".</comment>


Ask HN: Best house-pet training resources? - <AUTHOR>

Inspired by the discussions in https://news.ycombinator.com/item?id=10021268
What is the go-to resource that you use to train your house pet?
We just adopted a puppy and would love to hear this crowd's suggestions on websites/books/magazines/blogs or media regarding training and making your house pets life more wonderful. <comment><USER2> The best way to make "your house pet's life more wonderful" is to ensure that it is consistently obedient at any time without having to resort to bribery. IMO, the best training guide/philosophy is that of William Koehler. "The Koehler Method of Dog Training" is no longer printed, but you can easily find used copies on Amazon. The book is excellent and I've found his methods to be the most reliable way to get a well-behaved and happy dog.

Koehler trained dogs for the military during WWI and went on to train dogs in a number of Disney movies. His techniques are intended to produce consistent off the leash obedience. There is no question as to whether or not the techniques work. However, trainers following the "pure positive" and "death before discomfort" philosophies view Koehler's techniques as old-fashioned and cruel. Personally, I find it cruel, sometimes even dangerous, to allow a dog to determine when it will and will not be obedient.

Koehler's approach may not be for you, but you should at least educate yourself about it as an alternative.</comment>