Ask HN: What project are you currently working on? – June - <AUTHOR> What projects (hobby projects, side projects, startups) are you working on right now? Would love to hear what HN'ers are working on!<p>(ME : I'm working on currently to combine all the things that I've liked, bookmarked and up-voted - it's called "Marvelogs" - http://www.marvelogs.com)<p>How about you? <comment><USER2> I've just launched my first comic, Carcosa - [http://www.strangecompany.org/carcosa/](http://www.strangecompany.org/carcosa/) . It's made using CGI techniques, but post-processed to look like hand-drawn art. People seem to like it so far.

I'm working on 3 short films - one is literally hours away from completion, one I've just shot, and one I'm shooting in two weeks.

All of them are at least moderately interesting from a technical standpoint: the first is a two-hander between a live actor and a CGI motion-captured character. The second uses practical effects including the quite marvelous "el wire", but was made with a crew that goes beyond "tiny" and into "that shouldn't be possible", largely by utilising the latest film tech (Rode's new frequency-hopping wireless mic really saved our collective ass). And the third I'm using as a chance to really stretch the Sony a7s's legs and get close to Hollywood lighting without the Hollywood power bill, by using complex but very low light at an ISO that would normally be unusable.

I'm also continuing to poke at VR - most recently experimenting with stereo cubemaps, which are amazing. Here are my initial experiments with that: [http://www.strangecompany.org/3-new-virtual-reality- stereo-p...](http://www.strangecompany.org/3-new-virtual-reality-stereo- panoramas-for-the-oculus-rift-and-gearvr/)

</comment><comment><USER3> Practive

Elevator Pitch: Gives you enough insight into problems your customers are experiencing in realtime so that you can call them right away to make it right, instead of waiting for them to call you...or worse, leave your platform.

Integrated with Stripe, MixPanel, Mandrill, Gmail, MailChimp, Desk.com and Zendesk. Ingesting and correlating datapoints with customer profiles in order to trigger automated alerts.

Data is again siloed in third party partner systems. Time to bring it all back home and see the end to end customer experience and act on problems regardless of where they happen. Today's modern SaaS platforms generate enough data for customer service or success reps to spot problems in real time and be proactive about helping customers. We have the data, we just need to look at it.

<comment><AUTHOR> is there a demo?

<comment><USER3> Unfortunately not yet. Only 10% done. Is this something you need?

<comment><USER4> It's definitely a good idea and we're doing something similar but focused on other data sources.

<comment><USER3> I'd be curious to know what vendors you integrated with and for what purposes?

<comment><USER4> We're integrating with Forrester, Gartner, Nielsen, etc for analytical purposes.

</comment></comment></comment></comment></comment></comment><comment><USER5> Elevator Pitch: Carmatic - [http://carmatic.co](http://carmatic.co) (Car owner's best friend) At your request, takes your car to the repair shop for you and brings it back to your home or office. Think of 'Shyp for Car Repair'. All for just $25. We are operational in NYC.

Why: Because It sucks to be spending a lot time taking your car to the repair shop then either waiting there while its getting fixed or going back and forth to pick it up later. The experience is just frustrating!

Feedback: Would love HN'ers to give some feedback! (email: firstname.lastname@example.com)

</comment><comment><USER6> Just launched and signed on the first two customers (and a few early birds still running free) on [http://timeblock.com](http://timeblock.com)

It is a new working methodology that helps Makers beat procrastination, get more time in flow and have time for creating quality code while helping their Managers get a better overview and more sleep.

We are shooting for public launch after the summer holidays, currently the methodology is freely available if you sign up to our launch list. It's a three day email course.

</comment><comment><USER7> I managed to get audio and text working for my current project to build Space Invaders in C++.

It is objectively terrible but there it is: [https://bitbucket.org/kennethrapp/spaceinvaders](https://bitbucket.org/kennethrapp/spaceinvaders)

I need to spend a few weeks on Unity tutorials now so maybe I can learn to make something people will actually want to play.

I'm also working on a Corona app and a Facebook API think for a startup i'm interning/working at.

</comment><comment><USER8> Working on a past due invoice management platform in between consulting work. Basically it takes past due invoices and automates contact attempts. Emails, Letters, Voice Messages, Text Messages and Faxes on a predefined schedule. It also can be used for outbound calling campaigns and will soon have a built in call scripting engine. Long term is going to be more of a full debt collection platform.

If anyone is interested in more information feel free to email me at david at pastduereceivables.org

</comment><comment><USER9> Switching [http://soundshelter.net/](http://soundshelter.net/) over to Django

Been a surprisingly pleasurable experience so far.

<comment><USER10> What was it built in originally? What's motivating you to switch to Django?

<comment><USER9> PHP & MySQL

No real technical reason for switching, more a need to scratch my nerdy itch.

Python seems like a super-clean language so far.

</comment></comment></comment><comment><USER11> Creating a chrome extension that gives easy access to a user's playlists.

Here's the progress so far [http://imgur.com/rNHzfj4](http://imgur.com/rNHzfj4)

Motivation: I pretty much use Youtube as my music player. Great selection and I like watching the videos every now and then. The problem is that it doesn't really have the UI of a music player e.g. easy access to all of my 200+ playlists.

</comment><comment><USER12> I am working on a location bookmarking & discovery application, called kollekkt - [https://www.kollekkt.com](https://www.kollekkt.com) \- It lets you collect, share and discover the best places around the world. Users can also create their own, individual Collections of favourite places. It would be cool to know what you think of it.

<comment><USER13> How is it any different from the many other location bookmarking applications already available in the market?

<comment><USER14> I guess most bookmarking apps focus on people who want to bookmark their own stuff. Most of them never look at bookmarks from other users.

kollekkt seems to focus more on the discover part from how the site looks.

<comment><USER12> Thanks for your comments! Indeed I started this project to provide friends with a good solution to share and discover their favourite spots. So it's kind of a social website. Before I started this application, we used to share our individualised Google Maps among each other. I think it's more dedicated and accessible this way.

</comment></comment></comment></comment><comment><USER15> Three projects: \- Liquicipe, an Android app for storing E-Liquid recipes (primarily for testing C++ and Qt on Android)

\- A remake of my personal website with AngularJS and a Java Spring backend, primarily for testing Spring and adding a few larger features

\- Some kind of colors game on Android, using libgdx - primarily because I didn't know what to code

Marvelogs looks great. I'll keep an eye on it.

<comment><AUTHOR> any demos on your projects? Java Spring sounds a bit too heavy for a personal website no?

<comment><USER15> Not anything useful, yet. Liquicipe is open source, but I'm not happy with it yet: [https://github.com/andresmanz/liquicipe](https://github.com/andresmanz/liquicipe)

Personal website is a bit of a wrong term, maybe. I work as a freelancer, so I'll manage customers, code, invoices and all that stuff on the site. There are other tools for doing that, so it's really more about testing out Spring. :)

</comment></comment></comment><comment><USER16> Working on [https://zoomlee.com/](https://zoomlee.com/) to make it easier to travel internationally hassle free and leave worrying about visa/passport expiration dates, international tax tracking and safety concerns to technology :). Organize and securely store important documents

<comment><AUTHOR> WOW. Looks awesome. What's your userbase like? I HATE having to fill out my passport number in so many forms when traveling!

</comment></comment><comment><USER17> Nearshore/Offshore outsourcing concept. We match the talent to client needs.

Our bilingual team of software developers are based in Buenos Aires, Argentina. Very easy connect and chat during the business day via Skype.

Currently helping with Market Discovery & Business Development. Ping me for more details. > justin[@]accelone.com

</comment><comment><USER18> Practising some Common Lisp/functional programming with a text-based MUD ([https://github.com/veddox/atlantis](https://github.com/veddox/atlantis)). Don't expect the next nethack, but it's fun :-)

</comment><comment><USER19> I realized I didn't have anything in python that I could put on my resume, so I threw together a quick version of tail in python.

[https://github.com/gryftir/pytail](https://github.com/gryftir/pytail)

<comment><USER20> I've done quite a lot of python work, and so have it on my resume. I also have lots of sample code.

But even so no one seems to believe I really know python. I like python, I really do but even so I've completely given up on applying to python jobs.

</comment></comment><comment><USER21> \- Building a "strandbeest" bike with legs instead of a back wheel (for fun)

\- Creating a sports site that will generate odds based on crowd sourced content (as a hobby)

\- Developing a tool to manage data warehousing environments (as a side business)

<comment><AUTHOR> any demos??? :)

<comment><USER21> Soon. I'll have the strandbeest bike finished with videos up on reddit.com/r/diy by the end of the month. The other projects are a few months out.

</comment></comment></comment><comment><USER22> Still working on Storytella[0]. I'm currently working with a copywriter to get some much needed user guides in place.

[0] [https://storytel.la/](https://storytel.la/)

</comment><comment><USER23> Group collaboration environment. Work together wherever you are. Bullpens with people from different buildings, even different states, all talking and sharing and collaborating together, continuously.

</comment><comment><USER24> I am building my freelance business. Been working on my website, jcurcioconsulting.com, pretty heavily over the last couple of weeks.

</comment><comment><USER25> uLambda, just like AWS Lambda, but with REST API that you can actually use...

</comment><comment><USER20> The Global Computer Employer Index: [http://www.warplife.com/jobs/computer/](http://www.warplife.com/jobs/computer/)

It's not yet really "global" by any means however I do have some decent coverage of Portland, San Francisco, Seattle, San Luis Obispo, Santa Cruz and New York City.

I've stopped posting new companies while I work on some tools to automate the process. For example I keep new listings in a bunch of LibreOffice spreadsheets. Last week I learned the hard way that the number of tabs is limited to 95.

It is my hope that by doing this there will no longer be a need to find jobs through staffing agencies. When I was just starting out I just looked up "Computer Programming" in the Yellow Pages.

<comment><USER7> This looks great. I'll be watching the Austin list. When there is one.

<comment><USER20> I commonly say "I take requests". In part that's because it's easier for me to focus on specific locations but it is quite hard for me to choose as there are so many. In part it's because I already have at least a few companies for hundreds of locations.

One of my little "Trade Secrets" is what I suppose one could call breadth- first job hunting: rather than look for companies in Austin, I look for companies just about anywhere, however quite commonly companies will have more than one location. Just in the last few years it has become common for mid- sized companies to open small development shops - I particularly see lots of SFO/Portland/Seattle companies.

let me check... I have seven Austin companies however I know very well that there are quite a lot more.

I'll post what I have shortly, then dig up some more tomorrow evening or so.</comment></comment></comment>


Speed up your RAID: moving journal to SSD - <AUTHOR> http://insights.oetiker.ch/linux/external-journal-on-ssd/

<comment><USER2> So, I've done this, but its got one pretty important caveat.

If you're using a consumer level SSD, and doing a lot of writes, be prepared for your SSD to die quickly, like possibly within 6 months. Also, when that SSD dies, it may take your ext2/3fs with it (been there, glad I had backups).

If you're thinking about RAID1-ing the SSD, also remember that since this is solid state media, they're likely to die at the same time.

<comment><AUTHOR> The end of article contains "Reliability Impact" part that directly touching the topic deep enough

</comment></comment><comment><USER3> (2008)

Today I would just use bcache.</comment>


Ubuntu Forums has been hacked/defaced - <AUTHOR> http://ubuntuforums.org/signaturepics/Sput.html

<comment><AUTHOR> Here is a backup screenshot I took of the page, in case ubuntuforums is taken down and you want to see what the fuss was all about.

[http://imgur.com/hUX3f1F](http://imgur.com/hUX3f1F)

</comment><comment><USER2> what song is this? it's good

<comment><AUTHOR> If you view source on the page you can see a link to the song. It's being played from an invisible embedded youtube player, so you can get the link from there.</comment></comment>


Show HN: Figure My Bills - <AUTHOR> http://figuremybills.dandoes.net This is my first "real" app I've written in node.js.
This app came to be because the last time I changed jobs, I went from getting paid once a month to once every two weeks. Since two weeks doesn't quite line up the way once a month does, I had a hard time figuring out how much money I needed to set aside to get into the new rhythm. So, I wrote a little script that figured it out for me.
I wanted to play around with node.js, so it was a great excuse to create full app. Let me know what you think! <comment><AUTHOR> I'm not sure what happened to my comment, so here's the back story:

This is my first "real" app I've written in node.js.

This app came to be because the last time I changed jobs, I went from getting paid once a month to once every two weeks. Since two weeks doesn't quite line up the way once a month does, I had a hard time figuring out how much money I needed to set aside to get into the new rhythm. So, I wrote a little script that figured it out for me.

I wanted to play around with node.js, so it was a great excuse to create full app. Let me know what you think!</comment>


Phoneblocks - Ultra customizable smartphone - <AUTHOR> http://phoneblocks.com/

<comment><USER2> Needs a 301 redirect to [http://phonebloks.com/](http://phonebloks.com/)

And phonebloks.com and www.phonebloks.com need a 301 redirect to either www or no www.

And the page encoding is messed up</comment>


Too many choices -- good or bad -- can be mentally exhausting - <AUTHOR> http://www.physorg.com/news127404469.html

<comment><USER2> one trick to avoid exhaustion from too many choices: short circuit when you get a satisfactory outcome, even if it's not optimal. e.g., ordering food at a restaurant or watching tv, if i find a menu item or a program that's good enough and the outcome is not too important to me (those are the cases you should invest extra mental energy to find optimal solutions), i stop scanning there.

the general notion is that your mental energy is valuable currency and should be invested in ways that give you good return.

<comment><USER3> I have a ball of sticky tape, and I choose whatever it sticks too.

</comment></comment><comment><USER4> Barry Schwartz claims that choice makes us feel miserable. A very interesting TED talk, see: <http://www.ted.com/talks/view/id/93>

<comment><USER5> He does a great job of outlining the problem, but the only solution he suggests is spectacularly wrong. He says that redistribution of income will make us (those with "too much" income) happier because it will narrow our choices. But it won't. The great thing about money is that you can buy anything with it, including the reduction in number of choices. The cachet for many companies (e.g. Apple, Costco) is exactly that: consistently high quality and limited selection. And it always costs extra.

My biggest motivation for getting rich is not having to deal with insignificant (but unavoidable if you're poor) everyday problems. Being forced to choose is mostly a subset of those problems.

</comment></comment><comment><USER6> Of course, one should not use such research as an excuse to _eliminate_ choices (as many political types believe). All you need to do is _hide_ the other choices.

A great example of this is the choice of "Desktop or server?"

<http://www.ubuntu.com/>

In fact, that's barely even a choice. But if you want to choose between other options, you can (at the cost of an extra click):

<http://www.ubuntu.com/products/>

In my view, this is the proper way to deal with too many choices.</comment>


I Lost 20 KG with Intermittent Fasting and Strength Training - <AUTHOR> https://medium.com/@lexsapkota/i-lost-20-kg-with-intermittent-fasting-and-strength-training-201fa2405a72 <comment><USER2> Losing 20kg while eating less and training more, who would have thought.</comment>


Ask HN: Please review my project "ShowSpace" - <AUTHOR>

Hey everybody,
This is (almost) the first time I've posted here, but I hang around and read on Hacker News almost daily. After several months of work I've launched my project ShowSpace today in private beta today and I would value your feedback!
http://showspaceapp.com
ShowSpace is a system that lets users easily create niche affiliate storefronts, which are basically websites showing off products from a specific niche. It is targeted at affiliate marketers and the goal of the system is to remove all clutter and technical hurdles from creating such storefronts so that affiliates can focus on what they do best: come up with a niche, find great products, add unique product descriptions and promote the storefront! Some examples of storefronts a friend and I have created to test the system:
http://rosadinge.de/ (German) http://inherzform.de/ (German) http://kramnalle.se/ (Swedish) http://maskeradgarderoben.se/ (Swedish) http://heartshapedproducts.com/ (US)
I would especially like to get feedback on the storefront design and usability and possible pricing structure!
Cheers, Manuel <comment><USER2> Hi Manuel,

Speaking as a long-time affiliate (for better or worse), I'm glad to see you release this in the wild!

So, with that in mind, here's my thoughts:

1\. Overall, I like your concept of making a powerful turnkey storefront for affiliates. I especially like your outsource-ready concept. However, there are tons of others across the world, and many of them are working the work-from- home-churn-and-burn scam and have a hell of a lot of money to burn.

2\. A big difficulty is going to be content and keeping the storefronts valuable. You see, with SEO and PPC, you generally need to have a fairly unique site with some different content. Google will heavily penalize sites that are recycled data or affiliate links with little to no information in them. A good example is this HN post: <http://news.ycombinator.com/item?id=3099875>

Matt Cutts weighed in on this and his feedback was quite clear to that affiliate: "You have an autogenerated web site that consists of practically nothing other than affiliate links to Amazon. You can make an infinite number of autogenerated pages on your site and each autogenerated page consists of literally hundreds of affiliate links stuffed with keywords for unrelated products."

Remember that for search and PPC, the lifeblood of 99% of affiliates out there, content is king. You need to provide many ways for them to do this and site maps, tags and clean code don't go that far.

3\. You're going to face serious competition from the very established combo of Wordpress.com + Skimlinks. It is very easy to make a content-heavy affiliate site using that combo, as blogging is fairly simple and Skimlinks makes the monetization almost instant. If you want to hit those guys, really work on some content tools for affiliates. Writing product descriptions really isn't enough and reviews have to be really verbose to get ranked, so the more tools you provide for that, the better.

With all this in mind, do you have a winner here? Not yet, but you _are_ on to something. Really refine your product and add some core tools. I think you'd be best served by making a solution akin to WP+Skimlinks, but look more at what your users want because it is only my opinion :)

<comment><USER2> As another note, your pricing structure should be free. Be an affiliate for every merchant under the sun through CJ/LinkShare/ShareaSale/GAN/etc. and then have them be sub-affiliates. Keep about 20-30% of each commission, ala Skimlinks.

<comment><AUTHOR> I'm not a big fan of "free" I have to say. :)

I think a low monthly fee could keep away most of the "bad" affiliates who maybe just got into the game and think they will get rich overnight without doing any work.

I haven't decided at all on this point, though, and will talk to a lot more ShowSpace beta users and affiliates in general before making a decision.

</comment></comment></comment><comment><USER3> I guess this is a kind of Shopify with affiliates' data.

I think you're going to have to do a lot more with the designs, by which I mean you need a lot of good templates.

You could try to encourage designers to submit designs, you might have to curate them yourselves for quality, but some form of template structure would be required. It's going to be some "off topic" work to set that up, and you'll have to somehow provide an incentive for the designers to do something though.

Polyvore springs to mind, because actually it gives college fashion students a great way to experiment with colours, layout and design, I'm sure there's a complex engine behind that though. (Those guys are ex-Yahoo.) The point being, that students can be a great source of content, if you give them the right tools/incentive

<comment><AUTHOR> Shopify for affiliates... I like it! :)

Yes, creating a template library would definitely be one option. As you said, it would involve quite a lot of maintenance, though. Maybe creating 3-4 base themes and making it possible to style and adjust them in the backend would be a good start.

Thanks for your feedback!

</comment></comment><comment><USER4> If you're not aware of it already, make sure to show this off over at www.warriorforum.com I suspect they will go crazy for this kind of thing.

<comment><AUTHOR> Hehe, yeah, I am aware of WF. I think I will wait until ShowSpace is ready for prime time, though, WF strikes me more as the place to get loads of users than loads of good feedback. :)</comment></comment>


Ask HN: What made you decide to pursue a startup? - <AUTHOR>

I've been feeling pretty lost these past few days and could use some career advice.
I come from a finance and management consulting background and I've been thinking about ditching it all to pursue a start-up. I'm only 2 years out of college, however, I worry that if I leave the industry and begin working on a start-up full time that the opportunity cost might not be worth it.
It is incredibly scary ditching a well established resume to possibly work on something without any guarantee of success. I've been working on the start-up at night, but the truth is, there simply isn't enough hours in the day to do this effectively, especially when you're working 12 hours or more a day and constantly traveling for work.
I'm really not sure what to do. The thought of creating my own company sounds incredibly attractive, however, I might be missing out on a pretty decent lifestyle with a sizable paycheck. Obviously money isn't the only factor, but it definitely influences my mentality.
For those who've made the jump, what did it take? How do you know if you've made the right decision? Especially if you haven't scored the big exit deal?
My parents absolutely despise the idea of me doing a start-up and I question it myself, I'm just not sure. And I hate how for me, it really has to be one or the other, both my work and my start-up require extensive time commitments.
Any help or advice would be appreciated, thanks. <comment><USER2> First time was in '86 after working for a bank for a while and talking to my boss who laid out for me why I should go and do a thing on my own. Awesome guy (Thanks Eddy!).

I sold that business and traveled for a while after that, freelancing whenever the money pit was at low tide.

It was '93 last time I started from scratch, re-migrating to NL from PL, baby on the way, no outlook at getting a job (minor IT crisis in NL at the time).

Simply a need to put food on the table. Which I can guarantee you is a very powerful motivator, especially when it includes little humans.

I wouldn't worry ao much about what others think about it.

What is more important is that you cut down your costs to the absolute minimum (this may require a serious change in lifestyle) to give yourself as much 'runway' as you can get. It takes a while to get off the ground, and with a lower expense pattern your chances of success increase tremendously.

edit: You know you've made the right decision when you're happy. In spite of all the risks involved (no pension, no social security for me) I wouldn't want to be doing anything else.

</comment><comment><USER3> I've worked in companies for other people; time and time again I would find myself in situations where some sort of decision had to be made; some sort of meeting would be called where I would say "my opinion is that we should do X", people above me would say "no, we should actually do Y", Y would be done, a few months down the line it would be discovered that Y was the wrong thing to do and that X should've actually been done, and we would spend a lot of time backing out Y and doing X.

It sucks to sit at work and think "God damn it, I said we should've done this 6 months ago, and no one listened, and now we are doing it anyway."

I am not purposefully trying to toot my own horn - I just discovered time and time again that often my intuitions are correct, and I should trust them more. Once that became obvious, doing a startup was the only avenue that I could find that would allow me to make my own decisions, bask in the glory that their correctness would bring, and assume the responsibility for the failure that their incorrectness would bring.

<comment><USER2> So how has it been working out for you?

<comment><USER3> So far I've been taking lots of responsibility for my failures :)

<comment><USER2> Nothing wrong with that as long as you don't fail in exactly the same way twice.

Overall which part of running a startup is what you think is the hardest bit?

<comment><USER3> Balance. I end up working fourteen hour days when I feel motivated, burn out on those in about 2 weeks and then have a hard time getting myself back to getting anything done for another week. Over long-term, I would probably get a lot more done and be happier if I limited myself to being productive 6 hours a day every day rather than the other way, but getting the self-discipline to stop working and walk outside when you are on a roll is probably the most challenging thing.

I am not yet at a point where I can collect money from users; I suspect that once I get to that point, the hardest problem will be "how do I get more people to pay?" This seems to be completely different - it seems that right now I am my own enemy, whereas later it will be actual external forces.

</comment></comment></comment></comment></comment><comment><USER4> My experience, just for mulling over:

I've worked in my industry now for over ten years. Over that time, I've made a pretty good living. Now, I've got enough in the bank that I can take time off without worrying about rent or any other basic necessities. I started on my webapp last year part time, but I'll be going full bore on it once my current client gig ends.

On one hand, a good paycheck is nothing to sneer at. I'm self-funded. I'll have something to fall back on if things don't work out (the time off will be a "sabbatical"). On the other hand, time is more valuable than money. If you have something to do, the sooner the better.

Regardless of these two polar views, ultimately the combination of my current idea + opportunity + capability is what pushed me to make this change. I would have done it earlier if it had presented itself then. I'd be continuing on my current path if it never had.

Don't make the switch just because you feel you have to. Keep your eyes open, and jump on it when you see it.

</comment><comment><USER5> I can't remember a time when I didn't want to be CEO of a startup

</comment><comment><USER6> No matter how high up you go in a company, you're always ultimately building out someone else's ideas. My happiness has a direct connection with seeing my own ideas come to fruition.

The full process of coming up with an idea, planning and designing, grinding through all of the shit that comes along with actually tying up all of the loose ends and making it go live, and then being able to say "Yep, that is all my idea, I made it happen" is perfection in my mind.

You don't want to look back at yourself 5, 10, 20, 30 years from now and realize you're doing the same job you started doing. Take the leap, if it fails you'll always have that impressive resume to fall back on anyways.

Good luck.</comment>


How to allocate memory - <AUTHOR> http://geocar.sdf1.org/alloc.html <comment><USER2> If you'd like to learn more, here's an MIT research paper on fast memory allocation that has some really clever ideas: [http://supertech.csail.mit.edu/papers/Kuszmaul15.pdf](http://supertech.csail.mit.edu/papers/Kuszmaul15.pdf)

Abstract:

SuperMalloc is an implementation of malloc(3) originally designed for X86 Hardware Transactional Memory (HTM). It turns out that the same design decisions also make it fast even without HTM. For the malloc-test benchmark, which is one of the most difficult workloads for an allocator, with one thread SuperMalloc is about 2.1 times faster than the best of DLmalloc, JEmalloc, Hoard, and TBBmalloc; with 8 threads and HTM, SuperMalloc is 2.75 times faster; and on 32 threads without HTM SuperMalloc is 3.4 times faster. SuperMalloc generally compares favorably with the other allocators on speed, scalability, speed variance, memory footprint, and code size.

SuperMalloc achieves these performance advantages using less than half as much code as the alternatives. SuperMalloc exploits the fact that although physical memory is always precious, virtual address space on a 64-bit machine is relatively cheap. It allocates 2 MiB chunks which contain objects all the same size. To translate chunk numbers to chunk metadata, SuperMalloc uses a simple array (most of which is uncommitted to physical memory). SuperMalloc takes care to avoid associativity conflicts in the cache: most of the size classes are a prime number of cache lines, and nonaligned huge accesses are randomly aligned within a page. Objects are allocated from the fullest non-full page in the appropriate size class. For each size class, SuperMalloc employs a 10-object per-thread cache, a per-CPU cache that holds about a level-2-cache worth of objects per size class, and a global cache that is organized to allow the movement of many objects between a per-CPU cache and the global cache using O(1) instructions. SuperMalloc prefetches everything it can before starting a critical section, which makes the critical sections run fast, and for HTM improves the odds that the transaction will commit.

<comment><USER3> While it has some new material, it's also not very well-written, and some of the benchmarks could be argued as being unrealistic. For example, Brad should have also included results using applications such as Redis (which by default uses jemalloc), or MongoDB, or many others. In many other scenarios not detailed in the paper, the allocator can use much more memory compared to tcmalloc.

The "Vyukov" benchmark is an invented (and I would argue, contrived) scenario that Vyukov him or herself thought of to cause memory bloat in Intel's TBB just by examining the code. Whether or not it actually occurs in a real application is debatable.

If you may have noticed, nowhere in the paper is tcmalloc mentioned :) It is still a viable alternative today.

There are many other papers I would recommend in addition to this reading, e.g., "Dynamic Storage Allocation A Survey and Critical Review" Wilson et al. despite it being quite old.

This might get me down votes, but it is wise to remember that just because work has the MIT stamp on it, doesn't mean it's always the most top-quality.

<comment><USER4> If you haven't spent much time reading about allocation strategies and are looking for a good place to start, _Dynamic Storage Allocation A Survey and Critical Review_ is a fantastic start. One of my all-time favorite survey papers.

[http://www.cs.northwestern.edu/~pdinda/ics-s05/doc/dsa.pdf](http://www.cs.northwestern.edu/~pdinda/ics-s05/doc/dsa.pdf)

<comment><USER5> That's a good paper, but it's worth noting that most of the interesting work these days is in _multithreaded_ memory allocators, which weren't as important in 2005. Scaling well under multithreading (i.e. not taking a global malloc lock) changes the design space considerably: you need to have per-thread heaps and rebalance them from time to time, which is itself a very interesting problem.

<comment><USER6> For anyone interesting in this area, take a look at the memory allocator (and GC) for HotSpot. Specifically "Hierarchical PLABs, CLABs, TLABs in Hotspot" [0]

[0] [http://cs.uni-salzburg.at/~hpayer/](http://cs.uni-salzburg.at/~hpayer/)

</comment></comment></comment></comment></comment><comment><USER7> > One major limitation of malloc (and even the best implementations like > jemalloc and dlmalloc) is that they try to use a single allocator for each > data structure. This is a mistake: A huge performance gain can be had by > using a separate allocator for each of your data structures — or rather, for > each of your data usage patterns.

I stopped reading the article at this point. This statement has been disproven repeatedly over the years with Hoard and jemalloc. It is counter-intuitive but the data backs it up.

Custom per-data-structure allocators can fragment the global memory arena and cause more CPU cache misses as result of the extra code involved. The latest/greatest malloc/free implementations use a myriad of optimizations to achieve speed improvements that a custom allocator implementation would rarely use.

It's not an accident that jemalloc is so widely used in major applications - it works extremely well.

[https://github.com/jemalloc/jemalloc/wiki/History](https://github.com/jemalloc/jemalloc/wiki/History)

[https://github.com/jemalloc/jemalloc/wiki/Adoption](https://github.com/jemalloc/jemalloc/wiki/Adoption)

<comment><USER8> > This statement has been disproven repeatedly over the years with Hoard and > jemalloc. It is counter-intuitive but the data backs it up.

What data is that? I would honestly love to see it. TBH, it would not change my mind since I have practical experience of custom allocators working better for me over other general ones. But then again, I work in a severely memory constrained environment.

>Custom per-data-structure allocators can fragment the global memory arena and cause more CPU cache misses as result of the extra code involved.

Well, pretty much all modern allocators have techniques to avoid fragmentation. But that is besides the point.

But your comment did not make sense to me. Perhaps it does to others. As the developer you know the general sizes of your data structures, their usage patterns, their lifetimes and the frequency/rate at which they are allocated (all of which a general allocator has to guess or use some sort of rudimentary strategy which adds needless metadata). That is where you can design your custom allocator to actually reduce fragmentation. It may well be that the gain from a custom allocator isn't worth it, but I don't quite understand how it can get _worse_ as you claim.

>The latest/greatest malloc/free implementations use a myriad of optimizations to achieve speed improvements that a custom allocator implementation would rarely use.

So what exactly stops you from writing in those optimizations in a custom allocator? Certain domains, like game programming for e.g. People have been doing this for decades with excellent results.

>It's not an accident that jemalloc is so widely used in major applications - it works extremely well.

Sorry, you have no possible way of knowing _why_ everybody else uses something. That itself is not an argument for anything.

<comment><USER7> Custom allocators have the disadvantage of not knowing how the rest of the heap is managed. They are at a disadvantage to a modern well designed memory allocator like hoard or jemalloc.

Feel free to examine the many papers put out by Emery Berger on the subject of memory allocation as well as the design documents of jemalloc.

<comment><USER8> >Custom allocators have the disadvantage of not knowing how the rest of the heap is managed.

Sure, but they can work out of a fixed region statically allocated on startup.

>Feel free to examine the many papers put out by Emery Berger on the subject of memory allocation as well as the design documents of jemalloc.

[https://people.cs.umass.edu/~emery/pubs/berger- oopsla2002.pd...](https://people.cs.umass.edu/~emery/pubs/berger- oopsla2002.pdf)

Do you have links to more? I only found one paper from 14 years ago. It is interesting but it comes with a huge YMMV since they only examined specific projects which might not have a pattern that _requires_ a custom strategy anyway. A distribution of the objects along with time distribution would have been nice to include.

</comment></comment></comment></comment><comment><USER9> Not to be a downer, but this is poor and outdated C programming style.

Don't use sbrk(2) unless you're completely reimplementing the standard library.

Don't use alloca(3) unless you want weird headaches. Just use dynamically- sized arrays, which have been present in C for nearly two decades. They are block-scoped and easier to reason about.

None of the magic numbers are explained. `(p->si_addr + (16LL<<22)) & ~4095`, where do those come from? I'm guessing 4095 is the page size less one, but DON'T hardcode that!

Don't assign to lvalues in conditionals! (e.g. `while(j<31 && !(h=free_table[j]))`) It's hard to read and bug-prone.

Don't use `&h[1]` to refer to the space after a structure. I'm pretty sure that's undefined behavior, and it often gives the wrong alignment for whatever you want to put after h. Rather, add a final element to the structure of flexible array type (say `int data[]` if you're storing ints after the structure). That is guaranteed to have the semantics you're looking for.

Here's my advice:

1) Just use malloc(3). It's already fast and tuned to many allocation scenarios (including all four in this article!), and your application will continue to reap whatever performance improvements its maintainers make. Use aligned_alloc(3) if you need pages.

2) When working in the embedded world, it's often preferable to preallocate pools as large as your app will ever need for each kind of object you have collections of, and pin them to RAM (not necessarily for performance, but so you know you have the space). If you do so, you can reference the objects by index rather than pointer. It's much faster and more space-efficient, especially on 64-bit architectures.

3) Don't be afraid to realloc(3) to grow arrays. It incurs no asymptotic penalty.

4) Pay attention to data layout, to make sure that commonly-accessed stuff is not interspersed with rarely-accessed stuff. E.g. separate indexes from data where possible, if you tend to traverse the index rather than the data.

(Caveat: all my comments above apply to glibc. YMWV on other systems.)

<comment><USER10> _Don 't use `&h[1]` to refer to the space after a structure. I'm pretty sure that's undefined behavior_

This is legal. '&h[1]' is a synonym for 'h + 1', thanks to C99 section 188.8.131.52 paragraph 3:

        ... if the operand is the result of a [] operator, neither the & operator
        nor the unary * that is implied by the [] is evaluated and the result is
        as if the & operator were removed and the [] operator were changed to a
        + operator.)

and that is valid thanks to C99 section 6.5.6, paragraphs 7 and 8:

        7. For the purposes of these operators, a pointer to an object that is
        not an element of an array behaves the same as a pointer to the first
        element of an array of length one with the type of the object as its
        element type.
        8. When an expression that has integer type is added to or subtracted
        from a pointer, the result has the type of the pointer operand. [...]
        Moreover, if the expression P points to the last element of an array
        object, the expression (P)+1 points one past the last element of the
        array object [...] If both the pointer operand and the result point
        to elements of the same array object, or one past the last element of
        the array object, the evaluation shall not produce an overflow; otherwise,
        the behavior is undefined. If the result points one past the last element
        of the array object, it shall not be used as the operand of a unary *
        operator that is evaluated.

So is 'h' points to an object it is entirely legal to compute '&h[1]'; whether you can dereference it is a separate question (and will depend on memory layout et cetera).

<comment><USER9> Ah, I forgot that "one past the last element" is excepted. I stand corrected. (Similar tricks, such as &h[-1], are indeed undefined behavior.)

Regardless, my point stands that it can give you the wrong alignment, if the elements you're putting _after_ the header have greater alignment requirements than the header itself. E.g.:

        #include <stdint.h>
        struct foo { uint16_t a,b,c; }
        uint64_t *bar(struct foo *x) { return (uint64_t *) &x[1]; }

Gives you a misaligned pointer to an 8-byte object.

        #include <stdint.h>
        #include <stdint.h>
        struct foo { uint16_t a,b,c; uint64_t d[]; }
        uint64_t *bar(struct foo *x) { return x->d; }

That gives you a properly aligned pointer.

</comment></comment></comment><comment><USER11> I don't mean to be spammy about this, but I feel like I should repeat it. If you like this kind of content, consider joining the SDF. It's a free public access computing community with lots of artists, hackers, and grognards of all stripes. Your support is appreciated: [https://sdf.org/](https://sdf.org/)

</comment><comment><USER12> sizeof returns size_t, not sure why he's using `long long`, but to be precise it isn't a function, so it doesn't "return" anything. I guess it "yields" size_t.</comment>


A dashboard of Coronavirus outbreak in India - <AUTHOR> https://covidout.in/ <comment><USER2> Would India's relatively slow infection curve relate to poor reporting, control measures, or other factors - perhaps widespread chloroquine use?

<comment><USER3> I think poor reporting, governments is taking more control measures

</comment></comment><comment><USER3> Nice Design and thanks for this site with information :)</comment>


Building websites with science - <AUTHOR> http://codeascraft.etsy.com/2012/06/21/building-websites-with-science/

<comment><USER2> For an article that refers to data-based decisions, what was the purpose of white on black retina-burning colors for this blog? Is 'visitor should exit halfway through the article' a goal? :)

<comment><USER3> I think you might have been one of the 10% who got white on black -- they are just doing science.</comment></comment>


A Linux Compiler Deathmatch: GCC, LLVM, DragonEgg, Open64, Etc... - <AUTHOR> http://www.phoronix.com/vr.php?view=15657 <comment><USER2> Link to the results:

[http://global.phoronix-test- suite.com/index.php?k=profile...](http://global.phoronix-test- suite.com/index.php?k=profile&u=staalmannen-11555-17359-8826)

<comment><USER3> Scroll down for bar graphs of random polarity with occasional spurious entries using a pittance of available visual coding.

I think giving each compiler a hue and encoding the optimization level as saturation would have been a useful addition to the spatial organization. Rephrasing the data so that "big = good" would also save much of the mental "what am I looking at?" diversion at each chart view.

</comment></comment><comment><USER4> If only there was some sort of Standard Performance Evaluation Corporation out there with a wide variety of CPU intensive benchmarks to evaluate architectures and compilers.

Not getting a SPEC license makes this strictly amateur hour. And yes, I know compilers are tuned for SPEC, but you'd at least get repeatable high-quality results on a wide range of different programs.

SPEC isn't necessarily representative of every program out there, but his weird (and small) collection of microbenchmarks and random fragments of code is even less interesting than SPEC.

</comment><comment><USER5> Is there some actual content on the middle of all these ads? I gave up on page 2.

<comment><USER6> Some points that appear to come out of the data:

tcc (tiny c compiler) appears to be really good - it scores very well in both compilation time and execution speed

open64 (added in second batch of tests) also appears very good, for the same reasons

clang is quite slow.

Programs compiled with gcc -O2 often run faster than with -O3.

<comment><USER7> I think you're referring to another benchmark. AFAICS, this one doesn't test clang or tcc.

<comment><USER6> From the main article link I clicked through to the forum thread[1] where the tester posted his benchmarks. There's more than one there as he progressively tested different compilers and different setups; I looked at the first 2-3.

[1] [http://phoronix.com/forums/showthread.php?29090-CompilerDeat...](http://phoronix.com/forums/showthread.php?29090-CompilerDeathMatch- surprising-results)

<comment><USER8> The forum thread on page 3[1] however reads:

      side-by-side comparisons do however indicate that Clang performed as good or better than ICC or GCC when -O2 is used!

And through the link by yread you can see that clang really performs very well overall.

[1]: [http://phoronix.com/forums/showthread.php?29090-CompilerDeat...](http://phoronix.com/forums/showthread.php?29090-CompilerDeathMatch- surprising-results/page3)

</comment></comment></comment></comment></comment><comment><USER9> They're testing on an Atom platform, which is pretty horrible if you're comparing performance.

Many of the optimizations that compilers can do nowadays rely on better than an in-order "barely better than a Pentium" CPU.</comment>


Goldman Adds Pay, Perks in Silicon Valley War for Tech Talent - <AUTHOR> https://www.bloomberg.com/news/articles/2017-08-15/goldman-adds-pay-perks-in-silicon-valley-war-for-tech-talent <comment><USER2> > That would probably climb to more than $100,000

$100k for talent in NYC is not good.

<comment><USER3> For entry level, 100k is rather good.

<comment><USER4> If you consider all entry level engineering jobs I agree 100k is good. But I don't think GS is targeting all entry level engineers. They are targeting those that are considering Google, Facebook, Uber, AirBnB, etc. For that segment of the market, 100k is the minimum bar, plus you'll probably have to through in a nice signing bonus.

</comment></comment></comment><comment><USER5> They will never be able to offer the freedom of silicon valley.

<comment><USER6> What freedom is that?

<comment><USER5> Freedom to have a nap or work odd hours or work remotely. Freedom to wear the clothes you want. Freedom to leave your google job in the morning create a startup in the afternoon and get funded during dinner and have the company fold before last call.

You have the freedom to start a side project and find success or fail. You have the freedom to give something back to the open source community.

At some smaller places you can eat lunch with the owner beat them at pool and have a laugh over a beer on some sunny friday afternoon.

All things not possible working at gs because you work 16 hour days. The goal is to try to get a leg up on someone else because everything is based on the annual bonus.

<comment><USER7> Goldman relaxed it's dress code for tech employees: [http://www.reuters.com/article/us-goldman-sachs- technology-i...](http://www.reuters.com/article/us-goldman-sachs-technology- idUSKBN19Y17J)

I think banks are realizing that changes need to be made, so they're happening, albeit slowly.

</comment></comment></comment></comment><comment><USER8> > It's using HackerRank, a web platform that trains and grades people on > writing computer code

What is HN's opinion on HackerRank and the like?

<comment><USER9> I used it recently while preparing for interviews. I really liked it, especially their challenges which pair a small lesson with a coding problem. If you work your way through the different "tracks" it's a great way to focus on a particular area.

Also, I had the best experience while doing technical screens on HackerRank, every other platform had issues of one sort or another.</comment></comment>


Ask HN: Could e-gaming professionals become footballers? - <AUTHOR>

IdrA Fanclub interview: Description A quick interview of StarCraft 2 professional gamer EG… http://goo.gl/fb/uaXsQ<p>Hardly an interview by Charile Rose to be fair but this guy is really good in the game, has a fair following, there are now pro-gaming leagues, 24x7 replays and live casts on the net and on TV in Korea. Personalities like husky starcraft - is this the new celebrity in the next 5-10 years? Will they replace or add-on to the old? <comment><USER2> The link has a prob. <http://youtu.be/Ea8DlGM7e14>

Have you seen this Idra Interview? HaHaHa <http://youtu.be/pHG40AdYfJ0>

I can picture Day9, HD StarCraft, working at ESPN in the near future.

HD StarCraft's videos have over 57 Million views, and Google is listening.

Check out this Google Tech Talk. HDStarcraft and Painuser to discuss eSports. <http://youtu.be/Y0Stg0f1360></comment>


Bootstrap Vue – Bootstrap 4 components for Vue.js 2 - <AUTHOR> https://bootstrap-vue.github.io/ <comment><USER2> This is a promising project, however, in its current state it is at best a component library starter rather than a library you can build a full SPA with. A simple gap is any component relating to date and/or time input.

As a comparison, look at [http://element.eleme.io/#/en- US/component/layout](http://element.eleme.io/#/en-US/component/layout) and the number of components they support. Please note that Element has been posted on HN before and it also has its own set of limitations - not-mobile being the biggest issue.

At work we've built out an internal app with Element and found it to have saved enough time to be worth the compromises. Furthermore, how pretty it is on mobile is less of an issue for an internal application where the users have less choice.

BTW, another promising project is VueJS AdminLTE [https://github.com/rorikurniadi/vuejs- AdminLTE](https://github.com/rorikurniadi/vuejs-AdminLTE) or [https://github.com/misterGF/CoPilot](https://github.com/misterGF/CoPilot) however the issue with these are that they are project starters so you can't just import the components you need. Furthermore, they have different levels of componentisation completeness.

Although I may be being a bit harsh on the projects, but the underlying issue is that components should follow some level of atomic design philosophy so that we can build more complex components ourselves. However, componentising something like bootstrap takes time as one needs to make decisions on props, possible models, different internal states to support, ideally write tests, and keep everything as decoupled as possible so people can import what they need.

Such an undertaking is not just a large effort, but quit tedious as well. With how quickly JS moves, the likelihood that by the time someone completes a Bootstrap 4 to VueJS 2 implementation, that VueJS 3 will be released and they would have to start migrating again. Therefore, when looking at projects like these, I tend to prefer choosing ones that come out of a company that at least needs to scratch its own itch and maintain components needed for their use cases. It may not cover all, but at least it covers some real life scenarios.

<comment><USER3> While I agree in principle with your points, playing devil's advocate it seems to me that if you were wanting to integrate Bootstrap 4 with Vue 2 this project is laid out 110% better than most of us could do on a good day.

And who wouldn't want to use both of these projects together.

Seems like a sweet spot to me.</comment></comment>


Anatomy of a Failed Pitch at Startup Weekend - <AUTHOR> http://erik.randomdrivel.com/articles/12/anatomy-of-a-failed-pitch-at-startup-weekend-san-jose/

<comment><USER2> I'm a mentor at a program kind of similar to this (except it's 1 evening a month for 6 months or so).

I see this all the time. Everyone comes along with their own idea 90% of which are mediocre (and the 10% that aren't usually have their own team sorted out)

The real solution _isn't_ to try and sell your idea, it's to change your expectations.

It's _very_ unlikely anything you build on one of these programs will be any good unless you were already working on it with an existing team prior to the start of the program

But it _is_ likely that you will meet some people who - at some point in the future - might be interested in doing some - as yet undefined - project with you.

Treat it as a learning experience, where your expected outcome is to learn something and to hopefully meet some good people.

<comment><AUTHOR> The learning part of the experience was definitely the most important part... A handful of teams built really impressive shippable products.

The hope or fantasy that your idea or your team will be one of those teams is part of what fuels and attracts people to these events. I didn't expect a Wayne's World type ending or anything.

I just acknowledge that I could have done more hustling, and pitched better...

Had I done a better job pitching/mingling my weekend would have been much different, BUT I'm not sure I would trade the experience I DID have with the team I landed on for anything (well almost anything...)

</comment></comment><comment><USER3> One thing I learned from these startup weekends, too, is that it's just one weekend. You weren't necessarily forced or desperate to find a team or co- founder there, especially with it being your first startup weekend-style event. It's just a mindset.

<comment><USER4> Yeah...it's a great place to meet people, but it's a bit like going to a bar and looking to get married instead of just trying to get a phone number. You're likely to be disappointed if you're expecting a diamond ring.

<comment><AUTHOR> True... but sometimes I go to the bar just to hook up ;) But your analogy still holds true.

Managing expectations and emotions seems like an important piece to building a startup. Things like pride and ego can help fuel you, but can also get in the way of clear thinking.

</comment></comment></comment><comment><USER5> You looked like a lot of fun. I wish we'd been on the same team.

<comment><AUTHOR> thank you. On one hand I would have loved to have worked / launched <http://droprocks.com> this weekend, but I think I got much more (personal growth/learning) out of joining the team I did.

</comment></comment><comment><USER6> Eek, I imagine the dropbox guys would hate you for turning them into the RIAA's latest target

<comment><USER7> I'm pretty sure all the recent posts showing the RIAA how to determine what files exist on dropbox are going to do a much better job of that than his pitch will.</comment></comment>


Some of the story of ARM, including the visit to the 6502 creators - <AUTHOR> http://www.reghardware.com/2012/05/03/unsung_heroes_of_tech_arm_creators_sophie_wilson_and_steve_furber/

<comment><USER2> The first version of the BBC Arm based Archimedes came with a very simple demo called 'lander' by Braben (of Elite fame).

I got one of the machines through my contacts at the dutch importer of Acorn (ARM stands for "Acorn Risc Machine") and for days after it arrived I could do nothing else but fly around in this real time generated 3D scrolling landscape piloting a flying wedge and shooting up stuff.

It was mesmerizing. Compared to today's game titles it looked very simple but the responsiveness of the machine to the mouse while moving the craft was just amazing. People from all over town came around just to play with it. At that time a typical game that included '3D' would render hidden line removed wire frames at best. Contrast that with Lander displaying filled and shaded quads to generated the landscape, small polygons used to depict smoke and fire and other goodies. There were some short-cuts taken to be able to do the 3D transformations fast enough (for instance, only two directions to move in with a fixed camera angle, 'third person' viewpoint) but whatever the details Lander was a huge step in gaming and worked very well to show off the power of the ARM, even though the game that eventually got released based on that demo was disappointing (it was called virus).

<comment><USER3> Here is a video: <http://www.youtube.com/watch?v=653Ger80ros>

It is absolutely astounding for 1987. I guess that out of an SGI IRIS nothing could achieve anything similar back then.

</comment></comment><comment><USER4> Best bit:

"The development board plugged the chip into had a fault: there was no current being sent down the power supply lines at all. The processor was actually running on leakage from the logic circuits. So the low-power big thing that the ARM is most valued for today, the reason that it's on all your mobile phones, was a complete accident."

<comment><USER5> Worst bit:

Showing a photo of Wilson pre-transition. It added nothing to the story, being just a mug shot. I doubt she's happy about that...

<comment><USER6> Nobody can change the past. Pictures of faces are good in articles about people and events since humans are wired to respond to faces so I don't see that it didn't add anything. Therefore I'm not sure that it was out of line. But it's possible that I'm just tone deaf to to the polite/compassionate thing here.

<comment><USER5> If you just want to show Wilson's face, you use a current mug shot, not one dating from 20 years ago. If it was a photo of Wilson soldering up a circuit, or working at a computer during the ARN project, then sure, use the photo as it is showing something about the development of the ARM processor, the putative subject of the article. But even then I would hesitate with annotating the photo as being of Wilson, I'd probably tend towards using a generic term, like 'an Acorn employee' or 'one of ARM's designers'.

<comment><USER7> Why? It doesn't seem at all unusual to use from-that-time photos of the other people who worked on the project, does it?

</comment></comment></comment></comment></comment><comment><USER8> > ARM has now sold over ten billion processors, ten times more than Intel

I don't think ARM has sold processors for a long time. IP cores and designs, yes, but processors are actual physical items.

</comment><comment><USER9> Well, that was quite worth the read. Very motivating, what an individual and a small team can accomplish.</comment>


Show HN: Fetchful - Fetch page information previews from any link - <AUTHOR> http://www.fetchful.com

<comment><USER2> Great idea!

I see:

      Could not generate a preview, please check the URL is valid and   does not redirect then try again.

It seems like the service is down, so the error message is probably incorrect. That being said, you should follow redirects, this is a no-brainer. You can make HEAD requests to save bandwidth. Also, this one is minor, the enter key should submit the form.

<comment><AUTHOR> It does follow redirects but I think javascript redirects are causing a problem occasionally, still a couple of issues I need to work out. The occasional url does not get parsed properly for some reason which is probably causing this error.

</comment></comment><comment><USER3> Very cool, question: How do you know what's the official image of the page? I fetched google.com and it seems to "extract" the right image. However I don't see anything special in the <img> for google.com.

<comment><AUTHOR> A few different ways, it will favour the open graph image or meta 'image_src" tags if there are any. If not, it will take all the image tags on the page and then decide what's relevant based on the correct size and position on the page which is usually fairly accurate.

<comment><USER3> Nice job!

<comment><AUTHOR> Cheers :)

</comment></comment></comment></comment><comment><USER4> Is there a paid plan? I'd love to use something like this, but before I build on it, I'd like some assurance it will stay around for awhile.

<comment><AUTHOR> No paid plan at the moment, but if was something that people were genuinely interested in using, I would consider a (very cheap) paid plan. It will be sticking around 'as is' for a good while at the very least though.

</comment></comment><comment> saiko-chriskun how does this compare to embed.ly?

<comment><AUTHOR> It's similar, but I would like to think it's slightly faster due to it's simplicity. Also, as far as I can tell, embedly is not so good at finding relevant information from sites with no proper image and description metadata. It's really just a small project that I wanted for myself and thought others might find useful as well.

</comment></comment><comment><USER5> This is great. Thank you!

<comment><AUTHOR> No problem, do let me know if you decide to use it at all, I would be interested to see. I'm no javascript expert so my jquery examples with tooltips etc are pretty basic, I'm sure it could be used for more interesting things.

<comment><USER5> Can you add support for multiple URLs?

<comment><AUTHOR> It would be possible but I'm not sure that would be that much more useful than just sending a separate request for each url. If there were lots of them, they could just be looped through. Also, with lots of urls, the response could start getting quite slow. Maybe I'm missing the point of what you meant?</comment></comment></comment></comment>


Dashmap: Fast concurrent HashMap for Rust - <AUTHOR> https://github.com/xacrimon/dashmap <comment><USER2> If you're interested in this, you might like this live-coding session implementing Java's ConcurrentHashMap in Rust: [https://youtu.be/yQFWmGaFBjk](https://youtu.be/yQFWmGaFBjk)

</comment><comment><USER3> Nice! But a bit disappointing that read speed doesn't increase with number of threads. Curious what the bottleneck is here (first thought is it can't be the memory because CHT shows higher performance for reads in the graph). A comparison with a read-only hashmap would be nice here.

<comment><USER4> Hey! I do know what the issue is I think and it is something that will be resolved with v4 releasing later this year.

It's an atomic architecture specific thing.

</comment></comment><comment><USER5> I know this is a bit out of date, but I remember there was a port of the Scala Ctrie data structure to Rust using Hazard pointers. I would be quite interested in seeing how the performance of a snapshotable lock-free structure compares to this

[https://github.com/ballard26/concurrent- hamt](https://github.com/ballard26/concurrent-hamt)

<comment><USER4> The Contrie library is in the benchmarks and is a port. That said referring the note in the repo the benchmarks arent 100% scientific atm and will be revamped later this year.

</comment></comment><comment><USER6> I am curious how this differs from chashmap:

[https://gitlab.redox-os.org/redox-os/chashmap](https://gitlab.redox- os.org/redox-os/chashmap)

They provide benchmarks, but I would be more interested to know how the implementation differs.

<comment><USER4> Hi! I am the author of dashmap. CHashMap is essentially a table behind an rwlock where each slot is also behind its own rwlock. This is great in theory since it allows decently concurrent operations providing they don't need to lock the whole table for resizing. In practice this falls short quickly because of the contention on the outer lock and the large amount of locks and unlocks when traversing the map.

dashmap works by splitting into an array of shards, each shard behind its own rwlock. The shard is decided from the keys hash. This will only lock and unlock once for any one shard and allows concurrent table locking operations provided they are on different shards. Further, there is no central rwlock each thread must go thru which improves performance significantly.

</comment></comment><comment><USER7> I see Jon G referenced in the readme. Is this work based on his livestream series where he ported Java's concurrent Hashmap to Rust?

I love his youtube streams. He's extremely patient and thoughtfully thinks through problems in a similar way to me, making it easy to follow.

Regardless, kudos and great work. I'm sure I'll find a use for this in some of my tokio projects.

<comment><USER8> Nope, Dashmap is all xacrimon, and came on the scene long before my port. We've been collaborating on writing a shared benchmarking suite over at [https://github.com/jonhoo/bustle/](https://github.com/jonhoo/bustle/) though. For the time being, it looks like Dashmap outperforms the port of ConcurrentHashMap (called "flurry"), often by a significant amount. It seems to be mainly due to the garbage collection scheme flurry uses, but we're still digging into it (maybe you want to come help?).

In any case, I'm glad you enjoy the videos!

<comment><USER7> Ha "straight from the horse's mouth."

I'd love to help out if I can.

<comment><USER8> Awesome! Some good places to read up on and join the discussion are [https://github.com/jonhoo/bustle/issues/2](https://github.com/jonhoo/bustle/issues/2), [https://github.com/jonhoo/flurry/issues/50](https://github.com/jonhoo/flurry/issues/50), and [https://github.com/jonhoo/flurry/issues/80](https://github.com/jonhoo/flurry/issues/80). Happy to guide you further there!

</comment></comment></comment></comment><comment><USER9> Some feedback:

1\. Mind the difference between concurrency and parallelism. This is around safe parallel access. Concurrent access can happen on one thread without any synchronization.

2\. It's oftentimes an anti pattern to model thread safety around primitive data structures as opposed to higher level concerns. It forces all data that has to be consistent across thread boundaries to be in this one map. When circumstances change, you might want to have some of that data in different data structures and still provide consistent access to them. This change will be hard when you rely on data structure level thread safety.

<comment><USER10> I've always seen parallelism described as a special case of concurrency where the concurrent processing happens to run physically at the same time, while general case concurrency is about logically running at the same time (whether its actually at the same time or just appears that way).

The use of the word concurrent here is consistent with that definition and the literature, where concurrent data structures are data structures designed to for safe concurrent access (whether they are accessed literally at the same time or not).

</comment></comment><comment> asdf-asdf-asdf 11 uses of "unsafe".

(this is not a critique of this specific library, it's more a look at the rust ecosystem as a whole)

i keep looking at Rust, but at the end it seems it is not a language for me. Rust developers just seem to use more "unsafe" than what i am comfortable with. generally, if there could be a choice between using "unsafe", and taking a 2% performance penalty,i personally would go with the performance-penalty. of course, i can understand others have different priorities. the question is, what are the priorities of the rust ecosystem? i mean, can i find libraries that go with as-safe-as-possible or are most libraries as-fast-as-possible?

also, the claim that the rust language is fast and safe becomes harder to accept when the fast libraries use unsafe :) (i do understand code using "unsafe" can be safe if the developer does not make mistakes. the problem is, developers do make mistakes.)

<comment><USER11> I think... you miss the point.

Usage of unsafe is effectively flagging areas for peer review. You can't build everything in safe Rust - certain things _require_ the use of unsafe. Having it gated, reviewed, and so on is effectively a check on a class of bugs that can be hard to pin down.

IMHO, the community cares way, way too much about the mere sight of an unsafe in a codebase - it borders on religious zealotry. It's just a tool like anything else in the (wonderful) language.

<comment> asdf-asdf-asdf i don't think i missed the point here.

i wrote: "i do understand code using "unsafe" can be safe if the developer does not make mistakes. the problem is, developers do make mistakes."

you wrote: "Usage of unsafe is effectively flagging areas for peer review. You can't build everything in safe Rust - certain things _require_ the use of unsafe. Having it gated, reviewed, and so on is effectively a check on a class of bugs that can be hard to pin down."

it's the same thing. the difference is that you look at it from the glass- half-full point of view (it's good that must-be-verified-by-a-person blocks are limited here), and i do from the other end (it's bad that these blocks are necessary).

<comment><USER12> I think you missed parent's point. There are constructs that the current compiler _can 't_ prove is correct and to write such code you need unsafe. It is often not about a trade-off between speed/safety.

<comment><USER8> I actually gave a talk about exactly this a few weeks back that may be relevant: [https://youtube.com/watch?v=QAz- maaH0KM](https://youtube.com/watch?v=QAz-maaH0KM)</comment></comment></comment></comment></comment>