REST and long-running jobs (2014) - <AUTHOR> https://farazdagi.com/2014/rest-and-long-running-jobs/ <comment><USER2> One thing this piece leaves out is that if the operation/creation is in fact expensive, you want to make sure you have some kind of two-phase option so you don't inadvertently create multiple copies.

If the client crashes after the POST to create 'death star' succeeds and hence construction starts, but before it can read the response of the POST in the example and so never gets the pointer to the /queue/12345 from the response, if the client reissues the POST in the server should reject a second attempt to create a star named 'death star' just and give back the same pointer to /queue/12345\. Or, if you want to be able to have two stars named 'death star', you should first POST to some endpoint to get a unique identifier from the server first, and then use that to kick off the creation of your death star so the client can reason about the current state.

<comment><USER3> Or you could just generate an unique identifier on the client and save the roundtrip? Uuids aren't really expensive.

<comment><USER4> UUID is not enough to prevent duplicate resource creation. Consider the following steps: 1. Receive a request with uuid, 2. Persist the fact that system received this request, 3. who or what processes it? Even if you are working off atomic persistence, the resource creation process (the kick-off process) needs to maintain the state that it plucked one off the queue or table, atomically update the state in persistence as it creates the resource somewhere else and log the successful creation of this resource as yet another entry in the persistence store. This way, requests that exceed their SLA can be retried and the downstream systems must act with correct idempotent behavior in order for all of this to work correctly in the presence of failures.

It is all very technically possible, but it goes beyond just a UUID (in other words, UUID is necessary but not sufficient).

<comment><USER3> The issues you list don't really seem to be relevant in regard to duplicate resource creation? They're just about handling creating the resource in the first place.

As long as the uuid is persisted you can redirect all attempts to recreate that id to the right job/queue/whatever actually backs resources.

</comment></comment></comment></comment><comment><USER5> _> Your first instinct might be: "What if I return HTTP 201 Created immediately, but defer the actual creation to some later point?"... Well, you can't do that. If you do, you will be violating the HTTP/1.1: Semantics and Content protocol

> We know what status code to return, what about Location header? Simple. > Instead of the location to the actual resource (/stars/97865), API will > return location of the queued task that got created (/queue/12345)

> How to learn when resource is finally available? You need to query the > queued task... Once resource is created, API should respond with 303 See > Other status code on all the subsequent requests to the queued task... There > are two alternative ways to deal with the temporary task resource: API > client must issue DELETE request, so that server purges it. or, garbage > collection can be a server's job to do_

I've always wondered about how important it is to follow REST conventions, at the cost of simplicity, when you control both the server and client software.

In the example given above, the recommendation to use a 202 instead of 201, sounds perfectly reasonable. However, creating and returning to the client a new queue resource, which returns a 303 when complete, and the client then has to delete... Is it really so bad to just return /stars/98765 immediately? And when queried, have the relevant GET API return a response indicating that construction of the resource is still in progress? If the client needs to wait for construction to complete before doing something else, can't it just poll /stars/98765 instead of /queue/12345?

More generally speaking, if you're designing an API for use by your own client, and you're facing a tradeoff between interface/implementation simplicity vs following the official REST protocol, is it really worth choosing the latter?

<comment><USER6> HTTP is an inherently complex protocol, which has over time accrued many idiosyncratic, non-orthogonal features to support various use cases of the growing Web. Just consider that there exists an entire class, entire design space of libraries known as "HTTP routers" which boil down to _extracting arguments from the first line of an HTTP request_.

If you want simplicity, and you fully control both sides, and you don't care about the systemic advantages that REST purports to provide, then your best bet is to avoid HTTP altogether (which in practical terms may of course mean tunneling through it) and stick to a simple, modern RPC protocol.

<comment><AUTHOR> > Just consider that there exists an entire class, entire design space of > libraries known as "HTTP routers" which boil down to extracting arguments > from the first line of an HTTP request.

This comment sounds a bit disingenuous. Routing involves way more than extracting arguments from the first line of an HTTP request, but its complexity is not due to HTTP. Routing is based on content negotiation, and essentially everyone is free to design their own personal content negotiation process, and very often they do.

Take, for example, content type. Do file extensions matter? Does the Content- Type header mean anything? If both are used what should prevail? What should the router do if none was passed?

On top of that, then let's add HATEOAS, HAL, content type API versioning, custom headers, etc...

In the end developers need to map HTTP requests to an action, and HTTP is not the problem.

Libraries are helpful not because the problem is complex, but because ready- made solutions are better than rolling our own. There are plenty of libraries not because HTTP is complex, but because plenty of people have their personal preference.

</comment></comment></comment><comment><USER7> This author misses a key algorithm design option: the API does NOT offer a CREATE on such a resource, the API offers a CREATE on a JOB which generates the long-time-to-create resource. So the API client simply creates the Job, receives an ID immediately, which can be queried for status, forever. Long after the job completed it serves as a timestamped record of that specific resource's creation. It is only after the Job status reaches predefined status levels are the assets (files/data) associated with it available.

I've used this pattern with success for over a decade. First with on-demand digital products, which required compiling/generating time after customer request. This pattern also supports partial generation of long-time-to- generate resources. I used it with a custom game avatar service, where various modification and customization options are visualized without creating them. The visualization of the modified product is a 'milestone' asset generated by the resource creation pipeline. If the end-user chooses not to purchase said modification, that specific asset's sub-job is killed or never started.

With complex, multi-layered digital products this pattern works quite well. Even after a period of time, the end-user could return to a multi-leveled job and request an asset they previously denied. That sub-job gets timestamped just the same as if it ran originally. It just works.

<comment><AUTHOR> >This author misses a key algorithm design option: the API does NOT offer a CREATE on such a resource, the API offers a CREATE on a JOB which generates the long-time-to-create resource. So the API client simply creates the Job, receives an ID immediately, which can be queried for status, forever.

You should read the article because that's the approach it describes.

</comment></comment><comment><USER8> A coworker and I were discussing this very thing on Friday. He and I exhausted our Google-fu without success: Does anyone know of a framework/library that implements this? Don't really care about the language, but per other comments we're interested in the practical lessons learned that would show up in code.

</comment><comment><USER6> RFC 7540 also defines an optional way for the client to signal whether it wishes the request to be processed asynchronously: [https://tools.ietf.org/html/rfc7240#section-4.1](https://tools.ietf.org/html/rfc7240#section-4.1)

<comment><AUTHOR> My first impression is that RFC7540 isn't appropriate for this use case, because it is proposed as a way for the client to signal optional/prefered behavior when said behavior is already made mandatory by the server.

To put it differently, why would it matter if a client POSTs with Prefer: respond-async if the server response will always be async? Whether that header is present or not, the HTTP response is already designed to always return a Location header.

</comment></comment><comment><USER9> Someone once gave a talk on 'the Art of API design' that covered logical design of a RESTful API and one of their points was in close alignment with the article.

Can't find and slides to share but will edit/reply to self if I can, the talk was very informative and covered not just resource creation but destruction, Discovery, and even a bit about authentication.

</comment><comment><USER10> Anybody here who used this pattern in practice and found out it is not as good as this article portrays it?

<comment><USER11> Bottom line is if you're doing this via REST and not using web sockets (which is fine, of course!) then you're going to end up in a polling situation. That's the important thing to think about. I had a bulk export job that ran similar to this, minus a lot of the frills. On the initial POST you get back a token that would allow you to poll the job status. Until you went to download the (often several GB) payload, all requests were extremely fast. At most we would insert a single record into a DB and queue a tiny object onto the message bus. In the status poll it was a simple key lookup.

In my experience nobody seriously tries to figure out how to use your API by calling it, they'll prefer to look at your code examples and documentation.

Nobody is going to build a generic-enough REST client to make all the HATEOAS crap matter for their workflow. In my view it's a waste of time, but maybe others have found it measurably useful?

<comment><AUTHOR> > Bottom line is if you're doing this via REST and not using web sockets

Websockets is not a viable solution for this problem because long-running processes are long-running, and it would make no sense to keep a connection up for minutes/hours just to poll the process status.

<comment><USER11> Sure, good point. I'd argue this isn't exactly black and white but you won't find me defending web sockets for many applications at all anyway.

</comment></comment></comment></comment><comment><USER12> Instead you could also POST to /star-construction to get back a link to a /star-construction/foo resource, representing a long-running creation process.

This resource could be polled to get progress updates, and eventually a link to the created star resource, or perhaps an error describing why it didn't work.

The advantage is that you don't have to deal with two different types of resources being returned at the same URL, which complicates parsing the response. If creations are queued you also can get a view on the workload still in front of you.

</comment><comment><USER13> Better options would be:

1\. Send partial HTTP response (206) with progress details which may be handled using AJAX progress API at the client before sending final response. This is processing and network intensive but may be needed in certain scenarios like drawing graphics etc. You cannot recover from connection loss.

2\. Handover the task to a job scheduler and return the jobid. Client can then poll jobid using a status API or get the status pushed using a webhook. This is the more optimal approach for majority of cases.

<comment><USER14> 206 is only valid to respond to range requests, which can only be GET requests.

<comment><AUTHOR> Additionally, the 206 suggestion fails to take into account basic use cases such as having more than one queued job.

</comment></comment></comment><comment><USER15> I found this discussion a bit incomplete as it didn't really discuss the consequences or costs or benefits of modelling things one way or another.

Suppose we didn't use REST or HTTP for our API. What would be a good way to model this kind of thing?

<comment><USER16> This is essentially just an asynchronous response to some request. I expect there are probably three cases:

1\. Response is unimportant: do nothing

2\. Response timing is not important: client polls

3\. Response timing is important: server calls back into the client somehow

The article describes an implementation of point 2. Depending on the technologies available at the time, I'd argue 3 is probably more versatile. Both cases 1 and 2, and the blocking case can be implemented with case 3, if the available technology allows it.

</comment></comment><comment><USER17> This article is missing one detail: what if resource creation fails? How do you signal that failure when the client calls GET on the task queue resource, in a way that's distinguishable from a failure to retrieve the task queue status?

<comment><USER18> That's like asking whether you should send a 404 when you serve an article about a city that no longer exists.

HTTP status codes refer to the resource being requested (in this case, the creation task itself), not to something that resource refers.

</comment></comment><comment><USER19> "Once the originally desired resource is created, there are two alternative ways to deal with the temporary task resource:

API client must issue DELETE request, so that server purges it. Until then, server responds with 303 See Other status. Once deleted, 404 Not Found will be returned for subsequent GET /queue/12345 requests. or, garbage collection can be a server's job to do: once task is complete server can safely remove it and respond with the 410 Gone on subsequent GET /queue/12345 requests."

How do you distinguish between an id referring to a finished task that no longer exists (410 Gone), and an id that never referred to any task (404 Not Found)? Other than the obvious solution of keeping every task in the DB with a "finished" flag, which doesn't scale...

<comment><USER20> > Other than the obvious solution of keeping every task in the DB with a > "finished" flag, which doesn't scale...

Why doesn't that scale?

<comment><USER19> Because your storage requirements would grow indefinitely as you would need to store a record for every task ever performed.

<comment><USER21> If your IDs are completely unique, you could add them to a Bloom filter [0] when cleaning up the resource. Then you could use that to determine if an incoming ID has possibly been used before or definitely not.

[0] [https://en.wikipedia.org/wiki/Bloom_filter](https://en.wikipedia.org/wiki/Bloom_filter)

<comment><USER22> Bloom filters are probabilistic, meaning they don't always produce the right result. From the article you posted:

> False positive matches are possible, but false negatives are not – in other > words, a query returns either "possibly in set" or "definitely not in set". > Elements can be added to the set, but not removed (though this can be > addressed with a "counting" filter); the more elements that are added to the > set, the larger the probability of false positives.

So I don't think they're the right choice for this problem, if you actually depend on the result "not here" vs "never was here" being right.</comment></comment></comment></comment></comment>


Reflections on Rust, and the Sand Castle Metaphor - <AUTHOR> https://brandur.org/fragments/rust-reflections <comment><USER2> Very much matches my experience with the last ~2.5 years of Rust.

One of the really eye-opening ones for me was building a UI library on win32 and watching it _just work_ on Linux, OSX, Android and WASM(with some Canvas work). Obviously each platform took some work to build out their respective rendering primitives but the core engine(including embedding Lua via gcc crate) just worked.

As someone who's done x-platform stuff most of their career it was nothing short of incredible.

<comment><USER3> I have the same experiences with Go regarding xplat. I was particularly amazed that I could trivially cross compile by simply setting the GOOS and GOARCH env vars to the correct operating system and CPU architecture and then running `go build`.

EDIT: Curious why people are downvoting this. Is it really so taboo to suggest that Go could be nearly as pleasant as Rust at something? Do I really need to roll out my Rust fanboy creds every time I participate in a thread on the language?

<comment><USER2> [https://crates.io/crates/gcc](https://crates.io/crates/gcc) is even more amazing, it will cross-compile your _C_ dependencies(assuming you have the right compilers setup, which is mostly handled already via rustup or base install).

For Android all I had to do was point it to the NDK(for both Rust and the crate) and I got all my C dependencies for basically free. Anyone who's had to work with the NDK knows how nice that is. That's also with zero changes to my build script/code/ifdefs/whatever.

<comment><USER3> > it will cross-compile your C dependencies

That's really cool indeed! Rust's build tooling is truly world class.

</comment></comment></comment></comment><comment><USER4> It's not the borrow checker that the author considers a problem. It's move semantics and futures. The borrow checker was a huge breakthrough in language design. Some of the other new stuff is marginal. Trying to hammer Rust into a functional language via the template system was probably a mistake. Too many "see what cool things I can write in one line" features.

I was a big fan of Rust at first, but I bailed out a while back.

<comment><USER5> From the article

> This is especially true when it comes to the more complicated ones like > moves and futures, but also true for simpler ones like borrows. What I > didn't know when I wrote about it in frustration a month ago is that it > doesn't take a little longer longer to be effective in Rust compared to > other languages, it takes 10 to 20 times longer.

I think futures might be exacerbating the authors problems. There was a recent post [0] that does a good job explaining why futures are so hard and how the in-progress async/await features fix it.

[0] [https://aturon.github.io/2018/04/24/async- borrowing/](https://aturon.github.io/2018/04/24/async-borrowing/)

<comment><USER4> Too many articles about Rust talk about why the current language sucks, but some feature in progress will fix it.

Rust started out with roughly the complexity level of C++, and it's become more complex from there.

<comment><USER6> I agree with both of your comments, I've noticed exactly the same thing. Every feature that fix something brings more complexity and have probability that will not interact nicely with some other features which means you need to make another feature that will fix that and then that feature...

Soon people will use different dialects of the language like in C++ and like in C++ only few will be able to say that they really know the language.

How will managing huge enterprise projects will look like in future? With average developers maintaining it and not language experts like in case of Mozilla Servo?

Time will tell I suppose. I am little bit pessimistic but I hope to be proven wrong.

</comment></comment></comment></comment><comment><USER7> This is indeed very light in content. Also, while Rust fixes a whole set of problems, it doesn't mean that Rust fixes all possible problems.

It is relatively easy to create a situation where you overflow your program. See here: [https://github.com/rust- lang/rust/issues/50049](https://github.com/rust-lang/rust/issues/50049)

Googling around, there are many situations where bad code can lead to an overflow and your code breaking.

<comment><USER8> Who, ever, anywhere, has claimed that "Rust fixes all possible problems"? This is a straw man. As for bad code leading to overflows and crashes: yes, bad code causes overflows and crashes, definitionally. Rust does not, and has never claimed to protect against logic errors in this way. How could it?

<comment><USER9> The language in the article is very strong.

> Rust is another step into the beyond. When finishing a feature and its test > suite I'll run my program to see it in action, but just as a formality – I > already know it works. I also know that it's going to keep working because > meticulousness of the compiler is so good at catching regressions.

<comment><USER10> This is a bold statement, specially because not many other languages have such high quality guarantees at compile time.

> I already know it works

This is the part I don't agree all of the time, but depending on the scope and size of the project, I've seeing this be true. And the compiler is really good at catching regressions, submitting PRs for Rust projects is not easy in the beginning, but it is very hard to insert a regression.

</comment></comment></comment></comment><comment><USER11> This is not at all specific to rust: OP is just grateful for the compiler to find lots of problems one would have to write tests for in dynamic languages.

<comment><USER12> It's not even specific to compilers; it's any type-checker. Compilers just happen to include one.

<comment><USER13> Well it's more of a feature of the language really. The type of static type checking provided by Rust would be impossible to achieve in C or Python for instance, unless you add specific annotations to augment the language's type system.

</comment></comment></comment><comment><USER14> I have a hard time believing that Rust's guarantees make it safer than easier statically typed languages like Java or Go. Rust is hard because it is solving a different problem, memory management without a garbage collector, not because it has the strongest correctness guarantees.

<comment><USER2> What Rust gives you over Java is determinism.

Having a GC doesn't preclude you from leaking resources(file handles, textures, etc) _and_ you're at the mercy of whenever the finalizer decides to run once the GC has determined it's time to free an object. Usually this manifests as your Java application humming along until you hit some GC threshold cliff and then perf/memory plummets.

I also can't enforce ownership in Java which drives me up the wall. Once you hand out a ref anything is fair game and calling code is free to add that ref to the root GC set so that it never gets freed.

<comment><USER15> The comparison I like to make is that Rust is the safer/modern version of C++'s memory model. It's in a different class from most languages, but trying to apply all the latest lessons and techniques.

</comment></comment></comment><comment><USER16> I do think the difficulty of Rust is brushed aside by its proponents. This week I wrote a little script to read an XML file, get a path from it, read a file at that location, do some regexes on it, and then update the XML and write it back to disk.

I'd estimate it took me about 10 times longer to implement in Rust than Python. And I don't exactly know Python well - it wasn't just extra time Googling how to do things.

A lot of it is just how restrictive the borrow checker is. Often you have to structure your code in a really _weird_ way to satisfy it.

Dealing with strings is another pain point. I get why there is `&str` and `String`. But that doesn't explain why I can't add two `String`s together using +.

Another thing I've found is that because of the type inference it often gets _really_ confusing whether a variable is a reference or not. Doesn't help that code completion basically doesn't work at all at the moment.

I wish there were a simpler language, like Rust (no garbage collector, no runtime), but that was a _little_ more helpful and willing to do implicit things even if they are slightly slower than the most optimised code possible.

<comment><USER17> You need to type your Python code extremely fast to finish it 10 times faster than this Rust program:

        extern crate xpath_reader;
        extern crate regex;

        use std::io::prelude::*;
        use std::fs::File;

        use xpath_reader::Reader;
        use regex::Regex;

        fn main() {
          let mut contents = String::new();
          File::open("test.xml").expect("Unable to open the file").read_to_string(&mut contents).expect("Unable to read the file");
          //println!("File contents: {}", contents);

          let reader = Reader::from_str(&contents, None).expect("Cannot parse XML file");
          let files: Vec<String> = reader.read("//file").expect("Cannot find file entries in XML file");
          //println!("Files: {:?}", files);

          for file in files.iter() {
            println!("File: {}", file);
            let mut contents = String::new();
            File::open(file).expect("Unable to open the file").read_to_string(&mut contents).expect("Unable to read the file");
            //println!("File contents: {}", contents);

            let re = Regex::new(r"foo bar").expect("Cannot compile regex");
            let replaced = re.replace_all(&contents, "baz").to_string();
            //println!("File contents: {}", replaced);

            File::create(file).expect("Unable to overwrite the file").write(replaced.as_bytes()).expect("Unable to write to file");
          }
        }

<comment><USER18> This is a really silly counterpoint. I am not even sure what point you are trying to make. How can pasting some program show how long it would take someone to reason out how to write it?

</comment></comment></comment><comment><USER19> I like how author writes! And previous post on web dev in rust is very good as well

[https://brandur.org/rust-web](https://brandur.org/rust-web)

</comment><comment><USER20> this: "Software is partly a production problem, but it's mainly a maintenance problem."

</comment><comment><USER21> Doesn't have any objective content whatsoever. Not particularly useful for anybody?

Maybe I misunderstand</comment>


C++ Quirks, Remembered - <AUTHOR> http://robey.lag.net/2012/10/09/cpp-quirks-remembered.html

<comment><USER2> Thank you for calling out the face that generic classes must live in headers. I've always hated this, but rarely if ever see it mentioned.</comment>


Did Zipcar Get Swindled on Its Massive IPO? - <AUTHOR> http://www.theatlanticwire.com/business/2011/04/did-zipcar-get-swindled-on-its-massive-ipo/36681/

<comment><USER2> It's important to remember they only sold about 10% of the company, not the whole company, with the IPO. So most of that 50% gain also accrued to equity still held by the holders prior to the IPO. So yeah, maybe they could have priced 30% higher and pulled in an extra 5M in cash, but it would have also meant more risk that they sully the 90% they still hold with a bad first-day performance in the market.

I doubt they're crying in their milk tonight at Zipcar headquarters.

<comment><USER3> Auction IPOs are supposed to fix this problem, although I suppose clueless buyer's remorse could lead to a first-day drop in such a case.

<comment><USER2> It's very hard to innovate in IPOs, probably partially because these first day gains don't tend to bother issuers. Bankers would rather do things the traditional way (and make institutional buyers happy with big first day gains) and issuers tend to be relatively conservative about risking their only IPO on a less-proven method

Look at it this way: if you're the CEO, you're not going to get fired because of a 50% first-day gain. But if you botch things because you wanted to do a non-traditional IPO, you may be.

</comment></comment></comment><comment><USER4> I've worked both on the buy and sell side of the new issue trade. On the buy side, as a proprietary trader, who had the option to buy the new issue (or not). On the sell side, pricing and structuring deals with the equity capital markets team, and placing them into the market. (This means selling them to institutional clients.)

It's not surprising to me that this stock exhibited really high volatility on day one. It's not easy to value this particular block of equity. It could easily have traded down 50% if the pricing was overly aggressive.

Here's why:

\- Not profitable.

\- No easy comparables to help you know what metrics the market is going to care about, or what multiple it will apply.

\- Sub-majority block of equity being sold means public shareholders won't have control as a group.

\- Large overhang in the form of the significant amount of shares that aren't being sold in the IPO, but will eventually need to get sold.

Typically an IPO would be shopped around the street, meaning that multiple banks would have the opportunity to handle the sale.

If it's a fully underwritten deal, one parameter that gets negotiated is the underwriting price: bank X will guarantee that you sell the stock for no less than $15; bank Y will guarantee that you sell it for no less than $18. (Note: they take a % off the top. Plus commission when they sell it to institutions.)

In a market this starved for IPOs, I'd guess this was an underwritten deal, which means that the other banks probably wanted to place it at an even lower price.

</comment><comment><USER5> Doesn't anyone have a problem with a company going public that _has never turned a profit_?

I live in San Francisco and have been using Zipcar for years. They're a wonderful company, with convenient car locations all over the city, excellent customer service, and a very forgiving insurance policy (I was in an accident a couple years ago that totaled their car. I just had to pay $500). So I REALLY want Zipcar to stay in business.

But this just really is beginning to remind me of the crazy shit that was going on 11-15 years ago. Companies back in the day that probably could have built interesting long term businesses imploded when they went IPO and built up massive amounts of capital they didn't need.

Please, someone tell me I'm missing something.

<comment><USER6> On Marketplace this morning someone stated they are profitable where they operate (DC, NYC, and SF?) but have been spending money to grow.

<comment><USER5> Ok, now that does indeed make sense if true. Thanks.

</comment></comment></comment><comment><USER7> Is there an easy way to stay abreast of upcoming ipos? I'm kicking myself for missing this one.

<comment><USER8> As I understand it, the initial sales were first to institutional investors. By the time you or I had a chance to purchase any, it was already at $30/share (it's at $28/share now).

</comment></comment><comment><USER9> Funny it doesn't seem that long ago (okay, ten years ago) Henry Blodget was on the other side of the fence.

</comment><comment><USER10> No, the CEO made a bad deal with banks and is now using PR to shift the blame from himself to his bankers.

If everyone intrinsically knew that ZipCar was worth $30 then why did he sign a contract with the banks that allowed them to sell it for $18? Banks can't just a your company public on their own, they need the consent of the CEO and/or the board, when you sign this contract it outlines how the bank will take your company public.

If he didn't like that option he could always buy a shell listed on the exchange and do a reverse-merger. There are plenty of ways to take a company public and very few of them involve trusting GS to sell your company at whatever price they determine. If as CEO he allowed his company to be swindled out of $50 million he should resign in disgrace. He's so ingrained in the sausage making process he himself has been banned from trading.

As an aside after how badly GS and JPM fucked over the entire world it should give one pause as to whether you'd sign a contract with any entity who had demonstrated such moral hazard. They're called banksters for a reason.

Also, if you take IPO stock there is a gentleman's agreement that you won't sell it for 90 days, so it remains to be seen how well these investors actually did.

</comment><comment><USER11> How do you differentiate between undervaluation and over-exuberant, illogical investors? Neither is a good situation.</comment>


What Shaun White Teaches Business - <AUTHOR> http://www.businessweek.com/innovate/content/mar2010/id20100319_530576.htm

<comment><USER2> The Shaun White part of this article is the kind of thing I imagine is said at a third rate business conference. That being said, I have my own observation about Shaun White and business.

Shaun White is the most popular snowboarder right now. I recently started snowboarding and purchased my own gear. What surprised me was that I was not overwhelmed by links to the Shaun White model board + boots + jacket + etc. This is a kid who is sponsored by the biggest (?) snowboard company (Burton), has 2 gold medals, and his own video game. I would expect to see entry level, mid-level, and expert level boards all in the Shaun White line. I really don't understand how they can not market his gear that way. (I am still pretty new to snowboarding, but so far I have not even seen a Shaun White model.) Maybe another HN person has some insight on this (like my ad filter was completely blacklisting Shaun White or something ).

<comment><USER3> I think snowboarding is the kind of culture that wouldn't respond to something that direct. It's an outsider's culture, so the branding you want in snowboarders' heads is "the best riders ride Burton" not "be like Shaun by purchasing the new Shaun White Limited deck".</comment></comment>


Ask HN: Best compsci resources for someone without a degree - <AUTHOR> I have been writing software professionally for four years, but entered through non-traditional means. I've picked up bits and pieces over time, but never had any sort of formal education and want to try to rectify that. Most things I see people talk about though are sources like the gatech omscs program. That requires some sort of bachelors degree and structured time (wife/house/baby) that I don't have.<p>Does anyone have recommendations for places I can learn these things as a non college graduate with no required learning schedule? <comment><USER2> teachyourselfcs.com/

Wonderful recommendations, I am going through it right now myself.</comment>


Show HN: Trail Router – generate running routes that prefer greenery and nature - <AUTHOR> https://trailrouter.com/ <comment><AUTHOR> Author here. Trail Router is a running route planner that prefers greenery and nature in the routes it generates, and biases against busy roads. It can generate round-trip routes that meet a specified distance, as well as point- to-point routes.

I've written a detailed blog post about how Trail Router works here: [https://trailrouter.com/blog/how-trail-router- works/](https://trailrouter.com/blog/how-trail-router-works/)

I developed this because I am (or was...) a frequent traveller for work, and I like to run wherever I visit. But I don't enjoy running on busy main roads, and I found that without careful and time consuming route planning that is often what would end up happening if I didn't know the area. Trail Router is my solution to this problem.

After a previous discussion ([https://news.ycombinator.com/item?id=23177351](https://news.ycombinator.com/item?id=23177351)) received quite a lot of attention, dang reached out to suggest I post a Show HN to reach a wider audience. I've addressed some of the original feedback too - for example, you can now prefer hills in your routes, as well as avoid them.

<comment><USER2> We sorely need this for cycling. Komoot is the closest thing but it doesn't do mileage requests like your app here.

This is awesome!

EDIT: In addition to optimizing for green routes, optimizing for low speed car zones would be a killer feature.

<comment><AUTHOR> Thanks! Cycling is on the to-do list. But I'm conscious it probably wouldn't just be a single category - there would need to be separation for road cycling and mountain biking.

The routing algorithm does already prefer roads with low speed limits, but there is definitely more that could be done in this space (e.g. a slider to set a preference to avoid higher speed roads). It also depends on the availability of speed limit data (I use the OpenStreetMap database).

<comment> 076ae80a-3c97-4 The state of apps/web apps for planning road cycling routes is abysmal. If you can implement this you'll immediately be the best.

<comment><USER3> I am curious why you think this. I run [https://ridewithgps.com](https://ridewithgps.com) and always love constructive feedback to improve. There's always improvements to make, but I wouldn't quite go as far as calling things abysmal!

</comment></comment></comment></comment></comment><comment><USER4> Nice job, the algorithm is really effective! I tried this in multiple cities where it ended up recommending very similar routes to some of my established running routes, and even some new combinations :)

Love that your algorithm tries to exclude out-and-backs, I think that goes a long way to make the routes feel organic and appeal to runners. This is really polished and I wish you the best!

One feature suggestion: I wonder if you could have a flag to generate routes starting in parking lots/park entrances in a radius around the starting location. This could go a long way to recommend some common long-run routes where you need to travel a brief distance via car or public transportation but then can get 10-20 miles entirely in the woods.

<comment><AUTHOR> Thanks! The avoidance of out-and-backs was a must-have for me. It can definitely be improved though. For example, right now the algorithm tries to avoid traversing the same path twice, but it will happily use a path that is only a few metres away! So adding some buffer around that (to penalise paths that are close to one already taken, if not the same) is a future bit of work.

Your suggestion is a nice one. Here in the UK, where it's common to have trains connecting most towns and larger areas, I have thought about adding the ability to say "I want to run 20 miles, and finish at a train station that can get me home in under an hour". Assuming the user is using a car makes it more easily achievable though. Consider it on the todo list!

<comment><USER5> wait, it avoids out-and-backs? could have fooled me, all the routes it showed for my house were out and backs, some had multiple points of out and back(like a letter Y or E)... i guess thats why i hate running near home, everything is an out and back and even the computer cant find an alternative. :)

<comment><AUTHOR> It tries to! But if there's no good way to avoid them, then it will end up taking them anyway.

More specifically, when 'avoid repetition' is enabled (it is by default), then the routing algorithm has a very significant penalty weight applied to any existing path already taken.

</comment></comment></comment></comment><comment><USER6> This project was discussed in the "quarantine side project" mega-thread: [https://news.ycombinator.com/item?id=23177351](https://news.ycombinator.com/item?id=23177351).

Since people were interested, we invited Sam to repost it.

</comment><comment><USER7> I live in a small town that is getting grown over by suburbia. The routes ignore all the nice open farm fields to the east of me, and runs me down the busiest streets through some of the most dangerous intersections in my entire state, just to skim by the edge of a park in the city.

In that same 6 mile distance, in addition to running past the farms, it could have run me up to the base of the mountains, down the nice bike trail that goes through groves of trees and open field, then back home.

This is one reason I love testing apps like this - my town almost always messes them up.

<comment><AUTHOR> Would you mind sharing the name of your town? Feel free to email me directly if you wish, contact details are in my profile. It could be that OpenStreetMap data (which I use extensively in the algorithm) isn't very detailed for your area, but I'm keen to investigate further.

<comment><USER7> I don't mind sharing the town - Spanish Fork, UT 84660

Especially check out the east side of town, it seems like it wants to go into town for anything short of a 15 mile loop, whereas going east is far more natural.

<comment><AUTHOR> It's not finding any green 'features' in the vicinity (I know, the mountain range off to the east would beg to differ...), so it's falling back to guessing a direction in which to head. I can see that the Uinta-Wasatch-Cache National Forest is tagged properly in OpenStreetMap though ([https://www.openstreetmap.org/relation/3502235](https://www.openstreetmap.org/relation/3502235)), so this should be something I'm able to improve on. Thanks again for your feedback!

</comment></comment></comment></comment><comment><USER8> Wow, I'm actually impressed by how good the results were, I was not honestly expecting it! Lots of kudos! From the clean map interface and Strava heatmaps to defaulting to metric units, this is something I would even pay for!

<comment><AUTHOR> Quick anecdote... It defaulted to metric units globally when I first launched it ~8 months ago. But that quickly resulted in complaints from North American users, so it now defaults to imperial units in North America and metric everywhere else. Users can change it in the settings menu, of course.

Thanks for the kind words!

</comment></comment><comment><USER9> This is really well done, congrats. I love the level of automation to make it so easy to use. Also the attention to UX, not just in the narrow UI sense but the actual experience of the trail.

Some thoughts: \- Create a community out of it so that people can rate the routes, and therefore provide feedback for the algo? \- Use some google streetview or alternative for landmarks along the way so people remember what they should be seeing and don't get lost?

<comment><AUTHOR> UI/UX is most definitely not my forte, so I appreciate the kind words!

Your suggestion RE landmarks tallies with an issue I've been grappling with: how does a user actually run with the route that they've just planned? Yes, you can export the route to a GPX file, and some users will use that on watches/apps to guide them, but it's a bit clunky and isn't the kind of thing you'd necessarily do for a short 5km run in a new area.

I'd rather avoid building full run tracking functionality, as it'd just be repeating what so many other apps do (and often very well). Perhaps the answer is to enable easy syncing of the route to your other apps. Strangely, Strava (the most popular) don't allow importing of routes via their API.

<comment><USER10> Hi Sam, instead of syncing you could hook up the GraphHopper Android navigation demo. Although "app development" is not our focus I really like it to test the routing in real life. And for 2.0 it will be possible to directly have this in the server without a further dependency (see the recent pull request). I've mostly found minor OSM problems in my neighbourhood but also some bugs in GH.

<comment><USER10> Ah, I see you also have a mobile app. Nice! Is this mainly a mobile version or a native app?

<comment><AUTHOR> The mobile apps are really just full screen web views, with a few enhancements on top. I will definitely check out the GraphHopper Android project and the recent pull request. An improved native mobile app (possibly even with some offline routing, maybe without some features) would be my ideal endgame.

<comment><USER10> The Android demo is basically a fork from Mapbox and last year or so they even introduced offline routing (probably via valhalla), but only as a binary blob with a closed source license attached (we forked before so this does not affect us).

btw: are you using CH, LM or plain A* for your setup?

<comment><AUTHOR> I just had a proper look at GraphHopper Navigation - I can't believe I hadn't seen it before - that looks pretty much perfect for my use case! I will certainly be giving that a try.

I'm using plain A* currently. I understood from the Openrouteservice documentation & code that any time dynamic weights were being used, everything fell back to A* anyway. And I'm using dynamic weights heavily! My understanding could be flawed though, and I appreciate I am going to be a few GraphHopper releases behind now. The speed is okay for now - there's certainly bigger items on my todo list first.

I'm excited to try out GH 1.0 properly. Now that I've got much more familiarity with what I'm doing, I'm keen to switch to using GH directly and somehow try to avoid patching it (or try to get any patches merged back into the open source version).

</comment></comment></comment></comment></comment></comment></comment><comment><USER11> Love the idea. Great blog posts as well. This one alone [https://trailrouter.com/blog/how-trail-router- works/](https://trailrouter.com/blog/how-trail-router-works/) would have taken you several hours to write!

<comment><USER12> Hours? I would stress for a day on that! Bravo.

</comment></comment><comment><USER13> Ha, it makes the same route planning mistake I did years ago when I was in Rome.

I was staying near the Colosseum and for one of my runs, thought I'd run on the Appia Antica, because oldest road in the world, into Parco della Caffarella because it looked like a really nice park with lots of trails.

Well, next thing I know I'm running down a one-way cobblestone road with cars coming toward me and there's no entrance into the park because there's a huge wall between the road and the park. The road had walls on both sides in fact. I didn't want to turn around because I'd have the cars to my back. So I hugged that wall as close as I could and trusted the shit out of Roman drivers. Eventually I found a way into the park.

Here's the route Trail Router gave me:

• [https://ibb.co/k8m18xq](https://ibb.co/k8m18xq)

Here's the run I did that day:

• [https://ibb.co/pK6xqMX](https://ibb.co/pK6xqMX)

The road starts out looking great:

• [https://ibb.co/998GQk3](https://ibb.co/998GQk3) (my picture)

• [https://ibb.co/DYXkBt5](https://ibb.co/DYXkBt5) (google maps)

But very soon you're on this:

• [https://ibb.co/vJcCSXr](https://ibb.co/vJcCSXr) (google maps)

Well, it was all worth it because eventually I got into the park and I'm still alive:

• [https://ibb.co/52VgT1W](https://ibb.co/52VgT1W)

• [https://ibb.co/dbvG6f5](https://ibb.co/dbvG6f5)

• [https://ibb.co/ZTPknP3](https://ibb.co/ZTPknP3)

• [https://ibb.co/VWxrwnB](https://ibb.co/VWxrwnB)

I know you're at the mercy of map data. I don't know how unique this situation is. I always appreciate a new route planning tool. I usually use a combination of Footpath Route Planner, the heat maps from Strava, and Google Maps street view and satellite view. It's definitely a pain to do that for every trip. Maps are definitely not optimized for pedestrian safety, and things like sidewalks and shoulder-width are often inaccurate or just not there on the maps short of bringing up street view.

<comment><AUTHOR> Thanks for the feedback, appreciated. The reason it suggested that route was (1) the park, as you found, and (2) the Via Francigena hiking route ([https://www.viefrancigene.org/en/](https://www.viefrancigene.org/en/) & [https://www.openstreetmap.org/relation/6201614](https://www.openstreetmap.org/relation/6201614)) follows this road.

One item for consideration in the future is allowing users to flag unsafe portions of a route and then having that visible to other users (and also influence the algorithm).

<comment><USER13> Allowing users to flag unsafe portions sounds like a good work-around.

FWIW, I'm looking at the routes it's planning from my home and they aren't safe. It's sending me down heavily trafficked country roads with little to no shoulder. I'll email you some screenshots.

Thanks for the new app regardless!

<comment><AUTHOR> For anyone else experiencing something similar, try the 'Avoid potentially unsafe roads' option in the settings menu. This tells the routing algorithm to always avoid 'secondary' and 'tertiary' roads without explicit sidewalks (these are OpenStreetMap road classifications).

This isn't on by default because in some countries/regions, such roads are very reasonable to run on.

Clearly there's a better middle ground though, which might involve highlighting potentially problematic parts of the route in the UI.

Thanks again js2 for your email and all your feedback!

</comment></comment></comment></comment><comment><USER14> This is fantastic. Tried it where I live and the 5km and 10km routes were very similar to where I actually run for those distances, the only major differences being local paths that I know of which are poorly represented in available data.

One improvement I might suggest, it's not clear if this is being done already, is to not just bias towards greenery, but also boa against main roads. The routes generated for me spent a long time on roads that are quite major, where there are nicer alternatives available.

</comment><comment><USER15> It seems to work well for my location. I have been living in my current home for six months and I've made an effort to plan out greener routes around me for various distances. Trail Router comes up with similar routes when I put those distances in. It's not perfect – it seems to prefer joining up parks that are long distances away from one another as long as there's a river along the route instead of joining up parks that are closer together. But it's a lot better than I expected.

</comment><comment><USER16> Love the idea especially for when on holidays or in another town! I live I a very green environment and the maps generated are quite often 'wrong', that is not even near the routes I would suggest. Preferring urban areas to non-urban areas and seemingly not knowing of major walking paths in the forests nearby. The Strava map seems to know those paths, but OSM probably doesn't. Anybody know where I can dive deeper in OSM to find out what's going on?

An example: my neighborhood is surrounded by a free laying cycling and walking path. Perhaps diametrically only 30m-100m from houses, but 100% green and rustic. Then the mapper plots a route on sidewalks in urban areas. It did recognize the no. 1 park around though. But I'd hope it would get the woods as well.

<comment><AUTHOR> Thanks! My suggestion would to start out by checking out your area on openstreetmap.org for two things: Firstly to check to see if the trail you refer to is shown (it should show up as a yellow dashed line if it's a trail). Secondly to see if the woods are shown (this should show in a light green shaded area). Trail Router uses the presence of woods as a hint to prefer paths. Both of these items are surprisingly easy to edit on OSM.

</comment></comment><comment><USER17> This is great. I move around a lot, and one of the big headaches I always have is figuring out a decent route to run. The only thing I would request is having an option to have completely circular routes, rather then having routes that require running back the way you came.

</comment><comment><USER18> Looks very nice! Just started out running and especially the feature that it outputs multiple routes of similar lengths (which I can slowly increase) to have some variation is very appealing! From what I can see the routes all look very runnable with a lot of greenery.

</comment><comment><USER19> I am looking at the route it is generating from my place in New Orleans and comparing it against my own mental route generator for cycling and it's not doing so well. It's got me running _parallel_ to Jeff Davis Parkway instead of taking the trail along the very green parkway, then up the side of Bayou St. John with the _least_ greenery (one side has a lot, the other has next to none). Maybe in part because the unofficial desire trails up the green side of the bayou that I regularly see joggers on don't seem to exist on the map. Going though City Park makes sense; coming back I guess Cortez isn't too bad a choice if "avoid repetition" is on, but avoiding Jeff Davis just seems super weird.

</comment><comment><USER20> Pretty neat!

I did see a problem with stairs — not sure if algorithm does not take into consideration or if data is not properly marked up in OpenStreetMap, but locally, it recommended a route goong over two long sets of stairs: one of them being practically a goat trail I would recommend against running on, but careful trodding is ok. The other stairs are long and wide, so probably not perfect for running (hard to figure out your stride), but ok. I wonder if elevation rate could be taken into account, which would probably exclude the goat trail one.

<comment><AUTHOR> Steps are supported in OSM, but I don't know if your ones in particular are tagged. Either way, Trail Router doesn't consider them by default. If there is a lot of elevation gain/loss when using the stairs, you can adjust the Hill Preference slider (in the settings screen) to indicate whether you prefer hills or would rather avoid them. You can read more about that here: [https://trailrouter.com/blog/greenery-and-hills- sliders/](https://trailrouter.com/blog/greenery-and-hills-sliders/)

</comment></comment><comment><USER21> This is awesome! I've tried it on a few areas that I know well, and it really did a good job.

I really love running in new places as a way to explore my area as well as when I travel. I find there is something profoundly good about connecting with my surroundings that way. I use Strava Route Planner with the heatmap extensively, and now I think I'll be consulting this a lot.

My only suggestion is to show an elevation profile. There is already a hill preference slider in the settings, but the elevation profile would really make it great to look at.

<comment><AUTHOR> You're in luck, both of those things are already supported!

For viewing the elevation profile, click on the little up arrow on the miles/km distance box at the bottom of the screen.

You can configure a hills preference (either to prefer them or to avoid them) in the settings screen. Click on the cog icon to get there. More details are available at [https://trailrouter.com/blog/greenery-and-hills- sliders/](https://trailrouter.com/blog/greenery-and-hills-sliders/)

</comment></comment><comment><USER22> this is AMAZING! very fast and planned some interesting routes in Portland that I hadn't thought of. I think i'll start using this over my premium strava route planner

</comment><comment><USER23> I just loaded this and the map was centered on my house before I gave access to my location. I'm using current Firefox. Anyone know how that would happen?

<comment><AUTHOR> It uses geoIP data from Maxmind to approximately locate users even before they provide location information. It's not usually _that_ accurate though. It's just intended to get a user to the rough location that they're based in.

</comment></comment><comment><USER24> As a runner who only runs on trails (e.g. not on pavement) I make a point of asking other runners about places to run before I visit a new town.

I tend to get great advice and the benefit is that I often meet people to follow on Strava, where I build up a library of potential runs.

I'm not looking forward to a future where every conceivable off-road trail is available, instantly, in an app. I doubt anyone who appreciates nature really is.

</comment><comment><USER25> This is pretty cool, but there are definitely routes that pop up quickly that cross closed gates and trails that are not in good repair. I'd consider these to be good "Beta" routes, if you enjoy exploring and don't mind the occasional dead-end, but not super reliable if your time is limited.

Not a complaint, if I ran these are the sort of routes I'd love to explore.

<comment><AUTHOR> That's very fair, thanks. In cities I wouldn't consider this to be much of an issue as an occasional dead end (e.g. road works) is easily manually avoided. But in more rural locations it can be frustrating. If you're seeing it in your area, then you could consider submitting a correction to OpenStreetMap. It's surprisingly easy to do. I update Trail Router with the latest OSM data roughly once per month.

</comment></comment><comment><USER26> This is cool! Strava has a new feature where they'll plan a route for you: give a mileage range and a starting spot. When you tell Strava to prioritize trails vs paved, you get good results. Interesting very similar to yours, so that's really impressive given they're a large company with a big budget, or perhaps it says more about my location.

</comment><comment><USER27> This is really well done. If you use this, please consider donating. (I'm not affiliated in any way, I just saw on the about page that it's self funded and costs $200/mo). Here's the patreon page: [https://www.patreon.com/trailrouter](https://www.patreon.com/trailrouter)

</comment><comment><USER28> This is an amazing tool, I just downloaded the app and am excited to use it routinely. Thank you!

</comment><comment><USER29> AllTrails has a free trail creation tool. Check it out here, you just need to create a free account to save.

[https://www.alltrails.com/explore/map/new](https://www.alltrails.com/explore/map/new)

</comment><comment><USER30> Thanks for creating this, just tried it in my local area and it generated a route that I already enjoy !! I shall no doubt get a lot of use out of this, whilst it's not the intended use - I will cycle a few of the routes over the next few weeks !

</comment><comment><USER31> This is really fantastic. Thanks for making this!

One suggestion: Have changes to the route push state to the history. That way you can use the back button on the browser if you make a mistake. Right now it'll totally take you off of the page.

<comment><AUTHOR> Thanks very much! Personally I find it a little annoying if a site pushes state to the history very frequently. If I genuinely want to get off the site I end up clicking back loads, and then go too far!

I tried to solve this in Trail Router by putting an undo button on the screen (bottom right).

<comment><USER31> I would just follow how Google Maps works. I think that's how my brain was expecting Trail Router to work, since it visually and functionally is very similar to using the "Directions" feature of GMaps.

</comment></comment></comment><comment><USER32> It's quite amazing these days there's enough free data and software available you can create such a tool on your own! How do you plan to make it pay the server costs though?

<comment><AUTHOR> Fully agreed - the amount of data openly available is astounding. I think many people would be surprised at the richness of the tagging information within OpenStreetMap. Without OpenStreetMap, this project and countless others would not be possible.

Mapping costs would be by far the biggest one. However, I'm very lucky that after the last big spike in traffic, Mapbox very generously admitted me to their community programme. You can read more about that here: [https://trailrouter.com/blog/mapbox-community- programme/](https://trailrouter.com/blog/mapbox-community-programme/)

The server costs are quite modest at the moment (under $200/mo), which I'm lucky enough to be able to self-fund through my other work. If usage were to massively increase then maybe I'd need to find a way recoup costs. But even all of the hits from HN have barely put a dent in the server utilisation, so there's plenty of headroom for the foreseeable future.

</comment></comment><comment><USER33> I love to see that you rely a lot on open source. Using OSM instead of gmaps is a big plus in my list. Do you also plan to open source it?

</comment><comment> WD-42 Nice work! I tried it out for my local area and it ended up suggesting some routes that are very similar to my existing runs/rides.

</comment><comment><USER34> Looks interesting. It missed the multi-use trail that wraps around the area where I live - though its not on google maps either.

<comment><AUTHOR> It may be that the trail near you isn't on OpenStreetMap ([https://openstreetmap.org](https://openstreetmap.org)). If you're interested in fixing that (and thus the myriad of tools that rely on OpenStreetMap data), head to [https://wiki.openstreetmap.org/wiki/Getting_Involved](https://wiki.openstreetmap.org/wiki/Getting_Involved)

</comment></comment><comment><USER35> I tried to generate a 10km round trip route. It generated exactly the same route that I run normally :D

</comment><comment><USER36> Looks interesting! Have you considered providing a way to sync routes directly to Running watches?

<comment><AUTHOR> I actually have. Watch this space - there should be direct route syncing to Garmin and Suunto before terribly long. In the meantime, you can export routes to GPX/KML files already.

</comment></comment><comment><USER37> Wow. As soon as I gave it my location it plotted my favourite local return hike, very cool!

</comment><comment><USER38> Is there a way to show the elevation associated with a route?

<comment><AUTHOR> Yes - click on the little up arrow in the mile/km distance box at the bottom of the screen. It will show you elevation, as well as the type of paths taken.

</comment></comment><comment><USER39> Nice job! Two clicks on my phone and first route was spot on.

</comment><comment><USER40> very cool! i live close to a forest and it found a trail i had to manually tune on komoot.

feature request: an option to not cross country borders would be nice.

</comment><comment><USER41> Id want this but for bicycles.

</comment><comment><USER42> This gonna be a hit!</comment>


WireGuard now runs inside of the OpenBSD kernel - <AUTHOR> https://twitter.com/EdgeSecurity/status/1259667061942431744 <comment><USER2> Why are WireGuard implementations done in kernel rather than userspace?</comment>


2M Free Eclipse Glasses Coming to US Libraries - <AUTHOR> https://www.space.com/37310-public-libraries-give-away-eclipse-glasses.html#undefined.uxfs <comment><USER2> (June 26)

I can't find any mention of this in the three local library systems I checked.</comment>


Offer HN: My Book "Understanding SEO" – Free (Kindle) on Amazon for 24h - <AUTHOR> https://www.amazon.com/Understanding-SEO-Systematic-Approach-Optimization-ebook/dp/B07L3BSQHG/ <comment><AUTHOR> Hard, honest reviews would be awesome.</comment>


Ask HN: How do you monitor web apps and apis? - <AUTHOR> How do you monitor performance and errors of web apps and APIs? We get emails whenever there is an exception, but this is becoming less and less usable as we grow.
We process at least 500K requests a day. We have 20+ web applications on each environment and 100+ servers in several region. We use .NET.
We are trying out Stackify[stackify.com], but wondering if there are any better tools.
Thank you. <comment><USER2> I try to log most errors and exceptions, and send them to a log-management tool. I use Loggly (they make it easy to run reports or get automated notifications based on error-type) but have heard good things about Splunk as well.

To monitor the actual servers, I tend to use CloudWatch (since we are on AWS), but New Relic and DataDog are popular options.

I also use Pingdom to hit all our servers routinely (once a minute or so), and let us know if something goes down or is taking too long to respond. This is a bit of a redundant check considering the other metrics we have, but I like having some redundancy there.

Hope that helps.

</comment><comment><USER3> we are using [cat]([https://github.com/dianping/cat](https://github.com/dianping/cat)).

</comment><comment><USER4> use a log monitoring tool and define alerts to (send email, post in hipchat rooms) when exceptions are caught

Logentries is a good one.</comment>


Ask HN: Drupal or Django for my startup? and why - <AUTHOR> Hey there,
I wanted to know which choice is wiser - try to save up money with a CMS like Drupal or go with a flexible framework like Django or asp.NET
The project is a startup for cars that includes: - marketplace cars and all kinds of other vehicles - marketplace for autoparts - marketplace for tires - marketplace for all kinds of services around car - washing, repairing, etc - booking
- vin checking versus 3rd party databases - chat/messaging systems - ratings - statistics - owner tools - my garage - where you put data about your own car
- promo codes/vouchers etc. - ...
...in other words, it is huge. It is not the classical startup of 2 people in a garage, but instead a business with funding that aims to topple the leader of car selling in Bulgaria (small country).
It would be tremendously helpful to state what your experience is (how good you are at either), have you had specific experiences you didn't like with either and what other specific things might be a problem for one or the other.
Any thoughts or guidelines will be super helpful, so thanks in advance.
pp <comment><USER2> It sounds like you're not the technical founder because those two frameworks are completely different in nature and if you were you'd have a clear answer. I don't have professional experience with either. I do have experience with failed startup ideas because I started with a focus on technology and not problem. Btw, I am also Bulgarian so that's probably what we have in common. There are tens of technology stacks that can create a technical solution. However, there are very few founders that can create a strategy and build a business to topple an existing player. Just remember that. The technology should be the easy part to figure out.

Just my 2c.

</comment><comment><USER3> Wrote a 100k line of code financial app in Drupal 7. It's not compatible with D8, and expect the same with D9. Testing is less than an afterthought in the community, docs in D8 are usually empty pages when you dig in, the technology is loosing mindshare and not many are hiring. These are all red flags for a startup. It can be used to build apps but I would stay away.

Go with Django. Python is a more productive and clear language. Django is excellent at ensuring an upgrade path, provides great docs.

If you have well defined services, it sounds like you do, then make them apis. You could have separate Djano apps or use Flask, a great micro framework.

Overall my experience with Python and the community for building web apps has been great.

<comment><USER4> > Wrote a 100k line of code financial app in Drupal 7. It's not compatible > with D8, and expect the same with D9.

Actually D9 will not be a hard compatibility break for D8 thanks to the significant changes in Drupal 8: [https://www.drupal.org/blog/making-drupal- upgrades-easy-fore...](https://www.drupal.org/blog/making-drupal-upgrades- easy-forever)

I agree D8 has had a documentation problem but this change has a chance of helping with that too since developer attention won't be in trying to make a huge amount of sweeping changes all at once anymore.

> Testing is less than an afterthought in the community

Virtually every core patch has to provide tests in order to get accepted. There's better testing coverage of Drupal core than there's been ever before. Significant amount of work has gone into improve the way testing is done as well. There's definitely more to do since tests probably used to be less than an afterthought and the codebase has a long history but that statement isn't accurate at all.

What do you base "not many are hiring" on? Has seemed like the opposite is true to me.

</comment></comment><comment><USER5> The older I get the more I realize that language/framework decisions at many companies is a function of 1) what your existing team knows already and 2) the business requirements of your company. Furthermore I believe the goal of a founder is to solve problems with people (like fundraising, hiring, etc.) and not computers (by coding).

<comment><AUTHOR> Good point, though not on point ;)

</comment></comment><comment><USER6> Background: I work at a Fintech startup where the backend is built with Django. I have about 4 years of experience with it, and have worked on Django projects from several different industries, from education to ecommerce. I absolutely recommend that you go with Django. Your project sounds like will deal with many different types of problems, and using Django and Python will allow you to take advantage of the huge ecosystem of third-party libraries that will save you a ton of time. CMS systems like Drupal are great if your main activity is publishing content, but anything past that will leave you fighting with its internals while you should be writing business logic.

<comment><AUTHOR> Interesting. I am not arguing, just sharing: people say that Drupal now is a framework and you an actually write custom stuff, not just stack modules. Can that be, what do you think?

<comment><USER7> Yes - that is what is being said about Drupal 8 but I see it more of a shoehorning a CMS into a framework. Custom modules can be written - but doing the same in Django is much easier.

One approach that you can also look at is build the platform as an API in Django and build the wrappers / customizations for the sites in Drupal.

<comment><USER8> Actually it's always been a framework, it's just there's lots of shoehorning of CMS functionality. And it's less of shoehorning, it's just where the $s are flowing, I work with a team who do a lot of native CRM & BPM with Drupal it's just they haven't got $m so you haven't heard of them.

Drupal's fundamentally different than Django as not only are you able to code less due to the sharing of modules but you're also collaborating with thousands of others.

So with Django where you're coding coding coding, i.e. creating technical debt, the idea with Drupal is you only need to code what business logic hasn't already been built over the last 16 years. You're leveraging code and community, so you can focus on what you're good at and collaborate with thousands of others to fill the gaps.

</comment></comment></comment></comment><comment><USER9> I currently work for a large eCommerce site that uses Django/Python. Before that I worked with a large PHP CMS, though not Drupal. Some non-obvious, and practical things that Django does well:

* Dependency management is way easier with Python, so upgrading and deploying is less error prone

* Django scales really well thanks to excellent caching with e.g. Redis. Adding caching to views (pages) is child's play, and super easy to debug/purge.

* Speaking about scaling, writing a REST API or consuming a REST API is much easier in Django (django-rest-framework and requests, respectively)

* The Django ecosystem is great. If you have a problem you want to solve, someone has probably done it already and published a package

* The Drupal middleware is awful compared to Django middleware

* Testing

* The Django admin interface is amazing [0], and at least as good as any off-the-shelf CMS. It does permissions and custom actions. Your sales team needs to upload a CSV file of suppliers? No problem, just add it right in there.

* Debugging. The Django shell and pdb/ipdb are great tools.

Finally, thanks to some non-obvious pitfalls in PHP, you have to take more care when writing PHP to make it secure. Apart from the flame-wars, the only practical issue is when hiring. You need to make sure you hire PHP candidates who know the language inside out. With Django/Python, you can be a bit more lax - there's far fewer non-obvious pitfalls, especially with the templating system and ORM of Django. It also takes devs less time to do a thorough code review. IMO, it's easier to grow a Django team.

[0] [https://docs.djangoproject.com/en/dev/ref/contrib/admin/](https://docs.djangoproject.com/en/dev/ref/contrib/admin/)

</comment><comment><USER7> So I've experience of working with both and technically you are comparing apples to oranges. Drupal is a CMS built in PHP while Django is a Python framework to make web-apps (one of them being a CMS).

* A framework works at a lower level compared to a CMS. So if you have lot of customizations to be done which is not limited to how it looks - a framework is better * If most of the work needed is to change how a site looks and content only, a CMS might be a good fit * From your point it looks like there is a need for a strong platform * Which language are you programmers comfortable with (if you already have a tech team)?

For the company I was working in earlier, Drupal 7 was being used.

For my own startup I am using Django.

</comment><comment><USER10> You should compare something like laravel([https://laravel.com/](https://laravel.com/)) or symfony([https://symfony.com/](https://symfony.com/)) against the djangoproject([https://www.djangoproject.com/](https://www.djangoproject.com/)). I think this is a more appropriate choice. If you are not sure about the dev- performance of your team with a naked framework, you should consider something like the Oro- Platform([https://github.com/orocrm/platform](https://github.com/orocrm/platform)) or Odoo([https://github.com/odoo/odoo](https://github.com/odoo/odoo)). You can also work with a e-commerce/shop system like oscar([https://github.com/django- oscar/django-oscar](https://github.com/django-oscar/django-oscar)) or shuup([https://github.com/shuup/shuup](https://github.com/shuup/shuup)) as a base.

</comment><comment><USER8> I've been Drupaling for 13 years, before that I was J2EEing, before that I was AS/400 RPGing. Before that I was PASCALing at college, BASICing and Assemblering at home. I've seen all kinds of businesses and startups, and I've seen what works and what doesn't.

My advice to you is to use Drupal because you can get far fast. Use lean methodology (lean canvas etc.) and get your MVP out as fast as possible and start making money. Don't try and build lots of functionality that you don't know works, and by works I mean makes a profit.

Watch GaryV, as he says - the market is always right. Focus on selling cars or vin checking or whatever it is you're doing, nobody apart from techies give a hoot what your site is built on. Some of the biggest sites in the world are built on PHP, e.g. Facebook. Some of the biggest sites are built on Drupal, e.g. Weather.com (the most personalised site as every visitor sees a different output, and delivers data to mobile apps).

Your code won't help your business succeed, selling will.

<comment><USER8> Furthermore, use the Drupal Commerce modules for Drupal 8, releases are coming out thick and fast now. Commerce for 8 is awesome, you can have multiple stores (so like people have on eBay), product options are much easier than Drupal 7 to set up, and it's truly flexible so if when you A/B test or whatever you can easily alter the way your site works.

</comment></comment><comment><USER11> We migrated from Drupal to Django a few years ago and haven't been happier. Sites have become easier to maintain, new features are developed faster and designers find Django templates easier to work with.

We found Drupal's hooking system a pain to work with and it made debugging incredibly difficult.

We were using Drupal 6 at the time, not sure what has improved since then.

<comment><USER4> Hooks are going away in favor of a fully object-oriented architecture. Some hooks remain in Drupal 8, but they'll get replaced in time. It also uses a templating system called Twig that is similar to Django's.

</comment></comment><comment><USER12> Django. Stay away from php

<comment><AUTHOR> Why do you say that?

<comment><USER13> [https://eev.ee/blog/2012/04/09/php-a-fractal-of-bad- design/](https://eev.ee/blog/2012/04/09/php-a-fractal-of-bad-design/)

<comment><USER14> It's a few years old now, isn't PHP supposed to be better there days? (I am saying that as a Django dev).

</comment></comment></comment></comment><comment><USER15> I wrote this back in 2009 so it's a bit long in the tooth at this point, but still mostly accurate (and I'm still going strong with Django in 2017):

Drupal or Django? A Guide for Decision Makers [http://blog.birdhouse.org/2009/11/11/drupal-or- django/](http://blog.birdhouse.org/2009/11/11/drupal-or-django/)

</comment><comment><USER16> moine,

imho ... i worked with both of them :)

simplified

* drupal is a really nice, secure and extensible portal-system which can be enhanced with plugins written in php

* python-django is a web-framework based on common design-patterns

whats better for you!? it depends what you want to achive, but i think for a startup - and a maximum of flexibility + scalability - i would prefere python- django.

why? as a "general" webframework its more flexible than the portal/plugin approach of drupal + i would prefer python over php.

</comment><comment><USER17> Even though my experience is from a few years back, I absolutely cannot recommend Drupal for complex, custom applications. See my write-up here: [https://github.com/rshekhtm/VentureTap/blob/master/README.md](https://github.com/rshekhtm/VentureTap/blob/master/README.md), especially the Lessons Learned section.

<comment><USER4> So much has changed from Drupal 5 that I don't think your experience is all that relevant when assessing Drupal anymore. Just as one example, Drupal 8 now does use object-oriented PHP extensively.

</comment></comment><comment><USER18> I started with Drupal when it was 4.7, shortly to move to 5.x. We had a group blog, and it did the job pretty well, and it felt good to take advantage of things like maps modules and tagging. But niggling pains were there. Having to add extra modules to deal with duplicate pages ending in /, and while views was pretty simple point-and-click at the start, it seemed like it could do a lot more.

As time progressed, we added voting, rss aggregation, quizzes, and started extending views with URL arguments, relationships, fancy combinations of filters, login was via OAuth, and we were at 7.x by now. It was feeling unwieldy. Always aggressive in theming, what we wanted to do was taking an increasing amount of time, both in function and appearance.

A new project came along, and I thought, eh, Drupal can do that. And it surely could, seeing a nail, Drupal is a hammer that can do anything.

But it does it in a way where maintenance, or getting things exactly the way you want become incredibly time-consuming, and the code is a maze unless you're in it 24/7, and we had chosen Drupal not to be in code 24/7.

My background is not as a developer, but finance where using technology is an important skill. I can Bash, Ruby, JS, VBA/.Net, Python, R, and a few others to varying levels.

I shuddered a little when seriously considering Drupal as a solution to this new problem, mapping out the edge cases and exactly how many modules were needed (60+).

I ended up taking a week off learning NodeJS and using hapi as a framework. Just because I wanted a fresh start, not that I was enthralled with Node, but as so many others were using it, surely something was there.

This freshness gave some time to think about what I actually wanted, and to back-track from there, focusing on simplicity and well, simplicity. How to make everything as simple as possible, which is what a high level code framework offers.

You don't sound that technical, and I don't have experience with Django, however I recommend taking a code-approach, even if you don't want or need to be sitting in code, as you'll have access to and know how everything really works, not a pre-written GUI one where you're clutching at straws. Most web challenges are about the idea, the code implementation for what you sound like your doing, at an initial stage, is probably that not complex, and if you have success with the core idea, get technical talent onboard to optimise or do the things you find too hard.

A word of warning, however: Focus on making things simple. That will stop you going into a rabbit warren filled with code spaghetti.

tl;dr

Drupal is an excellent tool for publishing, and accommodates some complex publishing workflows that need multiple authors or layers of editing or approval. Core modules like comments or maps compliment this, but that's it. If your core need is greater than publishing (I suppose marketplace is publishing, and there are eCommerce modules), Drupal is a quick solution, but if your long-term roadmap means doing something remotely unique technically or in UX, roll your own using a framework to prevent re-inventing the wheel (and ending up with security nightmares). Bring on technical talent at an early stage when your idea seems proved. If you have funding, do this right away.

</comment><comment><USER19> Rails!

</comment><comment><USER20> Drupal 6 and 7 use Drupal crap function-based programming (not functional programming) style which over time leads to maintenance mess as you have tangled interdependencies of modules which are not 100% in terms of test coverage or quality. Drupal 8 bootstraps on top of that mess OOP. Drupal is engineered as garbage and gets unwieldy pretty quickly if you are going to be doing custom programming and straying away from the main modules. However for your case it's foreseeable you could stick to views and content types, and templates to customize those and not have to stray too much into actually writing custom code, however the UX of that sort of thing in Drupal is still shit and trying to customize that makes you do it the Drupal way which is often ugly and complicated. PHP 7 is a modern language I recommend a modern PHP framework or a modern Python framework like Django. In the end use what your technical founder knows best. Drupal makes it seem as if it's going to cut down on work, in the end it leads to more work to maintain, scale, and keep adding features and creating a better UX experience which users have come to expect. These days I would probably use some language for the backend and make the frontend with a responsive javascript framework like Angular, React, etc.</comment>


A New Form of Matter, Excitonium - <AUTHOR> https://physics.illinois.edu/news/article/24114 <comment><USER2> Phys.org is a dumpster fire, and the actual study is interesting.

[http://science.sciencemag.org/content/358/6368/1314](http://science.sciencemag.org/content/358/6368/1314)

 _Excitons—bound states of electrons and holes in solids—are expected to form a Bose condensate at sufficiently low temperatures. Excitonic condensation has been studied in systems such as quantum Hall bilayers where physical separation between electrons and holes enables a longer lifetime for their bound states. Kogar et al. observed excitons condensing in the three- dimensional semimetal 1T-TiSe2. In such systems, distinguishing exciton condensation from other types of order is tricky. To do so, the authors used momentum-resolved electron energy-loss spectroscopy, a technique developed to probe electronic collective excitations. The energy needed to excite an electronic mode became negligible at a finite momentum, signifying the formation of a condensate._

<comment><USER3> So is it an insulator, a perfect conductor or a superfluid?...

</comment></comment><comment><USER4> For a moment I thought it was yet another new ICO. I'm quite happy it's new science instead.

<comment><USER5> Any time a bitcoin transaction takes place, a hole is left in its place. The hole attracts other coins and when paired with an Ether, a new coin is formed: ExciteCoin

<comment><USER6> It's a quasi-cryptocurrency!

</comment></comment></comment><comment><USER7> I love hearing about macroscopic quantum phenomena. That's the path to warp drives.

<comment><USER8> I've sort of assumed that warp-drives == time-travel and are thus the same level of improbability. Is there a consistent universe where we have warp- drives and are NOT able to time-travel?

<comment><USER9> I Am Not Any Kind Of Physicist, but the way I've always considered it, is that if you can travel from point A to point B in less time than the speed of light permits, then that is essentially the same as traveling back in time.

<comment><USER10> Well, yea, it literally is traveling back in time in some inertial reference frame. So far as we know, the laws of physics are identical in one frame, it's possible in ours.

</comment></comment></comment></comment><comment><USER11> Alright, so it is a particle composed of a particle (an escaped electron) and the hole (simulation of positive particle = sum of all other electrons influence) it left in it's valance shell?

If I got that wrong (and possibly even if I got it right), can someone ELI5 this and its applications (if any)?

<comment><USER12> Depends, what does ELI5 mean?

      When an electron, seated at the edge of a
      crowded-with-electrons valence band in a
      semiconductor, gets excited and jumps over
      the energy gap to the otherwise empty
      conduction band, it leaves behind a "hole"
      in the valence band. That hole behaves as
      though it were a particle with positive
      charge, and it attracts the escaped
      electron. When the escaped electron with
      its negative charge, pairs up with the hole,
      the two remarkably form a composite particle,
      a boson.

So, it is something else entirely, after this whole process completes. It is transformed from being an electron, and is no longer an electron.

Electrons are Leptons, not Bosons.

The electron is transformed into a different thing, after it jumps out of its hole, only to land back in the hole it jumped out of.

When they made an electron jump out of it's valence shell, and then permitted it to bounce back into the void it left behind, landing back in the slot changes it enough that it's no longer an electron.

Apparently the orbit it resumes is different enough to be detected with instrumentation, which actually is pretty interesting, and probably matters in terms of optimizing materials for solid-state applications.

I can't tell if it's just that these are temporary ripples in the valence field that eventually resolve themselves, and that it's just a mild technicality, to be able to notice that the jumping electrons don't immediately meld back into their cloud, and that on paper, and due to mathematical descriptions, this "qualifies" the rebounded electron as "no- longer-an-electron" until it resumes a normal harmonious "orbit" with the atom's general electron cloud? (AKA: "decays back into an electron" in technical terms)

How does this kind of boson transition back to an ordinary electron? How long does that take? How often does this happen in nature? What does this mean in the grand scheme of things? Is the entire world, nay, _universe_ , slowly and irrevocably transforming all electrons into these things forever? How come the all the world's electrons haven't completely decayed into these sorts of bosons by now? Is this really very significant, or is it just a tidbit of technical trivia that persists for less than a few units of planck time?

The article does not answer such questions.

<comment><USER13> ELI5 means "explain like I'm five" or it's a way for asking for a layman's explanation of something

</comment></comment></comment><comment><USER14> Article at the university's site: [https://physics.illinois.edu/news/article/24114](https://physics.illinois.edu/news/article/24114)

<comment><USER15> Most useful link, thanks.

So, the idea seems to be that an electron that has crossed ("excited") into the conduction band may, instead of recombining with a hole it (or some other electron) left behind, it may, as the article puts it, "pair up" with the hole and thus form a bosonic quasi-particle.

One of the interesting aspects of this is that unlike Cooper pairing (of two electrons), the pairing of an electron with a hole is a collective effect, i.e. one that requires the participation of many electrons (as well as the atom's nucleus that accounts for the hole's positive charge).

Those who want to learn more about "collective electrodynamics" may consult a book by Carver Mead.</comment></comment>


Meat – Free Git collaboration platform - <AUTHOR> https://getmeat.io/

<comment><USER2> Interesting to note that this specifically says several times that it's free, but it doesn't appear to be open source, and the only availability seems to be an OVA [1] file.

Have I missed something, or is this a non-open free-as-in-beer tool?

I'm not saying that's necessarily a bad thing - I absolutely support the goal of companies/teams/groups having more control through better self-hostable tools (and any change that might have on the current mindshare-monopoly GitHub has)

[1] [https://en.wikipedia.org/wiki/Open_Virtualization_Format](https://en.wikipedia.org/wiki/Open_Virtualization_Format)</comment>


USB-C adds authentication protocol - <AUTHOR> http://www.theregister.co.uk/2016/04/13/usbc_adds_authentication_protocol/ <comment><USER2> DRM on USB devices. No thank you. I know - it is for the users. But we all know how it will turn out.</comment>