A Desperate Plea for a Free Software Alternative to Aspera (2018) - <AUTHOR> https://www.ccdatalab.org/blog/a-desperate-plea-for-a-free-software-alternative-to-aspera <comment><USER2> Ex Aspera Dev here. I did the encryption and early parallel work.

There is a lot of good science behind fasp. An advantage it has over IETF protocols is that both ends trust one another. Another advantage, until recently, was out-of-order delivery.

The protocol totally ignores drops, for flow control. Instead, it measures change in transit time. The receiver knows, the sender needs to know, but the useful lifetime of the measurement is less than the transit time. This should make an engineer think "control theory!", and did. So, the receiver reports a stream of transit time samples back to the sender, which feeds them into a predictor, which controls transmission rate. Simple, in principle, but the wide Internet is full of surprises.

If you think this wouldn't be able to go a thousand times faster than TCP, you have never tried moving a file to China or India over TCP. :-) (Customers used to report 5% drop rates.) Drops and high RTT are devastating to traditional TCP throughput on high-packet-rate routes; read about "slow-start" sometime, and do the math. Problem is that for untrusted peers, drops are the only trustworthy signal of congestion. Recent improvements where routers tag packets to say "I was really, really tempted to drop this!" help some.

Torrents get the out-of-order delivery and the lower sensitivity to drops, but its blocks are too big.. Others commented that opening lots of connections gets around some TCP bottlenecks, but that helps much only when the drop rate isn't too high (i.e. not to India or China).

<comment><USER3> Rate control sounds nearly identical to TCP Vegas then, no?

But out of order delivery and (I'm guessing) out of order retransmits appear to be unique to fasp. What did you mean about these being unique "until recently"? [edit - nevermind, saw below you were referring to sack]

PS. I remember looking at fasp ~10 years ago and it looked like a fantastic tech. Someone's taking things that many people only talked about and putting them into a product that really worked well. Cutting edge stuff. A job like - a dream for many :)

<comment><USER2> No, quite different from Vegas, if the graphs are to be believed. What I would see on real trans-oceanic links was a sub-second ramp up to a flat, stable rate, and stepwise rate changes during the session as other flows started and stopped, with no visible oscillation.

(I am looking at [https://blog.apnic.net/2017/05/09/bbr-new-kid-tcp- block/](https://blog.apnic.net/2017/05/09/bbr-new-kid-tcp-block/) .)

Fasp does not rely on RTT ACK timing, which is very noisy, but only outgoing delays.

The routers are applying their own algorithms to simulate sane queue timing, but they are doing all kinds of crazy shit under the hood to get better link performance. The queues, IOW, are a fiction provided to the endpoints to simplify their job and generate understandable, thus tunable full-route dynamics.

But the ACKs get completely different treatment, because traditionally nothing cared about ACK timing.

So, "onset of queuing", e.g., is pretty meaningless in practice, and fasp makes no attempt to detect it.

</comment></comment></comment><comment><USER4> Have you tried [https://github.com/openrq- team/OpenRQ](https://github.com/openrq-team/OpenRQ) or [https://github.com/harmony-one/go-raptorq](https://github.com/harmony-one/go- raptorq) or [https://github.com/cberner/raptorq](https://github.com/cberner/raptorq) (the last seems to have excellent decoding/encoding performance)?

(Raptor codes (a type of FEC) can essentially transfer over UDP at the _underlying line rate, even if packet drop is high_. In other words, if it is a 1 gbit/s link, with 5% packet loss, you will be able to send over 900 mbits/s without needing any TCP features). This is also helps high latency scenarios, as you don't need test/back off how much data to send...you just send as much Raptor encoded data as you want down the pipe, and as long as you send enough recovery packets with it, you will be able to reconstruct the original flawlessly).

(Also see [https://par.nsf.gov/servlets/purl/10066600](https://par.nsf.gov/servlets/purl/10066600)

[http://www1.icsi.berkeley.edu/~pooja/HowUseRaptorQ.pdf](http://www1.icsi.berkeley.edu/~pooja/HowUseRaptorQ.pdf)

[https://a1f9fb7d-b120-4c73-98e3-5cdb4ec8a2ab.filesusr.com/ug...](https://a1f9fb7d-b120-4c73-98e3-5cdb4ec8a2ab.filesusr.com/ugd/4c5b11_93bc499bed254bf29057bfb5f8c44867.pdf)

[https://youtu.be/bYPbat-FFTo](https://youtu.be/bYPbat-FFTo)

[https://github.com/mk-fg/python-libraptorq](https://github.com/mk-fg/python- libraptorq))

<comment><USER5> You don't need 5% packet loss on a high latency 1Gbps link to trash performance. Even .1% packet loss is enough to cut throughput massively on long fat networks (LFNs).

Erasure coding helps, but it demands much more CPU than simply having a very wide window for retransmits. For bulk data transfer of files, retransmitting any part of a 1GB window is trivial.

<comment><USER6> You mean with raptorQ 5% loss trashes performance by more than 5%?

<comment><USER4> No, the overhead is fairly minimal. A 5% packet loss (which is very bad) on a gigabit link where the sender is thousands of miles away might have an effective throughput of 930 mbits per second with RaptorQ. With regular TCP you would get only a few megabits/s, as it would back down until the loss rate was nearly zero.

</comment></comment></comment></comment><comment><USER7> This looks interesting as well as it's developed at CERN and uses plain tcp protocol [https://github.com/fast-data-transfer/fdt](https://github.com/fast- data-transfer/fdt)

<comment><USER8> Second this. When we needed to deal with somewhat similar issues (quickly transfer multi-gigabyte archives over massively unreliable link) , we ended up with FDT ; it can be easily scripted , and transfers data between two hosts (without using a network of torrent peers) at that hosts' link speed. Used it to transfer builds between US and Russia, and Russia and China .

<comment><USER9> +1

Used fdt to transfer a 6TB archive out of AWS very smoothly at full speed.

</comment></comment></comment><comment><USER10> The 'proof' shown that aspera is 200x faster shows an ETA of 2:15 compared to 00:09. First of all, this is 15x faster, not 200.

I tried downloading the file using axel -n 32, and it took 1:28, which is only 9x slower. Curious though, when it started out with all threads running it was transferring data at 29594.2KB/s and hit 50% after only a few seconds, but by the end with only a few threads left running it was only doing 5625.8KB/s.

It looks like the performance varies a lot, possibly due to multiple interfaces or wan links being used.. or issues with their server.

Using axel -n 64 over http I was able to fetch it in 50s, which is only 5x slower, and most of that time was spent from 98 to 100% as the last few connections finally finished.

A smarter client that more aggressively re-fetched chunks being downloaded over slow connections would likely match the same performance.

<comment><USER11> Yeah, and one window shows speeds in MB/s and the other in Mbps (a 10x difference in units right there).

<comment><USER10> ah, I bet they missed that which is how they ended up with 200x instead of under 20x

</comment></comment></comment><comment><USER12> Ex Aspera user here.

I worked in VFX, so I've been using aspera since before it was owned by IBM.

Depending on what line speed you have, but for a gig link we made a simple protocol using parallel TCP streams.

Basically, it chunked up the large file into configurable sized chunks, and assigned a chunk to each stream.

Another stream passed the metadata.

This has all the advantage of TCP, with less of the drawbacks of a custom UDP protocol.

For transferring files from london to SF we were getting 800mbit/s over a 1 gig link.

</comment><comment><USER13> Why not use torrents over wireguard? From an engineering perspective I'd personally be hard pressed to come up with a more optimal solution for private big data transfer.

If Aspera is as good as the parent suggests it's probably because they are operating a bittorrent-like network of geo-distributed peers. It would be very cheap to emulate that using cloud computing providers, especially if you are discarding the data after transfer.

<comment><USER12> Aspera uses multiple UDP streams to push data.

The special sauce is measuring the packet loss quickly enough to make sure your not over saturating the link

Apart from that, its a fairly simple protocol.

<comment><USER2> Aspera only uses multiple UDP streams if the line rate is too fast for one sending process to keep up. Very often the actual bottleneck is the receiving storage backend.

It absolutely does not measure packet loss at all. The only response to a dropped packet is to request a re-send.

The protocol is simple but depends on very smart sending rate control. Much of the complexity of the current version is to perform well when the receiver has a slow disk, or the link goes through a satellite.

</comment></comment></comment><comment><USER14> I benchmarked Aspera against alternatives for getting data from on-prem storage into the cloud (google cloud storage) a year or two ago on a 10Gbps link.

Aspera was faster on a single stream, but if you have a lot of files to move around (you usually do) you can just multiplex a bunch of tcp streams to get the same throughput.

So like in the example on their page, if you have 20 wget's hitting their ftp and there are no other bottlenecks, the throughput will be similar to using Aspera - and you have just a greater variety of free tooling for tcp based those protocols...

<comment><USER2> Sending on lots of TCP streams can get you to 80% line utilization if the drop rate is good and low. That's often good enough. On some lines, with the "wrong" number of TCP connections, the rate will oscillate, and you will be lucky to get 50%. Tuning that is a black art some people enjoy.

<comment><USER14> Yeah, I was probably working in an environment with pretty low packet loss.

Looking at the problem holistically - if you're going to spend money on Aspera to fix situations with high packet loss, maybe you're better off spending money on a higher quality connection and using tcp.

</comment></comment></comment><comment><USER15> Honestly speaking, a good way to solve this is not trivial. You would need to:

1\. Measure the existing performance on specific data sets.

2\. Understand how many of the bottlenecks come from the network latency vs. I/O limits vs. CPU bottlenecks (if using compression).

3\. See if any domain-specific compression is needed.

4\. Document the typical use cases. Understand why the current solution sucks (e.g. requires redundant user actions). Write down user interaction scenarios. Design the UI to be as efficient as possible for those scenarios.

This is a non-trivial amount of work that would require a lot of back-and- forth interaction and on-the-go requirement changes and I don't think it's entirely honest to ask someone to do this work for free in the name of cancer research (after all, you are not donating most of your paycheck to charities, are you?). If the existing solution by IBM sucks, how about making a Request For Proposal [0] and seeing if smaller software vendors could offer something better given that you are actually willing to pay for the work?

P.S. A student/hobbyist can probably whip out some sort of a parallel TCP-like thing with a large window for free, but you would get the same performance by just cranking up the TCP window size via sysctl and using multiple HTTP threads (htcat was suggested earlier in the comments).

[0] [https://en.wikipedia.org/wiki/Request_for_proposal](https://en.wikipedia.org/wiki/Request_for_proposal)

</comment><comment> yoz-y Way back when I had a problem on my network that was capping the FTP/SFTP speeds to about a fifth of my download speed. Torrents worked fine. To fix my problem I have cobbled up an extremely hacky solution which consisted of opening several listening sockets, ssh-ing to the remote machine and net- catting the file by chunks in several processes in parallel. It actually worked. The code is here [https://github.com/jlegeny/blazing- speed](https://github.com/jlegeny/blazing-speed) more for archiving purposes than to be used by anybody for anything.

</comment><comment><USER16> FTR, they say: "As you can see, the ascp client performs over 200x faster than FTP, and can download the whole file in 9 seconds. It's pretty magical." But the example shows MB/s vs Mb/s. It's not over 200x times faster, but about 17 times faster.

</comment><comment><USER17> Question at the meta-level: what software/platforms are producing, analyzing, modifying & consuming this data?

Is it possible some legacy systems that produce and consume these massive files on-site would more sensibly run in the cloud, directly & selectively accessing the data chunks they need over fast backbone connections?

Also, is there any room for an rsync type approach, sending compressed deltas rather than naively sending huge files that may be redundant?

Not to say disintermediating a BigCO expensive patented vendor-locked-in MLPOS (Market Leading Piece Of Shit) doesn't sound exciting -- it does. I'm just curious and a bit skeptical that it is always necessary to mass-copy all this data over and over.

<comment><USER18> The systems that produce massive DNA sequence files are machines reading the sequences of actual physical DNA molecules. The cloud can't sequence DNA, as far as I know.

<comment><USER2> Most DNA sequences are only slightly different from sequences you already have. There ought to be mature differs by now that would enable you to send just differences. If not, there is a huge opportunity for somebody.

<comment><USER18> Sure, you can compress the sequences by diffing against the published genome sequence, and many tools exist to do just that. But that alone maxes out at 50% compression, because the other half of the file is the quality scores that quantify the confidence level of each base call, for which there is no reference. Of course there are patterns in the quality scores, and you can compress these as well using the usual techniques for general compression, but it's going to limit how much you can compress without information loss. In addition, the sequencer assigns an unique ID to each sequence read. It's often ok to discard this information, but occasionally you need it in order to trace a specific problematic sequence through the entire analysis pipeline. So if you want lossless compression, you have to compress these as well. They should be rather compressible, since they are typically a prefix global to the file followed by a number that increments by 1 for each sequence. But for a general compressor you can't just automatically _assume_ that, because according to the (lack of a) file format spec, each ID could be anything as long as they're unique within the file. So if you want guaranteed lossless compression of any FASTQ file, your compressor needs to be sufficiently general.

Beyond that, compression of sequences by encoding only the differences from a specific reference sequence somewhat ties you down to using a specific reference, since that reference sequence is required for decompression. This is inconvenient because the reference sequence for a species is updated over time, and if you want to use the new reference, you'll need to decompress with the old reference and then re-compress with the new one. Do you want to do that for all your data, every time the reference is updated? Reference-based compression also means the compressed files are no longer self-contained, which may not be acceptable for certain use cases.

None of these issues are fundamentally impossible to address, but the point is that it's not nearly as simple as it sounds, and anything fancy you do to try and compress a sequence file will generally impose some additional requirements on what the receiver needs to do in order to read the file. And many receivers are not very technically inclined researchers whose plates are already full of other things they need to be doing besides figuring out how to decompress your new unfamiliar sequence compression format,

<comment><USER2> Thank you for this enlightening explanation. I should have guessed that it, like most things, is not as easy as it looks.

</comment></comment></comment></comment></comment><comment><USER19> I feel the author's pain - Aspera is a clunky 90s styled Ruby on Rails app that is about the most cloud unfriendly piece of software I've encountered, I'd love an alternative.

I honestly don't care about about their proprietary UDP protocol, it's nothing special, just another way to copy bits onto a wire. Dime a dozen.

The true value of Aspera is they provide an integrated browser plug-in that lets the technically challenged reliably upload large files. If the transfer is interrupted or either side changes addresses it deals with it gracefully. It's also a bridge to AWS S3.

I spent some time looking for a replacement and while there are numerous download managers that facilitate people downloading large files, I couldn't find any upload managers with the same level of integration and polish.

About the closet thing I could find is Cyberduck, but it's not integrated with the web browser, not as easy for technically challenged people to use, and there is no support (community support but seems really hit and miss). However it does make good use of Amazon's multiple upload api and will happily fill whatever wire it's connected to.

Torrent software has largely the same pros and cons Cyberduck does.

</comment><comment><USER20> I have seen this software before. I suppose the speed boost results from using multiple parallel connections, meaning it could make better use of aggregate links and multipath networks? Is there anything else to it?

<comment><USER21> It does what it says and it's expensive. There's a couple of commercial packages like this - two that come to mind are vcinity and Signiant. A solution can be made but every time I look at the market you find very few customers with a real need to move terabytes that can't solve their problem with a snowball-style solution. Those that do usually also have the economics to justify a commercial package.

<comment><USER2> Don't know about vcinity, but Signiant was always a marketing-heavy, engineering-light organization. Their focus was on user convenience features, and they almost ignored actual transit engineering.

</comment></comment></comment><comment><USER22> A proprietary protocol that somehow facilities throughout orders of magnitude than non proprietary alternatives. I call BS.

<comment><USER23> If they are like Riot and have their own fiber, it is not unlikely. There are tons of boring optimizations open source software also won't bother with without significant sponsorship (some could be specific to a particular file format). If you look at most network (and also video) protocols, they are accumulations of decades of improvements rather than neural network style overnight breakthroughs.

<comment><USER24> Do you know of any resources where I can learn more about these boring optimizations?

<comment><USER25> MIT open courseware has a course on Performance Engineering which details some of the types of low level optimizations that can be done. It requires a lot of measuring and problem specific solutions.

<comment><USER26> I've helped with that course in the past. It's awesome. It's completely unrelated to WAN networking. I found the IETF groups working on these problems more helpful.

<comment><USER24> Thanks. Any specific recommendations for someone fairly new to low-level networking? IETF docs can be a little dense :)

</comment></comment></comment></comment></comment></comment><comment><USER27> I've used UDT via UDR for rsync optimization and transfers over high-latency links: [https://github.com/LabAdvComp/UDR](https://github.com/LabAdvComp/UDR)

<comment><USER28> Nice! Thanks for the info.

</comment></comment><comment><USER29> Try looking at bbcp perhaps?

[http://pcbunn.cithep.caltech.edu/bbcp/using_bbcp.htm](http://pcbunn.cithep.caltech.edu/bbcp/using_bbcp.htm)

<comment><USER30> bbcp doesn't work so well for some reason. Where I used to work we had a senior dev that claimed that he could just repurpose bbcp for this. Then he gave up and wrote something custom in c++. Then he rewrote it in go, but used (among other things) non-multithreadsafe primitives, then when asked to make it encrypted, he used epoll with tls (which is apparently not a thing in go? I don't know). I told him he should just use DTLS or hell even a one-time-pad encrypted UDP stream with backpressure management, but he didn't listen to me. Then he rewrote it in C++ again and then went back to the go version. When I left the company, it still wasn't working.

Also, this senior dev never wrote unit tests.

On the other hand, maybe bbcp will work, and the senior dev just didn't know what he was doing. There was another senior dev who, seeing what was coming down the pike, left his job, and on his way out he was like, "yeah you can do it with bbcp, just you gotta tweak your tcp congestion rules on all the hops (which we _could_ do, but is probably not an option for OP)"

<comment><USER31> >On the other hand, maybe bbcp will work, and the senior dev just didn't know what he was doing

I mean... based on your description it sure sounds like he didn't know what he was doing.

<comment><USER30> Well yeah but the competent dev would have told you that it wouldn't work for OP.

</comment></comment></comment></comment><comment><USER32> In the past I see that Tsunami, UDT, or GridFTP were suggested. Any results?

<comment><AUTHOR> Yes, I got a similar list in response to my link on Twitter [https://twitter.com/fanf/status/1210858696558944256](https://twitter.com/fanf/status/1210858696558944256)

But it looks like GridFTP and Globus are basically dead [https://opensciencegrid.org/technology/policy/gridftp-gsi- mi...](https://opensciencegrid.org/technology/policy/gridftp-gsi-migration/)

Tsunami sounds good but maybe needs some updating? I haven't looked closely...

<comment><USER33> Globus is definitely not dead[1], it has an active community with strong uptake in the Materials Science world (where I work), among others. The Globus folks have a repo for many popular Linux distros, it's reasonably straightforward to get started. (I'm a user of it, not a developer or necessarily an advocate.)

It's not free for "managed" (Enterprise) applications, though, which is what the OP seems to be looking for, and I'm not sure it's the best choice for high-speed.

Something that I think _is_ free is the CERN FTS [2], which can use a GridFTP back-end, so possibly you can roll your own high-speed big-data infrastructure that way.

[1] [https://www.globus.org](https://www.globus.org) [2] [https://fts.web.cern.ch/](https://fts.web.cern.ch/)

<comment><USER34> I concur that Globus is definitely not dead! We're transferring 10s to 100s of TB per month with it. Globus is continuing to maintain GridFTP for this purpose, as part of Globus Connect Server and Globus Connect Personal, and it will happily saturate a multi-Gbps link.

And personally, I'd like to see more people getting a Globus subscription. It's cost-effective when compared to tools like Aspera, and helps fund the development of Globus software and features.

</comment></comment></comment></comment><comment><USER35> > Aspera is owned by IBM - does their client spy on what else is running on my > system and report it back to their headquarters as business intelligence? If > asked, would they share this information with the government? Without the > source code, it's impossible to tell!

I mean, with some reverse engineering, mentioned later in the post, it's really not.

</comment><comment><USER36> My group is about to start using Aspera as well.. I too wish there was a good alternative I could suggest to my bosses. And I'm still skeptical of their claims... but I haven't gotten to test it over an actual bad link yet (over a good link it does nothing, which a simple test will prove).

<comment><USER34> I don't know what your needs are, but Globus might be a good alternative.

[https://www.globus.org](https://www.globus.org)

It's being run out of UChicago, and I know they have at least one large company using them. Feel free to reach out to the Globus team, or shoot me an email!

</comment></comment><comment><USER37> There are other wan accelerators out there - like Riverbed.

But, I think Aspera uses some specialized compression for genomics data, and genomics compression is not easy.

My suggestion would be to talk to the telco that provides your wan. If you are using a direct internet connection my suggestion would be : don't.

<comment><USER38> The author is discussing intercontinental data transfers. Not many organizations have a WAN that can cross oceans without hitting the internet.

<comment><USER39> almost all HE and research institutions participate in nrens which provide 3xacrly those peerings

<comment><USER2> Yes, but that typically takes government cooperation, and you still don't get better latency -- even if you have neutrinos.

<comment><USER37> It doesn't, it takes buying access to a link.

Latency isn't the problem here - it's throughput.

<comment><USER2> Oh, how nice it would be to live in such a simple world.

Latency turns out to have a very great deal to do with how hard it is to get the nominal throughput you pay for.

</comment></comment></comment></comment></comment></comment><comment><USER40> What about UDT [1]? Several open source libraries exist already.

[1]: [https://en.wikipedia.org/wiki/UDP- based_Data_Transfer_Protoc...](https://en.wikipedia.org/wiki/UDP- based_Data_Transfer_Protocol)

</comment><comment><USER41> Would QUIC or TCP with BBR congestion control help?

<comment><USER42> TCP with PCC, even? I don't know how BBR compares with PCC.

Previously [https://news.ycombinator.com/item?id=8381480](https://news.ycombinator.com/item?id=8381480) (2014)

</comment></comment><comment><USER43> Another commercial alternative is M-Strean - non proprietary TCP based parallel streaming.

Limited to work with CIFS / SMB / NFS and S3 compatible / OpenStack Swift compatible / Ceph / Azure storage.

Commercial model is server based with no additional bandwidth charges .

Data stored in a non proprietary fashion so bi-modal access to data possible.

Not sure if it is working with QUIC or not as it is mentioned in blog posts.

Seems heavily focused on the media and entertainment industry.

[https://storagemadeeasy.com/M-Stream/](https://storagemadeeasy.com/M-Stream/)

</comment><comment><USER44> Facebook open sourced WDT, which seems to have the same goals: [https://github.com/facebook/wdt](https://github.com/facebook/wdt)

</comment><comment><USER45> There's a meta answer about the general approach to such problems: Ask what other research areas must have tackled it, and then find out what they do. The obvious answer in this case is HEP, which would lead to some of the tools mentioned. (There's also [https://www.psc.edu/index.php/hpn- ssh](https://www.psc.edu/index.php/hpn-ssh) but I don't know how it compares.)

It's unfortunate how many researchers don't do such research...

</comment><comment><USER46> Beyond more profound changes to the underlying transport protocol, bandwidth acceleration can be provide through a combination with fast compression.

A good setup here would be to pipe data to `tar`, then `zstd`, then `ssh`/`scp`, and do the reverse on the receiver side.

From an end-user standpoint, all this could be abstracted away if one could get `wget` (named in the article) to support zstd compression natively, which is possible since zstd is also a web compression standard.

Such compression support is, by the way, available in `wget2`.

</comment><comment><USER47> I wonder why something like [https://github.com/htcat/htcat](https://github.com/htcat/htcat) doesn't fit the bill.

</comment><comment><USER32> If you think about giving zmq a go, take a look at my library, here [1], [2].

Maybe we can talk about compiling a test to see how it compares?

[1] [https://pypi.org/project/p2p0mq/](https://pypi.org/project/p2p0mq/)

[2] [https://p2p0mq.readthedocs.io/en/latest/](https://p2p0mq.readthedocs.io/en/latest/)

<comment><USER48> ZMQ is too unreliable at transferring data which is a fraction of the stated use-case. Streaming realtime data, ZMQ had problems dropping data unpredictably under load testing, so ZMQ never made it out of testing for us. The protocol is too naive.

</comment></comment><comment><USER49> I think this lists some alternatives...[https://www.chpc.utah.edu/documentation/data_services.php](https://www.chpc.utah.edu/documentation/data_services.php)

I'm curious. How fast are these compared to netcat/socat/mbuffer?

<comment><USER50> It mentions rclone with the parallel option of "\--transfers=N". I wonder how well that does.

</comment></comment><comment><USER51> Try NKN (New Kind of Network) for this task

[https://medium.com/nknetwork/nkn-file-transfer-high- throughp...](https://medium.com/nknetwork/nkn-file-transfer-high-throughput- file-transfer-using-nkn-client-21d261d87658)

</comment><comment><USER52> Dat isn't designed for speed, but I think academic data is it's primary goal, so modifying it for speed seems within its charter.

[https://www.datprotocol.com/](https://www.datprotocol.com/)

</comment><comment><USER53> > _We'd have to fly all over the world with suitcases full of harddrives to be > able to collect our data!_

The postal system or commercial courier services seem like better options than buying plane tickets..

</comment><comment><USER54> How about mpscp from Sandia?

[https://www.osti.gov/servlets/purl/1146079](https://www.osti.gov/servlets/purl/1146079)

</comment><comment><USER16> Maybe OpenVPN + UDPspeeder could do something:

[https://github.com/wangyu-/UDPspeeder](https://github.com/wangyu-/UDPspeeder)

</comment><comment><USER55> I wonder how will aria2 stack up if you use it instead of wget?

[https://aria2.github.io/](https://aria2.github.io/)

</comment><comment><USER39> gridftp's the tool to use, and Globus makes it easy.

</comment><comment><USER56> Looks like the plea has been up since a while. Does anyone know if they had any progress on the problem? Is the problem still standing?

</comment><comment><USER17> Btw attempted to contact you at the email you put on your page. Interested in talking about this project.

</comment><comment><USER57> Can they not leverage torrents for this purpose?

</comment><comment><USER58> Question: if everybody used Aspera (instead of conventional transfer methods), would that negatively impact internet speeds overall?</comment>


Crimean Crisis Sanctions: Russia's 19 Richest People Lose $18.3B in Two Weeks - <AUTHOR> http://business.financialpost.com/2014/03/14/russian-ukrainian-oligarchs-brace-for-worst-as-western-threats-of-sanctions-intensify/

<comment><USER2> Putin might see this as an added bonus, since many of the "oligarchs" are not on friendly terms with the Kremlin.

<comment><USER3> Can you expand on that?</comment></comment>


Let's Abolish High School - <AUTHOR> http://www.youthrights.net/index.php?title=Let%27s_Abolish_High_School_%28article%29

<comment><USER4> > Teenagers are inherently highly capable young adults

I'm currently a teenager. This is mostly false.

\--

On a serious note, interesting idea. It won't happen, since the US education system (and the public sector in general) is not open to, well, any kind of change.

But as the founder of 9 startups, high school wasn't a huge help for me in any ways other than somewhere to socialize. _However,_ this is the _special case_. High school works decently enough for the rest of the 95%.

Dropping out is not an option. You 100% will have a much harder time with acceptance with most colleges (and asian parents.)

<comment><USER5> _I'm currently a teenager. This is mostly false._

I was a teenager -- I dropped out of school, moved out of my parents home, and took a full time technology job at 16, and began working my way up. This meant that by the time my peers graduated from college, I had 6 years of experience under my belt.

I wasn't _special_ , I was simply _motivated_. If my home and school life had been satisfactory, I probably would have been content to sit out my future high school and college years with very little responsibility, and I imagine this is the situation that most teenagers find themselves in.

 _High school works decently enough for the rest of the 95%._

95% of teenagers don't accept real responsibility and behave like capable adults because there is no external motivation or encouragement to do so.

Is this beneficial to their well-being?

 _Dropping out is not an option. You 100% will have a much harder time with acceptance with most colleges (and asian parents.)_

That depends on what you would like to do with your life. Returning to college later in life is surprisingly easy.

<comment><USER6> If you did something that 99.9999% of the population did not do then you are special by definition.

Teenager's have significantly different levels of cognitive ability compared to adults. There is a reason most people look back on their teenage years and say "why was I so stupid" and honestly it's because your brain was less developed on a biological level.

[http://books.google.com/books?id=2xpvp4ie_8MC&dq=hippoca...](http://books.google.com/books?id=2xpvp4ie_8MC&dq=hippocampus+development+teenagers&source=gbs_navlinks_s)

<comment><USER7> _> most people look back on their teenage years and say "why was I so stupid"_

As far as I can tell, that pattern holds for at least another decade.

<comment><USER5> Which is why I tend to believe it has quite a bit more to do with socially encouraged irresponsibility than teenage brain development.

<comment><USER8> This discussion highlights why these questions are so difficult and frightening: ethics _prevents_ good science.

Randomly sample a bunch of kids, hook them all up to monitoring devices, put them in different schools and societies that involve transitions into adulthood at different rates and onsets, set up a control group (Lord of the Flies setup), do the longitudinal study, and write it up. An associate professor and his 5 grad students would get a bunch of solid papers out of it. Theorists would have fun defining "optimality" in this context (to be fair, the philosophers have been gnawing at it for a while..)

This is unethical .. right? Then, you have to ask yourself: wouldn't it be _more_ ethical to do the study, figure out how best to raise kids, and let _all_ human young for posterity (or at least 'til the results are invalidated) benefit from the wisdom?

Hmm.. :)

<comment><USER9> It would only be more ethical if your ethical system was based on a utilitarian viewpoint.

In contrast, most capitalist democracies (all that I can think of anyway) are centered on individualism... the only ethically allowable sacrifice is the voluntary sacrifice... and children (anyone under 18) are not considered sentient enough to make decisions about their own lives and futures.

<comment> loup-vaillant You prove him right: by _assuming_ underage people aren't sentient enough, ethics prevent us to check. Not necessarily bad, but definitely restrictive, and not good science™.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER10> _The assertion that teenagers have an 'immature' brain that necessarily causes turmoil is completely invalidated when we look at anthropological research from around the world._

That children and teens have an immature brain is supported by biological research:

<http://www.npr.org/templates/story/story.php?storyId=1899715>

<http://harvardmagazine.com/2008/09/the-teen-brain.html>

[http://www.nytimes.com/interactive/2008/09/15/health/2008091...](http://www.nytimes.com/interactive/2008/09/15/health/20080915-brain- development.html)

I was not happy in high school (only ten years ago), so I think efforts to improve that period of people's lives are well spent. But I can't support something that starts from what I consider a faulty premise, which is the assumption that children and teens are miniature adults.

<comment><USER11> What does 'mature' even mean in this context? Don't mistake "not finished growing/changing" with lacking the ability to make mature, thoughtful decisions if given the opportunity/responsibility.

One of the reasons teenagers act so irresponsibly is because they've grown up with the idea that teenagers are inherently irresponsible and therefore aren't completely accountable for their actions. It becomes self-fulfilling.

<comment><USER10> Full maturation of the brain - and I do think that indicates being able to _reliably_ make "mature, thoughtful" decisions. I don't think you should deprive teenagers the opportunity to do so in the correct environment, but that's different from assuming they have the same mental faculties as adults.

Personally, I don't feel like I was a mature adult until about 22, and I was never considered an especially immature teen or young adult. I'm 28 now.

<comment><USER11> >I don't feel like I was a mature adult until about 22

Was that about the time you left college? I think a lot of feeling "mature" is actually having responsibility for yourself. If teenagers had responsibility for themselves, and didn't have the artificial environment of high school to pressure them in the wrong direction, I believe nearly all of them could be fully functioning adults.

I don't think mature brain connections correlates with mature decision making. People become adults when they're finally faced with adult decisions and have to deal with the consequences.

All one needs to make mature decisions is a sense of the consequences of actions and an ability to delay gratification to reach later goals. Both of which is fully within the grasp of teenagers, if they're taught properly growing up.

</comment></comment></comment></comment><comment><USER12> >John Taylor Gatto has long warned about the dangers of artificially extending childhood, and has blamed our schools for damaging families and stifling creativity and a love of learning.

I think it's ridiculous that it is now normal (or actually preferable) now to be in school until you are done with 1/4 to 1/3 of your life without seeing what "the real world" was like. This means the average person won't be able to do anything significant until they have already wasted their best years in school and low level positions.

I can get decent grades pretty easily (much like many people here) and so slacking off and procrastination was just a smart way of using my time. Now that I think I'm of the age where I can make something of myself, I need to unlearn a lot of the bad habits learned in school in order to get things done. Many smart, ambitious people I know do not try to start independant projects because they are still "just students," but then go on to do amazing things after graduating and finding their purpose in life. Much of this can be prevented with an early firsthand exposure to a life outside of the artificial punishment and reward system of school.

Chances are, even after reform people will probably still go on the same path -- highschool, college, graduate school, but they will have a much better reason to do well, since it will be what they chose to do.

</comment><comment><USER13> I think if a 15 year old was put into the position of having to make adult decisions and behave like an adult they would be perfectly capable.

It's just that they either don't have the chance or given the option, prefer to cruse through and muck around for as long as possible before being force to get serious.

<comment><USER14> The objective reality of the sort of decisions made by fifteen year-oldsters on a routine basis show this to be demonstably false.

<comment><AUTHOR> The point that many of us are attempting to make is that this is so because of how we currently treat teenagers. When you treat someone like a child they tend to act like a child.

<comment><USER15> I've known a couple of people who grew up too fast, and that can be kind of ugly sometimes. I'm all in favor of the emancipation of children and the abolishment of school, but atta same time, there's gotta be some kinda line.

</comment></comment></comment></comment><comment><USER16> Whatever happened to the idea of an 'apprenticeship'? Not all teenagers want to pursue 'traditional' studies and would be much better off learning a specific trade or craft that they are interested in.

</comment><comment><USER17> I like the ;?> on the top left. The title is a little misleading, the article is actually much more rational than the title sounds.

 _I'm a father of four children, and about 10 years ago I noticed—I couldn't help but notice—that my 15-year-old son was remarkably mature. He balanced work and play far better than I did, and he seemed quite ready to live on his own._

For parents that help educate their children, stimulate their interest and set an example this might be so, but for others, school might be a better alternative.

<comment><USER18> Agreed. I know if I, for example, had been let loose on the world at 15 things would have turned out BAD.

<comment><USER19> Did you go to normal school?

A lot of people in this discussion are trying to use the product of the putatively broken system (themselves or others they know) to prove that the putatively broken system must be left in place. This is simply begging the question (in the original sense), not an argument.

Edit: Fine, I'll spell it out. You can't counter an argument that says we have a system that is generating excessively dysfunctional 15-year-olds by pointing out that the 15-year olds generated by the system are excessively dysfunctional, and then follow it up with the conclusion that therefore we need the current system. That doesn't disprove the point, because the point in the first place is that the system is broken. You can cite millions of examples of actions taken by publicly-schooled 15-year-olds that were stupid, but none of them actually provide any useful information about the argument.

My post here isn't an argument either way about the issue; it's just _simple logic_. Arguing that of course 15-year olds are stupid and need to be in school is begging the question, or circular logic, and an invalid argument. It doesn't prove anything either way, it's not even evidence either way. Modding it down doesn't change that.

This isn't all spelled out in the exact post I'm replying to, but this bad logic is shot through this entire comment thread.

<comment><USER18> > Arguing that of course 15-year olds are stupid and need to be in school is > begging the question

Where did I argue that! Your putting words in my mouth. All I am saying is that, speaking from where I am now, at 15 I wasnt equipped to face the world. School was a good place for me.

This stuff _has sod all to do with the schools_ ; it's simple logic about maturity. Have you bothered to read the studies posted elsewhere about brain development??

The education system is broken: but that doesnt means it's a _bad_ idea. If we dont impart our knowledge then you end up with a problem. It could, ofc, be taught better than it is now.

<comment><USER19> You said that when you were 15, school was a good place for you. But you are talking about your 15-year-old self who had just spent 10-ish years in school; at this point I'm assuming that if that weren't entirely true you would have said. Since the discussion is about the hypothesis that school is an infantalizing place to be, saying that when you were 15 school was the right place to be is begging the question. If you had been in a hypothetical better environment, your 15-year-old self might have been much more capable of dealing with real challenges.

You don't "say" it. The idea that school is the only way a childhood can be is so deeply ingrained in your argument you can't even _perceive_ it. It's such a given that you consider it a logical axiom, and you end up arguing circularly without even realizing it, you and about half the other posters in this discussion.

Your last sentence would seem to me to reinforce the point. Maybe school can be "improved", but the idea that it is potentially fundamentally flawed doesn't seem to be thinkable. Mind you, I'm not saying that you (and others) think the idea, then reject it (which if done properly would be perfectly valid); people don't seem to be even capable of _thinking_ the idea.

Your picture of maturity is shaped by the system under question.

Again, I'm not actually attacking or defending the schools here. It's the logic I'm talking about. You can't justify the current system by using the current system; it's circular.

</comment></comment></comment></comment></comment><comment><USER20> Limited to yard work/baby sitting/uninteresting jobs? Where exactly?

After I turned 16 I enrolled in a community college, fully paid by high school, using a local dual enrollment program. When I was 15 I started working as a C++ developer, which was quite fun.

Yes, I was a freak and still am a completely asocial misanthrope. But no one is limited -- it is solely by choice (and probably fear of the unknown) that children stay in high school.

<comment><USER21> >> it is solely by choice (and probably fear of the unknown)

Or because of the social benefits. You state that you consider your former self a freak, and consider your current self a completely asocial misanthrope. I was following a very similar path to you, but realized that I was on my way to being a social outcast. To me, being a C++ programmer was not my purpose in life (although it was a very interesting field I ended up making my career). A great deal of what I see as the "purpose of life" consists of those social relationships. If I didn't have those, C++ wouldn't make me happy, so what would be the point of it all?

I don't mean to sound argumentative - I'm just pointing out that although I was capable of doing more, I chose to stay close to my own age group because of a legitimate quality-of-life benefit. You did state that it's by choice, but I think that's an aspect a lot of people don't consider, especially the author in question.

<comment><USER22> It is only because high school is the default that those who choose other options feel isolated. Those >25 usually find friends without resorting to spending most of their day in the same overcrowded fishbowl. If 16 year-olds were spread out across various activities, they would make the effort to find each other, but it's easy to exclude those few kids who aren't seen at their lockers everyday when school is the norm.

</comment></comment></comment><comment><USER11> I believe this is true. Children are adults-in-training and need to be treated as such. Yet we create these unnatural environments and wonder why teenagers act so irresponsibly. And when they finally do become "adults", they can't function as adults.

I believe the majority of teenagers will rise to the responsibility given to them, if it is genuine. You can't expect them to act responsibly on one hand but then subtly treat them as nothing more than overgrown children.

</comment><comment><USER23> There was a post on HN some time back from former house speaker Newt Gingrich, who is a fairly outspoken advocate on this issue:

[http://www.businessweek.com/magazine/content/08_45/b41070852...](http://www.businessweek.com/magazine/content/08_45/b4107085289974.htm)

</comment><comment><USER24> I came to the same conclusion (based on my own experience in teenage, and observations later) that there is a lot of talent waiting to be tapped in teenage years, and treating teenagers respectfully as young adults and giving them responsibility is good for them and good for society.

That is one of the reasons we like to hire 17 year olds in Zoho and combine education with work. Labor laws won't let us hire any sooner, but if it were my own family members, I would hire them sooner.

<comment><USER25> That is amazing. I work for the competition (FreshBooks) but I would have relished working right out of high school in a high tech position.

Good on ya.

</comment></comment><comment><USER26> There is a lot that independent learning by teenagers can do to help them grow faster and more happily. My son likes Project Euler

<http://projecteuler.net/>

as a framework for practicing computer programming related to pure mathematics, and runs his own literary discussion website both to learn about online community administration (including coding site features) and about creative writing. The most annoying aspects of his teen years, for him and for me, have been his closest approaches to "high school" experiences. Senior year of high school for him should be mostly dual enrollment at our friendly state university and a distance learning course in advanced microeconomics.

<comment><USER10> Is he not allowed to do that? I grew up in Virginia, and two of my friends chose to spend their senior years of high school by taking classes at a local university or the local community college.

(They were, in fact, brother and sister and had spent six years in Denmark as the children of foreign service officers. They did not take well to American high school.)

<comment><USER26> _Is he not allowed to do that?_

The dual enrollment is allowed here (one must apply to the college that offers the dual enrollment classes). This choice is not used by as many students as could benefit from it. Many seek the "socialization" of high school. Yeesh.

</comment></comment></comment><comment><USER27> It's not a terribly good use of time for people who actually like to learn. I dropped out of two high schools, including a gifted and talented program - <USER2> the hell knows how I got into that one, but I dropped it as well. It really seemed like obedience school to me - capricious, arbitrary, with a focus on discipline and not on learning. A number of times, I would fail a class where I was in the top five most talented people in it. I didn't see much point in doing what they prescribed - I'd rather go explore the world or play around on a computer or read books or go make some money.

I left home at 16, and eventually wound up graduating high school by going into an adult learning center at age 17 or so. I said, "I am fully emancipated, and living as an adult. I would like to complete the adult program." After some talking with a very nice fellow and being politely evasive, he agreed. I took a series of competency exams, completed the silliest multiple choice packet of American history I've ever seen, and I was awarded a diploma. I went and took the standardized tests and got into university. They were kind enough to loan or grant me almost my full tuition, though I dropped out of university once as well. Same reasons.

Anywho, with a quick perusal of history, it's pretty obvious that maturity does not come from age, it comes from having unique experiences and learning from them. If you see someone who is forced to become "head of household" at a young age due to parents illness and cares for his/her siblings, they are very mature and adultlike even in their mid-teens. Likewise, people dragged into a war move differently, with less naivete - characteristics you'd usually see in someone in their 40's. There's kids in their mid-20's with that look.

At least, that's the hypothesis I made: Unique experiences and learning from them = maturity. So I set out to have a lot of unique experiences, and I think I did rather a good job. I've had an interesting, enjoyable life. The only part I regret is wasting my age 13/14/15/16 years in middle school and high school. I had some fun with extracurricular activities and friends, but I could have done something so much more interesting during the actual classtime. Yuck.

One in particular that made me wake up when I read it was Tokugawa Ieyasu. He was third of Japan's great unifiers. In the 1600's, he founded the Tokugawa Shogunate - his family ruined in almost complete peace for 250 years after one of the most lawless and brutal times in Japanese history. Ieyasu married for the first time at age 13 or 14, and was Captain of a small division of troops around the 14/15 years. He did a pretty admirable job of it, grew up fast, and the rest is history.

But I think the flipside is true, too - if you don't let someone have responsibility, if you treat them like a child, they'll stay childlike and immature. Being an American who has traveled, I see people who are age 26-28 who are barely capable of functioning in the world, and I sometimes see people age 14-18 doing amazing things. This could be the norm, if we wanted it to.

I had a 17 year old fellow apply to work for me - he was half-generation American, if I've got the terminology right - he immigrated here with his parents around age 10 or so. I didn't wind up hiring him, but I became a mentor to him. We'd go get tea or coffee every couple weeks and I'd give him some generally no-BS advise. A very good guy, he's done some really impressive things in a short period of time - all he had to do was shake off the reigns and shackles, but that is harder than it sounds to do. At least in the average American school, they punish you swiftly, thoroughly, and publicly if you try to abstain or dare to question them. There's the occasional good teacher - <USER3> had maybe a dozen excellent professional teachers in my life, which I count myself quite lucky to have had. But the rest? Well, I guess we shouldn't be too hard on them. They're just doing what they were taught, too.

</comment><comment><USER28> If you get a crappy education in HS, despite your abilities, you're far behind your peers who got a very good one. Whether you then apply yourself in college, you remain behind those people because they keep on keepin on.

That's not so true today as it once was for kids who have access to the internet. If you've got a hunger to learn, there's plenty of fodder. My life would have been quite different if I'd had the net to learn from.

As to whether it's _necessary_ depends on what you feel life is all about, and/or what real talents (not addressed by mainstream ed) you have. Highly self-motivated people (or who fortune smiles on) can usualy live without HS. Most of the rest of us need it to cope in an ever-more-complex world.

</comment><comment><USER29> "and he had no choice but to attend high school for several more years"

This is a very common misconception. It's quite possible and legal in many (most?) US states to remove a child from school and keep them home to let them learn at their own pace.

<comment><AUTHOR> <http://www.hslda.org/laws/default.asp>

<comment><AUTHOR> NY for instance seems rather restrictive:

<http://www.hslda.org/laws/analysis/New_York.pdf>

</comment></comment></comment><comment><USER30> Maybe it isn't all about your profession. Maybe you should try to know some of the basics taught in high school in order to be a good citizen. Not to say that the high school I went to wasn't a waste of time, but that doesn't mean high school _has_ to be a waste of time.

People are good at being smug, and thinking they know everything. The best way to realize that there's actually some depth to a topic is to study it a bit. Since (from reading other comments here) none of us would have wanted to learn some of these topics, maybe we'd all still think that those _other_ topics are all a stupid waste of time, and people who do them are also useless.

</comment><comment><USER31> High schools are so unreliable here that we need an AA degree to compensate for it.

</comment><comment><USER32> I largely agree with his conclusion, but don't like it when people point out history to support it. Sure kids didn't have to go to school in 1850, but black people and women couldn't vote, most people didn't have indoor plumbing (let alone electricity), slavery was still legal.

The fact that kids didn't have to go to school back then is not at all an argument in support of them not doing so now and should be left out of a serious essay in favor of that position.

</comment><comment> b-man In case anyone is curious, there are proposals for systems that respect the individual, are libertarian and are passive in the sense that the individual is the active part, choosing when, how, and what he will learn.

Take a look at this article of mine for instance [http://cnxs.com.br/posts/an- emergent-participatory-design-fr...](http://cnxs.com.br/posts/an-emergent- participatory-design-framework-for-higher-education) .

</comment><comment><USER33> Without yet reading the actual post, let me put my thoughts in there.

After the age of 16, make school non-compulsory. At the end of the 10th grade year, have everyone take the GED/ACT/SAT and if they get a threshold level, congrats, you can now move on.

So when kids start misbehaving, you don't have to keep them there.

</comment><comment><USER34> I completely agree with this point and I wish he went into more detail about it:

"Finally, a wealth of data shows that when young people are given meaningful responsibility and meaningful contact with adults, they quickly rise to the challenge, and their inner adult emerges."

</comment><comment><USER35> Eliminating high school would cut the amount of taxes needed for public schooling, and would instead increase tax revenue by putting more people in the workforce. Not to mention that a more experienced (real-world experience) workforce might be more productive.

</comment><comment><USER36> See also Grace Llewellyn's Teenage Liberation Handbook:

[http://www.amazon.com/Teenage-Liberation-Handbook-School- Edu...](http://www.amazon.com/Teenage-Liberation-Handbook-School- Education/dp/0962959170/)

</comment><comment><USER37> Basically what they are proposing is the system of education currently in use in Germany. Once you hit high school you have the choice of preparing for a white or blue collar job. Blue collar jobs gets you out of school and into an apprenticeship at 15 or 16, white collar gets you into high school, or further education at least.

Seems to work quite well.</comment>


Amazon pitched facial-recognition system to ICE for use to target immigrants - <AUTHOR> http://www.chicagotribune.com/business/ct-amazon-facial-recognition-immigration-20181023-story.html <comment><USER2> >Unfortunately, our website is currently unavailable in most European countries...

</comment><comment><USER3> Great!</comment>


Ask HN: How do you organize your list of academic papers? - <AUTHOR> I started reading a lot of research papers as of late, especially in Machine Learning. I keep a list of links to papers I want to read, or have already read, so I can easily get back to them.
I've been using Google Keep to do this:
- 1 note per topic (e.g.: Recurrent Models, Vision, Memory, etc), with checkbox enabled, and labelled "Papers"
- 1 link + paper name per checkbox item, marked as "done" if I've read it
This is not an ideal system, since it...
- breaks Keeps' search functionality (Keep highlights notes that match your search, not individual checkbox items)
- makes it very hard to reorganize items in separate notes (you have to copy-paste)
So I was wondering if y'all knew of a better way to organize academic papers. Is there a better tool out there I should look at? Is there a better way to use Keep? I've been suggested to use 1 note per paper, which also allows me to write down thoughts after reading the paper, but that also means cluttering my Keep with tons of notes and it becomes harder to organize papers by topic. <comment><USER2> Zotero.</comment>


How costly is Chinese IP theft? - <AUTHOR> http://www.themoneyillusion.com/how-costly-is-china-ip-theft/ <comment><USER2> Yes. A $100 counterfeit handbag costs American companies at least $100 in lost profits, possibly a lot more if she would have bought a $500 genuine handbag instead.

It's interesting to see that there are enough chinese people working in the US that they are posting lots of stuff here defending at least some of their government's actions. I don't blame them, but its something to keep in mind when discussing what levels of immigration we as Americans feel is in our best interest.

You seem to post a lot of comments about China. I appologize if you aren't actually Chinese, I've just noticed a lot of Chinese people advocating for the interests of China on here lately.

<comment><AUTHOR> > Yes. A $100 counterfeit handbag costs American companies at least $100 in > lost profits

What do you think is the reason the economist sounds so dismissive of that idea?

<comment><USER2> Great question.

>"Displaced"? So let me get this right. If my wife buys a Coach handbag for $100, and it's a counterfeit from China, and she enjoys the handbag, then the "cost" to America is $100? I'm guessing that no actual economists participated in the writing of this government report.

This "economist" (your words) seems to think that the value his hypothetical wife gets from the handbag she or others thinks is a real Gucci offsets the fact that Gucci spent a fortune on their brand and China gets to take a big fat bite out of Gucci's sales because the US isnt willing to stand up to China on trade.

<comment><AUTHOR> lmao that ain't his reasoning muh brah. His point is about target demographics. The typical person who buys a knock-off -- is s/he such that, had the knock-off been unavailable, s/he would have purchased the real thing instead. Only if you answer to that question is yes will you assess the cost to be $100.

minor: 1. Coach (US), not Gucci (Italy). 2. The author is a legit economist (expert on monetary policy), content from his blog has been posted on HN regularly. He recently posted a series of comments on China. If you're interested check them out at:

[https://www.econlib.org/parmesan-cheese-and-sunbucks- coffee/](https://www.econlib.org/parmesan-cheese-and-sunbucks-coffee/)

[https://www.econlib.org/how-should-we-think-about-the- theft-...](https://www.econlib.org/how-should-we-think-about-the-theft-of- intellectual-property/)

[http://www.themoneyillusion.com/category/china/](http://www.themoneyillusion.com/category/china/)

<comment><USER3> > lmao that ain't his reasoning muh brah

Please don't do this here, and please don't use HN primarily for political or ideological or national battle, as described elsewhere.

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)</comment></comment></comment></comment></comment>


Facebook Friend Request Bug? - <AUTHOR>

Is anyone else getting this?<p>People on facebook who I've sent a Friend Request to (and have not accepted), are now appearing on my feed. Further inspection shows I can actually view their wall and photos. <comment><USER2> Yep, getting that. Was very surprised to click through onto their page and see 'Friend request sent' at the top. I suspect I'm probably just seeing items they've set to show to 'Everyone' though. Quick way of testing: log out or create a new Facebook account, navigate to their profile, and see what you can see.

</comment><comment><USER3> Their privacy settings are such that their Wall posts and photos are viewable to Everyone. Making a friend request basically results in a "follow" a la Twitter situation.

</comment><comment><USER4> I've had that as well. Always thought it was a bug. A lot of University students use it to stalk folks over here, which is kind of depressing..</comment>


Freelancer keeps degrading the coder experience - <AUTHOR> In the last few months Freelancer changed their modus operandi by trying to screw their coders in more than way:<p>* They bumped up the price for the standard membership from about $25 to $50 without a serious reason. They simply take advantage of the fact that the coders can't retaliate/protest in any way.<p>* They hide the buyers history (a freelancer can see the history of a buyer only after he bids and the buyer sends him a message). Given the fact that you have a limited number of bids per type of membership this is a really nasty move.<p>* The entire site is a huge money making machine with no concern whatsoever about coder's experience. All they care about is to make more money no matter how crappy the experience is for the coders. For e.g. they ask you money for cheating their own system (if you pay $2 your bid will be better placed than a bid of a coder with a better rating/history).<p>* They silently tolerate spam projects where buyers ask basically for free work (or they give no information about what they actually need until you place your bid). The idea is that Freelancer will get their cut no matter how a project ends (they charge both the buyer and the coder some fees). They don't actually care if a project is finished or not as long as they get their cut.<p>* Last but not least, they offer no financial protection for both buyers/coders. <comment><USER2> Freelancer does much, much worse things for your business _by dint of using Freelancer_ than any of the problems you have written about above.

You're tolerating a 50%+++ discount to your easily achievable potential rate _every day_. Next to that, $25 is nothing.

You're optimizing your business to churn out $X00 projects for clients who are, out of the universe of consulting clients, pretty bad. Your projects are, structurally, not going to leave you with any impressive accomplishments which you could take to other potential clients to justify your newer, higher rates. You're overwhelmingly going to be stuck doing the most boring work possible with your skill set.

You're tolerating a higher than average amount of "patholgoical customers" in your client pool -- people who have no business managing a software development project, with unreasonable expectations, poor managerial skills, and the ability to stiff you for virtually any reason.

You should start prospecting for clients yourself. You'll have a minor hit in billing efficiency while learning how to do this, but then be able to start selling days/weeks at a time of availability at _actual professional rates_ rather than competing with people charging $10 to $20 an hour. Turn some of those clients into referrals and some into recurring engagements / retainers / etc, and your business will be much, much more stable, more lucrative, and less stressful.

<comment><USER3> Where are you supposed to find clients if you're just starting out, and have no reputation?

<comment><USER2> It's not like business owners will stop talking to you if they find out your Klout score. Can you identify a business in your town which has previously purchased software development services? If not, go to a meetup, ask the people selling software development services who they have previously worked for.

An alternative: to a first approximation, any business with both millions in revenue and which pays for any professional services can also purchase software development services. Learn to estimate the revenue of a business from the outside ( _great_ life skill for consulting). One simple heuristic is "White collar employee count times $200k." Can you identify a business in your town which has 5+ white collar employees? Great.

Let's say you have a financial adviser firm in your neighborhood which has 8 white collar employees. Can you describe how a financial adviser makes money? If this isn't obvious to you from a lifetime of reading the Wall Street Journal, the answer is Googleable.

Find the guy at the firm who would have the authority to hire you. (At a firm with 8 people, "owner" is a fairly strong bet. You can identify him pretty easily, since he's plastered all over their website.) Talk to him. At a financial adviser firm, since his primary job is going to be sales, talking to him is _really easy_. You call them up and say "Hiya, I'm a software developer living in town and would like to speak to Bob. I read about him on the Internet and want to talk about financial advising." I predict a 98% success rate at successfully speaking to Bob.

You then talk to Bob like two business owners who each can learn something from the other. You can learn a lot about Bob about the financial advising industry. Bob can learn a lot from you about e.g. wild and whacky things computers can do these days, like email people. After you have learned from Bob how his firm works, you say "That's great, but there exist ways to make that even better with a bit of elbow grease. Let me outline how I'd get you more rich clients, directly increasing your revenue. It gets technical but the brief description is [pitch project here]. Interested in talking about this further?"

<comment><USER4>
      You then talk to Bob like two business owners
      who each can learn something from the other.

I realize this is largely my inexperience talking, but how does one do that? How do I make that not feel like I'm wasting their time? How do I signal (without losing their interest) that I'm (a) not a customer, and (b) interested in understanding their business in a way that isn't meant to compete with him?

This is probably something that comes with experience, but I fear like initial forays would be flaming piles of failure.

<comment><USER5> I think you will be surprised how willing people are to talk to you about the problems they have, if they think you might be able to help or offer advice.

Here are a few things that I've found work well:

1\. Call them up, don't email. But always start by asking if it's a good time and offering to set up better time to call if they're busy. The call is important because people ignore emails and since it's known to be "low cost" to send them it indicates that they're probably being generally targeted instead of being called specifically. But being polite and offering to call at another time indicates that you respect their time.

2\. Give them a quick idea of who you are and why you're calling them specifically. How did you find them? Why does what they do interest you? Tell them that you're interested in learning more about what they do and what kinds of frustrations they have with their technology. People like it when you are interested in what they do.

3\. Instead of telling them what you want to do, ask them what they would like to have someone do. People love to complain about their problems, and it's good information for you to know. Knowing your potential client's problems will make you more conversant when talking to future possible clients. It's okay for you to not be able to address their problem, just take extensive notes on anything they say, and instead of trying to solve their issue on the phone or set up a contract right away, just ask questions. "Are there any other things you find frustrating?" is a totally legitimate question, and has given me usually much more useful information than specific stuff.

4\. If they don't seem to be interested, or don't feel they themselves have interest in being involved, ask them if they have any ideas of who might be more in need of the kind of thing you can offer, or if they know anyone else who might be able to offer advice on what kind of services you could offer.

5\. Ask them if there are any questions you should have asked but didn't. "Is there anything else that you think I should be asking about but didn't?" is a totally normal and reasonable question.

After you call your first five possible clients, compile that information, research the things they mentioned, and use that to refocus the way you present what you might be able to help with.

<comment><USER6> For those who actually tried it, is cold-calling any effective? Because if someone I don't know called me up and tried to sell me "solutions", I wouldn't bother giving him the time of day.

Do other people respond differently?

<comment><USER7> It very much depends - for everyone, I suspect including you, although I could be wrong - on what solution the caller is offering.

There are a truly vast number of "solutions" which people might try to offer me that I'm just not interested in.

However, if someone was to call up and offer me a production-grade finger motion capture solution for under $1000, they'd definitely have my attention. Ditto a procedural solution for creating interiors in a fantasy or medieval setting for use with path tracing renderers. And I could reel off another half-dozen ideas.

That's why Patio11's "talk to them, find out what their problems are" approach is so darn effective - it narrows down to the solutions that will actually capture your interest, because they're solutions to problems you actually have.

(Re finger capture - Yes, I'm aware of the Leap Motion and Control VR. Both are interesting, Control VR slightly more so, but neither are quite production-grade yet.)

</comment></comment></comment></comment></comment></comment></comment><comment><USER8> I had a bad experience with Freelancer a few years ago. I deposited $250 for a small freelance work. As I found no interesting or potential candidates, I cancelled the project.

They asked me for my ID, and other details so they can refund the money. It took months of back and forth with their support team and then they decided that they are going to hold this indefinitely as unresolved.

Never got my money back. They are indeed a SCAM.

<comment><USER9> Should've done a chargeback. You tried to things the right way through them first, they essentially refused by not getting it done, so you're entitled to ask your card provider to get it back for you.

<comment><USER10> So many people are not aware of this option. I've done it 2-3 times in my life, generally because a company thinks they're going to jerk me around. One was an ACH that came from nowhere (still no idea how that happened, and the bank wasn't much help).

</comment></comment></comment><comment><USER11> The first simple answer here is to use a different platform. At one time that meant you could use Odesk and Elance as well, but now they are operating under the same umbrella. There are a lot of other platforms which don't have the same recognition, look around.

The second simple answer is that you don't have to use these platforms at all. Ultimately, anything which places you in a commodity / human cloud type of market is something you should avoid. If you can't avoid it now, then work hard to get away from it. Don't get too reliant on these sites. Keep trying new things to create new opportunities.

Aside from that, I'm not sure how much you really need of the features that Freelancer and the alternatives add. Outside the platform, you don't get things like buyer history or financial protection. Whatever fees Freelancer can charge is probably low considering they are providing leads which could turn out to be valuable over the lifetime of your relationship with the client.

You shouldn't need to rely on buyer history and filters against "spam" projects. You should be able to pick this up from the interactions you have with the client. Is the description a one paragraph mess full of misspellings? Does the description look like it was written by a professional? Does the listing really look like it has potential or are you just being overly optimistic?

Are you looking in an area which is full of trash? Are there other, more niche areas which have a better ratio of high quality postings to trash?

You need to be trying lots of things. You need to be spending time every day networking and marketing. There are lots of people here who have said these things far better than I could. Look through past HN posts on freelancing to pick up tips. This site is a gold mine of information.

ETA: For buyer protection, it's best to get paid as you go. Get a payment up- front. Get payments weekly. Make sure that at any given time, if the client flakes out, you aren't out much money.

Freelancing is a business. Like any business, there is a steep learning curve. There is a ton to learn and a lot of mistakes to be made. The effort is so large, that for many freelancers it doesn't even make sense to be doing this.

Spend a substantial amount of time researching info from others so that you don't have to make as many mistakes. Even then, you are going to make a lot of mistakes and you will need to try a lot of things to find out what works for you. All the issues you mentioned here are part of that. You just have to keep banging away and trying different approaches to find something which works.

</comment><comment><USER12> Check [http://www.guru.com/](http://www.guru.com/) , they take a percentage (10% for free, 5% with a membership), they show full feedback and payout histories, have a decent escrow system, and they only take a cut when you get paid.

<comment><USER13> equally BAD as freelancer.com and any other such sites. The inherent flaw is two folds: newphew-makes-websites crowd and $1-equals-$100--in-my-country.

The escrow system doesn't protect the coder at all, it's meant to protect the person paying the commission.

In order to apply to more projects you have to buy premium membership! You have to pay to work for peanuts!

<comment><USER12> I disagree. There are a lot of projects on there that have unrealistic expectations, but one tends to ignore those projects. It's a side effect of allowing employers to post for free. There certainly are projects that have employers who are willing to pay a reasonable amount for a quality project; they're unfortunately not the majority, so a lot of a freelancer's time goes into picking and choosing where to bid. Such is life.

The escrow system has protected me more than once. It benefits both sides of the equation. I even got help from them (on a one-time basis) when an employer went radio silent after I submitted a non-escrow invoice.

By the way, if it takes you more than the 10 free bids to land a single project, you may not be putting enough effort into your proposals, or you may be bidding on toxic projects.

</comment></comment></comment><comment><USER14> Just my 2cents, but I suspect the reason they front load so many charges is that a lot of people both employers/freelancers use freelancer for the initial connection and then move to paypal or similar for ongoing work.

Freelancer lets you establish trust between parties so to speak and then after that there isn't that much of an incentive to carry on using it if you're dealing with the same person.

<comment><AUTHOR> Actually, most buyers are not interested to pay you directly (Paypal or not) because they feel _safer_ on Freelancer.

Also, when a buyer asks you to go on Skype to discuss his project most of the time he will try to scam you into giving your work without being paid.

<comment><USER14> I talk as someone who's been there and done it. Perhaps I'm just picky with who I work for, but typically I do the initial job and then move to paypal, I've not had anyone not pay me yet for my work.

</comment></comment></comment><comment><USER15> Hello mmisu

Freelancer CEO here. I thought I'd address some of the issues you brought up.

We did increase the membership plan to $50/mo from $29/mo simply because the volume of projects have skyrocketed and as a result the volume of jobs performed by standard members had likewise increased. The $50/mo standard membership give a 50% discount on commissions to 5% from 10%, so there is tremendous value there. We have lots of users earning six figures (up to almost $1m per annum), so these memberships are quite reasonably priced for the benefits.

We removed the buyers name but not the feedback. We did that after an overwhelming number of requests from employers to take it off. The problem has been that freelancers have been spamming and harassing employers by guessing their contact details (e.g. trying similar skype names).

Of course we care about the coder's experience and we actively solicit ways to improve the site from our users. Your comment of "if you pay $2 your bid will be better placed than a bid of a coder with a better rating/history" is exactly how Google and Facebook works in terms of a small number of paid listings above organic results. We order freelancers from the best to the worst feedback in the organic results in the bid list. We have three slots available for paid sponsored bids above this. This gives people an opportunity who may be new but really want to stand out in the bid list a chance to break in to the market. We think this is a very fair way of doing things.

We don't tolerate spam projects for free work, how would that make sense? Our primary business model is to charge a commission for introductions- after both parties have talked, discussed the project details, pricing and a prepayment (milestone payment) has been made, not before.

We do offer financing protection for buyers and sellers through our milestone payment system, dispute resolution and arbitration systems. Your comment that we offer "no financial protection" is incorrect.

Regards Matt

</comment><comment><USER16> Lets not limit these to coders -- plenty of designers, writers, and others in industries that are seemingly far harder to survive in than this one rely on this site (and similar) too.

<comment><AUTHOR> True, I've seen a few writer jobs (projects) where the buyers expect to pay as low as $2 per article ... Looks worse than the coders situation were, if you have a good reputation, you can at least be paid a decent wage.

</comment></comment><comment><USER17> Stop using it.

<comment><AUTHOR> Your advice, while logical, has a small problem. A coder usually has a history/reputation on this kind of sites. Once you stop using it you start basically from zero. The reputation is not transferable to similar websites.

<comment><USER18> Tying your entire professional reputation to a single website is an unbelievably bad idea. Not only does it let companies get away with what Freelance et al are doing, but it keeps you making artificially low rates. Contribute to open source, network in your community and online, and do pro bono work for charities. If you're good, your reputation will rise and your business will not have a single point of failure.

<comment><USER2> _If you 're good, your reputation will rise_

Importantly: it is both not true that your reputation will necessarily be improved by working for charities/contributing to OSS, and not necessary that one has "a reputation" to successfully get consulting gigs.

To get a consulting gig, you need to a) identify a person who has the authority to say yes to a consulting gig, b) convince them to say yes. _Even if you have a reputation_ , you don't get to skip these steps. Reputation as being among the top in the field will get you more inbound leads, but you'll still have to sift down to people who can actually say yes to gigs, and then sell them.

How do you think your average accountant sells accounting services? By having a reputation as being one of the best accountants available for hire? No. They meet with business owners, winnow down to the ones who have money and problems with it, and then say that they'll trade solutions to problems for money.

</comment></comment></comment></comment><comment><USER19> OK... so go make a better Freelancer. We're uniquely gifted in that if we don't like or agree with something, we can go and do different.

<comment><AUTHOR> There are similar (some of them better) alternatives, the problem is that once you have a good history/reputation on one of these sites you can't transfer this to another one. You start from zero.

<comment><USER20> any suggestions as to which are better?

<comment><AUTHOR> vWorker (former Rent a Coder) was a lot better.

As of today, oDesk looks better, but I don't have a lot of experience with them.

<comment><USER6> I kinda dig oDesk. I got many decent clients off it. I was shooting for small, very short term projects - the kind that can be done within a day - so YMMV.

This very site's monthly freelancer thread is also worth checking out. I got a few decent leads out of it.

</comment></comment></comment></comment></comment><comment><USER21> This needs disrupting, in the way that StackOverflow disrupted expertsexchange.

</comment><comment><USER22> You can try out the software development side of TopCoder [1] which can be thought of as competitive freelancing.

[1] [http://www.topcoder.com/](http://www.topcoder.com/)

</comment><comment><USER23> How about creating a system which is fair for both clients as well as freelancers? May be one of us here can come up with a website which is better then freelancer and other such alternatives. Few things which I am not happy with these kind of sites are they are way too complex. When I joined freelancer (formerly know as getafreelancer) it was very simple to use and had only reviews and ratings. Things were simple yet worked well.

</comment><comment><USER24> Exactly great! Labour service or intellectual service inherently can not be bid, it will lead the price to much lower than the value, finally the market is fulfilled with poor service. Even if you as a coder really want to provide good services, but the market will hit you to the reality. So, it definitely is a very bad business model, nobody can get value added from the market.

</comment><comment><USER25> I would give Toptal a try. It seems like they are trying to establish a solid relationship between companies and freelancers and they care about both sides. They also make sure to properly screen all applicants before they allow them on the platform, which includes a Skype interview and a not so easy programming test.

*not affiliated with them in any way

<comment><USER26> Toptal is US only, right?

<comment><USER27> No, I know there are quote a few Croatian coders that work for them.

<comment><USER28> You stand correct. From toptal FAQ[1]:

 _Where are your developers located?

While we don't have any geographic preference, most of our engineers are physically located in North/Central/South America and Europe._

1\. [http://www.toptal.com/faq#where_are_your_developers_located](http://www.toptal.com/faq#where_are_your_developers_located)

</comment></comment></comment></comment><comment><USER29> The easiest way to render these problems irrelevant to you is to not support the product, either for providing work or buying it.

Freelancer and sites like it are is designed to provide the lowest common denominator in skills at the lowest possible price. It's a rat-race to feed on one's own body to prevent starving to death.

</comment><comment><USER30> No doubt Freelancer is pretty sucky, but: "The entire site is a huge money making machine", that's kind of the point. It's there to make money for Freelancer.

</comment><comment><USER31> Not to mention the constant flow of emails they send you everyday. I had to add a rule just to keep them from taking over my inbox! Some may consider that spam!

</comment><comment><USER32> freelancer are ruling the domain, because they had early days of advantage, i think all the freelancers should dump that moronic sites and move on to something betters. odesk or elance or some other.

</comment><comment><USER13> freelancer essentially caters to people who say "my nephew makes websites for free, why can't you?"

freelancer is full of people who can afford to work for far less than what is considered minimum wage in North America and Western Europe.

freelancer cares only about people who give them projects so they can take a cut. charging people to work was not enough.

it's time someone came up with a new freelance site. \- make it only accessible in North America and Western Europe. (the commodity prices work in other parts of hte world but not where the basic living costs are far higher) \- Guaranteed minimum hourly wage. \- rigorously filter projects being posted (no more build me an Android app for $5, "my newphew can do it bro why not you?") \- No bidding or fixed priced contracts. Hourly rate projects only.

<comment><USER33> >make it only accessible in North America and Western Europe

What for? Sure I understand that it is good for coders in North America and Western Europe because it protects them from the competition, but why should the consumers care? Although there is a negative stereotype about South Asian coders, I don't see a reason why a coder from outside of the developed word should be inherently low-skilled. My impression is that the main body of coders from South America and Eastern Europe, at least those who can speak English, are just as good as the main body of North American or Western European coders. Then there are a lot of really good people from South Asia, it's unfair and senseless to exclude them from this market. I am pretty sure any sane person would have problems with this kind of racial profiling even if there was no other way to establish whether or not given person is fit for the industry, which is not even the case.

Well, exclude regional profiling and basically you have invented toptal.</comment></comment>


Why 30 is the decade friends disappear – and what to do about it - <AUTHOR> https://www.vox.com/2016/7/12/12148938/friendship-adult-challenges-solutions <comment><USER2> TL;DR: kids and work. One would assume the solution is encouraging your kids to be more independent and pursuing work-life balance over increases in pay.</comment>


Ask HN: How desired is working at Apple? - <AUTHOR>

I'm a CS student looking for full-time employment (as a software developer) for after I graduate in May.
I'm strongly considering Apple. A couple of pieces have shown up over the past few weeks about the culture of privacy at Apple, and that's left me a little concerned. When people talk about large companies ideal for developers in the valley, I usually hear Google/Facebook/Twitter, and I'm wondering why I don't usually hear Apple on that list.
I'd really like to get a sense of some of the following from current/former employees or just people in Silicon Valley that have heard stories:
- is Apple desired more or less than Google/Facebook/Twitter?
- how selective is Apple compared to Google/Facebook/Twitter? i.e., is getting a job at Apple considered a big deal in the tech industry, or do people consider Google harder to get into? <comment><USER2> "is Apple desired more or less than Google/Facebook/Twitter?"

I think for a developer it is less simply because developers are not willing to deal with the perceived island culture. Developers are all about practicality and getting things done. Apple is all about style, design, and one button mice. Imagine stepping through the door on day one and being handed a black turtleneck, one button mouse, and being shown to your uncomfortable work futon, and btw, mandatory meditation hour is every day a 8, 10, and 3. That's probably not realistic, but that is the perception I have. But from a career perspective a job at any successful place like apple is going to be golden, it just may not fit every person's needs.

"how selective is Apple"

I think all the places you mention are probably going to be very selective, accepting anywhere from 0.5-5%.

</comment><comment><USER3> This is speculation but I assume that some of the following factors play a huge role in folks choosing which company to work for:

Pay, stocks, flexibility, benefits, etc...

I'd imagine Apple is probably the least appealing in many categories when compared to Google or Facebook even if they are one of the most awesome/innovative companies out there.

Then there are other factors to consider for some people such as network (folks they'll be connected to), environment, learning experience, etc...

I assume these also appear to be better at other companies.

Of course whether or not any of the above is better or comparable is a different story but it would seem if those are the things important to folks, Apple would be slightly lower on the list.</comment>


Flash, HTML, Ajax: Which will win the Web app war? - <AUTHOR> http://news.cnet.com/8301-17939_109-10011048-2.html?part=rss&tag=feed&subj=Webware <comment><USER2> I find a good ajax app to be far more usable than the other two.

Every time I start using Flash or Flex I run into annoying issues. The last one was that Firefox has a bug where it says "transferring data from yoursite.com" forever if your flash object does any network activity. Even after the network activity stops. The status bar doesn't return to normal again until a fresh page w/o flash network activity is loaded.

I have used a few 100% flash/flex apps and they are pretty annoying.

Even websites or blogs that load multiple youtube windows tend to crash Firefox way too frequently.

So I'm rooting for html/ajax.</comment>