How do I delete bytes from the beginning of a file? (2010) - <AUTHOR> https://devblogs.microsoft.com/oldnewthing/20101201-00/?p=12153 <comment><USER2> I swear that everytime someone posts a generic Microsoft blog entry, I automatically think: "oldnewthing".

I hovered over the link, and there it was.

Raymond Chen is a machine, the T-1000 of Windows development. I've never developed Windows applications using C++ and I still read his blog articles.

</comment><comment><USER3> Reading this, I couldn't help but try to find the rhythm of a poem to match the formatting.

<comment><USER4> I guess Raymond uses a line oriented editor like vim, and the original line breaks have been preserved. If you use one line per clause, it's much easier to change sentence and paragraph structures without having to seek around by the word.

<comment><USER2> Vim's not line oriented... at least as far I know. ed/ex, now those are line oriented.

But I wouldn't call vim line oriented. Unless you also want to call Notepad line oriented, since you can use Shift + End to do operations on an entire line :D

</comment></comment></comment><comment><USER5> I've often thought it would be useful (especially for software that manages container or archive file-formats, or for databases) to have exposed to the user a file-system object that operates _like_ a file, but which—instead of being exposed by OS as a seekable byte stream that can be appended/truncated on one end—is exposed by syscalls as a _vector_ of arbitrary, non-uniformly- sized extents, where the user is expected to ask the OS to pre-allocate extent buffers (think mmap(2) with anonymous private disk pages), and then, separately, stuff those into this vector object (which would be an operation almost exactly like hard-linking an existing file into a directory, in terms of its time and space complexity.)

Of course, for most filesystems (and especially ones with sparse-file support, and _especially_ ones with copy-on-write support), such an on-disk data structure is exactly what's already underlying the byte-stream abstraction. So this would just be a passthrough to allow people to directly manipulate that data structure. (In the process probably breaking certain preconditions the filesystem relies on, though, so it would need to track these "low-level extent vectors" as a separate filesystem object type. Programs would still be able to use regular file-abstraction syscalls on them, though.)

Interestingly, despite filesystems themselves not exposing the lower-level extent-vector abstractions, in some other systems that have file-stream-like abstractions, you _can_ operate on "files" this way.

Postgres's BLOBs, for example, are seekable byte-streams, which also (at least theoretically) allow you to insert into the middle of them. (I say theoretically because it's not an implemented API, but it's not exactly hidden from the user, either. BLOBs just get broken out into records in a table representing their extents; you can rewrite the keys of said records in that table to do whatever low-level operations you like.)

Or, for another example, S3 and its competitors let you arbitrarily compose objects as "components" of other virtual objects, which then read back as the concatenation of the objects they contain. (Sadly you don't get any other vector-manipulation ops than this, but if you keep references to the leaf extents around, that's often enough to rebuild your extent-vectors any time they change.)

Also, of course, back on the real filesystem, you can just manage your "vector of parts of a file" as multiple files in a directory, and then abstract over that directory that by using a FUSE server which exposes a view where the files are one contiguous file.

<comment><USER6> This, and more, has been tried on mainframes. When you could not afford to install a free RDBMS with a click of a mouse (the mouse being not available yet, too), you could really appreciate record-oriented files you could use as a database, and even have indexed access to the records.

<comment><USER7> Has anyone written a high-level introduction to mainframe tech that's accessible to people outside the mainframe culture? I've always been curious about it, but the IBM manuals I've come across have been somewhat impenetrable.

<comment><USER8> How about:

'Introduction to the New Mainframe: z/OS Basics' [http://www.redbooks.ibm.com/abstracts/sg246366.html?Open](http://www.redbooks.ibm.com/abstracts/sg246366.html?Open)

This, and the 'ABCs of Systems Programming' series were pretty good I thought.

</comment></comment></comment></comment><comment><USER9> Another approach I can think of is using rolling hash ([https://en.wikipedia.org/wiki/Rolling_hash](https://en.wikipedia.org/wiki/Rolling_hash)) to split file to different size of chunks, those chunks can be saved on different sectors which don't need to be continuous. The file keeps a list of indexes for all those chunks, just like inode did for blocks. When insert some bytes in the beginning of file, thanks for the rolling hash, most likely only the first several chunks need to be re-chunk and re-hash, thus the change is localised and can be cheaply done, all the rest chunks still untouched. When this combined with append-only storage, things are even easier because it only needs to deal with bytes around that beginning, and the update chunk index list in file.

This approach is already implemented in ZboxFS ([https://github.com/zboxfs/zbox](https://github.com/zboxfs/zbox)) to do content-based deduplication.

</comment><comment><USER10> I didn't get the "minus 100 points" reference, but it was not hard to find it on the web: [https://blogs.msdn.microsoft.com/ericgu/2004/01/12/minus-100...](https://blogs.msdn.microsoft.com/ericgu/2004/01/12/minus-100-points/)

</comment><comment><USER11> Why would you want to delete bytes from the beginning of a file? A file should be seen as a low-level data structure with a limited set of operations. If you want to implement a more complex data structure with a wider range of operations, there are usually multiple ways of implementing this on a low- level data structure with a limited set of operations. This is basically what a lot of software engineering is about. If you want to implement an array of bytes with delete and insert operations, there are many ways you could implement this on a low-level file. Which way is the best depends on so many other factors. Do you want undo-redo functionality, transaction properties, distributed access, and so on?

<comment><USER12> Because sometimes I want to remove old and irrelevant lines in a logfile but want to retain the recent lines.

> A file should be seen as a low-level data structure with a limited set of > operations.

Why should? For many people, a file is a basic way to permanently store data. Changing the data at any point of the file should be easy.

<comment><USER11> Actually, most file systems let you randomly read and write to files. Usually, they implement this by reading and writing blocks (of 256 bytes or a multiple). One could create file systems with more advanced operations, but most likely, these will be based on the more primitive operations. For example, to add the function to remove 10 bytes from the start of the file, one could read the whole file, block-by-block, and move the data 10 bytes. For a large log file, this will take a long time. Another solution is to have a begin of file offset for all files, this means that some additional information need to be stored with every the file and that this information needs to be accessed with all file operations, resulting in some performance loss for all file operations. Existing programs that access files as blocks of a certain size, might see significant performance penalties when the offset is not a multiple of the block size, because reading the data from one block, needs to read two actual blocks from the underlying medium.

If you often want to strip a logfile from old data, a solution might be to create new logfiles at a regular intervals and delete old logfiles. Or maybe even better, use a database to store the log messages. Gives you query functionality and allows you to perform more advanced operations, as removing certain types of log messages.

</comment></comment></comment><comment><USER13> See fallocate() for Linux/ext4. [http://man7.org/linux/man- pages/man2/fallocate.2.html](http://man7.org/linux/man- pages/man2/fallocate.2.html)

<comment><USER14> It looks like this will zero out data but not remove.

<comment><USER15> It will remove if you use FALLOC_FL_COLLAPSE_RANGE

<comment><USER14> But: " A filesystem may place limitations on the granularity of the operation, in order to ensure efficient implementation. Typically, offset and len must be a multiple of the filesystem logical block size, which varies according to the filesystem type and configuration. If a filesystem has such a requirement, fallocate() fails with the error EINVAL if this requirement is violated."

Seems it's limited.

<comment><USER15> Correct, my bad. Did some tests and AFAICT neither EXT4, XFS nor BTRFS allows arbitrary byte ranges here.

</comment></comment></comment></comment></comment><comment><USER16> Theoretically, if the number of bytes to be removed were a multiple of FS data block size (usually an inode size), it should be possible to unlink that inode as well, and rewrite the metadata. It probably involves fewer shifts and fewer bytes/time wasted.

In general, if you intend to create structures that allow one to reduce file size, reclaim the data, defragment - that sort of thing, a rewrite/coalesce happens to be the viable solution. In general, if allowed to increase, fragmentation will worsen with time.

</comment><comment><USER17> I'm not sure I really understood this. If you wanted to delete 10 bytes, couldn't you just read a sector, then copy the sector back sector[10:]?

Edit: then delete the last ten bytes.

<comment><USER14> Does the whole sector thing still make sense for SSDs? Do they still work by sector or can every byte be addressed directly?

<comment><USER18> The "native" sector size of an SSD is the erasure block size, typically 64KiB.

<comment><USER19> I'm fairly sure SSD erase block sizes are more like 2 MB or more. Even 8 MB wouldn't surprise me anymore.

</comment></comment></comment></comment><comment><USER20> To delete from the beginning of a file, overwrite offset 0 from offset n, 1 from n+1 and so on, and then truncate it when the read offset hits EOF.

</comment><comment><USER21> dunno about windows, but in llinux its fairly simple. I've had to do do this a lot of this sort of thing on big disk images my friend made with some weird program that pads the start of everything with 512 bytes of some random crap.

dd skip=10 iflag=skip_bytes if=original.file of=first10ktrimmed.file

despite being old, and the amount of insanely horrible things you can do when you mess up the syntax or mistype something using it, dd is fucking amazing.

to be fair, its not directly editing the file, but making a copy with the first 10k skipped.

you could have a jump/pointer that points to the shortened first block of the file and then at the end of that, a pointer/jump to pick up where the rest of the file continues, but you'd wind up with misalignment and fs fragmentation.

<comment><USER22> Linux doesn't implement sparse files.

<comment><USER23> Yes it does? The details are file system specific and if they support mmap then it's also likely there's a page alignment requirement but that's not atypical.

</comment></comment></comment><comment><USER24> Is this a problem that really needs a solution?

I can't recall any times where this would have been useful to me or think of a situation where it'd be a deal breaker. I could contrive a bunch of other operations that are inefficient on file systems like this (remove every other byte from the file, why not?), but doesn't mean that literally any idea we can think up should be supported by the file system apis.

<comment><USER25> One place this would be useful is a DVR. You want to record the last _n_ hours of video and discard anything older. And it's not inefficient. It just requires a lot of careful engineering that's probably not worth the bother.

<comment><USER26> It's unnecessary to create a filesystem-level support I think, since you could make a circular buffer holding a video stream in a file with some constant size that is approximately n hours of video. With some clever tricks you could even keep the format compatible with existing video container formats.

<comment><USER27> Or just use a set of files, each holding N minutes with, perhaps, some overlap + an index file. No real need to over-engineer things here.

</comment></comment></comment></comment><comment><USER28> The filesystem just needs a concept of a file start offset.

The initial value is 0, if you wanted to support truncating at the head, you just add to this offset the amount truncated.

In a sector-based filesystem, whenever this offset goes far enough into the file that crosses sector boundaries, you reclaim those sectors as free space. When it lands within a sector somewhere, that sector is pinned and you waste some space.

The problem is more that the userspace APIs don't expose well-supported mechanisms for doing this. Implementing it at the filesystem level is trivial.

<comment><USER29> I can think of a cool case for this.

What if you have an append-only system which is making remote backups. One service is writing the activity log append only, a second service is reading that, check-pointing, and then truncating the head of the file when the check- point has been committed. No need to do the tricky file-swap trickery.

<comment><USER30> Linux and fallocate() allows precisely this - we use this to "log locally" for applications, and a "tail this asap and publish to kafka then delete locally" helper.

See ( [https://gist.github.com/minaguib/1cbe29922b06d50755a2f580b8c...](https://gist.github.com/minaguib/1cbe29922b06d50755a2f580b8c343fa) ) for some test notes I took a couple of years ago.

<comment><USER31> fallocate() is not supported on network file systems, is it?</comment></comment></comment></comment>


Ask HN: What can be done about freelance sites? - <AUTHOR> Hi!
What in the existing freelance sites seem wrong to you, a problem needing solving which is a market opportunity?
Now in general, this market seem to be built in a way that prevents people's progress at every stage: starting out, growing as a professional, starting as a business, growing as a business.
At the begining we have a chicken and egg problem when nobody wants to hire you even if you are brilliant, until you have a stellar profile with a lot of experience - and lowering price doesn't really work, because the lower your price the higher the competition.
When the freelancer is past that stage, he has a problem of low rates in his history which makes him look like cheap labor, rising rates is very difficult.
And finally - it is difficulty to grow beyond an individual freelancer: many clients, for subjective reasons, don't want a company/agency, even those who definitely need one.
On the customer's side, it is a problem of fakes, and false impression of very low rates, which arise from a large number of profiles that bid with super low bids, but never get a job.
So, no surprise first advice to freelancers is usually 'avoid freelance sites'!
I fully understand that this is very hard since it is a two-sided market: to get freelancers you need clients and vice versa. But i think it is possible to make something to disrupt this market.
That is the biggest problem here? Anything you can think of on how to improve the market?
Thanks for inputs! <comment><USER2> Freelancer sites seem to end up in a hostile 3-way fight. Customers, clients, and the middle man each one trying to screw the other two groups.

And it gets worse with scale because it turns into a race to the bottom, the good players get lost and lumped together with the bad ones you can't spot/differentiate them among the sea of crap.

First some sites like Elance/Upwork/oDesk/Freelancer came along and they abused their popularity more and more every day.

Upwork has gone completely insane in the past few years treating freelancers like dirt.

Then from there you get another generation of sites that basically capitalised on "we are the good guys trust us" marketing.

They do some higher end marketing. Often (effectively) saying "yeah those cheap Upwork guys are terrible ... just try us we are different".

They also tend to be hollow, just a shiny store front, but in principle still the same troubling incentives are at play.

</comment><comment><USER3> Sounds like opposite side of a thread I was involved in recently:

[https://news.ycombinator.com/item?id=15898460](https://news.ycombinator.com/item?id=15898460)

I really don't think there is anything that can be done just short of a verification system like what Twitter use to do. You had to scan and email your driver's licence or state ID and send it to them. So you would need 'social security number' for job sites. I guess a Facebook, Twitter or Google login might work.

</comment><comment><AUTHOR> My ideas here revolve around enabling and promoting horizontal links between freelancers. When advanced ones legally outsource to n00bs, taking full responsibility for their jobs, as if it was done by themselves, and allowing money to be split in proportion invisible to clients. That is still not a finalized idea.</comment>


We launched: massify.com - <AUTHOR> http://massify.com/competitions/machine/

<comment><USER2> I'll have you know that your flash intro music was an unwelcome addition to the 3rd movement of Mahler's 2nd.

Anyway, I looked at the site and my one impression was: Skepticism. I mean:

 _...will produce one winning entry to be a feature film in theaters. We'll also be casting the movie using the site._

I mean, I know nothing about the art of film production, but I think that casting is usually done through trade journals and auditions for a reason. Likewise, don't studios usually base ideas on more than, "That sounds good", and look into what kind of connections the filmmakers bring with them, their track records, etc?

And this:

 _You are competing for the chance to work on a professional feature film - either as an actor or a producer and originator._

Huh? Isn't the producer the guy who drives everything, puts up the dough, etc? How do you win that sort of thing in a contest?

Skepticism, that's all.

<comment><AUTHOR> Well I could give you the marketing/PR spiel, but that's not my department.

I see the competitions as being only a single aspect of the site, and we are all hoping it grows beyond a competition and more into a vehicle for getting really talented people produced and promoted. The vision for that particular component is really more of a virtual indie production company - but we all have to start somewhere and I think this is a pretty good way to start.

It's not the only thing we do though. We've got three products in total. The professional networking product is still pretty infant, but we're obviously working on making that a value for indie talent and a means for them to network and get gigs - as well as provide tools for self promotion (for instance, the headshot generator does what many actors pay $400 or more for the exact same result).

Our final product, the next to roll out, are a suite of production development/management tools for indie/web/amateur filmmakers/videographers. It allows you to take your pitches and manage them all the way through post- production.

Producing for on-line is a growing market with different needs/goals than traditional film and we're aiming to help bolster it and become a hub for it. Hopefully, we'll get there.

ps. I don't write PR, I write code. This is honestly how I feel about the entire project, otherwise I couldn't have sunk the crazy amount of hours I have into it.

pss. sorry about the music ;)

<comment><USER3> I think it's a great concept - the web and digital tools are making both distribution and production costs much cheaper. However, no one (well, not until now) has really been tackling the middlemen - producers, talent agencies, etc. that still cost way too much and create barriers to entry.

Professional networking and casting assistance (crews and actors) has a lot of potential for growth. Great work getting together something I've been thinking a lot about but haven't actually worked on yet :)

</comment></comment></comment><comment><USER4> I have no idea what you guys are doing, but your designers rock!

<comment><USER5> That's kinda oxymoronic. If their designers rock, you'd know what they are doing no?

(I had to click on the tiny tiny "about" link in the footer to find out what they do)

<comment><USER6> Maybe. You can design for aesthetics, and/or for functionality. Sometimes, "rock" doesn't mean "is good"--think about the negative opinions some people have of [the term, or the actual] "rockstar programmers." In this case, I was thinking about "rock" as in a Hendrix guitar solo--doesn't really get a point across, but doesn't really need to.

</comment></comment></comment><comment><USER7> Congratulations!!

I'm an armchair film aficionado. I went to a pitch competition featuring pre- vetted aspiring film makers last year and it was fascinating enough to make me start wondering about the whole process and if I really could write a potential screenplay.

</comment><comment><USER8> I have a 12px minimum font size set in Firefox because I have a fairly high- resolution screen. This causes the content that should be in the middle of the page to be displayed offset to the right, starting for the right border of where it should be.

<comment><AUTHOR> Thanks for the heads up. Fixed and added as a test case to our QA. Thanks again :)

</comment></comment><comment><USER9> Just a heads up: Ghosts in the Machine is used in the header, but the end of the video reads Ghost in the Machine. No 's'. Not sure which version you were going for, but you might want to read through and double check all that.

</comment><comment><AUTHOR> Most difficult project of my career, almost a year and a half from idea to launch.

 _wipes sweat from brow_

Thanks to hacker news for keeping me occupied when I should have been coding.

</comment><comment><USER10> Wow! Great site. Kudos to you guys...

</comment><comment><USER11> Congrats! :]

</comment><comment><USER12> dig that design man.

</comment><comment><AUTHOR> Thanks everyone :)</comment>


Determination of the Hubble Constant Based on the Tip of the Red Giant Branch - <AUTHOR> https://arxiv.org/abs/1907.05922 <comment><USER2> Just a few days before that paper, this[1] paper was released where they used another independent method, which is in tension with the Planck results. Apparently the paper linked in the submission lies smack in the middle, but error bars prevents any firm conclusions.

There's some interesting discussion over at Physics Forums[2], including a nice illustration of the tension between different approaches in post #16.

[1]: [https://arxiv.org/abs/1907.04869](https://arxiv.org/abs/1907.04869)

[2]: [https://www.physicsforums.com/threads/new-h0licow-result- hub...](https://www.physicsforums.com/threads/new-h0licow-result-hubble- constant-is-73-3-1-7-1-8-km-s-mpc.974648/)</comment>


Ask HN: SaaS user management platform, would you use it? - <AUTHOR> Hi HN,
I am considering if building a small SaaS to manage users.
It will be a very simple and tiny api to register, login, send confirmation email,  reset password for users and manage stripe token.
I would also like in a second step to attach something like a key value store to each user.
Something like Auth0 or Stormpath but tinier and simpler.
It is something you guys would use?
Cheers, <comment><USER2> I am ~60,000 lines of code into this project. My version is NodeJS and Redis and is modular with packages that can be added for GitHub-like organizations, subscriptions using Stripe, and platforms using a full custom integration of Stripe Connect.

My software installs with NPM and may be positioned with your NodeJS app, or your application may be served by something else and any URLs my software doesn't handle proxy your application.

The installation of my software is an NPM install and then creating a two-line JavaScript file to start the app if you're not using NodeJS already.

It is completely API-driven and the API can be used by NodeJS or over HTTP, and has administrator and user branches. The UI has complete account and administrator functionality including creating sessions to impersonate a user for fine-grained controls.

I have complete test coverage, about 2/3 of all code is unit tests that cover internal APIs, the public APIs and the pages.

I am currently finishing some improvements to the Subscriptions and Connect modules and then I will be on version 1.

I'm just about ready to launch this as an open source project, maybe you'd like to have a chat on Skype - benlowry at outlook.com?

<comment><AUTHOR> That is interesting. Although I was looking to create something really really simple and dumbed down.

<comment><USER2> Despite the mountain of code it is hopefully simple by design:

\- about 40 NPM dependencies in total when all modules are added so it installs very quickly and node_modules is manageable

\- about 150 lines of CSS

\- one Redis database, with optional encryption at rest

\- no client-side javascript (but needs a couple bits for Stripe stuff, and the groundwork is in place for it to feel like an SPA eventually)

\- no build process, just node main.js so it starts very quickly

\- no templating language to learn, just ECMAScript string literals `like ${this + that}` in vanilla HTML files ([https://github.com/userappstore/dashboard/blob/master/src/ww...](https://github.com/userappstore/dashboard/blob/master/src/www/account/sessions.html))

\- DOM-like construction of HTML in NodeJS ([https://github.com/userappstore/dashboard/blob/master/src/ww...](https://github.com/userappstore/dashboard/blob/master/src/www/account/reset- codes.js#L74))

\- continually sweeping the software and modules to apply more consistent code formatting and tests

\- careful not to include text and formatting in code in preparation for localization

\- careful not to abstract to the point you require broad familiarity with the project to understand any page or nodejs

If you want to have a look check this out:

1) This is the basic dashboard software, this part by itself is intended to be all you need to create a web or mobile app with user registrations.

[http://github.com/userappstore/dashboard](http://github.com/userappstore/dashboard)

2) This is what the application emits when you start it up, after combining the dashboard, any modules you use, and any code you added to a single sitemap:

[https://github.com/userappstore/dashboard/blob/master/sitema...](https://github.com/userappstore/dashboard/blob/master/sitemap.txt)

3) This is the test output for the dashboard

[https://github.com/userappstore/dashboard/blob/master/tests....](https://github.com/userappstore/dashboard/blob/master/tests.txt)

Then the modules extend it:

[http://github.com/userappstore/organizations](http://github.com/userappstore/organizations)

[http://github.com/userappstore/stripe- subscriptions](http://github.com/userappstore/stripe-subscriptions)

[http://github.com/userappstore/stripe- connect](http://github.com/userappstore/stripe-connect)

[http://github.com/userappstore/maxmind- geoip](http://github.com/userappstore/maxmind-geoip)

My end goal is for this software to reach a very stable and mature place, concentrating on a very small set of responsibilities and doing them very well.

</comment></comment></comment><comment><USER3> frankly, probably not. Even stormpath closed shop and dropped their api. I probably wouldn't be able to trust a small startup to not do the same..

<comment><AUTHOR> Reasonable concern, thanks for the feedback.</comment></comment>


Disasteroids: Building a real time gravity space shooter in 48 hrs (NKO2012) - <AUTHOR> http://somethingcoded.net/

<comment><USER2> Love the game. Thanks for sharing the details of the competition and building it. It makes a fun read.

</comment><comment><USER3> Looks great guys, loved your submission last year too!</comment>


A Night at the Museum with Project Tango - <AUTHOR> http://www.pcmag.com/article2/0,2817,2499839,00.asp <comment><USER2> To compare with an iBeacons deployment in a museum: [https://www.brooklynmuseum.org/community/blogosphere/2016/02...](https://www.brooklynmuseum.org/community/blogosphere/2016/02/23/getting- visibility-on-the-ibeacon-problem/?imm_mid=0e1306&cmp=em-iot-na-na- newsltr_20160303)

</comment><comment><USER3> How do I get a job at Google[x]?!</comment>


ShowHN: GetURL - A CLI tool to get a public link for any file - <AUTHOR> https://github.com/uams/geturl

<comment><AUTHOR> Something I threw together over the weekend. I've been using filepicker on a website of mine. Realized that I could hack it to upload and share files in a simple tool.

Inspired by cloudApp, when I found myself wanting a command line version.

<comment><USER2> Well done! Also you deserve a big thanks from Filepicker.io - this is great inbound marketing for them.

</comment></comment><comment><USER3> Seems like this could be done with more respect for user/data sovereignty if done with Node.js. Essentially, you'd cat the file into the script, it would spin up an HTTP server, do some port knocking or whatever NAT-traversal-fu is necessary, then spit out a link with either your raw IP, or a preconfigured dynDNS domain name.

This way we aren't carelessly littering our data all over the "cloud".

<comment><USER4> Tooting my own horn again, but you just asked for <https://pagekite.net/>:

$ pagekite.py /path/to/file.blah yourname.pagekite.me

... send people a link to <https://yourname.pagekite.me/file.blah> and it streams from your disk. CTRL+C and it's offline with no copies stored anywhere in the cloud. Works with entire folders too (append +indexes to generate indexes), which is good for static HTML demos. If you want a harder-to-guess URL, append the +hide flag to the command above.

And yes, it's open source and you can run your own relay/reverse-proxy if you don't want to rely on me. Give it a try! :-)

<comment><USER5> Be proud, your domain has made BlueCoat's list of "proxy avoidance" sites and is blocked at my workplace.

<comment><USER4> Woah, I'm in the big leagues now! Blocking us actually makes perfect sense, PageKite can be used to avoid a lot of quite reasonable corp security policies.

Can you access sites on other domains which are connected via. PageKite? For example my personal site, <http://bre.klaki.net/> or my kite <http://bre.pagekite.me/> ?

<comment><USER5> Nothing on pagekite.me, will check the other tomorrow.

<comment><USER4> This probably doesn't belong on this thread, but I would really appreciate e-mail: bre at pagekite dot net.

</comment></comment></comment></comment></comment></comment><comment><USER6> Another:

        cat file | curl -F 'sprunge=<-' sprunge.us

<comment><USER7> Does sprunge work for non-textual data?

<comment><USER8> In the same vein, you can try <http://flag.io/>:

      curl -T /path/to/any/file flag.io
      # or
      cat some-file | curl -T - flag.io

Basically, pipe anything you want into it. It can also syntax-highlight your text files.

</comment></comment></comment><comment><USER9> Nice work and very handy.

I hadn't come across filepicker.io before and reading through the geturl code something jumped out at me:

APIKEY = check_output(['curl', '--silent', "%(fpurl)s/getKey?email=%(email)s" % {'fpurl': FPAPIURL, 'email': email}])

From that, it looks like any random person can fill up your filepicker.io space providing they have your API key or know the email address you used to register the account with. Made sense when I read a bit more about what filepicker.io actually does (i.e. a client-side embeddable javascript file uploader) but it's something to be aware of (especially if you link your account up to an S3 backend!).

<comment><USER10> One of the founders, just wanted to reply and say, yes, you are entirely right. We put up that endpoint for a separate purpose, didn't expect people to find it (underestimating people is clearly a bad strategy) and will be locking it down to require a password to create/find a apikey.

In general, the apikey doesn't actually provide very much security as is; by it's public by it's very nature as you have to put it client side and expose it to all your users. We've got HMAC and secret keys in the pipeline for next week :D

<comment><USER11> Will those security changes break GetURL?

Also, isn't it normal to check the referrer when using API keys? That's what Facebook does -- API keys only work from certain domains, which effectively restricts their access. The downside is that you need to maintain separate API keys for every domain (staging, sandbox, etc), but the advantage is that they don't rely on the honor system :P

</comment></comment></comment><comment><USER12> Neat, +1 on the 'brew' addition intent :)

<comment><USER13> Please submit to brew!

<comment><AUTHOR> Submitted to brew. Not sure if I did everything right (first time)

<https://github.com/mxcl/homebrew/pull/15229>

</comment></comment></comment><comment><USER14> I have no idea why I haven't installed xsel already. The help mentions a secondary clipboard. X has a secondary clipboard? Is there a way to paste that (like middle click for the primary)?

<comment><USER15> Following the old zero-one-infinity rule[1], X has an arbitrary number of clipboards; the most commonly used are PRIMARY (select+middle click), CLIPBOARD (usually C-c/C-v or edit→copy menus) and SECONDARY which, as far as I know, can only be pasted using applications that support it (e.g. xclip -o).

[1]: [http://www.catb.org/jargon/html/Z/Zero-One-Infinity- Rule.htm...](http://www.catb.org/jargon/html/Z/Zero-One-Infinity-Rule.html)

</comment></comment><comment><USER16> Since I have web hosting and use GitHub, here's my method:

        cp file_name ~/Projects/2012/ajf.me/ajf.me/imagedump/ && cd ~/Projects/2012/ajf.me/ajf.me/ && git add imagedump && git commit -m 'new file' && cd .. && ./update.sh

Elaborate, sure, but it does the job. update.sh runs git push and then does an SSH into my server and a git pull. (Because I'm too lazy to actually set up git on my own server)

<comment><USER17> You don't need to set up a git server on your own sever. Just create a git repo, add it as a remote revenue to your local repo, and use ssh+git to push changes directly.

But if you have your own server, why don't you just scp it directly there? That's the piece most people are missing.

<comment><USER11> SCP is really the solution here. But to be fair, I only discovered scp after a year or two of CLI work. I might have been tempted to hack something together like Taze's solution in my earlier days

</comment></comment></comment><comment><USER18> This is awesome, it actually makes filepicker.io useful for me. My main use- case for Dropbox has turned out to be for this kind of sharing... geturl is waaaaaay easier.

</comment><comment><USER19> This a nice nifty script. Thanks!

Something I noticed (<https://www.filepicker.io/pricing/>): Since filepicker.io charges by number of files and NOT by total size of files, a free account could potentially host a huge amount of data.

Eg: 5000 files, each of 1 GB = 5 TB!!!

</comment><comment><USER20> It couldn't handle files with spaces, plus the link it gives initiates a download for the file instead allowing you to view it in the browser. Considering most of the time I want to quickly upload a file in this manner the file is a screenshot, this is basically useless for me.

</comment><comment><USER21> Very nice! But no Python 2.6 support?

<comment><USER22> The key problem seems to be `check_output` in the `subprocess` module.

<comment><USER5> <https://github.com/uams/geturl/issues/7>

Edit: pull request submitted.

</comment></comment></comment><comment><USER23> What about storage usage?

      > To make it easier to get started, if you haven't
      > put in your S3 credentials we will store them on
      > our servers, but as your usage increases we will
      > ask you to move to your own storage.

</comment><comment><USER24> Nice tool, I made something similar for dropbox a while back that symlinks/copies files into your DB folder.

<https://github.com/BRMatt/dotfiles/blob/master/bin/dropup>

</comment><comment><USER25> Nice! There's an opportunity for some company sponsor this (a la localtunnel.com and twilio).

<comment><USER26> Given that it uses Filepicker.io, probably makes sense for us to take a look. The idea of turning files into URLs is one that we really like

</comment></comment><comment><USER27> Inspired by geturl, I built this: <http://news.ycombinator.com/item?id=4626779>

pro: no storage backend needed con: works only behind a UPnP router (most are ok)

</comment><comment><USER28> I really liked it, so I ported it to perl implementing a basic caching system (SHA1), I hope you don't mind: <https://github.com/psychotropic/Get--Url/>

</comment><comment><USER29> I still prefer Dropbox because you have better control about what you upload.

Simply copy your files into ~/Dropbox/Public, right-click and choose Dropbox > Copy Public URL. Voilà!

<comment><USER30> Except that Dropbox has now discontinued their public folders in exchange for there get link everywhere (which requires you to click add to dropbox or a download button rather than direct access).

<comment><USER29> I'm not sure what you mean. The method I wrote about still seem to work.

<comment><USER30> All new accounts no longer have a public folder.

<comment><USER31> Yes - all that means is that now you can move a file into anywhere in your Dropbox, right-click it, and click (in the Nautilus plugin) "Share Link" and get a publicly available link.

The only thing that the Public folder did differently was that it did the equivalent of "Share Link" for every file in it, and also had a public index of files in the folder.

</comment></comment></comment></comment></comment><comment><USER32> Very awesome tool, it's a great way of sharing nice and quickly!

</comment><comment><USER33> Nice utility!

Just a note:

28: exit("`curl` is requrired. Please install it")

<comment><AUTHOR> Haha. I'll fix that when I get a chance; pull requests are encouraged :D</comment></comment>


StepWise Pedometer for iPhone now FREE - <AUTHOR> https://itunes.apple.com/app/id788761724?mt=8&uo=4&at=10lwUJ&ct=hackernews

<comment><AUTHOR> Join the may users getting fit with StepWise... Free!</comment>


Colleges are reopening in fall, but many professors won't be present - <AUTHOR> https://www.nytimes.com/2020/07/03/us/coronavirus-college-professors.html <comment><USER2> This is looking more and more like an extinction event for academia.

Elite institutions can charge whatever they want for their brand name and selectivity, and for the environment of other equally motivated high potential students.

Other schools combine a school with a resort/cruise ship experience. Insofar as the school is no better than a bunch of videos, and the resort is mostly shut down, so long as employers don't care, it will be hard to justify spending 10x on a residential experience.

With live teaching going the way of life musicians 100 years ago, we can expect similar effects- fewer better teachers, and better access to them.

<comment><USER3> I'm leaning more strongly to "major impact / selection event / paradigm shift" than extinction. Education generally, and universities specifically, are highly durable institutions which have undergone massive changes over time, whilst retaining recognizable elements from their origins. Specific universities are remarkably durable, with the oldest (Bologna, Oxford, Salamanca, Cambridge) dating to the 11th and 12th centuries (Bologna will be 1,000 years old in 2088, within the lifetime of some now reading this). ([https://en.wikipedia.org/wiki/List_of_oldest_universities_in...](https://en.wikipedia.org/wiki/List_of_oldest_universities_in_continuous_operation))

Curricula, degrees conferred, methods, organisation, associated attached sport and medical industries, and defence and research programmes ... are largely newer. Starting with the Humboldtian reforms in Germany in the early 19th century, much of what we now think of as the modern universiy started taking shape ([https://en.wikipedia.org/wiki/Humboldtian_model_of_higher_ed...](https://en.wikipedia.org/wiki/Humboldtian_model_of_higher_education))

In-place instruction of both explicit and tacit knowledge, remains the most effective method attained. Mediated education, whether through old low-tech such as books, or high, such VR headsets, may have their affordances, but simply cannot match the full immersion of simultaneous group presence. More so for tacit knowledge in which simply ingesting words is not sufficient, where hands-on experience is required, and equipment, facilities, and materials needed.

There are several fundamental functions of the universiy which seem unlikely to vanish. Education; credentialling; open research; establishing curricula and pedagogy; relations with culture, industry, and government, document archival, basic research; keeping aging academics off the streets; and serving as transition institutions four young adults entering the world, all remain. No doubt individual campuses or systems will collapse, and many revered traditions will be abandoned, at least temporarily.

How this is accomplished given a few months, or years, or decades, of endemic disease, is a challenge. The institution's seen worse.

<comment><USER4> > Education generally, and universities specifically, are highly durable > institutions which have undergone massive changes over time

It's as though, if you concentrate a lot of smart and well-meaning people in one place, they make a great institution!

Maybe not any particular university, but the idea of elite, sequestered scholarship itself has long outlived neoliberal capitalism.

It was never about money. Moneyed interests expounding on the future of the academy are always dead wrong.

<comment><USER5> Finding other smart and well-meaning people and coordinating with them to meet at the same place is not trivial though.

</comment></comment></comment></comment><comment><USER6> Scott Galloway of NYU called it, and has been increasingly louder about it.

Like he says, it seems like this is the turning point for many middle-tier universities. Once you start viewing colleges as brands and luxury goods, especially the Stanfords and Harvards, it becomes clear there's a massive revenue shortfall for them.

>This fall, I'm scheduled to teach MKTG-GB.2365 (Brand Strategy) in KMC 2-60 Tuesday nights at 6-9pm. It's not going to happen. The room would normally be filled with 170 full- and part-time MBAs looking to garner the skills to build economic security for themselves and their families, and improve the world. Mostly the former. However, a room full of 170 NYT subscribers, sitting elbow to elbow, spells one thing — recurrence. So, while Liberty University will likely welcome kids back to campus, I speculate we, and anybody else that does not have their head up their a __, will not.[https://www.profgalloway.com/post-corona-higher- ed](https://www.profgalloway.com/post-corona-higher-ed)

</comment><comment><USER7> As a full-time resident of Boulder, CO (college town), this seems like an impossible situation. I can well imagine that the university here is under heavy pressure from local government to hold classes in the fall. This is because going a semester without all the students/faculty will be disastrous for the town's economy. On the other hand, the increase in coronavirus cases will also be disastrous. I don't have an answer. But, as a person who recently bought their first home in the area, I'm pretty bummed at the prospect of the local economy collapsing.

<comment><USER8> We're moving to Boulder next month. With houses starting at $800K and larger houses going for $1.5M+, it seems like the local economy must be driven as much by the tech companies that flocked to Boulder bc of the school as by the school itself.

<comment><USER7> I've lived here for about 30 years and I really doubt Boulder would last long without the university. And when I said "home", I meant condo, not house. The average tech employee in Boulder is still a long way away from house ownership in town. The real estate prices are not primarily explained by the tech scene.

</comment></comment></comment><comment><USER9> For universities to fully crumble and education to become significantly cheaper, in addition to all the advancements that have occurred in last decade, I believe we need a credible way to certify students online. Imagine if someone sitting in Tibet or slums of Bombay could prove to Googles of the world that he/she (and not some hired help) successfully completed a highly ranked test online, it could be transformative. If online testing could vouch for a person in the same way that universities do, it could open the doors to world-class opportunities for the entire world, cheaply.

<comment><USER10> Online tests are wide open to cheating if they aren't taken in person. Hybrids exist (take online test in a testing center, e.g. AWS certifications), but aren't all that useful for the slums of Bombay.

And in a way Google is already doing this in person with their interview process: they ask obscure questions aimed at knowlege gained in a computer science degree (or by good test preparation).

<comment><USER11> taking tests in a testing center is actually pretty viable and scales well. This is primarily how all those international students from developing countries write GRE, TOEFL and end up coming to study in the US.

</comment></comment></comment><comment><USER12> I'm an adjunct prof at a middle tier college. We have a couple classes planned to be in person for fall. I highly doubt it. Plus a majority of the student body is foreign and can't even travel to the USA. Essentially I'm sayin RIP to middle of the road universities. Guess I need to start looking for another side hustle.

</comment><comment><USER13> I'm doing a distance data science degree at a fraction of the cost and I would imagine I'm getting much more out of it than this September intake will. There's been a revolution in remote working, will education follow?

<comment><USER14> I hope the education is free. Because without the human presence, paying $250,000 USD for a university education sounds like a ripoff.

<comment><USER15> Is anyone genuinely paying a quarter of a million dollars for a degree? That seems extreme even for a non-scholarship medical doctorate degree, let alone a simple data-science degree.

<comment><USER16> I'm a current college student, and my school costs about $80k per year, counting tuition, room & board, supplies, etc., before financial aid. That would be $320k for 4 years, well over a quarter of a million.

I'm personally getting a significant amount of financial aid, so that it will end up costing much less than that for me, but I know people who are paying full price.

EDIT: I'm an undergrad at a 4-year university.

<comment><USER15> Obviously I believe you if you're saying that, but I've heard that nobody really pays these full sticker-prices and everyone's really getting some kind of aid unless they're literally a billionaire - is that not the case?

<comment><USER17> Typically, in the Ivy League / Stanford, a one-child family with annual income of over $150k to $200k (depending on the school) will pay full tuition. Really. And incomes of less than $65k-ish get you a free ride. Details vary.

<comment><USER18> This happened with my family. It was especially bad because earning $150k total for a household near an expensive urban area leads to purchasing power basically the same as a family with $65k household income in suburbia or small towns. Without adjusting parents income for cost of living (factors the kids in question have no control over), it's totally unfair.

This has a lot to do with "last place aversion" - the people making $150k near SF having to pay full price for their kid (who probably worked their ass off) to barely scrape into Stanford are (rightly or wrongly) going to be totally NIMBY / selfish in terms of voting and policy when it comes to the $65k families from far away whose kids can pay much less than half for Stanford, say.

<comment><USER19> I usually think of the opposite affect, someone making double money in San Fransisco pays the same price for what they order on amazon as someone working and living in a cheaper market. Also cars, airline tickets, vacations, etc. Cuts both ways, I guess.

My two cents: I'd blame San Fransisco, they are the ones with the weird housing market and shouldn't make others subsidize things around their propped up property prices.

<comment><USER18> I don't think you're right about that. Groceries, supplies, movie tickets, restaurants, fuel, transit, medical & dental resources, rent, and so on are all much more expensive in the city. The fraction of things you get on Amazon or through other means that allows paying a low competitive nationwide price is pretty tiny and doesn't really offer any meaningful advantage to city dwellers.

But on the flipside, state and federal taxes collected from city dwellers do heavily subsidize much more expensive infrastructure and operations costs in suburban and rural areas.

So rural residents get safe roads, remote snow removal, remote power lines, heavy freight supply shipping, equal prices for US mail shipping, school systems and so on, despite not having their own tax base capable of actually sustaining all the costs.

Basically, middle class and upper middle class urban workers subsidize pretty much everybody else. Any richer and you have access to tax avoidance resources and lobbying, and poorer and you consume much more in government resources than you pay in, especially in rural areas.

<comment><USER20> > I don't think you're right about that. Groceries, supplies, movie tickets, > restaurants, fuel, transit, medical & dental resources, rent, and so on are > all much more expensive in the city. The fraction of things you get on > Amazon or through other means that allows paying a low competitive > nationwide price is pretty tiny and doesn't really offer any meaningful > advantage to city dwellers.

It's primarily the discretionary, high-tech, and expensive items that are cheaper in a high-cost-of-living place. Want to buy an iPhone or a Tesla or an expensive camera or something like that? Those are all less expensive relative to your income in high-CoL places. Plus, interest rates from savings accounts and many retirement accounts are the same nationwide, meaning that having more total money is advantageous.

<comment><USER18> Inflation is faster in some urban centers though, so fixed interest rates are disproportionately bad for those urban dwellers. While that interest rate might grow your larger starting pile of money faster than it grows someone's money in rural Nebraska, the gains are eroded more quickly so they don't translate to purchasing power.

In those cities, your money has to grow fast or the loss of purchasing power is huge.

So I wouldn't say nationwide interest rates are definitely an advantage for people with bigger starting piles of money in urban areas. It's complicated.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER21> So many of these teachers have become unqualified to teach remote classes since they can't adapt to virtual engagement to help students succeed. They don't answer emails, don't chat, leaving students lost. You get more from a youtube video in many cases. Remote teaching has made these teachers lazy IMO.

<comment><USER22> I agree, based on this:

My high-school aged son was taking two in-person community college classes this last semester.

When the SIP hit, his Spanish class just "ended". Nothing online, nothing from his teacher, it was just over. His math teacher worked diligently to keep things going, and I could see the adaptations the math teacher attempted to evolve his efforts with the unexpected conditions.

If the CC is only online in the fall, we'll find something else for him. I don't think v1.0 of online CC is worth spending my money on, based on the v0.9 beta.

<comment><USER23> I'm not sure it's fair to characterise a change in circumstances as severe as what we've just been through, and the results of remote work, as a v0.9 beta. MVP feels fairer, and even that feels like a stretch for a lot of places.

<comment><USER22> Possibly -- but where a CC has a lot of adjunct profs, not staff, I'm not sure they're going to solve a lot of their issues over the summer.

Perhaps at a University, where tenured staff.. and those in the tenure-track, are likely to take the changed circumstances as a mandate to adjust and adapt.

One consideration we've had is signing him up for classes, and making a determination pretty quickly about how online-instruction ready the instructors he ends up with are. If we can drop enrollment for no, or little, $ outlay, we may give it a shot.

But our alternative -- returning to a tutor we used some time ago -- could be a more secure option, unless he's booked full. Which could be the case.

</comment></comment></comment></comment><comment><USER24> College currently sell an "experience" with clubs, restaurants, football teams, fraternities, housing, a nice campus with trees and lakes, etc. This isn't free. It has pushed up the price of education.

If education turns remote, I would expect prices to fall (and layoffs for the administrative staff), unless a similar experience part can be replicated online (which I strongly doubt)

This is the exact opposite of work vs remote work: if WFH becomes the default, I would expect wages to rise, as the most "experience" provided in the workplace (with nice offices, nice chairs, window view, in a nice campus with amenities like catering etc) will be footed by the employee in a WFH setting

<comment><USER25> If working from home becomes the default then wages will fall because employers will be able to select employees who live anywhere. Working in a physical office constrains the labor supply to workers who live nearby (or are able to relocate).

<comment><USER26> It will fall for people living in high cost areas like SF and NYC, but it will rise for people living in low and mid cost areas like Dallas or Cleveland because now companies in SF can easily hire the best developers in Dallas cheaper than average developers in SF by paying them more than the average Dallas rates.

</comment></comment></comment><comment><USER27> This is good. I think the commoditization of college had profound negative effects on society. Historic trust in the academic system was abused in order to, at best, sell degrees to people who were unqualified for them, and at worst, to indoctrinate two generations of children (anything outside of STEM is heavily politically slanted - it's the long march through the institutions that the infamous Weather Underground spoke of).

If intelligence (and general ability) is truly normally distributed then you cannot award degrees to the majority of the population without lowering historic standards.

<comment><USER28> You make a lot of good points here. What is expected of students in University these days is a joke compared to historical standards. Ironic given the more demanding job market expectations.

</comment></comment><comment><USER29> I suspect there will evolve a hybrid model:

\- a small number of elite institutions will offer largely the same product they do today, possibly in lower volume but even higher price. Some lectures will be online, combined with in person recitation, lab etc.

\- these same institutions' lectures will be available online, perhaps for free or paid. Why get a lecture from state U when you can get the MIT lecture?

\- local third parties will offer the recitation, lab etc. Perhaps you'll go to a completely independent one, perhaps you'll attend one that has licensed the GAtech name or has some sort of reciprocity deal in order to proctor exams and such.

\- the networking value of school attendance will vanish except at the elite places.

\- a few professors, the ones actually presenting the lecture, will earn a living wage, or perhaps even a significant wage. Everybody else student-facing in this ecosystem will earn adjunct/ post doc barely-living wage.

\- even the recitations could be conducted online, perhaps by third parties in India who can explain the material just as well or better than someone local. You could be getting 1:1 instruction his way.

</comment><comment><USER30> Let the students network and collaborate with each other. Classes and office hours can be done online. Value of universities is primarily in peer interactions.

</comment><comment><USER31> Online classes simply don't have the value that in person classes have. Extend that to degrees and that's the stigma you have to deal with when trying to get a job. Telling an employer you got your degree remotely V the in class person, who are you gonna hire?

<comment><USER32> > Online classes simply don't have the value that in person classes have.

I certainly agree with this (people aren't that stupid). But schools like Stanford and USC have been doing online MS programs for decades. As far as I know they are are just as valuable to your resume as an in-person MS from the same school.

The value is in the fact that your average student simply doesn't have the self-motivation to do fully-online classes (we certainly have been seeing this problem in the last few months). This is why they primarily have been for working professions part-time in the past, that's a group and mode of learning where people actually get it done.

<comment><USER33> I've benefited immensely from online education, but it's not a substitute for an in person liberal arts education.

</comment></comment></comment><comment><USER34> Not everyone is in a position to do it. And I don't know what the position of universities in deferring will be at this point. And obviously gap year options that are constructive/interesting are way limited at the moment.

But I'd have to seriously consider if it makes sense to enter a university as a freshman in September. Especially given a non-trivial possibility of things blowing up mid-semester with all sorts of attendant costs.

Obviously a really really tough decision. And this will probably result in a lot of students not entering or completing college which, despite some of the common wisdom in tech, will probably lead to some pretty severe downgrades to lifetime career opportunities and income to many.

</comment><comment><USER35> I have two daughters in college.

My older daughter just graduated in mathematics / machine learning from UCLA. Online classes were mostly ok, though there were some comical situations where the professor taught the whole class with his microphone on mute.

Her exams were really rough, though. The math professors were unwilling to make any accommodation for the virus. While in quarantine with demonstrations and national guard soldiers literally across the street, she was taking exams. Everyone had 24 hours to take the exams, so they made them extra hard. They graded on a curve, because taking that away would be sacrilegious. So you have a choice, either spend as much of 24 hours as possible on the test, cheat (like some groups of students did), or get a bad grade. And then she had three exams in the same 24 hour period. The goal of testing should be to ensure that you know the material, not this competition shit.

My younger daughter is studying at USC. She came back to Taiwan last semester, so she hasn't needed to deal with the virus. Online classes have been pretty chaotic, though. Watching async videos has actually been a plus, but the mandatory discussion sections at 3 AM have been rough. It would be nice if they would just open one up in Asia timezone. Especially because students from China are now blocked from going back to the US.

USC was planning to have everyone come back on campus next semester, but just canceled because their opening plan was rejected by LA government. I am happy that she will be safe.

Universities are really worried about their business model, and are willing to have people die to preserve it.

</comment><comment><USER36> I find it odd that even now, after all that's happened in recent months, and with the reserves of cash that some of the elite establishments (in the US at least) are not taking advantage of online degrees.

To give you an example. I am planning on starting a degree in October. I'm 45. It will be through the Open University. It will cost about £6000 to complete the degree.

I looked at a ton of Universities in the UK and their online offerings are abysmal. there are some in Europe that look ok but the US ones are in the $30k+ price range...

If I thought that Yale, Harvard etc. would do an online degree for a fraction of what an actual degree currently costs in person, I would jump at the chance... who wouldn't want that on their CV?

With the resources they have they could easily extend their degree programme to the rest of the world via online learning. Same lecturers, same material.

Instead, they're sitting on their hands while their entire foundation burns...

I agree with one of the other comments that this could be the beginning of the end for the bricks and mortar universities

<comment><USER37> A big part of the brand/prestige of Yale and Harvard is the exclusivity.

<comment><USER36> Valid point but there are hundreds of other institutions that don't have that exclusivity that have very little to offer when it comes to online learning.

From what I can see, the OU is miles ahead.

</comment></comment></comment><comment><USER38> > What [the students] might not expect: a lack of professors in the classroom.

Maybe they might not expect it, but will they mind it?

</comment><comment><USER39> I've thought about this quite a bit, really interesting what will happen to higher-ed.

I'm currently a graduate student at a public research uni, which I imagine are the type of uni's that may struggle (cause it's not like ivy tech or other elite schools, may struggle to tract people if they start competing online only).

Another interesting factor are the increasing amount of bootcamps like Lambda and others offering technical education for the low. I wonder if companies like those will try to expand to other less technical subjects.

Think future higher-ed will be up to the experience and faculty, also makes you kind of worry about these public research institutions if tuition starts drying up.

</comment><comment><USER40> Full professorships, irrespective of institution eliteness, are a tough job to get. If I were the provost, I'd just sack the folks who refuse to show up to work this fail. Most of them would be very easy replace (certainly if you value teaching skills over research skills).

Furthermore, the health risk to professors is much lower than it is to primary school teachers. 20 year olds can wear* masks for 45-90 minutes at a clip, 2nd graders won't be able to do that for six hours straight.

* or legally forced to, as they are adults.

<comment><USER41> As recent weeks have shown 20 year olds will be going to bars and parties and seeding new waves of infections. The vast majority of people in their age group have negligible mortality risk, and high incentives to socialize after being stuck at home with parents for months on end. The faculty on the other hand are in the demographic where COVID is a death sentence.

<comment><USER42> And when they don't die, it can ruin their academic & intellectual lives by making them long-term ill & disabled, including neurologically. Some of the lingering symptoms are nasty.

</comment></comment></comment><comment><USER43> This is the con of universities. The only way to justify the tuition is face- to-face teaching and to have access to all those amenities.

Once you remove those students would be asking for price reduction.

</comment><comment><USER44> [https://archive.is/D6RKy](https://archive.is/D6RKy)

</comment><comment><USER45> Administrative bloat and lax student standards should be geared up for older students and retraining during structural changes happening in the economy (right fucking now)

Sadly administrators have no skills to retrain people.

</comment><comment><USER46> Recently I read a post about education that was quite resonating. Universities have historically been "hard to enter, but easy to exit", meaning it's really difficult to get in but about the only thing that matter is getting in and no one really cares what your grades are if you are from Stanford/Harvard et.al.

On the flip side, med school or west point are less about getting in but more about exit value. It's not really that important as to whether you got in but whether you graduated from West Point. West Point / Navy Seals can essentially double the class size without losing any credibility. I think higher education is going in similar direction.</comment>


Binary Signals aka Option Trading What You Need to Put Your Income on Autopilot - <AUTHOR> http://www.bizocean.org/2014/05/Binary-signals-aka-option-trading-what-you-need-to-put-your-income-on-autopilot.html

<comment><USER2> is it really work for you ?</comment>


Mesmerizing maps show where the most educated Americans live - <AUTHOR> http://www.businessinsider.com/where-most-educated-americans-live-2017-3/ <comment><USER2> It would be interesting to see public transit routes plotted over these maps; at least in Boston, they explain the distribution of education pretty precisely. (E.g., check out the transportation to Roxbury versus to Somerville.)

<comment><USER3> Same for property value.</comment></comment>


Launch Pioneer Edition FreedomBox Home Servers - <AUTHOR> https://freedomboxfoundation.org/news/launching_sales/ <comment><USER2> Olimex is an amazing company. They make great boards, I've got a few that here that last +5 years already. Would recommend them to anyone.</comment>


ISPs Aim to Use Facebook Fracas to Saddle Silicon Valley with Crappy New Laws - <AUTHOR> https://www.techdirt.com/articles/20180409/09020639593/broadband-industry-aims-to-use-facebook-fracas-to-saddle-silicon-valley-with-crappy-new-laws.shtml <comment><USER2> The ISPs are in an all out attack on competitors like Facebook/Google/Netflix/others. ISPs bought their way into the market with the ISP privacy protections being removed [1][2] and net neutrality being killed rather than innovating. The privacy protections being removed helped lessen the FCC grip on privacy and helped their case, with legislators not the people, to remove FCC net neutrality protections.

It is obvious everywhere that massive PR pushes are going on and astroturfing to a heavy degree everywhere ISP competitors on the advertising/privacy/data space are.

Instead of innovating their way in with products people want and improving their networks so people pay more, the ISPs are trying to win this via bribing for legislation and mud slinging. They bought their way in with the ISP privacy protections being removed under the FCC to the FTC because they have more leverage there.

It is sad ISPs have resorted to this instead of innovating and creating products people want by improving them rather than trying to slow down competitors via bribing and lobbying.

ISPs were once a beacon of innovation bringing in broadband, now they are in the milking it phase.

[1] [https://eff.org/deeplinks/2017/03/five-creepy-things-your- is...](https://eff.org/deeplinks/2017/03/five-creepy-things-your-isp-could-do- if-congress-repeals-fccs-privacy-protections)

[2] [https://www.flake.senate.gov/public/index.cfm/2017/3/op- ed-f...](https://www.flake.senate.gov/public/index.cfm/2017/3/op-ed-for-the- wall-street-journal)

<comment><USER3> I find it ironic that two classes of monopolies (ISPs and Big Web) are being discussed here with diametrically opposed recommendations.

For ISPs, the council is that they should be regulated (e.g. net neutrality, data privacy).

For Facebook, Google, et al. the council is that they shouldn't be regulated (e.g. EU GDPR).

I have yet to see a cogent argument justifying that difference. My suspicion is that it ultimately falls down to "ISPs are unpopular, web companies are popular."

When it comes down to it, regulation promoting consumer privacy, price and TOS transparency, and access by new entrants to the market is almost always beneficial... to the consumer.

Because let's not kid ourselves that Facebook or Google are fundamentally different than someone who owns the last mile of your network connectivity.

Their current behavior might be (mostly) benevolent, but the centralization they've engineered means if they want to maximize monetization tomorrow...

<comment><USER4> _I find it ironic that two classes of monopolies (ISPs and Big Web) are being discussed here with diametrically opposed recommendations._

Technically, "Big web" would be perhaps a structural oligopoly [1] or you could say the market has monopolistic competition[2]. they're protected by the network effect, they provide unique products that are hard to duplicate. They are not actual monopolies as economists define them (and not really monopolies in the popular understanding either).

But ISPs are straight, government-granted monopolies [3]. Such companies have essentially one incentive, be good at influencing regulators and legislators.

Glibly equating these market structures distorts the situation.

As to why ISPs should be regulated in the name of neutrality and privacy? Well, they _already are highly regulated_. They are _creations_ of regulation. The public damn well should get something _out_ of that regulation, especially, the regulation is all about ameliorating the abusive situation their (actual, not pseudo) monopoly position invites.

[1] [http://www.economicsonline.co.uk/Business_economics/Oligopol...](http://www.economicsonline.co.uk/Business_economics/Oligopoly.html) [2] [https://en.wikipedia.org/wiki/Monopolistic_competition](https://en.wikipedia.org/wiki/Monopolistic_competition) [3] [https://en.wikipedia.org/wiki/Government- granted_monopoly](https://en.wikipedia.org/wiki/Government-granted_monopoly)

<comment><USER3> So your contention is that structural oligopolies (or markets with monopolistic competition) should _not_ be regulated?

I'd be curious to know why.

<comment><USER4> _So your contention is that structural oligopolies (or markets with monopolistic competition) should not be regulated?_

That's nowhere in my argument. I merely pointed that government grant monopolies pretty much automatically _should_ be regulated since they are virtually extensions of government fiat already ("we" give Comcast etc a monopoly on placement on phone poles and similar stuff, we should get something for that) and that this situation isn't comparable to monopolistic competition.

If you're curious, I think actual private companies should be regulated when their actions involve negative externalities [1] in their production process - chemical companies should not be able to dump toxins into rivers, broadly greenhouse gases should be limited, etc.

One might argue that Facebook selling people's data to the highest bidder is another kind of negative externality. I would argue that a better outcome is consumer education, gradually people learn whatever they post online has a significant risk of being public. Similarly, it's better for people to learn not to be manipulated by simple schemes than to imagine a firewall against the manipulations of evil people. Just consider that a firewall against manipulations of evil people would _let in the manipulation of "good people"_, right? Perhaps that's the whole idea.

[1] [https://en.wikipedia.org/wiki/Externality](https://en.wikipedia.org/wiki/Externality)

<comment><USER5> > we" give Comcast etc a monopoly on placement on phone poles and similar > stuff, we should get something for that) and that this situation isn't > comparable to monopolistic competition.

Comcast doesn't have a monopoly on placing things on utility poles. Poles are mostly owned by power companies or the local phone company, and those entities are required to lease pole space to everyone at non-discriminatory rates.

</comment></comment></comment></comment></comment></comment><comment><USER6> Silicon Valley needs new laws, accept that, and make the fight about ensuring those laws aren't burdensome, anticompetitive, and downright stupid. If you spend your energy trying to make those laws not happen at all, they'll be done to us and not with us. It's time to accept that a significant (more than 65%) of people want regulations in the tech space, and calmly, patiently explain what kinds of regulations can be helpful and which would be disastrous. Accept that it will be a matter of compromise (at best) and that means no one will be entirely happy.

<comment><USER7> Silicon Valley doesn't need new laws that's bullshit. People and privacy need strong laws that protect them. Irregardless of who's doing it.

<comment><USER8> That's a great point.

 _I_ need the protection. We don't need to go and start creating huge regulatory bodies around various industries, pitting one against the other. Make it so my information can't be captured -- or it can't be shared. Or make it so that keeping any information of mine past 30 days is a felony.

Then no matter who the businesses are, people can sue them and owners can go to jail.

Instead of trying to figure out exactly where the tech is right now, and how things might change? Bullshit. Protect _me_. With enough protections, the rest of it will work itself out.

<comment><USER9> I think the problem is the roll back of net neutrality for ISPs and at the same time calling for regulation on content providers. It doesn't seem like these interests are protecting the citizens.

Anyhow, I do agree we should be protecting peoples right to privacy and unfettered access to the internet.

<comment><USER8> You're right, of course, but as scottie_m points out, the main issue here is one around messaging.

The enemy of getting this fixed is complexity. Once we start targeting individual companies, types of internet connections, or hardware devices? We've lost. It's all over.

It's not these things don't need fixing, it's that we have about ten words or so to put into a slogan. That slogan needs to be repeated over and over again. I am a firm believer in Net Neutrality -- but that's not the right slogan to march under. If we get Net Neutrality and everything else stays the same, it's a complete loss.

By shifting the focus to each user, we can talk more directly about benefits. And by making penalties perhaps draconian, we can leave the management of the complexities to the vendors, where it belongs, instead of the public debate, where the lawyers win.

Privacy and anonymity are much bigger fish to fry than whether YouTube costs more than Instagram. If we could come up with a simple slogan to cover both of them, sign me up. Otherwise? We gotta stop everybody from putting telemetry on every detail of our lives.

Before the 60s in the U.S., there was pervasive and deeply-held racism, but most people didn't see it, even those who wanted to be morally superior to everybody else. (See the movie "The Help"). Many times I think of what that issue is for us. What will we look back on in 50 years as being barbaric and evil? What is our current equivalent to the Civil Rights campaigns?

It's storing tracking data about the minute parts of people's lives. We will either morally overcome this or we will enter a new Dark Ages. There's no middle ground. Here. Here is the moral battle.

<comment><USER9> It's not just the Net Neutrality rules, along with Obama era Net Neutrality rules that were set to be put in place, there were security and privacy rules that ISPs also overturned. If we simply regulate the content providers, we won't get privacy on the internet and we'll also reward ISPs for their lobbying and manipulation.

ISPs don't need to collect our data to provide service, yet they do. Content providers don't need to track us, yet they do. They both need to be addressed, or as you say, "we've lost." So, Privacy Now! Or some better clarion call.

[https://arstechnica.com/tech-policy/2017/10/verizon-asks- fcc...](https://arstechnica.com/tech-policy/2017/10/verizon-asks-fcc-to- preempt-any-state-privacy-or-net-neutrality-law/)

[https://arstechnica.com/tech-policy/2017/04/trumps- signature...](https://arstechnica.com/tech-policy/2017/04/trumps-signature- makes-it-official-isp-privacy-rules-are-dead/)

</comment></comment></comment></comment></comment></comment><comment><USER10> Can we please lump the data leaks of Equifax and the like in with the need for these new laws that shift user data from something companies are mining to something they are protecting and disclosing?

<comment><USER11> ISPs are planning to get into the data game. They aim to have laws that limit SV type startups, rather than increase their future liability.

<comment><USER12> The only good outcome will be for this to backfire _spectacularly_ with privacy protections that limit ISPs just as much as the tech companies.

The ISPs are already playing a dangerous game. Unlike their buddies at the FCC, congress is elected, and with enough popular pressure (and midterms coming up), they can be persuaded to do the right thing.

<comment><USER13> Particularly since the perceptions of the telecom industry are not exactly positive as a whole.

It's worth raising a question -- do the executives at AT&T/Comcast/Mediacom/Verizon really think that the public is on their side?

</comment></comment></comment></comment><comment><USER14> As someone that is programming a finance tech app, every regulation is bad for the future.

I am not afraid of regulation but its an annoying burden to keep big players big, and small players non-existent.

Most people do not have the money or motivation I have, and those people lose out because of regulations.

Never let a good tragedy go to waste.

<comment><USER15> If you don't regulate you allow bad players to keep doing bad things.

If you do regulate you force bad players to work harder to keep doing bad things.

I understand that regulation has negative side effects and unintended consequences. But truly free markets lead to child labor and depressed worker wages. It's impossible to craft perfect regulations, but it's implausible to assume "everything will just sort itself out if we let the free market be completely free" since that has literally never happened on a broad scale.

</comment></comment><comment><USER16> IF there is a war between ISPs and social networks maybe we get some benefits like the practices of ISPs and social networks getting more public for the regular person.

</comment><comment><USER17> This article feels so slanted it's unreadable. Conceding that Facebook and Google probably actually do need to be regulated happens near the very bottom of the article; until then you'd be forgiven for thinking that the ISPs made all their concerns about those companies up.

<comment><USER18> Agreed. Silicon Valley could frankly use some "crappy new laws". Of course, so could ISP's, and that didn't happen.

</comment></comment><comment><USER19> HN users have been pounding loudly on the anti-Facebook wardrums. Where did you imagine that going?

<comment><USER20> A place where Facebook changes their policy to take into account their users' privacy?

</comment></comment><comment><USER21> Is there any website that documents who all the ISPs sell our data to? Or at least can we know how much they earn from our data?

</comment><comment><USER22> ISPs and traditional media outlets (old media).

</comment><comment><USER23> "Oh dear how sad" hand back your licence then :-)

</comment><comment><USER24> Frankly as an end-user , one has to choose between bad and worse. Consumers are a true product now, readily available for exploitation

<comment><USER25> And to worsen the legal system doesn't come to rescue. Infact they joined hand with business and created a bible size EULA with legal jargon, which end-user have to accept and move forward.

</comment></comment><comment><AUTHOR> So does the news industry.

</comment><comment><USER26> Techdirt really doesn't live up to its name. When that site isn't actively avoiding major stories that make Silicon Valley megacorps look bad, they're busily defending the Surveillance Valley business model outright.

The "ISPs are bad, therefore Silicon Valley should be allowed to continue running roughshod over users' privacy" narrative is really idiotic. It tries to present this as a situation where we have a choice of either regulating ISPs, or regulating privacy-invading Silicon Valley surveillance corporations, but not both.

Edit: I just read the full article, and yes maybe my assumption wasn't charitable. That said, I had just got done reading this article where Mike Masnick encourages us to continue giving the benefit of the doubt to someone who has repeatedly violated trust for more than a decade, which left me _ass_ uming it would just be more of the same.

[https://www.techdirt.com/articles/20180408/22400539590/faceb...](https://www.techdirt.com/articles/20180408/22400539590/facebook- derangement-syndrome-company-has-problems-must-we-read-worst-into-absolutely- everything.shtml)

<comment><USER27> Your comment reflects a bad faith reading of the article and actually doesn't reflect the content of what was written.

Maybe read the article next time and not make such a low-effort comment.

<comment><USER26> Most of what I've read on techdirt over the past couple years seems to represent a bad faith effort to frame the current situation as "ISPs vs. Silicon Valley and the users", where the reality of the situation is "ISPs and Silicon Valley vs. the users" (and there's a recent situation in California where the ISPs teamed up with Silicon Valley to prove that point). It's hard to give the benefit of the doubt to that site at this point.

..and I say this as someone who used to be a big fan of that site. It's disappointing.

</comment></comment></comment><comment><USER28> The people who make up the tech industry been a huge proponent of regulating all sorts of things at all scales, from individual actions to small businesses to large corporations.

The use of the FB/CA "scandal" to promote knee jerk legislation that burdens the tech industry is just tech being on the receiving end of something they have no qualms about advocating for when it happens to a different industry.

<comment><USER29> This is a bit like the old adage about noticing how bad news reporting is in areas you're an expert, but then just blindly swallowing all other news. In this case it's regulation rather than news coverage.

</comment></comment><comment><USER30> Silicon Valley brought this to itself. I used to warn of this before, too, that they can't continue to do "evil stuff" and expect to have people's backing when the governments will come after them to regulate them for whatever reason. But they continued because they saw that nobody leaves them over the crap they pull so they thought there must be no consequences to their anti-consumer moves.

The only downside I see is that governments will use this excuse to bring more censorship and anti-encryption laws, too, and I _hate_ that Silicon Valley companies put is in a position where we have to either put up with their crap or side with governments and get some bad internet laws passed.

But right now, I think it's time to rein in on the Silicon Valley companies over their abuse of user data. Then we can deal with the abuse of user data by ISPs, data brokers, and everyone else. Ideally, this would be solved with a GDPR-like law, so that even ISPs "lose" in this scenario, and the consumers win. If we're lucky we may be able to stave off the government from trying to put an anti-encryption bill into the whole thing.

<comment><USER31> That's like what the game industry went through in the early 90s. When Mortal Kombat and the like were generating controversy and getting the attention of regulators, the industry proactively created the ESRB.

<comment><USER32> There's a big difference though. "The video game industry" wasn't really in the business of creating video games that offended cultural norms. It had a subcomponent of that industry that did it, sure, but even that subcomponent would have survived by making different games with regulation.

There is a lot of Silicon Valley right now whose core business is doing the things that are very likely to be regulated out of existence.

So I'd say self-regulation isn't an option at this point, as "self-regulation" is asking Facebook to return to unprofitability with no prospect of profit in the future, the destruction of huge swathes of the ad industry, any number of other companies based around data mining and selling and abusing information asymmetries... there's no way the wolves are going to self-regulate themselves into dogs.</comment></comment></comment>


Slowly changing dimensions - <AUTHOR> http://en.wikipedia.org/wiki/Slowly_changing_dimension

<comment><USER2> Good timing - this just came up at my work over the last few days. Being from an OLAP background makes it hard to wrap my head around this approach.</comment>