Julia issue #8839 closed - <AUTHOR> https://github.com/julialang/julia/issues/8839 <comment><USER2> Against all odds, the PhD thesis itself is both approachable and extremely interesting to read, even for someone who has never written in Julia (and in all likelihood never will): [https://github.com/JeffBezanson/phdthesis/blob/master/main.p...](https://github.com/JeffBezanson/phdthesis/blob/master/main.pdf)

<comment><USER3> came here to make the same comment. extremely readable and interesting thesis. proves that academic writing doesn't have to be unapproachable.

</comment></comment><comment><USER4> The only PhD defense I've seen with standing room only. Congrats!

<comment><USER5> Why was the PhD defense standing-room only?

<comment><USER6> There was an off-by-one error in calculating the required number of chairs.

<comment><USER7> Caused by Julia using 1-origin indexing. (I kid, I kid)

</comment></comment></comment></comment><comment><USER8> Congrats to the creator of Julia for completing his PhD on it :)

I note that the Acknowledgements page is empty. You goddamn did it on your own eh? ;)

<comment><USER9> That can be somewhat private, but is probably a result of the template he used.

</comment></comment><comment><USER10> "I can't reproduce this issue locally" hahaha

</comment><comment><USER11> I am on my way of getting my Master's afterwards I want to pursue a PhD, however, I have this feeling that my PhD will never be as great as this work. How do I get over this fear?

Besides that, this is obviously huge for Jeff, congratulations!! :)

<comment><USER12> i am doing a phd now and i think of it this way: most likely no one is ever going to read your papers and certainly not your thesis

<comment><USER13> If you are an aspiring actor, you will probably never become a star, but why is that to keep you from trying?

Getting a PhD...you should only do that if you want to try and make that big discovery (even if you'll more than likely fail), or are happy with standard academic work.

<comment><USER14> That doesn't mean you have to do that "big discovery" IN your PhD. PhD is just a paper so you can start your academic/research career.

<comment><USER13> Ya, you don't always get discovered in acting school. But you should have the dream while you are there.

</comment></comment></comment></comment></comment><comment><USER15> I suppose I should have seen it coming, but I never imagined using git as a method to fling theses.

Good work!

</comment><comment><USER16> PhD from MIT? I like him already.</comment>


Ehang 184 drone could carry you away one day - <AUTHOR> http://www.gizmag.com/ehang-184-aav-passenger-drone/41213/ <comment><AUTHOR> >full charge reportedly being sufficient to keep one passenger airborne for up to 23 minutes

>According to the Ehang rep we spoke to, the 184 is already fully functional, with a worldwide series of demo flights scheduled to begin soon. They claim that it should be commercially available later this year(!)

Seems kinda cool if dangerous</comment>


Proof of Computational Work - <AUTHOR> http://www.kapsi.de/blog/index.php/2011/07/proof-of-computational-work/

<comment><USER2> Ah yes, another person new on the Internet. Welcome. The blogger doesn't seem to be aware that this is an old idea, dating back to 1992 or earlier (well before the also old 1997 date given in WikiPedia for HashCash). See the references in this paper: <http://www.cl.cam.ac.uk/~rnc1/proofwork.pdf>

</comment><comment><USER3> The original "Pricing Via Processing or Combatting Junk Mail" paper by Dwork & Naor from 1992 is [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.75....](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.75.798&rep=rep1&type=pdf)

And the follow-on "On Memory-Bound Functions for Fighting Spam" (which is based on the notion that "since memory access speeds vary across machines much less than do CPU speeds, memory-bound functions may behave more equitably than CPU-bound functions") is: [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.85....](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.85.9764&rep=rep1&type=pdf)

</comment><comment><USER4> >A novel idea to fight Spam in Distributed Systems with no or minimum State.

It is not novel. It has been discussed extensively in the context of email spam.</comment>


Show HN: Gampets – App for watching pet videos - <AUTHOR> http://apple.co/1TFPsyv <comment><AUTHOR> Hey guys! Please give us feedback on this app. We are still working on adding a lot more cool features. This is just a beta version. Thank you!</comment>


Math Basics for Computer Science and Machine Learning [pdf] - <AUTHOR> http://www.cis.upenn.edu/~jean/math-basics.pdf <comment><USER3> The professor who wrote this is Jean Gallier, and I had him for advanced linear algebra at Penn. I am also pretty close to him in so far as a student can be close to a professor. On a personal note, he is one of the funniest professor I've had, and all math professors are characters.

For the people who are interested in ML, the thing to remember here is that he is a Serious mathematician, and he values rigor and in-depth understanding above all. A lot of his three star homework problems were basically impossible. He writes books first and foremost so _he_ can understand things better. In math books, there's the book you first read when you don't understand something _, then the book you read when you understand everything. This is book in the link.

_ for linear algebra, this:[https://www.amazon.com/Introduction-Linear- Algebra-Gilbert-S...](https://www.amazon.com/Introduction-Linear-Algebra- Gilbert- Strang/dp/0980232775/ref=asc_df_0980232775/?tag=hyprod-20&linkCode=df0&hvadid=312152840806&hvpos=1o1&hvnetw=g&hvrand=13794249926302782300&hvpone=&hvptwo=&hvqmt=&hvdev=c&hvdvcmdl=&hvlocint=&hvlocphy=9015292&hvtargid=pla-454800779501&psc=1&tag=&ref=&adgrpid=61316181319&hvpone=&hvptwo=&hvadid=312152840806&hvpos=1o1&hvnetw=g&hvrand=13794249926302782300&hvqmt=&hvdev=c&hvdvcmdl=&hvlocint=&hvlocphy=9015292&hvtargid=pla-454800779501))

<comment><USER4> As someone who never took an under graduate linear graduate course, videos of Strang's lectures got me through a couple graduate machine learning courses.

[https://ocw.mit.edu/courses/mathematics/18-06-linear- algebra...](https://ocw.mit.edu/courses/mathematics/18-06-linear-algebra- spring-2010/video-lectures/)

What he does with chalk and a blackboard is far more effective than anything done today with Powerpoint, fancy computer animations, or what have you.

</comment></comment><comment><USER5> From the start of Chapter 2:

"In the following four chapters, the basic algebraic structures (groups, rings, fields, vectorspaces) are reviewed, with a major emphasis on vector spaces. Basic notions of linear algebra such as vector spaces, subspaces, linear combinations, linear independence, [...], dual spaces,hyperplanes, transpose of a linear maps, are reviewed."

If anyone needs to start even earlier than this, I've actually found "3D Math Basics for Graphics and Game Development" to be a good true intro for linear algebra-related stuff. I think this would probably hold even if your primary interest is something other than graphics/game dev. Some of the text in that book's intro is a little cringey with its reliance on kind of juvenile game references, but I didn't find that sort of writing continuing during the actual text. So just push past that stuff.

I got a copy of it to act as a refresher before diving into Real-Time Collision Detection since it's been quite a long time since formal math for me (as in, high school, because I'm self-taught in CS). I've managed to make up a lot of ground by working hard and finding classes to audit online (Strang's linear alg course on OCW is a good one), but I have found that depressingly few math texts which claim to be "introductory" are actually truly introductory.

This isn't a slight against the linked work, I absolutely love when profs make resources such as this freely available.

"How to Prove It" and "Book of Proof" are also great intros to formal math, if less immediately practical.

<comment><USER6> I would disagree about the gamedev book reference, unless you are referring to the real basics of linear algebra.

The really important concepts for ML are least squares, eigenvalues and vectors, and SVD. Those concepts are not very relevant to game programming.

Well, least squares can be solved with projection, which is relevant for converting between coordinate spaces. But game dev isn't going to give you that intuition.

<comment><USER7> I believe the person you're replying to was attempting to help people like me, who haven't had a math lesson since leaving high school (in my case over 16 years ago) and whose level of math is roughly "You want me to multiply something? Let me get my phone."

So while the book in question might not be _the_ best resource, it probably is a better starting point than the linked doc.

<comment><USER8> I wish someone would put together the "basic math" resource for people like that. I didn't do any maths beyond 16. I was an maths idiot at school (me? the teacher? I dunno) but since then I've worked 20 years in programming and had to learn more and more maths just to make a living. However, my maths is a rickety hodge-podge house, with no real foundations. I can't be a total idiot, because I've done some decent work in distributed systems, even research in CRDTs (which made me learn some math) but still, I'd love a self-study plan that took me from 16 to today with the _real basics_.

Thanks for your comment, it made me feel less alone.

<comment><USER9> I've just finished the coursera's Mathematics for Machine Learning: Linear Algebra ([https://www.coursera.org/learn/linear-algebra-machine- learni...](https://www.coursera.org/learn/linear-algebra-machine- learning/home/welcome)) and I'd say its very good for beginners

</comment></comment></comment></comment></comment><comment><USER10> "Math Basics" is quite the misnomer—it gives the impression that one would need to study all of the contents of this book to be an effective practitioner in CS or ML. Memorizing every definition and theorem in this book would be neither necessary nor sufficient for that purpose.

Keep in mind it can take an hour, and sometimes way more, to _really_ absorb a single page of a math book like this (do the math). This is more of a reference text.

<comment><USER11> > "Math Basics" is quite the misnomer—it gives the impression that one would > need to study all of the contents of this book to be an effective > practitioner in CS or ML.

To me basics mean that if you study this entire book you won't be able to understand ML otherwise it would say comprehensive. Furthermore the math presented in this book are all taught in 1st year courses for most CS programs I've encountered.

> Keep in mind it can take an hour, and sometimes way more, to really absorb a > single page of a math.

Learning is a personal experience and happens at differing rates for different people. While I do agree this book is rather terse and would serve as a good reference any added explanations around the proofs would force a split to multiple publications so I can see why the authors chose to present it in the way they did.

Overall I have found this text easy to digest and well formulated and thank the authors and poster.

<comment><USER12> Which CS program? I think you're more than exaggerating. They don't teach topology, FEM and abstract algebra in "most" CS program. These are just three random examples; most of the book is wayyy more advanced than what I would expect to learn in freshman CS.

<comment><USER13> This book seems to be designed as a primer for PhD research students - <USER2> might include some talented final year undergrads heading in that direction.

If that's your level, it might reasonably be called "basic."

For everyone else - no.

</comment></comment></comment></comment><comment><USER14> Great book.

I think it's a good time to mention a couple of nice books (related)

1\. Elementary intro to math of machine learning [0]. Its style is a bit less austere than that of OP's. It also has a chapter on probability. It could possible serve as a great prequel to the book linked in the OP.

2\. The book on probability related topics of general data science: high- dimensional geometry, random walks, Markov chains, random graphs, various related algorithms etc [1]

3\. Support for people who'd like to read books like the one linked in the OP, but never seen any kind of higher math before [2]. This book has a cover that screams trashy book extremely skimpy on actual info (anyone who reads a lot of tech books knows what I am talking about), but surprisingly,it contains everything it says it does and in great detail. Not even actual math textbooks (say, Springer) are usually written with this much detail. Author likes to add bullet point style elaboration to almost every definition and theorem which is (almost) never the case with gazillions of books usually titled "Abstract Algebra", "Real Analysis", "Complex Analysis" etc. Some such books sometimes attach words like "friendly" to their title (say, "Friendly Measure Theory For Idiots") and still do not rise to the occasion. Worse yet, a ton (if not most) of these books are exact clones of each other with different author names attached. The linked book doesn't suffer from any of these problems.

[0] Mathematics For Machine Learning by Deisentoth, Faisal, Ong

[https://mml-book.github.io/book/mml-book.pdf](https://mml- book.github.io/book/mml-book.pdf)

[1] Foundations Of Data Science By Blum, Hopcroft, Kannan

[http://www.cs.cornell.edu/jeh/book%20no%20so;utions%20March%...](http://www.cs.cornell.edu/jeh/book%20no%20so;utions%20March%202019.pdf)

2] Pure Mathematics for Beginners: A Rigorous Introduction to Logic, Set Theory, Abstract Algebra, Number Theory, Real Analysis, Topology, Complex Analysis, and Linear Algebra by Steve Warner

[https://www.amazon.com/Pure-Mathematics-Beginners- Rigorous-I...](https://www.amazon.com/Pure-Mathematics-Beginners-Rigorous- Introduction/dp/0999811754)

<comment><USER15> I'll check out the last one. Currently I am teaching myself real analysis.

</comment></comment><comment><USER16> Very first sentence of 2.1 is full of notation, symbols and terms that I, as a prospective student, might not understand.

So many teachers seem incapable of stepping outside their sphere of knowledge and seeing what they know and others do not. And so much work went into this.

<comment><USER17> Here's what he is doing. He wants to start with the set of real numbers, intuitively the points on the line, usually denoted by R, maybe typed in some special font.

Then he wants to define, say, addition of real numbers. So, given two real numbers, x and y, that might be equal, he wants to define x + y.

So, here he wants to regard addition, that is, +, as an _operation_. Then, as is usual for defining operations, he wants an _operation_ to be just a special case of a function. So, he wants to call + a function. So, + will be a function of two variables, say, x and y. With usual function notation we will have

+(x,y) = x + y

The set of all (x,y) is the _domain_ of the function, and the set of all x + y is the _range_.

So, that defines the function + except commonly in pure math we want to be explicit about the range and domain of the function.

For function +, the range is just the set of all pairs (x,y) with x and y in R. That set is also the set theory _Cartesian_ product of set R with itself and written R x R. So, the domain of + is R x R. The range is just R. Then to be explicit about the range and domain of function +, we can write

+: R x R --> R

which says that + is a function with range R x R and domain R.

We learned how to add in, what, kindergarten? So, why make this so complicated?

Well, he wants to regard the real numbers as just one example of lots of different _algebraic systems_ , e.g., groups, fields, vector spaces, and much more, with lots of operations and, possibly, more that could be defined. E.g., later in his book he will want to add vectors and matrices, take an _inner product_ of two vectors, and multiply two matrices.

So, back to addition on the real numbers, he wants to regard that as just a special case of an _operation_ on an _algebraic system_.

IMHO there's not much benefit for making adding two real numbers look so complicated.

Whatever he did in that chapter for defining addition on the reals, soon he is discussing matrix multiplication with no definition at all -- assuming the reader already understands that, that is defined and discussed many pages later in his book.

So, in his notation

+: R x R --> R

and matrix multiplication, he is using material before he has defined it, even before he has motivated, explained, exemplified, indicated the value of, and defined it. In good math writing and in good technical writing more generally, that practice is, in non-technical language, a bummer.

But from the table of contents, it appears that the book has quite a long list of possibly interesting narrow topics. And maybe for the routine material, his proofs and presentation are good -- maybe. I thought enough of the book to keep a copy of the PDF. It's there; if someday I want a discussion of some narrow topic, maybe I'll try his book!

In mathematical writing, it used to be common for the word processing to be much more work than the mathematics! Now with TeX and LaTeX, and I'm assuming that the book used one of these two, the flood gates are open!

<comment><USER18> Is there a book that explains things in the manner you have? I enjoy math but have trouble reading it.

<comment><USER17> Most of the best math is not trivial and, thus, usually takes some effort to understand.

There are some good authors of math for, say, calculus, linear algebra, differential equations, advanced calculus, advanced calculus mostly for applications, real analysis, optimization, probability, some topics in stochastic processes, introductory statistics, various more advanced topics in statistics.

Mostly the books are short on motivation and applications, and as a result it is too easy to spend time on material likely not worth the time unless you can be sure both to live forever and remember forever.

For calculus I liked Johnson and Kiokemeister. I taught from Protter and Morrey, and it was easier than J&K. Lots of people liked Thomas.

For linear algebra, I liked E. Nering and, then, P. Halmos, _Finite Dimensional Vector Spaces_ which really is _baby Hilbert space_ theory. Take Nering seriously -- he was a student of E. Artin at Princeton. His treatment of linear algebra is balanced and polished. For one of his editions, he has some group representation theory in the back, good, and some linear programming, really bad.

A lot of people like the MIT Strang book.

For advanced calculus to help when studying physics, especially electricity and magnetism and engineering, I very much liked

Tom M. Apostol, 'Mathematical Analysis: A Modern Approach to Advanced Calculus', Addison-Wesley, Reading, Massachusetts, 1957.

He has more recent versions, but for physics and engineering I like the 1957 version and don't like the later versions at all.

For ordinary differential equations, I liked

Earl A. Coddington, 'An Introduction to Ordinary Differential Equations', Prentice-Hall, Englewood Cliffs, NJ, 1961.

He makes variation of parameters look really nice -- then can understand the remark in the old movie _The Day the Earth Stood Still_. Ordinary differential equations is a huge, old field, and there is some question about how much of that deserves study now. Do notice that for _systems_ of ordinary differential equations, get to apply some linear algebra in cute ways.

For advanced calculus for applications, there is the old MIT Hildebrand -- he knows what he is talking about, is easy enough to read, and a good place to go if need one of his topics.

In recent decades, the pure math departments wanted to teach _advanced calculus_ as the theorems and proofs for freshman calculus. So there is Rudin, _Principles of Mathematical Analysis_ , third edition (not the first two, maybe a later edition if there is one). So here's what is going on: He wants to develop the Riemann integral which is the one in freshman calculus. For that he wants to integrate over a closed interval on the real line, that is, some [a,b] which for real numbers a <= b is the set of a real numbers x so that a <= x <= b. Rudin will argue that [a,b] is a _compact_ subset of the reals and show that the Riemann integral exists on all compact sets. So, the first chapters are big on compact sets. Then he talks about functions that are continuous and then ones that are uniformly continuous. With uniform continuity, the Riemann integral follows right away. Later he does some infinite sequences and series and then uses these for careful treatments of some important results, the exponential function, the number e, the sine and cosine, etc. Later he does integration of functions of several variables on manifolds for Stokes theorem and the related divergence theorem, a fully careful treatment of these theorems used in E&M. He does the Cartan _exterior algebra_ : What is going on is that he wants to integrate a function g: M --> R where M is a _manifold_ , that is, the range of some function f from some box, triangle, etc. to the space with the M. So, for this need the formula for change of variable for integrating with several variables, and that is a determinant of a square matrix. This integration is a multidimensional version of the line integral where direction of integration is important -- the exterior algebra is the multi-dimensional version of that. Can see that again in some treatments of general relativity in physics.

I like Rudin's third edition: Once know what the heck he is driving at and how he is getting there, say, as above, then his high precision is welcome.

For statistics, I suggest using some popular elementary book as a start. Then learn probability really well and from then on study particular topics in statistics as needed. The current directions in machine learning promise to make lots of particular topics important.

For a first book on statistics, consider

George W. Snedecor and William G. Cochran, 'Statistical Methods, Sixth Edition', ISBN 0-8138-1560-6, The Iowa State University Press, Ames, Iowa, 1971.

My wife did really well with that. So, get a good start on statistics and, then, get to learn some analysis of variance (experimental design), an underrated topic.

For a second book on statistics, consider

Alexander M. Mood, Franklin A. Graybill, and Duane C. Boas, 'Introduction to the Theory of Statistics, Third Edition', McGraw-Hill, New York, 1974.

Here, go quickly and get only the high points and don't expect the math to be very good -- in places it's pretty bad.

For regression analysis and linear multivariate statistics more generally, there are several books, Maurice M.\ Tatsuoka, Donald F.\ Morrison, William W.\ Cooley and Paul R.\ Lohnes, N.\ R.\ Draper and H.\ Smith. So, in particular, get enough to understand that regression is a perpendicular projection and, thus, get the Pythagorean theorem again.

For more on such statistics aimed at machine learning, get the Breiman CART -- _Classification and Regression Trees_ , maybe much of the start of ML.

With that much in statistics, will have seen a lot of applied probability and may be ready for the real stuff. For that, need measure theory, e.g., the first half of Rudin, _Real and Complex Analysis_ or Royden, _Real Analysis_. Then read Breiman, _Probability_. After that might read some of Chung, Loeve, Neveu, and maybe some more. Then return to applications including statistics with a really solid foundation in probability, random variables, the classic limit results, and much more. Then can read and/or write lots of advanced topics in statistics.

For optimization, a similar review is possible, but it's getting late.

<comment><USER18> Thank you very much for the list. If you care for any further reviews I would greatly appreciate it. Your list as is will take me years anyways.

</comment></comment></comment></comment></comment><comment><USER19> I love that they have problems you can solve as well at the end of (almost) every chapter.

This IS a lot of math (1,962 pages) and it's missing a preface/introduction which would have been helpful to understand if I need to go linear or if a la carte is okay. At the moment I'd assume each major section is independent.

Awesome find! Wonder how It's used. (One of) the author(s) seems pretty prolific too - [http://www.cis.upenn.edu/~jean/](http://www.cis.upenn.edu/~jean/)

<comment><USER20> > _a la carte_

Yeah, I wish we had an online resource (other than Wikipedia) anyone could learn any sort of math from in a systematic way... Oh well.

<comment><USER21> I'm wondering - is this a genuine request, or a snarky, implicit reference to an online resource for learning math somewhere?

I'd love to know about the existing resource, if it exists. (The only thing that comes to mind is Wolfram Alpha, which didn't seem 'systematic' the last time I skimmed the main page)

<comment> house9-2 Maybe they were referring to Khan Academy?

[https://www.khanacademy.org/math](https://www.khanacademy.org/math)

<comment><USER22> I think Khan academy for higher math/physics/chemistry /CS would be invaluable. Maybe it could be crowdsourced in some way?

<comment><USER23> Well, there's always 3Blue1Brown: [https://www.youtube.com/channel/UCYO_jab_esuFRV4b17AJtAw](https://www.youtube.com/channel/UCYO_jab_esuFRV4b17AJtAw) Take a look at his course, Essence of Calculus: [https://www.youtube.com/watch?v=WUvTyaaNkzM&list=PLZHQObOWTQ...](https://www.youtube.com/watch?v=WUvTyaaNkzM&list=PLZHQObOWTQDMsr9K-rj53DwVRMYO3t5Yr) I'm not fluent in math, but I find it fascinating to watch, for some reason. NancyPi is also great: [https://www.youtube.com/channel/UCRGXV1QlxZ8aucmE45tRx8w](https://www.youtube.com/channel/UCRGXV1QlxZ8aucmE45tRx8w) Oh, and don't forget the Mathologer: [https://www.youtube.com/channel/UC1_uAIS3r8Vu6JjXWvastJg](https://www.youtube.com/channel/UC1_uAIS3r8Vu6JjXWvastJg) What a great guy! While I'm at it, go watch Numberphile too: [https://www.youtube.com/user/numberphile](https://www.youtube.com/user/numberphile) Is there anyone I forgot? :D

</comment></comment></comment></comment></comment></comment><comment><USER24> Whoah, this covers a lot. I was expecting some linear algebra, calculus, and discrete math, but there's actually some stuff in there I don't know after doing a masters in math.

<comment><USER25> That makes me feel somewhat better - I saw the title of 'Maths Basics' and thought 'Great!'... then I saw it's 1,962 pages - if that's the basics, how much is the intermediate and advanced bit?

</comment></comment><comment><USER26> I love professor gallier! He's an incredible person.

That being said, this is faaaaaar beyond basics. It'd be more appropriate to call this an incomplete (aiming to be comprehensive) guide to almost everything you need to know in computer science (related to math).

<comment><USER17> My look at the table of contents looked like the book is short on both probability and statistics.

</comment></comment><comment><USER27> That's almost 2,000 pages of math...I don't know why and how, but somehow I forgot most of the Statistics knowledge I obtained as a graduate student (in Stat) 10 years ago.

I remembered that I took an advanced course about Bayesian Inference, and one course about Multivariate Statistics (PCA, Factor analysis, these kind of things), and my project is about Bernstein Polynomial. That's it...

<comment><USER28> You forget complex things that you don't use regularly. Math, spoken languages, written languages, coding...of course you can re-learn it, and re- learning is faster than learning it for the first time.

Based on speaking to my managers in the past, it seems like a year-long lapse is enough for you to lose an incredible amount of retained knowledge/skill. But it's not a permanent loss.

<comment><USER27> Yeah agreed, sometimes reading a research paper from the DS team would actually ring a bell somewhere and I know where to look at. I'm re-learning Statistics from bottom up at the moment lol but this 2,000-page book really looks daunting. I'm pretty sure I didn't take any advanced optimization course back in university.

</comment></comment></comment><comment><USER29> In basic calculus one can burn countless hours memorizing mechanical rules to derive and integrate different function forms, or one can just plug the function into something like wolfram-alpha and get, for a lot of useful cases, a symbolic answer, or at least some approximate answer for a point or interval.

The point is, understanding integrals and derivatives doesn't require one to memorize all the mechanical rules. Using software to compute those functions can be a huge time saver. No one should go with pen an paper double checking if that polynomial integral is correct or not!

With a book almost 2000 pages long, I wonder if this books leans more heavily on the mechanical-rules side of math. In my mind, is the difference between writing a book such that you can _write your own wolfram alpha_ , or writing a book so you can just _use it_.

<comment><USER30> You don't need to memorize rules when studying math. Just like you don't need to spend any time to memorize syntax for programming languages. You automatically remember things you use a lot.

Once you have spent countless hours doing exercises to the extent that you understand the math, you already remember the rules. If you have not spent countless hours doing exercises, you don't understand anything at this level.

You don't hire a programmer who has read all the books and 'understands' programming but has never programmed. It's the same with math. You don't just read a math book from start to finish. You can use wolfram alpha for visualizing functions, not for learning math.

<comment><USER31> \+ 100. I can't upvote this enough.

Programmers have spent countless hours practising programming to the point where they have forgotten how difficult it was in the beginning. A non programmer might think of programming as "memorizing hundreds of rules" to get anything done, but one doesn't learn programming by sitting around explicitly memorizing hundreds of rules and _then_ begin to program.

Actually writing programs with a minimal set of 'rules' memorized and then adding more as needed is how one typically learns programming.

<comment><USER23> I've been teaching programming for five or six years now. I always start with HTML, then add CSS, and then add JavaScript. That way they experience mastery all the way, and see how they can be creative with the code. It's so great to see a pupil "get it" \-- and sometimes even pupils that "suck at math" or even pupils who have problems spelling the most basic sentence correctly. In fact I've found that there's a strange correlation where pupils who have dyslexia often seem to be better than others at programming.

<comment><USER32> > I always start with HTML, then add CSS, and then add JavaScript

So when do you get around to teaching programming, then? ; P

/ducks

<comment><USER20> Starting with HTML and JavaScript is one of the most efficient ways to get children interested in programming.

</comment></comment></comment></comment></comment></comment><comment><USER33> Haha 1900 pages on "basics"?

I suspect there are better resources for each topic covered (e.g Gilbert Strang books and OCW lectures for Linear Algebra), but it is definitely interesting to peruse and get a sense of relevant topics.

</comment><comment><USER34> "Math Basics"

It's 2000 pages long....

<comment><USER35> Chapter 1 can't be more basic, a great summary of what will remain intuitively after reading the whole opus

<comment><USER36> lol

</comment></comment></comment><comment><USER37> Even if you were ambitious and manage to read 5 pages per day everyday this would take you more than one year to read this from start to finish.

</comment><comment><USER38> Nice to see wavelets in here - but it's a shame that he seems to be encouraging people to actually use Haar wavelets. They're fine for teaching - but there are usually better choices in real life. Daubechies are a good default

</comment><comment><USER39> The writing style of this book i.e. rigorous math notation and proposition/proof presentation is going to put off the great majority of potential CS and ML readers. At almost 2000 pages it sure makes a great door- stop though.

</comment><comment><USER40> Me: Oh, good! I've always been pretty good at math but I want to learn how to make sense of the math I encounter in CS and ML.

[Reads the first paragraph of the 2nd chapter]

Me: I don't know anything about math. At all.

</comment><comment><USER41> Basics should be concepts that get you to 80% and tell you where to look for the rest 20%. This book tries to get you directly to 95% and is best treated as a reference book.

</comment><comment><USER42> Strangely, probability theory is completely omitted.

<comment><USER23> Is there a probable cause?

</comment></comment><comment><USER43> I really can't figure out who the target audience for this book is, if it has a target audience at all.

</comment><comment><USER44> 403 forbidden. Can someone host this?

<comment><USER45> [https://web.archive.org/web/20190730230113/https://www.cis.u...](https://web.archive.org/web/20190730230113/https://www.cis.upenn.edu/~jean/math- basics.pdf)

</comment></comment><comment><USER46> I look at this and profoundly thank the people who make ml libraries for us the rest of us.

<comment><USER47> The vast vast vast vast vast majority of this book (which is more of a reference and encyclopedia than an actual book for learning) is not required for implementing most ML libraries.

</comment></comment><comment><USER48> This is more like courses a talented undergrad math major would take through 4 years.

</comment><comment><USER49> Okay, so this looks potentially awesome. But given that it is a reference work rather than some introductory "basic" little quick read-through, I'd prefer to have it in paper form.

Any hope of that happening?

</comment><comment><USER50> this is an incredible reference for a machine learning researcher who wants to fill in some gaps in their existing mathematical knowledge.

But I would be shocked if this would be of any use for someone trying to learn a little linear algebra in order to play with neural networks. For that I think you still want Strang.

I think "foundations" might have been a better word than "basics" here. "Basics" in any case is not in the printed title, only in the filename.

</comment><comment><USER51> 403 forbidden, can someone help us out?

<comment><USER52> It now redirects to google drive. Simply refresh. It redirects to [0].

[0] [https://drive.google.com/file/d/1sJvLQwxMyu89t2z4Zf9tD7O7efn...](https://drive.google.com/file/d/1sJvLQwxMyu89t2z4Zf9tD7O7efnbIUyB/view)

</comment></comment><comment><USER53> Can someone recommend me a book on Linear Algebra, Statistics and Probability?

<comment><USER54> Second this. If it has exercises, even better. About to undertake my MSc Comp Sci and would like to learn Discrete Math as I'm coming from a non-math background (did B.IT at school and only took two math courses).

</comment></comment><comment><USER55> That's... quite a lot of math

</comment><comment><USER56> This book looks great, I've been looking for a resource like that.

</comment><comment><USER57> Great to see Prof. Gallier featured on here!

</comment><comment><USER58> Math "Basics" in nearly 2k pages!

</comment><comment><USER59> Q: Is this all necessary for ML?

<comment><USER60> No, you can be an ML practitioner with just an intuitive understanding of, say, gradient descent works and you would do fine. You can even pick up that intuitive understanding on a strictly need-to-know basis, when it's needed for learning an ML technique. That's what fast.ai teaches.

For being more than a practitioner, like an implementer of new ML libraries or a researcher, of course you'd need to know more.

</comment></comment><comment><USER61> is Linear Algebra something you need to work with machine learning?

</comment><comment><USER62> This is a good book

</comment><comment><USER63> This is nice.

</comment><comment><USER64> Wow!</comment>


Can A Company Keep An Employee's LinkedIn Account After Employment Ends? - <AUTHOR> http://www.techdirt.com/articles/20111229/03500917224/can-company-keep-employees-linkedin-account-after-theyre-no-longer-employed.shtml <comment><USER2> Reading only the title I would answer: "Of course not", but there are more questions to this.

Why the original owner allowed company to use his own account? By the fact that he handed down password and permissions to the company to operate this account, one could argue that this is not his own account anymore.

However, whoever wins this suit, the right thing to do for LinkedIn would be to just disable/delete the account for misuse. The fact that multiple people inside a company had been sharing the same account, changing person name, picture and position is a bit fishy, and potentially a scam against other linkedin users. I wouldn't be happy to connect to "John D" and write recommendation to him, only to find out that few weeks later this is some "Alex F" I don't know.

<comment><USER3> _By the fact that he handed down password and permissions to the company to operate this account, one could argue that this is not his own account anymore._

I've given colleagues the keys to my appartment, the password to my privately- owned laptop, ipad and phone...

Sometimes these weren't work-related and just happened to involve colleagues, other times they were 100% work related.

I don't believe that giving a colleague access to something of yours indicates it is owned by the company. Hell, why not argue the other way, the company gave me access to all their servers so don't they sort of belong to me now?

<comment><USER4> Yes, but did you give them the keys so that they could hold a meeting with investors in your living room? It seems that she meant for others in the company to use the account as a means of interacting with clients, etc.

I think the lawsuit is ridiculous, but I'd enjoy taking the plaintiff's side in a mock trial.

<comment><USER3> No I didn't, but if I had done that would it make a difference? It's certainly something I would do if the situation came up (and if I didn't live just down the road from our offices).

Certainly I've given access to my blackberry to answer calls from clients before, I still claim I fully own this phone.

<comment><USER4> > No I didn't, but if I had done that would it make a difference?

I think it makes a difference. As an admittedly weak analogy (only one cup of tea so far this morning), I can write off a dinner as a business expense if I dine with a client, but I can't if I'm just taking my family out and talking about business.

<comment><USER3> I can expense a meal I buy for a colleague and myself too, not just if a third party client is involved.

So imagine our offices are busy as hell and there isn't room for one of my colleagues to meet with a client who is coming in. I say "take my keys, there's scotch in the usual place, take him into the living room". Maybe I'd even expense the bottle of scotch afterwards. Would you now argue that the company owns my home because it was used in this way?

<comment><USER4> Depends.

If you were the CEO, and your small house faced a busy street, and had the business name plastered all over it, and you made a habit of inviting customers, coworkers, clients, business partners, etc. - perhaps! (Again, in a mock trial sort of situation, not in one where I'd actually potentially make you homeless.)

But also IANAL, so I'm sure there must be a precedent for this sort of thing, both online and real-state wise.

<comment><USER5> You're way off. Possession, contrary to popular cliché, is not 9/10ths of the law. Property, real or personal, has an owner. In the case of real estate and certain other property (such as a car), that ownership is usually even on record with the county or state.

Short of actual abandonment, which is a narrow, and generally hard-to-prove case, ownership of property in any form cannot be transferred without some sort of contract. Although some property might be transferred by oral contract, that still requires evidence that a contract was formed, which requires a "meeting of the minds". You actually have to prove there was an _intent_ by the owner to transfer the property.

That's a high bar to clear, especially with something as inherently personal as a linkedin account, where any reasonable person would assume the account belongs to the individual, not the corporation.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER6> I've had the experience of having company contact data stolen via a Linkedin account.

I hired a salesman for one of my businesses. He had limited sales experience but I was willing to train. His Linkedin account had maybe on or two dozen contacts. I watched as he systematically added contacts to the account over a year. Nearly all of these contacts came out of our business activities. By the end of the year he was in the hundreds of contacts. The relationship wasn't working out and I had to let him go. He went to work for a competitor. Not only was he armed with the knowledge from this one year "training program" but now he had a who's-who of the industry in his LinkedIn account. Business can really suck sometimes.

Some of these contacts were developed after non-trivial financial and monetary investments. For example, we were spending about $250K per year going to tradeshows. Some of these contacts came from relationships built over making contact with people repeated times at various tradeshows spread over several years. In some cases the tradeshow activity was followed-up by in-person visits at the customer site. One could easily argue that some of the contacts easily cost tens of thousands of dollars to develop.

Who owns these contacts and relationships? I think it is very clear that the business does. I also understand that this is not a trivial problem. These days it is nearly impossible to control the exodus of internal data, be it contacts, code or design data (electronics, mechanical). One personal Dropbox account and an employee can move hard drives worth of data out of your business. Agreements not to do so are only worth the paper they are on. In the end it might just be a matter of personal morals and ethics.

<comment><USER7> Why do you think it's "clear" that the business owns the contacts and relationships? Contacts and relationships form the basis of a salesman's job, much as language and API expertise are the basis of a programmer's job. You may invest considerably in a programmer's training, only to watch him leave for a competitor, but you will have a difficult time claiming ownership of his knowledge and experience.

I think the lesson is that, from a business perspective, you need to guard access to key contacts and companies where possible, if the likely departure of an employee who knows them will cause the business undue hardship. How else would you propose to structure a job where the employee forfeits relationships upon separation? (And how would you get any decent salesperson to accept such a job?)

<comment><USER8> IMHO: relationships are personally unless explicitly called out in a contract. If I met someone, I met them, you can't take that back. If a relationship is that vital to the company then safe guards of some sort need to be in place.

</comment></comment></comment><comment><USER9> Can't LinkedIn clarify this in their terms of service?

As in, only individuals (not organizations) are allowed to create personal accounts, and LinkedIn will treat the person who crated the account as the owner, it is non-transferable, and masquerading as another individual is prohibited? Etc, etc.

<comment><USER10> It would make sense. Airlines had to deal with this issue years ago when frequent flyer programs were introduced.

Companies wanted to get the miles and points of their employees when they traveled for business. Airlines pushed a hard line and said that the miles and points are associated with the person who flies, not the company that pays for the ticket.

</comment></comment><comment><USER11> _"Here's a scenario for you: Your boss encourages you to sign up for a Linkedin profile which you start to actively use in your work as well as socially. The time comes when you and your company part ways for whatever reason. Your boss now says that the account that was set up belongs to the company and you have to give it up. Does it sound like an unlikely scenario? It has happened to lots of people out there and it will happen again"_

[http://theundercoverrecruiter.com/content/social- networking-...](http://theundercoverrecruiter.com/content/social-networking- and-job-change-7-tips-keeping-your-contacts)

<comment><USER12> A lot of staffing firms are suing former employee's over such.

On my own research, you are "safer" if you never allow the company to pay for premium options, you had the LN account prior to employment, and honor your non-compete post-employment.

All very tricky. I no longer use my LN for staffing to avoid this. [http://takethisjobnshoveitblog.com/2010/06/01/employee- sued-...](http://takethisjobnshoveitblog.com/2010/06/01/employee-sued- claiming-activity-on-linkedin-was-violation-of-non-compete-agreement/)

[http://dailyblogma.com/featured/employers-battle-workers- lin...](http://dailyblogma.com/featured/employers-battle-workers-linkedin- contacts/) _In Minnesota, the IT staffing firm TEKsystems Inc. is suing three former employees and their new employer for allegedly violating non- competition, non-solicitation and non-disclosure agreements. What did the employees do? They continued to use their LinkedIn accounts, which they had set up while working at TEKsystems, after they had left the company. The lawsuit alleges the employees used LinkedIn to communicate with TEKsystems clients and contractors in violation of their departure agreements._

[http://thenextweb.com/socialmedia/2011/12/30/think-going- aft...](http://thenextweb.com/socialmedia/2011/12/30/think-going-after-a- twitter-account-is-bad-this-company-wants-employee-linkedin-accounts/)

A former Sr contract recruiter with Microsoft was told she had to give up her LN account by their HR department (cannot find at the moment) after leaving she told them no, given she already had the LN account.

</comment></comment><comment><USER13> Siscom[1] do "information management".

Their page on "end user computing" solutions seems apt:

> _For many businesses and governments, the end-user computing environment has > evolved in an ad-hoc fashion. Many organizations need to refresh and > simplify their environments to bring down the cost of management, improve > availability and reduce time to market for new capabilities. In addition, > companies are looking to improve end-user productivity by upgrading their > desktop and messaging infrastructures to support newly available mobile and > collaboration solutions. For these organizations, piecemeal and one-off > initiatives cannot produce sufficient business value; they need a more > transformational approach to their end-user computing environments._

> _SISCOM works closely with organizations to fill those gaps in experience, > and to provide the methods, tools and tactics to achieve the highest > performance through a transformed end-user computing environment: all at > lower cost, and with greater business payback._

People make mistakes, but come on.

[1] (<http://www.siscom.com.sa/>)

</comment><comment><USER14> As far as the legality goes I would assume LinkedIn gets to decide. It's a 3rd party service, I haven't read their TOS but I'm assuming they own everything and merely permit users to use their service. Sure there are terms but legal ownership is LinkedIn's.

Curious to see how LinkedIn handles the account that's actually been taken over and replaced with a new name. I doubt the idea of one person building a network and then selling the packaged contacts to another person fits at all with LinkedIn's plans for the future. The last think LinkedIn needs right now is people to start goldfarming.

</comment><comment><USER15> The problem here is when you allow an employee's linkedin account to be your CRM.

</comment><comment><USER16> The first rule of working for an organization: You are not the organization. This goes for your time, your resources, your accounts.

People blur this line for various reasons. They're youthful and "motivated"/optimistic. They substitute the organization to lesser or greater extent for a personal life. The organization pressures them to co-opt their personal life in varying manners and degrees.

If an organization wants an account, _they_ should start one. If that's not within the terms of service of the provider, that's the organization's problem.

If you work for an organization that continues to pressure you -- directly or indirectly -- to blur these lines, it's time to start planning for and heading for the exit.

I used to think my "detached" colleagues were unmotivated. In some cases, perhaps; but in others, I've come to see they were perhaps simply wiser and/or more experienced. (Meaning, with the ones who otherwise demonstrated significant intelligence and ability.)

When you work for someone else, it's your _job_. You may -- hopefully do -- greatly enjoy it. Nonetheless, don't forget to understand, demarcate, and respect its boundary -- and your own.

<comment><USER17> Work ethics aside, this situation poses a problem for LinkedIn, since the idea that a company can edit their employees' accounts or create fictitious people is counter to the whole idea of identity-based social networks (quite different than FB or G+ company pages).

Ultimately this points to a problem for social networks, in that "your" page on a social network really isn't "owned" by you, it's just your dossier that you have some rights to edit/control, but is really the property of site/network that hosts it.

This won't change unless regulation/legislation/legal precedent creates real legal protections. Given how much of our lives are "on the Internet" these days, Cyberethics [1] is a growing issue.

[1] <http://en.wikipedia.org/wiki/Cyberethics>

<comment><USER16> I'm suddenly put in mind of the "corporate personhood" debate that's picked up once again in the U.S., recently.

Who is a "person", and what right or permission do or don't they have to be an account holder?

As for your point, it's a reason I'm moving back to self-hosting. I've had enough of "share-cropping".

</comment></comment></comment><comment><USER18> The one resource the world will never run short on is total morons, as evidenced by this story.</comment>


File Integrity 6: Which image format is most resilient? - <AUTHOR> https://eclecticlight.co/2020/04/21/file-integrity-6-which-image-format-is-most-resilient/ <comment><USER3> In general, most compressed file formats don't even bother to be resilient against errors, since error correction mostly runs against the point of compression (to make files as small as possible). Some file formats do have error detection - for example, PNG provides a CRC-32 of every chunk, and compressed formats like Zip and gzip have similar checksums.

If you happen to know (or guess) that a compressed file with a checksum only has 1-2 bytes in error, it is definitely possible to recover the original file through bruteforce. You can probably even do this to a file without a checksum just by testing which possibilities parse correctly.

However, corruption on filesystems rarely takes the form of a small number of bytes; oftentimes the file's metadata gets corrupted such that entire blocks of the file (anywhere from 512 bytes to 64KB) are swapped, dropped or lost. Instead of worrying about corruption-resistant image formats (since storing everything as uncompressed TIFF/BMP is incredibly wasteful), it's probably worth using a more corruption-resistant filesystem instead (i.e. not FAT!).

<comment><USER4> What's surprising to me is that no file system (even ZFS) or database utilize error correction controls. Well aside from 1 bit/byte ECC style codes or raid. There are a few forms of forward error correction [1], which are already widely used in hardware [2]. There are some projects trying for long term fixity of storage [3], but nothing default or built in, afaict.

I assumed RocksDB or some other modern KV engine would include FEC, at least the meta-data. Using DB's on embedded devices reveals how easily (and regularly) a whole DB of data can be corrupted. A few bytes of error and an entire IoT DB can be lost. FEC's can be cpu expensive but not that expensive nowadays.

1: [https://en.wikipedia.org/wiki/Forward_error_correction](https://en.wikipedia.org/wiki/Forward_error_correction) 2: [https://www.electronicdesign.com/technologies/communications...](https://www.electronicdesign.com/technologies/communications/article/21789969/use- forward-error-correction-to-improve-data-communications) 3: [https://github.com/lrq3000/pyFileFixity](https://github.com/lrq3000/pyFileFixity)

<comment><USER5> I would expect that filesystems do not integrate conventional FEC (e.g. block codes), because FEC is poorly suited to the kind of errors that affect filesystems.

FEC works against errors that have a predictable expected error rate, where errors are evenly distributed among the data. This applies to block codes (where FEC will have a maximum decodable error rate per block) and convolutional codes which don't have 'blocks', but still have a maximum local error density they can reliably correct. For media where burst errors are a possibility, FEC is almost always paired with interleaving to increase the likelihood hor a ~uniform distribution of errors.

FEC works pretty well when the problem is "I wrote some data to disk that I can't read back correctly, but I can read the other data back just fine". At the media level, hard drives have used FEC to protect individual bits/sectors for a long time. Likewise at the disk level, RAID 4/5/6 do use FEC to protect against individual disk failures.

Errors experienced at a filesystem level are not usually due to 'random' corruption of the data, but to invalid operations on the data structure e.g. a crash while flushing the data to disk which leaves the filesystem in an inconsistent state. In other words, the filesystem-specific errors are more likely to be "I wrote the wrong data to disk" or "I only wrote some of the data I meant to".

FEC will not defend against that well, because any FEC must be written to the disk along with the data to prevent the FEC+data from getting into an inconsistent state. For example, the 'RAID 5 write hole' (which also happens with RAID 1 and RAID 6) exists because physical disks will not commit data to the disk at exactly the same time even when the write commands are issued at the same time (which mostly won't happen anyway, since write commands tend to be serialized on your SCSI/Fiberchannel/whatever interface to multiple drives). That means there exists a time where the data has been updated but the parity has not (or vice versa), and if a drive fails at that time, the RAID array may not be cleanly recoverable.

So for practical reasons, if you write the wrong data to disk, you'll probably write the wrong FEC too, and FEC won't help recover much. If your allocation table gets corrupted and you write part of one file on top of another, you'll be overwriting the FEC too. For conventional FEC.

BUT... error correction isn't just "conventional FEC". Error correction in general is any time you add redundancy to data to make it more tolerant to errors. In the filesystem space, this happens, but it isn't usually done with parity matricies or convolutional state machines. Instead, redundancy is added by things like log-structured file systems. The 'write what you plan to do, then write the actual data, then write that you successfully did it (in that order)' behavior of log structured file systems does add redundancy and does make error recovery more possible.

In summary, "error correction" in general is used at several levels for file storage:

\- block FEC at the bit level

\- block FEC at the drive level if RAID is used

\- robust CRC error detect + automatic repeat request over your PCIe/NVMe link or over a network connection

\- Redundant metadata in your filesystem journal

but the key is to use the right kind of error correction for the expected error conditions.

<comment><USER6> I suggest you read a bit more about reliable data transmission and FEC. Burst error FECs absolutely available and used (mostly FEC + interleaving in this order).

</comment></comment></comment></comment><comment><USER7> JPEG can be made almost as resilient as an uncompressed format by adding in restart markers.

Restart markers duplicate the JPEG headers within the file, at however many intervals you like, so the bitstream can be, well, restarted, limiting the effect of any corruption.

`jpegtran` can insert them losslessly.

<comment><USER8> Interesting, didn't know. For arbitrary data I found "rsbep".

</comment></comment><comment><USER9> I wrote my little consistency checker ([https://github.com/jwr/ccheck](https://github.com/jwr/ccheck)) specifically for checking my image archives against corruption. I am appalled that in this day and age we still (mostly) use tools and filesystems that have no way of knowing if data has been corrupted.

Before I wrote the tool, I searched online and couldn't find anything that would be small, simple, easily maintained, and would last for at least 10 years.

I now use it with all my archives, which means I know if a given replica is correct.

</comment><comment> mark-r I like the idea of a utility that can add ECC to any arbitrary file. You'd need to ensure you had an uncorrupted copy of the utility that you could use in the future though, or the images become unreadable even if they're uncorrupted themselves - unless the ECC is contained in a sidecar file.

<comment><USER10> PAR2 or most file compression formats can fit the bill. Of course if it's going to stay put a filesystem that does this might be more convenient.

<comment> mark-r Yes, it might be more convenient when built into a filesystem. But now you must keep alive an OS that can handle that filesystem. That seems like a much greater burden for the long haul.

</comment></comment></comment><comment><USER11> If you're afraid of file corruption, install par2 with homebrew.

My procedure for pictures is that monthly, I copy them off of my phone onto my storage. Then I run par2 in that directory to add a parity file. That all goes onto the backup.

<comment><USER12> I do the same. To help with 'maintenance' (adding par to newly added files, checking and handling integrity for existing ones) I wrote a small utility you may like: [https://github.com/brenthuisman/par2deep](https://github.com/brenthuisman/par2deep)

<comment><USER11> Very nice, thanks!

</comment></comment></comment><comment><USER13> Genuine question: why would you want that? Isn't that the role of the file system or the data transport?

</comment><comment><USER14> This might be a stupid question, but with a container format such as HEIF could you simply store multiple identical copies of an image inside the container? If one image gets corrupted you have the other to fall back on.

Two HEIF or AVIF images should be about the same size as a single JPEG too.

<comment><USER15> It's both much less wasteful and much more useful to add an error correction (FEC) block instead. With random errors spread across two copies, how do you know which chunks of each are correct? Sure, you could use a Merkle tree or similar, but FEC is a better solution. For size penalties of order 10% in practical use, FEC blocks can detect and correct errors not just in the data but in the FEC blocks themselves too.

Their biggest downside is that their computational complexity can be high. This is only a problem during creation and decoding on error; if the data is checksummed, you only need to take the slow path and touch the FEC block if an error is encountered. This is a great tradeoff for archival uses, and good for many other uses.

</comment></comment><comment><USER16> Is "vandal" by chance an open source utility? I'd like to test some other files and formats.

<comment> mark-r It doesn't sound like it would be hard to write your own. Not sure how useful it is though, since it doesn't make any attempt to replicate the kind of errors you would see in the wild.

</comment></comment><comment><USER17> Would you outperform uncompressed TIFF with sufficient copies of a compressed file? Or compressed file plus parity file?

</comment><comment> mark-r There used to be a format that was highly resilient to errors. It was called film.

<comment><USER18> Of course singular - or linear, say, from bad projection device - defects in that are rather final, and this format has a host of inconveniences like fire hazard, inability to get transferred over networks, comparatively huge costs of copying, large physical size demands etc.

We can perhaps use a convenient digital format for images and add to it, ahem, anti-corruption measures. Single file is still going to be vulnerable - <USER2> though to a less extent - if you, say, add some Reed-Solomon protection; but now you can use distributed networks the size of a planet, or even more. Organization of such a storage could be an interesting problem - at least partially solved already.

<comment> mark-r I'm not saying film is perfect. But it's an often overlooked choice that has worked well for the last 50 years.</comment></comment></comment>


US court rules against FCC on net neutrality - <AUTHOR> http://finance.yahoo.com/news/US-court-rules-against-FCC-on-apf-78990100.html?x=0&.v=4

<comment><USER2> The problem with what the FCC did here was to take upon itself the role of having a roving commission to regulate carrier activities on the internet.

By law, the FCC is an administrative agency - that is, it is charged with administering an act of Congress that gives it express authority to regulate an industry. An administrative agency can only regulate things over which it has been given such authority. In this case, the FCC did not have any _express_ authority in any statute empowering it to impose rules of net neutrality concerning a carrier's attempt to restrict peer-to-peer usage across the carrier's network. The FCC admitted as much and argued that it had so-called "ancillary" authority to make the ruling that it did in this case. In other words, it claimed _implied_ authority to act as it did. The court disagreed.

The Second Circuit's decision goes through a painstaking analysis of technical doctrines of administrative law relating to the FCC's attempt to justify its decision based on "ancillary" authority. The court concluded that the FCC, in failing to tie this assertion of authority to any "statutorily mandated responsibility," had overstepped its proper function. Accordingly the court struck the order down as illegal.

A reasonably informed legal discussion of the case appears here ([http://www.law.com/jsp/article.jsp?id=1202447593360&Appe...](http://www.law.com/jsp/article.jsp?id=1202447593360&Appeals_Court_Finds_FCC_Lacks_Authority_to_Regulate_Internet)). As noted in that piece, this is a "decision with far-reaching implications for the future of the Internet and the role of the Federal Communications Commission."

Whatever the merits of net neutrality, the FCC cannot disregard the rule of law. Congress _can_ empower the FCC to regulate the net in this manner and that is where this battle should be fought.

<comment><USER3> Can I just say that having grellas active here on HN is one of the best developments of the last year for this community? Thanks, sir.

<comment><USER4> Concur!

</comment></comment></comment><comment><USER5> This is nuts. Maybe the court ruling is correct that the FCC lacks authority to enforce net neutrality, but we have to somehow figure out how to put an end to this madness.

It is a case that has much greater implications than just unfair treatment of traffic, and no I do not mean interference with the national broadband initiative. I am speaking of the precedent it sets.

Unfortunately, still a great deal of the country has no idea about this issue, what this means and it will go unnoticed. Even if it does come into the lime- light of issues (doubtful), because of the lack of understanding by the majority of constituents in this country it will be such an easy topic to spin.

Depressing.

<comment><USER4> I hope this thing gets settled in congress, as it has been suggested and attempted by civil-rights groups. Whatever precedence gets set, it will spread to Europe too, and I really hope a mistake like this doesn't determine the way westerners get to use internet in the future :-S

</comment></comment><comment><USER6> If anyone here supports Net Neutrality, I challenge you to explain why you want the FCC to have more power. Do you approve of its regulation of content? Would you like to see such regulation occur on the internet?

The FCC regulating the internet is a sure path to speech restrictions (censorship, etc.) and taxation of the internet. Why? Because it becomes a one stop shop for cultural conservatives who want the internet regulated.

<comment><USER7> Slippery slope arguments are almost never persuasive.

Giving the FCC a mandate to enforce openness is in no way giving it a mandate to enforce censorship. The world doesn't exist on a 1-dimensional plane between zero FCC power and ultimate FCC power. There's other stuff going on.

<comment><USER6> What else is going on? Currently there are lots of entrenched lobbyists who influence FCC policy. This is why things like nipple slips or use of the word "ass" on prime time TV are a big deal.

Do you think those lobbyists wouldn't immediately start lobbying for internet "decency" regulations the minute the FCC gained additional power to regulate the internet?

Such restrictions will start in obvious ways on largely uncontroversial subjects -- perhaps a law that appears at first to target only pedophiles or hard core pornography viewing by minors -- then more and more ambiguity creeps in until you have horrible censorship going on.

Have you read quotes by recent FCC chairmen? They generally consider themselves to be stewards of good taste. This is scary stuff.

<comment><USER8> Grandolf, assuming the FCC doesn't fall down that slope, do you support forced net neutrality?

Remember in the case that censorship does become an issue you can fight that, but there's nothing about net neutrality that gives the FCC that power. I want net neutrality and I'll be quite specific about what the FCC can do about that. Plus if the FCC is already headed towards the censorship department, blocking net neutrality isn't going to stop that.

<comment><USER6> We have net neutrality today in fact, just not in law.

This battle is about big firms duking it out in the court of public opinion by using FUD about a non-free internet.

Business is important too, and if someone can start a business delivering 100MB fiber to my door for $20 per month but Bing handles all search traffic... so be it. I might very possibly want to buy that service.

Without the _freedom_ to have firms make such deals and offer such services, we're giving up a lot of potential innovation (in services and in pricing bundles).

I don't think there is anything inherently bad about the sorts of contractual restrictions that may arise on the internet vs the rest of the universe.

<comment><USER8> Good point. But also consider the same thing can happen on the other extreme.

For example say that internet service provider gains a monopoly through its cheap service. Now it can unfairly suppress new search engines by only allowing Bing.

The thing about the net neutrality law would be that it is equally applied across the board. As long as it doesn't unfairly benefit a specific group, it's fair. And it benefits people in general because it prevents companies from leveraging power from short term tactics which have long term negative repercussions on society.

</comment></comment></comment></comment></comment></comment><comment><USER9> Let's just keep in mind, the court is not ruling against Net Neutrality itself, only that the FCC in particular does not have the authority/jurisdiction to enforce Net Neutrality. A legislative action, or possibly even an executive order, could mandate net neutrality and/or expand the FCC's powers to make that decision on its own. Time for you web startup types to call up your senator and express concern about your small business being shafted. Put it in terms they can understand: if my web traffic is a second class citizen and my page loads 0.1 second slower, I lose $X,000 a month.

<comment><USER6> Small businesses are the ones most likely to benefit from preventing net neutrality -- via an acquisition by a content provider like Comcast or Verizon or AOL, etc.

It is the companies with > 30% market share that stand to lose, since ISPs would be able to pick winners and fragment the market further. This is why Google supports net neutrality.

<comment><USER10> If I write on my blog about Comcast doing something bad, what is stopping Comcast from simply blocking all connections to my blog?

<comment><USER11> Further - what is preventing them from changing the html and removing their name from whatever they serve?

<comment><USER6> Nothing, yet they don't do that. I think you proved my point.

<comment><USER12> Just like MSNBC didn't block Wikileaks when Wikileaks made their parent company, GE, look bad.

Oh wait, they _did_ do that.

<comment><USER6> How is that relevant to net neutrality?

<comment><USER12> What if GE was an ISP and not just a search engine?

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER13> Can we revoke their rights to location based monopolies?

<comment><USER5> Yes, in a way. You would have to work with the mayor or counsel members of your township in the U.S. They are responsible for controlling what companies may lay wire and services in your township.

Note: Townships get a nice kickback per subscriber within their borders. Yes, another money game. And yes, that kickback is a negotiable price-per- subscriber...

</comment></comment><comment><USER6> I think it's important to point out that firms like Google which have supported net neutrality, have done so to support their own business models.

Microsoft 'cloned' google and created Bing in a very short time. As computing power increases this task gets easier and easier. Plausibly, comcast could create its own "google" and direct all search traffic there. It could sell ads and use the ad revenue to lower monthly subscription fees to its customers.

If the service works well enough (i.e., if google is actually clonable) then most consumers wouldn't care. That's precisely Google's concern. Why should it give up its revenue so that Comcast can start its own revenue stream?

Think about Google's share price and all the billions of dollars of ad revenue. That's what this issue is about. Google has invested hugely in becoming "infrastructure" but the price of cloning that infrastructure is quickly decreasing... so Google is trying to stop it by any means necessary.

<comment><USER14> _Plausibly, comcast could create its own "google" and direct all search traffic there. It could sell ads and use the ad revenue to lower monthly subscription fees to its customers._

Except they wouldn't do that. They would use QoS to make sure their search engine was the only one usable to their customers and then they would charge extra for access to the search engine as an added feature.

<comment><USER6> What's wrong with that? Do you really think there would be so little benefit from building a good search engine that Comcast would build a lousy one? They could even license one from any number of third parties.

<comment><USER15> Well sure, that's the point - there would be little benefit from building a good search engine if you couldn't choose a different one. Google has to be great because otherwise somebody will get all its traffic. A monopoly can give lousy service because you can't leave.

<comment><USER6> True, but if the search engine is lousy there won't be much traffic. Google has gone to great lengths to build a service that lots of us wouldn't want to live without. So why worry?

</comment></comment></comment></comment></comment><comment><USER16> Ideally, whoever grants the telcos monopoly power (Is it the FCC that does that? I don't know.) should be able to enforce net neutrality. If that's the FCC, they should be able to enforce it; if it's someone else, they should.

<comment><USER6> So you are arguing that the FCC should have more power? Fuck I'd better hurry up and swear on the internet while it's legal.

</comment></comment><comment><USER17> This situation looks similar to natural gas transportation. There needs to be separate companies who provide internet access and who own and operate transmition cables. Cable operators would not care about what kind of traffic flows through their wires as long as it's paid for. ISP's would rent bandwidth from cable providers and re-sell it to customers. Since there will be less of an entry barrier to become and ISP there would be more competition and customers in an area would not depend on the whim of one company.

</comment><comment><USER18> I may not be up to speed with the whole argument, but is there a middle ground that neither party seems to want to meet at?

Net Neutrality proponents seem to want to do away with QOS entirely, but this seems to be on the grounds of ensuring the big operators don't use it for evil.

QOS can be an extremely beneficial thing. How many times has anyone been on a Skype call when the video dropped or the audio came through garbled? How many of you want to ensure that when you're playing Call of Duty that the very milisecond you hit the trigger for a headshot, your packet doesn't get dropped and you miss your target? On the flip side, how many of you don't care if an email you send gets sent that very milisecond or one or two seconds later? And who would care if a podcast you have downloading from iTunes overnight takes 15 minutes or 20 minutes?

This kind of QOS ensures that video, audio, and gameplay get priority over other forms of data that are not as time-sensitive.

Please correct me if I'm wrong, because this overall issue does seem a bit complex.

<comment><USER6> You are right. The reason Google supports net neutrality (for example) is to prevent Comcast from making a deal with Vimeo to offer QOS on Vimeo videos, allowing them to play at noticeably higher quality.

Comcast could even make a deal with a yet unheard of startup and suddenly Google would lose lots of market share to an upstart who was willing to accept Comcast's terms.

<comment><USER18> YouTube isn't exactly a shining example of high-quality grade video, but I totally get your point.

I'm just hoping that a tiering of QOS from a technical standpoint is not entirely ignored. Perhaps it's something already being addressed by Net Neutrality proponents.

Edit: Here's an article from a Cisco podcast that discusses this same issue I've mentioned: [http://www.ciscohandsontraining.com/2009/09/need-for-qos- ver...](http://www.ciscohandsontraining.com/2009/09/need-for-qos-versus-net- neutrality.html)

<comment><USER6> I hope so too. Surprisingly, I was watching youtube 1080p content via AppleTv the other day and was quite impressed. I certainly know why Comcast opposes net neutrality :)

</comment></comment></comment></comment><comment><USER19> All I can say is that I feel sorry for you in the US. In Norway, I get 16MBit DSL at my cottage (yes, half an hours walk from the nearest dirt road). We don't even have any net neutrality laws, and there are only two nationwide DSL providers.

<comment><USER20> The general bandwidth you get is irrelevant. The central question from the perspective of net neutrality is whether there is any traffic shaping upstream of you.

<comment><USER21> Yeah, well, I still envy him his ability not to live in town.

</comment></comment></comment><comment><USER11> This commentary is interesting (and non-alarmist): [http://balkin.blogspot.com/2010/04/whats-next-for-network- ne...](http://balkin.blogspot.com/2010/04/whats-next-for-network- neutrality.html)

</comment><comment><USER22> WTF??? I may not be a fancy big-city lawyer, but I would think that even someone as dense as a Supreme Court Justice (and obtuseness seems to be a job requirement these days), let alone a clearer-thinking lower court judge, should be able to see the clear applicability of common carrier statute and precedent here. Unless and until the ISPs actually create, procure and provide exclusive content and _only_ their own content (goods) via their delivery means, they are bound by common carrier. That goes back to what, Domesday?

<comment><USER23> Don't the FCC's own regulations say that ISPs aren't common carriers? Don't recent acts of Congress (e.g. the Telecommunications Act) overrule ancient common law precedents?

<comment><USER22> Only if the adjudicators agree with statute and regulation -- and there is clear parallel with every other application of common carrier law, so what the courts should be finding is that the ISPs are _de facto_ and _de jure_ common carriers, despite legislation to the contrary. To put it technically, they are the same in essence, and the rule of law says that when two things are the same in essence, they cannot be treated differently.

</comment></comment></comment><comment><USER24> If you support net neutrality, but are against FCC content censorship, you should applaud this ruling. It checks the agency's ability to conjure new powers for itself. That overreach is the mechanism by which "neutrality" could be subtly redefined, over time, by the next generation of righteous busybodies.

Without strict by-the-letter-of-the-law limits, future populist regulators could easily decide that packets aren't truly "neutral" unless they advance neutrality, equality, and justice for all Americans. Packets which contain hate speech, pornography, anonymous speech, content inaccessible to the handicapped, political campaigning beyond the spending limits of the FEC, or an insufficient proportion of 'underrepresented' viewpoints all twist the net against the public interest. So why shouldn't fair-minded regulators issue extra rules to protect us? Demand more proof of compliance and 'good faith' from operators to stay in business? And once an agency is reviewing business practices for compliance with political ideals, where does it stop?

The diversity of political viewpoints available from a mass media is roughly inversely correlated with the amount of government regulation. Broadcast TV -- requiring FCC licenses for using the "public" airwaves -- has the tamest commentary. Cable TV -- federally-unregulated but locally-licensed, with a lot of operator overlap with regulated broadcasting -- is slightly more diverse. Books and periodicals -- there is no FCC of print, thankfully -- are the most diverse.

Right now the internet is most like books and periodicals. Anything that gives the FCC review power over net businesses will push it to be tamer, more like cable or broadcast. (This is even if there are no explicit content regulations; simply trying to keep the bipartisan regulators on your good side with regard to future disputes means self-censorship.)

Even if you think that's a risk worth taking for "neutrality", you ought to at least require that authority to come from explicit legislation. Get a clear definition of new rules and authorities in writing; have that definition pass Congress and constitutional review. Don't let an appointed agency of 3 Dems/2 Reps (and sometimes, 3 Reps/2 Dems) assign themselves this new authority.

</comment><comment><USER25> This was a clear victory for property rights. They own the equipment; they should get to control how their property is used (in accordance with their contracts).

The problem here isn't the lack of regulation, it's the lack of competition (as other's have pointed out). No ISP can afford to be shackling their users in the face of unshackled competition.

I hope we'll have a vibrant 4G broadband market in the next 5 years.

<comment><USER26> Then I want my chunk of the property back, the amount that the taxpayers shelled out to subsidize the telcos laying down said equipment, and which Comcast and others readily accepted.

<comment><USER27> If parent means to assert or imply that taxpayers in the U.S. paid for more than a tiny fraction of the costs of laying down the copper and fiber over which residential internet service is delivered in the U.S., I would like references supporting that assertion. The argument parent probably wants is that in much or most of the U.S., local governments granted the telcos and cable companies a government-created monopoly in the local jurisdiction, in exchange for which it is reasonable for government to expect the companies to submit to certain rules, especially since the rules are essentially the same as the rules by which monopoly power in the railroad, telegraph and telephone industries were somewhat-successfully ameliorated, and especially since there exists in the fundamental architecture of the internet a bright line in the form of the boundary between the IP layer (and everything below it) and services that ride atop it that can serve as the target of network-neutrality rules, and especially since most of the architects or "technical founding fathers", many of whom are libertarians and lovers of private-property rules, BTW, of the internet support the imposition of network-neutrality rules. Reader who doubt the existence of a "bright line" at the IP layer should make sure they have read the classic paper "The End-to-end argument in system design" and the history of the internet since the publication of that paper. (In other words, the "quality of service" arguments used by the telcos etc to argue the impracticality of network-neutrality rules have always been on the losing side of debates for the first 30 or 35 years of the history of the internet, and consequently suffer from a high burden of proof.)

For these reasons, even a lover of property rights and constitutional limits on government like myself can support network neutrality after having familiarized myself with the pro-network-neutrality arguments by people with a track record for good calls in internet design and internet policy and with the history of industries with very strong networks effects, like the railroad, telegraph, telephone, cable and residential-internet-service industries.

And I have to say that the largeness of the number of opinions arrived at with almost no knowledge in this comment section is very depressing.</comment></comment></comment>


Experimental Philosophy: The New New Philosophy - <AUTHOR> http://www.nytimes.com/2007/12/09/magazine/09wwln-idealab-t.html?ex=1354856400&en=42f8ef6876e20ada&ei=5090&partner=rssuserland&emc=rss

<comment><AUTHOR> I noticed the same gig in Pittsburgh, but they seemed to have a division there between the xphi's and armchairs. It seemed weird to me that the xphi's could do science and still call themselves philosophers. Why not just call themselves scientists? That's the problem with philosophy these days, all the good questions have been taken. So they're stuck doing literary analyses.

<comment><USER2> All the interesting questions left are epistemological ones about language games, so literary works are a sort of test data.

And, yes, there are lot of hacks writing badly about these questions, but in what fields are there not a preponderance of posers?

</comment></comment><comment><USER3> _[Philosophers assert that a certain idea is Obvious]: well, that may be so at Princeton or Rutgers. On the other side of the planet, it might seem [that it isn't]. What should philosophers make of that?_

That their discipline is the impacted wisdom tooth of the modern academy.

<comment><USER4> Well the big push in xphi is coming from places like Rutgers, where guys like Stich, Maudlin, Goldman, just do very theoretical science.</comment></comment>