text:
Today I am running an experiment and opening my blog to machine learning questions.
Please ask questions in the comments section.
I will try to answer them in the blog over the weekend of the next week or so.
Thanks in advance.
Today I am running an experiment and opening my blog to machine learning questions.
Please ask questions in the comments section.
I will try to answer them in the blog over the weekend of the next week or so.
Thanks in advance.
Another Holiday Blog: Feedback and Questions are very welcome.
I have a client who is using Naive Bayes pretty successfully, and the subject has come up as to 'why do we need fancy machine learning methods?' A typical question I am asked is:
Why do we need Regularization ?
What happens if we just turn the regularizer off ?
Can we just set it to a very small, default value ?
After all, we can just measure what customers are doing exactly. Why not just reshow the most popular items, ads, etc to them. Can we just run a Regression? Or Naive Bayes ? Or just use the raw empirical estimates?
Do we Need Hundreds of Classifiers to Solve Real World Classification Problems?
Sure, if you are simply estimating historical frequencies, n-gram statistics, etc, then a simple method like Naive Bayes is fine. But to predict the future..
the Reason we need Regularization is for Generalization.
After all, we don't just want to make the same recommendations over and over.
Unless you are just in love with ad re-targeting (sic). We want to predict on things we have never seen.
And we need to correct for presentation bias when collecting data on things we have seen.
Most importantly, we need methods that don't break down unexpectedly.
In this post, we will see why Regularization is subtle and non-trivial even in seemingly simple linear models like the classic Tikhonov Regularization. And something that is almost never discussed in modern classes.
We demonstrate that 'strong' overtraining is accompanied by a Phase Transition–and optimal solutions lies just below this.
The example is actually motivated by something I saw recently in my consulting practice–and it was not obvious at first.
Still, please realize–we are about to discuss a pathological case hidden in the dark corners of Tikhonov Regularization–a curiosity for mathematicians.
See this Quora post on SVD in NLP for some additional motivations.
Let's begin when Vapnik [1], Smale [2], and the other great minds of the field begin:
(Regularized) Linear Regression:
The most basic method in all statistics is linear regression
It is attributed to Gauss, one of the greatest mathematicians ever.
One solution is to assume Gaussian noise and minimize the error.
The solution can also be constructed using the Moore-Penrose PseudoInverse.
If we multiply by to the left on both sides, we obtain
The formal solution is
which is only valid when we can actually invert the data covariance matrix .
We say the problem is well-posed when
Otherwise we say it is singular, or ill-posed [1].
In nearly all practical methods, we would not compute the inverse directly, but use some iterative technique to solve for . In nearly all practical cases, even when the matrix seems invertible, or even well posed, it may have numerical instabilities, either due to the data or the numerical solver. Vapnik refers to these as stochastically ill posed problems [4]
Frequently XTX can not be inverted..or should not be inverted..directly.
A trivial solution is to simply 'regularize' the matrix by adding a small, non-zero value to the diagonal:
Naively, this seems like it would dampen instabilities and allow for a robust numerical solution. And it does…in most cases.
If you want to sound like you are doing some fancy math, give it a Russian name; we call this Tikhonov Regularization [4]:
This is (also) called Ridge Regression in many common packages such as scikit learn.
The parameter is the regularization parameter values, usually found with Cross Validation (CV). While a data science best practice, CV can fail !
Grid searching , we can obtain the infamous L-curve:
The L-curve is a log-log plot of the the norm of the regularized solution vs. the norm of the residual. The best lies somewhere along the bottom of the L, but we can't really tell where (even with cross validation).
This challenge of regularization is absent from basic machine learning classes?
The optimal is actually hard to find. Cross Validation (CV) only works in ideal cases. And we need a CV metric, like , which also only works in ideal cases.
We might naively imagine can just be very small–in effect, turning off the regularizer. This is seen in the curve above, where the solution norm diverges.
The regularization fails in these 2 notable cases, when
Today we will examine this curious, spurious behavior in case 2 (and look briefly at 1)
As von Neumann said once, "With four parameters I can fit an elephant, and with five I can make him wiggle his trunk"
To understand where the regularization can fail, we need to distinguish between the cases in which Ridge Regression is commonly used.
Say we have features, and instances. We distinguish between the High Bias and High Variance regimes.
This regime is overdetermined, complicated models that are subject to overtraining. We find multiple solutions which satisfy our constraints.
Most big data machine learning problems lie here. We might have 100M documents and 100K words. Or 1M images and simply all pixels as the (base) features.
Regularization lets us pick the best of solution which is the most smooth (L2 norm), or the most sparse (L1 norm), or maybe even the one with the least presentation bias (i.e. using Variance regularization to implement Counterfactural Risk Minimization)
This is the Underdetermined regime, and any solution we find, regularized or not, will most likely generalize poorly.
When we have more features than instances, there is no solution at all (let alone a unique one)
Still, we can pick a regularizer, and the effect is similar to dimensional reduction. Tikhonov Regularization is similar truncated SVD (explained below)
Any solution we find may work, but the predictions will be strongly biased towards the training data.
But it is not only just sampling variability that can lead to poor predictions.
In between these 2 limits is an seemingly harmless case, however, this is really a
This is a rare case where the number of features = the number of instances.
This does come up in practical problems, such as classifying a large number of small text phrases. (Something I have been working on with one of my larger clients)
The number of phrases ~ the number of unique words.
This is a dangerous case … not only because it seems so simple … but because the general theory breaks down. Fortunately, it is only pathologicial in
We examine how these different regimes behave for small values of
Let's formalize and consider the how the predicted accuracy behaves.
We analyze our regression under Gaussian noise . Let
where is a Gaussian random variable with unit mean and variance
This simple model lets us analyze predicted Accuracy analytically.
be the optimal Estimate, and
be the Ground Truth, and
be expected (mean).
We would like define, and the decompose, the Generalization Accuracy into Bias and Variance contributions.
Second, to derive the Generalization Error, we need to work out how the estimator behaves as a random variable. Frequently, in Machine Learning research, one examines the worst case scenario. Alternatively, we use the average case.
Define the Estimation Error
Notice that by Generalization Error, we usually we want to know how our model performs on a hold set or test point :
In the Appendix, we work out the generalization bounds for the worst case and the average case. From here on, however, when we refer to Generalization Error, we mean the average case Estimation Error (above).
For Ridge Regression, the mean estimator is
and its variation is
We can now decompose the Estimation Error into 2 terms:
, where
is the Bias, and
is the Variance,
and examine each regime in the limit
The Bias is just the error of the average estimator:
and the Variance is the trace of the Covariance of the estimator
We can examine the limiting behavior of these statistics by looking at the leading terms in a series expansion for each. Consider the Singular Value Decomposition (SVD) of
Let be the positive singular values, and be the right singular vectors.
We can write the Bias as
With no regularization, and the number of features exceeds the number of instances, there is a strong bias, determined by the 'extra' singular values
Otherwise, the Bias vanishes.
So the Bias appears to only matter in the underdetermined case.
(Actually, this is misleading; in the overdetermined case, we can introduce a bias when tuning the regularization parameter too high)
In contrast, the Variance
can diverge if the minimal singular value is small:
That is, if the singular values decrease too fast, the variance can explode.
And this can happen in the danger zone
When the variance can be infinite!
Which also means the Central Limit Theorem (CLT) breaks down…at least how we normally use it.
Traditionally, the (classical) CLT says that the infinite sum of i.i.d. random variables converges to a Gaussian distribution–when the variance of the is finite. We can, however, generalize the CLT to show that, when the variance is infinite, the sum converges to a Levy or -stable, power-law distribution.
These power-law distributions are quite interesting–and arise frequently in chemistry and physics during a phase transition. But first, let's see the divergent behavior actually arise.
Ryota Tomioka [3] has worked some excellent examples using standard data sets
[I will work up some ipython notebook examples soon]
The Spambase dataset is a great example. It has features, and instances. We run a Ridge Regression, with , with .
As the data set size increases, the accuracy sharps to zero at , and then increases sharply again, saturating at $latex N_{f}\sim 10^{3} $.
Many other datasets show this anomalous behavior at , as predicted by our analysis above.
Where Ridge Regression fails, Logistic Regression bounces back. Below we show some examples of RR vs LR in the danger zone
Logistic Regression still has problems, but it no where near as pathological as RR.
The reason for this is subtle.
Traditional theory (very losely) bounds for quantities like the generalization error and (therefore) the variance.
We work out some of examples in the Appendix.
Clearly these bounds don't work in all cases. So what do they really tell us? What's the point?
These theories gives us some confidence that we can actually apply a learning algorithm–but only when the regularizer is not too small and the noise is not too large.
They essentially try to get us far away from the pathological phase transition that arises when the variance diverges.
In practice, we would never just cross our fingers set .
We pick a hold out set and run Generalized Cross Validation (GCV) . Yet this can fail when
Hansen [5, 7] has discussed these issue in detail…and he provides an alternative method, the L-curve, which attempt to balance the size of the regularized solution and the residuals.
For example, we can sketch the relative errors for the L-curve and GVC method (see 7). Above we see that the L-curve relative errors are more Gaussian than GCV.
In other words, while most packages provide a small choice of regression metrics; as data scientists, the defaults like may not be good enough. Even using the 2-norm may not represent the errors well. As data scientists, we may need to develop our own metrics to get a good for a regression. (or maybe just use an SVM regression).
Hansen claims that,"experiments confirm that whenever GCV finds a good regularization parameter, the corresponding solution is located at the corner of the L-curve." [7]
This means that the optimal regularized solution lives 'just below' the phase transition, where the norm diverges.
What do we mean by a phase transition?
The simplest thing is to we plot a graph and look a steep cliff or sharp change.
Here, generalization accuracies drops off suddenly as
In a physical phase transition,the fluctuations (i.e. variances and norms) in the system approach infinity.
Imagine watching a pot boil
We see bubbles of all sizes, small and large. The variation in bubble size is all over the map. This is characteristic of a phase transition: i.e. water to gas.
When we cook, however, we frequently simmer our foods–we keep the water just below the boiling point. This is as hot as the water can get before it changes to gas.
Likewise, it appears that in Ridge Regression, we frequently operate at a point just below the phase transition–where the solution norm explodes. And this is quite interesting. And, I suspect, may be important generally.
In chemical physics, we need special techniques to treat this regime, such as the Renormalization Group. Amazingly, Deep Learning / Restricted Boltzmann Machines look very similar to the Variational Renormalization Group method.
In our next post, we will examine this idea further, by looking at the phase diagram of the traditional Hopfield Neural Networ, and the idea of Replica Symmetry Breaking in the Statistical Mechanics of Generalization. Stay tuned !
1. Vapnik and Izmailov, V -Matrix Method of Solving Statistical Inference Problems, JMLR 2015
2. Smale, On the Mathematical Foundations of Learning, 2002
3. https://github.com/ryotat
4. http://ttic.uchicago.edu/~gregory/courses/LargeScaleLearning/lectures/proj_learn1.pdf
5. The L-curve and its use in the numerical treatment of inverse problems
6. Discrete Ill-Posed and Rank Deficient Problems
7. The use of the L-curve in the regularization of discrete ill-posed problems, 1993
when ,
be the data covariance matrix. And , for convenience, let
The Expected value of the optimal estimator, assuming Gaussian noise, is given using the Penrose Pseudo-Inverse
The Covariance is
The regularized Covariance matrix arises so frequently that we will assign it a symbol
We can now write down the mean and covariance
[more work needed here]
Consider the Generalization Error for a test point
We can now obtain some very simple bounds on the Generalization Accuracy (just like a bona-a-fide ML researcher)
where we assume the 'covariance*' is
We can use the SVD approach to make stronger statements about our ability to solve the inversion problem
I will sketch an idea called the "Discrete Picard Condition" and provide some intuition
Write as a sum over singular vectors
Introduce an abstract PseudoInverse, defined by
Express the formal solution as
Imagine the SVD of the PseudoInverse is
and use it to obtain
For this expression to be meaningful, the SVD coefficients must decay, on average, faster than the corresponding singular values .
Consequently, the only coefficients that carry any information are larger than the noise level. The small coefficients have small singular values and are dominated by noise. As the problem becomes more difficult to solve uniquely, the singular values decay more rapidly.
If we discard the singular components, we obtain a regularized, unsupervised solution called
which is similar in spirit to spectral clustering.
Deep Learning is amazing. But why is Deep Learning so successful? Is Deep Learning just old-school Neural Networks on modern hardware? Is it just that we have so much data now the methods work better? Is Deep Learning just a really good at finding features. Researchers are working hard to sort this out.
Recently it has been shown that [1]
Unsupervised Deep Learning implements the Kadanoff Real Space Variational Renormalization Group (1975)
This means the success of Deep Learning is intimately related to some very deep and subtle ideas from Theoretical Physics. In this post we examine this.
An AutoEncoder is a Unsupervised Deep Learning algorithm that learns how to represent an complex image or other data structure . There are several kinds of AutoEncoders; we care about so-called Neural Encoders–those using Deep Learning techniques to reconstruct the data:
The simplest Neural Encoder is a Restricted Boltzman Machine (RBM). An RBM is non-linear, recursive, lossy function that maps the data from visible nodes into hidden nodes :
The RBM is learned by selecting the optimal parameters that minimize some measure of the reconstruction error (see Training RBMs, below)
RBMs and other Deep Learning algos are formulated using classical Statistical Mechanics. And that is where it gets interesting!
In machine learning (ML), we map (visible) data into (hidden) features
The hidden units discover features at a coarser grain level of scale
With RBMs, when features are complex, we may stack them into a Deep Belief Network (DBM), so that we can learn at different levels of scale
and leads to multi-scale features in each layer
Deep Belief Networks are a Theory of Unsupervised MultiScale Feature Learning
We call a flow map
eIf we apply the flow map to the data repeatedly, (we hope) it converges to a fixed point
Notice that we usually expect to apply the same map each time , however, for a computational theory we may need more flexibility.
The simplest example of a flow map is the simple linear map
so that
where C is a non-negative, low rank matrix
We have seen this before: this leads to a Convex form of NonNegative Matrix Factorization NMF
Convex NMF applies when we can specify the feature space and where the data naturally clusters. Here, there are a few instances that are archetypes that define the convex hull of the data.
Amazingly, many clustering problems are provably convex–but that's a story for another post.
Near a fixed point, we commonly approximate the flow map by a linear operator
This lets us capture the structure of the true data manifold, and is usually described by the low lying eigen-spectra of
In the same spirit, Semi & Unsupervised Manifold Learning, we model the data using a Laplacian operator , usually parameterized by a single scale parameter .
These methods include Spectral Clustering, Manifold Regularization , Laplacian SVM, etc. Note that manifold learning methods, like the Manifold Tanget Classifier, employ Contractive Auto Encoders and use several scale parameters to capture the local structure of the data manifold.
In chemistry and physics, we frequently encounter problems that require a multi-scale description. We need this for critical points and phase transitions, for natural crashes like earthquakes and avalanches, for polymers and other macromolecules, for strongly correlated electronic systems, for quantum field theory, and, now, for Deep Learning.
A unifying idea across these systems is the Renormalization Group (RG) Theory.
Renormalization Group Theory is both a conceptual framework on how to think about physics on multiple scales as well as a technical & computational problem solving tool.
Ken Wilson won the 1982 Nobel Prize in Physics for the development and application of his Momentum Space RG theory to phase transitions.
We used RG theory to model the recent BitCoin crash as a phase transition.
Wilson invented modern multi-scale modeling; the so-called Wilson Basis was an early form of Wavelets. Wilson was also a big advocate of using supercomputers for solving problems. Being a Nobel Laureate, he had great success promoting scientific computing. It was thanks to him I had access to a Cray Y-MP when I was in high school because he was a professor at my undergrad, The Ohio State University.
Here is the idea. Consider a feature map which transforms the data to a different, more coarse grain scale
The RG theory requires that the Free Energy is rescaled, to reflect that
the Free Energy is both Size-Extensive and Scale Invariant near a Critical Point
This is not obvious — but it is essential to both having a conceptual understanding of complex, multi scale phenomena, and it is necessary to obtain very highly accurate numerical calculations. In fact, being size extensive and/or size consistent is absolutely necessary for highly accurate quantum chemistry calculations of strongly correlated systems. So it is pretty amazing but perhaps not surprising that this is necessary for large scale deep learning calculations also!
If we (can) apply the same map, , repeatedly, we obtain a RG recursion relation, which is the starting point for most analytic work in theoretical physics. It is usually difficult to obtain an exact solution to the RGE (although it is illuminating when possible [20]).
Many RG formulations both approximate the exact RGE and/or only include relevant variables. To describe a multiscale system, it is essential to distinguish between these relevant and irrelevant variables.
Let's say the feature map is a simple linear rescaling
We can obtain a very elegant, approximate RG solution where F(x) obeys a complex (or log-periodic) power law.
This behavior is thought to characterize Per-Bak style Self-Organized Criticality (SOC), which appears in many natural systems–and perhaps even in the brain itself. Which leads to the argument that perhaps Deep Learning and Real Learning work so well because they operate like a system just near a phase transition–also known as the Sand Pile Model--operating at a state between order and chaos.
Leo Kadanoff, now at the University of Chicago, invented some of the early ideas in Renormalization Group. He is most famous for the Real Space formulation of RG, sometimes called the Block Spin approach. He also developed an alternative approach, called the Variational Renormalization Group (VRG, 1975), which is, remarkably, what Unsupervised RBMs are implementing!
Let's consider a traditional Neural Network–a Hopfield Associative Memory (HAM). This is also known as an Ising model or a Spin Glass in statistical physics.
An HAM consists of only visible units; it stores memories explicitly and directly in them:
We specify the Energy — called the Hamiltonian — for the nodes. Note that all the nodes are visible. We write
The Hopfield model has only single and pair-wise interactions.
A general Hamiltonian might have many-body, multi-scale interactions:
The Partition Function is given as
And the Free Energy is
The idea was to mimic how our neurons were thought to store memories–although perhaps our neurons do not even do this.
Either way, Hopfield Neural Networks have many problems; most notably they may learn spurious patterns that never appeared in the training set. So they are pretty bad memories.
Hinton created the modern RBM to overcome the problems of the Hopfield model. He used hidden units to represent the features in the data–not to memorize the data examples directly.
An RBM is specified Energy function for both the visible and hidden units
This also defines joint probability of simultaenously observing a configuration of hidden and visible spins
which is learned variationally, by minimizing the reconstruction error…or the cross entropy (KL divergence), plus some regularization (Dropout), using Greedy layer-wise unsupervised training, with the Contrastive Divergence (CD or PCD) algo, …
The specific details of an RBM Energy are not addressed by these general concepts; these details do not affect these arguments–although clearly they matter in practice !
It turns out that
Introducing Hidden Units in a Neural Network is a Scale Renormalization.
When changing scale, we obtain an Effective Hamiltonian that acts on a the new feature space (i.e the hidden units)
or, in operator form
This Effective Hamiltonian is not specified explicitly, but we know it can take the general form (of a spin funnel, actually)
The RG transform preservers the Free Energy (when properly rescaled):
The RG theory provides a way to iteratively update, or renormalize, the system Hamiltonian. Each time we add a layer of hidden units (h1, h2, …), we have
We imagine that the flow map is attracted to a Critical Trajectory which naturally leads the algorithm to the fixed point. At each step, when we apply another RG transform, we obtain a new, Renormalized Manifold, each one closer to the optimal data manifold.
Conceptually, the RG flow map is most useful when applied to critical phenomena–physical systems and/or simple models that undergo a phase transition. And, as importantly, the small changes in the data should 'wash away' as noise and not affect the macroscopic / critical phenomena. Many systems–but not all–display this.
Where Hopfield Nets fail to be useful here, RBMs and Deep Learning systems shine.
We now show that these RG transformations are achieved by stacking RBMs and solving the RBM inference problem!
Kadanoff's Variational Renormalization Group
As in many physics problems, we break the modeling problem into two parts: one we know how to solve, and one we need to guess.
The joint Hamiltonian, or Energy function, is then given by
The Correlation V(v,h) is defined so that the partition function is not changed
This gives us
(Sometimes the Correlation V is called a Transfer Operator T, where V(v,h)=-T(v,h) )
We may now define a renormalized effective Hamilonian that acts only on the hidden nodes
so that we may write
Because the partition function does not change, the Exact RGE preserves the Free Energy (up to a scale change, we we subsume into
We generally can not solve the exact RGE–but we can try to minimize this Free Energy difference.
What Kadanoff showed, way back in 1975, is that we can accurately approximate the Exact Renormalization Group Equation by finding a lower bound using this formalism
Deep learning appears to be a real-space variational RG technique, specifically applicable to very complex, inhomogenous systems where the detailed scale transformations have to be learned from the data
We will now show how to express RBMs using the VRG formalism and provide some intuition
In an RBM, we simply want to learn the Energy function directly; we don't specify the Hamiltonian for the visible or hidden units explicitly, like we would in physics. The RBM Energy is just
We identify the Hamiltonian for the hidden units as the Renormalized Effective Hamiltonian from the VRG theory
To obtain RBM Hamiltonians for just the visible or hidden nodes, we need to integrate out the other nodes; that is, we need to find the marginal probabilities.
To train an RBM, we apply Contrastive Divergence (CD), or, perhaps today, Persistent Contrastive Divergence (PCD). We can kindof think of this as slowly approximating
In practice, however, RBM training minimizes the associated Free Energy difference … or something akin to this…to avoid overfitting.
In the "Practical Guide to Training Restricted Boltzmann Machines", Hinton explains how to train an RBM (circa 2011). Section 6 addresses "Monitoring the overfitting"
"it is possible to directly monitor the overfitting by comparing the free energies of training data and held out validation data…If the model is not overfitting at all, the average free energy should be about the same on training and validation data"
Modern variants of Real Space VRG are not "'forced' to minimize the global free energy" and have attempted other approaches such as Tensor-SVD Renormalization. Likeswise, some RBM / DBM approaches do likewise may minimize a different objective.
In some methods, we minimize the KL Divergence; this has a very natural analog in VRG language [1].
The Renormalization Group Theory provides new insights as to why Deep Learning works so amazingly well. It is not, however, a complete theory. Rather, it is framework for beginning to understand what is an incredibly powerful, modern, applied tool. Enjoy!
[1] An exact mapping between the Variational Renormalization Group and Deep Learning, 2014
[2] Variational Approximations for Renormalization Group Transformations, 1976
[3] A Common Logic to Seeing Cats and Cosmos
[5] A Fundamental Theory to Model the Mind
[6] A Practical Guide to Training Restricted Boltzmann Machines, 2010
[7] On the importance of initialization and momentum in deep learning, 2013
[8] Dropout: A Simple Way to Prevent Neural Networks from Overfitting, 2014
[9] Ken Wilson, A Scientific Appreciation
[10] http://www-math.unice.fr/~patras/CargeseConference/ACQFT09_JZinnJustin.pdf
[11] Training Products of Experts by Minimizing Contrastive Divergence, 2002
[12] Training Restricted Boltzmann Machines using Approximations to the Likelihood Gradient, 2008
[13] http://www.nonlin-processes-geophys.net/3/102/1996/npg-3-102-1996.pdf
[14] THE RENORMALIZATION GROUP AND CRITICAL PHENOMENA, Ken Wilson Nobel Prize Lecture
[15] Scaling, universality, and renormalization: Three pillars of modern critical phenomena
[16] The Potential Energy of an Autoencoder, 2014
[17] Contractive Auto-Encoders: Explicit Invariance During Feature Extraction, 2011
[18] Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion, 2010
[19] Quora: What is Renormalization group theory?
Why does Deep Learning work?
This is the big question on everyone's mind these days. C'mon we all know the answer already:
"the long-term behavior of certain neural network models are governed by the statistical mechanism of infinite-range Ising spin-glass Hamiltonians" [1] In other words,
Multilayer Neural Networks are just Spin Glasses? Right?
This is kinda true–depending on what you mean by a spin glass.
In a recent paper by LeCun, he attempts to extend our understanding of training neural networks by studying the SGD approach to solving the multilayer Neural Network optimization problem [1]. Furthermore, he claims
None of these works however make the attempt to explain the paradigm of optimizing the highly non-convex neural network objective function through the prism of spin-glass theory and thus in this respect our approach is very novel. And, again, this is kinda true
But here's the thing…we already have a good idea of what the Energy Landscape of multiscale spin glass models* look like–from early theoretical protein folding work by Peter Wolynes to modern work by Ken Dill, etc [2,3,4]). In fact, here is a typical surface:
*[technically these are Ising spin models with multi-spin interactions]
Let us consider the nodes, which above represent partially folded states, as nodes in a multiscale spin glass–or , say, a multilayer neural network. Immediately we see the analogy and the appearance of the 'Energy funnel' In fact, researchers have studied these 'folding funnels' of spin glass models over 20 years ago [2,3,4]. And we knew then that
as we increase the network size, the funnel gets sharper
Note: the Wolynes protein-folding spin-glass model is significantly different from the p-spin Hopfield model that LeCun discusses because it contains multi-scale, multi-spin interactions. These details matter.
Spin glasses and spin funnels are quite different. Spin glasses are highly non-convex with lots of local minima, saddle points, etc. Spin funnels, however, are designed to find the spin glass of minimal-frustration, and have a convex, funnel shaped, energy landscape.
Spin Funnels have been characterized and cataloged by theoretical chemists and can take many different convex shapes
This seemed to be necessary to resolve one of the great mysteries of protein folding: Levinthal's paradox [5]. If nature just used statistical sampling to fold a protein, it would take longer than the 'known' lifetime of the Universe. It is why Machine Learning is not just statistics.
Deep Learning Networks are (probably) Spin Funnels
So with a surface like this, it is not so surprising that an SGD method might be able to find the Energy minima (called the Native State in protein folding theory). We just need to jump around until we reach the top of the funnel, and then it is a straight shot down. This, in fact, defines a so-called 'folding funnel' [4]
So is not surprising at all that SGD may work.
Recent research at Google and Stanford confirms that the Deep Learning Energy Landscapes appear to be roughly convex [6], as does LeCuns work on spin glasses.
Note that a real theory of protein folding, which would actually be able to fold a protein correctly (i.e. Freed's approach [7]), would be a lot more detailed than a simple spin glass model. Likewise, real Deep Learning systems are going to have a lot more engineering details–to avoid overtraining (Dropout, Pooling, Momentum) than a theoretical spin funnel.
[Indeed, what is unclear is what happens at the bottom of the funnel. Does the system exhibit a spin glass transition (with full blown Replica Symmetry Breaking, as LeCun suggests), or is the optimal solution more like a simple native state defined by but a few low-energy configurations ? Do DL systems display a phase transition, and is it first order like protein folding? We will address these details in a future post.] In any case case,
It is not that Deep Learning is non-convex–is that we need to avoid over-training
And it seems modern systems can do this using a few simple tricks, like Rectified Linear Units, Dropout, etc.
Hopefully we can learn something using the techniques developed to study the energy landscape of multi-scale spin glass/ spin funnels models. [8,9], thereby utilizing methods theoretical chemistry and condensed matter physics.
In a future post, we will look in detail at the folding funnel including details such as misfolding, the spin glass transition in the funnel, and the relationship to symmetry breaking, overtraining, and LeCun's very recent analysis. These are non-trivial issues.
I believe this is the first conjecture that Supervised Deep Learning is related to a Spin Funnel. In the next post, I will examine the relationship between Unsupervised Deep Learning and the Variational Renormalization Group [10].
Learn more about us at: http://calculationconsulting.com
[1] LeCun et. al., The Loss Surfaces of Multilayer Networks, 2015
[2] Spin glasses and the statistical mechanics of protein folding, PNAS, 1987
[3] THEORY OF PROTEIN FOLDING: The Energy Landscape Perspective, Annu. Rev. Phys. Chem. 1997
[5] From Levinthal to pathways to funnels, Nature, 1997
[6] QUALITATIVELY CHARACTERIZING NEURAL NETWORK OPTIMIZATION PROBLEMS, Google Research (2015)
[7] Mimicking the folding pathway to improve homology-free protein structure prediction, 2008
[8] Funnels in Energy Landscapes, 2007
[9] Landscape Statistics of the Low Autocorrelated Binary String Problem, 2007
[10] A Common Logic to Seeing Cats and Cosmos, 2014
This post is finishing up some older work of an R&D / coding project in Transductive Learning.
Why are SVMs interesting? It is just a better way to do Logistic Regression? Is it the Kernel Trick? And does this even matter now that Deep Learning is everywhere? To the beginning student of machine learning, SVMs are the first example of a Convex Optimization method. To the advanced practitioner, SVMs are the starting point to creating powerful Convex Relaxations to hard problems.
Historically, convex optimization was seen as the path to central planing an entire economy. A great new book, Red Plenty, is "about the scientists who did their genuinely brilliant best to make the dream come true …" [amazon review].
It was a mass delusion over the simplex method, and it is about as crazy as our current fears over Deep Learning and AI.
Convex optimization is pretty useful, as long as we don't get crazy about it.
The prototypical method convex relaxation is for Transductive Learning and the Transductive SVM (TSVM)
Vapnik proposed the idea of Transduction many years ago; indeed the VC theory is proven using Transduction. I would bet that he knew a TSVM could be convexified–although I would need a job at Facebook to verify this.
A good TSVM has been available since 2001 in SvmLight, But SvmLight is not opensource, so most people use SvmLin.
Today there are Transductive variants of Random Forests, Regression, and even Deep Learning. There was even a recent Kaggle Contest–the Black Box Challenge (and, of course, the Deep Learning method won). Indeed, Deep Learning classifiers may benefit greatly from Transductive/SemiSupervised pretraining with methods like Pseudo Label [8], as shown in the Kaggle The National Data Science Bowl contest.
We mostly care about binary text classification, although there is plenty of research in convex relaxation for multiclass transduction, computer vision, etc.
Transductive learning is essentially like running a SVM, but having to guess a lot of the labels. The optimization problem is
where is the loss function , the regularization function,
and the binary labels are only partially known.
The optimization is a non-convex, mixed-integer problem. Amazingly, we can reformulate the TSVM to obtain a convex optimization!
This is called a Convex Relaxation, and it lets us guess the unknown labels…
to within a good approximation, and using some prior knowledge.
Proving an approximation is truly convex is pretty hard stuff, but the basic idea is very simple. We just want to find a convex approximation to a non-convex function.
It has been known for a while that the TSVM problem can be convexified [5]. But it has been computationally intractable and there is no widely available code.
We examine a new Convex Relaxation of the Transductive Learning called the Weakly Labeled SVM (WellSVM) [2,3].
In the Transductive SVM (TSVM) approach, one selects solutions with minimum SVM Loss (or Slack)
and the maximum margin
Let us consider the standard SVM optimization
which has the dual form
To keep the notation simpler, we will not consider the Kernalized form of the algorithm. Besides, we are mostly interested in text classification, and Kernels are not needed.
In a TSVM, we have to guess the labels and select the best solution (for a given set of regularization parameters). There are way too many labes to guess, so
we need to constrain the label configurations by balancing the guesses
We assume that we have some idea of the total fraction of positive (+) labels
This is, however, a critical piece of prior information we need. We define the space of possible labels as
i.e, for exactly half positive / negative labels, then
and the true mean label value is zero
So we are saying that if we know
Then we know almost all the labels exactly! And that is powerful.
Note: iI is critical that in any transductive method, we reduce the size of the label configuration space. The Balancing constraint is the standard constraint–but it may be hard to implement in practice. To me, personally, the problem resembles the matrix product states method in quantum many body theory. And I suspect that, eventually, there will be a related method found for transducitive learning. Perhaps even Deep Learning has found this.
The popular SvmLin method use a kind of Transductive Meta-Heuristics that set the standard for other approaches. The problem is, we never really know if we have the best solution. And it is not easy to extend to multiclass classification.
Convex methods have been the method of choice since Dantzig popularized the simplex method in 1950
Although linear programming itself was actually invented in 1939 by Leonid Kantorovich [7] — "the only Soviet scholar ever to win the Nobel Prize for Economics"
A convex method lends itself to production code that anyone can run.
More generally, we need to solve a non-convex min-max problem of the form
where the G matrix is
where α lies in the convex set
Notice that G is concave in α and (can be made) linear in the y's [3]. We seek a convex relaxation of this min-max problem. And, as importantly, we want to code the final problem using an off-the-shelf SVM solver with some simple mods. The steps are
For details, see [4,5], although it was originally posed by John von Neumann in his work on Game Theory.
One could spend a lifetime stuyding von Neumann's contributions. Quantum mechanics. Nuclear Physics. Etc.
Here we scratch the surface.
The Minimax thereom lets us switch the order of the min/max bounds.
The original problem is an upper bound to this. That is
To solve this, we
We convert the search over possible label configurations into the dual max problem, so that the label configurations become constraints.
This linear in α and θ. In fact, it is convex.
There are an exponential number of constraints in .
Even though it is convex, we can not solve this exactly practice. And that's…ok.
Not all possible labelings matter, so not all of these constraints are active (necessary) for an optimal solution.
We just need an active subset, , which we can find by …
The Cutting Plane, or Gomory Chvatal, method. (Remember, if you want to sound cool, give your method a Russian name).
To proceed, we construct the Lagrangian and then solve the convex dual problem.
You may remember the method of Lagrange multipliers from freshman calculus:
The Lagrangian is
When we set the derivative w.r.t. to 0, we find
This lets us rewrite the TSVM as
where the set of allowable multiplers lies in the simplex
The resulting optimization is convex in µ and concave in α.
This is critical as it makes a Kernel-TSVM a Multiple Kernel Learning (MKL) method.
"WellSVM maximizes the margin by generating the most violated label vectors iteratively, and then combines them via efficient multiple kernel learning techniques".
For linear applications, we need only consider 1 set of .
Now replace the inner optimization subproblem with its dual. Of course, the dual problem is a lower bound on the optimal value. We then
to obtain a new min-max optimization–a convex relaxation of the original
When we restrict the label configurations to the working set ,we have
A key insight of WellSVM is that the core label search
is equivalent to
To me, this is very elegant!
We search the convex hull of the document-document density matrix, weighted by the Langrange multipliers for the labels.
How can we solve a SVM with exponential constraints? Take a page from old-school Joachim's Structural SVMs [6].
This is the goto-method for all Mixed Integer Linear Programming (MILP) problems. On each iteration, we
This grows the active set of label configurations , learning from previous guesses. We expect $latex N_{\mathcal{C}}\ll N_{\mathcal{B}} $
the PseudoCode is
The cutting plane algo finds a new constraint, or cuts, on each iteration, to 'chip away' at a problem until the inner convex hull is found. It usually finds the most violated constraint on each iteration, however,
With SVMs we can get good results just finding any violated constraint.
Here, we seek , a violated label assignment. The WellSVM paper [2,3] provides a great solution.
For any violation , and for any pair of label configurations , we have
This lets us compute in two steps:
First, compute , by searching the current, active, and usually small set
Second, compute , by searching the set of all balanced label configurations
We will review the existing code and run some simulations soon
The goal is to modify scikit learn interface to liblinear to output the Lagrange multipliers . then to implement WellSVM in an ipython notebook. stay tuned.
[1] The Semi-Supervised Learning Book (2006)
[2] Convex and Scalable Weakly Labeled SVMs , JMLR (2013)
[3] Learning from Weakly Labeled Data (2013)
[4] Kim and Boyd, A Minimax Theorem with Applications to Machine Learning, Signal Processing, and Finance, 2007
[5] Minimax theorem, game theory and Lagrange duality
[6] Cutting-Plane Training of Structural SVMs
[7] "This is an example of Stigler's law of eponymy, which says that "no scientific discovery is named after its original discoverer." Stigler's law of eponymy is itself an example of Stigler's law of eponymy, since Robert Merton formulated similar ideas much earlier Quote from Joe Blitzstein on Quora
[8] Pseudo-Label : The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks
Last week, On Jan 14, 2015 Bitcoin dropped below $200 (USD)
Moreover, several Bitcoin exchanges have shut down due to hacks and/or criminal activity, and key BitCoin players are on trial. This begs the question:
Why is BitCoin Crashing ?
Markets crash all the time. Stock markets. Currency Markets. Even the Dutch Tulip market of 1637 crashed–although even this is still hotly debated.
Fraud? Speculation? Mania? Lack of Regulation? What gives?
In 1996, researchers at the University of Chicago[1] and elsewhere [2] independently proposed that market crashes resemble physical crashes such as earthquakes, ruptures, and avalanches. Such phenomena arise from long range, collective fluctuations — i.e. herding behavior — that overtakes a system as it approaches a critical point…and/or in the aftershocks.
The theory relies upon a fairly generous application of the theory of critical phenomena and the Renormalization Group (RG) theory. It provides both a qualitative description of the phenomena and quantitative predictions such as when a crash may occur or how long a bear market will continue.
Most notably, the strongest proponent, Didier Sornette, accurately estimated the long bear run of the Japanese markets–what he terms a Financial Anti-Bubble:
He has written numerous papers, a book, and even a TED Talk.
We also introduced these ideas in a previous post: Noisy Time Series II: Earth Quakes, Black Holes, and Machine Learning
It is well known that financial markets display non-Gaussian, long tail, power law-like behavior. After a crash model, the price series itself may follow a power law
where is the time of the crash, the time , and is a real number.
We can fit the recent Bitcoin EOD (End of Day) prices to a power law, starting at it's peak on Nov 11, 2013:
We obtain a modestly good least squares fit, with and . Clearly there is a lot of noise–or is there?
RG theory suggests that near a crash, the power law may become complex (i.e a form of 'symmetry breaking'). We will now sketch how this comes about, using the nobel prize winning Renormalization Group theory.
When examining a price series , or some other phenomena, say , a critical point occurs when we observe a rapid, sharp spike–i.e. Bitcoin's fast rise up to Nov 29, 2013.
Here, we will find that as
where is a complex critical exponent
We say becomes singular near a crash when there exists a k-th derivative that becomes infinite as :
We also know that near a critical point, physical systems exhibit scale invariant (i.e. fractal) behavior. The noise, or fluctuations, we observe on small time scales look just like fluctuations on larger time scales .
We call the RG flow map. The flow map describes these change of time scales, and the RG equation describes the invariances.
The fundamental RG equation (eqn) is:
We assume is continuous–which is non trivial for price series –and that is differentiable. is the non-singular (regular) part of .
The magic of the RG theory is that it allows us to describe the behavior near a critical point knowing only the flow map and the regular function . The formal solution is an infinite order power series
We will sketch a very simple, leading order approximation to that leads to a complex power law.
We start by first assuming power law behavior (at lowest order):
We seek the simplest RG solution. Let us assume the RG flow map is linear in :
where (for technical reasons to ensure the critical point is an unstable fixed point of the RG flow).
We will ignore the regular solution and write the simplified RG eqn:
For our simple power law, we have
This is easy to satisfy if
We seek the most general solution, applicable to discrete physical systems, such as earthquakes, ruptures in materials–and financial market crashes. That is, we expect to take on discrete values:
We call this Discrete Scale Invariance (DSI)
The most general solution of DSI is
where is a general periodic function. As above, we express as the limit of an infinite power series
This is a classic Weierstrass function–a pathological function that, in the infinite limit, is continuous everywhere and yet differentiable nowhere. It is used to model fractals, natural systems that are scale invariant, and highly discontinous but structured time series.
In a crash scenario, we expect that the true power series for will diverge; i.e. the k-th derivative explodes. So here, we only take the leading term of our linearized model
Note that it takes the form
so , simple power law behavior, is like the regular part of the solution.
We now see that we have a complex critical exponent :
We consider the real part of the complex power law
which gives the following log periodic formula for the price series
Instead of fluctuating randomly around the underlying power law drift, the price series exhibits log-periodic oscillations–the DSI signature of a crash.
This model appears to work for 1-2 oscillations before (or after) the crash. To model the longer time anti-bubble behavior, Sornette has developed an extended formula, based on the third-order Landau expansion (as opposed to, say, additional terms in the power series defined above). We leave these details and further analysis for later.
We now fit the latest Bitcoin behavior, assuming the crash / Bear market started on Nov 29, 2013 :
We readily find a DSI fit, with .
It is actually quite difficult to get a very tight fit, and usually advanced methods are needed. This is a simple, crude fit done to illustrate the basic ideas.
We see that Bitcoin, like other financial markets, displays log periodic behavior, characteristic of a self-organized crash.
These kinds of crashes are not caused by external events or bad players–they are endemic to all markets and result from the cooperative actions of all participants.
We can try to detect them, perhaps even profit off them, but it is unclear how to avoid these Self-organized Critical points that wreck havoc on our finances.
Bitcoin is an attempt to create a new kind of currency–a CryptoCurrency–that is free from both institutional control and individual corruption. It is hoped that we can avoid the kind of devastating inflation, devaluation, and crashes that occur all too frequently in our current worldwide banking systems. The promise is that Bitcoin, backed by the Blockchain, can remove our dependence on a single, faulty institution and replace it with a decentralized, distributed form of trust. But
"Real life operates at a critical point between order and chaos"
Our analysis here shows that even the Bitcoin economy still appears to behave like a traditional market–prone to kinds of crashes that frequently arise in all natural self organized systems.
This Self-Organized Criticality is perhaps just how nature works [6,7].
And while Bitcoin is certainly interesting and exciting, perhaps it too is subject to the natural laws of physics.
[1] 1995 James A. Feigenbaum and Peter G.O. Freund, Discrete Scaling in Stock Markets Before Crashes
[2] TED Talk: How We Can Predict the Next Financial Crises
[3] What will upend the economy next? Economist didier sornette explains
[4] D. Sornette, Critical market crashes
[5] 1999 Predicting Financial Crises using Discrete Scale Invariance
[6] 1999 A. Johansen and D. Sornette, Financial "Anti-Bubbles": Log-Periodicity in Gold and Nikkei collapses
[7] Anders Johansen and Didier Sornette, Evaluation of the quantitative prediction of a trend reversal on the Japanese stock market in 1999 , 2000
[8] 2003 Wei-Xing Zhou and Didier Sornette Renormalization group analysis of the 2000–2002 anti-bubble in the US S&P500 index: explanation of the hierarchy of five crashes and prediction
[9] 20122 Didier Sornette, Ryan Woodard, Wanfeng Yan, and Wei-Xing Zhou Clarifications to Questions and Criticisms on the Johansen-Ledoit-Sornette Bubble Model
[10] http://guava.physics.uiuc.edu/~nigel/courses/563/essay/lin.ps.gz
[11] Per Bak, "How Nature Works: The Science of Self-Organized Criticality", 1996
Recently Facebook hired Vapnik, the father of the Support Vector Machine (SVM) and the Vapnik-Chrevoniks Statistical Learning Theory.
Lesser known, Vapnik has also pioneered methods of Transductive and Universum Learning. Most recently, however, he has developed the theory of the Learning Using Privileged Information (LUPI), also known as the SVM+ method.
And this has generated a lot of interest in the community to develop numerical implementations.
In this post, we examine this most recent contribution.
Once again, we consider the problem of trying to learn a good classifier having a small set of labeled examples .
Here, we assume we have multiple views of the data; that is, different, statistically independent feature sets , that describe the data.
For example, we might have a set of breast cancer images, and some holistic, textual description provided by a doctor. Or, we might want to classify web pages, using both the text and the hyperlinks.
Multi-View Learning is a big topic in machine learning, and includes methods like SemiSupervised CoTraining, Multiple Kernel Learning, Weighted SVMs, etc. We might even consider Ensemble methods as a kind of Multi-View Learning.
Having any extra, independent info about our data is very powerful.
In LUPI, also called the SVM+ method, we consider situation where we only have extra information about the labeled examples .
Vapnik calls Privileged Information — and shows us how to use it
If we are going to discuss Vapnik we need to mention the VC theory. Here, we note an important result about soft and hard-margin SVMs: when the training set is non-separable, the VC bounds scale as :
where is the training error, and h is the VC dimension
But when the training set is separable, i.e. , the VC bounds scale as
This means we can use say L=300 labeled examples as opposed to L=900,000, which is a huge difference.
It turns out we can also achieve scaling–if we have an Oracle that tells us, a priori, the slacks. Hence the term Oracle SVM. The Oracle tells us how much confidence we have in each label.
So if we can get the slacks , or, equivalently, the weights, for each instance, L can be much smaller.
[Of course, this is only true if we can sample all the relevant features.]
In practice, we usually assume data is noisy and non-separable. So we use a soft-margin SVM where we can adjust the amount slack with the (cost) parameter,
Note that we also have the bias , but we can usually set to zero (still, be careful doing this!).
Let us write the soft-margin SVM optimization as
subject to the constraints
Notice that while we formally add slack variables to max-margin optimization–they eventually vanish. That is, we don't estimate the slacks; we replace them with the maximum margin violation, or the Hinge Loss error
We then solve the unconstrained soft-margin SVM optimization, which is a convex upper bound to the problem stated above–as explained in this nice presentation on LibLinear.
or, in simpler terms
And this works great–when we have lots of labeled data L.
What if we could actually estimate or assign all the L slack variables ? In principle, we can far use labeled examples L. This is the promise of LUPI.
In LUPI, we use the privileged information to learn the slack variables :
We model the slack as linear functions of the privileged information.
Effectively, the privileged information provides a measure of confidence for each labeled instance.
To avoid overtraining, we apply a max-margin approach. This leads to an extended, SVM+ optimization problem, with only 2 adjustable regularization parameters (and 2 bias params):
subject to the constraints
The SVM+/LUPI problem is a convex (quadratic) optimization problem with linear constraints–although it does require a custom solver. The current approach is to solve the dual problem using a variant of SMO. There are a couple versions (below) in Matlab and Lua. The Matlab code includes related SVM+MTL (SVM+ MultiTask Learning). I am unaware of a open source C or python implementation similar to Svmlin or Universvm.
To overcome the solver complexity, other methods, such as the Learning to Rank method, have been proposed. We will keep an eye out for useful open source platforms that everyone can benefit from.
The SVM+/LUPI formalism demonstrates a critical point about modern machine learning research. If we have multiple views of our data, extra information available during training (or maybe during testing), or just several large Hilbert spaces of features, then we can frequently do better than just dumping everything into an SVM, holding our nose, and turning the crank.
or how many ways can we name the same method?
Learning with Hidden Information
The SVM+ method: MIT Reading Notes
Learning with Non-Trivial Teacher
Learning to Rank Using Privileged Information
On the Theory of Learning with Privileged Information
SMO-style algorithms for learning using privileged information
Learning Using Privileged Information: SVM+ and Weighted SVM
Learning with Asymmetric Information
Comparison of SVM+ and related methods (SVM+MTL, rMTL, SVM+ regression, etc.)
Proof and Implementation of Algorithmic Realization of Learning Using Privileged Information (LUPI) Paradigm: SVM+
Learning with Privileged Information


text:
Select Local Merchants
Tech City BowlRead MoreTech City Bowl offers a fusion of modern and classic entertainment, as 170-inch high-definition screens hover above the center's 32 lanes. The facility hosts open-bowling hours throughout the week, and allows customers to earn redemption tickets that may be redeemed for video games, radio-controlled helicopters, cameras, and more. They also offer league opportunities for adults and children, as well as interactive bowling games such as Rocky Road Race. Every Friday and Saturday night, Tech City Bowl takes on fluorescent hues and black-lit lanes for Cosmic X-Bowl. Laser lights and fog machines fill the alley with special effects, and a live DJ spins dance-worthy tunes that are broadcast on the high-definition screens. In the adjacent arcade, players can test abilities on air hockey and video games, and the onsite sports bar fills empty glasses with classic libations and seasonal microbrews. Bowlers can refuel in between rounds at the center's grill, which serves a full menu to keep hungry players from cracking open their bowling balls onto frying pans.13033 Northeast 70th PlaceKirkland, WA US
Lucky StrikeRead MoreStrike, spare, who cares? Bowling is always fun at Bellevue's Lucky Strike. Don't worry about packing a snack. This alley has a great restaurant so you can keep the fun going. Cheer on your favorite team at Lucky Strike, a local restaurant with TVs. Treat yourself to your favorite tasty beverage or one of our flavorful signature drinks. There's often a DJ spinning as well, and patrons are encouraged to step out on the dance floor. A tad noisy, the alley is well-suited for those who don't mind a little extra hustle and bustle. Weekend customers, beware! The alley is busiest on Friday and Saturday, so getting seated will take some time. At Lucky Strike, you can easily find parking in the lot next door.700 Bellevue Way NortheastBellevue, WA US
Skyway Park Bowl & CasinoRead MoreTime has a way of slowing down inside Skyway Park Bowl. People hold their breath for those few moments after they release their bowling balls. In the Lucky Dragonz Poker Room, dealers run frequent Texas Hold'em tournaments. The air is ripe with anticipation, especially when there's an opportunity to win some money, as happens regularly. Other evenings bring more special events. On Thursdays, cosmic bowl turns on the black lights. Karaoke takes place every night, and live bands show up on weekends, like office workers who want to beat Monday traffic. The nightclub room doubles as a dining destination, where an onsite restaurant serves BBQ duck noodle soup and other Vietnamese cuisine. The nightclub room can also be rented for weddings, birthday parties, and other events.11819 Renton Ave. SSeattle, WA US
West Seattle BowlRead MoreWest Seattle Bowl's lanes are a piece of West Seattle history, even if they don't look it at first glance. Strip away the modern scoring system, the updated decor, and 18 of the now 32 lanes, and it's easy to imagine the families of pilgrims bowling at the alley's grand opening in 1948. Since those early days. West Seattle Bowl has benefited the community beyond just giving it a fun place to rack up Xs and /s. The owners regularly host charity-bowling events, which have brought in roughly $100,000 annually during recent years. West Seattle Bowl is as much about specials as it is about good old-fashioned bowling. Weekenders can take advantage of Sunday mornings' "Breakfast & Bowl," for example, where the on-site Highstrike Grill serves up two complimentary games of bowling with its eggs.4505 39th Avenue SouthwestSeattle, WA US
Woodland Park Lawn Bowling ClubRead MoreKnock them pins down at Woodland Park Lawn Bowling Club in Seattle! Check out this alley's restaurant, where you can stay for lunch or grab your meal to go. You won't need to get a sitter before heading to this alley — kids are more than welcome at this family-friendly establishment. Parking is plentiful, so visitors can feel free to bring their vehicles. When it comes to bowling in Seattle, you're sure to find your lucky strike at Woodland Park Lawn Bowling Club.6018 Whitman Ave NSeattle, WA US
Professional Bowlers AssociationRead MoreA perfect place for a family night or a romantic date, head on over to Seattle's Professional Bowlers Association for some bowling excitement! Parking is plentiful, so visitors can feel free to bring their vehicles. So head on over to Professional Bowlers Association in Seattle and toss all your worries down the lane.615 2nd AvenueSeattle, WA US

subset: commoncrawl


text:
Pinned alphabet cards on wall.
Clothes Pin, Card Display, Alphabet Cards, Kidsroom, Kid S Room, Flash Card, Kids Rooms
ten ways to display alphabet cards :: another playroom idea - could trade out cards for other types of flash cards as well
super cute and cheap idea. just buy the flashcards on amazon and string it up with twine and clothespins.
either the playroom or the nursery... simple display for the vintages alphabet cards I just got ... (or do the same thing in the upstairs hallway, but w/ longer rows. I could alternate my french flash cards w/ old postcards b/w photos)
Alphabet cards. Could hot glue clothes pin and attach large flash cards
***hung alphabet cards flash cards like this in B's room. so cute!***
alphabet cards as decor lines and clothespins to hold art!
alphabet cards laundry line in kids room.
alphabet flashcards hung by clothespins
Look! Alphabet Card Display, Take 2
Animal alphabet cards for the nursery.
Alphabet Cards Nursery, Baby Boy Nursery Idea, Baby Boy Nursery Room Ideas, Playroom, Baby Boy Bedroom Ideas, Baby Room, Flash Card
A super cute addition to a baby's room. Decorations for when they are small, and learning for when they are growing.
http://www.namebrandbabyclothes.com/category/changing-table/ Animal alphabet cards for the nursery.
Animal alphabet cards for the playroom - something similar but larger letters and plain background.
I have these cards! Can maybe string them up on a cord over the crib or changing table?
Animal alphabet cards for the nursery. Just buy flash cards!
Animal alphabet cards for the nursery. Play room?
Flash card (Animal alphabet)!
Would be cute in playroom
Play room.
A Unique Way To Dress Up Tables And Chairs — Shelterness
20 DIY Adorable Ideas for Kids Room
For Kids, Kidsroom, Children, Kids Table, Table Legs, Kids Rooms, Ikea Hack
Happy Socks Table and chairs for kids :) maybe kids won't scratch the wood when we need to pull out the "kid's table"
#kidsroom #kidstableandchairs 20 DIY Adorable Ideas for Kids Room | Daily source for inspiration and fresh ideas on Architecture, Art and Design
20 DIY Adorable Ideas for Kids Room | Daily source for inspiration and fresh ideas on Architecture, Art and Design - Love the yarn wrapped table legs to dress up a plain craft table
sock table legs. cute for a kids' room. really cute on the table legs...not so much on the chair. I do like how it would help prevent scratches on the floors
luxury furniture, living room ideas, home furniture, contemporary furniture, contemporary living room, high end furniture, entryway furniture. kid room decor ideas.
Easy Ikea Hacks For Kids! This one is more whimsical, but I love the pop of color.
20 DIY Adorable Ideas for Kids Room | Ideas para decorar el cuarto de los niños
Children's socks...good way to preserve floors and make a kids room look fun!
20 DIY Adorable Ideas for Kids Room / play room
Every Child is an Artist Wall Decal Large - Children Artwork Display Decal…
Love this for the art wall!!!
Kids Room, Kidsroom, Wall Decal, Children, Artwork Display, Art Wall
I was thinking about doing this in the play room for the kids artwork :) Every Child is an Artist Wall Decal Large by StephenEdwardGraphic, $24.00
Freshen up the art wall at home... Every Child is an Artist Wall Decal Medium - Children Artwork Display - Picasso Quote. $17.00, via Etsy.
Every Child is an Artist Wall Decal Large - Children Artwork Display Decal - Picasso Quote di StephenEdwardGraphic su Etsy https://www.etsy.com/it/listing/94724200/every-child-is-an-artist-wall-decal
Every Child is an Artist Wall Decal Large - Children Artwork Display - Picasso Quote. $24.00, via Etsy.
Great idea for wall decor. #kidsrooms #wallart
artwork display for kids room
Kids room or playroom idea❤
Beautiful Playhouse
Kids Room, Kidsroom, Blue House, Children, Dollhouse, Playhouse, For Kid
One of the most lamentable aspects of having it all in the twenty-first century is that with so many gadgets, doohickeys, and thingamabobs, we often unwittingly discourage children from using their imaginations. A lightbulb went off when we spotted this open concept playhouse on Finnish designer Jenni Juurinen's website.
Wireframe playhouse: It could not be more simple, which is what we love about it. From one minute to the next, it could be the children's reading area, their shop, their house or a fire station. For more enclosure, you could easily cover the walls and roof with fabric, and then remove it on whim. A weekend's worth of work for a lifetime of memories.
little blue house: http://www.scandinaviandeko.com/blog/diy-make-your-own-playhouse/
Wooden play house can be a lot of fun - 20 DIY Adorable Ideas for Kids Room
#kidsroom DIY #indoorplayhouse #playhouse Could make a cool bedframe.
dollhouse and indoor playhouse Jenni Juurinen designer
Open Ended Playhouse for Kid's Room
This super modern doll house by the Finnish Designer Jenni Juurinen is a wonderful open ended play option for kids. I'm not really into dolls houses, but this simple and minimalistic v…
Healthy finishes and furnishing are especially important for young children. When designing a child's room include zero VOC paints, organic bedding, and furniture without harmful chemicals such as flame retardants. Keeping the background neutral will make it easy to change the space with accessories. Instead of having to repaint the whole bedroom as kids grow up and styles change, add color and design with pillows, bedding, and items such as stuffed animals. Other features such as work spaces can also be designed to adjust as they grew older. Including enough hide away storage can make it easier for the kids to keep their room tidy. Space under the bed, in closets, and bookcases with doors can be utilized to store toys and books that aren't being used. Teepees and clubhouses can be a great addition to the design of a kid's bedroom. We included a Teepee Roundup last week with some of our favorites. Keep it chic. Interior design for kid's rooms can be achieved by striking a balance between their space and the rest of the house. A clean, well-executed design can create a room that kids love living in without being unrelated to the home as a whole.
Alphabet Match
Alphabet Match 1
Bottle Caps, Abc, Bottle Tops, Foam Letter, Milk Bottle, Alphabet Activities
Alphabet Match using Milk Bottle Tops.. Love this and would change it up by having Upper case and lower case letters!
Alphabet Match - matching upper case and lower case letters. So simple to make using bottle tops and foam letters! learning4kids
Alphabet match - use bottle tops foam letters to match upper lower case... you could also do alphabet bingo!
Alphabet Match - matching upper case and lower case letters. So simple to make using bottle tops and foam letters! {learning4kids} - Laminate the sheet of paper - cute!
Alphabet match - use bottle tops & foam letters to match upper & lower case (",)
abc milk cap matching activity. use small foam letter stickers on milk caps
Alphabet match with bottle caps and foam letter stickers
Alphabet Match with bottletops
Fun Alphabet Activities for Kids: Alphabet match is a fun hands-on activity for kids to help recognise and learn their alphabet letters.
Sand table using a plastic bin, just put the top on when finished. This is a good idea... we have one of the cute turtle sand boxes but we had to stop using it because the cover doesn't seal tight, so ants, spiders and other bugs would still be able to crawl in with the cover on it.
Sandbox Idea, Backyard Sand, Water Table, Plastic Bins, Grandkid, Outdoor Kids, Kids Outdoor
Plastic bins are great for a backyard sand table. Just put the lid on when you're done. Would also work for a water table.
Plastic bins are great for a backyard sand table or water table. Cover with the lid to keep out cats, raccoons, and etc.
Plastic bins are great for a backyard sand table. Just put the lid on when you're done! That way once she is bored of it, or out grown it we can easily get rid of the sand. Also could change it up to be a water table.
Portable sand table for the grandkids.....use a large plastic bin, then put the lid on when done.
Plastic bins are great for a backyard sand table or water table.
Sandbox Idea
82 Gratuitous Photos Of Drawer Porn
kids can keep it all set up and put away on drawer shelves
Kids Room, Kidsroom, Lego Creation, Storage Idea, Lego Drawer, Lego Storage
Lego Drawer Storage for keeping everything set-up. This would be awesome for the kids' rooms. Great Idea!
for keeping stuff set up More Playmobil, Lego Sets, Drawers Storage, Lego Creations, Lego Storage, Storage Ideas, Drawers Shelves, Toys Storage, Kids Rooms Kids room organizing The Best DIY and Decor Place For You: Playmobil Drawer Storage for keeping everything set-up. Would be great lego storage too! This would work perfectly for Lego Creations Playmobil Drawer Storage Legos Creations could be set up and left up until the next idea comes along. Toy storage idea. Drawer Storage for keeping ...
Toy storage idea. A closet would work too! Woah! cool for lego creations!
Wow ! Drawers to keep the playmobil cities on ! #KidsRoom #Genius
for keeping stuff set up More Playmobil, Lego Sets, Drawers Storage, Lego Creations, Lego Storage, Storage Ideas, Drawers Shelves, Toys Storage, Kids Rooms Kids room organizing The Best DIY and Decor Place For You: Playmobil Drawer Storage for keeping everything set-up. Would be great lego storage too! This would work perfectly for Lego Creations Playmobil Drawer Storage Legos Creations could be set up and left up until the next idea comes along. Toy storage idea. Drawer Storage for keeping Playmobile or Lego sets up. Use closet organizer with pull out drawers, or remove sides and part of front from dresser drawer shelves for play toy storage - perfect for all our Playmobil!
Cute lego storage ideas great for the family room/ play room.
Cute lego storage ideas ~ for a couple of the lower shelves
self-cleaning kids's rooms/toy storage ideas | Yahoo DIY
cute playroom idea!!
Playroom - Ikea Expedit shelf and ABC alphabet banner. Cards are "My Favorite Things Flash Cards" I got from Amazon.
Playrooms Ideas, Kids Playrooms, Kids Room, Kids Playroom Ideas, Flash Card
Playroom - Ikea Expedit shelf and ABC alphabet banner. Cards are "My Favorite Things Flash Cards" I got from Amazon. Check this out Austin Mommas. Done by a former coworker/friend of ours. It's a GREAT playroom!
Love the ABC banner! I can make this for Dani! Playroom - Ikea Expedit shelf and ABC alphabet banner. Cards are "My Favorite Things Flash Cards" I got from Amazon.
Playroom - Ikea Expedit shelf and ABC alphabet banner. Cards are "My Favorite Things Flash Cards" I got from Amazon.
Cute playroom idea! ABC banner could easily be DIY.
kids playroom with Ikea - Google Search
Hang up flashcards we already have.
Playroom decoration
Cute in kids room!
play room idea ;@)
tent//kids room..
Girl Room, Kids Bedroom, Kidsroom, Kids Rooms
kid's bedroom idea - Home and Garden Design Ideas What a great kid's room concept! #zacbacon #zacbaconproperties #luxury #kids #kidsroom #bedroom #creative
kid's bedroom idea - Home and Garden Design Ideas SAM can you make this? just les pink more blue and green ;)
kid's bedroom idea - Home and Garden Design #living room design| http://kitchenstuffscollections.blogspot.com
kid's bedroom idea - Home and Garden Design #kitchen design #living room design #kitchen design ideas #kitchen interior #modern kitchen design| http://kitchenstuffscollections.blogspot.com
kid's bedroom idea - Home and Garden Design #kitchen design #kitchen design ideas #kitchen design| http://kitchenstuffscollections.blogspot.com
kids bedroom playroom idea / idées pour chambre d'enfants / gloewen et scrat
kid's bedroom idea.. Love the Cloud wallpaper!
tent//kids room.. Or a reading nook!!!
Makeover Tour: A Colorful and Quirky "Big Girl Room" Bedroom Makeover
art wall
Art Work, Wall Art, Display Kids Artwork, Cute Ideas, Artwork Display, Children S Art, Kid Art Displays, Art Walls
Really cute idea to display the kids art or school work~ Frames for a buck each at a yardsale and wooden clipboards.
Really cute idea to display the kids art or school work~ found 3 frames for a buck each at a yardsale, wooden clipboards from walmart. Ta-da!
kids art display by WCS.....cute idea for grandchildren art at our house.
gallerie dart pour tout petits/Really cute idea to display the kids art
art wall cute idea, maybe something different than the clipboards
Kids art wall! What a cute idea!! #YourHomeMatters
Cute idea to display kid artwork
Artwork display
I recently created an Art Wall in my bedroom from some pieces that I have been collecting over the years. Art Walls are a great focal point for a room and is a very telling way to make a personal statement about your home. All the images on my Art Wall that I have collected say something about my personality and my interests. Here is my finished Art Wall (please excuse my little mess and terrible cell phone images): Creating an art wall is very easy! I was actually surprised at how easy the pieces all fit together. If you are having a hard time you can always lay out your pieces of art on the floor to judge how you would like them once they are hung. You can use paintings, photographs, black and white and color. Or you could even have a whole wall of mirrors (which I personally think would look really snazzy) if that is what you prefer! There is no limit to the possibilities. Once you have your art laid out on the floor take a snap shot of the lay out with your cell phone for reference. Then experiment with a few different layouts and take a picture of each one. Now you can go back and choose the layout that you like best. Once you have chosen your lay out it is easiest to start in the center and work your way out. Find the inner most piece of art and hang that first. That piece is your focal point (like my unicorn image) as well as how you will judge where to place each piece of art from the picture of the lay out you have chosen on your cell phone. Its best to keep your pieces at eye level. I went for a little more drastic effect and almost touched the ceiling with some of my pieces. Look at me, I am a little rule breaker. You really can do what ever you like, but you might want to try not to get too close to furniture. Keep a buffer of say, 8-10 inches. Now I think all I need is a rug! I hope these tips help you in the process of creating your very own unique art wall! Good luck! If you have an art wall and would like to have it featured on Misanthropy Creations you can send a picture to email@example.com Related articles Refresh Your Home with Wall Art 6 Ways to Set Up a Gallery Wall Modern Kitchen Layout Wall Art Museum, art gallery to share common entrance Make a big impact in your home with big art
I like the reading area
Girl Room, Reading Area, Girlsroom, Kidsroom, Light Fixture, Children S Room, Kids Rooms
#kidsroom teen bedroom #decorating girls room, wallpaper, theme, boys room, storage, bunk beds, colour, interior design, ceiling, bed, workspace
little girl's room #kids #play #room #home #family #children #prettyperfectkidsroom | Pretty Perfect Living |
Scandinavian nursery. More wallpaper ideas for kids rooms at www.justkidswallpaper.com #kidsrooms #kidswallpaper
Tiny Little Pads - Interiors for Kids: Scandinavian Retro Kids Room Inspiration.
I love the pillows on the floor for a reading area! the boo and the boy: decor
Gorgeous girlsroom with ceiling lamp from Normann Copenhagen
Girls' Rooms | Get a load of that organization!
little girl's room - LOVE the light fixture
Sweet kids space
Kids Bedrooms, Girl Room, Girlsroom, Girls Bedroom, Kidsroom, Kid S Room, Kids Rooms
Beautiful little girls room. F too #Electronic Toys| http://amazingelectronictoys.blogspot.com
Beautiful little girls room. F too
Colorful girl's bedroom decor
kid bed and room idea
Kidsroom Kinderzimmer
colorful kid's room
Kids Bedroom ideas
bedroom green... colorful room bedside styling
Kid's room <3
Wall Colour, Girl, Wall Color, Living Room, Vintage Kids Rooms, Kid S Room, Paint Colour
.#modern home design #interior design #living room design #home design| http://graphic-design-collections-johnson.blogspot.com
.#living room design #home decorating #home design ideas #home design #room designs
WALL COLOR- Working Space by Martine à la Maison, via Flickr
#werkplek #Kidsroom | Martine/Omstebeurt via Flickr
Nice paint colour for a girls room.
Cute wall color for a girls room
Blush pink,vintage kids room
Girl's room paint color
Love the wall colour
Great wall display
Brick Wall, For Kids, Kidsroom, Vintage Kids Room, Crate Shelves, Nursery, Kid S Room, Kids Rooms
Creative crates: Vintage crates are perfect to contain toys and collectibles and become a display in themselves! Find lots of cool vintage lights for kids' rooms at www.fatshackvintage.com.au!
Crate Shelves - perfect for extra storage without using up floor space! Paint in colors to coordinate in kids' rooms or playroom! You could also place them higher up for display items or keeping things away from little hands, or above a changing table for supplies.
wooden crates. cajas de madera. kids room. www.yourbox.bigcartel.com
The brick wall, the boxes... Great idea for kids room
chambre enfant, jolie bibliothèque/déco récup
❥ Cute crates for kids' rooms #house #decor
❥ DIY box shelves for kid's room
crate shelves for a kids rooms
Una delle mie passioni è il riciclo creativo e le cassette di legno si prestano benissimo a questo uso, sopratutto nella camera dei bambini. nicety.livejournal.com maruska21.blogspot.com.es Scatoloni di cartone si possono trasformare in una cucina completamente accessoriata o in una playhouse pieghevole per poterla riporre quando non viene usata. declutterorganizerepurpose.wordpress.com 25.media.tumblr.com handmadecharlotte.com Anche i vecchi giocattoli che spesso rimangono abbandonati in un angolo possono tornare a nuova vita e ad un nuovo uso. recyclart.org kidsroom.dk creaturecomfortsblog.com Via Pinterest
kids rhs
Kids Room, Kidsroom, Playroom, Baby Room, Playhouse
Castle hideaway for a little king or queen | 10 Fun & Friendly Playrooms Part 2 - Tinyme Blog
Cool corner for kids room. #children #kids #diy #kidsroom #idea
See more kids room design ideas here: www.delightfull.eu
10 Fun & Friendly Kids Playrooms Part 2 - Tinyme Blog
would be cute as a corner bird-style playhouse
Play Room Castle for Kids Liapela.com
Play house
baby room
Ready, set, play! Create the ultimate fun kids play space for your little ones with Part 2 in our collection of kids playroom inspiration...
love the walls
Floral Wallpapers, Ds Kids, Girls Room, Kidsroom, Cute Wallpapers, Girl Rooms, Kids Rooms
FabGoose Hookie hangers retro & sweet kid's room decoration: http://www.fabgoose.com/shop/junior_hookie_hangers/hookie-hanger-junior-3-color-pack-girl/
Lovely vintage look RHS - pretty pink and blue english country cottage - style floral wallpaper
Floral wallpaper in little girl's bedroom. Lovely
Kids room - Vintage wallpaper - Smilerynker
changing table and cute wallpaper
wallpaper, kids room
DS Kids brocante...
Little girls room
Cute wallpaper
Chalkboard Wall
Kids Room, Chalk Board, Kidsroom, Playroom, Children, Vintage Chalkboard
Kids' space with chalkboard wall http://www.unitednow.com/search.aspx?searchterm=Unframed+chalkboard
A collections of vintage chalkboards is a genius idea for decorating a kid's room. Great for kids who like to draw on walls.
nice somehow. rustic, retro and modern! the dollhouse and the pink chair <3 the dark floor, the different chalk boards are clever!
I had that dollhouse!!!!!! Children's room - Vintage chalkboard Wall - Smile And Wave. www.amamillo.com
chalkboard wall in kids rooms. Több keretes rész, több gyereknél jó
Children's room - Vintage chalkboard Wall - Smile And Wave.
Arty with chalk boards. Love the combo with the pink chair.
#Chalkboard #Wall #kidsroom #playroom #kids
assorted chalk boards- Fun for a playroom
Shared Room, Kids Room, Kidsroom, Children, Baby Room, Half Wall, Black Wall, White Room
Kids Room from Scotland #design #interior #kids #kidsroom #forkids #children #decor #ideas
cute shared room idea Oliver & Sebastian's room via AMM blog @Deborah Moir
white and black wall... Interesting color blocking for kids room
white floors, black walls, kids room via Remodelista
black and white room painting in children room
shared room | half and half wall
Half-wall in kid's room.
cute baby's room idea
Shared Kids Rooms Ideas
ABC... limited edition giclee print of an original illustration (8.5 x 11in)
Alphabet Helen, Animal Alphabet, Alphabet Poster, Helen Dardik
Alphabet Print from Helen Dardik – Limited Edition ABC Art – Nursery Decor from Etsy | Small for Big
Helen Dardik illustrated alphabet (LOVE THIS)
Illustrated alphabet - Helen Dardik
Adorable ABCs print by Helen Dardik
Helen Dardik illustrated alphabet
alphabet poster.
Helen Dardik alphabet print
Helen Dardik Animal Alphabet
Kid's Room Decorating Ideas


text:
oakley europe Madrid Helsinki ray bans prescription ry returning from Questionnaire in command。Year,Chengdu Brand new China examine results doubled,The variety of workers offers doubled。
Chengdu groove level of relaxation
Zhao Jingdong
thousand votes。Reporters then"Buy aged train"In the online seek out your keyword,There happen to be numerous ticketing website,At firsstained glass window panelsdding to that they may be well-built and ideal in my lanky go.
The only real drawback are the comments from 'funny' individuals who throw up the typical 'hilarious' Matrix feedback. I am just thinking of taking some fighting styles class so that I wi mcm bags genuinesure
to talk about independence along with democracy?What skills possess commitments in order to globe tranquility as well as prosperity?
Ask:Abe stated right after visiting the Yasukuni Shrine,He been to the actual Yasukuni Shrine has become a politics and diplomatic issue regrets,He did not hurt in、The idea of ????the sentiments from the Han people,Hope regarding the、With respect Han,Build friendly relations。What is the comm
price.zol.net.cn tru Hong Kong, China oakley sunglasses coupons
ram affirmation yet again.
A: breach associated with promise, it really is to show on the business owner seriously isn't stated to the customer to supply solutions, towards the detriment of the protection under the law and also interests regarding purchasers.Encouraging to be able to grievance launched problems: trading shut down inside 20 days right after (including cash-on-delivery deals).Violating claims problem dealing with time: resolve for protection under the law following your launch, customer care might help the actual ionline glasses retailer sky wherever the,People don't pay attention during the early,No remnants in the parts fall,And complement seem colorless all together the
t could be good for a starter as if you. The truth is, an earlier visit is a must.
as well as micro companies with regards to an optimistic signal。"Because the eye price management before,More distorted price tag signals。Banks regarding Thre oakley frog skin qualityamework goes
A century, nevertheless is now functioning in the 21st century!
2015 summertime brand-new fashion pine sun glasses men's oculos p sol glasses holbrook for females model custom goblet together with unique company logo
2015 Top Trend Sale RB 2140 Wayfarer Males Sun glasses Polarized Summertime Fashion Points Girls Sunshine Supra Shoes sale
glasses frames for womengether with extreme footwear shortages, to develop seashoes desalination and primary standby time with the long-term solution regarding a
ld involving assets。It is actually understood,Articles associated with your execution of the authorities is actually brewing。After the rules promulgated,Credit property regarding financial instituti ray ban usa shop for salesed together with the product packaging the eyeglasses were only available in (the particular eyeglasses were percolate covered along with
completely protected any time sent to myself). For the price tag I do believe these glasses are great and more importantly they're sturdy as well! I might actually advise these kind of sunglasses since they are UV400 security and look fantastic!
good spectacles and also very good value. come to one day. they're indeed big but they complete the task throughout protecting eye coming from sun's rays. ar


text:
By: Billy Cokit
Location: Mississauga, Toronto Canada
I was hiking with my friends Tom and Daniel up a mountain. My friends and I always took risks and made a meeting each week about what dangerous stunt we should do next.
It was Tom's idea to go hiking, so we searched for the best mountain we could find to climb. No luck finding one as big as Mount Everest, but we still found one a good one 1626 metres high.
The mountain was orange rock and quite challenging to hold onto, my friends and I ONLY used our hands when climbing. It was helpful that there were little jagged rocks sticking out of the mountain that we could hold onto.
After climbing about 1000 metres, we all took out our water bottles from our bags and drank while we stood on the little jagged rocks. They were thin so we had to stand on one foot.
I was concetrating so much on my drinking that I did not notice my weight pushing me down. I was falling from the sky 1000 metres up until my body started to stop little by little and float. I was back up the mountain, but Tom and Daniel never noticed what had happened.
You did not have to see to believe. Because even though I didn\'t see the angel, I know it was there. I just know it.
Some may never see with earthly eyes these heavenly beings or know that they exist. Many are unaware of the numerous ways they have protected and guided us. We would like to share with you the stories of people who take pleasure in expressing how real angels are.
At 33 Ambrose had it all - a successful career as a lawyer, an important position as governor of Milan, the approval and friendship of the emperor, and a large estate. Then the bishop ... continue reading
St. John Vianney, Priest (Patron of priests) Feast day - August 4 Universally known as the "Cure of Ars)," St. John Mary Vianney was ordained a priest in 1815. Three years later ... continue reading
The remarkable woman who would be known as Mother Theresa began life named Agnes Gonxha Bojaxhiu. Born on August 26, 1910 in Skopje, she was the youngest child born to Nikola and Drane ... continue reading
At the end of the sixth century anyone would have said that Augustine had found his niche in life. Looking at this respected prior of a monastery, almost anyone would have predicted he would spend ... continue readingMore Saint of the Day
St. Bernadette was born in Lourdes, France on January 7, 1844. Her parents were very poor and she was the first of nine children. She was baptized at St. Pierre's, the local parish church, on January 9. As a toddler, Bernadette contracted cholera and suffered extreme ... continue readingMore Female Saints
Saint Michael the Archangel isn't a saint, but rather he is an angel, and the leader of all angels and of the army of God. This is what the title "Archangel" means, that he is above all the others in rank. St. Michael has four main responsibilities or offices, as we ... continue reading
There is a romantic legend that the mother of Thomas Becket was a Saracen princess who followed his father, a pilgrim or crusader, back from the Holy Land, and wandered about Europe repeating the only English words she knew, "London" and "Becket," until she found him. ... continue reading
St. Catherine of Alexandria, Virgin and Martyr whose feast day is November 25th. She is the patroness of philosophers and preachers. St. Catherine is believed to have been born in Alexandria of a noble family. Converted to Christianity through a vision, she ... continue reading
By Marshall Connolly (CALIFORNIA NETWORK)
On September 4, 2016, Pope Francis will canonize Mother Teresa as a saint of the Roman Catholic Church. Her canonization is an important event for Catholics and all people around the world. Here's 3 reasons why. LOS ANGELES, CA (Catholic Online) - When Mother Teresa is ... continue readingMore Christian Saints & Heroes


text:
Stop Child Abuse
upload image
Stop Child Abuse images They never thought for this life HD wallpaper and background photos
Login or Join Fanpop to Download
Already have an account?
Sign In to Fanpop
Sign In to Fanpop
They never thought for this life. . HD Wallpaper and background images in the Stop Child Abuse club tagged: sad mean hurt crying children children rights.
Submitted by mjpeterpan7 over a year ago
Around the Web


text:
About Sweet_Meli : Bonjour ,
My name is Melissa ,
I'm a Medical Assistant
Currently Pursuing Marine Biology
I'm very athletic , and I'm addicted to Zumba
French & Mexican ;)
Socially Awkward by nature ;)
Write me .
About Sweet_Meli : Bonjour ,
Sweet_Meli's FML badges
You have looked through 5 pages of the website. That's a start.
Checking you out
You checked out the profile page belonging to one of the last people to have a look at your profile.
An insomniac or a creature of the dark
You commented on an FML between 1 and 3 am.
Sweet_Meli's favorite FMLs
- 1
- « Previous page
- Next page »
- Today, while messaging this guy I like on Facebook, I learned that you can set an emoji as the name… Today, my dog ran out into the middle of the street. A car was coming so I ran out to grab my dog.… Today, after an awful birthday I returned to my work week looking forward to keeping my mind busy.…
- Today, on my way home to Bordeaux after a weekend in Paris, I had the pleasure of being sat next to… Today, because I'm on my period, I asked my boyfriend to turn around so I could change my clothes.… Today, I'm a French teacher abroad, and as my beard has a huge hole near my chin, my students call…


text:
Intermarkets' Privacy Policy
Donate to Ace of Spades HQ!
Recent Entries
Overnight Open Thread (28 May 2016)
Saturday Afternoon Chess/Open Thread 05-28-2016 [OregonMuse]
Ace of Spades Pet Thread [Misanthropic Humanitarian]
Saturday Gardening Thread: Flags [KT]
How Many Transgenders In America? 0.006% = About 20,000. Total. [CBD]
Saturday Morning Oddities [Misanthropic Humanitarian]
Saturday Morning Open Thread - Niedermeyer's Dead Horse
Overnight Open Thread (27 May 2016)
Why Trump Won, and Why the Current Crop of Purported Leaders Must Be Swept Aside
Filler/Open Thread [CBD]
« Palin At Tea Party Convention | Main | Greenpeace UK call for Pachauri resignation, then retracts/clarifies/says it was "out of context"/whateveh »
February 06, 2010
Overnight Open Thread - Saturday Edition
Hello, hello all M&Ms. Welcome to your weekly Saturday night ONT.
Oh and kudos to holger for bringing a little old english poetry to the ONT last night. He gets 2 MaetPoints for that. Collect enough and you can get the steak knife set!
The hottest women of Super Bowl media
So who's up for some socialism?
Why Oscar is really a grouch
Sex as explained by pens
Capt. Phil Harris now breathing on his own
Microsoft to patch 17-year-old computer bug
The Yahoo AoSHQ group. Bla bla bla.
Tonight's post brought to you by speaking notes :
Notice: Posted by permission of AceCorp LLC. Please e-mail overnight open thread tips to maetenloch at gmail. Otherwise send tips to Ace.
| Access Comments
Recent Comments
G6loq: "Who's the chick behind Trump on the video? ..."
Mike Hammer, etc., etc.: " there's pie? I'm listenin'... Posted by: Alto ..."
Nevergiveup: " Obama Regime Launches Muslim Outreach Office ..."
ccoffer: "The Shapiro types imagined they were some sort of ..."
AltonJackson: "there's pie?I'm listenin'... ..."
speedster1: "131 MH Food thread's tomorrow. Moose out fro ..."
Mike Hammer, etc., etc.: "Wait... I thought this was a pie thread. This isn ..."
Skip: "128- Because I've read WWII history as long as I'v ..."
rickl: "I'm still waiting to hear whether SpaceX has manag ..."
Gushka can haz kittys what plays fetch!: "240 Gushka- Looking to expand my horizons. Have h ..."
Barry HUSSEIN 0Bola, ex-POTUS: ">>>But as I wrote yesterday my sympathy for what h ..."
OregonMuse: "[i]Posted by: Dirty Randy at May 28, 2016 09:10 PM ..."
Recent Entries
Overnight Open Thread (28 May 2016)
Saturday Afternoon Chess/Open Thread 05-28-2016 [OregonMuse]
Ace of Spades Pet Thread [Misanthropic Humanitarian]
Saturday Gardening Thread: Flags [KT]
How Many Transgenders In America? 0.006% = About 20,000. Total. [CBD]
Saturday Morning Oddities [Misanthropic Humanitarian]
Saturday Morning Open Thread - Niedermeyer's Dead Horse
Overnight Open Thread (27 May 2016)
Why Trump Won, and Why the Current Crop of Purported Leaders Must Be Swept Aside
Filler/Open Thread [CBD]
Polls! Polls! Polls!
Election Projection
Federal Review
Real Clear Politics
Rasmussen Reports
FoxNews Polls
Sabato/Crystal Ball
Frequently Asked Questions
The (Almost) Complete Paul Anka Integrity Kick
Primary Document: The Audio
Paul Anka Haiku Contest Announcement
Integrity SAT's: Entrance Exam for Paul Anka's Band
AllahPundit's Paul Anka 45's Collection
AnkaPundit: Paul Anka Takes Over the Site for a Weekend (Continues through to Monday's postings)
George Bush Slices Don Rumsfeld Like an F*ckin' Hammer
Top Top Tens
Democratic Forays into Erotica
New Shows On Gore's DNC/MTV Network
Nicknames for Potatoes, By People Who Really Hate Potatoes
Star Wars Euphemisms for Self-Abuse
Signs You're at an Iraqi "Wedding Party"
Signs Your Clown Has Gone Bad
Signs That You, Geroge Michael, Should Probably Just Give It Up
Signs of Hip-Hop Influence on John Kerry
NYT Headlines Spinning Bush's Jobs Boom
Things People Are More Likely to Say Than "Did You Hear What Al Franken Said Yesterday?"
Signs that Paul Krugman Has Lost His Frickin' Mind
All-Time Best NBA Players, According to Senator Robert Byrd
Other Bad Things About the Jews, According to the Koran
Signs That David Letterman Just Doesn't Care Anymore
Examples of Bob Kerrey's Insufferable Racial Jackassery
Signs Andy Rooney Is Going Senile
Other Judgments Dick Clarke Made About Condi Rice Based on Her Appearance
Collective Names for Groups of People
John Kerry's Other Vietnam Super-Pets
Cool Things About the XM8 Assault Rifle
Media-Approved Facts About the Democrat Spy
Changes to Make Christianity More "Inclusive"
Secret John Kerry Senatorial Accomplishments
John Edwards Campaign Excuses
John Kerry Pick-Up Lines
Changes Liberal Senator George Michell Will Make at Disney
Torments in Dog-Hell
Greatest Hitjobs
The Ace of Spades HQ Sex-for-Money Skankathon
A D&D Guide to the Democratic Candidates
Margaret Cho: Just Not Funny
More Margaret Cho Abuse
Margaret Cho: Still Not Funny
Iraqi Prisoner Claims He Was Raped... By Woman
Wonkette Announces "Morning Zoo" Format
John Kerry's "Plan" Causes Surrender of Moqtada al-Sadr's Militia
World Muslim Leaders Apologize for Nick Berg's Beheading
Michael Moore Goes on Lunchtime Manhattan Death-Spree
Milestone: Oliver Willis Posts 400th "Fake News Article" Referencing Britney Spears
Liberal Economists Rue a "New Decade of Greed"
Artificial Insouciance: Maureen Dowd's Word Processor Revolts Against Her Numbing Imbecility
Intelligence Officials Eye Blogs for Tips
They Done Found Us Out, Cletus: Intrepid Internet Detective Figures Out Our Master Plan
Shock: Josh Marshall Almost Mentions Sarin Discovery in Iraq
Leather-Clad Biker Freaks Terrorize Australian Town
When Clinton Was President, Torture Was Cool
What Wonkette Means When She Explains What Tina Brown Means
Wonkette's Stand-Up Act
Wankette HQ Gay-Rumors Du Jour
Here's What's Bugging Me: Goose and Slider
My Own Micah Wright Style Confession of Dishonesty
Outraged "Conservatives" React to the FMA
An On-Line Impression of Dennis Miller Having Sex with a Kodiak Bear
The Story the Rightwing Media Refuses to Report!
Our Lunch with David "Glengarry Glen Ross" Mamet
The House of Love: Paul Krugman
A Michael Moore Mystery (TM)
The Dowd-O-Matic!
Liberal Consistency and Other Myths
Kepler's Laws of Liberal Media Bias
John Kerry-- The Splunge! Candidate
"Divisive" Politics & "Attacks on Patriotism" (very long)
The Donkey ("The Raven" parody)
Powered by
Movable Type 2.64