text:
Understanding scan() in Theano
This is just short post since the meat is in an ipython notebook. I discuss the scan function in Theano, used for looping, iteration over tensors and recurrences. If you are a complete beginner to theano, check out my previous post. It seemed a bit obscure when I first started trying to understand it, so I thought I would share my learning process and hopefully speed up yours!
My sources for learning were these two great resources: this and that.
OK check it out here and let me know what you think.
Support Vector Machines and the Kernel Trick
Today we will be discussing a machine learning algorithm that is probably the most mathematically sophisticated of all the algorithms we have looked at so far.
Don't let that daunt you though! Not all the details are essential for understanding the general idea, and it can certainly be used without understanding the maths, although if you are content with this then there's not much point reading a blog like this!
This post is based on what I have learned from reading the relevant chapter in the excellent book: Elements of Statistical Learning.
The Setup
The premise is exactly the same as the perceptron algorithm, we have input data with class labels .
In order to classify the inputs, we seek a seperating hyperplane . Recall that a hyperplane is just a plane that may not pass through the origin, and can be described as
for some and .
It is handy to assume that is a unit vector, and geometrically it is the normal vector to the hyperplane. I will refer to as the bias. If the bias is zero, then is just a regular plane, and if it is nonzero is parallel to this plane, the bias controls how far 'above' or 'below' the origin is. In fact if , then .
If we define
then .
Moreover, is the signed distance of from . Note that if was not assumed to be a unit vector, it would instead be proportional to the signed distance. If we say that is above , and if we say that is below .
To classify a point , we say it has label if above , and if below.
The Problem(s)
If this is possible, then the perceptron algorithm will converge to a hyperplane that correctly classifies all of the points.
One problem is that there are infinitely many such hyperplanes, intuitively we would like to choose one to maximize the distances of all the points from it. If there are points close to the decision boundary, then it is likely that unseen points from the same distribution might up on the wrong side of the hyperplane and be misclassified. The minimum distance of all the points to is called the margin of the decision boundary, so we would like to maximise the margin.
Another problem is that the data might be too mixed up for there to be a seperating hyperplane, no matter which one you choose some points get misclassified. What we need is a way of choosing the hyperplane that does the best job of trying to seperate the points.
Finally it may be that a hyperplane is a bad approximation to the true decision boundary, we need something wiggilier.
Let's solve these problems in turn.
Maximizing the Margin
The first step in solving a machine learning problem is to figure out how to phrase it as an optimization problem, so let's try that.
Leting denote the margin, we would like to
maximize over ,
subject to for , and .
Note that implies that is correctly classified since their signs must be the same for the product to be positive.
Now we just do a litle finessing, notice that
, ie
is the same as
and so the optimization problem is equivalent to:
maximize over ,
subject to for , and ,
and so maximizing is the same as minimizing the norm of in this formulation, giving:
minimize over ,
subject to for .
Finally minimizing the norm is the same as minimizing half the square of the norm, and the latter is much more convienient since it doesn't have any of those pesky square roots:
minimize over ,
subject to for .
Now that we have the optimization problem setup, we use the Karush-Kuhn-Tucker conditions, normally when trying to minimize a function you look for places where the derivative is zero, and this the extension of that idea when you have constraints.
The conditions firstly say that
for .
The last condition shows that only for those such that , these are the that are 'on the margin', those input vectors closest to the optimal hyperplane.
This means that is determined only by these on the margin, we call these the support vectors.
The optimization problem can be solved using quadratic programming techniques, it is also sometimes useful to look at the equations given by the KKT conditions and convert it into the 'dual' optimization problem in terms of the .
Life isn't always easy, and sometimes the data is not linearly seperable. It is desirable that our algorithm be able to tolerate some errors and try to minimize them, otherwise, even if the data are linearly seperable, the algorithm may fall over itself trying to fit a few bad apples in the data, rather than going for a more natural fit that will hopefully generalize better.
We do this by introducing slack variables for and changing our constraint from
and as before this changes to
Of course we want to minimize the errors, and so we can recast our optimization problem as
minimize ,
subject to , for ,
where is a cost parameter that we need to choose, controlling our tolerance for mistakes.
The KKT conditions for this new problem are
where for .
Instead of trying to minimize an objective function subject to several complex constraints, it is often easier to collect the constraints into the Lagrangian and minimize that, subject to simpler constraints, this is called the penalty method.
In our case the Lagrangian is
and minimizing our objective function is equivalent to minimizing with respect to whilst at the same time maxmizing with respect to . This is why this is called a saddle-point method.
The constraints for this equivalent problem are and , the others are taken care of automagically.
Now we can substitute the KKT condition equations above to simplify , giving the dual Lagrangian
So our new problem is
subject to and ,
but doesn't appear in ! But then we notice that the KKT condition
is equivalent to constraining .
So our final formulation is
subject to and .
A much simpler formulation! In fact this is a convex quadratic programming problem, so it can be solved using out of the box methods from a library.
The Kernel Trick
We've now seen how to cope with errors by using a cost parameter , this is a form of regularization that can control overfitting. But what if the true decision boundary is geninuely something a bit more complex than a hyperplane? The solution to this problem is to change coordinates from using some mapping , then fit a hyperplane in this new space, then the pullback can be a nonlinear surface in .
This is of course a standard practice for any machine learning algorithm, feature engineering. What is special about support vector machines is that the function to be minimized is
notice that you can compute this using only inner products in whatever coordinate system you use. This means that if you have a function
such that
then there is no need to work directly in the transformed space at all! In fact you don't even need to know what is, as long as you have .
This is a big deal because this can be much more computationally efficient if you want to transform to a very high dimensional space. This is what is called the kernel trick, because we get to have the benefits of a richer feature space on the cheap.
So in order to do this, you just have to dream up a kernel , to make sure that your actually defines an inner product in some (Hilbert) space, by Mercer's Theorem what you need to check is that, for any choice of vectors , the matrix is positive semi-definite.
Some common choices are:
dth-degree polynomial kernel: ,
radial basis kernel: .
I have decided not to do a python implementation this time, because I would have to get in to quadratic programming to do it properly, perhaps I will write a post on this topic in the future.
Theano, Autoencoders and MNIST
Recently I have finally upgraded my ancient laptop. Other than the ability for me to play the occasional video game, this means that I now have a dedicated Nvidia graphics card, in particular one that supports something called CUDA.
Now that I have the hardware, I thought that it would be fun to check out this python library theano. As I understand it, it has two very interesting aspects.
One is that it does symbolic computation, that is it represents variables as in mathematics, and their manipulations.
A variable in computer science is something like a integer or a string or a list etc…, a piece of information stored somewhere in memory with a name made up by the programmer, and the compiler takes care of associating the human understandable name to the address of the information in memory.
A variable in mathematics is like in , an indeterminate quantity that can be manipulated using arithmetic operations, or using calculus and so on. So what in mathematics we call a variable corresponds roughly to a function in computer science.
The difference between symbolic computation and its variables, and regular functions, is that the programming language is aware of the mathematical nature of a variable, such that when you apply operations to variables to create new variables, it does this in an efficient way. The best way to understand this is a simple example. Let's suppose we have two functions . Now we can combine these functions by adding them to make a new function .
Now in python say, when you call with an argument, it will pass this argument to both and and then add the result and return it.
If we are doing symbolic computation, and were variables with , then the variable constantly. And so the computer does no work at all when using , because it knows that it must always be zero.
So one important aspect is that symbolic computation can simplify and hence often optimize functions automatically.
Another is that theano can automagically differentiate variables with respect to variables! This might not sound like a big deal, but it is. In machine learning, we often create a complicated function, like in logistic regression or neural networks, and then derive a helpful form for the derivative to do gradient descent to optimize it. In theano, all you have to do is specify the objective function, and it will let you do gradient descent automatically.
The second amazing thing about theano is that it compiles its theano functions into C, and so it often runs much much faster than native python or numpy. It can also make it easier to run computations on a gpu which can mean a 100x speedup for the right cases!
Hopefully I've made you curious, so click here for an installation guide. On this linux machine it was very simple. I am dual booting windows 8.1 and Ubuntu at the moment, it is very tricky to get the gpu aspect to work on windows, but here is a guide for the brave. To check that the GPU aspect is working, run the test code here.
Introduction to Theano
I recommend that you read the tutorials here, and also this excellent ipython worksheet here.
Let's little play with the basic functionality.
import theano as th from theano import tensor as T #To make a variable, specify the type of variable from the tensor #library. #We can have scalars. x=T.dscalar(name='scalar_x') #Note that the name is just a string that theano will use #to communicate to us about the variable. print x print x.type #We can also have vectors. v = T.dvector(name='vector_v') print v print v.type #And matrices. A = T.dmatrix(name='matrix_A') print A print A.type #We can also make new variables using standard mathematical operations. x_2 = x*x print x_2 print x_2.type #We can also make python functions which return new variables. def power(variable, n): return variable**n x_10 = power(x,10) print x_10 print x_10.type #We can of course do standard linear algebra operations also. Av = T.dot(A,v) print Av print Av.type #Of course when a variable is actually evaluated, you must #ensure that the dimensions are correct, or you will #get an error. #To see the value of a variable for a particular value of the variables #comprising it, we make a theano function. f = th.function([A,v], [Av]) #The syntax is a list of input variables, and then a list of output variables. #The python code takes a little longer to run initially, because thenao #compiles the function into C, but thereafter it will run extremely fast. #Let's try using the function. import numpy as np m=np.ones((2,2)) print m vec = np.asarray([1,2]) print vec print f(m,vec) #Now we can try computing gradients. #First let's make a scalar variable by taking an inner product. w=T.dvector(name='w_vector') vTw = T.dot(v,w) #Now we take the gradient with respect to w. vTw_grad = T.grad(vTw,w) print vTw_grad print vTw_grad.type #Now let's test it. vec1 = np.asarray([1,2]) vec2 = np.asarray([0,0]) #To evaulate a variable given inputs, there is another syntax #in additon to creating a thenano function. print vTw_grad.eval({w:vec1,v:vec2}) #Next we discuss theano's shared variables, #these differ from regular theano variables in that #theano variables only have a value within a theano function #whereas shared theano variables have a value independent of #being called in a function. w=T.shared(name='shared_matrix', value=np.ones((2,3))) print w print w.type print w.get_value() #You can also set a shared variable's value. w.set_value(np.zeros((2,3))) print w.get_value() #You can also have theano functions update shared variables, #we illustrate this with a silly example that updates a matrix #like in gradient descent. x=T.dvector('x') wx=T.dot(w,x) cost = wx.mean() cost_grad = T.grad(cost, w) f=th.function(inputs=[x], outputs=[wx, cost], updates=[(w, w-0.1*cost_grad)]) #Notice the syntax of updates argument, should be a list #of two tuples of the form: (variable_to_be_updated, updated_variable). print f([1,1,1]) print w.get_value()
If you want to tinker with this, get the sourcecode here.
I don't want to go into too much detail here, but an autoencoder is a feedforward neural network with a single hidden layer with the twist being that its target output is exactly the same as its input. If the number of hidden neurons is less than the number of input/output neurons, then the network will be forced to learn some sort of compression.
Even when the number of hidden neurons is large, there are various tricks in cost functions you can use to ensure that the network does something more interesting than learning the identity function.
The hope of an autoencoder is that it learns some structure in the data, so that the hidden layer can be thought of as a feature space that one can use for a supervised learning algorithm.
Basically then we have the hidden layer, then , where are bias vectors, are matrices, and are element-wise non-linearities like the logistic function.
A simplification that is often made is to enforce . This gives the model less free parameters to learn, and so is a form of regularization. It also uses less memory. This is referred to as an autoencoder with tied weights.
Autoencoders with Theano
Thanks to automatic differentiation, backprogation is effortless with theano, and thanks to gpu computation, much faster also! I recommend having a look at my python implementation of backpropagation just so you can see the effort saved. This implementation benefited greatly from this tutorial.
import numpy as np import theano as th from theano import tensor as T from numpy import random as rng class AutoEncoder(object): def __init__(self, X, hidden_size, activation_function, output_function): #X is the data, an m x n numpy matrix #where rows correspond to datapoints #and columns correspond to features. assert type(X) is np.ndarray assert len(X.shape)==2 self.X=X self.X=th.shared(name='X', value=np.asarray(self.X, dtype=th.config.floatX),borrow=True) #The config.floatX and borrow=True stuff is to get this to run #fast on the gpu. I recommend just doing this without thinking about #it until you understand the code as a whole, then learning more #about gpus and theano. self.n = X.shape[1] self.m = X.shape[0] #Hidden_size is the number of neurons in the hidden layer, an int. assert type(hidden_size) is int assert hidden_size > 0 self.hidden_size=hidden_size initial_W = np.asarray(rng.uniform( low=-4 * np.sqrt(6. / (self.hidden_size + self.n)), high=4 * np.sqrt(6. / (self.hidden_size + self.n)), size=(self.n, self.hidden_size)), dtype=th.config.floatX) self.W = th.shared(value=initial_W, name='W', borrow=True) self.b1 = th.shared(name='b1', value=np.zeros(shape=(self.hidden_size,), dtype=th.config.floatX),borrow=True) self.b2 = th.shared(name='b2', value=np.zeros(shape=(self.n,), dtype=th.config.floatX),borrow=True) self.activation_function=activation_function self.output_function=output_function def train(self, n_epochs=100, mini_batch_size=1, learning_rate=0.1): index = T.lscalar() x=T.matrix('x') params = [self.W, self.b1, self.b2] hidden = self.activation_function(T.dot(x, self.W)+self.b1) output = T.dot(hidden,T.transpose(self.W))+self.b2 output = self.output_function(output) #Use cross-entropy loss. L = -T.sum(x*T.log(output) + (1-x)*T.log(1-output), axis=1) cost=L.mean() updates=[] #Return gradient with respect to W, b1, b2. gparams = T.grad(cost,params) #Create a list of 2 tuples for updates. for param, gparam in zip(params, gparams): updates.append((param, param-learning_rate*gparam)) #Train given a mini-batch of the data. train = th.function(inputs=[index], outputs=[cost], updates=updates, givens={x:self.X[index:index+mini_batch_size,:]}) import time start_time = time.clock() for epoch in xrange(n_epochs): print "Epoch:",epoch for row in xrange(0,self.m, mini_batch_size): train(row) end_time = time.clock() print "Average time per epoch=", (end_time-start_time)/n_epochs def get_hidden(self,data): x=T.dmatrix('x') hidden = self.activation_function(T.dot(x,self.W)+self.b1) transformed_data = th.function(inputs=[x], outputs=[hidden]) return transformed_data(data) def get_weights(self): return [self.W.get_value(), self.b1.get_value(), self.b2.get_value()]
And that is it! Now we should test this puppy out, so let's find some data.
The MNIST data is famous in machine learning circles, it consists of single handwritten digits. Nearly every paper written on neural networks and so on tests their contribution on the data. The task is to classify the digits, but we will just test our autoencoder. First we will steal a very nice helper function to load the data set, this will download it for you if you don't have it already. You can find it here.
import cPickle import gzip import os
def load_data(dataset): ''' Loads the dataset :type dataset: string :param dataset: the path to the dataset (here MNIST) ''' ############# # LOAD DATA # ############# # Download the MNIST dataset if it is not present data_dir, data_file = os.path.split(dataset) if data_dir == "" and not os.path.isfile(dataset): # Check if dataset is in the data directory. new_path = os.path.join(os.path.split(__file__)[0], "..", "data", dataset) if os.path.isfile(new_path) or data_file == 'mnist.pkl.gz': dataset = new_path if (not os.path.isfile(dataset)) and data_file == 'mnist.pkl.gz': import urllib origin = 'http://www.iro.umontreal.ca/~lisa/deep/data/mnist/mnist.pkl.gz' print 'Downloading data from %s' % origin urllib.urlretrieve(origin, dataset) print '... loading data' # Load the dataset f = gzip.open(dataset, 'rb') train_set, valid_set, test_set = cPickle.load(f) f.close() #train_set, valid_set, test_set format: tuple(input, target) #input is an numpy.ndarray of 2 dimensions (a matrix) #which row's correspond to an example. target is a #numpy.ndarray of 1 dimensions (vector)) that have the same length as #the number of rows in the input. It should give the target #target to the example with the same index in the input. return (train_set, valid_set, test_set) path="/home/harri/Dropbox/Work/Blogs/triangleinequality/Theano/data/mnist.pkl.gz" data=load_data(path)
We also want a plotting function to visualize the features learned, I won't explain this here, but if you google "visualizing features of neural network" you can learn plenty about it.
def plot_first_k_numbers(X,k): from matplotlib import mpl,pyplot m=X.shape[0] k=min(m,k) j = int(round(k / 10.0)) fig, ax = pyplot.subplots(j,10) for i in range(k): w=X[i,:] w=w.reshape(28,28) ax[i/10, i%10].imshow(w,cmap=pyplot.cm.gist_yarg, interpolation='nearest', aspect='equal') ax[i/10, i%10].axis('off') pyplot.tick_params(\ axis='x', # changes apply to the x-axis which='both', # both major and minor ticks are affected bottom='off', # ticks along the bottom edge are off top='off', # ticks along the top edge are off labelbottom='off') pyplot.tick_params(\ axis='y', # changes apply to the x-axis which='both', # both major and minor ticks are affected left='off', right='off', # ticks along the top edge are off labelleft='off') fig.show()
Now we put it together.
def m_test(data): X=data[0][0] activation_function = T.nnet.sigmoid output_function=activation_function A = AutoEncoder(X, 500, activation_function, output_function) A.train(20,20) W=np.transpose(A.get_weights()[0]) plot_first_k_numbers(W, 100, False) plot_first_k_numbers(W,100, True) m_test(data)
Running this, with 500 hidden units for 20 epochs with minibatch size of 20, the following features were learned:
Nothing too interesting yet, in order to get more useful features we could change the cost function to encourage sparsity, or introduce noise into the inputs, or use a different activation function. Hopefully this has been helpful for those just starting out with theano. Here is the source code.
How to Balance your Binary Search Trees – AVL Trees
Last time we introduced the binary search tree (BST) and saw that they could do inserts and deletions in time where is the height of the tree. We also saw that it could find the successor or predecessor to a node in the same time, and hence that it could sort a list in time, where is the length of the list.
Ideally we would have , and this is often the case 'on average', but the worst case is , which will occur if we start with a sorted list! So today we are going to discuss how to have our tree try to balance itself automagically so as to keep its height .
Recall that the defining characteristic of a BST is the so-called 'search property': for any subtree rooted at a node, the stuff on the left is smaller, and the stuff on the right is greater.
We now introduce another property or invariant that we will insist upon and call it the 'plus or minus property' (pm) or 'balance property', and it is satisfied by a node if the height of its left subtree differs from the height of its right subtree by no more than one. We call a BST that satisfies pm an 'AVL tree' (AVL).
So we will give each node an extra field node.height, and node.balance, where node.balance is equal to the height on the left minus the height on the right. Hence it will be 0 when they are equal, +1 when the left is larger, and -1 when the right is larger.
Before we go any further, you might be wondering whether or not this property forces the tree to have height, and indeed it does, so let's prove that to ourselves before coercing our trees to have this property.
Proof That AVL Trees are Balanced
Define to be the smallest number of nodes you can build a AVL of height with. That is, any AVL of height must have at least nodes.
Now suppose that we had shown that , or in particular that for all . Now take any AVL tree of height and having nodes. Then as desired.
So we just need to show that . The way to do this is by setting up a recurrence relation. Indeed we will see that .
Now using this notice that:
And so if then applying this repeatedly yields:
On the other hand if then we apply till we get to giving:
Combining these gives . Now taking logs base gives
And we are done.
Keeping Track of the AVL Property
Now that we agree that the AVL or pm property is a thing worth having, we should alter our code for a BST so that the nodes have height and balance fields, and so that the insertion and deletion methods update this accordingly. Once we have done that we will look at how to enforce the balance property.
Now we could use class inheritance and modify the BST class we built last time, but some of the code is a bit kludgy and so I will implement this from scratch, although I won't go into detail about the stuff that is the same from last time.
We begin with the AVLNode and gift it with a sense of balance.
class AVLNode(object): def __init__(self, key): self.key=key self.right_child=None self.left_child=None self.parent=None self.height=0 self.balance=0 def update_height(self, upwards=True): #If upwards we go up the tree correcting heights and balances, #if not we just correct the given node. if self.left_child is None: #Empty left tree. left_height = 0 else: left_height = self.left_child.height+1 if self.right_child is None: #Empty right tree. right_height = 0 else: right_height = self.right_child.height+1 #Note that the balance can change even when the height does not, #so change it before checking to see if height needs updating. self.balance = left_height-right_height height = max(left_height, right_height) if self.height != height: self.height = height if self.parent is not None: #We only need to go up a level if the height changes. if upwards: self.parent.update_height() def is_left(self): #Handy to find out whether a node is a left or right child or neither. if self.parent is None: return self.parent else: return self is self.parent.left_child
I think the code, and comments, should explain everything if you read my last post. Let's also pretend that we had a magic function balance(node), which will balance a node and then move up ancestors and do the same, we won't worry about what exactly it will do, but add it into our insert routine.
class AVLTree(object): def __init__(self): self.root =None def insert(self, key, node=None): #The first call is slightly different. if node is None: #First call, start node at root. node = self.root if node is None: #Empty tree, create root. node = AVLNode(key=key) self.root=node return node else: ret= self.insert(key=key, node=node) self.balance(ret) return ret #Not a first call. if node.key ==key: #No need to insert, key already present. return node elif node.key >key: child = node.left_child if child is None: #Reached the bottom, insert node and update heights. child = AVLNode(key=key) child.parent=node node.left_child = child node.update_height() return child else: return self.insert(key=key, node=child) elif node.key < key: child = node.right_child if child is None: #Reached the bottom, insert node and update heights. child = AVLNode(key=key) child.parent=node node.right_child = child return child else: return self.insert(key=key, node=child) else: print "This shouldn't happen." def find(self, key, node=None): if node is None: #First call. node=self.root if self.root is None: return None else: return self.find(key, self.root) #Now we handle nonfirst calls. elif node.key == key: #Found the node. return node elif key < node.key: if node.left_child is None: #If key not in tree, we return a node that would be its parent. return node else: return self.find(key,node.left_child) else: if node.right_child is None: return node else: return self.find(key, node.right_child) def delete(self, key, node=None): #Delete key from tree. if node is None: #Initial call. node = self.find(key) if (node is None) or (node.key != key): #Empty tree or key not in tree. return if (node.left_child is None) and (node.right_child is not None): #Has one right child. right_child = node.right_child left = node.is_left() if left is not None: parent=node.parent if not left: parent.right_child=right_child else: parent.left_child=right_child right_child.parent =parent self.balance(parent) else: right_child.parent=None self.root = right_child #No need to update heights or rebalance. elif (node.left_child is not None) and (node.right_child is None): #Has one left child. left_child = node.left_child left= node.is_left() if left is not None: parent=node.parent if left: parent.left_child=left_child else: parent.right_child=right_child left_child.parent =parent self.balance(parent) else: left_child.parent=None self.root=left_child elif node.left_child is None: #Has no children. parent = node.parent if parent is None: #Deleting a lone root, set tree to empty. self.root = None else: if parent.left_child is node: parent.left_child =None else: parent.right_child=None self.balance(parent) else: #Node has two childen, swap keys with successor node #and delete successor node. right_most_child = self.find_leftmost(node.right_child) node.key = right_most_child.key self.delete(key=node.key,node=right_most_child) #Note that updating the heights will be handled in the next #call of delete. def find_rightmost(self, node): if node.right_child is None: return node else: return self.find_rightmost(node.right_child) def find_leftmost(self, node): if node.left_child is None: return node else: return self.find_leftmost(node.left_child) def find_next(self, key): node = self.find(key) if (node is None) or (node.key != key): #Key not in tree. return None else: right_child = node.right_child if right_child is not None: node= self.find_leftmost(right_child) else: parent = node.parent while(parent is not None): if node is parent.left_child: break node = parent parent = node.parent node=parent if node is None: #Key is largest in tree. return node else: return node.key def find_prev(self, key): node = self.find(key) if (node is None) or (node.key != key): #Key not in tree. return None else: left_child = node.left_child if left_child is not None: node= self.find_rightmost(left_child) else: parent = node.parent while(parent is not None): if node is parent.right_child: break node = parent parent = node.parent node=parent if node is None: #Key is largest in tree. return node else: return node.key #I also include a new plotting routine to show the balances or keys of the node. def plot(self, balance=False): #Builds a copy of the BST in igraphs for plotting. #Since exporting the adjacency lists loses information about #left and right children, we build it using a queue. import igraph as igraphs G = igraphs.Graph() if self.root is not None: G.add_vertices(1) queue = [[self.root,0]] #Queue has a pointer to the node in our BST, and its index #in the igraphs copy. index=0 not_break=True while(not_break): #At each iteration, we label the head of the queue with its key, #then add any children into the igraphs graph, #and into the queue. node=queue[0][0] #Select front of queue. node_index = queue[0][1] if not balance: G.vs[node_index]['label']=node.key else: G.vs[node_index]['label']=node.balance if index ==0: #Label root green. G.vs[node_index]['color']='green' if node.left_child is not None: G.add_vertices(1) G.add_edges([(node_index, index+1)]) queue.append([node.left_child,index+1]) G.vs[index+1]['color']='red' #Left children are red. index+=1 if node.right_child is not None: G.add_vertices(1) G.add_edges([(node_index, index+1)]) G.vs[index+1]['color']='blue' queue.append([node.right_child, index+1]) index += 1 queue.pop(0) if len(queue)==0: not_break=False layout = G.layout_reingold_tilford(root=0) igraphs.plot(G, layout=layout)
If you want to make sure it works, I have written a small test function.
def test(): lst= [1,4,2,5,1,3,7,11,4.5] B = AVLTree() for item in lst: print "inserting", item B.insert(item) B.plot(True) print "End of inserts" print "Deleting 5" B.plot(True) B.delete(5) print "Deleting 1" B.plot(True) B.delete(1) B.plot(False) print B.root.key ==4 print B.find_next(3) ==4 print B.find_prev(7)==4.5 print B.find_prev(1) is None print B.find_prev(7)==4.5 print B.find_prev(2) is None print B.find_prev(11) == 7
Great now we know the balance of each node, but how to balance the tree?
Tree Rotations
Before solving our balancing problem I want to introduce a new tool, an operation called a tree rotation that respects the search property. Here's what a tree rotation looks like:
So there are two moves that are inverses of one another, a right rotation and a left rotation. Staring at the diagram for a little bit you can also see that these moves preserve the search property. Let's implement this, and then consider when to use it.
We'll first create a right rotation, then swap the lefts for rights and vice versa to make a left rotation.
The argument to the function will always be the root, that is the parent of the two nodes, so in our diagram above, the label root is appropriate for the right-rotation.
We then move around the the parent-child pointers. Notice that the relation between the root and , and the relation between pivot and , is unchanged. The changes we have to make are that changes parents, and that root and pivot swap places from parent/child to child/parent.
So the left-child of root, pivot, is changed to the right-child of pivot, , the right-child of pivot is changed to root, the parent of root is changed to pivot and finally the previous parent of root, if any, becomes the new parent of pivot.
def right_rotation(self, root): left=root.is_left() pivot = root.left_child if pivot is None: return root.left_child = pivot.right_child if pivot.right_child is not None: root.left_child.parent = root pivot.right_child = root pivot.parent = root.parent root.parent=pivot if left is None: self.root = pivot elif left: pivot.parent.left_child=pivot else: pivot.parent.right_child=pivot root.update_height(False) pivot.update_height(False)
And analogously:
def left_rotation(self, root): left=root.is_left() pivot = root.right_child if pivot is None: return root.right_child = pivot.left_child if pivot.left_child is not None: root.right_child.parent = root pivot.left_child = root pivot.parent = root.parent root.parent=pivot if left is None: self.root = pivot elif left: pivot.parent.left_child=pivot else: pivot.parent.right_child=pivot root.update_height(False) pivot.update_height(False)
Now we ought to test it so using the same list as last time:
def test_rotation(): lst= [1,4,2,5,1,3,7,11,4.5] print "List is ",lst B = AVLTree() for item in lst: print "inserting", item B.insert(item) node=B.find(4) B.plot() B.left_rotation(node) B.plot() B.right_rotation(node.parent) B.plot() test_rotation()
Producing the following two plots:
Finally we show how to use tree rotations to keep a AVL tree balanced after insertions and deletions.
Bringing Balance to the Force
After an insertion or deletion, heights can change by at most 1. So assuming that the tree was balanced prior to the operation, if a node becomes unbalanced, it will have a balance of .
Let's look at the case where we have a node with balance with some pictures, the other case is the same modulo lefts and rights.
This is called the left-left case. Alternatively we could have the left-right case:
This takes to a similar case to before:
To implement this is simple now that we have the right tools. We want to function to start with the changed node, either the inserted node in insertion, or the parent of the deleted node in deletion, and move up through the ancestors, updating heights and balancing as it goes.
def balance(self, node): node.update_height(False) if node.balance == 2: if node.left_child.balance != -1: #Left-left case. self.right_rotation(node) if node.parent.parent is not None: #Move up a level. self.balance(node.parent.parent) else: #Left-right case. self.left_rotation(node.left_child) self.balance(node) elif node.balance ==-2: if node.right_child.balance != 1: #Right-right case. self.left_rotation(node) if node.parent.parent is not None: self.balance(node.parent.parent) else: #Right-left case. self.right_rotation(node.right_child) self.balance(node) else: if node.parent is not None: self.balance(node.parent)
We might as well make a sorting routine to wrap the AVL Tree now.
def sort(lst, ascending=True): A = AVLTree() for item in lst: A.insert(item) ret=[] if ascending: node=A.find_leftmost(A.root) if node is not None: key = node.key else: key=node while (key is not None): ret.append(key) key=A.find_next(key) else: node=A.find_rightmost(A.root) if node is not None: key = node.key else: key=node while (key is not None): ret.append(key) key=A.find_prev(key) return ret
And that's it! Please let me know if you spot any mistakes. You can download the source here.
Binary Search Trees
So in preparation for starting a PhD in Machine Learning/Data Science at the University of Edinburgh .
I have been watching some of the lectures on algorithms and data structures at MIT here.
One of the data structures discussed is the binary search tree (BST), and so in this post I will explain what they are and give a python implementation.
As you might expect, a BST is a way of organizing data in a binary tree. It consists of a collection of nodes, each of which has a key, corresponding to a piece of data, say an integer. Importantly the keys must be comparable, so that for any two nodes either or .
Each node also has (at most) three pointers: parent, left child and right child.
Most importantly we have the invariant of the BST. An invariant is, of course, something that stays the same, a property that is not altered under any of the data structures allowable operations. The invariant or search property is that for any given node all of the nodes in the left subtree rooted at have keys less than , and all of the nodes in the right subtree have keys greater than .
Any useful data structure has operations. Some obvious things to want to do are to insert and delete. Of course in order to do these things we have to take care not to mess up the search property. To insert we begin at the root and at each stage we ask whether the key to be inserted is lesser, greater or equal to the parent, in which case we move right, left or nowhere respectively. It's easiest just to see an example:
Now I think you get the idea, so let's start coding this up. First we will make the BST node.
class TreeNode(object): #A tree node has a key, which can be compared with other keys, #and possibly 'pointers' to a parent, left child or right child. def __init__(self, key, parent=None, left_child=None, right_child=None): self.parent=parent self.left_child=left_child self.right_child=right_child self.key=key
Next we will start making the BST. The BST remembers the root, and the pointers do the rest of the work. We begin by implementing a helpful method called 'find', which returns a node with a given key, or that could (but doesn't) have a child with that key.
class BinarySearchTree(object): def __init__(self, root=None): self.root=root def find(self, key): #Returns a node corresponding to a key. #If key is the key of some node, returns that node. #If tree is empty, returns None. #Otherwise returns a leaf that could accept #a node with that key as a child. #This function wraps a call to a recursive function self._find. if self.root is None: return None else: return self._find(key, self.root) def _find(self, key, node): #This is a recursive function that does all the work for #self.find. if key == node.key: return node elif key < node.key: if node.left_child is None: return node else: return self._find(key, node.left_child) else: if node.right_child is None: return node else: return self._find(key, node.right_child)
Using 'find' we can then implement 'insert'.
def insert(self, key): #Inserts a node with given key. #If key already in tree, then returns the node with that key. #Otherwise creates a new node and returns it. #This takes time of order the height of the tree. node = self.find(key) if node is None: self.root=TreeNode(key) return self.root elif key == node.key: return node elif key < node.key: left_child = TreeNode(key, parent=node) node.left_child=left_child return left_child else: right_child = TreeNode(key, parent=node) node.right_child=right_child return right_child
So far so good!
The next thing we want to do is be able to delete keys.
If a key belongs to a leaf node then this is simple we just delete the node.
If the node has one child, then if it has a parent then when we delete the node we must connect the parent to the child of the deleted node.
If the node has two children things are a bit more tricky. If we delete the node then we break the tree. What we need to do instead is replace the node's key with something else in the tree. What we replace it with is the rightmost node of the subtree rooted at the node's left child, as we shall see later this is the successor of the node in the BST. The following diagram should explain:
So we swap the node's key for this new rightmode node's, then delete the rightmost node. Since it is rightmost it cannot have a right child, and so it is one of our base cases for the deletion operation. If this is still not clear to you, do some examples on paper and you will soon see the idea. Let's implement this.
def delete(self, key): #Delete key from tree. #If key is not in BST does nothing. #Otherwise it calls a semi-recursive function _delete. node = self.find(key) if (node is None) or (node.key != key): return else: self._delete(node) def _delete(self, node): #If the node has less than two children we can delete the node #directly by removing it and then gluing the tree back together. #If node has two children, it swaps a key lower down in the tree to #replace the deleted node, and deletes the lower down node. if (node.left_child is None) and (node.right_child is not None): #Has one right child. right_child = node.right_child parent = node.parent if parent is not None: parent.right_child=right_child right_child.parent =parent else: right_child.parent=None if node is self.root: self.root=right_child elif (node.left_child is not None) and (node.right_child is None): #Has one left child. left_child = node.left_child parent = node.parent if parent is not None: parent.left_child=left_child left_child.parent =parent else: left_child.parent=None if node is self.root: self.root=left_child elif node.left_child is None: #Has no children. parent = node.parent if parent is None: self.root = None else: if parent.left_child is not None: left = parent.left_child is node else: left = False if left: parent.left_child =None else: parent.right_child=None else: right_most_child = self.find_leftmost(node.right_child) node.key = right_most_child.key self._delete(right_most_child) def find_rightmost(self, node): if node.right_child is None: return node else: return self.find_rightmost(node.right_child) def find_leftmost(self, node): if node.left_child is None: return node else: return self.find_leftmost(node.left_child)
Ok great, we have now covered the basics of how to modify a BST. Now we want to know how to extract information from it. In particular given a key, we want to be able to find the next largest or smallest key in the BST, if any. Calling the successor (predecessor) operation repeatedly, starting from the left(right)-most node, will give us the sorted keys in ascending(descending) order.
I will explain how to find the successor, the predecessor case is of course very similar. Let's suppose we want to find the successor of node with right-child . In this case we choose the left-most child of which we will call .
Note that in a subtree rooted at some node, the left-most node is the smallest element in the subtree, and the right-most node is the largest.
So in the subtree rooted at , the right-subtree rooted at contains all the elements in the subtree greater than . The left-most element of the right-subtree is then the smallest element in the subtree greater than and is a good candidate for a successor. Let's draw a picture and try to see why this must in fact be the correct node.
OK, but what if has no right child? In this case we move up through the ancestors of , searching for the first ancestor that is a left-child of its parent. If there is no such node, ie we get to the root, then is the right-most, and hence largest, element in the tree, and has no successor. To see that this is correct employ a similar argument to the previous case.
Let's code that up.
def find_next(self, key): node = self.find(key) if (node is None) or (node.key != key): #Key not in tree. return else: ret= self._find_next(node) if ret is None: #Key is largest in tree. return ret else: return ret.key def _find_next(self, node): right_child = node.right_child if right_child is not None: return self.find_leftmost(right_child) else: parent = node.parent while(parent is not None): if node is parent.left_child: break node = parent parent = node.parent return parent def find_prev(self, key): node = self.find(key) if (node is None) or (node.key != key): #Key not in tree. return else: ret= self._find_prev(node) if ret is None: #Key is smallest in tree. return ret else: return ret.key def _find_prev(self, node): left_child = node.left_child if left_child is not None: return self.find_rightmost(left_child) else: parent = node.parent while(parent is not None): if node is parent.right_child: break node = parent parent = node.parent return parent
By now I think we deserve to see the fruits of our hard work, so let's whip up a plotting routine using igraphs. Don't worry about the details: since I can't find a binary tree plotting style in igraphs, I have colored the root green, left children red and right children blue.
In order to preserve information about left and right children, the plotting routine builds the graph in igraphs using a queue.
def plot(self): #Builds a copy of the BST in igraphs for plotting. #Since exporting the adjacency lists loses information about #left and right children, we build it using a queue. import igraph as igraphs G = igraphs.Graph() if self.root is not None: G.add_vertices(1) queue = [[self.root,0]] #Queue has a pointer to the node in our BST, and its index #in the igraphs copy. index=0 not_break=True while(not_break): #At each iteration, we label the head of the queue with its key, #then add any children into the igraphs graph, #and into the queue. node=queue[0][0] #Select front of queue. node_index = queue[0][1] G.vs[node_index]['label']=node.key if index ==0: #Label root green. G.vs[node_index]['color']='green' if node.left_child is not None: G.add_vertices(1) G.add_edges([(node_index, index+1)]) queue.append([node.left_child,index+1]) G.vs[index+1]['color']='red' #Left children are red. index+=1 if node.right_child is not None: G.add_vertices(1) G.add_edges([(node_index, index+1)]) G.vs[index+1]['color']='blue' queue.append([node.right_child, index+1]) index += 1 queue.pop(0) if len(queue)==0: not_break=False layout = G.layout_reingold_tilford(root=0) igraphs.plot(G, layout=layout)
Now let's test it!
def test(): lst= [1,4,2,5,1,3,7,11,4.5] B = BinarySearchTree() for item in lst: B.insert(item) B.plot() B.delete(5) B.delete(1) B.plot() print B.root.key print B.find_next(3) print B.find_prev(7) print B.find_prev(1) test()
This produced the following output:
None" to the console, as desired.
You can probably see that the operations take time where is the height of the tree. This means that we can sort a length list in time. On average , but if given a monotonic list we will get a linear tree with . So next time we will see how to 'balance' the tree using tree rotations to achieve fast operations.
Please feel free to leave any comments, suggestions or corrections! Here is the source code.
Entropy: Expect a Surprise
Defining Surprise
Let's suppose we have a system with states which happen with probabilities . On the other hand we have our model for the system which encode our beliefs or predictions about the states' probabilities .
How surprised would we be if happens? What properties should the surprise have? Let's make a wish list:
- First the surprise should be a function that takes a probability to a real number representing the surprise.
- Secondly if , then we will be infinitely surprised, so in this case.
- Similarly if , then we won't be surprised at all, so .
- We would also like to be a nice function to work with, so let's have it be continuously differentiable.
One final property is that I would like surprise to be additive or extensive , this means that for independent events , I would like to be . This is the main difference between probability and surprise, surprises add up, whereas probabilities multiply. I think it easier to think about adding so this makes me happy. So in general we want .
If surprise doesn't seem to be additive to you, most likely because you have been reared on probabilities, you can substitute 'surprise' for 'information'.
If you play around with these constraints you will find you need to be a negative logarithm, that is . If we choose a our base, as mathematicians prefer, we measure it in nats, if we work base 2, as computer scientists like, we have bits. This is also referred to as the self-information.
Defining Entropy
A natural quantity to compute is
the expected surprise. This is only defined when , and whenever , by convention we let . If this offends you then you should alter the definition to take the sum over the support of !
Natural quantities deserve special notation and so we write
for ,
and refer to it as the cross entropy.
In the special case where , where the model matches the true probabilities, we call the entropy of and write or for this quantity. Using Jensen's inequality, we can show that the entropy is maximised iff for each outcome, in which case .
To get a feel for this maximization of entropy, we plot the binary case where we have just two outcomes. So letting the probability of the first outcome be , let's see how the surprise changes with , and below that how the entropy changes:
As you can see, you can make the first outcome very surprising or informative, but you pay for that by making the complementary outcome less surprising, the lower graph shows that the optimum compromise comes from having both outcomes be equally likely.
Intuitively any approximate model will increase the surprise so that .
The difference is then a measure of much more surprised we are when using the model, this quantity is written and called the Kullback-Leibler divergence, which is a kind of oriented distance between distributions, notice that it is not symmetric and so not a metric.
Our intuition that can then rewritten as , and in this form it is known as Gibbs' inequality.
Proof of Gibbs' Inequality
This is pretty simple, since is a concave function, is convex, and we can use Jensen's inequality, writing and we have:
Also notice that if so the inequality is sharp, and moreover, since we are dealing with a strictly convex function the divergence is zero if and only if .
Conditional Entropy
Now for two random variables we can consider the entropy of , conditioned upon . We can call this , now as ever we take the expectation to give the conditional entropy of given .
Including more information can only mean less surprise so we must have
We also have something called the chain rule:
where is the joint entropy.
So we have
and using the definition of conditional probability this becomes
which is just the joint entropy minus
and this is just the entropy, and so
as required.
Mutual Information
Similarly to the KL-divergence, we can test how much this inequality is observed, defining
which we call the mutual information of and .
In fact the analogy between mutual information and KL-divergence can be made precise if we write for the joint density and for the respective marginal densities, we have
to prove this is not hard and so I won't bother. Notice that Gibb's inequality then implies what we guessed before, that conditioning does not lose information (or increase surprise).
What is the interpretation? Well if and only if and are independent. So we are measuring the how much more surprised we will be if we assume that are independent, alternatively, how much information is lost by modelling as being independent, so it is a measure of dependence.
Fascinating! It is interesting to compare this with the correlation of and . This is also a measure of dependence, recording the extent to which large values of correspond to large values of for positive correlation, or large values of correspond to small (negative) values of . It is possible to be uncorrelated but be dependent.
By contrast mutual information is totally oblivious to large and small values, it cares only about the probabilities and so in my opinion is much more beautiful. What's more, by Gibb's inequality, the mutual information is zero if and only if are independent! So it is a purely informational measure of dependency, in that it fails to encode anything about how the dependency is seen from the perspective of the metric. So it really depends on the context which is more useful. For making estimates of random variables, the values are important, and so the correlation or covariance will likely be more useful.
An Example: Guess the Number
To illustrate the usefulness of entropy we will play a game. I will pick a number uniformly at random from the unit interval .
You can then make a guess , and I will tell you whether or . After I tell you, you have narrowed down the location of the number to if or if .
If the aim is to minimize the expected length of the interval after a guess, we should guess . This is very intuitive, but prove this we note that the expected length is
Since was chosen uniformly at random, and . So the expected length is
Note that and so , and . Therefore any extrema are minima, and since is the unique solution to , we know that guessing minimizes the expected length of the interval.
Now if is the score I give your guess, a random variable taking values so that , then we have already seen that to maximize the entropy of we should have each outcome be equally likely and so take . So the idea is when performing an experiment, or asking a question, the better you are able to predict the answer, the less you learn, and vice versa. So we see that maximizing our entropy is the best strategy as we've proved using calculus.
What about if we are allowed to make guesses ? If we choose to maximize our entropy with each guess, guessing halfway along each interval, then we will have narrowed down to an interval of width , but is this still the best strategy? Intuitively it does still seem like the best plan, but it isn't quite as obvious as in the case of 1 guess.
Any deterministic strategy (that doesn't repeat guesses!) can be described as narrowing down to intervals of lengths summing to , and the probability of the strategy ending with an interval of length is . The expected length of an interval is then .
This suggests a geometric interpretation, what we are looking for is the point on the standard simplex closest to the origin. Now if you read about projecting onto the standard simplex, you will see that we can immediately conclude that we should take .
If you don't believe that, what you might believe using your geometric intuition is that we are looking for a sphere around the origin with radius so that the sphere is tangent to the simplex, touching it at a single point. In the 1 guess case, this looks like this:
the sphere in this case is just a circle, shown in blue. The constraint set is shown in red, the set of points with nonnegative coordinates summing to one. You can see that if you start with a small circle and expand until it hits the red line segment, the circle will hit it at the point .
So we are minimizing a function, the radius, subject to a constraint that the function be tangent to the set of points satisfying . This is precisely the situation encountered in the method of Lagrange multipliers, if you want to understand this idea I recommend checking out this blog post, Lagrangians for the Amnesiac.
In standard notation for this technique, let and . So we want to minimize subject to . The method implies that we then looking for a so that
which is precisely what the geometry tells us – that needs to be tangent to the standard simplex.
Solving this gives
for each ,
and so we must have
for each ,
implying that is the same for each , which forces .
So yet again the best policy is to maximize the entropy at each turn! I invite you to think about stochastic strategies.
Next Time
In the future we might look at how entropy is related to compression, or how the KL-divergence can be used to induce sparsity in an autoencoder.
Neural Networks Part 2: Python Implementation
Ok so last time we introduced the feedforward neural network. We discussed how input gets fed forward to become output, and the backpropagation algorithm for learning the weights of the edges.
Today we will begin by showing how the model can be expressed using matrix notation, under the assumption that the neural network is fully connected, that is each neuron is connected to all the neurons in the next layer.
Once this is done we will give a Python implementation and test it out.
Matrix Notation For Neural Networks
Most of this I learned from here.
In what follows, vectors are always thought of as columns, and so the transpose a row.
So first off we have , our input vector, and our output vector.
Our neural network will have layers with the input layer, and the output layer.
For each layer :
- write for the number of neurons in that layer (not including the bias neurons);
- write for the layer's activation function;
- write for vector of derivatives of the activation functions;
- write for the vector of inputs to the layer;
- write for the vector of biases;
- write for the vector of outputs from the layer;
- write ;
- write for the real matrix containing the weights of the edges going from (not including bias neurons), so that .
Feeding forward in vector notation
So to feedforward the input we have
And we call this output .
Backpropagation in vector notation
Now write for the error at the layer defined .
Then , and , where the multiplication is component wise.
Write for the matrix with entries , for non-bias weights.
Then we have , the outer product of and .
And for the biases write for the vector with component .
Then .
Python Implementation
We will create a class NeuralNetwork and perform our calculations using matrices in numpy.
import numpy as np class NeuralNetwork(object): def __init__(self, X, y, parameters): #Input data self.X=X #Output data self.y=y #Expect parameters to be a tuple of the form: # ((n_input,0,0), (n_hidden_layer_1, f_1, f_1'), ..., # (n_hidden_layer_k, f_k, f_k'), (n_output, f_o, f_o')) self.n_layers = len(parameters) #Counts number of neurons without bias neurons in each layer. self.sizes = [layer[0] for layer in parameters] #Activation functions for each layer. self.fs =[layer[1] for layer in parameters] #Derivatives of activation functions for each layer. self.fprimes = [layer[2] for layer in parameters] self.build_network() def build_network(self): #List of weight matrices taking the output of one layer to the input of the next. self.weights=[] #Bias vector for each layer. self.biases=[] #Input vector for each layer. self.inputs=[] #Output vector for each layer. self.outputs=[] #Vector of errors at each layer. self.errors=[] #We initialise the weights randomly, and fill the other vectors with 1s. for layer in range(self.n_layers-1): n = self.sizes[layer] m = self.sizes[layer+1] self.weights.append(np.random.normal(0,1, (m,n))) self.biases.append(np.random.normal(0,1,(m,1))) self.inputs.append(np.zeros((n,1))) self.outputs.append(np.zeros((n,1))) self.errors.append(np.zeros((n,1))) #There are only n-1 weight matrices, so we do the last case separately. n = self.sizes[-1] self.inputs.append(np.zeros((n,1))) self.outputs.append(np.zeros((n,1))) self.errors.append(np.zeros((n,1))) def feedforward(self, x): #Propagates the input from the input layer to the output layer. k=len(x) x.shape=(k,1) self.inputs[0]=x self.outputs[0]=x for i in range(1,self.n_layers): self.inputs[i]=self.weights[i-1].dot(self.outputs[i-1])+self.biases[i-1] self.outputs[i]=self.fs[i](self.inputs[i]) return self.outputs[-1] def update_weights(self,x,y): #Update the weight matrices for each layer based on a single input x and target y. output = self.feedforward(x) self.errors[-1]=self.fprimes[-1](self.outputs[-1])*(output-y) n=self.n_layers-2 for i in xrange(n,0,-1): self.errors[i] = self.fprimes[i](self.inputs[i])*self.weights[i].T.dot(self.errors[i+1]) self.weights[i] = self.weights[i]-self.learning_rate*np.outer(self.errors[i+1],self.outputs[i]) self.biases[i] = self.biases[i] - self.learning_rate*self.errors[i+1] self.weights[0] = self.weights[0]-self.learning_rate*np.outer(self.errors[1],self.outputs[0]) self.biases[0] = self.biases[0] - self.learning_rate*self.errors[1] def train(self,n_iter, learning_rate=1): #Updates the weights after comparing each input in X with y #repeats this process n_iter times. self.learning_rate=learning_rate n=self.X.shape[0] for repeat in range(n_iter): #We shuffle the order in which we go through the inputs on each iter. index=list(range(n)) np.random.shuffle(index) for row in index: x=self.X[row] y=self.y[row] self.update_weights(x,y) def predict_x(self, x): return self.feedforward(x) def predict(self, X): n = len(X) m = self.sizes[-1] ret = np.ones((n,m)) for i in range(len(X)): ret[i,:] = self.feedforward(X[i]) return ret
And we're done! Now to test it we generate some synthetic data and supply the network with some activation functions.
def logistic(x): return 1.0/(1+np.exp(-x)) def logistic_prime(x): ex=np.exp(-x) return ex/(1+ex)**2 def identity(x): return x def identity_prime(x): return 1
First we will try to get it to approximate a sine curve.
#expit is a fast way to compute logistic using precomputed exp. from scipy.special import expit def test_regression(plots=False): #First create the data. n=200 X=np.linspace(0,3*np.pi,num=n) X.shape=(n,1) y=np.sin(X) #We make a neural net with 2 hidden layers, 20 neurons in each, using logistic activation #functions. param=((1,0,0),(20, expit, logistic_prime),(20, expit, logistic_prime),(1,identity, identity_prime)) #Set learning rate. rates=[0.05] predictions=[] for rate in rates: N=NeuralNetwork(X,y,param) N.train(4000, learning_rate=rate) predictions.append([rate,N.predict(X)]) import matplotlib.pyplot as plt fig, ax=plt.subplots(1,1) if plots: ax.plot(X,y, label='Sine', linewidth=2, color='black') for data in predictions: ax.plot(X,data[1],label="Learning Rate: "+str(data[0])) ax.legend() test_regression(True)
When I ran this it produced the following:
Next we will try a classification problem with a nonlinear decision boundary, how about being above and below the sine curve?
def test_classification(plots=False): #Number samples n=700 n_iter=1500 learning_rate=0.05 #Samples for true decision boundary plot L=np.linspace(0,3*np.pi,num=n) l = np.sin(L) #Data inputs, training X = np.random.uniform(0, 3*np.pi, size=(n,2)) X[:,1] *= 1/np.pi X[:,1]-= 1 #Data inputs, testing T = np.random.uniform(0, 3*np.pi, size=(n,2)) T[:,1] *= 1/np.pi T[:,1] -= 1 #Data outputs y = np.sin(X[:,0]) <= X[:,1] #Fitting param=((2,0,0),(30, expit, logistic_prime),(30, expit, logistic_prime),(1,expit, logistic_prime)) N=NeuralNetwork(X,y, param) #Training N.train(n_iter, learning_rate) predictions_training=N.predict(X) predictions_training= predictions_training <0.5 predictions_training= predictions_training[:,0] #Testing predictions_testing=N.predict(T) predictions_testing= predictions_testing <0.5 predictions_testing= predictions_testing[:,0] #Plotting import matplotlib.pyplot as plt fig, ax=plt.subplots(2,1) #Training plot #We plot the predictions of the neural net blue for class 0, red for 1. ax[0].scatter(X[predictions_training,0], X[predictions_training,1], color='blue') not_index = np.logical_not(predictions_training) ax[0].scatter(X[not_index,0], X[not_index,1], color='red') ax[0].set_xlim(0, 3*np.pi) ax[0].set_ylim(-1,1) #True decision boundary ax[0].plot(L,l, color='black') #Shade the areas according to how to they should be classified. ax[0].fill_between(L, l,y2=-1, alpha=0.5) ax[0].fill_between(L, l, y2=1, alpha=0.5, color='red') #Testing plot ax[1].scatter(T[predictions_testing,0], T[predictions_testing,1], color='blue') not_index = np.logical_not(predictions_testing) ax[1].scatter(T[not_index,0], T[not_index,1], color='red') ax[1].set_xlim(0, 3*np.pi) ax[1].set_ylim(-1,1) ax[1].plot(L,l, color='black') ax[1].fill_between(L, l,y2=-1, alpha=0.5) ax[1].fill_between(L, l, y2=1, alpha=0.5, color='red') test_classification()
When I ran this it looked like this.
The top plot shows the performance on the training data, and the bottom the performance of on the testing data. The points are colored according to the net's predictions, whereas the areas are shaded according to the true classifications.
As you can see neural networks are capable of giving very good models, but the number of iterations and hidden nodes may be large. Less iterations are possible using good configurations of the network i terms of sizes and numbers of hidden layers, and choosing the learning rate well, but it is difficult to know how to choose these well.
It ended up being quicker to choose a larger number of hidden nodes, a large number of iterations, and a small learning rate, than to experiment finding a good choice. In the future I may write about how to use momentum to speed up training.
The code for this post is available here, please comment if anything is unclear or incorrect!


text:
Every other Monday from 1:00 p.m.-4:00 p.m. The next game is Monday, August 10. All player levels are welcome.
Every Thursday from 10:00 a.m. to 12:00 p.m. at TBY. Contact Naomi Glass at email@example.com for more information.
3rd Thursday of each month from 7:00 p.m. to 10:00 p.m. in the Youth Lounge.
On a private court in Corona Del Mar, please call Winnie at
(949) 644-5178 if you would like to play.

subset: commoncrawl


text:
While My Guitar Gently Weeps - Jake Shimabukuro
Probably one of the most viewed uke videos anywhere on the internet, and with good reason. Jake re-works The Beatles and makes us all take the ukulele very seriously indeed.
Theme From 'Shaft' - The Ukulele Orchestra Of Great Britain
Not only is it highly amusing hearing a northerner pretend to be Isaac Hayes, but the entire band play beautifully, and the arrangement straddles the line between hilarity and genius.
Between The Devil And The Deep Blue Sea - George Harrison
George Harrison re-works an American classic with aplomb on his final album. Check out the young Jools Holland tinkling the ivories in this one...
George Formby - When I'm Cleaning Windows
The original ukulele superstar. Witty, charming and unique, there's no one quite like George Formby. This song exists somewhere in the DNA of every Englishman.
Super Mario Theme - James Hill
For both the originality of the choice of cover and the fluidity of the playing, James Hill makes the cut for our ukulele megamix.
Somebody That I Used To Know - Sungha Jung
Whatever you think of the song, there's no denying that Sungha Jung is an exceptional talent. Awe-inspiring.
The Scarlet Tide - Elvis Costello
With nothing but a uke for company, Elvis blows away the audience with a phenomenal performance of a beautiful song (taken from the Cold Mountain soundtrack, fact fans).
Led Zeppelin meets Beethoven - Taimane Gardner
It's the ultimate classical rock crossover, and Taimane works the stage like the uke hero she surely is.
California Worldfest - Brittni Paiva and Tommy Emmanuel
Tommy Emmanuel has a great eye for talent and regularly gives young players a break. Check out Brittni Paiva's effortless interplay with bit Tom here.
Something - Paul McCartney, ELO and Eric Clapton
Macca has been performing Something on ukulele as a tribute to George Harrison for some time now, but here is surely the ultimate version, as ELO and Clapton join in half way through.
Kris Fuchigami - Can't Take My Eyes Of You
A flawless performance of a classic, Kris Fuchigami has got incredible feel and a nice hat, which is really all anyone needs.
Flight Of The Conchords - Mermaids
Sometimes funny is enough. Although it helps if you can write great tunes, too...
Want to play like some of these guys? Don't forget that Play Guitar Now: Ukulele is available now from WHSmiths and all good newsagents, or you can order a copy online. Now get strumming!


text:
What's an #OilDork?
Well, once you're using doTERRA oils, and once you realize their full potential for good in people's lives you start digging and learning and carrying these little bottles everywhere, dripping these hopeful liquids on anyone who will allow it. You read and research and find that there's always more to fall in love with. And then one day you realize you're pretty dang nerdy about all of this natural wellness stuff. You're an #oildork!
And you embrace that because in our team you'll be surrounded by some of the BEST, smartest, kindest, hippest, coolest dorks you'll ever want to know. People who will embrace you and support you and be just as excited about our mission as anyone ever could be.
Our doTERRA team is a family of #oildorks all proud to be just a smidge on the obsessed side of natural wellness and holistic living. Sure, it doesn't happen overnight for everyone, but by now you'll certainly admit intrigue, yes?
Thinking offering hope by the drop sounds like a pretty amazing way to make a living? Unbelievable even? It is. It's incredible, keep reading to see what that can look like. It'd be an honor to have you join us.
Work with us! a brilliant way to earn.
Through our passion for sharing these health-promoting, good-from-the-earth & good-for-the-earth products, we have grown a wonderful business that supports our growing family and farm. What started as empowering our family with simple tools that were accessible at all hours of the day, we now also offer others financial healing as well.
It began with one class–I just had to share these simple tools! That turned into 1 class a month, then 1 a week. We first paid for our oils, then payed for a few goats and fencing, and then we stepped it up. Through diligence, hard work and consistent commitment we have replaced and grown our income in ways we never thought possible! As our financial limitations slip away, our dreams grow. As our freedom grows, our passion for helping others achieve the same does also.
If you're interested in starting a business that involves helping others on a path toward natural healing and empowerment–JOIN US! We are looking for folks who are interested in this opportunity, who are also eager to teach others and enable them with these gifts of Creation.
We offer several bundles to help Entrepreneurs get started, along with perks, support and training opportunities that will help you blossom as we have in this amazing industry. CONTACT US to get a list of specials and more information on the business.
We also offer continued support through our online forums, monthly live team calls, strategy sessions, weekly continuing education and business trainings, email campaigns and much more.


text:
Cooper's Gifts Presents Their Annual Girls Night Out!
Attention ladies: Have you been longing for a night out? Do you need some "me time"? Do you feel that you need to reconnect with some of your girl friends? Are you looking to make some new friends or enjoy social interaction? Are you looking for something fun to do while still being able to help out others? If you've answered yes to any of these questions then you are in serious need of a "Girls Night Out"! What better way to celebrate, then to attend "Cooper's Girls Night...Read More »


text:
We know the times are rough so we have taken care and cut back on our pleas for help.
But that doesn't mean our need for help has lessened.
We have had some great animals some of which have required extensive veterinarian care.
You can read more about Alaska, Jean, Starlette, Almond Joy, Carilynn, and there are more.
If you would like some suggestions for items that we can put to good use…
.. here is the list!
Dog toys and rawhides
Purina One dog food
Purina One cat food
Dog treats – training treats, biscuits, any size or type of treat
Blankets – polyester/cotton, clean
Beds: dog or cat: small and medium are best
Monies to help with vet bills
Participate in the Gift Program ($50)
Gas cards to help volunteers with all the driving they do to get animals to
and from vets/adoption events/etc
Donation to the Daylight – prison program.
Small, soft treats for prison program
any cat furniture
Scratching posts
Office Supplies
Cleaning supplies: paper towels, Bleach, Vinegar, more paper towels, laundry soap
Bigger items:
Micro scope
Computer able to handle large memory programs
Thank You!


text:
2017 Average Cost To Finish A Basement Remodel, Bathroom, Framing
systems out there to make basement conversions easier. These ease of installation, after all, comes at a worth.
With nearly 30 years of expertise within the trade, Fitch Total Basement Finishing knows the best practices when it comes to basement ending for householders in Rochester and all through all the encircling areas. We are committed to excellence on every job and we work arduous to make sure our prospects are satisfied. If you are trying to acquire priceless house in your home, schedule a free basement finishing estimate with Fitch Total Basement Finishing today! Suspended ceiling are the norm. A suspended ceiling is essentially the most expedient approach of including a ceiling to your basement while preserving entry to providers located close to the joists (wiring, and so forth.).
Let us provide you a free, detailed basement ending and transforming estimate and consultation. We share goals with owners of Rochester …


text:
The Council is comprised of leading luxury homebuilders from none competing markets across the country. NCBC members combine a passion for the custom homebuilding business with many years of operational leadership experience. Our members have banded together since 1998, offering an exceptional educational opportunity to improve our best business practices, fiscal performance and share information on cutting edge technology and construction methods.
The Council Members have demonstrated the highest level of integrity, craftsmanship and service to our clients and our industry. Our members build extraordinary homes in most all the leading markets across America. While our homes represent a wide variety of architectural styles and geographic diversity, they have in common an uncommon level of craftsmanship and quality.
Our use of technology, process development and relationship building is setting a new standard of excellence for the industry, and improving the ways our members run their businesses. Our forums improve the collective skills, wisdom and network connections of the best in the custom building industry, and through our semi-annual council conferences and ongoing dialogue with industry leaders, our members are enabled to reach their highest level of performance, and bring additional value to our clients.
The National Custom Builders Council offers its non-competing members educational opportunities to share experiences, seek advice and counsel, and add to or expand business processes that enhance management skills, improve operations, and bring greater profitability to each member.
Don't Forget to Visit Our Website:


text:
Little Box of Sweets
Set in a country coming of age, Little Box of Sweets is a simple love story about a young girl's journey to fight for her dreams and the love of the man who inspired her.
A little village in North India is Asha's world. Whatever she knows and wants is right here in her humble existence with her grandparents and her soul mate Lalli.
Then Seth comes along, the handsome son of an Indian Commissioner and Sheila, the beautiful daughter of a British missionary.
Seth and Asha's common love for poetry draws them closer - awakening Asha to a dream to be more like him and strive for a better future. But it comes at a price.
- Year
- 2007
- Type of film
- Features
- Running Time
- 98 mins
- Format
- S16mm
- Director
- Meneka Das 1st Feature
- Producer
- Vijay Das
- Co-Producer
- Vijay Das
- Editor
- Huw Jenkens, David Charap
- Screenwriter
- Meneka Das
- Director of Photography
- Dusan Todorovic
- Production Designer
- Sharad Singh
- Sound
- Ben Norrington, Sven Taites
- Music
- Andrew T MacKay
- Principal Cast
- Joe Anderson, Meneka Das, Sheenu Das
Production status
Please let us know if we need to make any amendments to this Film entry mailing us directly at
Last updated 16th November 2006


text:
96 hours of weaving time, since January done, I have completed weaving the Sunrise Christmas Morning from Schweitzer Mountain tapestry. Plus, Agent Carter is shaping up into a fun tapestry!
Allied Arts Month of May Show: Book Arts of the Tri-Cities
Cheese Louise: Urban Sketchers Tri-Cities
Art Before and After Work! lol
Patti Kirch


text:
19-04-2016 #1
Just a heads-up on some ballscrews which are so good I can't help but share the link...
Take a look here on ebay...
Brand new THK precision-class ground, 32 diameter 10 mm pitch, long pre-loaded nut... 300 mm usable travel or a smidge over. The details don't seem to be in the current THK catalogue but going by the model code on the packaging, they seem to be a slightly 'custom' version of the EIK model listed in this catalogue in German (page 108):
The end machining is a bit short for standard BK25 blocks but I think by removing one seal (and making other arrangements to keep crap out) it'll work...
I ordered one and when it came ordered two more. They come in the original THK boxes with an inspection certificate. Lovely smooth, tight action when turning by hand. For £71.50 inc postage each it's a steal...
I have no connection with the seller except as a happy customer. He says he has plenty of them - they were spares for a now discontinued motion platform (he has two of those too if anyone wants one - hexapod mill maybe?):
19-04-2016 #2
Whats the preload class on these?https://emvioeng.com
Machine tools and 3D printing supplies. Expanding constantly.
19-04-2016 #3
Not entirely sure. The feel is of a sort of light/medium preload I'd say - tight but not notchy at all. The PDF I linked to above says (in the table on p. 108) that the preload (Vorspannung) is 1.3 kN - but I'm not sure how that is measured. P. 49 says the preload is achieved through pitch offset of the different races within the single nut.
Last edited by bikepete; 19-04-2016 at 02:23 PM.
19-04-2016 #4
In case it helps the entire model number code is:
and the one I have in front of me has serial number KX04K05797 - that is also the only thing etched onto the ballnut itself.
Mmm, just found more in another THK catalogue (p. 15-248):
S is presumably 'Special'
6 is six ball circuits
RR is labyrinth seals both ends
G0 is "zero or less" axial clearance i.e preload
535.5L is threaded length
C5 is accuracy class
Last edited by bikepete; 19-04-2016 at 02:46 PM.
19-04-2016 #5
Very nice indeed! I keep having to remind myself to stop drooling over things like this for a while . After all I am having a clearout.https://emvioeng.com
Machine tools and 3D printing supplies. Expanding constantly.
19-04-2016 #6
Thread Information
Users Browsing this Thread
There are currently 1 users browsing this thread. (0 members and 1 guests)
Similar Threads
FOR SALE: Pssst. Compressor - grab a bargain!By Wal in forum Items For SaleReplies: 2Last Post: 15-03-2014, 12:32 AM
Bargain for someone on eBayBy irving2008 in forum General DiscussionReplies: 2Last Post: 17-07-2010, 09:48 PM
Is this a bargain?By irving2008 in forum General DiscussionReplies: 13Last Post: 25-05-2010, 05:53 PM
Bought myself a lathe.....bargain!!By HiltonSteve in forum Milling Machines, Builds & ConversionsReplies: 7Last Post: 28-12-2009, 05:09 PM
Hexapod Robot CNC routerBy Lee Roberts in forum General DiscussionReplies: 2Last Post: 08-01-2009, 10:54 PM


text:
Soon-to-be bride Bethan Jones is having her beloved long hair chopped off the day after her wedding to raise money for a cancer charity.
The 25-year-old student nurse, who lives in Melton with fiancee James Cooke (29), is giving the sponsorship proceeds from the effort and her shorn locks to theLittle Princess Trust, which provides wigs for children who lose their hair following chemotherapy treatment.
Bethan said: "I'm having about 30cm cut off the day after my wedding in October.
"I love my hair but there are children who could really benefit from having it."
Go to www.justgiving.com/fundraising/B-Jones9 to sponsor Bethan, who has so far raised £75 of her £200 target.


text:
The Ely papers are not one but a host of scattered deposits in Public Records Office of Northern Ireland, amounting in total to about 1,550 documents, covering the period 1630-1928 and documenting the estates, mainly in Cos Fermanagh, Wexford and Dublin, the business affairs and the frequent accessions of peerage honours of the Loftus family, from 1751 Barons and Viscounts Loftus and Earls and Marquesses of Ely.
Tottenham Family Links
The Loftus family has, for a number of years, had a prescence on the web. The linked page contains their family tree.


text:
- This event has passed.
Running with Ed Relay
May 18
The 10th annual Running with Ed (RWE) relay is May 18th, 2019. This 38-mile, 10 leg race traverses spectacular Park City, and gets you home in time to have a playdate or a date night. The course offers everything from trail to pavement, challenging climbs to fast downhills, easy legs for beginners and tough ones for experts. Winding past Olympic venues, iconic landmarks, neighborhood schools, and historic Main Street, this is Park City's best!


text: On 3-Jun-08, at 1:25 PM, jo at frot.org wrote: > This idea of selling the "OSGeo Teach-In" as a "brand" to sponsors > keeps the decision clean. There's already a range of sponsorship > benefits listed at http://www.osgeo.org/sponsorship ... a Supporting > Sponsor could have the right to host events labelled "OSGeo Teach- > in", > and there would end our involvement. Jo, I don't see how becoming a sponsor can help in these cases. It's come up a couple times but I think there's a misunderstanding of sponsor benefits here. The rights of sponsors don't really include using the OSGeo brand as part of their own events. It merely lets them identify themselves as sponsors. Are you thinking of specific wording on that page that suggests more? I'm thinking of the "Right to use OSGeo Sponsor logo and promote named sponsorship level" Tyler


text:
Think twice before your feet leave the ground. The paragliding launch is vital to having a safe flight. It may sound confusing at first – after all, how does the process on ground affect your in-flight experience – but don't be fooled. The most important steps happen before your feet even leave the ground. So, minimize your chances of having an accident by following proper paragliding launching techniques you should have perfect within your paragliding lessons.
Envision yourself as a pilot of an aircraft. Often, new paragliders pilots hop into their launch, and this is poor technique. Your paragliding launch should be similar to an airplane taking off. There's no hopping, bouncing or skipping. It should be a smooth, gradual lift off. Doing the opposite can cause your paraglider to be ahead or behind you, which increases your risk of a launching accident. Your wing always needs to be directly above you for a proper launch. So act like a pilot, not like a bunny because doing the latter will only cause problems.
When you're paragliding, you never want to rush into your launch. You just have to move into the rising airflow. However, to pull off this proper paragliding launch technique, your glider has to be directly overhead. If it's behind you or even a little ahead, the lift-off will be bumpy and dangerous with the possibility of no lift at all. Ground handling is a critical part that you learn to perfect throughout your paragliding lessons.
With the prior mentioned, proper paragliding launching technique requires you to take off directly into the wind. Remember, your launch should be like an airplane taking off; flowing down the runway at a smooth, gradual increase of the speed. Not being in full control and, for example: being pulled from your launch, can result in higher stress level, which could be detrimental to your entire paragliding experience. So, go with the flow in full control of the process; never let it pull.
A major component of proper paragliding launch technique is to have your glider fully loaded. Anything short of that can make your entire take off quite unpredictable, as it can cause deflation close to the ground. A loaded wing is also faster and reacts better to the input you give with the brakes, so it's easier to control. Learning to optimize using your weight in the harness while running will instantly minimize your risk of paragliding launching accidents.
Now, if any of these come to you as a surprise, it's time to sign up or re-do a paragliding lesson. These launching techniques should be infused into your brain before you even consider taking a flight.
Whether it's your first day training to become a paraglider, or your first day taking flight, it's undeniably an exciting moment! And with excitement tends to come some nerves, so let us take care of that for you. Knowing what to expect on the first day can be all you need to feel confident as you head towards this new adventure that is going to let you see the world in a whole new way.
We can tell you what not to expect on the first day of paragliding training, and that is to not expect to jump in a harness and take off. There is a lot of training required before you are able to take flight, and even more before you're able to paraglide independently.
The good news is that the training portion of paragliding is fun! It's not like sitting in class, studying about stuff you'll probably never use in the real world. Instead, you're learning the essential skills and steps needed to take this new hobby of yours to all new heights.
Whether you're starting with your very first Paragliding Introductory Lesson or are heading onto a new set of training, the only things you need to bring include:
Expect to feel an adrenaline rush like you never have before. The first day flying a paraglide will be with a supervisor. It takes several lessons and extensive training before you can fly independently. So you will be strapped into a harness alongside your instructor.
Your paraglider will first inflate first on the ground before taking flight. You won't be leaping off the side of a cliff or anything of that sort. Launch is a meticulous task, so although you'll be extremely anxious to get going, patience is required.
All flying equipment is provided, but you will have to register at the flying site, which may come with additional (minimal) fees. Other than that, all you need is the following:
And of course, definitely don't forget to attach your Go-Pro!
You can check the Let's Go Paragliding videos collection, but this video gives a good overview of what we offer. Everyone dreams of being able to fly at least once in their lifetime, but very few realize that you can fly. You just need to take some paragliding lessons. However, if this adrenaline-rushing sport has you feeling a little intimidated, and you're not sure what to expect, let this paragliding video give you a good look at some of the class activities.
The first paragliding classes start on the training hill. However, it doesn't take long before you're able to get your feet off the ground and make launch. Of course, you won't be flying to all kinds of altitudes, but you will get a feeling for paragliding and the entire launching process. This paragliding video also shows you some excellent techniques of doing just that.
There's always the option of being towed within your paragliding lessons. These are typically incorporated when you've perfected your launch and understand some paragliding techniques, but don't have the experienced needed to experience higher mountain flying. Being towed is an exciting milestone within your paragliding training because you get an authentic feeling of flying. I allow you to make more flights in a day, creating a routine, and setting up good reflexes. You're up in the sky, controlling your wing all on your own (under radio supervision).This paragliding video shows you how fun this portion of your paragliding training is.
Now, we know you're eager to get off the tow and set your soaring aspirations to all new heights. Everyone is! But you will only be able to do so once you pass several levels of paragliding lessons, and have perfected the necessary skills to paragliding. However, watching this video can be the ultimate way to keep you motivated throughout your training, while also reminding you of the end goal. Plus, the scenery is astonishing. You just can't go wrong.
As you increase your paragliding experience, you will quickly see that every level of paragliding is fun. You learn new things every day, and it just gets better as you hike up the ladder to achieving your desired pilot license.
Are you ready? Head on over to Let's Go Paragliding and sign up for some lessons! You don't have just to sit at home and watch other people paragliding and the beautiful sights they see. You can achieve it too!
As an avid paragliding enthusiast, you probably love checking out other people's adventures. Even if you have yet to soar the sky in a paraglide, you just can't deny how awesome it must be to literally fly. Paragliding videos are an excellent way to take a look at this adrenaline-rushing sport, and there's no denying that you'll want to sign up for our paragliding lessons after checking the following videos out. Here are our top picks for epic paragliding videos.
This paragliding video is from our own collection, and it's definitely a must-see. The sun began to set during flight, and it's truly a stunning view that very few people get to see. The landscape is just incredible.
We really like this video from EasyGliderful because it shows a unique perspective to paragliding. Instead of viewing the flight as if you were in the flying-seat, you get to see what it looks like if you were up there in the sky, paragliding with someone else. This is a really cool video and will certainly entice your paragliding aspirations.
This paragliding video from Luu is an excellent one to check out if you want to see the entire paragliding process. The video shows the paraglider's hiking to their spot, taking flight and soaring the skies.
This is another clip from the Let's Go Paragliding video collection, and it's stunning. Take a look and you'll see both photos and video from a soaring height that is just memorizing to watch.
This paragliding video from GoPro is really well put together and it's absolutely mesmerizing. The paragliders take a hot air balloon before leaping off. However, it's important to note that this isn't standard paragliding procedures. Usually, the wing is completely inflated before taking flight, but it looks like these paragliders got a little innovative.
You're definitely ready to take some paragliding lessons now, aren't you? Let's Go Paragliding is a highly reputable school that can give you all the high quality training you need to become a successful and safe paraglider. The trainer was even voted the United States Instructor of the Year in 2010. So, you can train easy knowing that you're learning from one of the best in the industry!
We also have plenty more paragliding videos available on our Let's Go Paragliding Youtube account. It's a channel that's definitely worth subscribing to! Here's to many more flights (or even your first flight) in your near future!
The season started with great vibes. We already have novice pilots making mountain flights and soars with our local friends, the vultures.
That day from the video we managed to hike twice to the top of Mount Brace and do two beautiful flights.
When all the elements are present—the sun is here but not too hot, the wind is blowing in the right direction at the right intensity, a group of friends is around to share the hike and flight—those kinds of days are just worth all the effort and investment we've made to get there.
Of course becoming a paragliding pilot and being able to enjoy that kind of day requires a bit of time and effort as in any sport. You don't go on double black–diamond ski runs on your first day. That wouldn't be a smart move.
Paragliding is similar; it takes some time on a training hill with an instructor to master basic skills to control the paraglider in a different kind of air (turbulent or laminar), and velocity (from 0 to 15+ mph).
When your launching technique is under control, landing into the wind softly is another skill to master.
When your ground skills, take off, and landing are controlled, then comes the big picture—the big playground. It's similar to learning scuba diving in a swimming pool and then being ready to jump out into the big ocean.
At Mount Brace, even after your Novice certification, you will fly under the supervision of an "Observer" (a more advanced pilot) who will act like a big brother, helping you make the right choices depending on the weather conditions.
It also takes time to see the big picture, understand the wind, the turbulences, reading the clouds, and riding thermals. So having somebody more experienced to guide you through the process is a must for a long and safe pilot life.
But once you get there, then the kind of days that look like the one in the video are rewards that make all the previous efforts worth it when you take into account the experience you get in return.
Let's Get You Flying at the Paragliding Introduction Day
You finally decided to take that leap of adrenaline and fly across the skies into a new hobby that will provide you with a lifetime of memories. Congratulations! We are just as excited to have you choose our Paragliding School as your ultimate guide to teaching you how to conquer the mind-blowing activity of paragliding. We know that your heart is sailing with adventure and ready to take flight, so let's quit the chit chat and get to the good stuff of paragliding!
What is Paragliding?
You've likely seen it before in the sky – it's a bird, it's a plane, no, it's a paraglider! You may even know the basics of paragliding, but it is important to know exactly what this astonishing activity is all about before heading off to your Introduction Day. This is an outdoor sport that allows you to explore the world in a way that only birds get to. It's a pure rush and extremely safe, with the proper training which of course, you will be receiving on Introduction Day and throughout all the other classes of your choice. Paragliding is similar to flying a kite, only you're sitting below the kite instead of steering it from the ground. The wind currents act somewhat like the 'engine' of the glider and you basically soar through the sky, enjoying the sights.
Your First Paragliding Class
We offer a huge range of different training programs to ensure that your desires are met fully – and even more importantly, to keep your paragliding experience furthering. However, before you advance on to flying on your own or becoming an expert under the sail, you certainly need a bit of an introduction to this adrenaline-pumping action. This is where you will get all the basics that you need, including equipment. The slow and steady approach within the training will provide you with a thorough look into what paragliding is all about, and if you decide that it just isn't for you, this is the perfect time and place to discover that. However, we are absolutely convinced that you will be ecstatic with the full day of outdoor fun, and can't wait to provide you with the thrill of flying. During the Introduction Day training class, you will also learn the basic skills to kitting your glider, how the air currents is the secret to flying, and you will even make your very first flight… from the training hill, of course.
Prepare For Your First Paragliding Class
The first thing you need to do to prepare for your first paragraph class is to get exciting! This is an experience of a lifetime, and you certainly want to enjoy it to the fullest. So, get a well night's rest and a delicious and nutritious breakfast in before packing up your bag. Some things you definitely don't want to forget at home include your; hiking boots, light gloves, water and snacks. Depending on the weather, you may also want to add in your sunglasses, a long sleeve shirt and some long pants just to ensure your comfort. Other than that, you're ready to go. Let's Go Paragliding will provide you with all necessary equipment such as the glider itself, harness, helmet and radios. You just have to bring yourself and your excitement – and a couple of other things, as mentioned.


text:
Online Community
FrederickCounty.com welcomes matthewcluver, the newest member to our growing online community!
Haven't joined FrederickCounty.com yet?
Get Involved
Add Your BusinessSubmit Content
events, articles, etc...
Latest Activity
Business Listings: Gold, Silver & Platinum
Gold, Silver & Platinum Dealers
Did you know?
Your Listing For Gold, Silver & Platinum Could Be Here?
(Spots Available First Come)
Click for Details
Bonanza Coins
Julian M. Leidman - Maryland Rare Coin Dealer
940 Wayne Avenue
Silver Spring, MD, 20910View Map - Get Directions
Free Listings:
Newtown Antiques and Pawn $ Coupon!
We Buy Gold & Silver In Any Condition
375 Fairfax Pike
Stephens City, VA, 22655View Map - Get Directions
Boscawen Gold & Silver Exch
41 W Boscawen St
Winchester, Virginia , 22601
(540) 667-6065
View Map - Get Directions
Submit a FREE business listing to this category
If you appreciate the content published here please "Like" us below: