Simplify Calculus for Machine Learning with SymPy
Photo by Kevin Ku on Unsplash

Simplify Calculus for Machine Learning with SymPy

A quick look at calculus for Machine learning and how to add it to your code with SymPy

Machine learning requires some calculus. Many of the online Machine learning courses don’t always cover the basics of calculus assuming the user already has a foundation. If you are anything like me, you might need a bit of a refresher. Let’s take a look at a few basic calculus concepts and how to write them in your code using SymPy.

Most of the time, we need calculus to find the derivatives in optimization problems. This helps us to decide whether to increase or decrease weights. Our end goal is to find an extreme point(s) that will be the local minimum or maximum point(s) in a function.

Let’s go through the process of finding an extreme point by taking the following steps.

Installing and learning the basics of Sympy. Finding the slope of a linear function. Discovering tangent and secant lines. Using our slope and tangent line knowledge to find limits. Understanding what the derivative of a function is. Using the derivative to find the extreme point. Deciding whether the extreme point is a local minimum or a maximum point.

Getting Started With SymPy

SymPy is a Python library that lets you use symbols to compute various mathematic equations. It includes functions to calculate calculus equations. It also includes many other functions for some higher-level mathematics.

Installing SymPy is simple you can find full installation instructions here. If you are already using Anaconda, SymPy is included. With Anacona you can make sure your SymPy is up to date with a simple:

conda update sympy

If you aren’t using Anaconda, pip is a great way to install new Python libraries.

pip install sympy

SymPy depends on the mpmath library so you’ll need that installed too.

conda install mpmath

#or

pip install mpmath

With SymPy we can create variables like we would in a math equation. We need to set these variables as symbols so SymPy knows to treat them differently than regular Python variables. This is simple and accomplished using the symbols() function.

import sympy

x2, y = sympy.symbols('x2 y')

Now that we have SymPy installed let’s take a step back and look at the foundations of calculus.

Linear Equations and the Slope

As mentioned above one of the main reasons we need calculus is to find the extreme point(s). To illustrate this, let’s pretend every year you enter the Annual Potato Cannon Contest. Every year you lose to the terrible Danny MacDougal. This year you hire a coach to help you beat Danny. To beat Danny, the coach needs you to give him three things.

1. When Danny’s potato is at it’s highest point.

2. How long it takes the potato to get to the highest point.

3. The slope of the potato at its highest point.

Let’s talk first about the slope.

If Danny built a magic cannon where the potato ascended forever finding the slope would be easy but there wouldn’t be a maximum height. This type of potato flight path is would be a linear equation like y = 3x + 2 (in y-intercept form).

We can visualize this linear function potato flight using NumPy and Matplotlib. If you don’t have NumPy and Matplotlib installed the process is like the SymPy installation above. See here and here for more details.

import matplotlib.pyplot as plt

import numpy as np #create 100 values for x ranging from 0 to 6

x = np.linspace(0,6,100) #our linear function

y = 3*x + 2 #add some aesthetics to out plot

plt.grid(color='b', linestyle='--', linewidth=.5)

plt.plot(x,y, label="potato flight")

plt.xlim(0, 6)

plt.ylim(0,20)

plt.legend(loc='best')

plt.xlabel("Seconds")

plt.ylabel("Feet(x10)") #show the plot we created

plt.show()

We want to examine the slope so we will add to marks on two random points: (1,5) and (4,14).

plt.plot(1, 5, 'x', color='red')

plt.plot(4, 14, 'x', color='red')

potato flying in a linear function

When we have a linear function our slope is constant and we can calculate it by looking at any two points and calculating the change in y divided by the change x. Let’s look at the two points we marked earlier.

equation to find the slope

In y-intercept form (y = mx + b) the m is always the slope. This checks out from our previous function of y = 3x + 2.

Potatoes Must Come Down — Nonlinear Functions

The slope of a linear function is easy but the potato must come down. We need a way to calculate a slope that changes with each point. Let’s start by visualizing a more realistic potato flight path.

Our coach is amazing and he knows the function that represents the flight of the potato is f(x) = -(x²) +4x.

Once again let’s visualize this potato flight path with Matplotlib.

x = np.linspace(0,5,100)

y = -(x**2) + 4*x plt.xlim(0, 4.5)

plt.ylim(0,4.5)

plt.xlabel("Seconds")

plt.ylabel("Height in Feet(x10)")

plt.grid(color='b', linestyle='--', linewidth=.5)

plt.plot(x, y, label="potato")

plt.legend(loc='best') plt.show()

non-linear flight of a potato

From the visualization of the function, we see that in around 2 seconds the potato is at its height of around 40 feet. The graph is helpful but we still need the slope of the maximum height. Plus we need some hard evidence to bring back to the coach. Let’s move forward proving this height and time with calculus.

Secant Lines

A secant of a curve is a line that intersects the curve at a minimum of two distinct points. When we have nonlinear functions we can still find the slope between two points, or a secant line. Since (2,4) looks like the top of our potato path, let’s look at 2 points on our new nonlinear potato path: (1,3) and (2,4).

#adding this code to our above plot

x2 = np.linspace(1,2,100)

y2 = x2 + 2

plt.plot(x2,y2, color='green')

secant line x + 2

We can see that the slope from (1,3) and (2,4) is 1. Let’s take this a step further and see what happens when we try to find the slope of just the point(2,4). To do so we need a line that will be able to represent going through one point.

Tangent Lines

Let’s see what happens to the slope of our lines the closer we get to (2,4). To so we’ll draw a few more secant lines. We’ll keep the endpoint at (2,4) but one line will start at (1.5, 3,75) and the other one at (1.95, 3.9975). This gives us two more secant lines y3 = .5x + 3 and y4 = .05x + 3.9.

two more secant lines

When we finally reach (2,4) as our starting point, a line that forms, which is y = 0x +4. This is the tangent line for (2,4). The tangent line is calculated by solving the limit and plugging it into the y-intercept linear equation. More on the limit in a little bit. Every point in a function has a tangent line, which is how we can calculate the slope for every point in a function.

The tangent line for the point(2,4)

Remember, our main goal is to find an extreme point, the maximum height of the potato. Extreme points will be when the slope of the tangent line is zero as this will signify when a function changes directions. For instance, let’s look at the slope for the tangent line going through (1,3) and one going through (3,3).

3 tangent lines

Let’s look at what happened to the slope of the tangent line for these three points.

point (1,3), the tangent line is y = 2x + 1, slope is 2

point (2,4), the tangent line is y = 0x + 4, slope is 0

point (3,3), the tangent line is y = -2x + 9, slope is -2

After 0 the slope changes directions. Perfect, we need to find the point or points that have a tangent line with a slope of zero. These points will tell us the maximum or minimum points as the direction of the tangent line slopes will always change after 0.

Great! So we need to find a point on a function where the slope is zero. If only there was a to create a function that would give us the slope of any point in our original function. There is, and that is what the derivative is! Before we jump into derivatives let’s look at Limits.

Limits

Limits allow us to find the slope of a function as it approaches a certain x value. If we are solving the limit as a secant line it’s easy. We plug in our two x values and two y values and we can solve the slope equation. This sort of limit defined. But, we want to find an undefined limit because we want the slope of the tangent line to the point (2,4).

As we approach (2,4) we need the slope. We can’t get the slope of the secant line from (2,4) to (2,4) because if we plug those numbers into our slope equation we get 0/0. This won’t work because as a math teacher once said: “ There are two things in life you can’t do, nail jello to a wall and divide by zero”. Since we can’t divide by zero we need to find the undefined limit.

Limits are solved by plugging our function into the slope equation and factoring it out. We can use our slope formula from before and substitute our point(2,4) for the x1/y1 values and then substitute the f(x) and x for the x2/y2 values. Our limit value is the x value we are approaching, which in this case is 2. Our new slope equation for limit 2 of the function -(x²) +4x is :

Our goal is to get rid of the x on the denominator, so let’s expand the -(x²) and cancel out the x on the denominator.

And now that we have removed the x from the denominator we can put plug 2 back into x.

Now that we know how a limit is solved let’s fire up SymPy so we can solve the limit in our code.

SymPy has a function named limit() which has 3 parameters.

the function we’re finding the limit for

the input variable

the number the input variable is approaching

Since our limit is undefined we need to substitute our x and y values as we did above.

import sympy

x2, y = sympy.symbols('x2 y')

#store our substituted function as a y variable

y = (-(x2**2) + 4 * x2 -4) / (x2 -2)

limit = sympy.limit(y, x2, 2)

// 0

Our limit is 0!

Derivatives

The derivative is a function that will give us the slope of a tangent for any point in our function.

Now we understand what the limit and tangent lines are we can head towards our end goal of using derivatives to find the extreme point(s) in the function. The process of finding a function’s derivative is differentiation.

Solving the derivative uses some algebra and our slope formula from above. Because we aren’t solving for a specific point we won’t substitute any values. For this example let’s also replace x1 and x2 with the more common form, which is to use x and x +h. This will give us the following formula to solve for the derivative with f′(x) meaning the derivative of the function of x:

If we plug our function into this we get:

We can then solve this similarly to how we solved for the limit. We can also use the power rule to solve to easily find our derivative using the following equation.

Using that rule we can see the derivative of our function -(x²) +4x is -2x +4.

Instead of going through the steps to find the derivative by hand let’s find the derivative using SymPy.

SymPy gives us a function called diff() which will perform the differentiation process and returns the derivative.

The diff function takes two parameters:

the function we’re finding the derivative for

the input variable

Let’s give this function a try using our original function, -(x²) +4x.

# set x as the variable

x = sympy.symbols('x') #help make the out easier to read

sympy.init_printing(use_unicode=True) #enter the our argumnets to th diff funtion

d= sympy.diff(-(x**2) + 4*x, x) #print our derivative

print(d)

//-2*x + 4

Our extreme points in a function will be where our derivative is equal to zero. This is because when the derivative is equal to 0 the direction of the function has changed, as we explored above.

To find the x value we set our derivative to equal 0 and solve for x, -2x + 4 = 0.

This is solved with SymPy by using the function solveset(). Solvest takes two parameters:

the Eq function which takes two parameters: the equation and the value the equation needs to equal

the variable we are trying to solve

Solvset will return a set for all numbers that solve the equation.

Using solvset to find the x value when the derivative is equal to 0 will look like this:

answer = sympy.solveset(sympy.Eq(d, 0),x)

print(answer)

//{2}

Perfect! Our x value is 2 and if we plug that into the original function we get 4 as our y value.

Now we are certain that at 2 seconds MacDougal’s potato is exactly 40 feet in the air and the slope at that point is 0! We can take that to the coach and we are sure to win the next potato cannon contest!

Is our Extreme Point a Min or a Max?

We know our potato’s extreme point must be a max point because these potato cannons aren’t designed to shoot the potato down. But if we didn’t have a graph or know the direction of the function, how would we see if the extreme point(s) is a local minimum value or a local maximum. We already know that setting the x value of the derivative to 2 results in a slope of 0. What happens if we plug two more numbers into our derivative function, one number will be larger than 2 and the other will be less than 2. We will try this with 1 and 3.

test1 = -2*1 + 4

test2 = -2*3 + 4

print(test1)

//2

print(test2)

//-2

We see that before our extreme point the slope is positive and then after the extreme point our slope is negative. A change in slope from positive to negative tells us the extreme point is a maximum value. If the slope had gone from negative to positive we would know the extreme point is a minimum point. If there are multiple extreme points we would want to choose a value between each of the points. For instance, if we had extreme points 1 and 5, we would repeat this process three times:

choose a random number less than 1

choose a random number greater than 1 but less than 5

choose a random number greater than 5

Most likely nonlinear functions won’t have only one extreme point. In this case, all of the steps are the same but when we solve for the derivative being equal to zero we get multiple solutions.

Conclusion

SymPy vast library and is a great way to find the derivative and the local extreme point(s) in a function. SymPy is easy to use and simple to read adding simplicity and readability to any Machine learning project that requires calculus. We’ve only touched on a few of the many functions available with this library. I encourage you to explore it further and see what else can be used to incorporate mathematics into your data science projects.
['Jeremiah Lutes']
['Python', 'Data Science', 'Calculus', 'Code', 'Machine Learning']