Implementation of perceptron, inspired by Rashka
Style

Python comes with an official Style Guide often just called PEP8, and especially as a beginnger it's a good starting point to get going. Of course coding style is often a matter of choice, however, there are aspects you should definitely follow.

Whitespace
As you know, Python's code structure is build upon indentation, so some parts are already language-defined. The style guide also has recommendations on how to use whitespace within statements and expressions. For example, , should always be followed be a single space character, while = is preceded and followed by a single space when used in assignments (there should be no whitespace around = if used as keyword arguments to functions such as foo(batz='bar')). So you would go from

gradL_W,gradL_b=self.backward(y_i,y_hat,x_i)
self.w_,self.b_=self.update(gradL_W,gradL_b)


to

gradL_W, gradL_b = self.backward(y_i, y_hat, x_i)
self.w_, self.b_ = self.update(gradL_W, gradL_b)


which does look way cleaner IMHO.

Variable names
It's always good practice to use descriptive variable names. Simply writing self.weights instead of self.w_ wont hurt that much. Apart from that, the Style Guide also recommends to use a single leading underscore for non-public variable names. Note, this is only a convention, since Python has no "real" private member variables as you may know from other languages.

Documentation
There is a single function in your perceptron class that has a) documentation and b) also follows the docstring-style from the Style Guide. You should apply that to the other functions and maybe even to the class as well. Your current project might be simple, but if projects become more complex, good documentation will save you a lot of headache. Following the official docstring style also has the nice benefit that Python's help(...) function as well as most IDEs will easily find it.

The code

Apart from the stylistic issues mentioned above, there are some more direct code-focused aspects you should work on.

Code duplication
There is some duplicated code in your implementation of the perceptron model. E.g., you implement the scalar product at the input of the perceptron twice! The first time at net_input, which is likely where it's supposed to go, and the second time in predict. predict also reimplements pretty much the whole point of activation. It's pretty straightforward to implement predict using net_input and activation:

def predict(self, X):
    s = self.net_input(X)
    y_hat = list(map(self.activation, s))
    return y_hat


If you use Python for a while you will also learn to use list comprehensions more often than the list(map(...)) construct, simply because it's more flexible and readable. Rewriting the calculation of y_hat as list comprehension would look like y_hat = [self.activation(i) for i in s]. After you have arrived at

def predict(self, X):
    s = self.net_input(X)
    y_hat = [self.activation(i) for i in s]
    return y_hat


you may also find the similarity between predict and forward quite striking. forward can now be boiled down to simply

def forward(self, x):
    return self.predict(x)[0]


Getting the "first" result of the return value using [0] is mainly to ensure that the new version just returns a single number as it was before. Otherwise it would be a list with a single element.

Explicit is better than implicit

(Taken from the Zen of Python)

Using code like 

heaviside = lambda x: (1, -1)[x &lt; 0]
return heaviside(z)


might be clever, but clever does not equal good or readable most of the time. The same thing can also be expressed as return -1 if z &lt; 0 else 1, which is almost a verbatim translation of the mathematical definition of the step function as presented by Rashka. If you don't like the inline-if, you may also write it as "normal" if condition:

if z &lt; 0:
    return -1
else:
    return 1


gradL_b = error * 1 is another instance of cleverness. This time, I am not even sure what you are actually trying to accomplish here. Depending on the type of your labels y, error will either be of type int or maybe also float, and multiplying it with 1 will not change that.

errors += (y_hat != y_i) * 1 is more obvious, but could also be expressed explicitly as errors += int(y_hat != y_i). Python should even be able to do errors += y_hat != y_i without changing the end result.

Return values
I'm not 100% sure what you are trying to accomplish with some of your function return values. For example, I cannot see the point in fit returning self. updateis another instance of that topic. I can see you're using it as self.w_, self.b_ = self.update(gradL_W, gradL_b), but there is not a real reason to do this, since update already modifies the weights internally.

You will likely have to think about that topic as you further progress in your journey into the depths of neural networks (pun intended). Often, the multilayer perceptron is  one of the next logical steps in this process. At this stage, you will need the gradients of previous layers to be able to propagate errors through the network. But time will tell1.

Until then: Happy coding!



1You will also probably soon find out that modelling single neurons is not sufficient (and computationally efficient) for what there is to come. Depending on your learning materials, matrix operations will pop up more and more often. This is the time where the NumPy library will help you shine.