How does this example violate Liskov substitution principle, which then causes violation of the open-closed principle?
The listing does not appear to be a good illustration of LSP violation, since no clear reason is given why the draw method couldn't be a virtual method of Shape. If it was, then there would be no need to do any type query in drawShape: calling draw would work for all subclasses of Shape.

The Liskov Substitution Principle is typically described in terms of contracts. For example: the contract of the square root function requires that the input is non-negative and ensures that the square of the output is equal to the input.

Hence, an overridden method in a subclass is substitutable for the corresponding method in the superclass if it:


requires no more of its input than the superclass contract
ensures no less of its output


A subclass is LSP-conforming if all its methods are LSP conforming.

Hence, a better illustration of LSP violation might be a rotate method, in which all shapes except Square can be rotated by any angle, but Square is only rotatable by some multiple of 90% (such a requirement would actually demonstrate that the Square should not be a subclass of Shape).

In general, LSP violation causes OCP violation because there is no meaningful action that a 'closed' function can take without needing knowledge of specific subclasses in the type hierarchy. 

The standard example of this is to have a Circle class be a subclass of an Ellipse class that has a resize(xaxis,yaxis) method. Unless the semantics of resize are so diluted as to be largely useless ('maybe resize yourself or throw an exception') then subclasses can end up in a state that is inconsistent with anything meaningful/useful to the 'closed' function that is calling its methods.

A good discussion of the LSP (including the 'Circle-Ellipse' issue) can be found here