What are the challenges in predicting lithology directly from acoustic impedance?
Yes, lithology interpretation/classification directly from seismic inversion products is something people do.  I've seen it work very well in oil and  gas exploration and development of fields. 

I personally think that there are two main challenges involved in predicting lithology from seismic inversion impedances. It would be a bit of a cop out to say that the whole inversion and lithology prediction process is challenging, so I won't, even though it is. So, assuming that you achieve good inversion results, by which I mean that your inversion synthetics match input seismic angle stacks and your inversion products (estimated elastic properties) match calibration points (upscaled well logs), I think that the challenges are the uncertainties associated with the:


ability to discriminate rock types in study area, and
low-frequency model


Below I go into a little more detail regarding these two factors. Just remember that I'm talking about lithology prediction from an absolute inversion. Also, I will talk about "elastic properties" a lot, by which I mean the P-impedance, S-impedance and density, or their derivatives like Lambda, Mu and Poisson's ratio if you're that way inclined. 

Discrimination of rock types

Before you attempt to predict lithologies from seismic inversion results, you need to know the expected elastic properties of the rocks in your study area. If the various rock types in your study area have the exact same elastic properties, distinguishing them from inversion products is going to be impossible, no matter how good your inversion is. To make a good lithology prediction you need to know the elastic properties of the measured or predicted rocks in your study area. If the rocks have similar (e.g. overlapping) rock properties you need to account for this uncertainty in your lithology prediction. 

For example, say you are targeting gas and the predicted reservoir is sandstone. Does a gas filled sand have different elastic properties to a brine filled sand? What about from the sealing shale? What are the expected range of elastic properties for these rocks? By doing forward modelling experiments (e.g. Gassman fluid substitution and/or lithology mixing) you could answer questions that address risk-cases like: what are the expected elastic properties of a low saturation gas sand or a high porosity brine sand? If high porosity brine sands and gas-sands have similar/overlapping elastic properties and I end up predicting a gas-sand from my inversion, what is the probability that the gas-sand is 'real'? 


(source)

The above image shows the elastic properties for three lithologies: sandstone, shale and limestone, in terms of P-wave velocity** (Vp) and the ratio of P- to S- velocity (Vp/Vs). Provided your inversion results are accurate, these lithologies would be distinguishable because their property populations do not overlap. 

The mean P-wave velocity (Vp) for the limestone is approximately 5800 m/s. But what if the mean Vp was 4000 m/s? The limestone population would then overlap with the shale, meaning that it would be difficult to discriminate between the two lithologies from your inversion results with great confidence. 

A thorough rock physics study is necessary before attempting any seismic inversion project to understand the elastic properties of your rocks and to ensure that your target rocks are distinguishable. If your rocks have sufficiently different elastic properties then lithology prediction from seismic inversion is feasible.  You can use your rock physics study to define how your lithologies can be classified from the inversion results (e.g. depth dependent probability density functions of P-impedance versus Vp/Vs for each lithology/fluid type). 

** Vp can only be estimated from seismic inversion if you also get a good density result, which is uncommon. I'd prefer it if this graph showed P-impedance (Vp * density), but beggars can't be choosers. 

Low-frequency model

So, you know the elastic properties of the rocks in your study area and you know that they can be theoretically discriminated, and you want to predict lithologies from seismic inversion results? I think that the next challenge in lithology prediction are the uncertainties associated with the low-frequency model, because the low-frequency model: 


determines the absolute value of estimated elastic properties


so has a very big influence on the predicted lithology

is built by interpolating/extrapolating sparse data (e.g. well logs or rock physics trends)


i.e. a lot of the spatial variation of the low-frequency model is not measured



A low-frequency model is used to fill in the low-frequency elastic property gap not measured by seismic data, and allow absolute elastic properties to be estimated. For conventional marine seismic with a bandwidth of, say, 10-60Hz, the low-frequency model will provide the inversion with elastic properties of the study area between 0-10Hz (see picture below). What kind of geological information is captured within this low-frequency range? Large scale geological changes, like compaction trends, or the elastic property changes associated with low-order sequence stratigraphy cycles. 


(source)

Without a low-frequency model, inversion can only provide the relative impedances of two layers; this is called a relative inversion. Inversion with a low-frequency model is called an absolute inversion, and it seeks to estimate the total elastic property of the layer, independent of impedance of above/below layers. If you get the low-frequency model wrong, your absolute elastic properties will also be wrong and therefore you will predict incorrect lithologies. 

A low-frequency model can be built by extrapolating low-pass filtered well logs (P- and S-sonic and density), rock physics trends, etc throughout the study area, creating a 3D distribution of elastic properties. How can you be sure that you have a good low-frequency model?  


Blind well testing can give you confidence in your model. 
Use a robust method for extrapolating your well logs or rock physics trends. 
Even better would be to improve the low-frequencies of your seismic data through deghosting, to reduce the influence of the low-frequency model on your final interpretation.


Final thoughts

You can create different criteria for classifying lithologies, run an absolute inversion and predict lithology fairly easily these days. But if you don't have confidence in your ability to discriminate between different rock types or you're not sure if your estimated absolute elastic properties are correct, you cannot be sure that the lithologies you have predicted are useful. So what to do? Firstly you need to do a really good rock physics study first to understand the elastic properties of the rocks, and rocks you predict to be present, in your study area. Secondly put in a lot of effort to making sure you have a robust low-frequency model.