Modeling price vs demand
I've tangled with modeling pricing systems over the last two years and one of my key learnings applies here:
Available sales data is often a bad basis for straight-forward prediction tasks and the reason for this is fairly simple:
If you classify all prices (or transactions of a given product at a price x) into &quot;Accepted&quot; and &quot;Not accepted&quot; by the customer you will realize that the data provided by your customer only contains instances of &quot;Accepted&quot; prices.
Therefore a straight-forward modeling of y ~ x with y = demand and x = price is impossible because your y does not vary!
There are several ways around this however. In my comment I mentioned discount being a valuable information!
You have already noticed that discounts are not depended on logical variables, they are seemingly random, this isn't true!
Discounts in most organizations are very, very flexible and often applied manually based on negotiations. This means they are a great indicator of our target y &quot;Acceptance&quot;/&quot;Non Acceptance&quot;.
Consider this:
Discount = Demand x Undiscounted_Price
This means that large discounts indicate that the demand is low / negative for the undiscounted price and low discounts indicate demand is high.
To truly discover this relation you might need to model codependent factors and then remove them by training new models on the residuals.
Edit:
An important thing to add especially for the B2B domain. Demand for a certain product is almost set in stone for a customer. Unlike consumers companies do not buy surplus or refrain from purchase due to the prices.
What they do is switch suppliers! This means that there is a really important unknown variable &quot;Customer Demand for product X&quot;. You do not want model this variable but you need it to model what you actually want to do:
Share of Wallet or the percentage of the fixed customer demand that was satisfied by your company with the goal being to identify the price that will optimize that percentage. This is helpful to keep in mind because it constrains the performance of any model as you never know whether the historic demand you have in your data is already 0% or 100% of the total demand and thus could not decrease/increase regardless of price.