What statistical model should I use to analyze the likelihood that a single event influenced longitudinal data
For the record, I think this is the type of question that's perfect for the data science Stack Exchange. I hope we get a bunch of real world examples of data problems and several perspectives on how best to solve them.

I would encourage you not to use p-values as they can be pretty misleading (1, 2). My approach hinges on you being able to summarize traffic on a given page before and after some intervention. What you care about is the difference in the rate before and after the intervention. That is, how does the number of hits per day change? Below, I explain a first stab approach with some simulated example data. I will then explain one potential pitfall (and what I would do about it).

First, let's think about one page before and after an intervention. Pretend the intervention increases hits per day by roughly 15%:

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

def simulate_data(true_diff=0):
    #First choose a number of days between [1, 1000] before the intervention
    num_before = np.random.randint(1, 1001)

    #Next choose a number of days between [1, 1000] after the intervention
    num_after = np.random.randint(1, 1001)

    #Next choose a rate for before the intervention. How many views per day on average?
    rate_before = np.random.randint(50, 151)

    #The intervention causes a `true_diff` increase on average (but is also random)
    rate_after = np.random.normal(1 + true_diff, .1) * rate_before

    #Simulate viewers per day:
    vpd_before = np.random.poisson(rate_before, size=num_before)
    vpd_after = np.random.poisson(rate_after, size=num_after)

    return vpd_before, vpd_after

vpd_before, vpd_after = simulate_data(.15)

plt.hist(vpd_before, histtype="step", bins=20, normed=True, lw=2)
plt.hist(vpd_after, histtype="step", bins=20, normed=True, lw=2)
plt.legend(("before", "after"))
plt.title("Views per day before and after intervention")
plt.xlabel("Views per day")
plt.ylabel("Frequency")
plt.show()




We can clearly see that the intervention increased the number of hits per day, on average. But in order to quantify the difference in rates, we should use one company's intervention for multiple pages. Since the underlying rate will be different for each page, we should compute the percent change in rate (again, the rate here is hits per day).

Now, let's pretend we have data for n = 100 pages, each of which received an intervention from the same company. To get the percent difference we take (mean(hits per day before) - mean(hits per day after)) / mean(hits per day before):

n = 100

pct_diff = np.zeros(n)

for i in xrange(n):
    vpd_before, vpd_after = simulate_data(.15)
    # % difference. Note: this is the thing we want to infer
    pct_diff[i] = (vpd_after.mean() - vpd_before.mean()) / vpd_before.mean()

plt.hist(pct_diff)
plt.title("Distribution of percent change")
plt.xlabel("Percent change")
plt.ylabel("Frequency")
plt.show()




Now we have the distribution of our parameter of interest! We can query this result in different ways. For example, we might want to know the mode, or (approximation of) the most likely value for this percent change:

def mode_continuous(x, num_bins=None):
    if num_bins is None:
        counts, bins = np.histogram(x)
    else:
        counts, bins = np.histogram(x, bins=num_bins)

    ndx = np.argmax(counts)
    return bins[ndx:(ndx+1)].mean()

mode_continuous(pct_diff, 20)


When I ran this I got 0.126, which is not bad, considering our true percent change is 15. We can also see the number of positive changes, which approximates the probability that a given company's intervention improves hits per day:

(pct_diff &gt; 0).mean()


Here, my result is 0.93, so we could say there's a pretty good chance that this company is effective.

Finally, a potential pitfall: Each page probably has some underlying trend that you should probably account for. That is, even without the intervention, hits per day may increase. To account for this, I would estimate a simple linear regression where the outcome variable is hits per day and the independent variable is day (start at day=0 and simply increment for all the days in your sample). Then subtract the estimate, y_hat, from each number of hits per day to de-trend your data. Then you can do the above procedure and be confident that a positive percent difference is not due to the underlying trend. Of course, the trend may not be linear, so use discretion! Good luck!