Experimental Evaluation of Residential Demand Response in California

arXiv:1710.03190v3 [physics.soc-ph] 10 Jan 2018

Datong P. Zhou∗ , Maximilian Balandat† , and Claire J. Tomlin†

Abstract— We evaluate the causal effect of hour-ahead price
interventions on the reduction of residential electricity consumption, using a large-scale experiment on 7,000 households
in California. In addition to this experimental approach, we
also develop a non-experimental framework that allows for
an estimation of the desired treatment effect on an individual
level by estimating user-level counterfactuals using time-series
prediction. This approach crucially eliminates the need for a
randomized experiment. Both approaches estimate a reduction of ≈0.10 kWh (11%) per Demand Response event and
household. Using different incentive levels, we find a weak
price elasticity of reduction. We also evaluate the effect of an
adaptive targeting scheme, which discriminates users based on
their estimated responses in order to increase the per-dollar
reduction ratio by 30%. Lastly, we find that households with
smart home automation devices reduce significantly more than
households without, namely 0.28 kWh (37%).

I. I NTRODUCTION
This paper studies the causal effect of incentivizing residential households to participate in Demand Response (DR)
to temporarily reduce electricity consumption. DR has been
promoted by the introduction of demand-side management
programs (DSM) after the 1970s energy crisis [1], enabled
by the integration of information and communications technology in the electric grid. The rationale behind DSM is the
inelasticity of energy supply due to the slowness of power
plants’ output adjustment, which causes small increases and
decreases in demand to result in a price boom or bust,
respectively. Since utilities are obligated to provide end-users
with electricity at a quasi-fixed tariff at all times [2], e.g.
Time-of-Use pricing, they have to bear price risks. Therefore,
DSM attempts to protect utilities against such price risks by
partially relaying them to end-users, which increases market
efficiency according to the economic consensus [3].
In 2015, the California Public Utilities Commission
(CPUC) launched a Demand Response Auction Mechanism
(DRAM) [4], requiring utilities to procure a certain amount
of reduction capacity from DR providers. These aggregators
incentivize their customers (also called “Proxy Demand
Resource” (PDR) [5]) under contract to temporarily reduce
their consumption relative to their projected usage without
intervention, referred to in this context as counterfactual or
baseline, based on which compensations for (non-)fulfilled
reductions are settled: If the consumer uses less (more)
energy than the baseline, she receives a reward (incurs a
penalty). Figure 1 illustrates the interactions between agents.
The estimation of the materialized reduction arguably is
the most critical component of the DR bidding process. If
∗ Department of Mechanical Engineering, University of California,
Berkeley. † Department of EECS, University of California, Berkeley.
[datong.zhou, balandat, tomlin]@berkeley.edu

Wholesale Market
Scheduling Coordinator

DR Provider

Electric Utility
PDR

End-Use Customers

Fig. 1: Interactions of Agents in Residential Demand Response

the reductions are estimated with a biased counterfactual,
either the DR provider or the utility clearing the bids is systematically discriminated against. If the baseline is unbiased
but plagued by high variance, the profit settlement is highly
volatile. Existing baselines employed by major power grid
operators in the United States (e.g. California Independent
System Operator (CAISO), New York ISO) are calculated
with simple arithmetic averages of previous observations [5]
and therefore are inaccurate. The estimation of more accurate
baselines is a significant contribution of this paper.
A. Contributions
We estimate the average treatment effect (ATE) of hourahead notifications on the reduction of electricity consumption by evaluating a Randomized Controlled Trial (RCT) on
≈ 5000 residential households in California serviced by the
three main electric utilities (PG&E, SDG&E, SCE). This
experiment is funded by the California Energy Commission,
which − to the best of our knowledge − is the first one
to experiment with hour-ahead notifications on a residential
household level. We estimate an ATE of −0.10 kWh per DR
Event and user and further discover notable geographic and
temporal heterogeneity among users, as the largest estimated
reductions occur in summer months as well as in regions with
warmer climate.
In addition to this experimental approach, we also develop
a non-experimental method for estimating this causal effect
on an individual user level, which is easily aggregated
into an ATE. Importantly, we find that the identified ATEs
in both cases are close to each other. Interestingly, the
non-experimental approach even achieves tighter confidence
intervals of the estimated causal effect. This suggests that
our methodology is capable of identifying the causal impact
of any intervention in settings with high-frequency data,
thereby circumventing financial and ethical constraints which
frequently arise in clinical trials, transportation, or education.
Lastly, we design an adaptive targeting method to exploit
the heterogeneity in users’ responses to incentive signals
to assign differing price levels to different subsets of the
treatment population. Specifically, we separate users based

on their previous responses into two distinct groups, each
of which either only receives low or high incentives. This
method yields an increase of the per-dollar yield of ≈30%.
This paper is structured as follows: Section II explains the
experimental setup and provides summary statistics on the
RCT data. We then develop the non-experimental estimation
framework in Section III, where we pay particular attention
to estimation bias and empirical de-biasing methods (Section
III-C). Non-experimental estimation results are provided in
Section IV. Next, in Section V we estimate the ATE using a
classical Fixed-Effects Estimator [6]. Section V-F compares
the estimates obtained by both approaches. Lastly, the effect
of adaptive targeting is discussed in Section VI. Section VII
concludes. Additional figures and numeric data are relegated
to the appendix.
B. Related Work
Causal inference seeks to extrapolate the effect of interventions to general settings, however, many experiments are
infeasible or unethical due to budget or ethical factors. With
the rapid growth of collected user data, non-experimental
estimates become more and more valuable, as the new hope
is to use such estimates in place of experiments. These
facts have spurred research at the intersection of machine
learning and economics, whose general idea is to partition
observations under treatment and control in order to fit a
nominal model on the latter set, which, when applied on the
treatment set, yields the treatment effect of interest.
Examples for such models are [7], who evaluates welfare
effects of home automation by calculating the KolmogorovSmirnov Statistic between users, or [8], who constructs a
convex combination of US states as the counterfactual estimate for tobacco consumption to estimate the effect of a tobacco control program in California on tobacco consumption.
In [9], the estimators are random forests trained by recursive
partitioning of the feature space and novel cross-validation
criteria. [10] develops Bayesian structural time series models
combined with a Monte-Carlo sampling method for treatment
effect inference of market interventions.
Fitting an estimator on smart meter time-series is essentially a short-term load forecasting (STLF) problem, whose
goal is to fit estimators on observed data to predict future
consumption with the highest possible accuracy. Within
STLF, tools employed are ARIMA models with a seasonal
component [11] and classic regression models where support vector regression (SVR) and neural networks yield the
highest accuracy [12], [13]. A comprehensive comparison
between ML techniques for forecasting and differing levels
of load aggregation is provided in [14].
In the context of smart meter data mining, much of the
existing work focuses on disaggregation of energy consumption to identify contributions of discrete appliances from the
total observed consumption [15] and to learn consumption
patterns [16], [17]. Studies in applied economics typically
emphasize the estimation of ATEs of experimental interventions. To increase precision of the estimates, the employed
regression models often employ unit-level fixed effects [18],

[19], which is an implicit way of training models for the
consumption of individual consumers. In this work, we make
these user-level models explicit, allowing for more general
ML techniques. Importantly, our approach is original as it
permits to perform causal inference on the level of individual
treatment effects in a straightforward fashion by employing estimators from STLF. To the best of our knowledge,
this paper is the first of its kind to analyze the potential
of Demand Response interventions on a residential level,
combining ideas at the intersection of causal inference from
econometrics and Machine Learning for estimation.
II. E XPERIMENTAL S ETUP AND DATA C HARACTERISTICS
A. Setup of the Experiment
The experiment is carried out by OhmConnect, Inc., using
funds provided by the California Energy Commission. Figure
2 draws a flowchart of the experimental setup.
All Participants
Treatment−
Encouraged

Treatment−
Non-Encouraged
Phase 1 (90 days)

Control

Estimate Responses
Targeted-High

Targeted-Low
Non-Targeted
Phase 2 (90 days)
Fig. 2: Setup of Experiment

Over the course of the experimental time period (Nov.
2016 - Dec. 2017), each consumer that signs up for the study
is randomly assigned to one of the following groups:
• Treatment−Encouraged: The user receives an average
number of 25 DR events in the 90 days following the
signup, with incentive levels being randomly chosen
from the set R = {0.05, 0.25, 0.50, 1.00, 3.00} USD
kWh .
Additionally, the user is given a rebate for purchasing
a smart home automation device.
• Treatment−Non-Encouraged: Same as in TreatmentEncouraged, but without smart home automation rebate.
• Control: Users do not receive any notifications for DR
events for a period of 90 days after sigup.
These three groups form Phase 1 of the experiment.
Users in the control group that have reached 90 days
of age are removed from the study. Users in either
the Treatment−Encouraged or Treatment−Non-Encouraged
groups that have reached 90 days of age are pooled and
systematically assigned to one of the following groups for
Phase 2 interventions:
• Targeted-High: The user receives an average number of
25 DR events for a period of 90 days after being rolled
over into Phase 2. Each reward level is randomly drawn
from the set {1.00, 3.00} USD
kWh .
• Targeted-Low: Same as in Targeted-High, but rewards
are randomly drawn from {0.05, 0.25, 0.50} USD
kWh .
• Non-Targeted: Same as in targeted groups, with rewards
drawn from {0.05, 0.25, 0.50, 1.00, 3.00} USD
kWh .

Users with completed Phase 2 are removed from the study.
In Sections III-V, we evaluate Phase 1 of the experiment
whereas Section VI is dedicated to adaptive targeting (Phase
2). In the remainder of this paper, we use the term “treatment
users” to refer to users in the “Treatment-Encouraged” and
“Treatment-Non-Encouraged” group. Users receive notifications of a DR event with the incentive level up to 20 minutes
into an hour, which lasts until the end of the hour.
B. Summary Statistics
Table I reports the number of users by experiment group
and proportion of users for which we were able to scrape
historical smart meter reading data. The table shows that
the randomized assignment of users to groups roughly follows a 1:2:2 ratio (Control vs. Treatment−Encouraged vs.
Treatment−Non-Encouraged).
Historical Smart Meter Data Availability by Group
Group
# Enrolled
# With Data
# With DR
Control
3266
1373
−
Treatment−Encouraged
6735
2953
2921
Treatment−Non-Enc.
6689
3014
2985
TABLE I: Number of Total Users Enrolled by Group, Data Availability, and
Users with DR Events.

Users without DR events or for which we were unable to
scrape historical data are omitted from the study. Since the
assignment of users into the different experimental groups
was randomized (see Section II-D), dropping such users does
not affect the evaluation of the experiment. Figure 3 shows
the geographic distribution of the remaining users.
Geographic Distribution of Users
PG&E: 2969 Users
SCE: 1675 Users
SDG&E: 969 Users

D. Balance Checks
To verify that users were randomly assigned to control
and treatment groups, we perform a balance check on the
distribution of observed air temperatures and electricity consumptions across both groups. Notice that the relatively large
sample size renders a classical differences-in-means t-test
inappropriate. Therefore, we utilize Cohen’s d to estimate the
effect size based on the differences between means, which
is insensitive to the large sample size. Given two discrete
distributions P and Q with sample sizes n1 /n2 and means
x̄1 /x̄2 , Cohen’s d is defined as

1/2
(n1 − 1)s21 + (n2 − 1)s22
x̄1 − x̄2
, (1)
, s=
d=
s
n1 + n2 − 2
where s1 and s2 are the sample standard deviations for
distributions P and Q, respectively. In addition, we use
the Hellinger Distance H as a nonparametric comparison
to quantify the similarity between the distributions [22]:
1
H(P, Q) = √
2

k
X
√
√
( pi − qi )2

!1/2
(2)

i=1

where P = {p1 , . . . , pk } and Q = {q1 , . . . , qk }. To compute
(1) and (2), we discretize the temperature and consumption
distributions appropriately. Table VI in the Appendix provides these metrics together with the differences in means
for a selected subset of hours of the day, which was chosen
to coincide with those hours of the day for which DR events
were observed (see Figure 10). We omit the metrics for
the remaining hours of the day as they are very similar
to the listed ones. As the Hellinger Distance H ∈ [0, 1],
with 0 corresponding to a perfect similarity and 1 to total
dissimilarity, we can assume that the assignment of users
into treatment and control group is as good as random.
III. N ONEXPERIMENTAL T REATMENT E FFECT
E STIMATION
A. Potential Outcomes Framework

Fig. 3: Geographic Distribution of Users

C. Weather Data
Hourly measurements of ambient air temperature are
scraped from the publicly accessible California Irrigation
Management Information System [20]. As there are fewer
weather stations than distinct user ZIP codes, we linearly
interpolate user-specific temperatures at their ZIP codes from
the two closest weather stations in latitude and longitude by
calculating geodesic distances with Vincenty’s formulae [21].

To estimate the effect of the DR intervention program,
we adopt the potential outcomes framework introduced by
Rubin (1974) [23]. Let I = {1, . . . , n} denote the set of
users. The indicator Dit ∈ {0, 1} encodes the fact whether
or not user i received DR treatment at time t. Each user is
endowed with a consumption time series yi = {yi1 , . . . , yiτ }
and associated covariates Xi = {xi1 , . . . , xiτ } ∈ ×τi=1 Xi ,
Xi ⊂ Rnx , where time is indexed by t ∈ T = {1, . . . , τ }
0
and nx is the dimension of the covariate space Xi . Let yit
1
and yit denote user i’s electricity consumption at time t for
Dit = 0 and Dit = 1, respectively. Let Ci and Ti denote the
set of control and treatment times for user i. That is,
Ci = {t ∈ T | Dit = 0},

Ti = {t ∈ T | Dit = 1}.

(3)

The number of treatment hours is much smaller than the
number of non-treatment hours. Thus 0 < |Ti |/|Ci |  1.

Di,t = {(xit , yit ) | t ∈ Ti }, Di,c = {(xit , yit ) | t ∈ Ci }. (4)
The one-sample estimate of the treatment effect on user i at
time t, given the covariates xit ∈ Rnx , is
1
0
βit (xit ) := yit
(xit ) − yit
(xit ) ∀ i ∈ I, t ∈ T,

t∈Ti

The average treatment effect on the treated (ATT) follows
from (6):
1 X 1 X 1
0
(yit − yit
). (7)
ATT := Ei∈I [βi ] =
|I|
|Ti |
t∈Ti

Since users were put into different experimental groups in a
randomized fashion, the ATT and the average treatment effect (ATE) are identical [24]. Lastly, the conditional average
treatment effect (CATE) on x̃ is obtained by marginalizing
the conditional distribution of one-sample estimates (5) on
x̃ over all users and treatment times, where x̃ ∈ Rñx is a
subvector of x ∈ Rnx , 0 < ñx < nx :
h
i
1
0
CATE(x̃) := Ei∈I Et∈Ti (yit
− yit
) x̃it = x̃ . (8)
The CATE captures heterogeneity among users, e.g. with
respect to specific hours of the day, the geographic distribution of users, the extent to which a user possesses “smart
home” appliances, group or peer effects, etc. To rule out
the existence of unobserved factors that could influence the
assignment mechanism generating the complete observed
data set {(xit , yit , Dit ) | i ∈ I, t ∈ T}, we make the
following standard assumptions:
Assumption 1 (Unconfoundedness of Treatment Assignment). Given the covariates {xit }t∈T , the potential outcomes
are independent of treatment assignment:
0
1
(yit
, yit
) ⊥ Dit | xit

∀i ∈ I, t ∈ T.

0
1
0
yit = yit
+ Dit · (yit
− yit
) ∀ t ∈ T.

(5)

which varies across time, the covariate space, and the user
population. Marginalizing this one-sample estimate over the
set of treatment times Ti and the covariate space Xi yields
the user-specific Individual Treatment Effect (ITE) βi
i
h
1 X 1
0
1
0
yit − yit
. (6)
xit =
βi := EXi Et∈Ti yit
− yit
|Ti |

i∈I

outcomes. That is, the conditional distribution of the potential
outcomes, given covariates, remains constant.
The fundamental problem of causal inference [26] refers
to the fact that either the treatment or the control outcome
can be observed, but never both (granted there are no missing
observations). That is,

(9)

Thus, the ITE (6) is not identified, because one and
only one of both potential outcomes is observed, namely
1
0
{yit
| t ∈ Ti } for the treatment times and {yit
| t ∈ Ci } for
the control times. It therefore becomes necessary to estimate
counterfactuals.
B. Non-Experimental Estimation of Counterfactuals
Consider the following model for the estimation of such
counterfactuals:
yit = fi (xit ) + Dit · βit (xit ) + εit ,

∀i ∈ I, t ∈ T.

(10)

Assumption 1 is the “ignorable treatment assignment”
assumption introduced by Rosenbaum and Rubin [25]. Under
this assumption, the assignment of DR treatment to users is
implemented in a randomized fashion, which allows the calculation of unbiased ATEs (7) and CATEs (8). Assumption 2,
motivated by the time-series nature of the observational data,
ensures that the set of observable covariates {xit | t ∈ T} can
capture seasonality effects in the estimation of the potential

(12)

where εit denotes noise uncorrelated with covariates and
treatment assignment. fi (·) : Rnx 7→ R is the conditional
mean function and pertains to Dit = 0. To obtain an estimate
0
for fi (·), denoted with fˆi (·), control outcomes {yit
| t ∈
Ci } are first regressed on {xit | t ∈ Ci }, namely their
observable covariates. In a second step, the counterfactual
0
ŷit
for any t ∈ Ti can be estimated by evaluating fˆi (·)
on its associated covariate vector xit . Finally, subtracting
0
1
ŷit
from yit
isolates the one-sample estimate βit (xit ), from
which the user-specific ITE (6) can be estimated. Figure 4
illustrates this process of estimating the reduction during a
1
DR event by subtracting the actual consumption yit
from
0
the predicted counterfactual ŷit = fˆi (xit ). Despite the fact
that consumption can be predicted for horizons longer than a
single hour, we restrict our estimators fi (·) to a single hour
prediction horizon as DR events are at most one hour long.
Consumption/Temperature
materialized
consumption
0
fˆi (xit ) = ŷit

Estimated
Reduction
ambient air
actual temperature
consumption

Assumption 2 (Stationarity of Potential Outcomes). Given
the covariates {xit }t∈T , the potential outcomes are independent of time, that is,
0
1
(yit
, yit
) ⊥ t | xit

(11)

Covariates xit
for Estimation

1
yit

estimated
counterfactual

DR Event

Further, let Di,t and Di,c denote user i’s covariate-outcome
pairs of treatment and control times, respectively. That is,

t
8:00

9:00

10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00

0 using Treatment Covariates x
Fig. 4: Estimation of the Counterfactual ŷit
it
0 − y1
and Predicted Reduction ŷit
it

To estimate fi (·), we use the following classical regression
methods [27], referred to as estimators:
E1: Ordinary Least Squares Regression (OLS)
E2: L1 Regularized (LASSO) Linear Regression (L1)
E3: L2 Regularized (Ridge) Linear Regression (L2)
E4: k-Nearest Neighbors Regression (KNN)

E5: Decision Tree Regression (DT)
E6: Random Forest Regression (RF)
DT (E5) and RF (E6) follow the procedure of Classification
and Regression Trees [28]. We compare estimators (E1)-(E6)
to the CAISO 10-in-10 Baseline (BL) [5], which, for any
given hour on a weekday, is calculated as the mean of the
hourly consumptions on the 10 most recent business days
during the selected hour. For weekend days and holidays, the
mean of the 4 most recent observations is calculated. This
BL is further adjusted with a Load Point Adjustment, which
corrects the BL by a factor proportional to the consumption
three hours prior to a DR event [5].
Since users tend to exhibit a temporary increase in consumption in the hours following the DR intervention [1], we
remove nr = 8 hourly observations following each DR event
in order to prevent estimators (E1)-(E6) from learning from
such spillover effects. This process is illustrated in Figure 5.
Consumption

t
Fig. 5: Separation of consumption time series into training set (green), DR
Events (grey), and Periods of Spillover Effects (blue)

Hence the training data Di,tr ⊂ Di,t used to estimate
the conditional mean function fˆi (·) (12) consists of all
observations leading up to a DR event, excluding those that
are within 8 hours of any DR event. To estimate user i’s
0
counterfactual outcome ŷit
during a DR event t ∈ Ti , we
use the following covariates:
• 5 hourly consumption values preceding time t
• Air temperature at time t and 4 preceding measurements
• Hour of the day, an indicator variable for (non-)business
days, and month of the year as categorical variables
Thus, the covariate vector writes
0
xit = [yit−1

···

0
yit−5

Tit

C(HoDit ) : C(is Bdayit )

···

Tit−4

C(MoYit )].

(13)

•

Seasonal and temporal bias: Due to the experimental
design, DR events for a particular user are concentrated
within a period of 180 days after signing up. Further,
DR events are called only in the afternoon and early
evening (see Figure 10). Thus, fitting an estimator on
all available historical data is likely to introduce bias
during these time periods of interest.

To deal with these challenges, we use the de-biasing procedure presented in Algorithm 1, which was first introduced
in [29]. We first separate a subset of non-DR events from
Algorithm 1 Unbiased Estimation of Counterfactuals
Input: Treatment data Di,t , control data Di,c , Estimator, η
1: Split Di,c into training data Di,tr and placebo data Di,pl
according to empirical distribution of Ti . Split control
times Ci into training times Ci,tr and placebo times Ci,pl
2: Compute weights for Di,tr = {(xit , yit ) | t ∈ Ci,tr }
according to (15a)-(15c), using η
3: Fit conditional mean function fˆi on Di,tr with weights
0
4: Estimate placebo counterfactuals {ŷit
| t ∈ Ci,pl }
5: Compute bias on placebo treatment set
0
6: Estimate treatment counterfactuals {ŷit
| t ∈ Ti }
7: Subtract placebo treatment bias from estimated treatment
counterfactuals:
X
1
0
0
0
0
ŷit
← ŷit
−
(ŷiτ
− yiτ
) ∀ t ∈ Ti (14)
|Ci,pl |
τ ∈Ci,pl

user i’s control data Di,c , which we call the placebo set
Di,pl with associated placebo treatment times Ci,pl (we chose
Di,pl to be of size 25). This placebo set is drawn according
to user i’s empirical distribution of Phase 1 DR events by
hour of day and month of year. Next, the non-experimental
estimator of choice is fitted (using cross-validation to find
hyperparameters to minimize the mean squared prediction
error) on the training set Di,tr . Importantly, to account for
temporal bias, we assign weights to the training samples,
ensuring that samples in “similar” hours or seasons as actual
DR events are assigned larger weights. Specifically, the
weights wit are determined as follows:

In (13), Tit denotes temperature, HoDit hour of day,
is Bdayit an indicator variable for business days, and MoYit
the month of year (all for user i at time t). “C” denotes
dummy variables and “:” their interaction.

HoD MoY
wit ∝ wit
wit ,
X
HoD
wit = η +
1(HoDit = HoDiτ ),

C. Placebo Treatments and De-biasing of Estimators

MoY
wit
=η+

As previously mentioned, a crucial element of an estimator is unbiasedness. If an estimator systematically predicts
counterfactuals that are too large (small), users receive an
excess reward (are paid less) proportional to the amount of
prediction bias. For a fair economic settlement, it is thus
desirable to minimize the amount of bias. In our application,
such prediction bias is caused by the following two factors:
• Inherent bias of estimators: With the exception of OLS
(E1), (E2)-(E6) are inherently biased, which is justified
due to the well-known bias-variance tradeoff.

(15a)
(15b)

τ ∈Ci,tr

X

1(MoYit = MoYiτ ),

(15c)

τ ∈Ci,tr

where η > 0 is a constant to be chosen a-priori.
Then, the fitted model is used to predict counterfactuals
associated with placebo events. This yields a set of |Ci,pl |
paired samples from which we can obtain a proxy of the
estimation bias that remains even after assigning sample
weights according to the previous step. Finally, to obtain an
empirically de-biased estimate of actual Phase 1 DR events,
we simply subtract this proxy of the estimation bias from
predicted Phase 1 DR event outcomes.

D. Estimation of Individual Treatment Effects
To obtain point estimates for user i’s ITE βi , we simply
average all one-sample estimates (5) according to (6). To
obtain an estimate of whether or not a given user i has
actually reduced consumption, we utilize a nonparametric
permutation test with the null hypothesis of a zero ITE:
H0 : βi = 0,

H1 : βi 6= 0.

(16)

0
1
Given user i’s paired samples {zit = ŷit
− yit
| t ∈ Ti }
during DR periods, the p-value associated with H0 (16) is
P
1(D̄ ≤ β̂i )
.
(17)
p = D∈Pi |T |
2 i

In (17), D̄ denotes the mean of D. Pi denotes the set of all
possible assignments of signs to the pairwise differences in
1
0
the set {zit = yit
− ŷit
| t ∈ Ti }. That is,
Pi = {s1 zi1 , . . . , s|Ti | zi|Ti | | sj ∈ {±1}, 1 ≤ j ≤ |Ti |} (18)
which is of size 2|Ti | . Finally, the p-value from (16) is
calculated as the fraction of all possible assignments whose
means are less than or equal the estimated ITE β̂i . In
practice, as the number of DR events per user in Phase 1
is about 25 (see Figure 2), the number of total possible
assignments becomes computationally infeasible. Thus, we
randomly generate a subset of 105 assignments from Pi to
compute the p-value in (17). Moreover, we use the percentile
bootstrap method [30] to compute a confidence interval of
the estimated ITE for user i around the point estimate β̂i .
IV. N ONEXPERIMENTAL E STIMATION R ESULTS
A. Average Treatment Effect
Figure 6 shows ATE point estimates and their 99% bootstrapped confidence intervals conditional on differing reward
levels for all estimators as well as the CAISO BL. Due to the
empirical de-biasing procedure (see Section III-C), the point
estimates for estimators E1-E6 are close to each other. BL
appears to be biased in favor of the DRP, as it systematically
predicts smaller reductions than E1-E6.

the results give rise to a notable correlation between incentive
levels and reductions.
To compare the prediction accuracy of the estimators,
Table II reports the width of the confidence intervals for
each method and incentive level. The inferiority of the
CAISO baseline compared to the non-experimental estimators, among which RF achieves the tightest confidence
intervals, becomes apparent. Therefore, in the remainder
of this paper, we restrict all results achieved with nonexperimental estimators to those obtained with RF.
Width of CATE Confidence Intervals (kWh) by Incentive Level
BL
KNN
OLS
L1
L2
DT
RF

0.25 USD
kWh
0.0273
0.0219
0.0217
0.0211
0.0214
0.0251
0.0210

0.50 USD
kWh
0.0269
0.0235
0.0230
0.0233
0.0214
0.0247
0.0212

1.00 USD
kWh
0.0289
0.0227
0.0219
0.0212
0.0219
0.0241
0.0211

3.00 USD
kWh
0.0266
0.0214
0.0219
0.0225
0.0218
0.0250
0.0205

TABLE II: Width of 99 % Confidence Intervals around ATE Point Estimate
Conditional on Incentive Level, All Estimators

B. Individual Treatment Effects
Figure 7 plots ITEs for a randomly selected subset of
800 users who received at least 10 DR events in Phase 1,
estimated with RF. Users are sorted by their point estimates
(blue), whose 95% bootstrapped confidence intervals are
drawn in black. Yellow lines represent users with at least
one active smart home automation device. By marginalizing
the point estimates over all users with at least 10 events, we
obtain an ATE of −0.104 kWh (−11.4%), which is close to
−0.105 kWh as reported earlier. The difference ensues from
only considering users with at least 10 DR events. The 99%
ATE confidence interval is [−0.115, −0.093] kWh.
800

700

Estimated ITEs of 800 Randomly Selected Users with ≥ 10 DR Events, RF
95% ITE Confidence Interval
Point Estimate
Zero
Mean Reduction
99% ATE Confidence Interval

600

Estimated Reductions [kWh] and 99% Confidence Intervals

500

User Number

−0.08

Reductions [kWh]

0.05 USD
kWh
0.0277
0.0218
0.0214
0.0217
0.0221
0.0255
0.0211

−0.09
−0.10
−0.11
−0.12

400

300

−0.13

BL

−0.14

0.05

KNN

OLS

0.25

L1

0.50

L2

DT

1.00

RF

3.00

Incentive [USD/kWh]

200

Fig. 6: CATEs by Incentive Level with Bootstrapped Confidence Intervals

The ATE averaged over the predictions of estimators E1E6 is 0.105 kWh / 11.5%. The intercept and the slope of
the demand curve are −0.099 kWh / −0.013 kWh/USD,
meaning that users reduce an additional 0.013 kWh per
dollar offered, which is only a small change. Due to the
idiosyncratic nature of the CATE for r = 0.5 USD
kWh , the slope
and intercept have to be interpreted with caution. However,

100

0
−2.5

−2.0

−1.5

−1.0

−0.5

0.0

0.5

1.0

1.5

Absolute Reduction [kWh]

Fig. 7: Distribution of ITEs with Bootstrapped Confidence Intervals

Table III reports estimated ATEs for users with or without
active smart home automation devices, which are obtained

by aggregating the relevant estimated ITEs from Figure 7.
We notice larger responses as well as a larger percentage of
estimated reducers among automated users.
ATEs Conditional on Automation Status for Users with ≥ 10 DR Events
# Users
% Reducers
ATE (kWh)
ATE (%)
Automated
451
79.2
−0.279
−36.7
4491
63.6
−0.087
−9.62
Non-Automated
All
4942
65.0
−0.105
−11.5
TABLE III: Estimated CATEs by Automation Status, RF Estimator (E6)

C. Estimation by Month of the Year
The CATE by month of the year is found in a similar
fashion to the CATE by hour of the day:
Xit = [is treatit

BLit

C(MoY) : is DRit ] . (23)

Tit

D. Role of Smart Home Automation
The CATE by automation status is determined by introducing the indicator is autoit :

Table IV reports the percentage of significant reducers for
different confidence levels, obtained with the permutation
test under the null (16). From Tables III and IV, it becomes Xit = [is treatit BLit Tit C(is autoit ) : is DRit ] . (24)
clear that automated users show larger reductions than nonE. Effect of Automation Uptake Encouragement
automated ones, which agrees with expectations.
Lastly, the effect of incentivizing users to purchase a smart
Fraction of Significant Reducers (among sample of size 4942)
home automation device on energy consumption during DR
1 − α = 0.9
1 − α = 0.95
1 − α = 0.99
# Automated
225
205
159
events is determined as follows:
45.5
1162
25.9
1367
27.7

35.3
829
18.5
988
20.0

Xit = [is encit

TABLE IV: Estimated Percentage of Significant Reducers according to
Permutation Test, RF Estimator (E6)

V. ATE E STIMATION WITH F IXED E FFECTS M ODELS
To estimate the ATE of DR interventions on electricity
consumption, we consider the following fixed-effects model
with raw consumption (kWh) as the dependent variable:
kWhit = Xit · β + αit + uit .

(19)

In (19), subscripts i and t refer to user i at time t, respectively. Xit is a row vector of observable covariates, αit are
unobserved fixed effects, and uit is the noise term which is
assumed to be uncorrelated with the regressors and Gaussian
distributed with zero mean and finite variance. The fixed
effects term αit removes persistent differences across users
in their hourly and monthly consumption interacted with a
business day indicator variable:
αit ∼ C(HoDit ) : C(is Bdayit ) + C(MoYit ).

(20)

A. Estimation by Incentive Level
To estimate the CATE by incentive level, the covariate
matrix Xit in (19) is specified as follows:
Xit = [is treatit

BLit

Rit = [1(rit = 0.05)

Tit
···

Rit ] ,
1(rit = 3.00)] .

(21a)
(21b)

In (21a), is treatit is an indicator set to one for all treatment
users (and zero for all control users). BLit is the CAISO
baseline for user i at time t, which is necessary to control
for the non-random assignment of reward levels to users, Tit
is the ambient air temperature, and Rit is the reward level.
B. Estimation by Hour of the Day
To estimate the CATE by hour of the day, we pool all
reward levels into the indicator variable is DRit , which is
one if user i received treatment at time t and zero otherwise:
Xit = [is treatit

BLit

Tit

is nonencit

is encit · is DRit

C(HoD) : is DRit ] . (22)

BLit

Tit

is nonencit · is DRit ].

(25)

In (25), the indicators is enc and is nonenc are 1 for all
users in the “Treatment-Encouraged” and in “TreatmentNon-Encouraged”, respectively, and zero otherwise.
F. Comparison of Estimation Methods
We now benchmark the results obtained from the best nonexperimental estimator (RF) to those from the fixed effects
model with specification (20).
Figure 8 compares the point CATEs by reward levels
and their 95% confidence intervals. It can be seen that
the point estimates are close to each other (−0.101 kWh
aggregated for fixed effects vs. −0.105 for non-experimental
approach with RF, a less than 5% difference), a finding
that suggests that our non-experimental estimation technique
produces reliable estimates comparable to the experimental
gold standard. The fact that the confidence intervals are
notably tighter for RF corroborates this notion.
Estimated Reductions [kWh] and 95% Confidence Intervals
−0.07

Reductions [kWh]

49.9
1382
30.8
1607
32.5

% of Total
# Non-Automated
% of Total
# All
% of Total

ITE Estimates
FE Regression

−0.08
−0.09
−0.10
−0.11
−0.12
−0.13

0.05

0.25

0.50

1.00

3.00

Incentive [USD/kWh]

Fig. 8: CATEs by Incentive Level with Confidence Intervals, Comparison
Fixed Effects Estimators and Non-Experimental Estimators

VI. E FFECT OF A DAPTIVE TARGETING
The goal of adaptive targeting is to maximize the reduction
per dollar paid to the users, which is achieved by either
minimizing the payout and/or maximizing users’ reductions.
We evaluate the reduction by reward ratios for the targeted
(ρt ) and non-targeted (ρn ) groups by averaging the per-event

reductions β̂it (xit ) (5) normalized by the reward rit :
P
P
(2) β̂it (xit )/rit
i∈St
t∈Ti
ρt =
,
(26a)
P
(2)
i∈St |Ti |
P
P
(2) β̂it (xit )/rit
i∈Sn
t∈Ti
ρn =
,
(26b)
P
(2)
i∈Sn |Ti |
(2)

where Ti

denotes user i’s set of Phase 2 DR events.

A. Targeting Assignment Algorithm
Algorithm 2 describes the targeting assignment algorithm
on a given set of users, which we denote with S. Users are
Algorithm 2 Adaptive Targeting of Users
Input: Set of users S with completed Phase 1
Output: Groups Sn , Sl , and Sh of proportion 2 : 1 : 1
1: Randomly split S = Sn ∪ St , −1 ≤ |Sn | − |St | ≤ 1
2: Estimate ITE β̂i with E6 for each user i ∈ St .
3: Sort {βi }i∈S in ascending order
4: Split St = Sl ∪ Sh , such that −1 ≤ |Sl | − |Sh | ≤ 1 and
β̂i < β̂j ∀ i ∈ Sl , j ∈ Sh
5: Assign users in Sl and Sh to low- and high-targeted,
respectively
transitioned into Phase 2 on a weekly basis. That is, for a
particular week, all users who have reached 90 days of age in
Phase 1 form the current weekly cohort, which is randomly
split into a non-targeted group Sn and targeted group St of
equal size (ties are broken randomly). For each user in St ,
we calculate the ITE based on Phase 1 events. These ITEs
are then sorted in ascending order. The 50% of the largest
reducers (with the most negative ITEs) are defined to be the
low-targeted group Sl , whereas the other half is assigned to
high-targeted group Sh . This targeting scheme appears to be
a double-edged sword: On the one hand, the DRP pays less
money to large reducers and also achieves larger reductions
for previously small reducers, increasing the desired ratio.
On the other hand, previously large reducers now reduce
less (in response to smaller rewards) and previously small
reducers are paid more money for increased reductions,
thereby counteracting the desired goal. However, the latter
factors are dominated by the gains from the former ones, as
we show in Section VI-C.
B. Validation of Adaptive Targeting
Algorithm 2 exploits the fact that users are relatively price
inelastic (indeed Figure 8 only shows a weak negative slope
of the demand curve) to assign large incentives to low reducers (and small incentives to high reducers) to minimize the
total payout from DRP to users. The attentive reader might
wonder why the targeting criterion had been determined to be
the estimated ITE rather than any other criterion. Indeed, we
performed a targeting exercise to determine which criterion is
most suitable for maximizing ρt (26a). The idea is to assign
users into one of two targeted groups, based on one of the
following criteria estimated from Phase 1 responses:
• ITE

ITE normalized by average reward level received
Intercept of estimated individual demand curve
• Slope of estimated individual demand curve
Each of these four criteria are computed in kWh and %
values, and after a sufficient number of iterations it was
determined that the ITE β̂i indeed is the criterion that
maximizes ρt (26a).
•
•

C. Results of Adaptive Targeting
Table V reports the targeting metrics together with the
CATE by treatment group as well as the number of observations for targeted nt and non-targeted users nn . We restrict
our analysis to samples obtained after June 27, 2017, as we
observe larger effects of targeting in summer months.
Estimator
BL
RF

ρt
0.483
0.372

Targeting
ρn
0.421
0.287

Metrics for
CATEt
−0.071
−0.037

Phase 2
CATEn
−0.088
−0.056

nt
25438
25438

nn
26488
26488

TABLE V: Targeting Results for 2, 704 users between June 27, 2017 kWh
December 31, 2017. [ρ] = USD
, [CATE] = kWh.
kWh
RF predicts a difference of ρt − ρn ≈ 0.085 USD
, or an
increase of about 30% compared to the non-targeted scheme.
For BL, this increase is smaller (15%). However, due to
the biasedness of the BL (see Figure 6), the RF estimate is
more reliable. We can observe the tradeoff between smaller
reductions (indeed RF predicts a CATE for targeted users
that is 34% smaller compared to non-targeted users) and
a reduced average payout, which decreases by 85% (not
reported in Table V). The latter effect dominates the decrease
in net reductions, resulting in the 30% increase of the
reduction per reward ratio (26a).

VII. C ONCLUSION
We analyzed Residential Demand Response as a humanin-the-loop cyber-physical system that incentivizes users
to curtail electricity consumption during designated hours.
Utilizing data collected from a Randomized Controlled Trial
funded by the CEC and conducted by a Demand Response
provider in the San Francisco Bay Area, we estimated the
causal effect of hour-ahead price interventions on electricity
reduction. To the best of our knowledge, this is the first major
study to investigate DR on such short time scales.
We developed a non-experimental estimation framework
and benchmarked its estimates against those obtained from
an experimental Fixed-Effects Linear Regression Model. Importantly, the former does not depend on the existence of an
experimental control group to construct counterfactuals that
are necessary to estimate the treatment effect. Instead, we
employ off-the-shelf regression models to learn a consumption model on non-DR periods, which can then be used to
predict counterfactuals during DR hours of interest. We find
that the estimated treatment effects from both approaches
are close to each other. The estimated ATE is −0.10 kWh
(11%) per Demand Response event and household. Further,
we observe a weak positive correlation between the incentive
level and the estimated reductions, suggesting that users are
only weakly elastic in response to incentives.

R EFERENCES
[1] P. Palensky and D. Dietrich, “Demand Side Management: Demand
Response, Intelligent Energy Systems, and Smart Loads,” IEEE Transactions on Industrial Informatics, vol. 7, no. 3, pp. 381–388, 2011.
[2] Federal Energy Regulatory Commission, “Assessment of Demand
Response and Advanced Metering,” Tech. Rep., 2016.
[3] S. Borenstein, “The Long-Run Efficiency of Real-Time Electricity
Pricing,” The Energy Journal, 2005.
[4] “Public Utilities Commission of the State of California: Resolution
E-4728. Approval with Modifications to the Joint Utility Proposal for
a Demand Response Auction Mechanism Pilot,” July 2015.
[5] “California Independent System Operator Corporation (CAISO): Fifth
Replacement FERC Electric Tariff,” 2014.
[6] P. J. Diggle, P. Heagarty, K.-Y. Liang, and S. L. Zeger, Analysis of
Longitudinal Data. Oxford University Press, 2013, vol. 2.
[7] B. Bollinger and W. R. Hartmann, “Welfare Effects of Home Automation Technology with Dynamic Pricing,” Stanford University, Graduate
School of Business Research Papers, 2015.
[8] A. Abadie, A. Diamond, and J. Hainmueller, “Synthetic Control
Methods for Comparative Case Studies: Estimating the Effect of
California’s Tobacco Control Program,” Journal of the American
Statistical Association, vol. 105, no. 490, pp. 493–505, 2012.
[9] S. Athey and G. W. Imbens, “Recursive Partitioning for Heterogeneous
Causal Effects,” Proceedings of the National Academy of Sciences of
the United States of America, vol. 113, no. 27, pp. 7353–7360, 2016.
[10] K. Brodersen, F. Gallusser, J. Koehler, N. Remy, and S. Scott, “Inferring Causal Impact Using Bayesian Structural Time-Series Models,”
The Annals of Applied Statistics, vol. 9, no. 1, pp. 247–274, 2015.
[11] J. W. Taylor and P. E. Sharry, “Short-Term Load Forecasting Methods:
An Evaluation Based on European Data,” IEEE Transactions on Power
Systems, vol. 22, no. 4, pp. 2213–2219, 2007.
[12] T. Senjyu, H. Takara, K. Uezato, and T. Funabashi, “One-HourAhead Load Forecasting Using Neural Network,” IEEE Transactions
on Power Systems, vol. 17, no. 1, pp. 113–118, 2002.
[13] E. E. Elattar, J. Goulermas, and Q. H. Wu, “Electric Load Forecasting
Based on Locally Weighted Support Vector Regression,” IEEE Transactions on Systems, Man, and Cybernetics, vol. 40, no. 4, 2010.
[14] P. Mirowski, S. Chen, T. K. Ho, and C.-N. Yu, “Demand Forecasting
in Smart Grids,” Bell Labs Technical Journal, 2014.

[15] F. Chen, J. Dai, B. Wang, S. Sahu, M. Naphade, and C.-T. Lu, “Activity
Analysis Based on Low Sample Rate Smart Meters,” Proceedings
of the 17th ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining, pp. 240–248, 2011.
[16] A. Molina-Markham, P. Shenoy, K. Fu, E. Cecchet, and D. Irwin,
“Private Memoirs of a Smart Meter,” Proceedings of the 2nd ACM
Workshop on Embedded Sensing Systems for Energy-Efficiency in
Building, pp. 61–66, 2010.
[17] D. Zhou, M. Balandat, and C. Tomlin, “A Bayesian Perspective on
Residential Demand Response Using Smart Meter Data,” 54th Allerton
Conference on Communication, Control, and Computing, 2016.
[18] H. Allcott, “Rethinking Real-Time Electricity Pricing,” Resource and
Energy Economics, vol. 33, no. 4, pp. 820–842, 2011.
[19] K. K. Jessoe, D. L. Miller, and D. S. Rapson, “Can High-Frequency
Data and Non-Experimental Research Designs Recover Causal Effects?” Working Paper, 2015.
[20] “California Irrigation Management Information System,” 2017.
[21] T. Vincenty, “Geodetic Inverse Solution Between Antipodal Points,”
Tech. Rep., 1975.
[22] M. S. Nikulin, “Hellinger distance,” ”http://www.encyclopediaofmath.
org/index.php?title=Hellinger distance&oldid=16453”.
[23] D. B. Rubin, “Estimating Causal Effects of Treatments in Randomized
and Non-Randomized Studies,” Journal of Educational Psychology,
vol. 66, no. 5, pp. 688–701, 1974.
[24] J.-S. Pischke and J. D. Angrist, Mostly Harmless Econometrics, 1st ed.
Princeton University Press, 2009.
[25] P. R. Rosenbaum and D. B. Rubin, “The Central Role of the Propensity Score in Observational Studies for Causal Effects,” Biometrika,
vol. 70, no. 1, pp. 41–55, 1983.
[26] P. W. Holland, “Statistics and Causal Inference,” Journal of the
American Statistical Association, vol. 81, no. 396, pp. 945–960, 1986.
[27] T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical
Learning. Springer New York, 2009.
[28] L. Breiman, J. Friedman, C. Stone, and R. A. Olshen, “Classification
and Regression Trees,” CRC Press, 1984.
[29] M. Balandat, “New Tools for Econometric Analysis of HighFrequency Time Series Data - Application to Demand-Side Management in Electricity Markets,” University of California, Berkeley, PhD
Dissertation, 2016.
[30] B. Efron and R. J. Tibshirani, An Introduction to the Bootstrap. CRC
Press, 1994.

A PPENDIX
A. Summary Statistics
Figures 9-12 illustrate the distribution of the number of
DR events received among users with completed Phase 1, as
well as the total number of DR events broken out by hour
of the day, day of the week, and month of the year.
Number of Phase 1 DR Events Across Completed Phase 1 Users

2000

Number of Users

The fact that the estimates obtained from both approaches are close to each other is encouraging, as our
non-experimental framework permits to go a step further
compared to the experimental method in that it allows
for an estimation of individual treatment effects. From an
economic perspective, being able to differentiate low from
high responders allows for an adaptive targeting scheme,
whose goal is to minimize the total payout to users while
maximizing total reductions. We utilize this fact to achieve
an increase of the reduction-per-reward ratio of 30%.
Lastly, we emphasize that our non-experimental estimation
framework presented in this paper has to potential to generalize to similar human-in-the-loop cyber-physical systems
that require the incentivization of users to achieve a desired
objective. This is because our non-experimental framework,
whose techniques are general rather than specific to Demand
Response, admits results on an individual user level, which
could be of particular interest in the incentivization of users
in transportation or financial systems.
Future work includes the analysis of adversarial user
behavior (baseline gaming) and advanced effects including
peer and network effects influencing Demand Response.
Also, we intend to investigate the effect of moral suasion and
other non-monetary incentives on the reduction of electricity
consumption of residential households.

1500
1000
25.14
500
0

0

5

10

15

20

25

30

Number of Messages

Fig. 9: Distribution of Number of Phase 1 DR Events Across Users with
Completed Phase 1

B. Balance Checks
Table VI provides the balance metrics introduced in Section II-D.

Total Number of DR Dispatches by Hour of Day

Count

30000
20000
10000
0

10

11

12

13

14

15

16

17

18

19

20

21

22

23

Hour of Day

Fig. 10: Distribution of DR Events by Hour of the Day Across Users with
Completed Phase 1
Total Number of DR Dispatches in Phase 1 by Day of the Week
Number of Users

20000
15000
10000
5000
0

TABLE VI: Balance Checks for Users in Control and Treatment Group
Monday

Tuesday

Wednesday Thursday

Friday

Saturday

Sunday

Day of the Week

Fig. 11: Distribution of DR Events by Day of the Week Across Users with
Completed Phase 1

C. Fixed Effects Regression Tables
Tables VII-IX provide the results of the Fixed Effects
Regressions presented in Section V. The point estimates of
interest are printed in boldface and are accompanied by the
standard errors as well as their 95% confidence intervals. The
t-value of the regression gives rise to the p-value, where we
use (∗), (∗∗), (∗ ∗ ∗) to denote statistical significance at the
90%, 95%, 99% confidence level, respectively.
D. Comparison of Estimation Methods
Figure 8 visually compares the ATEs broken out by incentive level, and it can be seen that both methods produce similar estimates. Figure 13 does the same for month of the year.
Agreeing with intuition, the reductions are notably larger in
summer months compared to winter periods. Conditional on
the automation status, Table IX states that the reductions are
−0.331 and −0.103 kWh for automated and non-automated
users, respectively, compared to −0.332 and −0.121 kWh
calculated by the non-experimental case. These values are
close to each other. Lastly, no significant difference in the
magnitude of reductions can be found between encouraged
and non-encouraged users.
E. Correlation of Temperature and ITE
As mentioned in the previous subsection, larger reductions
are estimated in warm summer months. To test the hypothesis
Total Number of DR Dispatches by Month of Year

Count

40000
30000
20000
10000
0

Balance Metrics for Control and Treatment Group
Cohen’s D
Hellinger Dist. Diff. Mean
kWh, HoD = 13
0.023
0.002
0.022
kWh, HoD = 14
0.026
0.002
0.026
kWh, HoD = 15
0.017
0.002
0.017
kWh, HoD = 16
0.006
0.002
0.006
kWh, HoD = 17
0.002
0.002
0.002
kWh, HoD = 18
0.002
0.002
−0.007
kWh, HoD = 19
0.002
0.002
−0.010
kWh, HoD = 20
0.002
0.002
−0.008
air temp, HoD = 13
0.013
0.005
0.032
0.013
0.005
0.032
air temp, HoD = 14
air temp, HoD = 15
0.015
0.005
0.038
air temp, HoD = 16
0.015
0.005
0.039
0.016
0.004
0.041
air temp, HoD = 17
air temp, HoD = 18
0.016
0.003
0.041
air temp, HoD = 19
0.018
0.003
0.043
0.018
0.002
0.042
air temp, HoD = 20
# historical obs. (hours)
4.632
0.037
303.7

Jan

Feb

Mar

Apr

May

Jun

Jul

Aug

Sep

Oct

Nov

Dec

Month of Year

Fig. 12: Distribution of DR Events by Month of the Year Across Users with
Completed Phase 1

Effect of DR by Incentive Level on Electricity Consumption
Estimate
t-Value
95% Conf. Int.
p-value
(Std. Err.)
-0.006
is treatit
-2.100
[-0.013, 0]
0.047∗∗
(0.003)
0.879
88.89
[0.859, 0.900]
<0.001∗∗∗
BLit
(0.010)
0.0205
Tit
10.79
[0.017, 0.024]
<0.001∗∗∗
(0.002)
1(is DRit ,
-0.120
-8.532
[-0.148, -0.091]
<0.001∗∗∗
rit = 0.05)
(0.014)
-0.121
1(is DRit ,
-6.910
[-0.157, -0.085]
<0.001∗∗∗
rit = 0.25)
(0.018)
1(is DRit ,
-0.115
-7.369
[-0.147, -0.083]
<0.001∗∗∗
rit = 0.50)
(0.016)
1(is DRit ,
-0.124
-6.219
[-0.166, -0.083]
<0.001∗∗∗
rit = 1.00)
(0.020)
1(is DRit ,
-0.136
-12.95
[-0.157, -0.114]
<0.001∗∗∗
rit = 3.00)
(0.010)
Parameter

TABLE VII: Fixed Effect Regression Results by Incentive Level
Effect of DR by Month
Estimate
Parameter
(Std. Err.)
-0.007
is treatit
(0.003)
0.879
BLit
(0.016)
0.021
Tit
(0.006)
-0.041
1(is DRit ,
MoYit = 1)
(0.010)
1(is DRit ,
-0.022
MoYit = 2)
(0.009)
1(is DRit ,
-0.076
MoYit = 3)
(0.004)
1(is DRit ,
-0.062
MoYit = 4)
(0.004)
1(is DRit ,
-0.094
MoYit = 5)
(0.005)
1(is DRit ,
-0.155
MoYit = 6)
(0.008)
1(is DRit ,
-0.227
MoYit = 7)
(0.007)
1(is DRit ,
-0.159
MoYit = 8)
(0.008)
1(is DRit ,
-0.179
MoYit = 9)
(0.014)
1(is DRit ,
-0.029
MoYit = 11)
(0.009)
1(is DRit ,
-0.022
MoYit = 12)
(0.010)

of Year on Electricity Consumption
t-Value

95% Conf. Int.

p-value

-1.962

[-0.014, 0.001]

0.078∗

55.52

[0.844, 0.915]

<0.001∗∗∗

3.326

[0.007, 0.034]

0.008∗∗∗

-4.298

[-0.063, -0.020]

0.002∗∗∗

-2.571

[-0.041, -0.003]

0.028∗∗

-18.62

[-0.085, -0.067]

0.002∗∗∗

-15.14

[-0.071, -0.053]

<0.001∗∗∗

-20.07

[-0.104, -0.083]

<0.001∗∗∗

-20.25

[-0.172, -0.138]

<0.001∗∗∗

-32.68

[-0.242, -0.211]

<0.001∗∗∗

-19.39

[-0.177, -0.141]

<0.001∗∗∗

-9.071

[-0.218, -0.142]

<0.001∗∗∗

-3.055

[-0.050, -0.008]

0.012∗∗

-2.172

[-0.045, 0.001]

0.055∗

TABLE VIII: Fixed Effect Regression Results by Month of Year

Estimated Reductions [kWh] and 95% Confidence Intervals

Electricity Consumption
95% Conf. Int.

p-value

[-0.013, 0]

0.047∗∗

[0.859, 0.900]

<0.001∗∗∗

[0.017, 0.024]

<0.001∗∗∗

[-0.418, -0.243]

<0.001∗∗∗

[-0.132, -0.074]

<0.001∗∗∗

Reductions [kWh]

Effect of Home Automation on
Estimate
t-Value
Parameter
(Std. Err.)
-0.006
is treatit
-2.101
(0.003)
0.879
88.94
BLit
(0.010)
0.021
Tit
10.79
(0.002)
-0.331
1(is DRit ,
-7.800
is autoit )
(0.042)
1(is DRit ,
-0.103
-7.310
¬is autoit )
(0.014)

−0.10
−0.15

ITE Estimates
FE Regression

−0.20

1

2

3

4

5

6

7

8

9

10

11

12

Month of the Year

Fig. 13: CATEs by Month of Year with Confidence Intervals, Comparison
Fixed Effects Estimators and Non-Experimental Estimators
Correlation between Temperature and Individual Treatment Effects

TABLE IX: Fixed Effect Regression Results by Automation Status

0.0

Estimated ITE [kWh]

Effect of Automation Uptake Incentive on Electricity Consumption
Estimate
Parameter
t-Value
95% Conf. Int.
p-value
(Std. Err.)
-0.005
is encit
-1.422
[-0.012, 0.002]
0.168
(0.003)
-0.008
is nonencit
-2.485
[-0.015, -0.001]
0.021∗∗
(0.003)
0.9366
38.38
[0.886, 0.987]
<0.001∗∗∗
BLit
(0.024)
0.0206
Tit
10.794
[0.017, 0.024]
<0.001∗∗∗
(0.002)
1(is DRit ,
-0.121
-7.703
[-0.153, -0.088]
<0.001∗∗∗
is encit )
(0.016)
1(is DRit ,
-0.125
-8.304
[-0.156, -0.094]
<0.001∗∗∗
is nonencit )
(0.015)

0.00
−0.05

−0.5
−1.0
−1.5

Zero
Estimated Slope
5

10

15

20

25

30

35

40

Mean Temperature [◦C]

Fig. 14: Correlation between Average Ambient Air Temperature and ITEs.
CATT [kWh] by Location, RF

TABLE X: Fixed Effect Regression Results by Automation Uptake Encouragement

whether or not there exists such a correlation, Figure 14
scatter plots estimated ITEs as a function of the average
ambient air temperature observed during the relevant DR
events. We can notice a notable positive correlation of
ambient air temperature and the magnitude of reductions.
Indeed, a subsequent hypothesis test with the null being a
zero slope is rejected with a p-value of less than 1e − 9.
To support this notion, we marginalize ITEs for each ZIP
code to obtain the geographic distribution of CATEs by
location, see Figure 15, and it is visually striking that users
in coastal areas in California show smaller reductions than
users in the Central Valley, where the climate is hotter.

0
−10−2

−10−1

−100

Fig. 15: Correlation between Average Ambient Air Temperature and CATEs.

