Slow select Queries on Log shipping standby server
Statistics in read only databases.

TL;DR

In Read only databases, regular permanent statistics can be leveraged to satisfy query plans. If sql server needs new statistics for the read only db  or these permanent statistics in the database have become stale, then temporary statistics can be created / updated. These statistics reside in TempDB and are managed by SQL Server (you can only drop them).

There are two kinds of temporary statistics, these that get created due to missing statistics and those that get 'updated'.  

The creation of temporary statistics can be removed by adding the statistics manually on the primary database, or generating the estimated execution plans of the reporting queries (also on the primary instance/db). More on that further in this answer.

Permanent statistics can be updated and 'transformed' to temporary statistics on a read only database.

The temporary stat updates can be resolved by either updating your statistics on the primary db more often or disabling the auto stat updates on the read only db after log restores are done. 

To disable the auto update stats you could execute this on the read only database: ALTER DATABASE [Database] SET AUTO_UPDATE_STATISTICS OFF; This will stop the updates of temporary statistics on this Read only database.

Another important part regarding Restores to a standby database &amp; temporary statistics is that when applying log backups, the temporary statistics get  updated again, even though they still  exist in sys.stats. 

In your case explaining the daily issues with Statman queries (recalculating the temp statistics each day if they need to be created / updated).



Norecovery --> Standby --> Norecovery ... removes temporary statistics

Another interesting part regarding temporary statistics is that they will be gone when the db state changes to restoring using RESTORE DATABASE ... WITH NORECOVERY.

use MASTER
GO
RESTORE DATABASE [ReadOnly2] with NORECOVERY
RESTORE DATABASE [ReadOnly2] WITH STANDBY = 'D:\temp\ReadOnly_Standby.bak'


Effectively flushing the temporary statistics on all objects 

SELECT * From sys.stats where is_temporary = 1;


And recalculating the same 2 stats between each state change + test query running.



Both of these permanent statistics show up in the xml of our execution plan

   &lt;StatisticsInfo Database="[ReadOnly2]" Schema="[dbo]" Table="[Bla]" Statistics="[IX_Bla_indexedval]" ModificationCount="12000000" SamplingPercent="15.8812" LastUpdate="2019-06-12T10:52:32.25" /&gt;
      &lt;StatisticsInfo Database="[ReadOnly2]" Schema="[dbo]" Table="[Bla]" Statistics="[PK__Bla__3214EC075017BD54]" ModificationCount="12000000" SamplingPercent="15.2345" LastUpdate="2019-06-12T10:52:35.34" /&gt;


With the modificationcount, samplingpercent &amp; lastupdate changed after executing the query again with the temporary statistics 'updated'.

ModificationCount="0" SamplingPercent="5.71018" LastUpdate="2019-06-13T11:32:36.5" 




Creation of temporary statistics

Regular, non temporary statistics do not get updated / you cannot update (not even temporary) statistics  on read only databases. 

What you are seeing on the reporting instance is the creation / 'updating' of temporary statistics. 

These statistics reside in TempDB &amp; SQL Server Creates &amp; Updates them.

Replicating the behaviour

I was able to replicate the behaviour of temporary statistics creation on a table with 100M rows in a read only database (Addendum #1)


With the troublesome StatMan Queries you mentioned.

SELECT StatMan([SC0], [SB0000]) FROM (SELECT TOP 100 PERCENT [SC0], step_direction([SC0]) over (order by NULL) AS [SB0000]  FROM (SELECT [NonIndexedVal] AS [SC0] FROM [dbo].[Bla] TABLESAMPLE SYSTEM (7.707678e-001 PERCENT) WITH (READUNCOMMITTED) ) AS _MS_UPDSTATS_TBL_HELPER ORDER BY [SC0], [SB0000] ) AS _MS_UPDSTATS_TBL  OPTION (MAXDOP 16)


Maxdop 16 (max cores) due to me having MAXDOP set as 0 on my test machine, YMMV

Restarting the instance

When I restart the instance I see the same behaviour, showing that these are in fact, temporary statistics. Some Q&amp;A on temporary statistics can be found here.

Now what we are seeing is the creation of temporary statistics and they get created before the query runs.

For big tables, they do note that sample rate is still applied to temporary statistics.


  Other point to note is that statistics created as part of auto-stats
  use data sampling so the creation of these statistics is fast and does
  not depend on the size of the table


Source



How could you resolve the creation of the statistics when sql server restarts / the database is restored?

If it is possible to do so, you could script them out and create them on the main 'primary' database.

Finding the temporary statistics

SELECT OBJECT_ID, name, auto_created,
user_created, is_temporary
FROM sys.stats
WHERE is_temporary = 1;


Scripting the statistics

Without T-SQL
Or by using the T-SQL answer on the question How to script statistics in Sql Server? (using T-SQL) By Martin Smith

Which could solve your main issue with the statistics being created time and time again.

Another idea if the issue is due to temp stats being created &amp; you cannot script them out correctly, could be creating the estimated execution plans for the reporting queries on the primary database. This should create the statistics needed when auto create statistics = on.



Temporary statistic updates

The other problem that could arise are stale permanent statistics.
As noted in this aforementioned blog, stale permanent statistics can be updated and set to is_temporary=1. 

This means that permanent statistics on a read only database can become temporary statistics until the instance is restarted. When you update the statistics on the primary it should get carried over to the secondary when the log is applied.



Asynchronous temporary statistic updates

We see the asynchronous statistic updates also working for these temporary statistics!

After running addendum #1, we run the next snippet:

   USE MASTER
GO
ALTER DATABASE [ReadOnly] SET  READ_WRITE;

ALTER DATABASE [ReadOnly] SET AUTO_UPDATE_STATISTICS OFF 

USE [ReadOnly]
GO
INSERT INTO dbo.Bla WITH(TABLOCK)(Indexedval,NonIndexedVal) 
SELECT TOP(10000000) --10M
        ROW_NUMBER() OVER(ORDER BY (SELECT NULL)) as rownum1,
        ROW_NUMBER() OVER(ORDER BY (SELECT NULL)) as rownum2
FROM master..spt_values spt1
CROSS JOIN master..spt_values spt2
CROSS JOIN master..spt_values spt3;

ALTER DATABASE [ReadOnly] SET AUTO_UPDATE_STATISTICS ON 

USE MASTER
GO
ALTER DATABASE [ReadOnly] SET  READ_ONLY;
SELECT Indexedval 
FROM dbo.Bla
WHERE  Indexedval =999999
AND 1= (SELECT 1); 


The above SELECT runs instantly.

After running the query, the aftermath is shown.



The temp statistic updates run after query execution.

SELECT StatMan([SC0]) FROM (SELECT TOP 100 PERCENT [Indexedval] AS [SC0] FROM [dbo].[Bla] WITH (READUNCOMMITTED)  ORDER BY [SC0] ) AS _MS_UPDSTATS_TBL  OPTION (MAXDOP 1)




Disabling auto update statistics on read only databases

You could disable the updating of permanent statistics to temporary statistics by running the next statement, you can change this setting on a read only database and it will still work.

ALTER DATABASE [ReadOnly] SET AUTO_UPDATE_STATISTICS OFF;


Solution for the temporary stat upgrades.


  Also, will the update stats done on primary (once a week) be restored
  during LS restore?


Updating your stats will be brought over to the secondary, updating them more where possible will result in less stale statistics and less stale permanent statistics being converted to temporary statistics. 

Looking into both of these would be my go to in tackling this problem. 



Disabling temporary statistics by using a traceflag

When massively enabling traceflags to help a colleague and digging deeper between 2 and 3 thousand, it appears that traceflag 2362 can be used to disable temporary statistics.

You can enable them like this:

DBCC TRACEON(2362,-1);


And all newly temp stats will not be created. Existing temporary statistics will remain until they are removed. For example by setting the db offline and online again.



Applying log backups to a standby database &amp; Temporary statistics

Addendum 2

When running the queries in addendum #2, and applying log backups to the standby database the stats get updated after each restore. 

Even after applying an 'empty' log backup.

In addendum 2, between each log backup restore the following query runs:

SELECT Indexedval 
FROM dbo.Bla
WHERE  Indexedval =999999
AND 1= (SELECT 1); 


These trigger temp stat updates each time.

Proof




All this means that applying logs at night will make the temp stat updates run each day, without a restart of the instance.

Resolving this issue


set auto update stats off on the standby db after applying logs. ALTER DATABASE [ReadOnly2] SET AUTO_UPDATE_STATISTICS OFF;
Run your reporting queries when the restores finish to create the temp stats
Update the statistics on your primary database more often before
applying the logs.
Enabling traceflag 2362


When restoring a log backup they still exist:

SELECT name, is_temporary From sys.stats where is_temporary = 1;

name    is_temporary
PK__Bla__3214EC075017BD54   1
IX_Bla_indexedval   1


But they do get recalculated when rerunning the query.




Addendum #1 (Table with 100M records in a read only db.)

CREATE DATABASE [ReadOnly]
 CONTAINMENT = NONE
 ON  PRIMARY 
( NAME = N'ReadOnly', FILENAME = N'C:\Program Files\Microsoft SQL Server\MSSQL14.SQL02\MSSQL\DATA\ReadOnly.mdf' , SIZE = 8192KB , FILEGROWTH = 65536KB )
 LOG ON 
( NAME = N'ReadOnly_log', FILENAME = N'C:\Program Files\Microsoft SQL Server\MSSQL14.SQL02\MSSQL\DATA\ReadOnly_log.ldf' , SIZE = 8192KB , FILEGROWTH = 65536KB )
GO
ALTER DATABASE [ReadOnly] SET COMPATIBILITY_LEVEL = 140
ALTER DATABASE [ReadOnly] SET AUTO_CREATE_STATISTICS ON(INCREMENTAL = OFF)
ALTER DATABASE [ReadOnly] SET AUTO_UPDATE_STATISTICS ON 
ALTER DATABASE [ReadOnly] SET AUTO_UPDATE_STATISTICS_ASYNC ON 
ALTER DATABASE [ReadOnly] SET  READ_WRITE 
ALTER DATABASE [ReadOnly] SET RECOVERY SIMPLE 
ALTER DATABASE [ReadOnly] SET  MULTI_USER 
ALTER DATABASE [ReadOnly] SET PAGE_VERIFY CHECKSUM  

USE [ReadOnly]
GO
CREATE TABLE dbo.Bla(Id INT IDENTITY(1,1) PRIMARY KEY NOT NULL, Indexedval INT,NonIndexedVal INT);
CREATE INDEX IX_Bla_indexedval on dbo.Bla(Indexedval);

INSERT INTO dbo.Bla WITH(TABLOCK)(Indexedval,NonIndexedVal) 
SELECT TOP(10000000) --10M
        ROW_NUMBER() OVER(ORDER BY (SELECT NULL)) as rownum1,
        ROW_NUMBER() OVER(ORDER BY (SELECT NULL)) as rownum2
FROM master..spt_values spt1
CROSS JOIN master..spt_values spt2
CROSS JOIN master..spt_values spt3;
GO 10

USE MASTER
GO
ALTER DATABASE [ReadOnly] SET  READ_ONLY;

USE [ReadOnly]
GO
SELECT NonIndexedVal 
FROM dbo.Bla
WHERE  NonIndexedVal = 999999;


#Addendum 2 

ALTER DATABASE [ReadOnly] SET  READ_WRITE;

ALTER DATABASE [ReadOnly] SET RECOVERY FULL

BACKUP DATABASE [ReadOnly] to disk = 'D:\temp\ReadOnly.bak'
WITH COMPRESSION, STATS=5


RESTORE FILELISTONLY FROM DISK = 'D:\temp\ReadOnly.bak'

RESTORE DATABASE [ReadOnly2] FROM disk = 'D:\temp\ReadOnly.bak'
WITH MOVE  'ReadOnly' to 'D:\temp\ReadOnly2.mdf'
,MOVE 'ReadOnly_log' to 'F:\temp\ReadOnly_log2.ldf'
, STANDBY = 'D:\temp\ReadOnly_Standby.bak'





USE [ReadOnly2]
GO
ALTER DATABASE [ReadOnly2] SET AUTO_UPDATE_STATISTICS ON

SELECT Indexedval 
FROM dbo.Bla
WHERE  Indexedval =999999
AND 1= (SELECT 1); 


USE [ReadOnly]

INSERT INTO dbo.Bla WITH(TABLOCK)(Indexedval,NonIndexedVal) 
SELECT TOP(2000000) --2M
        ROW_NUMBER() OVER(ORDER BY (SELECT NULL)) as rownum1,
        ROW_NUMBER() OVER(ORDER BY (SELECT NULL)) as rownum2
FROM master..spt_values spt1
CROSS JOIN master..spt_values spt2
CROSS JOIN master..spt_values spt3;


USE MASTER
GO
BACKUP LOG [ReadOnly] to disk = 'D:\temp\ReadOnlyLog.trn'
WITH COMPRESSION, STATS=5


RESTORE LOG [ReadOnly2] FROM DISK='D:\temp\ReadOnlyLog.trn'
WITH STANDBY = 'D:\temp\ReadOnly_Standby.bak'

USE [ReadOnly2]
SELECT Indexedval 
FROM dbo.Bla
WHERE  Indexedval =999999
AND 1= (SELECT 1); 

BACKUP LOG [ReadOnly] to disk = 'D:\temp\ReadOnlyLog2.trn'
WITH COMPRESSION, STATS=5


RESTORE LOG [ReadOnly2] FROM DISK='D:\temp\ReadOnlyLog2.trn'
WITH STANDBY = 'D:\temp\ReadOnly_Standby.bak'

USE [ReadOnly2]
SELECT Indexedval 
FROM dbo.Bla
WHERE  Indexedval =999999
AND 1= (SELECT 1); 
SELECT * From sys.stats where is_temporary = 1