Command line

I have a test database called TPCH, and these are the scripts to perform a full backup without compression, and one with compression:

--Backup without compression 
BACKUP DATABASE [TPCH] TO DISK = 'c:\SQLBackup\TPCH_uncompressed.bak'
GO 

--Backup with compression 
BACKUP DATABASE [TPCH] TO DISK = 'c:\SQLBackup\TPCH_compressed.bak' 
WITH COMPRESSION
GO 

To see the difference, I can look in msdb.dbo.backupset, which shows you both the backup_size without compression, as well as the compressed_backup_size:

select 
database_name, 
backup_size, 
compressed_backup_size,
 (backup_size-compressed_backup_size)/backup_size*100 AS SavingsInPercent 
FROM msdb.dbo.backupset 

The output from this is:

My first backup show the same value in both backup_size and compressed_backup_size columns, and that is because bacup compression wasn’t enabled. The second one shows two different values, and a calculated saving of 58%. This value varies depending on the content of the database, but it’s not uncommon to see savings in the 80-85% area. Not bad, considering the small effort for enabling it

GUI

If you are using the GUI to take a manual backup, you will see the standard backup windows. If you go to the Options pane, you will see this:

Notice the compression part in the bottom. The default setting is “Use the default server setting”, which is “Do not compress backup”. You can select the “Compress backup”, and achieve the same result as with the “WITH COMPRESSION” in the t-sql script. But hey! If there is a “use the default server setting” option, then how can we change that? Simply right click the sql server, go to properties and open up the “Database Settings” tab:

Here you see the “Compress backup” checkbox. If you tick this, the backup’s will automatically be compressed, unless you specifically choose not to perform compression. Pretty nifty!

IBM TDP for SQL Server

If you are using IBM Tivoli Data Protection for SQL Server to perform your database backups, the default server setting above is unfortunately ignored. But if you are running TDP 5.5.4 or newer, you can actually enable it anyways. TDP for SQL Server ususally has the following root directory: c:\progra~1\Tivoli\TSM\TDPSql\ and in here you find TDPSql.cfg. In this file you simply add this line to the bottom:

SQLCOMPression          YES

Now restart your scheduler service, and you are good to go. If you would like to verify your own savings after enabling this, you can use the script from before.

Benefits

There are a few things worth mentioning. If you enable backup compression, you put extra load on the SQL Server cpu, because it needs to perform the compression. But at the same time you offload your network, because you only need to transfer 15-50% of your usual data volume. And with smaller backups you have room to hold more backups on your storage system.

Depending on your setup, the backup and restore speed will probably be a lot better as well. The backups will be faster because you need to transfer and write less data to your destination. And restores will be faster because you need to read and transfer less data from your backup source.

After testing, if you are unable to receive notification emails, you could use the below Database Mail views for troubleshooting. This first one contains one single row for each message successfully sent by Database Mail.

select * from msdb.dbo.sysmail_sentitems 

This next is the complete opposite, here we’ll se a row for each message with an unsent or retrying status.

select * from msdb.dbo.sysmail_unsentitems 

And finally here a view that shows one row for each Database mail message with status failed

select * from msdb.dbo.sysmail_faileditems 

Personally I’m not a big fan of the GUI, if You share that passion with me, here is a view that will give You the Database Mail event log.

select * from msdb.dbo.sysmail_event_log 

And finally, here is the view that shows it all.

select * from msdb.dbo.sysmail_allitems 

Database Mail has two queues, a status queue and a mail queue. The mail queue tells us the number of mail in queue waiting to be sent. The status queue stores the status of items that have already been sent. Here is a Stored Procedure that can show You the status of the two queues.

exec msdb.dbo.sysmail_help_queue_sp; exec msdb.dbo.sysmail_help_queue_sp @queue_type = 'mail'; exec msdb.dbo.sysmail_help_queue_sp @queue_type = 'status'; 

Did this not help You? Well here is a link to the official Microsoft documentation, that might help – happy troubleshooting.

What some times can be a challenge is to have access to the same SMTP server again and again, and especially when I am testing something with Database Mail, SQL Server alerts, Operators or something else that uses a SMTP server from SQL Server to send out an email.

Because of these challenges I normally use a Gmail account to send mails from, Gmail requires authentication to send mails via their SMTP servers, so an account type like this can be used from anywhere.

So many people have blogged about this before me, including myself actually – I blogged some thing like this a few years back – in DANISH. I always struggle to find my script to configure these things, so that is why I am blogging about this again – to refresh my brain, and to share my code with You.

Before we can start playing with Database mail, we have to enable the feature, that is done with the following code:

EXEC sp_Configure 'Database Mail XPs', 1
GO

RECONFIGURE
GO

With the feature enabled, all there is left to do before sending mails from SQL Server, is to changes the following script so that is takes you account info, and not the bogus info that I have entered:

-- Create a Database Mail account
EXEC msdb.dbo.sysmail_add_account_sp
      @account_name = 'MyMailAccount'
    , @email_address = 'me@gmail.com'
    , @display_name = 'My Servers Database Mail Account'
    , @description = 'MyMailAccount'
    , @mailserver_name = 'smtp.gmail.com'
    , @port = 587
    , @username = 'me@gmail.com'
    , @password = 'mYpASSWORD'
    , @enable_ssl = 1

-- Create a Database Mail profile
EXECUTE msdb.dbo.sysmail_add_profile_sp
    @profile_name = 'MyPublicProfile',
    @description = 'My Public Profile';

-- Add the account to the profile
EXECUTE msdb.dbo.sysmail_add_profileaccount_sp
    @profile_name = 'MyPublicProfile',
    @account_name = 'MyMailAccount',
    @sequence_number = 1;

-- Configuring global profile, and setting default mail profile
EXECUTE msdb.dbo.sysmail_add_principalprofile_sp
    @profile_name = 'MyPublicProfile',
    @principal_name = 'public',
    @is_default = 1;
GO

After execution this script, we are ready to send out an test mail, to check that everything is working, here is the code to do that:

EXECUTE msdb.dbo.sp_send_dbmail
    @subject = 'Test Database Mail Message',
    @recipients = 'me@myemail.com',
    @query = 'set nocount on;SELECT Getdate() as ServerTime';

Wait one second and you should receive an email from you SQL Server. In the next blog post we’ll look at how to monitor and debug Database mail – stay tuned for that next week.

Before I finish this blog post totally, I’ll just show you a few functions and views that you can use to look at the profiles and accounts created. And another god thing to have handy is how to delete the account/profile just created.

exec msdb.dbo.sysmail_delete_account_sp @account_name = 'MyMailAccount'
exec msdb.dbo.sysmail_delete_profile_sp @profile_name = 'MyPublicProfile'

select * from msdb.dbo.sysmail_account
select * from msdb.dbo.sysmail_profile

One of the features that mirroring gives us, almost never make it to our discussions. And since this feature is actually pretty nice and powerful, I thought it was worth sharing – and perhaps we will remember to take it into account the next time we battle about HA
The feature is called “Automatic Page Repair”, and is documented here: http://msdn.microsoft.com/en-us/library/bb677167.aspx

If a page corruption has been detected on the prinical server, it can automatically read that specific page from the mirror database and fix the principal. How cool is that?!” Using a failover cluster, the database files are only located in one location, and if something happens with the underlying storage system corrupting your data, the only way of fixing it is to restore a backup as soon as possible after you identify the corruption. No need to say, that this is not optimal.

Demo

To show you how this works, I have created a mirror between two local instances on my computer. I won’t go into details about the creation of the mirror, but now I have this on my computer:

The mirrored database is MyData, and sqlmaster\sql2008 instance is currently the principal.

I created a dummy table using this script:

use MyData go create table TestData (id int identity, col1 char(2000)) go insert into TestData (col1) values (replicate('a', 2000)) go 4096

Now I will simulate a disk failure on the sqlmaster\sql2008 server, so I stop the service and open the .mdf file in a hex editor. Somewhere down the file, this show up:

All the a’s are the data column, and the part in the middle is probably a page header. The details are not important for this test, but I’ll replace a bunch of the data with zeros:

Save the file, and start up the sql server.

Now I perform a table scan by simply issuing a select * from… and this is what happens:

Msg 824, Level 24, State 2, Line 1 SQL Server detected a logical consistency-based I/O error: incorrect checksum (expected: 0xb4f55a1a; actual: 0xa4a264f1). It occurred during a read of page (1:840) in database ID 5 at offset 0×00000000690000 in file ‘C:\DemoData\MyData.mdf’. Additional messages in the SQL Server error log or system event log may provide more detail. This is a severe error condition that threatens database integrity and must be corrected immediately. Complete a full database consistency check (DBCC CHECKDB). This error can be caused by many factors; for more information, see SQL Server Books Online.

That is a message we normally don’t want to see. It states that page 840 is corrupt, but because the database is part of a mirror, some cool magic has happened. If I run the select again, I get a similar error – but this time for page 841. And third time I execute the select *, it succeeds! What has happened is the “Automatic Page Repair” mechanism kicking in. SQL Server provides a DMV called sys.dm_db_mirroring_auto_page_repair to see if repairs like that have happened, so let’s have a look at that:

SELECT DB_NAME(database_id) as DatabaseName, * FROM sys.dm_db_mirroring_auto_page_repair; 

Here we can actually see that page_id 840 and 841 were repaired automatically.
The automatic page repair feature does not prevent an error to occur, but when a query tries to access a corrupt page, it will be repaired automatically so it is ready to use for the next query. So the impact of disk errors will be limited.
If you are using mirroring in your setup, try to have a look at this DMV. If a lot of errors show up here, it might be an indication of bad disks, and you should take action to fix it.

So if you find yourself needing arguments in favor of mirroring, this is a feature worth remembering.

The blog post from last week will be the basic setup for this one, so the steps to create the table and to collect the data will not be repeated in this one. If you need them you’ll have to go back and get it.

Today I’ll encapsulate the TSQL logic in a view, this is not necessary, but it makes it much easier from Excel to connect to a predefined view rather than having to write the TSQL. I will call the view for V_WaitStats, and here is the code to create it:

CREATE VIEW V_WaitStats
AS

WITH cte AS
(
SELECT
    id, wait_type, wait_time_ms, SampleTime,
    ROW_NUMBER() OVER(PARTITION BY wait_type ORDER BY SampleTime) as rn
FROM InstanceWideWaitStats
)
SELECT
 t1.wait_type,
 t2.wait_time_ms - t1.wait_time_ms as WaitTimeInMS,
 t2.SampleTime
FROM cte t1
INNER JOIN cte t2 on (t1.wait_type = t2.wait_type) and (t1.rn = t2.rn -1)
WHERE t2.wait_time_ms - t1.wait_time_ms <> 0

What I want to do next is connect to my SQL Server View from Excel, I am no way an Excel Guru, but this is so easy, and such a powerful tool at visualize data like wait stats. Here is some screenshots from the process.

First step is to enter the name of the SQL Server where the data is stored

Then is it time to select database and the View that we created earlier in this blog post. And the in the end it is just a matter of choosing the right option.

And then just hit the OK button. Then it is time to show some Pivot Table magic, Excel gives us so many opportunities to play around with the Wait Stats data, further down you’ll find some examples that shows some of them. Selecting a subset of wait types or periods is easy with Excel, and the Graph changes straight away.

I use this quite often, and the information that You get with this is very useful. This solution that we have described in these two blog posts, should be in every DBA’s toolbox.

One of the first things that I always start looking at is wait stats, by looking at a systems wait statistics I get a quick idea about where to start looking next – is it IO (Slow IO Sub system or missing indexes), is it memory or something complete different. To do this I use sys.dm_os_wait_stats, this gives me instance wide wait stats, accumulated since SQL Server was last restarted or since the stats was cleared. You can read more about the sys.dm_os_wait_stats DMV here.

Looking at the raw output from sys.dm_os_wait_stats doesn’t give you much to work with, you need to apply some kind of black magic. I am not going to take the credits for the following script, it comes from a blog that belongs to guy called Glen Berry. Glen publishes a collection of scripts every months that are very useful. Here is the magic:

WITH Waits AS (
SELECT wait_type, wait_time_ms / 1000. AS wait_time_s,
    100. * wait_time_ms / SUM(wait_time_ms) OVER() AS pct,
    ROW_NUMBER() OVER(ORDER BY wait_time_ms DESC) AS rn
 FROM sys.dm_os_wait_stats
 WHERE wait_type NOT IN('KSOURCE_WAKEUP', 'SLEEP_BPOOL_FLUSH', 'BROKER_TASK_STOP',
 'XE_TIMER_EVENT', 'XE_DISPATCHER_WAIT', 'FT_IFTS_SCHEDULER_IDLE_WAIT',
 'SQLTRACE_BUFFER_FLUSH', 'CLR_AUTO_EVENT', 'BROKER_EVENTHANDLER', 'LAZYWRITER_SLEEP',
 'BAD_PAGE_PROCESS', 'BROKER_TRANSMITTER', 'CHECKPOINT_QUEUE', 'DBMIRROR_EVENTS_QUEUE',
 'LAZYWRITER_SLEEP', 'ONDEMAND_TASK_QUEUE', 'REQUEST_FOR_DEADLOCK_SEARCH', 'LOGMGR_QUEUE',
 'SLEEP_TASK', 'SQLTRACE_BUFFER_FLUSH', 'CLR_MANUAL_EVENT', 'BROKER_RECEIVE_WAITFOR',
 'PREEMPTIVE_OS_GETPROCADDRESS', 'PREEMPTIVE_OS_AUTHENTICATIONOPS', 'BROKER_TO_FLUSH')
 )
SELECT W1.wait_type,
  CAST(W1.wait_time_s AS DECIMAL(12, 2)) AS wait_time_s,
  CAST(W1.pct AS DECIMAL(12, 2)) AS pct,
  CAST(SUM(W2.pct) AS DECIMAL(12, 2)) AS running_pct
FROM Waits AS W1
INNER JOIN Waits AS W2 ON W2.rn <= W1.rn
GROUP BY W1.rn, W1.wait_type, W1.wait_time_s, W1.pct
HAVING SUM(W2.pct) - W1.pct < 95;

and if we take a look at the output that this script gives on my no-action laptop, it look like this:

the wait types that you see here is more or less useless, there is no SQL Server activity on my laptop, but try running the script on your system, and hopefully you’ll se something way different.

To get more info about Wait Types, and what they mean to your system, here is an excellent link that explains that in detail.

Wait stats over time

The above example show how you can look at wait stats here and now, but how about the user that calls You Monday morning and complains about poor performance during the weekend – the above script is not really going to help you with that. I a situation like that is would be really nice with wait stat information from the weekend, even better if they could be broken down into small timeslots. If you could see the change in the wait stats between 11:00 and 11:15 that might give you an idea where to start to look for the problems.

Lets try to build a solution that makes such information available for you. First off all, I will create a table to store the sample data in.

Create table InstanceWideWaitStats
(
    id bigint identity(1,1) primary key,
    wait_type nvarchar(120),
    waiting_tasks_count    bigint,
    wait_time_ms bigint,
    max_wait_time_ms bigint,
    signal_wait_time_ms    bigint,
    SampleTime datetime2 default sysdatetime()
)

Now let’s write some TSQL that writes some data into this table:

insert into InstanceWideWaitStats (wait_type, waiting_tasks_count, wait_time_ms, max_wait_time_ms, signal_wait_time_ms)
select
    wait_type, waiting_tasks_count, wait_time_ms, max_wait_time_ms, signal_wait_time_ms
from sys.dm_os_wait_stats

This code is super simple, the only thing that I actually do is to sample the DMV that I mentioned earlier in this blog post. What I normally do it the schedule this to run every 15 minutes with the SQL Server Agent. When that is done we are ready to have a look at the data.

The information that I am interested in is the column called wait_time_ms – the same column from the last sample that we did. This gives me the amount of wait time of a given Wait Type between two samples, which is 15 minutes if You run the job every 15 minutes. Let’s have a look at the TSQL:

with cte as
(
select
    id, wait_type, wait_time_ms, SampleTime,
    ROW_NUMBER() OVER(PArtition by wait_type order by SampleTime) as rn
from InstanceWideWaitStats
)
select
 t1.wait_type,
 t2.wait_time_ms - t1.wait_time_ms as WaitTimeInMS,
 t2.SampleTime
from cte t1
inner join cte t2 on (t1.wait_type = t2.wait_type) and (t1.rn = t2.rn -1)
where t2.wait_time_ms - t1.wait_time_ms <> 0

This is SQL Server 2008 syntax, SQL Server 2012 comes with a new feature called LAG / LEAD. If you are running 2012 your code would be much more simple, and could look something like this:

SELECT
 wait_type,
 SampleTime,
 wait_time_ms - LAG(wait_time_ms, 1, null) OVER(PArtition by wait_type order by SampleTime)
FROM InstanceWideWaitStats
where wait_type = 'BROKER_TO_FLUSH'

In a real world examples you would consider to have a where clauses or something on the SampleTime column, to avoid to get to much data and only to look at the period you need. Let’s have a look at the output from the 2008 syntax query:

wuupa, here goes the magic – with this little simple setup we are now able to look at the wait stats and how they looked earlier. As mentioned earlier, this is not a silver bullet, this is just a very good place to start looking when you have performance issues with your SQL Server.

In the next blog post I’ll demonstrate how you can visualize this data, and make it quite nice and easy to look at with Excel.

Some of you may have seen this “numbers table” technique in use before, but I thought I would share a few simpel examples to those of you that haven’t seen the light yet

Lets start with a classic problem: Converting a delimited string into a table.

In many old setups I have seen, it’s not uncommon to have the need for passing multiple values to a stored procedure, to look up data for multiple values in one go. Let me build my demo table, and show how such a stored procedure could look like:

CREATE TABLE MyFavoriteMovies
(
    MovieId INT IDENTITY(1,1) PRIMARY KEY,
    MovieTitle VARCHAR(100),
    NumberOfViews INT
)
GO
INSERT INTO MyFavoriteMovies (MovieTitle, NumberOfViews)
VALUES
    ('Gone with the Wind', 23),
    ('Driving Miss Daisy', 9),
    ('Martha', 16),
    ('Fætrene på Torndal', 33),
    ('Casablanca', 8),
    ('Dumb & Dumber', 6),
    ('Wargames', 24),
    ('Geniiius, the movie', NULL)
GO

CREATE PROCEDURE FetchMovieTitles
@MovieIds VARCHAR(1000)
AS
DECLARE @sqlcmd VARCHAR(MAX) = '
    SELECT * FROM MyFavoriteMovies
    WHERE MovieId IN (' + @MovieIds + ')'
EXEC (@sqlcmd)
GO

I created a table to hold a list of my favorite movies, and then I have a stored procedure i can execute, that takes a comma separated list of MovieId’s as parameter. It’s not pretty, I know, but nevertheless it’s a template I have seen numerous times over the time.

So why is this not ideal?

The obvious reason is, that because the stored procedure generates the sql command dynamically, and then executes it without any check whatsoever, it’s open for sql injections. On top of that security issue, it also generates a new execution plan for every different parameter that the procedure is executed with. If my query was more complex in the procedure, the dynamic sql could easily becomre nasty to work with. Think about the single quote which serves both as the escape character but also surrounds strings. An example of this could look like this:

DECLARE @sqlcmd VARCHAR(MAX) = '
    SELECT * FROM MyFavoriteMovies
    WHERE MovieId IN (' + @MovieIds + ')
    AND MovieTitle = ''Wargames'''

Ugly, right?

So how can we fix this you ask? Imaging we could convert the list of MovieId’s to a table, then the query could look something like this:

SELECT * FROM MyFavoriteMovies
WHERE MovieId IN (SELECT MovieId FROM #MovieIds)
AND MovieTitle = 'Wargames'

Now we got rid of the dynamic sql, and it’s much prettier to read. But we don’t have the #MovieIds table yet, so we need to find a way to convert our delimited string to a table.

Let’s start to build the solution up, by looking at this:

DECLARE @MovieIds VARCHAR(1000) = '1,3,5'
SELECT SUBSTRING(@MovieIds, 1, len(@MovieIds))
SELECT SUBSTRING(@MovieIds, 2, len(@MovieIds))
SELECT SUBSTRING(@MovieIds, 3, len(@MovieIds))
SELECT SUBSTRING(@MovieIds, 4, len(@MovieIds))
SELECT SUBSTRING(@MovieIds, 5, len(@MovieIds))

Here I have five select statements, which performs a substring on the variable, but with an increasing offset going from 1 to 5. Notice that the length of the @MovieIds string is exactly 5 characters. Does this get my any closer to the solution? Can you spot the pattern? The first character in the first, third and fifth result is actually the values I’m looking for. But how could I identify these? What if we instead of selecting the rest of the string in each of the substring command, could stop at the next comma?

DECLARE @MovieIds VARCHAR(1000) = '1,3,5'
SELECT SUBSTRING(@MovieIds, 1, CHARINDEX(',', @MovieIds+',', 1) - 1)
SELECT SUBSTRING(@MovieIds, 2, CHARINDEX(',', @MovieIds+',', 2) - 2)
SELECT SUBSTRING(@MovieIds, 3, CHARINDEX(',', @MovieIds+',', 3) - 3)
SELECT SUBSTRING(@MovieIds, 4, CHARINDEX(',', @MovieIds+',', 4) - 4)
SELECT SUBSTRING(@MovieIds, 5, CHARINDEX(',', @MovieIds+',', 5) - 5)

The length of the substring is now replaced by the CHARINDEX function. Once again notice the pattern with the increasing values from 1 to 5. The result now almost look like what I want. If I could filter out the second and fourth row, and merge the rest into a table – then I’m done.

Enter Numbers table

CREATE TABLE Numbers (
    n INT PRIMARY KEY
)
GO
INSERT INTO Numbers (n)
VALUES (1), (2), (3), (4), (5)
GO
SELECT n FROM Numbers
GO

Do you see a pattern that looks like the substring queries above? Now we can actually use the n column from the Numbers table instead of the hardcoded values for the increasing numbers 1 to 5. That would look like this:

DECLARE @MovieIds VARCHAR(1000) = '1,3,5'
SELECT    SUBSTRING(@MovieIds, n, CHARINDEX(',', @MovieIds+',', n) - n)
FROM    Numbers

Now I select from the Numbers table, and the only coulumn I select, is the substring function from before – but now with column n instead of the hardcoded values 1 to 5. The output:

It’s getting warm… we are almost there.

To filter out the rows I don’t want, I need to find something I can use in a where clause. The easiest is to simply use the very same substring line, and the find the rows with a value <> ‘’. That would look like this:

DECLARE @MovieIds VARCHAR(1000) = '1,3,5'
DECLARE @Delimiter VARCHAR(5) = ','
SELECT    SUBSTRING(@MovieIds, n, CHARINDEX(@Delimiter, @MovieIds+@Delimiter, n) - n) as MyCol
FROM    Numbers
WHERE    SUBSTRING(@MovieIds, n, CHARINDEX(@Delimiter, @MovieIds+@Delimiter, n) - n) <> ''

This looks very much like what we need. If we put this in a function called dbo.SplitToTable() that takes the parameters @Input and @Delimiter, we can write our original procedure like this:

CREATE FUNCTION dbo.SplitToTable(@Input VARCHAR(1000), @Delimiter VARCHAR(5))
RETURNS TABLE
AS RETURN
(
SELECT    SUBSTRING(@Input, n, CHARINDEX(@Delimiter, @Input+@Delimiter, n) - n) as val
FROM    Numbers
WHERE    SUBSTRING(@Input, n, CHARINDEX(@Delimiter, @Input+@Delimiter, n) - n) <> ''
)
GO

ALTER PROCEDURE FetchMovieTitles
@MovieIds VARCHAR(1000)
AS
SELECT * FROM MyFavoriteMovies
WHERE MovieId IN (SELECT SUBSTRING(val AS INT) FROM dbo.SplitToTable(@MovieIds, ','))
GO

EXEC FetchMovieTitles @MovieIds = '1,3,5'
GO

Now I have accomplished my goal to get rid of the dynamic sql to avoid security risks, and to be able to reuse cached execution plans.

There are a few corrections that need to be made to make this work for longer inputs. First of all we need to have more values in our numbers table. 1 to 5 only covers input string up to a length of 5. The where clause in the function can also be optimized a bit. I won’t go into the details, but the setup I often implement uses the code below:

CREATE TABLE Numbers (
    n INT PRIMARY KEY
)

--Fill data in Numbers table (this needs only to be done 1 time)
;WITH
  Pass0 as (select 1 as C union all select 1), --2 rows
  Pass1 as (select 1 as C from Pass0 as A, Pass0 as B),--4 rows
  Pass2 as (select 1 as C from Pass1 as A, Pass1 as B),--16 rows
  Pass3 as (select 1 as C from Pass2 as A, Pass2 as B),--256 rows
  Pass4 as (select 1 as C from Pass3 as A, Pass3 as B),--65.536 rows
  Pass5 as (select 1 as C from Pass4 as A, Pass4 as B),--4.294.967.296 rows
  Tally as (select row_number() over(order by C) as Number from Pass5)
INSERT Numbers (n)
SELECT Number
FROM Tally
WHERE Number < 1000000 --Fill 1 million rows (or as many as you like)
                       --This will fill approximately 13 MB
GO

CREATE FUNCTION [dbo].[SplitToTable]
(
    @Input  varchar(8000),
    @Delimiter varchar(5)
)
RETURNS TABLE
AS

RETURN
(
    SELECT
        SUBSTRING(@Input+@Delimiter, n, CHARINDEX(@Delimiter, @Input+@Delimiter, n) - n) AS Val
    FROM Numbers
    WHERE
        n <= LEN(@Input)
        AND SUBSTRING(@Delimiter + @Input, n, 1) = @Delimiter
)
GO

SELECT * FROM dbo.SplitToTable('hello|world|pipe|delimited|string', '|')
GO

To use the Numbers table one needs to think a bit differently about things, than the normal way of solving problems. The way I described the solution in this blog post, pretty much covers how I often attach a new problem, where my gut tells me that the numbers table might come in handy. What if I need a table with all dates the next 30 days? Simple:

SELECT DATEADD(DD, n, '2012-04-01')
FROM Numbers
WHERE n <= 30

Sometimes I end up with pretty nifty solutions to crazy problems, that otherwise was really nasty to solve. This SplitToTable function is just one of the many examples of things to use the Numbers table for. If you would like to learn more about this, just google for auxiliary numbers table, and a lot of different solutions and ideas pop up.

Our good old friend sys.sysprocesses actually gives us this kind of information. Let me use the same setup as I did in part 1 to trigger a blocking scenario. From connection 1 I’m execution the following, which leaves an open transaction that holds a lock on the DummyTable:

CREATE TABLE DummyTable (col1 INT)
INSERT INTO DummyTable (col1) VALUES (1)
GO

BEGIN TRAN
GO
DELETE FROM DummyTable
WAITFOR DELAY '00:01:00'
GO

From connection 2 I’m executing the following, which will keep running until connection 1 either performs a commit or a rollback:

DELETE FROM DummyTable
GO

From a third connection I’m running this query agains sys.sysprocesses:

SELECT
    procs.spid,
    procs.blocked as BlockingSPID,
    DB_NAME(procs.dbid) as DatabaseName,
    procs.program_name,
    procs.loginame,
    procs.waitresource,
    procs.lastwaittype
FROM sys.sysprocesses procs
WHERE spid IN (54,55)

SPID = 54 is my connection 1, and SPID = 55 is my second connection.

The output shows that that my connection 2 (spid 55) is blocked by spid = 54, while waiting to acquire a LCK_M_U on the resource RID:6:1:169:0. This is pretty usefull information, if we could just get the sql text returned as well. And luckily we can

We can extend the query to OUTER APPLY sys.dm_exec_sql_text. This will give us the text of the queries, so lets see how that goes:

SELECT
    procs.spid,
    procs.blocked as BlockingSPID,
    DB_NAME(procs.dbid) as DatabaseName,
    procs.program_name,
    procs.loginame,
    procs.waitresource,
    procs.lastwaittype,
    text
FROM sys.sysprocesses procs
OUTER APPLY sys.dm_exec_sql_text(sql_handle)
WHERE spid IN (54,55)

Now we are actually able to see the exact query that is causing the blocking.
Because I knew that the blocking was going on for at least 30 seconds, I simply needed to sample this data every 15. seconds, untill I had cought my sinner. I simply create a temporary table to hold the data, and the created a loop that would run until I stopped it manually. By creating a recursive query, I could also find the blocking chain – so the final query looked like this:

CREATE TABLE #Blocking
(
    Timestamp DATETIME,
    SPID SMALLINT,
    BlockingSPID INT,
    DatabaseName VARCHAR(255),
    LoginName VARCHAR(255),
    ProgramName VARCHAR(255),
    BlockingStatement VARCHAR(MAX),
    Waitresource VARCHAR(500),
    lastwaittype VARCHAR(500),
    RowNo BIGINT,
    LevelRow INT
)

--Run until manually stopped
WHILE (1=1)
BEGIN
    ;WITH Processes
    AS
    (
    SELECT
        procs.spid,
        procs.blocked AS BlockingSPID,
        DB_NAME(procs.dbid) AS DatabaseName,
        procs.program_name AS ProgramName,
        procs.loginame AS LoginName,
        procs.waitresource,
        procs.lastwaittype,
        CAST(text AS VARCHAR(MAX)) AS Text
    FROM
        sys.sysprocesses procs
        CROSS APPLY sys.dm_exec_sql_text (sql_handle)
    WHERE
        procs.spid > 50
    ), Blocking
    AS
     (
        --Define base of recursive query
        SELECT
            procs.SPID,
            procs.BlockingSPID,
            procs.DatabaseName,
            procs.LoginName,
            procs.ProgramName,
            procs.Text,
            procs.Waitresource,
            procs.lastwaittype,
            ROW_NUMBER() OVER(ORDER BY procs.SPID) as RowNo,
            0 AS LevelRow
        FROM
            Processes procs
            JOIN Processes procs2 ON procs.SPID = procs2.BlockingSPID
        WHERE
            procs.BlockingSPID = 0
        UNION ALL
        --UNION the recursive step
        SELECT
            ProcsR.SPID,
            ProcsR.BlockingSPID,
            ProcsR.DatabaseName,
            ProcsR.LoginName,
            ProcsR.ProgramName,
            ProcsR.Text,
            ProcsR.Waitresource,
            ProcsR.lastwaittype,
            d.RowNo,
            d.LevelRow + 1
        FROM
            Processes ProcsR
            JOIN Blocking d ON ProcsR.BlockingSPID = d.SPID
         WHERE
            ProcsR.BlockingSPID > 0
    )
    INSERT INTO #Blocking
    SELECT GETDATE() as Timestamp, * FROM Blocking

    --Wait for 15 seconds before starting over
    WAITFOR DELAY '00:00:15'
END

This query is using multiple leves of CTE’s, with the first level named Processes beeing pretty much identical to the query showed before. The next level CTE called Blocking is used to create the recursive structure making it possible to identify the blocking chain. My connection 1 and 2 were still running, so running the above query for just a few seconds, samples the data I want:

--After stopping the sampling manually:
SELECT * FROM #Blocking
ORDER BY Timestamp, RowNo, LevelRow

Each sample will have the same values in the Timestamp column. For each sampling each blocking chain will have the same value in RowNo, and LevelRow orders the statements in the blocking chain.
In my example, I only ran the sampling query long enough to get one sample. And the only blocking happening on the system was my two connections, so only one blocking chain was captured… the one with RowNo = 1.
LevelRow = 0 is the root of the blocking chain, and in my example this is my connection 1.

So by running the above query on the production system, and yet again waiting for the timeouts to occur, I was actually able to identify the root cause for all the timeouts I was debugging. The blocking query was holding a table lock for a long time, which perfectly explained what was happening. Fixing the root cause is another story, which is not in scope of this post.

Wrap up

A system was experiencing intermittent query timeouts. My debugging process was:

1. Use a classic server side trace to catch the queries that timed out, to figure out if they were using up resources for more than 30 seconds, or if they were waiting for something.
2. It turned out they were waiting for something, so I ran a XEvent session to track all wait events with a duration > 200 ms (part 1 of this blog post series)
3. The XEvent session revealed that the queries were waiting to acquire an intent exclusive lock, so the next step was to sample blocking chains to find the root query (This blog post)
4. The blocking sampling revealed the query that caused heavy table locks, and now it was easy to go ahead and fixing the problem.

But remember..

The debugging process will not work for hunting down the root cause for all scenarios. The outcome of each step determines what the next step should be. So if my initial server side trace had shown heavy cpu usage for the entire 30 seconds, then my next step would be to look into execution plans etc. So in that case it wouldn’t make sense to use the XEvent sessions to capture wait_info and continue down that path.
Nevertheless I hope this little blog post series gave you an idea how you could hunt down your own nasty queries

So I knew that the queries did not complete in 30 seconds, but I also knew that the queries were executed thousands of times during a day, with no problem at all.

I didn’t know if the queries were actually hitting the cpu/disks heavily, or if they were blocked by something. If they were waiting for something, I didn’t know if it was physical resources like disk, cpu, network or if it was a lock on an object. If they were hitting the physical resources, it might be a bad execution plan, messed up statistics or a sick index that was the cause.

To answer some of the unknowns, I started a basic sql server trace, and caught all queries with a duration > 25 seconds. After running that trace a couple of days, the timeouts finally happened again. From the trace I could see that the queries had a duration of ~30.000 ms, but a cpu value of only 1-200 ms. The number of reads were also close to zero. This meant, that a bad execution plan wasn’t the cause, but some waiting/blocking was going on. I just couldn’t see what it was from the sql server trace.

X-Events to the rescue!

X-Event Sessions is the new tracing method introduced with SQL Server 2008, which is more lightweight than a classic profiling/tracing, and it is also able to give more detailed information. I won’t go into the details of XEvents in this blog post, but just show you how I used it for this specific task.

The goal

The goal for this X-Event tracing was to be able to find out what the queries were waiting for. It could either be other queries that had a lock on objects, or it could be a lack of physical resources in the server.

The code

To create the X-Event session, you simply need to run this:

CREATE EVENT SESSION [WaitEvents] ON SERVER
ADD EVENT sqlos.wait_info
(
    ACTION(
        sqlserver.database_id,
        sqlserver.session_id,
        sqlserver.sql_text,
        sqlserver.plan_handle
        )
    WHERE
        opcode = 1 AND duration > 200 AND session_id > 50
)
ADD TARGET package0.asynchronous_file_target
    (SET FILENAME = N'C:\XEvents\WaitEvents.xel',
    METADATAFILE = N'C:\XEvents\WaitEvents.xem')
WITH (EVENT_RETENTION_MODE=ALLOW_SINGLE_EVENT_LOSS,
    MAX_DISPATCH_LATENCY=5 SECONDS)
GO

Just adjust the path to the files to match your server.
The session triggers on events of the type sqlos.wait_info. This triggers an event on every wait event that is happening on the system.
The ACTION part defines the information I want extracted. In this case I just want the database_id, session_id, sql_text as well as the plan_handle to be able to see the actual execution plan for that specific query at that specific time.
The opcode=1 filters out so we only get the “end” events.
session_id > 50 ensures that we only get events caused by user sessions. If the root cause happens to be some SQL Server internal stuff, then we might miss it – but my gut feeling told me that it was ok in this case.
duration > 200 ensures that we are not getting spammed with a lot af small wait events for all io operations. If the root cause is a lot of slow IO’s, then  it might not be caught by this XEvent session. If that’s the case, it’s just a matter of adjusting the values. The lower value, the more events you will collect.

Because I need to have this X-Event session running for a couple of days, I want to store the events in files, so the ADD TARGET simply does that.

The session isn’t running per default, so we need to start it manually:

ALTER EVENT SESSION [WaitEvents] ON SERVER
STATE = START
GO

Stopping the session is as simple as:

ALTER EVENT SESSION [WaitEvents] ON SERVER
STATE = STOP
GO

And dropping it entirely from the server:

DROP EVENT SESSION [WaitEvents] ON SERVER

Like a server side trace, we are able to leave the session definition on the server, and then just starting and stopping it when we need to collect data.

Generating dummy events

While the above XEvent session was running on my test setup, I ran the following from connection 1:

CREATE TABLE DummyTable (col1 INT)
INSERT INTO DummyTable (col1) VALUES (1)
GO

WAITFOR DELAY '00:00:01'
GO

BEGIN TRAN
GO
DELETE FROM DummyTable
GO

This leaves an open transaction which holds a lock on the DummyTable. From connection 2 I then ran this:

DELETE FROM DummyTable
GO

I then aborted the query from connection 2, and committed the transaction from connection 1. I then stopped the X-Event session with the STOP command above, and was ready to analyze the data.

Analyzing the data

To analyze the data we need to do some XQuery magic. To be honest, I’m not too comfortable writing these queries from scratch, but luckily it’s easy to find tons of examples on the internet. If you haven’t found a query that matches your XEvent session 100%, then it’s most often easy to adjust the scripts to do so.

The query I ended up with, was this (Don’t worry, it’s not as nasty as it might seem )

;WITH MyXEventData AS (
SELECT
DATEADD(hh,
DATEDIFF(hh, GETUTCDATE(), CURRENT_TIMESTAMP),
XEventData.value('(event/@timestamp)[1]', 'datetime2')) AS [timestamp],
COALESCE(XEventData.value('(event/data[@name="database_id"]/value)[1]', 'int'),
XEventData.value('(event/action[@name="database_id"]/value)[1]', 'int')) AS database_id,
XEventData.value('(event/action[@name="session_id"]/value)[1]', 'int') AS [session_id],
XEventData.value('(event/data[@name="wait_type"]/text)[1]', 'nvarchar(4000)') AS [wait_type],
XEventData.value('(event/data[@name="duration"]/value)[1]', 'bigint') AS [duration],
XEventData.value('(event/action[@name="plan_handle"]/value)[1]', 'nvarchar(4000)') AS [plan_handle],
XEventData.value('(event/action[@name="sql_text"]/value)[1]', 'nvarchar(4000)') AS [sql_text]
FROM (
    SELECT CAST (event_data AS XML) AS XEventData
    FROM sys.fn_xe_file_target_read_file (
        'C:\XEvents\WaitEvents*.xel',
        'C:\XEvents\WaitEvents*.xem', null, null)) RawData
)
SELECT *
FROM MyXEventData

The query uses a CTE (Common Table Expression) to wrap the nasty xml stuff, and making it way easier to write simple queries against the CTE in the end.
Let’s break this down a bit. The following part simply reads the content of the files, and returns the entire xml as a column called XEventData:

SELECT CAST (event_data AS XML) AS XEventData
FROM sys.fn_xe_file_target_read_file (
        ‘C:\XEvents\WaitEvents*.xel’,
        ‘C:\XEvents\WaitEvents*.xem’, null, null)) RawData

This sub-select is what we execute the XQuery expressions against, to fetch the relevant data from within the xml document. Since this is not a blog post about XML and XQuerying, let’s just skip the details, and go to the resulting output from the entire query:

The first event was my WAITFOR DELAY query, which by design was waiting for 1 second. The wait_type is WAITFOR. The duration column says 1001 ms, which is pretty close to the expected 1 sec.

The second event was our DELETE statement from connection 2. This was waiting for 2166 ms before I manually aborted the query. The wait_type was LCK_M_U.

Back to my hunting experience

Using the XEvent session as demonstrated above, I ran it on the production system to find out what the timed out queries were waiting for. After a bunch of timeouts had happened, I found all the matching wait events, and they all had a matching wait event of type LCK_M_IX with a duration of ~30000 ms. The LCK_M_IX is an intent exclusive lock, which confirms my gut feeling.

Know I knew for certain, that some other query was blocking for a long time period. And since the affected queries were both inserts, updates and deletes, I thought I was dealing with a table lock. But I didn’t know for sure, and I didn’t know which query was the sinner.

Next step

The next is to identify the query that caused the blocking. I will show you how to do that in part 2 of this blog post, so keep and eye out for it.

The Quest

The Quest was to be able to provide a system with undo/redo capabilities, so if a user had entered (and committed) incorrect data, it should be possible to rollback to any given time before that happened. Backup/restore wasn’t an option, because only a specific subset of data should be rolled back (let’s say a specific UserId)

The idea

The idea was to either build a tracking system manually using triggers, or to look into the CDC possibilities. The idea I will try out with this small demo, is to use CDC to read the data from a previous point in time, and then use the MERGE statement to revert the data.

The solution

First I’ll create a database and enable CDC:

USE master
go
CREATE DATABASE TestCDC
GO
USE TestCDC
GO
EXEC sys.sp_cdc_enable_db
GO

Next I will create a table “Timelog”, and enable CDC on it. The Timelog table is the one we want the undo/redo capabillity on.

CREATE TABLE TimeLog (
    TimeLogId INT IDENTITY PRIMARY KEY,
    UserId INT NOT NULL,
    TimeStart DATETIME NOT NULL,
    TimeEnd DATETIME NULL,
    Description VARCHAR(200),
    Created DATETIME DEFAULT GETDATE(),
    Modified DATETIME DEFAULT GETDATE()
)
GO

EXEC sys.sp_cdc_enable_table
@source_schema = N'dbo',
@source_name   = N'TimeLog',
@role_name     = NULL,
@filegroup_name = N'PRIMARY',
@supports_net_changes = 1
GO

The sys.sp_cdc_enable_table procedure gives the possibilities to control which filegroup the change data should be stored on, as well as a role to control access to the data. For this demo I’ll just store the data in the PRIMARY filegroup. CDC relies on SQL Server Agent to be running, so make sure it is so before running the sys.sp_cdc_enable_table procedure.

Next up is inserting some data to simulate one user performing a few correct time registrations, and another user registering some wrong data. I have put in a WAITFOR DELAY, which simulates that some time has past on since the last registration. It does not matter if it is seconds or months.

/*
Two users - userid 1 and 2
User 1 makes 3 registrations, but the third (the update) is wrong
User 2 makes 2 registrations, both are correct.
User 1 makes 1 correct registration

*/
--Correct registration
INSERT INTO TimeLog (UserId, TimeStart, Description)
VALUES (1, '2012-01-01 10:00:00', '1st work day')
GO
WAITFOR DELAY '00:00:02'

--Correct registration
INSERT INTO TimeLog (UserId, TimeStart, Description)
VALUES (1, '2012-01-02 10:00:00', '2nd work day')
GO
WAITFOR DELAY '00:00:02'

--Wrong registration/update
UPDATE TimeLog
SET
    TimeEnd = '2013-01-01 19:00:00',
    Modified = GETDATE()
WHERE
    UserId = 1
    AND TimeStart = '2012-01-01 10:00:00'
GO
WAITFOR DELAY '00:00:02'

--Correct registration
INSERT INTO TimeLog (UserId, TimeStart, Description)
VALUES (2, '2012-01-01 09:00:00', '1st work day')
GO
WAITFOR DELAY '00:00:02'

--Correct registration
INSERT INTO TimeLog (UserId, TimeStart, Description)
VALUES (2, '2012-01-02 09:00:00', '2nd work day')
GO
WAITFOR DELAY '00:00:02'

--Correct registration
INSERT INTO TimeLog (UserId, TimeStart, Description)
VALUES (1, '2012-01-03 10:00:00', '3rd work day')
GO

Let’s see how the data looks now:

--Notice TimeLogId = 1.
--The TimeEnd shouldn't be in 2013.
SELECT * FROM TimeLog

CDC adds a few functions that we can use to select the changes that has happened. The first we will use, is sys.fn_cdc_map_time_to_lsn, which takes a datetime as input, and returns the lsn (Transaction log sequence number) that matches the timestamp. It is also fed with a string value that tells if it should return the lsn that is

• largest less than
• largest less than or equal
• smallest greater than
• smallest greater than or equal

I’m interested in all changes that has happened on the table within the last 24 hours, so I’ll use the following lines to find those lsn’s:

DECLARE @begin_time datetime, @end_time datetime, @begin_lsn binary(10), @end_lsn binary(10);
SET @begin_time = GETDATE()-1
SET @end_time = GETDATE()
SELECT @begin_lsn = sys.fn_cdc_map_time_to_lsn('smallest greater than', @begin_time);
SELECT @end_lsn = sys.fn_cdc_map_time_to_lsn('largest less than or equal', @end_time);

Now I have the lsn of the smallest lsn that has happened after the point in time GETDATE()-1 which is 24 hours ago. I also have the lsn of the last lsn that has happened before (or equal to) GETDATE(). This leads me to the next functions available. After enabling CDC on the table, a few functions and system tables was automatically added to my database:

The next function I’ll use, is the one called cdc.fn_cdc_get_all_changes_dbo_TimeLog. It’s called with the two lsn’s found above like this:

SELECT *
FROM cdc.fn_cdc_get_all_changes_dbo_TimeLog(@begin_lsn, @end_lsn, 'all update old')

I now get all the changes that has happened, and updates will be listed as a trigger solution would, by having both a delete and insert row to hold the data from before the update and after.

Now UserId = 1 realizes that invalid registrations have found its way in the table. If we look at the CDC data, then UserId = 1 created an error at the transaction with lsn = 0x0000002C0000013B0003 which happened at ‘2012-03-05 19:38:04.727’. Another function can now be used to find the netto change up until right before that time. And because we started the cdc on an empty table, the netto change correspond to how the table looked at that time. The function is similar to the cdc.fn_cdc_get_all_changes_dbo_TimeLog() we used before, now it’s just called _net_ instead of _all_. So let’s try it out:

DECLARE @begin_time datetime = GETDATE()-1
DECLARE @end_time datetime = '2012-03-05 19:38:04.727'

DECLARE @begin_lsn binary(10) = (select sys.fn_cdc_map_time_to_lsn('smallest greater than', @begin_time))
DECLARE @end_lsn binary(10) = (select sys.fn_cdc_map_time_to_lsn('largest less than', @end_time))
SELECT *
FROM cdc.fn_cdc_get_net_changes_dbo_TimeLog(@begin_lsn, @end_lsn, 'all')
WHERE UserId = 1

Now I filtered to only show changes for UserId = 1, and only up until right before the time when the wrong data was entered. This is actually the data as we would like to revert back to. Remember, that we don’t want to change anything for UserId = 2!

The way we can do this, is by using a MERGE statement, that uses the select above as source, and the TimeLog table as destination. Like this:

--Reset UserId = 1 to 2012-03-05 19:38:04.727
DECLARE @UserId INT = 1
DECLARE @begin_time datetime = GETDATE()-1
DECLARE @end_time datetime = '2012-03-05 19:38:04.727'

DECLARE @begin_lsn binary(10) = (select sys.fn_cdc_map_time_to_lsn('smallest greater than', @begin_time))
DECLARE @end_lsn binary(10) = (select sys.fn_cdc_map_time_to_lsn('largest less than', @end_time))

SET IDENTITY_INSERT TimeLog ON
MERGE INTO TimeLog tgt
USING (
    SELECT *
    FROM cdc.fn_cdc_get_net_changes_dbo_TimeLog(@begin_lsn, @end_lsn, 'all')
    WHERE UserId = @UserId
) src ON tgt.TimeLogId = src.TimeLogId
WHEN MATCHED THEN
    UPDATE SET
        tgt.TimeStart = src.TimeStart,
        tgt.TimeEnd = src.TimeEnd,
        tgt.Description = src.Description,
        tgt.Created = src.Created,
        tgt.Modified = src.Modified
WHEN NOT MATCHED BY TARGET THEN
    INSERT (TimeLogId, UserId, TimeStart, TimeEnd, Description, Created, Modified)
    VALUES (src.TimeLogId, src.UserId, src.TimeStart, src.TimeEnd, src.Description, src.Created, src.Modified)
WHEN NOT MATCHED BY SOURCE AND tgt.UserId = @UserId THEN
    DELETE;
SET IDENTITY_INSERT TimeLog OFF
GO

1. The select in the USING part, is the same as the one we just saw. I’m joining the source and target on TimeLogId. If the clause matches, then we simply update all the columns of the target (TimeLog table) to the values from the source (the CDC data).
2. The rows that exists in the source, but not the target, is being inserted (notice the IDENTITY_INSERT at the top).
3. The rows that does not exist in the source, but is present in the target and has UserId = 1 is deleted.

This is just a basic text book MERGE statement, so no fancy thing here.

Now let’s see the data available in TimeLog:

SELECT * FROM TimeLog

All the registrations for UserId = 2 is still available, but the last two changes for UserId = 1 is gone. That was the wrong update on the “1st work day”, as well as the “3rd work day”.

If we wan’t to perform a redo (an undo of the rollback), then we simply need to execute the exact same code, but use the EndTime from right before we did the rollback. Simple, right?

The conclusion

The solution I came up with, actually solved the problem. But I haven’t done any testing using huge data volumes. And since I have absolutely no real world experience using CDC, I cannot say if this idea is truly stupid, or if it actually might work. I haven’t considered how the undo feature should be build using transactions to ensure integrity, so the solution will probably be a bit more complex than the one above.

But at least now I have an idea what CDC is and what it might be used for, and hopefully you have too