Archive

Archive for the ‘Administration’ Category

Stored Procedure to Automate Restores – Mark IV

March 3, 2014 Leave a comment

Here is the latest version of sp_AutomateDBRestores. Previous versions used system tables to build the MOVE clause for the RESTORE statement and to identify valid backup files. That works fine if your instance is online and healthy. This version allows a list of backup files to be input as a table-valued parameter, in the event that your primary server is unavailable and you need to restore to another location. RESTORE FILELISTONLY is used to identify logical and physical data files and RESTORE HEADERONLY validates the backup files. Code to define the table-valued parameter is contained in the procedure. Its definition is:


CREATE TYPE backupfiletype AS TABLE 
(backupfilename VARCHAR(255)
,backupfiletype VARCHAR(4));

The procedure expects backupfiletype to equal FULL, DIFF, or LOG. Any other value will generate an error.

This is an example of calling the procedure with the table-valued parameter and a STOPAT date:


DECLARE @backuptvp AS backupfiletype

INSERT INTO @backuptvp
VALUES
('C:\Backup\AdventureWorks2012\FULL\AdventureWorks2012_FULL_20140303_110000.bak', 'FULL'),
('C:\Backup\AdventureWorks2012\DIFF\AdventureWorks2012_DIFF_20140303_110432.bak', 'DIFF'),
('C:\Backup\AdventureWorks2012\LOG\AdventureWorks2012_LOG_20140303_110438.trn', 'LOG'),
('C:\Backup\AdventureWorks2012\LOG\AdventureWorks2012_LOG_20140303_110444.trn', 'LOG'),
('C:\Backup\AdventureWorks2012\LOG\AdventureWorks2012_LOG_20140303_110500.trn', 'LOG')

EXEC sp_automateDBRestoresWithFileList 
@sourcedbname = 'AdventureWorks2012',
@restoredbname = 'AdventureWorks2012test',
@droprestoredb = 0,
@noexec = 1,
@stopat = '2014-03-03 11:04:43.878', 
@TVP = @backuptvp

Currently, the MOVE clause will use the physical data file of the original backup. If you need to restore to a different location, execute the procedure with @noexec = 1 and manually edit the restore commands. I will be posting an update shortly that will allow physical file names to be passed in as parameters.

As usual, I hope you can put this code to use. If you have questions or comments, use the comment link on this page.


USE [master]
GO
 
 /****** Object:  StoredProcedure [dbo].[sp_automateDBRestoresWithFileList]    Script Date: 2/16/2014 8:48:07 PM ******/
IF EXISTS(SELECT 1 FROM master.sys.objects WHERE name = 'sp_automateDBRestoresWithFileList')
BEGIN
 
    DROP PROCEDURE [dbo].[sp_automateDBRestoresWithFileList]
 
END
GO

USE [master]
GO

--If the backupfiletype data type does not exist, create it
IF NOT EXISTS
(SELECT 1 FROM sys.types
WHERE name = 'backupfiletype')
BEGIN

		CREATE TYPE backupfiletype AS TABLE 
		(backupfilename VARCHAR(255)
		,backupfiletype VARCHAR(4));

END
 
/****** Object:  StoredProcedure [dbo].[sp_automateDBRestoresWithFileList]    Script Date: 2/16/2014 8:48:07 PM ******/
SET ANSI_NULLS ON
GO
 
SET QUOTED_IDENTIFIER ON
GO
 
CREATE PROCEDURE [dbo].sp_automateDBRestoresWithFileList
    @sourcedbname SYSNAME, 
    @restoredbname SYSNAME = NULL,
    @droprestoredb TINYINT = 1,
    @noexec TINYINT = 0,
    @stopat DATETIME = NULL,
	@TVP backupfiletype READONLY
AS
       
SET NOCOUNT ON
  
BEGIN TRY
  
--Drop temp tables if they exist
IF OBJECT_ID('tempdb..#backupfiles') IS NOT NULL
BEGIN
   
    DROP TABLE #backupfiles
       
END
   
IF OBJECT_ID('tempdb..#databasefiles') IS NOT NULL
BEGIN
   
    DROP TABLE #databasefiles
       
END

IF EXISTS(SELECT 1 FROM tempdb.sys.objects WHERE name LIKE '%headeronly%')
BEGIN

	DROP TABLE #headeronly

END

IF EXISTS(SELECT 1 FROM tempdb.sys.objects WHERE name LIKE '%headeronlyfile%')
BEGIN

	DROP TABLE #headeronlyfile

END

IF EXISTS(SELECT * FROM tempdb.sys.objects WHERE name LIKE 'FileListOnly%')
BEGIN

	DROP TABLE #FileListOnly

END
   

--Create temp tables
CREATE TABLE #tvpfiles
(backupfilename VARCHAR(2000)
,backupfiletype VARCHAR(4))

CREATE TABLE #backupfiles
(backupstartdate DATETIME
,backupfinishdate DATETIME
,physicalfilename VARCHAR(2000)
,backuptype CHAR(4))
   
CREATE TABLE #databasefiles
(DatabaseName SYSNAME
,LogicalName SYSNAME
,PhysicalName VARCHAR(2000)
,filetype TINYINT)

CREATE TABLE #headeronly
(BackupName NVARCHAR(128)
,BackupDescription NVARCHAR(255)
,BackupType SMALLINT
,ExpirationDate DATETIME
,Compressed BIT
,Position SMALLINT
,DeviceType TINYINT
,UserName NVARCHAR(128)
,ServerName NVARCHAR(128)
,DatabaseName NVARCHAR(128)
,DatabaseVersion INT
,DatabaseCreationDate DATETIME
,BackupSize NUMERIC(20,0)
,FirstLSN NUMERIC(25,0)
,LastLSN NUMERIC(25,0)
,CheckpointLSN NUMERIC(25,0)
,DatabaseBackupLSN NUMERIC(25,0)
,BackupStartDate DATETIME
,BackupFinishDate DATETIME
,SortOrder SMALLINT
,CodePage SMALLINT
,UnicodeLocaleId INT
,UnicodeComparisonStyle INT
,CompatibilityLevel TINYINT
,SoftwareVendorId INT
,SoftwareVersionMajor INT
,SoftwareVersionMinor INT
,SoftwareVersionBuild INT
,MachineName NVARCHAR(128)
,Flags INT
,BindingID UNIQUEIDENTIFIER
,RecoveryForkID UNIQUEIDENTIFIER
,Collation NVARCHAR(128)
,FamilyGUID UNIQUEIDENTIFIER
,HasBulkLoggedData BIT
,IsSnapshot BIT
,IsReadOnly BIT
,IsSingleUser BIT
,HasBackupChecksums BIT
,IsDamaged BIT
,BeginsLogChain BIT
,HasIncompleteMetaData BIT
,IsForceOffline BIT
,IsCopyOnly BIT
,FirstRecoveryForkID UNIQUEIDENTIFIER
,ForkPointLSN NUMERIC(25,0)
,RecoveryModel NVARCHAR(60)
,DifferentialBaseLSN NUMERIC(25,0)
,DifferentialBaseGUID UNIQUEIDENTIFIER
,BackupTypeDescription NVARCHAR(60)
,BackupSetGUID UNIQUEIDENTIFIER
,CompressedBackupSize BIGINT
,Containment BIT)

CREATE TABLE #FileListOnly (
    LogicalName NVARCHAR(128),
    PhysicalName NVARCHAR(260),
    [Type] CHAR(1),
    FileGroupName NVARCHAR(128),
    Size NUMERIC(20,0),
    MaxSize NUMERIC(20,0), 
    FileID BIGINT, 
    CreateLSN NUMERIC(25,0), 
    DropLSN NUMERIC(25,0),
    UniqueID UNIQUEIDENTIFIER, 
    ReadOnlyLSN NUMERIC(25,0),
    ReadWriteLSN NUMERIC(25,0),
    BackupSizeInBytes BIGINT, 
    SourceBlockSize INT, 
    FileGroupID INT, 
    LogGroupGUID UNIQUEIDENTIFIER,
    DifferentialBaseLSN NUMERIC(25,0),
    DifferentialBaseGUID UNIQUEIDENTIFIER, 
    IsReadOnly BIT, 
    IsPresent BIT, 
    TDEThumbprint VARBINARY(32) 
 ) 

--Declare local variables
DECLARE @filecount SMALLINT
DECLARE @fullfilecount SMALLINT
DECLARE @loopcount SMALLINT
DECLARE @datafileloopcount SMALLINT
DECLARE @backuploopcount SMALLINT
DECLARE @backupfile VARCHAR(2000)
DECLARE @tvpbackupfile VARCHAR(2000)
DECLARE @sqlstr VARCHAR(MAX)
DECLARE @backupfiletype CHAR(4)
DECLARE @tvpbackupfiletype CHAR(4)
DECLARE @datafilecount SMALLINT
DECLARE @logfilecount SMALLINT
DECLARE @logbackupcount SMALLINT
DECLARE @diffbackupcount SMALLINT
DECLARE @datafile VARCHAR(2000)
DECLARE @moveparm VARCHAR(2000)
DECLARE @fromstatement VARCHAR(2000)
DECLARE @logfile SYSNAME
DECLARE @physlogfile VARCHAR(2000)
DECLARE @physdatafile VARCHAR(2000)
DECLARE @fullbackupdate DATETIME
DECLARE @logbackupdate DATETIME
DECLARE @logbackuplimit DATETIME
DECLARE @errormessage VARCHAR(1000)
DECLARE @tvplimit SMALLINT
DECLARE @tvploopcount SMALLINT
  
--Initialize variables
SET @moveparm = ' WITH'
SET @fromstatement = ' FROM '
SET @loopcount = 1
SET @datafileloopcount = 1
SET @backuploopcount = 1
SET @tvploopcount = 1

--Get backup file name and file type from the table-valued parameter, if they exist
INSERT INTO #tvpfiles
SELECT backupfilename, backupfiletype FROM @TVP

--Get the count of rows from the table-valued parameter
SET @tvplimit = @@ROWCOUNT

SELECT @tvplimit AS TVPLimit
  
--Drop the restored database if it exists and @droprestoredb = 1
IF EXISTS(SELECT 1 FROM sys.databases WHERE name = @restoredbname) AND @droprestoredb = 1
BEGIN
	
	--Drop the existing database if @sourcedbname and @restoredbname are different or if they are the same, but @TVP has been passed in
	IF (@sourcedbname <> @restoredbname) OR (@sourcedbname = @restoredbname AND @tvplimit > 0)
	BEGIN

		SET @sqlstr = 'DROP DATABASE ' + @restoredbname
		EXEC (@sqlstr)
       
	END
	ELSE
	--This condition will drop the existing database, making sys.database_files unavailable
	--An error is raised and the procedure stops.
	BEGIN

		RAISERROR ('You are attempting to restore over an existing database and have set @droprestoredb = 1.  This will cause the existing database to be dropped, making sys.database_files unavailable.  Please change the @restoredbname or set @droprestoredb = 0.', -- Message text.
		16, -- Severity.
		1 -- State.
		);

	END
END

--If a table-valued parameter is not passed in, get file information from sys.database_files   
IF @tvplimit = 0
BEGIN

	--Build dynamic SQL to get a list of data and log files for the source database
	SET @sqlstr = 'INSERT INTO #databasefiles SELECT ' + '''' + @sourcedbname + '''' + ' AS DatabaseName, name, physical_name, type 
	FROM ' + @sourcedbname + '.sys.database_files'
 
 
	--Execute dynamic SQL to return data and log files for the source database
	EXEC (@sqlstr)
   
	--Get counts of data files and log files
	SELECT @datafilecount = COUNT(*) FROM #databasefiles WHERE filetype = 0
	SELECT @logfilecount  = COUNT(*) FROM #databasefiles WHERE filetype = 1

END
--If table-valued parameter is passed in get database file information from RESTORE FILELISTONLY
--Insert the backupfilename and backupfiletype into #backupfiles
ELSE
BEGIN

	SELECT @tvpbackupfile = backupfilename FROM #tvpfiles
	WHERE backupfiletype = 'FULL'

	SET @sqlstr = 'RESTORE FILELISTONLY FROM DISK = ' + '''' + @tvpbackupfile + ''''

	PRINT @sqlstr

	INSERT INTO #FileListOnly
	EXEC(@sqlstr)
	
	INSERT INTO #databasefiles
	SELECT 
	@sourcedbname, 
	LogicalName, 
	PhysicalName, 
	CASE [Type]
	WHEN 'D' THEN 0
	WHEN 'L' THEN 1
	END
	FROM #FileListOnly


	--Loop through files in #tvpfiles and run RESTORE HEADERONLY to get the BackupStartDate and BackupFinishDate for each file
	WHILE @tvploopcount <= @tvplimit
	BEGIN

		SELECT TOP 1 @tvpbackupfile = backupfilename, @tvpbackupfiletype = backupfiletype
		FROM #tvpfiles

		IF @tvpbackupfiletype NOT IN ('FULL', 'DIFF', 'LOG')
		BEGIN

			RAISERROR ('You have passed an invalid backup file type.  Valid backup file types are FULL, DIFF, and LOG.  Please review your data.', -- Message text.
			16, -- Severity.
			1 -- State.
			);

		END

		SELECT @sqlstr = 'RESTORE HEADERONLY FROM DISK = ' + '''' + @tvpbackupfile + ''''

		INSERT INTO #headeronly
		EXEC(@sqlstr)

		INSERT INTO #backupfiles
		SELECT BackupStartDate, BackupFinishDate, @tvpbackupfile, @tvpbackupfiletype
		FROM #headeronly

		SELECT @datafilecount = COUNT(*) FROM #databasefiles WHERE filetype = 0
		SELECT @logfilecount  = COUNT(*) FROM #databasefiles WHERE filetype = 1

		SET @tvploopcount += 1

		DELETE FROM #tvpfiles
		WHERE backupfilename = @tvpbackupfile

		DELETE FROM #headeronly

	END

END

--Raise an error if the database contains more than one log file  
IF @logfilecount > 1
BEGIN
  
    RAISERROR ('You have more than one log file, which is a really bad idea.  No restore for you.  See this link for more information.  http://www.sqlskills.com/blogs/paul/multiple-log-files-and-why-theyre-bad/', -- Message text.
    16, -- Severity.
    1 -- State.
    );
      
END

--If @droprestoredb = 1 and @restoredbname <> @droprestoredb. replace file extensions
IF @sourcedbname <> @restoredbname 
BEGIN

	UPDATE #databasefiles
	SET PhysicalName = REPLACE(PhysicalName,'.mdf','_testrestore.mdf')
	WHERE PhysicalName LIKE '%mdf%'

	UPDATE #databasefiles
	SET PhysicalName = REPLACE(PhysicalName,'.ldf','_testrestore.ldf')
	WHERE PhysicalName LIKE '%ldf%'

END

--Get the logical and physical log file names
SELECT @logfile = LogicalName, @physlogfile = PhysicalName
FROM #databasefiles
WHERE filetype = 1
   
--If the database contains one data file, build the MOVE clause for a single file
IF @datafilecount = 1
BEGIN
     
    SELECT @datafile = LogicalName, @physdatafile = PhysicalName
	FROM #databasefiles
    WHERE filetype = 0
     
    SET @moveparm = @moveparm + ' MOVE ' + '''' + @datafile + '''' + ' TO ' + '''' + @physdatafile + '''' + ',' + ' MOVE ' + '''' + @logfile + '''' + ' TO ' + '''' + @physlogfile + '''' + ', NORECOVERY'

END
ELSE
--If the database contains more than one data file, loop through the list of files and build a multi-file MOVE statement
IF @datafilecount > 1
BEGIN
       
    --SELECT @datafileloopcount, @datafilecount
    WHILE @datafileloopcount <= @datafilecount
    BEGIN
           
        --SELECT TOP (1) LogicalName, REPLACE(PhysicalName,@sourcedbname,@restoredbname) FROM #databasefiles WHERE filetype = 0
        SET @physdatafile = (SELECT TOP (1) PhysicalName FROM #databasefiles WHERE filetype = 0)
        SET @datafile = (SELECT TOP (1) LogicalName FROM #databasefiles WHERE filetype = 0)     
        SET @moveparm = @moveparm + ' MOVE ' + '''' + @datafile + '''' + ' TO ' + '''' + @physdatafile + '''' + ','
        DELETE TOP (1) FROM #databasefiles WHERE filetype = 0
        SET @datafileloopcount += 1
           
    END
       
    --Append the MOVE command for the log file to the MOVE clause and set it to NORECOVERY
    SET @moveparm = @moveparm + ' MOVE ' + '''' + @logfile + '''' + ' TO ' + '''' + @physlogfile + '''' + ', NORECOVERY'
           
END
   
IF @droprestoredb = 0
BEGIN
   
    SET @moveparm = @moveparm + ', REPLACE'
       
END

--If the table-valued parameter is not passed in, build the #backupfiles table from msdb tables 
IF @tvplimit = 0
BEGIN

	SELECT @fullbackupdate = MAX(c.backup_finish_date)
			FROM msdb.dbo.backupset c 
			INNER JOIN msdb.dbo.backupmediafamily d
			ON c.media_set_id = d.media_set_id 
			WHERE c.database_name = @sourcedbname
			AND c.type = 'D'
         
	SELECT @logbackupdate = MAX(c.backup_start_date)
			FROM msdb.dbo.backupset c 
			INNER JOIN msdb.dbo.backupmediafamily d
			ON c.media_set_id = d.media_set_id 
			WHERE c.database_name = @sourcedbname
			AND c.type IN ('D', 'I')
 
	--Get the most recent full backup file for the source database and insert it into the #backupfiles temp table
	INSERT INTO #backupfiles
	SELECT
	a.backup_start_date, a.backup_finish_date, b.physical_device_name, 'FULL' AS [backuptype]
	FROM msdb.dbo.backupset a join msdb.dbo.backupmediafamily b
	  ON a.media_set_id = b.media_set_id
	WHERE a.database_name = @sourcedbname
	AND backup_finish_date = @fullbackupdate
	ORDER BY a.backup_finish_date ASC
  
	--Get a list of differential backup files older than the most recent full backup
	INSERT INTO #backupfiles
	SELECT TOP 1
	a.backup_start_date, a.backup_finish_date, b.physical_device_name, 'DIFF' AS [backuptype]
	FROM msdb.dbo.backupset a join msdb.dbo.backupmediafamily b
	  ON a.media_set_id = b.media_set_id
	WHERE a.database_name = @sourcedbname
	AND a.type = 'I'
	AND backup_finish_date > @fullbackupdate
	ORDER BY a.backup_finish_date DESC
  
	--Get all log backupfiles run after the latest full or differential backup and insert it into the #backupfiles temp table
	INSERT INTO #backupfiles
	SELECT
	a.backup_start_date, a.backup_finish_date, b.physical_device_name,'LOG'  AS [backuptype]
	from msdb.dbo.backupset a join msdb.dbo.backupmediafamily b
	  on a.media_set_id = b.media_set_id
	where a.database_name = @sourcedbname
	AND a.type = 'L'
	AND backup_finish_date > @logbackupdate
	order by a.backup_finish_date ASC
 
	IF @stopat < @fullbackupdate
	BEGIN
 
		SET @errormessage = 'The @stopat values provided ' + CAST(@stopat AS VARCHAR(26)) + 'is older than the most recent full backup date of ' + CAST(@fullbackupdate AS VARCHAR(26)) + '.  Please check the value and rerun.'
 
		RAISERROR (@errormessage, -- Message text.
		16, -- Severity.
		1 -- State.
		);
     
	END

END     

--If a @stopat date has been passed, delete LOG files newer than @stopat 
IF @stopat IS NOT NULL
BEGIN
 
    SELECT TOP 1 @logbackuplimit = backupfinishdate FROM #backupfiles
    WHERE backupfinishdate > @stopat
     
    DELETE FROM #backupfiles 
    WHERE backupfinishdate > @logbackuplimit
     
END
  
--Get counts of all backup files, full backup files, differential backup files, and log backup files for use in loop processing below
SELECT @filecount = COUNT(*) FROM #backupfiles
SELECT @fullfilecount = COUNT(*) FROM #backupfiles WHERE backuptype = 'FULL'
SELECT @diffbackupcount = COUNT(*) FROM #backupfiles WHERE backuptype = 'DIFF'
SELECT @logbackupcount = COUNT(*) FROM #backupfiles WHERE backuptype = 'LOG'

--If no FULL backups exists. error out  
IF @fullfilecount = 0 
BEGIN
  
    RAISERROR ('There are no full backup files for your database.', -- Message text.
    16, -- Severity.
    1 -- State.
    );
  
END
   
--While backup files exist, loop through them to generate the restore statement
WHILE @loopcount <= @filecount
BEGIN
      
    --Get the top backup file from the temp table   
    SET @backupfiletype = (SELECT TOP 1 backuptype FROM #backupfiles)
       
    --If it is a full backup file, build the restore statement
    IF @backupfiletype = 'FULL'
    BEGIN
  
        --If there are no log backups to apply, replace NORECOVERY with RECOVERY in the MOVE clause
        IF (@logbackupcount = 0 AND @diffbackupcount = 0)
        BEGIN
           
            SET @moveparm = REPLACE(@moveparm, 'NORECOVERY', 'RECOVERY')
               
        END
       
        --If there is a single full backup file, build the RESTORE statement with that file
        IF @fullfilecount = 1
        BEGIN
           
            SET @backupfile = (SELECT TOP 1 physicalfilename FROM #backupfiles)
           
            SET @sqlstr = 'RESTORE DATABASE ' + @restoredbname + ' FROM DISK = ' + '''' + @backupfile + '''' + @moveparm
            --PRINT @sqlstr
            DELETE TOP (1) FROM #backupfiles
               
        END
        --If there is more than one full backup file, build the RESTORE command for all full backup files
        ELSE IF @fullfilecount > 1
        BEGIN
               
            WHILE @backuploopcount <= @fullfilecount
            BEGIN
               
                SET @backupfile = (SELECT TOP 1 physicalfilename FROM #backupfiles WHERE backuptype = 'FULL')
                SET @fromstatement = @fromstatement + 'DISK = ' + '''' + @backupfile + '''' + ','
                DELETE TOP (1) FROM #backupfiles WHERE backuptype = 'FULL'
                SET @backuploopcount += 1
               
            END
               
            --Truncate the trailing comma from the FROM clause
            SET @fromstatement = SUBSTRING(@fromstatement,1,(LEN(@fromstatement) - 1))
            SET @sqlstr = 'RESTORE DATABASE ' + @restoredbname + @fromstatement + @moveparm
            SET @loopcount += 1
            --PRINT @sqlstr
               
        END
               
               
    END
        --One RESTORE statement will be built for the most recent differential backup
    ELSE IF @backupfiletype = 'DIFF'
    BEGIN
          
        IF @diffbackupcount = 1 AND @logbackupcount = 0
        BEGIN
          
            SET @moveparm = REPLACE(@moveparm, 'NORECOVERY', 'RECOVERY')
              
        END
       
        SET @backupfile = (SELECT TOP 1 physicalfilename FROM #backupfiles)
   
          
        SET @sqlstr = 'RESTORE DATABASE ' + @restoredbname + ' FROM DISK = ' + '''' + @backupfile + '''' + @moveparm
        --PRINT @sqlstr
        DELETE TOP (1) FROM #backupfiles
           
    END
    --If it is a log backup file, build a RESTORE log statement
    --One RESTORE statement will be built for each log backup file
    ELSE IF @backupfiletype = 'LOG'
    BEGIN
      
        SET @backupfile = (SELECT TOP 1 physicalfilename FROM #backupfiles)
   
        SET @sqlstr = 'RESTORE LOG ' + @restoredbname + ' FROM DISK = ' + '''' + @backupfile + ''''
           
        --If there is another log backup file, set the RESTORE to WITH NORECOVERY
        IF (SELECT COUNT(*) FROM #backupfiles) > 1
        BEGIN
             
            SET @sqlstr = @sqlstr + ' WITH NORECOVERY'
                                 
        END
        --If there is not another log backup file, set the RESTORE to WITH RECOVERY
        ELSE
        BEGIN
         
            IF @stopat IS NOT NULL
            BEGIN
                --CAST(@stopat AS VARCHAR(26))
                SET @sqlstr = @sqlstr + ' WITH NORECOVERY, STOPAT = ' + '''' + CONVERT(VARCHAR(30),@stopat,121) + ''''
             
            END
            ELSE
            BEGIN
             
                SET @sqlstr = @sqlstr + ' WITH RECOVERY'
                 
            END
           
        END
           
        DELETE TOP (1) FROM #backupfiles
   
       
    END
       
    --If @noexec = 1, PRINT @sqlstr
    IF @noexec = 1
    BEGIN
       
        PRINT @sqlstr
       
    END
    --If @noexec <> 1, execute @sqlstr
    ELSE
    BEGIN
       
        EXEC (@sqlstr)
           
    END
       
    --Increment the loop count and continue processing backup files
    SET @loopcount += 1
       
END

--If @stopat has been passed and a LOG backup exists, restore database with recovery 
IF @stopat IS NOT NULL AND @logbackupcount > 0
BEGIN
 
    SET @sqlstr = 'RESTORE DATABASE ' + @restoredbname + ' WITH RECOVERY'
     
    IF @noexec = 1
    BEGIN
     
        PRINT @sqlstr
         
    END
    ELSE
    BEGIN
     
        EXEC(@sqlstr)
         
    END
     
END
   
END TRY
BEGIN CATCH
  
    SELECT
    @sourcedbname AS DatabaseName,
    ERROR_NUMBER() AS ErrorNumber,
    ERROR_SEVERITY() AS ErrorSeverity,
    ERROR_STATE() as ErrorState,
    ERROR_PROCEDURE() as ErrorProcedure,
    ERROR_LINE() as ErrorLine,
    ERROR_MESSAGE() as ErrorMessage;
  
  
END CATCH 
GO

Stored Procedure to Automate Restores – Mark III

February 16, 2014 Leave a comment

sp_AutomateDatabaseRestores, now with STOPAT! I've added parameter @stopat to allow generation and execution of a RESTORE statement WITH STOPAT. Check the Presentation Slides and Scripts link for a script to demonstrate this functionality.

USE [master]
GO

/****** Object:  StoredProcedure [dbo].[sp_automateDBRestores]    Script Date: 2/16/2014 8:48:07 PM ******/
IF EXISTS(SELECT 1 FROM master.sys.objects WHERE name = 'sp_automateDBRestores')
BEGIN

	DROP PROCEDURE [dbo].[sp_automateDBRestores]

END
GO

/****** Object:  StoredProcedure [dbo].[sp_automateDBRestores]    Script Date: 2/16/2014 8:48:07 PM ******/
SET ANSI_NULLS ON
GO

SET QUOTED_IDENTIFIER ON
GO

CREATE PROCEDURE [dbo].[sp_automateDBRestores]
    @sourcedbname SYSNAME, 
    @restoredbname SYSNAME,
    @droprestoredb TINYINT = 1,
    @noexec TINYINT = 0,
    @stopat DATETIME = NULL
AS
      
SET NOCOUNT ON
 
BEGIN TRY
 
--Drop temp tables if they exist
IF OBJECT_ID('tempdb..#backupfiles') IS NOT NULL
BEGIN
  
    DROP TABLE #backupfiles
      
END
  
IF OBJECT_ID('tempdb..#databasefiles') IS NOT NULL
BEGIN
  
    DROP TABLE #databasefiles
      
END
  
--Create temp tables
CREATE TABLE #backupfiles
(backupstartdate DATETIME
,backupfinishdate DATETIME
,physicalfilename VARCHAR(2000)
,backuptype CHAR(4))
  
CREATE TABLE #databasefiles
(DatabaseName SYSNAME
,LogicalName SYSNAME
,PhysicalName VARCHAR(2000)
,filetype TINYINT)
  
--Declare local variables
DECLARE @filecount SMALLINT
DECLARE @fullfilecount SMALLINT
DECLARE @loopcount SMALLINT
DECLARE @datafileloopcount SMALLINT
DECLARE @backuploopcount SMALLINT
DECLARE @backupfile VARCHAR(2000)
DECLARE @sqlstr VARCHAR(MAX)
DECLARE @backupfiletype CHAR(4)
DECLARE @datafilecount SMALLINT
DECLARE @logfilecount SMALLINT
DECLARE @logbackupcount SMALLINT
DECLARE @diffbackupcount SMALLINT
DECLARE @datafile VARCHAR(2000)
DECLARE @moveparm VARCHAR(2000)
DECLARE @fromstatement VARCHAR(2000)
DECLARE @logfile SYSNAME
DECLARE @physlogfile VARCHAR(2000)
DECLARE @physdatafile VARCHAR(2000)
DECLARE @fullbackupdate DATETIME
DECLARE @logbackupdate DATETIME
DECLARE @logbackuplimit DATETIME
DECLARE @errormessage VARCHAR(1000)
  
--Drop the restored database if it exists and @droprestoredb = 1
IF EXISTS(SELECT 1 FROM sys.databases WHERE name = @restoredbname) AND @droprestoredb = 1
BEGIN
  
    SET @sqlstr = 'DROP DATABASE ' + @restoredbname
    EXEC (@sqlstr)
      
END

 
--Initialize variables
SET @moveparm = ' WITH'
SET @fromstatement = ' FROM '
SET @loopcount = 1
SET @datafileloopcount = 1
SET @backuploopcount = 1
  
  
--Build dynamic SQL to get a list of data and log files for the source database
SET @sqlstr = 'INSERT INTO #databasefiles SELECT ' + '''' + @sourcedbname + '''' + ' AS DatabaseName, name, physical_name, type 
FROM ' + @sourcedbname + '.sys.database_files'


--Execute dynamic SQL to return data and log files for the source database
EXEC (@sqlstr)
  
--Get counts of data files and log files
SELECT @datafilecount = COUNT(*) FROM #databasefiles WHERE filetype = 0
SELECT @logfilecount  = COUNT(*) FROM #databasefiles WHERE filetype = 1
 
IF @logfilecount > 1
BEGIN
 
    RAISERROR ('You have more than one log file, which is a really bad idea.  No backup for you.  See this link for more information.  http://www.sqlskills.com/blogs/paul/multiple-log-files-and-why-theyre-bad/', -- Message text.
    16, -- Severity.
    1 -- State.
    );
     
END
--Get the logical and physical log file names
SELECT @logfile = LogicalName, @physlogfile = REPLACE(PhysicalName,'.ldf','_testrestore.ldf')
FROM #databasefiles
WHERE filetype = 1

SELECT @physlogfile AS PhysLogFile
  
--If the database contains one data file, build the MOVE clause for a single file
IF @datafilecount = 1
BEGIN
	
	SELECT @datafile = LogicalName, @physdatafile = REPLACE(PhysicalName,'.mdf','_testrestore.mdf') 
    FROM #databasefiles
    WHERE filetype = 0
    
	SELECT @physdatafile AS PhysDataFile
    
    SET @moveparm = @moveparm + ' MOVE ' + '''' + @datafile + '''' + ' TO ' + '''' + @physdatafile + '''' + ',' + ' MOVE ' + '''' + @logfile + '''' + ' TO ' + '''' + @physlogfile + '''' + ', NORECOVERY'
END
ELSE
--If the database contains more than one data file, loop through the list of files and build a multi-file MOVE statement
IF @datafilecount > 1
BEGIN
      
    --SELECT @datafileloopcount, @datafilecount
    WHILE @datafileloopcount <= @datafilecount
    BEGIN
          
        --SELECT TOP (1) LogicalName, REPLACE(PhysicalName,@sourcedbname,@restoredbname) FROM #databasefiles WHERE filetype = 0
        SET @physdatafile = (SELECT TOP (1) REPLACE(PhysicalName,'.mdf','_testrestore.mdf') FROM #databasefiles WHERE filetype = 0)
        SET @datafile = (SELECT TOP (1) LogicalName FROM #databasefiles WHERE filetype = 0)     
        SET @moveparm = @moveparm + ' MOVE ' + '''' + @datafile + '''' + ' TO ' + '''' + @physdatafile + '''' + ','
        DELETE TOP (1) FROM #databasefiles WHERE filetype = 0
        SET @datafileloopcount += 1
          
    END
      
    --Append the MOVE command for the log file to the MOVE clause and set it to NORECOVERY
    SET @moveparm = @moveparm + ' MOVE ' + '''' + @logfile + '''' + ' TO ' + '''' + @physlogfile + '''' + ', NORECOVERY'
          
END
  
IF @droprestoredb = 0
BEGIN
  
    SET @moveparm = @moveparm + ', REPLACE'
      
END


SELECT @fullbackupdate = MAX(c.backup_finish_date)
		FROM msdb.dbo.backupset c 
		INNER JOIN msdb.dbo.backupmediafamily d
		ON c.media_set_id = d.media_set_id 
		WHERE c.database_name = @sourcedbname
		AND c.type = 'D'
        
SELECT @logbackupdate = MAX(c.backup_start_date)
		FROM msdb.dbo.backupset c 
		INNER JOIN msdb.dbo.backupmediafamily d
		ON c.media_set_id = d.media_set_id 
		WHERE c.database_name = @sourcedbname
		AND c.type IN ('D', 'I')

--Get the most recent full backup file for the source database and insert it into the #backupfiles temp table
INSERT INTO #backupfiles
SELECT
a.backup_start_date, a.backup_finish_date, b.physical_device_name, 'FULL' AS [backuptype]
from msdb.dbo.backupset a join msdb.dbo.backupmediafamily b
  on a.media_set_id = b.media_set_id
where a.database_name = @sourcedbname
AND backup_finish_date = @fullbackupdate
    --(SELECT MAX(c.backup_finish_date)
    --    FROM msdb.dbo.backupset c 
    --    INNER JOIN msdb.dbo.backupmediafamily d
    --    ON c.media_set_id = d.media_set_id 
    --    WHERE c.database_name = @sourcedbname
    --    AND c.type = 'D')
order by a.backup_finish_date ASC
 
--Get a list of differential backup files older than the most recent full backup
INSERT INTO #backupfiles
SELECT TOP 1
a.backup_start_date, a.backup_finish_date, b.physical_device_name, 'DIFF' AS [backuptype]
from msdb.dbo.backupset a join msdb.dbo.backupmediafamily b
  on a.media_set_id = b.media_set_id
where a.database_name = @sourcedbname
AND a.type = 'I'
AND backup_finish_date > @fullbackupdate
    --(SELECT MAX(c.backup_finish_date)
    --    FROM msdb.dbo.backupset c 
    --    INNER JOIN msdb.dbo.backupmediafamily d
    --    ON c.media_set_id = d.media_set_id 
    --    WHERE c.database_name = @sourcedbname
    --    AND c.type = 'D')
order by a.backup_finish_date DESC
 
--Get all log backupfiles run after the latest full or differential backup and insert it into the #backupfiles temp table
INSERT INTO #backupfiles
SELECT
a.backup_start_date, a.backup_finish_date, b.physical_device_name,'LOG'  AS [backuptype]
from msdb.dbo.backupset a join msdb.dbo.backupmediafamily b
  on a.media_set_id = b.media_set_id
where a.database_name = @sourcedbname
AND a.type = 'L'
AND backup_finish_date > @logbackupdate
    --(SELECT MAX(c.backup_start_date)
    --    FROM msdb.dbo.backupset c 
    --    INNER JOIN msdb.dbo.backupmediafamily d
    --    ON c.media_set_id = d.media_set_id 
    --    WHERE c.database_name = @sourcedbname
    --    AND c.type IN ('D', 'I'))
order by a.backup_finish_date ASC

IF @stopat < @fullbackupdate
BEGIN

	SET @errormessage = 'The @stopat values provided ' + CAST(@stopat AS VARCHAR(26)) + 'is older than the most recent full backup date of ' + CAST(@fullbackupdate AS VARCHAR(26)) + '.  Please check the value and rerun.'

    RAISERROR (@errormessage, -- Message text.
    16, -- Severity.
    1 -- State.
    );
    
END
	

IF @stopat IS NOT NULL
BEGIN

	SELECT TOP 1 @logbackuplimit = backupfinishdate FROM #backupfiles
	WHERE backupfinishdate > @stopat
	
	DELETE FROM #backupfiles 
	WHERE backupfinishdate > @logbackuplimit

	SELECT * FROM #backupfiles
	
END
 
--Get counts of all backup files, full backup files, differential backup files, and log backup files for use in loop processing below
SELECT @filecount = COUNT(*) FROM #backupfiles
SELECT @fullfilecount = COUNT(*) FROM #backupfiles WHERE backuptype = 'FULL'
SELECT @diffbackupcount = COUNT(*) FROM #backupfiles WHERE backuptype = 'DIFF'
SELECT @logbackupcount = COUNT(*) FROM #backupfiles WHERE backuptype = 'LOG'
 
IF @fullfilecount = 0 
BEGIN
 
    RAISERROR ('There are no full backup files for your database.', -- Message text.
    16, -- Severity.
    1 -- State.
    );
 
END
  
--While backup files exists, loop through them to generate the restore statement
WHILE @loopcount <= @filecount
BEGIN
     
    --Get the top backup file from the temp table   
    SET @backupfiletype = (SELECT TOP 1 backuptype FROM #backupfiles)
      
    --If it is a full backup file, build the restore statement
    IF @backupfiletype = 'FULL'
    BEGIN
 
        --If there are no log backups to apply, replace NORECOVERY with RECOVERY in the MOVE clause
        IF (@logbackupcount = 0 AND @diffbackupcount = 0)
        BEGIN
          
            SET @moveparm = REPLACE(@moveparm, 'NORECOVERY', 'RECOVERY')
              
        END
      
        --If there is a single full backup file, build the RESTORE statement with that file
        IF @fullfilecount = 1
        BEGIN
          
            SET @backupfile = (SELECT TOP 1 physicalfilename FROM #backupfiles)
          
            SET @sqlstr = 'RESTORE DATABASE ' + @restoredbname + ' FROM DISK = ' + '''' + @backupfile + '''' + @moveparm
            --PRINT @sqlstr
            DELETE TOP (1) FROM #backupfiles
              
        END
        --If there is more than one full backup file, build the RESTORE command for all full backup files
        ELSE IF @fullfilecount > 1
        BEGIN
              
            WHILE @backuploopcount <= @fullfilecount
            BEGIN
              
                SET @backupfile = (SELECT TOP 1 physicalfilename FROM #backupfiles WHERE backuptype = 'FULL')
                SET @fromstatement = @fromstatement + 'DISK = ' + '''' + @backupfile + '''' + ','
                DELETE TOP (1) FROM #backupfiles WHERE backuptype = 'FULL'
                SET @backuploopcount += 1
              
            END
              
            --Truncate the trailing comma from the FROM clause
            SET @fromstatement = SUBSTRING(@fromstatement,1,(LEN(@fromstatement) - 1))
            SET @sqlstr = 'RESTORE DATABASE ' + @restoredbname + @fromstatement + @moveparm
            SET @loopcount += 1
            --PRINT @sqlstr
              
        END
              
              
    END
        --One RESTORE datement will be built for the most recent differential backup
    ELSE IF @backupfiletype = 'DIFF'
    BEGIN
         
        IF @diffbackupcount = 1 AND @logbackupcount = 0
        BEGIN
         
            SET @moveparm = REPLACE(@moveparm, 'NORECOVERY', 'RECOVERY')
             
        END
      
        SET @backupfile = (SELECT TOP 1 physicalfilename FROM #backupfiles)
  
         
        SET @sqlstr = 'RESTORE DATABASE ' + @restoredbname + ' FROM DISK = ' + '''' + @backupfile + '''' + @moveparm
        --PRINT @sqlstr
        DELETE TOP (1) FROM #backupfiles
          
    END
    --If it is a log backup file, build a RESTORE log statement
    --One RESTORE datement will be built for each log backup file
    ELSE IF @backupfiletype = 'LOG'
    BEGIN
     
        SET @backupfile = (SELECT TOP 1 physicalfilename FROM #backupfiles)
  
        SET @sqlstr = 'RESTORE LOG ' + @restoredbname + ' FROM DISK = ' + '''' + @backupfile + ''''
          
        --If there is another log backup file, set the RESTORE to WITH NORECOVERY
        IF (SELECT COUNT(*) FROM #backupfiles) > 1
        BEGIN
			
			SET @sqlstr = @sqlstr + ' WITH NORECOVERY'
								
        END
        --If there is not another log backup file, set the RESTORE to WITH RECOVERY
        ELSE
        BEGIN
        
			IF @stopat IS NOT NULL
			BEGIN
				--CAST(@stopat AS VARCHAR(26))
				SET @sqlstr = @sqlstr + ' WITH NORECOVERY, STOPAT = ' + '''' + CONVERT(VARCHAR(30),@stopat,121) + ''''
			
			END
			ELSE
			BEGIN
			
				SET @sqlstr = @sqlstr + ' WITH RECOVERY'
				
			END
          
        END
          
        DELETE TOP (1) FROM #backupfiles
  
      
    END
      
    --If @noexec = 1, PRINT @sqlstr
    IF @noexec = 1
    BEGIN
      
        PRINT @sqlstr
      
    END
    --If @noexec <> 1, EXEC @sqlstr
    ELSE
    BEGIN
      
        EXEC (@sqlstr)
          
    END
      
    --Increment the loop count and continue processing backup files
    SET @loopcount += 1
      
END

IF @stopat IS NOT NULL
BEGIN

	SET @sqlstr = 'RESTORE DATABASE ' + @restoredbname + ' WITH RECOVERY'
	
	IF @noexec = 1
	BEGIN
	
		PRINT @sqlstr
		
	END
	ELSE
	BEGIN
	
		EXEC(@sqlstr)
		
	END
	
END
  
END TRY
BEGIN CATCH
 
    SELECT
    @sourcedbname AS DatabaseName,
    ERROR_NUMBER() AS ErrorNumber,
    ERROR_SEVERITY() AS ErrorSeverity,
    ERROR_STATE() as ErrorState,
    ERROR_PROCEDURE() as ErrorProcedure,
    ERROR_LINE() as ErrorLine,
    ERROR_MESSAGE() as ErrorMessage;
 
 
END CATCH 
GO

Stored Procedure to Delete Rows in Batches Based on a Date Column

August 8, 2013 2 comments

I've posted previously about log space reservation (here and here). I've also posted about running dynamic T-SQL in a WHILE loop (here). This morning, I wrote a procedure that combines the two topics.

Because SQL Server may have to roll an active transaction back, it reserves space in the log file in addition to the space used to hold the log records. This reserved space is released when the transaction commits.

One of the places I've seen this cause problems is a routine that deletes historical data from a table. Imagine you have an application that has been active for two years. It contains 150 million rows and has never been purged. A business decision is made to purge all data older than two months, which accounts for approximately 140 million rows. The simplest way to code this is as follows:

USE [databasename]
GO

DELETE FROM historytable
WHERE insertdate < DATEADD(MONTH,-2,GETDATE());
GO

This will work, but it will consume a huge amount of log space because all 140 million deletes will happen in a single transaction. So, in addition to the log space required for the deletes and any associated allocations, SQL Server will hold on to the reserved space for all 140 million records for the entirety of the transaction. A better solution is to delete records in smaller batches, with each delete wrapped in an explicit transaction.

That brings us to the procedure I wrote this morning for a situation very similar to the one I described above. The procedure will delete rows from a table based on the value of a date column. It allows you to set the number of rows to delete in each batch along with the threshold for deletion. See below for a sample execution.

-- =============================================
-- Create uspBatchDelete
-- Created : 2013-08-08
-- Created By: Frank Gill
-- Performs a batch delete of rows from a table 
-- Inputs - @tablename - the table to delete from
--			@datecolumn - the date column to compare against
--			@increment - the number of rows to delete in each batch
--          @numberofdays - the number of days of data you want to maintain
--			(unless your database has a time machine, the last parm should be negative)
-- =============================================

-- Drop stored procedure if it already exists
IF EXISTS (
  SELECT * 
    FROM INFORMATION_SCHEMA.ROUTINES 
   WHERE SPECIFIC_SCHEMA = N'dbo'
     AND SPECIFIC_NAME = N'usp_Batch_Delete' 
)
   DROP PROCEDURE dbo.uspBatchDelete
GO

CREATE PROCEDURE dbo.uspBatchDelete
	@tablename SYSNAME, 
	@datecolumn SYSNAME,
	@increment BIGINT,
	@numberofdays INT
AS
	
	BEGIN TRY
	
	-- Declare local variables
	DECLARE @sqlstr NVARCHAR(2000);
	DECLARE @rowcount BIGINT;
	DECLARE @loopcount BIGINT;
	DECLARE @ParmDefinition nvarchar(500);
	
	-- Set the parameters for the sp_executesql statement
	SET @ParmDefinition = N'@rowcountOUT BIGINT OUTPUT';
	-- Initialize the loop counter
	SET @loopcount = 1;
	
	-- Build the dynamic SQL string to return the row count
	-- Note that the input parameters are concatenated into the string, while the output parameter is contained in the string
	-- Also note that running a COUNT(*) on a large table can take a long time

	SET @sqlstr = N'SELECT @rowcountOUT = COUNT(*)
    FROM ' + @tablename + ' WITH (NOLOCK)
    WHERE ' + @datecolumn + ' < DATEADD(DAY,' + CAST(@numberofdays AS VARCHAR(4)) + ',GETDATE())';
	
	-- Execute the SQL String using sp_executesql, passing in the parameter definition and defining the output variable
	EXECUTE sp_executesql
    @sqlstr
    ,@ParmDefinition
    ,@rowcountOUT = @rowcount OUTPUT;
	 
	-- Perform the loop while there are rows to delete
	WHILE @loopcount <= @rowcount
	BEGIN
		 
		 -- Build a dynamic SQL string to delete rows
		 SET @sqlstr = 'DELETE TOP (' + CAST(@increment AS VARCHAR(10)) + ') FROM ' + @tablename + '
		 WHERE ' + @datecolumn + ' < DATEADD(DAY,' + CAST(@numberofdays AS VARCHAR(4)) + ',GETDATE())';
		 
		 -- Execute the dynamic SQL string to delete a batch of rows
		 EXEC(@sqlstr);
		 
		 -- Add the @increment value to @loopcount
		 SET @loopcount = @loopcount + @increment;
		 
		 PRINT CAST(@increment AS VARCHAR(10)) + ' rows deleted.'
		 
	END
	
	END TRY
	
	BEGIN CATCH
		
		SELECT
		ERROR_NUMBER() AS ErrorNumber
		,ERROR_SEVERITY() AS ErrorSeverity
		,ERROR_STATE() AS ErrorState
		,ERROR_PROCEDURE() AS ErrorProcedure
		,ERROR_LINE() AS ErrorLine
		,ERROR_MESSAGE() AS ErrorMessage;

	END CATCH
GO

In this example, the procedure will delete rows from tablename in batches of 50,000 where datecolumnname is older than two months. If you are dealing with a scenario similar to the one described above, deleting 22 months of data in a single run may take longer than you want. In that case, you can run the procedure multiple times changing the @numberofdays parm with each run. You will need to do some testing to determine the optimal values for the @increment and @numberofdays parameters.

-- =============================================
-- Example to execute the stored procedure
-- =============================================
EXECUTE dbo.uspBatchDelete @tablename = 'tablename',
@datecolumn = 'datecolumnname', 
@increment = 50000, 
@numberofdays = -60;
GO

I hope you can put this code to use.