Notesale: Turn your study into money

Document Preview

Extracts from the notes are below, to see the PDF you'll receive please use the links above

---

Expert SQL Server
2008 Development

Alastair Aitchison
Adam Machanic

Expert SQL Server 2008 Development
Copyright © 2009 by Alastair Aitchison and Adam Machanic
All rights reserved
...

ISBN-13 (pbk): 978-1-4302-7213-7
ISBN-13 (electronic): 978-1-4302-7212-0
Printed and bound in the United States of America 9 8 7 6 5 4 3 2 1
Trademarked names may appear in this book
...

President and Publisher: Paul Manning
Lead Editor: Jonathan Gennick
Technical Reviewer: Evan Terry
Editorial Board: Clay Andres, Steve Anglin, Mark Beckner, Ewan Buckingham, Gary Cornell,
Jonathan Gennick, Jonathan Hassell, Michelle Lowman, Matthew Moodie, Duncan Parkes,
Jeffrey Pepper, Frank Pohlmann, Douglas Pundick, Ben Renow-Clarke, Dominic Shakeshaft,
Matt Wade, Tom Welsh
Coordinating Editor: Mary Tobin
Copy Editor: Damon Larson
Compositor: Bytheway Publishing Services
Indexer: Barbara Palumbo
Artist: April Milne
Cover Designer: Anna Ishchenko
Distributed to the book trade worldwide by Springer-Verlag New York, Inc
...
Phone 1-800-SPRINGER, fax 201-348-4505, e-mail orders-ny@springersbm
...
springeronline
...

For information on translations, please e-mail info@apress
...
apress
...

Apress and friends of ED books may be purchased in bulk for academic, corporate, or promotional
use
...
For more information, reference our
Special Bulk Sales–eBook Licensing web page at http://www
...
com/info/bulksales
...
Although every
precaution has been taken in the preparation of this work, neither the author(s) nor Apress shall have
any liability to any person or entity with respect to any loss or damage caused or alleged to be caused
directly or indirectly by the information contained in this work
...
apress
...
You will need to
answer questions pertaining to this book in order to successfully download the code
...
iv
Contents
...
xvi
About the Technical Reviewer
...
xviii
Preface
...
1
Chapter 2: Best Practices for Database Programming
...
49
Chapter 4: Errors and Exceptions
...
101
Chapter 6: Encryption
...
159
Chapter 8: Dynamic T-SQL
...
235
Chapter 10: Working with Spatial Data
...
321
Chapter 12: Trees, Hierarchies, and Graphs
...
419

iv

Contents
Contents at a Glance
...
v
About the Author
...
xvii
Acknowledgments
...
xix
Chapter 1: Software Development Methodologies for the Database World
...
1
Coupling
...
4
Encapsulation
...
5
Interfaces As Contracts
...
6

Integrating Databases and Object-Oriented Systems
...
10
Business Logic
...
12

The “Object-Relational Impedance Mismatch”
...
13
Modeling Inheritance
...
17
v

CONTENTS

Introducing the Database-As-API Mindset
...
19
Performance
...
20
Maintainability
...
21
Allowing for Future Requirements
...
22
Best Practices for Database Programming
...
23
Defensive Programming
...
24
Why Use a Defensive Approach to Database Development?
...
28
Identify Hidden Assumptions in Your Code
...
33
Testing
...
39
Validate All Input
...
42
Limit Your Exposure
...
43
Comments
...
45
If All Else Fails
...
46
Summary
...
49
Approaches to Testing
...
50
Unit Testing Frameworks
...
55

Guidelines for Implementing Database Testing Processes and Procedures
...
56
What Kind of Testing Is Important?
...
57
Will Management Buy In?
...
58
Real-Time Client-Side Monitoring
...
60
System Monitoring
...
62
Extended Events
...
65

Analyzing Performance Data
...
67
Big-Picture Analysis
...
68
Fixing Problems: Is It Sufficient to Focus on the Obvious?
...
70
Chapter 4: Errors and Exceptions
...
Errors
...
72
Statement-Level Exceptions
...
73

vii

CONTENTS

Parsing and Scope-Resolution Exceptions
...
76
The XACT_ABORT Setting
...
78
Error Number
...
79
Error State
...
80
SQL Server’s RAISERROR Function
...
82
Creating Persistent Custom Error Messages
...
85
Monitoring Exception Events with Traces
...
85
Why Handle Exceptions in T-SQL?
...
86
SQL Server’s TRY/CATCH Syntax
...
89
Rethrowing Exceptions
...
91
Using TRY/CATCH to Build Retry Logic
...
93

Transactions and Exceptions
...
96
XACT_ABORT: Turning Myth into (Semi-)Reality
...
99

Summary
...
101
The Principle of Least Privilege
...
103
Server-Level Proxies
...
104
Data Security in Layers: The Onion Model
...
105
Basic Impersonation Using EXECUTE AS
...
110
Privilege Escalation Without Ownership Chains
...
112
Stored Procedure Signing Using Certificates
...
117

Summary
...
121
Do You Really Need Encryption?
...
121
What Are You Protecting Against?
...
123
The Automatic Key Management Hierarchy
...
124
Database Master Key
...
125
Alternative Encryption Management Structures
...
126
Removing Keys from the Automatic Encryption Hierarchy
...
127

Data Protection and Encryption Methods
...
129
Symmetric Key Encryption
...
134
Transparent Data Encryption
...
139
Implications of Encryption on Query Design
...
148
Wildcard Searches Using HMAC Substrings
...
157

Summary
...
159
Bridging the SQL/CLR Gap: The SqlTypes Library
...
161
The Problem
...
161
A Simple Example: E-Mail Address Format Validation
...
163
Security Exceptions
...
165
The Quest for Code Safety
...
168
Working with Host Protection Privileges
...
173
Granting Cross-Assembly Privileges
...
175
Strong Naming
...
TSQL
...
179
Calculating Running Aggregates
...
183

x

CONTENTS

Enhancing Service Broker Scale-Out with SQLCLR
...
185
XML Deserialization
...
187
Binary Deserialization
...
194
Chapter 8: Dynamic T-SQL
...
Ad Hoc T-SQL
...
Ad Hoc SQL Debate
...
197
Compilation and Parameterization
...
200
Application-Level Parameterization
...
203

Supporting Optional Parameters
...
206
Going Dynamic: Using EXECUTE
...
218
sp_executesql: A Better EXECUTE
...
223

Dynamic SQL Security Considerations
...
230
Interface Rules
...
232
Chapter 9: Designing Systems for Application Concurrency
...
236
Isolation Levels and Transactional Behavior
...
239
xi

CONTENTS

READ COMMITTED Isolation
...
239
SERIALIZABLE Isolation
...
241
READ UNCOMMITTED Isolation
...
242
From Isolation to Concurrency Control
...
243
Progressing to a Solution
...
249
Application Locks: Generalizing Pessimistic Concurrency
...
259
Embracing Conflict: Multivalue Concurrency Control
...
269
Controlling Resource Allocation
...
277
Controlling Concurrent Request Processing
...
281
Chapter 10: Working with Spatial Data
...
283
Spatial Reference Systems
...
286
Projected Coordinate Systems
...
288
Datum
...
288
Projection
...
290

xii

CONTENTS

Geography vs
...
292
Standards Compliance
...
294
Technical Limitations and Performance
...
296
Well-Known Text
...
297
Geography Markup Language
...
298

Querying Spatial Data
...
304
Finding Locations Within a Given Bounding Box
...
313
How Does a Spatial Index Work?
...
315

Summary
...
321
Modeling Time-Based Information
...
322
Input Date Formats
...
325
Efficiently Querying Date/Time Columns
...
329
Truncating the Time Portion of a datetime Value
...
332
How Many Candles on the Birthday Cake?
...
336
Dealing with Time Zones
...
343
Using the datetimeoffset Type
...
346
Modeling and Querying Continuous Intervals
...
354
Overlapping Intervals
...
362

Modeling Durations
...
366
Summary
...
371
Terminology: Everything Is a Graph
...
373
Constraining the Edges
...
376
Traversing the Graph
...
388
Finding Direct Descendants
...
391
Ordering the Output
...
396
Traversing up the Hierarchy
...
401
Deleting Existing Nodes
...
402

Persisted Materialized Paths
...
406
Navigating up the Hierarchy
...
408
Relocating Subtrees
...
411
Constraining the Hierarchy
...
412
Finding Subordinates
...
414
Inserting Nodes
...
416
Deleting Nodes
...
417

Summary
...
419

xv

About the Author
Alastair Aitchison is a freelance technology consultant based in Norwich, England
...
He has implemented various SQL Server
solutions requiring highly concurrent processes and large data warehouses in the financial services
sector, combined with reporting and analytical capability based on the Microsoft business intelligence
stack
...
He speaks at user groups and conferences, and is a highly active
contributor to several online support communities, including the Microsoft SQL Server Developer
Center forums
...
His past and current clients include the State
of Idaho, Albertsons, American Honda Motors, and Toyota Motor Sales, USA
...
He has also been the technical reviewer of several Apress books
relating to SQL Server databases
...
com
...
I am particularly lucky to have the assistance once more of two hugely talented
individuals, in the form of Jonathan Gennick and Evan Terry
...
Evan not only
provided the benefit of his wealth of technical knowledge, but also his authoring expertise, and at times
he simply provided a sensible voice of reason, all of which helped to improve the book significantly
...
Thank
you all
...
I couldn’t do anything without them
...
I hope that you find the content
interesting, useful, and above all, enjoyable to read
...
One thing I have noticed is that, with every
new release, SQL Server grows ever more powerful, and ever more complex
...
SQL Server developers
are no longer simply expected to be proficent in writing T-SQL code, but also in XML and SQLCLR (and
knowing when to use each)
...
The
types of information stored in modern databases represent not just character, numeric, and binary data,
but complex data such as spatial, hierarchical, and filestream data
...
Instead, I’m going to concentrate on
what I believe you need to know to create high-quality database applications, based on my own practical
experience
...
Nor will I insult your intelligence by laboriously
explaining the basics – I'll assume that you're already familiar with the straightforward examples covered
in Books Online, and now want to take your knowledge further
...
I
promise not to show you seemingly perfect solutions, which you then discover only work in the
artificially-cleansed "AdventureWorks" world; as developers we work with imperfect data, and I'll try to
show you examples that deal with the warts and all
...

Finally, I hope that you enjoy reading this book and thinking about the issues discussed
...
While you shouldn't let this search for perfection detract
you from the job at hand (sometimes, "good enough" is all you need), there are always new techniques
to learn, and alternative methods to explore
...


xix

CHAPTER 1

Software Development
Methodologies for the
Database World
Databases are software
...
Yet, all too often, the database is thought of as a
secondary entity when development teams discuss architecture and test plans, and many database
developers are still not aware of, or do not apply, standard software development best practices to
database applications
...
Many developers go beyond
simply persisting application data, instead creating applications that are data driven
...

Given this dependency upon data and databases, the developers who specialize in this field have no
choice but to become not only competent software developers, but also absolute experts at accessing
and managing data
...
Without the data, there is no need for the application
...
These pages stress rigorous testing, well-thoughtout architectures, and careful attention to interdependencies
...

In this chapter, I will present an overview of software development and architectural matters as they
apply to the world of database applications
...
Still, I encourage you to think carefully about these issues rather than
taking my—or anyone else’s—word as the absolute truth
...
Only through careful reflection on a case-by-case basis can you hope to identify and understand
the “best” possible solution for any given situation
...
The truth is that writing first-class software doesn’t involve nearly as
much complexity as many architects would lead you to believe
...
The three most important
concepts that every software developer must know in order to succeed are coupling, cohesion, and
encapsulation:
•

Coupling refers to the amount of dependency of one module within a system
upon another module in the same system
...
Modules, or systems, are said
to be tightly coupled when they depend on each other to such an extent that a
change in one necessitates a change to the other
...
Software developers
should strive instead to produce the opposite: loosely coupled modules and
systems, which can be easily isolated and amended without affecting the rest of
the system
...
Strongly
cohesive modules, which have only one function, are said to be more desirable
than weakly cohesive modules, which perform many operations and therefore
may be less maintainable and reusable
...
As you will see, this concept is essentially the
combination of loose coupling and strong cohesion
...


Unfortunately, these qualitative definitions are somewhat difficult to apply, and in real systems,
there is a significant amount of subjectivity involved in determining whether a given module is or is not
tightly coupled to some other module, whether a routine is cohesive, or whether logic is properly
encapsulated
...

Generally, developers will discuss these ideas using comparative terms—for instance, a module may be
said to be less tightly coupled to another module than it was before its interfaces were refactored
...
Let’s take a look at a couple of examples to
clarify things
...
This is one of those
areas that management teams tend to despise, because it adds no tangible value to the application from a
sales point of view, and entails revisiting sections of code that had previously been considered “finished
...
The following class might be defined to
model a car dealership’s stock (to keep the examples simple, I’ll give code listings in this section based
on a simplified and scaled-down C#-like syntax):

class Dealership
{
// Name of the dealership
string Name;
// Address of the dealership
string Address;
// Cars that the dealership has
Car[] Cars;
// Define the Car subclass
class Car
{
// Make of the car
string Make;
// Model of the car
string Model;
}
}
This class has three fields: the name of the dealership and address are both strings, but the
collection of the dealership’s cars is typed based on a subclass, Car
...
Take the owner of a car, for example:

class CarOwner
{
// Name of the car owner
string name;
// The car owner's cars
Dealership
...
Car; in order to own a car, it
seems to be presupposed that there must have been a dealership involved
...
Doing so would mean that a CarOwner would be coupled to a Car, as would a
Dealership—but a CarOwner and a Dealership would not be coupled at all
...


3

CHAPTER 1

SOFTWARE DEVELOPMENT METHODOLOGIES FOR THE DATABASE WORLD

Cohesion
To demonstrate the principle of cohesion, consider the following method that might be defined in a
banking application:

bool TransferFunds(
Account AccountFrom,
Account AccountTo,
decimal Amount)
{
if (AccountFrom
...
Balance -= Amount;
else
return(false);
AccountTo
...

That’s not much of a problem in itself, but now think of how much infrastructure (e
...
, error-handling
code) is missing from this method
...
The TransferFunds method has been made
weakly cohesive because, in performing a transfer, it requires the same functionality as provided by the
individual Withdraw and Deposit methods, only using completely different code
...
Should the withdrawal succeed, followed by an unsuccessful deposit, this
code as-is would result in the funds effectively vanishing into thin air
...
There
is no room for in-between—especially when you’re dealing with people’s funds!

Encapsulation
Of the three topics discussed in this section, encapsulation is probably the most important for a
database developer to understand
...
Balance >= Amount)
{
AccountFrom
...
But what if an error existed in Withdraw, and some code path allowed Balance to be
manipulated without first checking to make sure the funds existed? To avoid this situation, it should not
have been made possible to set the value for Balance from the Withdraw method directly
...
By doing so, the class would control its own data
and rules internally—and not have to rely on any consumer to properly do so
...


Interfaces
The only purpose of a module in an application is to do something at the request of a consumer (i
...
,
another module or system)
...
Therefore, a system must expose interfaces, well-known methods and properties
that other modules can use to make requests
...

Interface design is where the concepts of coupling and encapsulation really take on meaning
...
In such a
situation, any change to the module’s internal implementation may require a modification to the
implementation of the consumer
...
The contract
states that if the consumer specifies a certain set of parameters to the interface, a certain set of values
will be returned
...
For instance, a stored procedure that returns additional
columns if a user passes in a certain argument may be an example of a poorly designed interface
...
This means that the input
parameters are well defined, and the outputs are known at compile time
...
In these cases, it is up to the developer to ensure that the expected outputs are well
documented and that unit tests exist to validate them (see Chapter 3 for information on unit
testing)
...


Interface Design
Knowing how to measure successful interface design is a difficult question
...
If, in six months’ time, you were to completely
rewrite the module for performance or other reasons, can you ensure that all inputs and outputs will
remain the same?
For example, consider the following stored procedure signature:

CREATE PROCEDURE GetAllEmployeeData
--Columns to order by, comma-delimited
@OrderBy varchar(400) = NULL
Assume that this stored procedure does exactly what its name implies—it returns all data from the
Employees table, for every employee in the database
...

The interface issues here are fairly significant
...
In this case, a consumer of this stored procedure might expect that, internally, the commadelimited list will simply be appended to a dynamic SQL statement
...

Secondly, the consumer of this stored procedure must have a list of columns in the Employees table
in order to know the valid values that may be passed in the comma-delimited list
...
What about a Photo column, defined as varbinary(max), which
contains a JPEG image of the employee’s photo? Does it make sense to allow a consumer to specify that
column for sorting?
These kinds of interface issues can cause real problems from a maintenance point of view
...
And what should happen if the query is initially implemented as
dynamic SQL, but needs to be changed later to use static SQL in order to avoid recompilation costs? Will

6

CHAPTER 1

SOFTWARE DEVELOPMENT METHODOLOGIES FOR THE DATABASE WORLD

it be possible to detect which applications assumed that the ASC and DESC keywords could be used,
before they throw exceptions at runtime?
The central message I hope to have conveyed here is that extreme flexibility and solid, maintainable
interfaces may not go hand in hand in many situations
...
But remember that in most cases there are perfectly
sound workarounds that do not sacrifice any of the real flexibility intended by the original interface
...
One such version follows:

CREATE PROCEDURE GetAllEmployeeData
@OrderByName int = 0,
@OrderByNameASC bit = 1,
@OrderBySalary int = 0,
@OrderBySalaryASC bit = 1,
-- Other columns
...
So if a consumer passes a
value of 2 for the @OrderByName parameter and a value of 1 for the @OrderBySalary parameter, the result
will be sorted first by salary, and then by name
...

This version of the interface exposes nothing about the internal implementation of the stored
procedure
...
In addition, the consumer has no need for knowledge of the actual
column names of the Employees table
...
Or, there may be two columns, one containing a first name and one a last
name
...
Note that this same reasoning can also be applied
to suggest that end users and applications should only access data exposed as a view rather than directly
accessing base tables in the database
...

Note that this example only discussed inputs to the interface
...
g
...
I recommend always
using the AS keyword to create column aliases as necessary, so that interfaces can continue to return the
same outputs even if there are changes to the underlying tables
...
Doing so can create stored procedures that are difficult to test and maintain
...
Many methods throw
well-defined exceptions in certain situations, but if these exceptions are not adequately documented, their
well-intended purpose becomes rather wasted
...
It is almost always better to
follow a code path around a potential problem than to have to deal with an exception
...
Object-oriented frameworks and
database systems generally do not play well together, primarily because they have a different set of core
goals
...

It’s clear that we have two incompatible paradigms for modeling business entities
...
To that end, it’s important that database developers know what belongs where,
and when to pass the buck back up to their application developer brethren
...
How should you decide between implementation in the database vs
...
” Sadly, try as we might,
developers have still not figured out how to develop an application without the need to implement
business requirements
...
Does “business logic” belong in the database? In the
application tier? What about the user interface? And what impact do newer application architectures
have on this age-old question?

A Brief History of Logic Placement
Once upon a time, computers were simply called “computers
...
Back then there wasn’t much of a difference between an
application and its data, so there were few questions to ask, and fewer answers to give, about the
architectural issues we debate today
...
” Smaller and cheaper than the mainframes, the “minis” quickly
grew in popularity
...
Plus, these machines were
inexpensive enough that they could even be used directly by end users as an alternative to the previously
ubiquitous dumb terminals
...

The advent of the minis signaled multiple changes in the application architecture landscape
...
Instead of harnessing only the power of one server, workloads could now be
distributed in order to create more scalable applications
...
However, the client/server-based
architecture that had its genesis during the minicomputer era did not die; application developers found that
it could be much cheaper to offload work to clients than to purchase bigger servers
...
Web servers replaced the mainframe systems as centralized data and
UI systems, and browsers took on the role previously filled by the terminals
...

The latest trend toward cloud-based computing looks set to pose another serious challenge to the
traditional view of architectural design decisions
...

Vendors such as Amazon, Google, and Microsoft already offer cloud-based database services, but at the
time of writing, these are all still at a very embryonic stage
...
However, there is growing momentum
behind the move to the cloud, and it will be interesting to see what effect this has on data architecture
decisions over the next few years
...

Database developers must strive to ensure that data is sufficiently encapsulated to allow it to be
shared among multiple applications, while ensuring that the logic of disparate applications does not
collide and put the entire database into an inconsistent state
...

Rules and logic can be segmented into three basic groups:
•

Data logic

•

Business logic

•

Application logic

9

CHAPTER 1

SOFTWARE DEVELOPMENT METHODOLOGIES FOR THE DATABASE WORLD

Figure 1-1
...


Data Logic
Data logic defines the conditions that must be true for the data in the database to be in a consistent,
noncorrupt state
...
Data rules do not dictate
how the data can be manipulated or when it should be manipulated; rather, data rules dictate the state
that the data must end up in once any process is finished
...
Therefore, data rules must mirror all rules that drive the business
itself
...

In order to properly enforce this rule for both the current application and all possible future
applications, it must be implemented centrally, at the level of the data itself
...

As a general guideline, you should try to implement as many data rules as necessary in order to
avoid the possibility of data quality problems
...
Any
validation rule that is central to the business is central to the data, and vice versa
...


10

CHAPTER 1

SOFTWARE DEVELOPMENT METHODOLOGIES FOR THE DATABASE WORLD

Where Do the Data Rules Really Belong?
Many object-oriented zealots would argue that the correct solution is not a database at all, but rather an
interface bus, which acts as a façade over the database and takes control of all communications to and
from the database
...
First of all, this
approach completely ignores the idea of database-enforced data integrity and turns the database layer into
a mere storage container, failing to take advantage of any of the in-built features offered by almost all
modern databases designed specifically for that purpose
...
Writing such an interface layer may eliminate some database code, but it only defers the
necessity of working with the database
...
While applications and application architectures
come and go, databases seem to have an extremely long life in the enterprise
...
All of these issues are probably one big reason that although I’ve heard
architects argue this issue for years, I’ve never seen such a system implemented
...
In
other words, this term is overused and has no real meaning
...
Business logic, for the purpose of this text, is defined as any rule or process
that dictates how or when to manipulate data in order to change the state of the data, but that does not
dictate how to persist or validate the data
...
The raw data, which we might assume has already been
subjected to data logic rules, can be passed through business logic in order to determine the
aggregations and analyses appropriate for answering the questions that the end user might pose
...

So does business logic belong in the database? The answer is a definite “maybe
...
Other factors
(such as overall application architecture) notwithstanding, this means that in general practice you
should try to put the business logic in the tier in which it can deliver the best performance, or in which it
can be reused with the most ease
...


11

CHAPTER 1

SOFTWARE DEVELOPMENT METHODOLOGIES FOR THE DATABASE WORLD

Performance vs
...
Reality
Architecture purists might argue that performance should have no bearing on application design; it’s an
implementation detail, and can be solved at the code level
...

Performance is, in fact, inexorably tied to design in virtually every application
...
In many cases, these performance flaws can be identified—and fixed—during the design phase,
before they are allowed to materialize
...


Application Logic
If data logic definitely belongs in the database, and business logic may have a place in the database,
application logic is the set of rules that should be kept as far away from the central data as possible
...
Given the
application hierarchy discussed previously (one database that might be shared by many applications,
which in turn might be shared by many user interfaces), it’s clear that mingling user interface data with
application or central business data can raise severe coupling issues and ultimately reduce the
possibility for sharing of data
...

Doing so certainly makes sense for many applications
...

Whenever possible, make sure to create different tables, preferably in different schemas or even entirely
different databases, in order to store purely application-related data
...


The “Object-Relational Impedance Mismatch”
The primary stumbling block that makes it difficult to move information between object-oriented
systems and relational databases is that the two types of systems are incompatible from a basic design
point of view
...
Object-oriented systems, on the
other hand, tend to be much more lax in this area
...

For example, consider the following class, for a product in a retail system:

class Product
{
string UPC;
string Name;
string Description;
decimal Price;

12

CHAPTER 1

SOFTWARE DEVELOPMENT METHODOLOGIES FOR THE DATABASE WORLD

datetime UpdatedDate;
}
At first glance, the fields defined in this class seem to relate to one another quite readily, and one
might expect that they would always belong in a single table in a database
...

In the database, the data could be modeled as follows:

CREATE TABLE Products
(
UPC varchar(20) PRIMARY KEY,
Name varchar(50)
);
CREATE TABLE ProductHistory
(
UPC varchar(20) FOREIGN KEY REFERENCES Products (UPC),
Description varchar(100),
Price decimal,
UpdatedDate datetime,
PRIMARY KEY (UPC, UpdatedDate)
);
The important thing to note here is that the object representation of data may not have any bearing
on how the data happens to be modeled in the database, and vice versa
...


Are Tables Really Classes in Disguise?
It is sometimes stated in introductory database textbooks that tables can be compared to classes, and
rows to instances of a class (i
...
, objects)
...
They can also define (loosely) a set of methods for an
entity, in the form of triggers
...
The key foundations of an object-oriented system are
inheritance and polymorphism, both of which are difficult if not impossible to represent in SQL
databases
...
An entity in an object-oriented system can “have” a child entity, which is
generally accessed using a “dot” notation
...
Books;
In this object-oriented example, the bookstore “has” the books
...
Rather
than the bookstore having the books, the relationship between the entities is expressed the other way
around, where the books maintain a foreign key that points back to the bookstore:

CREATE TABLE BookStores
(
BookStoreId int PRIMARY KEY

13

CHAPTER 1

SOFTWARE DEVELOPMENT METHODOLOGIES FOR THE DATABASE WORLD

);
CREATE TABLE Books
(
BookStoreId int REFERENCES BookStores (BookStoreId),
BookName varchar(50),
Quantity int,
PRIMARY KEY (BookStoreId, BookName)
);
While the object-oriented and SQL representations can store the same information, they do so
differently enough that it does not make sense to say that a table represents a class, at least in current
SQL databases
...
g
...
g
...
In an SQL database, “has-a” relationships are quite common, whereas “is-a” relationships
can be difficult to achieve
...
This table may have columns (attributes) that typically belong to a product, such
as “price,” “weight,” and “UPC
...
However, the company may sell many subclasses of products, each with their own
specific sets of additional attributes
...

Subclassing in the object-oriented world is done via inheritance models that are implemented in
languages such as C#
...
This makes it possible to
seamlessly deal with both books and DVDs in the checkout part of a point-of-sale application, while
keeping separate attributes about each subclass for use in other parts of the application where they are
needed
...
The following code listing shows one way that
it can be approached:

CREATE TABLE Products
(
UPC int NOT NULL PRIMARY KEY,
Weight decimal NOT NULL,
Price decimal NOT NULL
);
CREATE TABLE Books
(
UPC int NOT NULL PRIMARY KEY
REFERENCES Products (UPC),
PageCount int NOT NULL
);

14

CHAPTER 1

SOFTWARE DEVELOPMENT METHODOLOGIES FOR THE DATABASE WORLD

CREATE TABLE DVDs
(
UPC int NOT NULL PRIMARY KEY
REFERENCES Products (UPC),
LengthInMinutes decimal NOT NULL,
Format varchar(4) NOT NULL
CHECK (Format IN ('NTSC', 'PAL'))
);
The database structure created using this code listing is illustrated in Figure 1-2
...
Modeling CREATE TABLE DVDs inheritance in a SQL database
Although this model successfully establishes books and DVDs as subtypes for products, it has a
couple of serious problems
...
A single UPC can belong to both the Books and DVDs subtypes simultaneously
...

Another issue is access to attributes
...
However, that is not the case in the model presented here
...
This really
breaks down the overall sense of working with a subtype
...
One method of guaranteeing
uniqueness among subtypes involves populating the supertype with an additional attribute identifying
the subtype of each instance
...
The relationship is
further enforced in each subtype table by a CHECK constraint on the ProductType column, ensuring that
only the correct product types are allowed to be inserted
...
A view
can be created for each subtype, which encapsulates the join necessary to retrieve the supertype’s
attributes
...
The indexing helps with
performance, and the triggers allow the views to be updateable
...
Mapping object-oriented data into a database (properly) is often not at all straightforward, and for
complex object graphs can be quite a challenge
...
This
design is fraught with issues and should be avoided
...
For example, it is impossible to look at the
table and find out what attributes belong to a book instead of a DVD
...
Furthermore, data
integrity is all but lost
...


16

CHAPTER 1

SOFTWARE DEVELOPMENT METHODOLOGIES FOR THE DATABASE WORLD

ORM: A Solution That Creates Many Problems
One solution to overcoming the problems that exist between relationship and object-oriented systems is
to turn to tools known as object-relational mappers (ORMs), which attempt to automatically map objects
to databases
...
Each of these tools comes with its own features and functions, but the basic idea is the same
in most cases: the developer “plugs” the ORM tool into an existing object-oriented system and tells the
tool which columns in the database map to each field of each class
...
This is all done automatically and somewhat seamlessly
...
These tools work based on the assumption that classes and tables can be mapped in oneto-one correspondence in most cases, which, as previously mentioned, is generally not true
...

One company I did some work for had used a popular Java-based ORM tool for its e-commerce
application
...
The
Java developers working for the company were forced to insert fake orders into the system in order to
allow the firm to sell new products
...
Aside from the issues with the tools that create
database tables based on classes, the two primary issues that concern me are both performance related:
First of all, ORM tools tend to think in terms of objects rather than collections of
related data (i
...
, tables)
...
This means that (depending on how
connection pooling is handled) a lot of database connections are opened and
closed on a regular basis, and the overall interface to retrieve the data is quite
“chatty
...

Second, query tuning may be difficult if ORM tools are relied upon too heavily
...
The current
crop of ORM tools does not intelligently monitor for and automatically fix possible
issues with poorly written queries, and developers using these tools are often taken
by surprise when the system fails to scale because of improperly written queries
...
However, even in the most recent version of the Microsoft Entity Framework
(
...
0 Beta 1), there are substantial deficiencies in the SQL code generated that lead to database
queries that are ugly at best, and frequently suboptimal
...


17

CHAPTER 1

SOFTWARE DEVELOPMENT METHODOLOGIES FOR THE DATABASE WORLD

Introducing the Database-As-API Mindset
By far the most important issue to be wary of when writing data interchange interfaces between object
systems and database systems is coupling
...
This is important in both worlds; if a change to the database
requires an application change, it can often be expensive to recompile and redeploy the application
...

To combat these issues, database developers must resolve to adhere rigidly to a solid set of
encapsulated interfaces between the database system and the objects
...

An application programming interface (API) is a set of interfaces that allows a system to interact
with another system
...

In database terms, this means that an API would expose public interfaces for retrieving data from,
inserting data into, and updating data in the database
...
This set of
interfaces should completely encapsulate all implementation details, including table and column
names, keys, indexes, and queries
...

In order to define such an interface, the first step is to define stored procedures for all external
database access
...
Stored procedures are the only construct available in SQL
Server that can provide the type of interfaces necessary for a comprehensive data API
...
Many software shops have discovered that web services are a good way to
provide a standard, cross-platform interface layer, such as using ADO
...
Whether using web services is superior to using other
protocols is something that must be decided on a per-case basis; like any other technology, they can
certainly be used in the wrong way or in the wrong scenario
...

By using stored procedures with correctly defined interfaces and full encapsulation of information,
coupling between the application and the database will be greatly reduced, resulting in a database
system that is much easier to maintain and evolve over time
...
In order to reinforce the idea that the database must be
thought of as an API rather than a persistence layer, this topic will be revisited throughout the book with
examples that deal with interfaces to outside systems
...
But, when developing a piece of software, there are hard limits that constrain what can actually
be achieved
...

The database is, in most cases, the center of the applications it drives
...

Likewise, the database is often where applications face real challenges in terms of performance,
maintainability, and other critical success factors
...

Attempting to strike the right balance generally involves a trade-off between the following areas:
•

Performance

•

Testability

•

Maintainability

•

Security

•

Allowing for future requirements

Balancing the demands of these competing facets is not an easy task
...


Performance
We live in an increasingly impatient society
...
We want fast food, fast cars, and fast service, and are constantly in
search of instant gratification of all types
...
Users continuously seem to feel that applications just aren’t performing as fast as they
should, even when those applications are doing a tremendous amount of work
...

The problem, of course, is that performance isn’t easy, and can throw the entire balance off
...
Functionality might have to be
trimmed (less work for the application to do means it will be faster), security might have to be reduced
(less authorization cycles means less work), or inefficient code might have to be rewritten in arcane,
unmaintainable ways in order to squeeze every last CPU cycle out of the server
...
Most of the time, if we find ourselves in a position in
which a user is complaining about performance and we’re going to lose money or a job if it’s not
remedied, the user doesn’t want to hear about why fixing the performance problem will increase
coupling and decrease maintainability
...


19

CHAPTER 1

SOFTWARE DEVELOPMENT METHODOLOGIES FOR THE DATABASE WORLD

A fortunate fact about sticking with best practices is that they’re often considered to be the best way
to do things for several reasons
...
And on those few occasions where
you need to break some “perfect” code to get it working as fast as necessary, know that it’s not your
fault—society put you in this position!

Testability
It is inadvisable, to say the least, to ship any product without thoroughly testing it
...
Many
of these problems result from attempts to produce “flexible” modules or interfaces—instead of properly
partitioning functionality and paying close attention to cohesion, it is sometimes tempting to create “allsinging, all-dancing,” monolithic routines that try to do it all
...
The
combinatorial explosion of possible use cases for a single routine can be immense—even though, in
most cases, the number of actual combinations that users of the application will exploit is far more
limited
...
Does it
really need to be that flexible? Will the functionality really be exploited in full right away, or can it be
slowly extended later as required?

Maintainability
Throughout the lifespan of an application, various modules and routines will require maintenance and
revision in the form of enhancements and bug fixes
...

When determining how testable a given routine is, we are generally only concerned with whether
the interface is stable enough to allow the authoring of test cases
...
From a
maintainability point of view, the most important interface issue is coupling
...

The issue of maintainability also goes beyond the interface into the actual implementation
...
Generally speaking, the more lines of code in a routine, the more difficult
maintenance becomes; but since large routines may also be a sign of a cohesion problem, such an issue
should be caught early in the design process if developers are paying attention
...
On one hand, flexibility of an interface can increase coupling between routines by requiring
the caller to have too much knowledge of parameter combinations, overrideable options, and the like
...
In some cases, making routines as generic as possible can result in fewer total
routines needed by a system, and therefore less code to maintain
...
Oftentimes, therefore, it may be advantageous
early in a project to aim for some flexibility, and then refactor later when maintainability begins to suffer
...
Breaking changes are not as much of an issue when tests exist that can
quickly validate new approaches to implementation
...
Security is, however, also one of the most
complex areas, and complexity can hide flaws that a trained attacker can easily exploit
...
From a testing standpoint, a developer needs to consider whether a given
security scheme will create too many variables to make testing feasible
...
The more complex a given routine is, the more difficult
(and therefore more expensive) it will be to maintain
...
The security responsibilities of the data tier or database will generally include areas such as
authentication to the application, authorization to view data, and availability of data
...


Allowing for Future Requirements
In a dynamic environment, where you face ever-changing customer requirements and additional feature
requests, it is easy to give too much attention to tomorrow’s enhancements, instead of concentrating on
today’s bugs
...
” Developers want to believe that adding complexity up front will
work to their advantage by allowing less work to be done later in the development process
...
These pieces of code must be carried around by the development team and kept
up to date in order to compile the application, but often go totally unused for years at a time
...
For example, according to Books Online,
the NUM_INPUT_PARAMS, NUM_OUTPUT_PARAMS, and NUM_RESULT_SETS returned by the sp_stored_procedures
stored procedure in SQL Server 2008 are reserved for future use (http://msdn
...
com/enus/library/ms190504
...
microsoft
...
90)
...

In one 15-year-old application I worked on, the initial development team had been especially active
in prepopulating the code base with features reserved for the future
...
The remaining
developers who had to deal with the 2 million–line application were afraid of removing anything lest it
would break some long-forgotten feature that some user still relied upon
...


21

CHAPTER 1

SOFTWARE DEVELOPMENT METHODOLOGIES FOR THE DATABASE WORLD

Although that example is extreme (certainly by far the worst I’ve come across), it teaches us to
adhere to the golden rule of software development: the KISS principle (keep it simple, stupid)
...
Adding new features tomorrow should
always be a secondary concern to delivering a robust, working product today
...

By understanding architectural concepts such as coupling, cohesion, and encapsulation, database
developers can define modular data interfaces that allow for great gains in ongoing maintenance and
testing
...

This chapter has provided an introduction to these ideas
...


22

CHAPTER 2

Best Practices for
Database Programming
Software development is not just a practical discipline performed by coders, but also an area of
academic research and theory
...
Various methodologies have emerged, including test-driven development
(TDD), agile and extreme programming (XP), and defensive programming, and there have been
countless arguments concerning the benefits afforded by each of these schools of thought
...
However, the topics discussed here
can be applied just as readily in any environment
...

I do not intend to provide an exhaustive, objective guide as to what constitutes best practice, but
rather to highlight some of the standards that I believe demonstrate the level of professionalism that
database developers require in order to do a good job
...


Defensive Programming
Defensive programming is a methodology used in software development that suggests that developers
should proactively anticipate and make allowances for (or “defend against”) unforeseen future events
...

Defensive programming essentially involves taking a pessimistic view of the world—if something
can go wrong, it will: network resources will become unavailable halfway through a transaction; required
files will be absent or corrupt; users will input data in any number of ways different from that expected,
and so on
...
This means that potential error conditions can be detected and handled before an
actual error occurs
...
In many

23

CHAPTER 2

BEST PRACTICES FOR DATABASE PROGRAMMING

cases, it may be possible to identify and isolate a particular component responsible for a failure, allowing
the rest of the application to continue functioning
...
Applications are not made
powerful and effective by their complexity, but by their elegant simplicity
...


•

“If it ain’t broke, fix it anyway
...


•

Be challenging, thorough, and cautious at all stages and development
...


•

Extensive code reviews and testing should be conducted with different peer
groups, including other developers or technical teams, consultants, end users, and
management
...


•

Assumptions should be avoided wherever possible
...


•

Applications should be built from short, highly cohesive, loosely coupled modules
...
Reusing
specific code modules, rather than duplicating functionality, reduces the chances
of introducing new bugs
...


Attitudes to Defensive Programming
The key advantages of taking a defensive approach to programming are essentially twofold:
•

•

24

Defensive applications are typically robust and stable, require fewer essential bug
fixes, and are more resilient to situations that may otherwise lead to expensive
failures or crashes
...

In many cases, defensive programming can lead to an improved user experience
...
Exceptions
can be isolated and handled with a minimum negative effect on user experience,
rather than propagating an entire system failure
...

However, as with any school of thought, defensive programming is not without its opponents
...
In each case, I’ve tried to give
a reasoned response to each criticism
...

It is certainly true that following a defensive methodology can result in a longer up-front development
time when compared to applications developed following other software practices
...
Coding itself takes longer
because additional code paths may need to be added to handle checks and assertions of assumptions
...
All these factors contribute to the fact that the overall development and release
cycle for defensive software is longer than in other approaches
...
However, this does not necessarily mean that defensive code takes
longer to develop when considered over the full life cycle of an application
...


Writing code that anticipates and handles every possible scenario makes defensive
applications bloated
...
Defensive
code protects against events that may be unlikely to happen, but that certainly doesn’t mean that they
can’t happen
...
Defensive applications may
contain more total lines of code than other applications, but all of that code should be well designed,
with a clear purpose
...
Such actions lead to code that is both complex and rigid
...


Defensive programming hides bugs that then go unfixed, rather than making them
visible
...
By explicitly
identifying and checking exceptional scenarios, defensive programming actually takes a very proactive
approach to the identification of errors
...
To demonstrate this in practical terms, consider the following code listing, which
describes a simple stored procedure to divide one number by another:
CREATE PROCEDURE Divide (
@x decimal(18,2),
@y decimal(18,2)
)
AS BEGIN
SELECT @x / @y
END;
GO
Based on the code as written previously, it would be very easy to cause an exception using this
procedure if, for example, the supplied value of @y was 0
...
Exception hiding such as this can be
very dangerous, and makes it almost impossible to ensure the correct functioning of an application
...
This means asserting such things as making
sure that values for @x and @y are supplied (i
...
, they are not NULL), that @y is not equal to zero, that the
supplied values lie within the range that can be stored within the decimal(18,2) datatype, and so on
...
In real life,
these code paths may handle such assertions in a number of ways—typically logging the error, reporting
a message to the user, and attempting to continue system operation if it is possible to do so
...

Defensive programming can be contrasted to the fail fast methodology, which focuses on
immediate recognition of any errors encountered by causing the application to halt whenever an
exception occurs
...


Why Use a Defensive Approach to Database Development?
As stated previously, defensive programming is not the only software development methodology that
can be applied to database development
...
So why have I chosen to focus on just defensive programming in this chapter, and
throughout this book in general? I believe that defensive programming is the most appropriate approach
for database development for the following reasons:
Database applications tend to have a longer expected lifespan than other
software applications
...
Web applications, for example, may
be revised and relaunched on a nearly annual basis, in order to take advantage of
whatever technology is current at the time
...
As a
result, it is easier to justify the greater up-front development cost associated with
defensive programming
...

Users (and management) are less tolerant of bugs in database applications
...
While undoubtedly a cause of frustration, many people are routinely in

27

CHAPTER 2

BEST PRACTICES FOR DATABASE PROGRAMMING

the habit of hitting Ctrl+Alt+Delete to reset their machine when a web browser
hangs, or because some application fails to shut down correctly
...
Recent highly publicized scandals in which
bugs have been exploited in the systems of several governments and large
organizations have further heightened the general public’s ultrasensitivity toward
anything that might present a risk to database integrity
...
It can be argued that people are absolutely
right to be more worried about database bugs than bugs in other software
...
But an
unexpected error in a database may lead to important personal, confidential, or
sensitive data being placed at risk, which can have rather more serious
consequences
...


Designing for Longevity
Consumer software applications have an increasingly short expected shelf life, with compressed release
cycles pushing out one release barely before the predecessor has hit the shelves
...
Well-designed, defensively programmed applications can continue to operate for
many years
...
Despite only being required for an immediate post-merger
period, the (rather unfortunately named) Short Term Management Information database continued to be
used for up to ten years later, as it remained more reliable and robust than subsequent attempted
replacements
...
As in any methodology, defensive programming is more concerned with the mindset with
which you should approach development than prescribing a definitive set of rules to follow
...
I’ll try to keep the actual examples as simple as possible in every case, so
that you can concentrate on the reasons I consider these to be best practices, rather than the code itself
...
Once these assumptions have been identified, the function can either
be adjusted to remove the dependency on them, or explicitly test each condition and make provisions
should it not hold true
...

To demonstrate this concept, consider the following code listing, which creates and populates a
Customers and an Orders table:
CREATE TABLE Customers(
CustID int,
Name varchar(32),
Address varchar(255));
INSERT INTO Customers(CustID, Name, Address) VALUES
(1, 'Bob Smith', 'Flat 1, 27 Heigham Street'),
(2, 'Tony James', '87 Long Road');
GO
CREATE TABLE Orders(
OrderID INT,
CustID INT,
OrderDate DATE);
INSERT INTO Orders(OrderID, CustID, OrderDate) VALUES
(1, 1, '2008-01-01'),
(2, 1, '2008-03-04'),
(3, 2, '2008-03-07');
GO
Now consider the following query to select a list of every customer order, which uses columns from
both tables:
SELECT
Name,
Address,
OrderID
FROM
Customers c
JOIN Orders o ON c
...
CustID;
GO

29

CHAPTER 2

BEST PRACTICES FOR DATABASE PROGRAMMING

The query executes successfully and we get the results expected:
Bob Smith

Flat 1, 27 Heigham Street

1

Bob Smith

Flat 1, 27 Heigham Street

2

Tony James

87 Long Road

3

But what is the hidden assumption? The column names listed in the SELECT query were not qualified
with table names, so what would happen if the table structure were to change in the future? Suppose
that an Address column were added to the Orders table to enable a separate delivery address to be
attached to each order, rather than relying on the address in the Customers table:
ALTER TABLE Orders ADD Address varchar(255);
GO
The unqualified column name, Address, specified in the SELECT query, is now ambiguous, and if we
attempt to run the original query again we receive an error:
Msg 209, Level 16, State 1, Line 1
Ambiguous column name 'Address'
...
The simple
practice that could have prevented this error would have been to ensure that all column names were
prefixed with the appropriate table name or alias:
SELECT
c
...
Address,
o
...
CustID = o
...
However, sometimes you may not be so fortunate, as shown in the following example
...
The following code demonstrates this structure, together with a mechanism to
automatically populate the ChangeLog table by means of an UPDATE trigger attached to the MainData table:
CREATE TABLE ChangeLog(
ChangeID int IDENTITY(1,1),
RowID int,
OldValue char(3),
NewValue char(3),
ChangeDate datetime);
GO
CREATE TRIGGER DataUpdate ON MainData
FOR UPDATE
AS
DECLARE @ID int;
SELECT @ID = ID FROM INSERTED;
DECLARE @OldValue varchar(32);
SELECT @OldValue = Value FROM DELETED;
DECLARE @NewValue varchar(32);
SELECT @NewValue = Value FROM INSERTED;
INSERT INTO ChangeLog(RowID, OldValue, NewValue, ChangeDate)
VALUES(@ID, @OldValue, @NewValue, GetDate());
GO
We can test the trigger by running a simple UPDATE query against the MainData table:
UPDATE MainData SET Value = 'aaa' WHERE ID = 1;
GO
The query appears to be functioning correctly—SQL Server Management Studio reports the following:
(1 row(s) affected)

(1 row(s) affected)

31

CHAPTER 2

BEST PRACTICES FOR DATABASE PROGRAMMING

And, as expected, we find that one row has been updated in the MainData table:
ID

Value

1

aaa

2

def

3

ghi

4

jkl

and an associated row has been created in the ChangeLog table:
ChangeID

RowID

OldValue

NewValue

ChangeDate

1

1

abc

aaa

2009-06-15 14:11:09
...
Within the trigger logic, the
variables @ID, @OldValue, and @NewValue are assigned values that will be inserted into the ChangeLog table
...
770

2

2

def

zzz

2009-06-15 15:18:11
...
Had this scenario been actively considered, it would have been easy to recode the procedure to
deal with such an event by making a subtle alteration to the trigger syntax, as shown here:
ALTER TRIGGER DataUpdate ON MainData
FOR UPDATE
AS
INSERT INTO ChangeLog(RowID, OldValue, NewValue, ChangeDate)
SELECT i
...
Value, i
...
ID = d
...
In programming terms, there are often shortcuts that provide a convenient, concise
way of achieving a given task in fewer lines of code than other, more standard methods
...
Most commonly, shortcut methods require less code
because they rely on some assumed default values rather than those explicitly stated within the
procedure
...

By relying on a default value, shortcut methods may increase the rigidity of your code and also
introduce an external dependency—the default value may vary depending on server configuration, or

33

CHAPTER 2

BEST PRACTICES FOR DATABASE PROGRAMMING

change between different versions of SQL Server
...

To demonstrate, consider what happens when you CAST a value to a varchar datatype without
explicitly declaring the appropriate data length:
SELECT CAST ('This example seems to work ok' AS varchar);
GO
The query appears to work correctly, and results in the following output:
This example seems to work ok
It seems to be a common misunderstanding among some developers that omitting the length for
the varchar type as the target of a CAST operation results in SQL Server dynamically assigning a length
sufficient to accommodate all of the characters of the input
...
In the second example, the input string is silently truncated to 30 characters, even though
there is no obvious indication in the code to this effect
...

Another example of a shortcut sometimes made is to rely on implicit CASTs between datatypes
...
9 * @x / @y;

SELECT 1000 * @Rate;
GO
In this example, @Rate is a multiplicative factor whose value is determined by the ratio of two
parameters, @x and @y, multiplied by a hard-coded scale factor of 1
...
When applied to the value 1000, as
in this example, the result is as follows:
1060

34

CHAPTER 2

BEST PRACTICES FOR DATABASE PROGRAMMING

Now let’s suppose that management makes a decision to change the calculation used to determine
@Rate, and increases the scale factor from 1
...
The obvious (but incorrect) solution would be to
amend the code as follows:
DECLARE
@x int = 5,
@y int = 9,
@Rate decimal(18,2);
SET @Rate = 2 * @x / @y;
SELECT 1000 * @Rate;
GO
1000
Rather than increasing the rate as intended, the change has actually negated the effect of applying
any rate to the supplied value of 1000
...
In integer mathematics, this equates to 1
...
9 caused an implicit cast of both @x and @y parameters to the decimal type, so
the sum was calculated with decimal precision
...
To avoid
these complications, it is always best to explicitly state the type and precision of any parameters used in
a calculation, and avoid implicit CASTs between them
...
If we cannot tell what a line of code is meant to do, it is incredibly hard to test
whether it is achieving its purpose or not
...
But there are actually several
shortcuts used here that add ambiguity as to what the intended purpose of this code is:
The first shortcut is in the implicit CAST from the string value '03/05/1979' to a
datetime
...
In the United Kingdom it
means the 3rd of May, but to American readers it means the 5th of March
...

Even if the dd/mm/yyyy or mm/dd/yyyy ordering is resolved, there is still
ambiguity regarding the input value
...

However, perhaps it was not the developer’s intention to specify an instance in
time, but rather the whole of a calendar day
...

When applied to a date value, the + operator adds the given number of days, so this
appears to be a shortcut in place of using the DATEADD method to add 365 days
...

The combination of these factors has meant that it is unclear whether the true intention of this simple
line of code is
SELECT DATEADD(DAY, 365, '1979-03-05');
which leads to the following result:
1980-03-04 00:00:00
...


Perhaps the most well-known example of a shortcut method is the use of SELECT * in order to
retrieve every column of data from a table, rather than listing the individual columns by name
...
At best, this may result in
columns of data being retrieved that are then never used, leading to inefficiency
...
There are many other reasons why SELECT * should be avoided, such as the addition of
unnecessary rows to the query precluding the use of covering indexes, which may lead to a substantial
degradation in query performance
...
In order to defend against situations that might occur in a live
production environment, an application should be tested under the same conditions that it will
experience in the real world
...
In addition to performance testing, there are functional tests and unit tests to consider,
which ensure that every part of the application is behaving as expected according to its contract, and
performing the correct function
...

When testing an application, it is important to consider the sample data on which tests will be
based
...
If the application is expected to perform against production data, then it
should be tested against a fair representation of that data, warts and all
...
Random sampling methods can be
used to ensure that the test data represents a fair sample of the overall data set, but it is also important
for defensive testing to ensure that applications are tested against extreme edge cases, as it is these
unusual conditions that may otherwise lead to exceptions
...
Some exceptional
circumstances only arise in a full-scale environment
...
Nor should you simply assume that the performance of your
application will scale predictably with the number of rows of data involved
...
The performance
of other applications, however, may degrade exponentially (such as when working with Cartesian
products created from CROSS JOINs between tables)
...

Another consideration when testing is the effect of multiple users on a system
...
However,
these same tests can fail in the presence of concurrency—that is, multiple requests for the same resource
on the database
...

CREATE TABLE XandY (
x int,
y int,
v rowversion);
INSERT INTO XandY (x, y) VALUES (0, 0);
GO
The following code executes a loop that reads the current values from the XandY table, increments
the value of x by 1, and then writes the new values back to the table
...

SET NOCOUNT ON;
DECLARE
@x int,
@y int,
@v rowversion,

37

CHAPTER 2

BEST PRACTICES FOR DATABASE PROGRAMMING

@success int = 0;
WHILE @success < 100000
BEGIN
-- Retrieve existing values
SELECT
@x = x,
@y = y,
@v = v
FROM XandY
-- Increase x by 1
SET @x = @x + 1;
SET TRANSACTION ISOLATION LEVEL READ COMMITTED;
BEGIN TRANSACTION
IF EXISTS(SELECT 1 FROM XandY WHERE v = @v)
BEGIN
UPDATE XandY
SET
x = @x,
y = @y
WHERE v = @v;
SET @success = @success + 1;
END
COMMIT;
END
GO
Executing this code leads, as you’d expect, to the value of the x column being increased to 100,000:
x

y

v

100000

0

0x00000000001EA0B9

Now let’s try running the same query in a concurrent situation
...
It then writes both values back to the table, as before
...
While it is still executing, execute the
second script, which updates the value of y
...
An explanation of why
this has occurred, and methods to deal with such situations, will be explained in Chapter 9
...
The two activities of automated testing
and human code review are complementary and can detect different areas for code improvement
...
In these cases, code review is a more effective approach
...
'
AND CHARINDEX('
...
',REVERSE(LTRIM(RTRIM(@email_address)))) >= 3
AND (CHARINDEX('
...
',@email_address ) = 0)
)
PRINT 'The supplied email address is valid';
ELSE
PRINT 'The supplied email address is not valid';
This code might well pass functional tests to suggest that, based on a set of test email addresses
provided, the function correctly identifies whether the format of a supplied e-mail address is valid
...
NET Base Class Library, such as shown here:
SELECT dbo
...
_%+-]+@[A-Z0-9
...
NET System
...
RegularExpressions
...
While both
methods achieve the same end result, rewriting the code in this way creates a routine that is more
efficient and maintainable, and also promotes reusability, since the suggested RegExMatch function could
be used to match regular expression patterns in other situations, such as checking whether a phone
number is valid
...
One of the advantages of well-encapsulated code is that
those modules that are most likely to benefit from the exercise can be isolated and reviewed separately
from the rest of the application
...
g
...


40

CHAPTER 2

BEST PRACTICES FOR DATABASE PROGRAMMING

A good defensive stance is to assume that all input is invalid and may lead to exceptional
circumstances unless proved otherwise
...
” For example, bad characters can be replaced or escaped
...
Silently
modifying input affects data integrity and is generally not recommended unless it
cannot be avoided
...
For example, input should not
be allowed to contain SQL keywords such as DELETE or DROP, or contain
nonalphanumeric characters
...
From a UI point of view, you can consider this as equivalent to
allowing users to only select values from a predefined drop-down list, rather than
a free-text box
...


All of these approaches are susceptible to flaws
...
You might expect the
result of the following to reject the input:
DECLARE @Input varchar(32) = '10E2';
SELECT ISNUMERIC(@Input);
Most exceptions occur as the result of unforeseen but essentially benign circumstances
...
Perhaps the most widely known defensive programming techniques concern the prevention
of SQL injection attacks
...

SQL injection attacks typically take advantage of poorly implemented functions that construct and
execute dynamic SQL-based on unvalidated user input
...
sysusers WHERE name = ''' + @Input +
'''';
EXECUTE(@sql);
END
The intended purpose of this code is fairly straightforward—it returns the status of the user supplied
in the parameter @Input
...
sysusers WHERE name = 'public' OR 1 = 1;
The condition OR 1 = 1 appended to the end of the query will always evaluate to true, so the effect
will be to make the query list every row in the sys
...

Despite this being a simple and well-known weakness, it is still alarmingly common
...


Future-proof Your Code
In order to prevent the risk of bugs appearing, it makes sense to ensure that any defensive code adheres
to the latest standards
...

Deprecated features refer to features that, while still currently in use, have been superseded by
alternative replacements
...
Consider the
following code listing:
CREATE TABLE ExpertSqlServerDevelopment
...
Deprecated (
EmployeeID int DEFAULT 0,
Forename varchar(32) DEFAULT '',
Surname varchar(32) DEFAULT '',
Photo image NULL
);
CREATE INDEX ixDeprecated ON Deprecated(EmployeeID);
DROP INDEX Deprecated
...
dbo
...
dbo
...
dbo
...
EmployeeID
SET ROWCOUNT 0;
This query works as expected in SQL Server 2008, but makes use of a number of deprecated features,
which should be avoided
...
dm_os_performance_counters dynamic management view (DMV) maintains a count of every time a
deprecated feature is used, and can be interrogated as follows:
SELECT
object_name,
instance_name,
cntr_value
FROM sys
...
Many such features exist in SQL Server—the following code listing
demonstrates the undocumented sp_MSForEachTable stored procedure, for example, which can be used
to execute a supplied query against every table in a database
...
Undocumented features, in contrast, may break at any time without warning, and there may be
no clear upgrade path
...


Limit Your Exposure
If defensive programming is designed to ensure that an application can cope with the occurrence of
exceptional events, one basic defensive technique is to limit the number of such events that can occur
...
Don’t grant EXTERNAL_ACCESS to an assembly when SAFE will do
...

All users should be authenticated, and only authorized to access those resources that are required,
for the period of time for which they are required
...
Doing so reduces the chance of the system being compromised
by an attack, and is discussed in more detail in Chapter 5
...
Different encryption methods are discussed in Chapter 6
...
I have chosen to include it here, partly because I
consider it so vital that it can never be restated too often, but also because the nature of defensive
programming emphasizes these areas more than other approaches, for the following reasons:

43

CHAPTER 2

BEST PRACTICES FOR DATABASE PROGRAMMING

As stated previously, the aim of defensive programming is to minimize the risk of
errors occurring as a result of future unforeseen events
...
By
creating clear, well-documented code now, you enhance its future
understandability, reducing the chances that bugs will be accidentally introduced
when it is next addressed
...
When they are next reviewed some
years later, the development team responsible may be very different, or the original
developers may no longer remember why a certain approach was taken
...

Code that is well laid out often goes hand in hand with code that is well thought
out
...
Most IDEs and code editors provide layout features
that will automatically apply a consistent format for tabs, whitespace,
capitalization and so on, and these settings can normally be customized to match
whatever coding standards are in place in a given organization
...
If the code needs to be revised, it will be much easier to quickly establish
the best method to do so
...

For these reasons, I believe exercising good code etiquette to be a key part of defensive programming
...


Comments
Everybody knows that comments are an important part of any code, and yet few of us comment our
code as well as we should (one reason commonly put forward is that developers prefer to write code
rather than writing about code)
...
Well-written
comments make it easier to tell what a function is aiming to achieve and why it has been written a
certain way, which by implication means that it is easier to spot any bugs or assumptions made that
could break that code
...
The following
comment, for example, is not helpful:
-- Set x to 5
SET @x = 5;

44

CHAPTER 2

BEST PRACTICES FOR DATABASE PROGRAMMING

In general, comments should explain why a certain approach has been taken and what the
developer is aiming to achieve
...
In general it is not necessary to simply comment what a built-in
function does, but there may be exceptions to this rule
...
In
case you haven’t figured it out, the result gives you the date of Easter Sunday in any given year (specified
using the variable @y)
...
Consider the following code:
SELECT DATEPART(Y, '20090617');
In most programming languages, the character Y used in a date format function denotes the year
associated with a date
...
To explain the actual result of
168, the code could have easily been made self-documenting by replacing the Y with DAYOFYEAR (for
which it is an abbreviation):
SELECT DATEPART(DAYOFYEAR, '20090617');

Indentations and Statement Blocks
Code indentations and liberal use of whitespace can help to identify logical blocks of code, loops, and
batches, creating code that is understandable, easily maintained, and less likely to have bugs introduced
in the future
...
It
is therefore vitally important that the visual layout of code reinforces its logical behavior, as poorly
presented code may actually be misleading
...

To avoid such misleading situations, I always recommend the liberal use of statement blocks
marked by BEGIN and END, even if a block contains only one statement, as follows:
IF 1 = 1
BEGIN
PRINT 'True';
END
ELSE
BEGIN
PRINT 'False';
END
PRINT 'Then Print This';
Another misleading practice that can easily be avoided is the failure to use parentheses to explicitly
demonstrate the order in which the components of a query are resolved
...


If All Else Fails
...
It can be argued that, if the ideal of defensive programming were ever truly
realized, it would not be necessary to implement exception-handling code, since any potential scenarios
that could lead to exceptions would have been identified and handled before they were allowed to occur
...
For a
detailed discussion of exception and error handling in SQL Server, please refer to Chapter 4
...
Successful defensive development is most
likely to occur when coding is a shared, open activity
...
Different people will be able to critically
examine code from a number of different points of view, which helps to identify any assumptions that
might have gone unnoticed by a single developer
...
If only one developer knows the intricacies of a particularly complex section
of code and then that developer leaves or is unavailable, you may encounter difficulties maintaining that
code in the future
...
Coders may seek to ensure that only they understand
how a particular section of complex code works, either as a way of flaunting their technical knowledge,
for reasons of personal pride, or as a way of creating a dependence on them—making themselves
indispensable and ensuring their future job security
...

Managers responsible for development teams should try to foster an environment of continued
professional development, in which shared learning and best practice are key
...
In order to make sure that applications remain cutting edge, individual training of developers and
knowledge-sharing between peers should be promoted and encouraged
...
For example, a company may implement a
reward scheme that pays individual bonuses for any developer that discovers and solves bugs in live
applications
...

Another factor affecting the success of defensive development concerns the way in which budget
and project deadlines are managed
...
It is an unfortunate fact that, when deadlines are brought forward or budgets slashed, it is
defensive practices (such as rigorous testing) that management regard as nonessential, and are among
the first to be dropped from the scope of the project
...
These are unlikely to use defensive
programming and will not stand up to rigorous testing
...
For these reasons, true defensive
programming might be seen as an ideal, rather than an achievable objective
...
Given the typical expected lifespan of database applications
and the potential severity of the consequences should a bug occur, it makes sense to adopt a defensive

47

CHAPTER 2

BEST PRACTICES FOR DATABASE PROGRAMMING

approach to ensure that the applications remain robust over a long period of time, and that the need for
ongoing maintenance is kept to a minimum
...

Throughout the rest of the book, I will continue to show in more detail how to adopt a defensive stance
across a range of development scenarios
...
The hallmark of a truly great developer, and what allows these
qualities to shine through, is a thorough understanding of the importance of testing
...
Carefully designed functional tests ensure compliance
with business requirements
...

Unfortunately, like various other practices that are better established in the application
development community, testing hasn’t yet caught on much with database professionals
...

There is no good reason that database developers should not write just as many—or more—tests
than their application developer counterparts
...
Software testing is a huge field, complete
with much of its own lingo, so my intention is to concentrate only on those areas that I believe to be
most important for database developers
...
The internal workings of the module are not exposed to (or required
by) the tester—hence they are contained within a “black box
...
White box testing is also called “open-box” testing, as the tester is
allowed to look inside the module to see how it operates, rather than just
examining its inputs and outputs
...
Examples of black box tests include unit tests, security
tests, and basic performance tests such as stress tests and endurance tests
...

From a database development perspective, examples of white box tests include functional tests that
validate the internal working of a module, tests that perform data validation, and cases when
performance tuning requires thorough knowledge of data access methods
...


Unit and Functional Testing
Developing software with a specific concentration on the data tier can have a benefit when it comes to
testing: there aren’t too many types of tests that you need to be familiar with
...
This is the purpose of unit tests and functional tests
...
For
instance, a unit test of a stored procedure should validate that, given a certain set of
inputs, the stored procedure returns the correct set of output results, as defined by
the interface of the stored procedure being tested
...
It means “correct” only insofar as what is defined as
the contract for the stored procedure; the actual data returned is not important
...
Phrased another
way, unit tests test the ability of interfaces to communicate with the outside world
exactly as their contracts say they will
...
In testing nomenclature, the term functional test has a much vaguer
meaning than unit test
...
For a simple stored procedure that selects data from the
database, this asks the question of whether the stored procedure returning the
correct data? Again, I will carefully define the term correct
...
The
logic required for this kind of validation means that a functional test is a white box
test in the database world, compared to the black box of unit testing
...
Consider the following stored
procedure, which might be used for a banking application:
CREATE PROCEDURE GetAggregateTransactionHistory
@CustomerId int
AS
BEGIN
SET NOCOUNT ON;

50

CHAPTER 3

TESTING DATABASE ROUTINES

SELECT
SUM
(
CASE TransactionType
WHEN 'Deposit' THEN Amount
ELSE 0
END
) AS TotalDeposits,
SUM
(
CASE TransactionType
WHEN 'Withdrawal' THEN Amount
ELSE 0
END
) AS TotalWithdrawals
FROM TransactionHistory
WHERE
CustomerId = @CustomerId;
END;
This stored procedure’s implied contract states that, given the input of a customer ID into the
@CustomerId parameter, a result set of two columns and zero or one rows will be output (the contract
does not imply anything about invalid customer IDs or customers who have not made any transactions)
...


What if the Customer Doesn’t Exist?
The output of the GetAggregateTransactionHistory stored procedure will be the same whether you
pass in a valid customer ID for a customer that happens to have had no transactions, or an invalid
customer ID
...
Depending on the requirements of a particular
situation, it might make sense to make the interface richer by changing the rules a bit, only returning no
rows if an invalid customer ID is passed in
...

A unit test against this stored procedure should do nothing more than validate the interface
...
No
verification of data is necessary; it would be out of scope, for instance, to find out whether the aggregate
information was accurate or not—that would be the job of a functional test
...
Is the interface working as documented, providing the
appropriate level of encapsulation and returning data in the correct format?

51

CHAPTER 3

TESTING DATABASE ROUTINES

Each interface in the system will need one or more of these tests (see the “How Many Tests Are
Needed?” section later in the chapter), so they need to be kept focused and lightweight
...
In the case of the
GetAggregateTransactionHistory stored procedure, writing a functional test would essentially entail
rewriting the entire stored procedure again—hardly a good use of developer time
...
These frameworks generally
make use of debug assertions, which allow the developer to specify those conditions that make a test
true or false
...
It accepts an expression as input and throws an exception if the expression is
false; otherwise, it returns true (or void, in some languages)
...
If a
routine expects that a variable is in a certain state at a certain time, an assertion can be used in order to
help make sure that assumption is enforced as the code matures
...

In unit testing, assertions serve much the same purpose
...
If any assertion throws an exception in a unit test, the
entire test is considered to have failed
...
net/projects/tsqlunit)
...
NET language using the
...
nunit
...

Providing an in-depth guide to coding against unit testing frameworks is outside the scope of this
book, but given that unit testing stored procedures is still somewhat of a mystery to many developers, I
will provide a basic set of rules to follow
...


2
...
What are the result sets that will be returned? What are
the datatypes of the columns, and how many columns will there be? Does the
contract make any guarantees about a certain number of rows?
Next, write code necessary to execute the stored procedure to be tested
...
NET to fill a DataSet with the result of the stored procedure,
where it can subsequently be interrogated
...
You might be
tempted to call the stored procedure using the same method as in the
application itself
...
Given that you only

CHAPTER 3

TESTING DATABASE ROUTINES

need to fill a DataSet, recoding the data access in the unit test should not be a
major burden, and will keep you from testing parts of the code that you don’t
intend to
...


Finally, use one assertion for each assumption you’re making about the stored
procedure; that means one assertion per column name, one per column
datatype, one for the row count if necessary, and so on
...


The following code listing gives an example of what an NUnit test of the
GetAggregateTransactionHistory stored procedure might look like:
[TestMethod]
public void TestAggregateTransactionHistory()
{
// Set up a command object
SqlCommand comm = new SqlCommand();
// Set up the connection
comm
...
CommandText = "GetAggregateTransactionHistory";
comm
...
StoredProcedure;
comm
...
AddWithValue("@CustomerId", 123);
// Create a DataSet for the results
DataSet ds = new DataSet();
// Define a DataAdapter to fill a DataSet
SqlDataAdapter adapter = new SqlDataAdapter();
adapter
...
Fill(ds);
}
catch
{
Assert
...

// There must be exactly one returned result set
Assert
...
Tables
...
Tables[0];
// There must be exactly two columns returned
Assert
...
Columns
...
IsTrue(
dt
...
IndexOf("TotalDeposits") > -1,
"Column TotalDeposits does not exist");
Assert
...
Columns
...
IsTrue(
dt
...
DataType == typeof(decimal),
"TotalDeposits data type is incorrect");
Assert
...
Columns["TotalWithdrawals"]
...
IsTrue(
dt
...
Count <= 1,
"Too many rows returned");
}
Although it might be disturbing to note that the unit test is over twice as long as the stored
procedure it is testing, keep in mind that most of this code can be easily turned into a template for quick
reuse
...
Many hours can be wasted debugging working code trying to figure out why the unit
test is failing, when it’s actually the fault of some code the unit test is relying on to do its job
...
In essence, they help you as a
developer to guarantee that in making changes to a system you didn’t break anything obvious
...
Developing against a system with a well-established set of unit tests is a joy, as
each developer no longer needs to worry about breaking some other component due to an interface
change
...


Regression Testing
As you build up a set of unit tests for a particular application, the tests will eventually come to serve as a
regression suite, which will help to guard against regression bugs—bugs that occur when a developer

54

CHAPTER 3

TESTING DATABASE ROUTINES

breaks functionality that used to work
...
For the intentional changes, the
solution is to rewrite the unit test accordingly
...

Experience has shown that fixing bugs in an application often introduces other bugs
...
By building a regression suite, the cost of
fixing these “side effect” bugs is greatly reduced
...

Regression testing is also the key to some newer software development methodologies, such as agile
development and extreme programming (XP)
...


Guidelines for Implementing Database Testing Processes
and Procedures
Of all the possible elements that make up a testing strategy, there is really only one key to success:
consistency
...
e
...
Inconsistency, or a lack of knowledge concerning
those variables that might have changed between tests, can mean that any problems identified during
testing will be difficult to trace
...
These tests should be automated and easy to run
...


Continuous Testing
Once you’ve built a set of automated tests, you’re one step away from a fully automatic testing
environment
...
Many software development shops use this technique to run their tests several times a day, throwing
alerts almost instantly if problem code is checked in
...
A great free tool to help set up continuous integration in
...
NET, available at http://sourceforge
...

Testers must also pay particular attention to any data used to conduct the tests
...
Such
a set of data can guarantee consistency between test runs, as it can be restored to its original state
...


55

CHAPTER 3

TESTING DATABASE ROUTINES

It’s also recommended that a copy of actual production data (if available) be used for testing near
the end of any given test period, rather than relying on artificially generated test data
...


Why Is Testing Important?
It can be argued that the only purpose of software is to be used by end users, and therefore the only
purpose of testing is to make sure that those end users don’t encounter issues
...


•

Testing ensures that no problems need to be fixed
...
If not fully tested by developers or a quality assurance team,
an application will be tested by the end users trying to use the software
...

Testing by development and quality assurance teams validates the software
...
Since the database is an increasingly important component in
most applications, testing the database makes sense; if the database has problems, they will propagate
to the rest of the application
...
Databases should be tested for the following issues:
•
•

Data availability and authorization tests are similar to interface consistency tests,
but more focused on who can get data from the database than how the data
should be retrieved
...
These kinds of tests are only important if the database is
being used for authenticating users
...


Performance tests are important for verifying that the user experience will be
positive, and that users will not have to wait longer than necessary for data
...


CHAPTER 3

•

TESTING DATABASE ROUTINES

Regression testing covers every other type of test, but generally focuses on
uncovering issues that were previously fixed
...


How Many Tests Are Needed?
Although most development teams lack a sufficient number of tests to test the application thoroughly, in
some cases the opposite is true
...
It’s important to balance the need for thorough testing with the
realities of time and monetary constraints
...
For example, consider the following stored procedure interface:
CREATE PROCEDURE SearchProducts
SearchText varchar(100) = NULL,
PriceLessThan decimal = NULL,
ProductCategory int = NULL
This stored procedure returns data about products based on three parameters, each of which is
optional, based on the following (documented) rules:
•

A user can search for text in the product’s description
...


•

A user can combine a text search or price search with an additional filter on a
certain product category, so that only results from that category are returned
...
This condition should
return an error
...


In order to validate the stored procedure’s interface, one unit test is necessary for each of these
conditions
...
The unit tests for the invalid combinations of
arguments should verify that an error occurs when these combinations are used
...

In addition to these unit tests, an additional regression test should be produced for each known
issue that has been fixed within the stored procedure, in order to ensure that the procedure’s
functionality does not degenerate over time
...
The individual tests will have to do nothing more than pass the
correct parameters to a parameterized base test
...
Many software shops, especially smaller ones, have no dedicated quality assurance staff, and such

57

CHAPTER 3

TESTING DATABASE ROUTINES

compressed development schedules that little testing gets done, making full functionality testing nearly
impossible
...
On the contrary, time and money is actually wasted by
lack of testing
...
A developer who is currently working on enhancing a given module has an
in-depth understanding of the code at that moment
...
If defects are
discovered and reported while the developer is still in the trenches, the developer will not need to
relearn the code in order to fix the problem, thereby saving a lot of time
...

If management teams refuse to listen to reason and allocate additional development time for proper
testing, try doing it anyway
...
Adopting a testing strategy—with or without management approval—can mean better,
faster output, which in the end will help to ensure success
...
Performance testing is imperative for ensuring a positive user experience
...

Performance testing relies on collecting, reviewing, and analyzing performance data for different
aspects of the system
...
SQL Server 2008
provides a number of in-built tools that allow DBAs and developers to store or view real-time
information about activity taking place on the server, including the following:
•

SQL Server Profiler

•

Server-side traces

•

System Monitor console

•

Dynamic Management Views (DMVs)

•

Extended Events

•

Data Collector

There are also a number of third-party monitoring tools available that can measure, aggregate, and
present performance data in different ways
...


58

CHAPTER 3

TESTING DATABASE ROUTINES

Note Access to performance monitoring tools in many organizations is restricted to database or system
administrators
...


Real-Time Client-Side Monitoring
The Profiler tool that ships with SQL Server 2008 is extremely useful and very easy to use
...
However, for most performance monitoring
work, there are only a few key events that you’ll need to worry about
...
Each of these events fires on completion of queries; the only difference
between them is that RPC:Completed fires on completion of a remote procedure call (RPC), whereas
SQL:BatchCompleted fires on completion of a SQL batch—different access methods, same end result
...
Due to the fact that this column includes
compilation time, it is common to see the reported amount of time drop on
consecutive queries, thanks to plan caching
...
A
logical I/O occurs any time SQL Server’s query engine requests data, whether from
the physical disk or from the buffer cache
...
However, even reading data from memory does cost
the server in terms of CPU time, so it is a good idea to try to keep any kind of reads
to a minimum
...

This means that only writes that were actually persisted to disk during the course
of the query will be reported
...

The duration of a query is a direct reflection on the user experience, so this is
generally the one to start with
...


By reviewing the high-level information contained in these columns, you can identify potential
candidates for further investigation
...
What is the maximum amount of time that a query can be allowed to run? What should the
average amount of run time be? By aggregating the Duration column, you can determine whether these
times have been exceeded
...
For instance, the Scan:Started
event can be used to identify possible queries that are making inefficient use of indexes and therefore
may be causing I/O problems
...


Server-Side Traces
While SQL Server Profiler is a convenient and useful tool, it does have some limitations
...
As such,
the very act of attempting to monitor performance may have a negative effect on the performance of the
system being measured, leading to biased results (the “observer effect”)
...
A server-side
trace runs in the background on the SQL server, saving its results to a local file on the server instead of
streaming them to the client
...
A better approach is to create a server-side trace based on a trace
definition exported from the SQL Server Profiler tool, as explained in the following steps:
1
...


2
...


3
...

This file contains the T-SQL code required to start a trace based on the
parameters supplied
...


Edit the following line of the script, by specifying a valid output path and file
name for the trace results where indicated:

exec @rc = sp_trace_create @TraceID output, 0, N'InsertFileNameHere',
@maxfilesize, NULL

Note The specified file name should not include an extension of any kind
...

5
...
Increasing the maximum file size will help to
minimize the number of rollover files created during the trace
...


6
...
The trace will begin
collecting data in the background, and the generated script will return a trace
identifier, TraceID, which you should make note of as it will be required to
control the trace later
...
When you are done tracing, you must
stop and close the trace by using the sp_trace_setstatus stored procedure, supplying the TraceID trace

60

CHAPTER 3

TESTING DATABASE ROUTINES

identifier returned when the trace was started
...
fn_trace_gettable function can be used to read the
data from the trace file
...
trc extension automatically added by SQL Server—and the maximum number of rollover files to
read
...
trc
...
fn_trace_gettable('C:\Traces\myTrace
...
It can be
inserted into a table, queried, or aggregated in any number of ways in order to evaluate which queries
are potentially causing problems
...
These counters can be read using the System Monitor console (aka Performance
Monitor, or perfmon
...
Similar to SQL Server trace events, there are hundreds of counters from which to choose—
but only a handful generally need to be monitored when doing an initial performance evaluation of a
SQL Server installation
...
If this counter is above 70 percent during peak
load periods, it may be worthwhile to begin investigating which routines are
making heavy use of CPU time
...
Disk Queue Length indicates whether processes have to wait to
use disk resources
...
e
...
Too many simultaneous
requests results in wait times, which can mean query performance problems
...


•

PhysicalDisk:Disk Read Bytes/sec and PhysicalDisk:Disk Write Bytes/sec report
the number of bytes read from and written to the disk, respectively
...
Disk Queue
Length can help to explain problems
...
Slow DML queries coupled with high physical writes and
high queue lengths are a typical indication of disk contention, and a good sign that
you might want to evaluate how to reduce index fragmentation in order to
decrease insert and update times
...
Decreasing lock contention can be quite a
challenge, but it can be solved in some cases by using either dirty reads (the READ
UNCOMMITTED isolation level) or row versioning (the SNAPSHOT isolation level)
...


•

SQLServer:Buffer Manager:Page life expectancy is the average amount of time, in
seconds, that pages remain in buffer cache memory after they are read off of the
disk
...
Either way, values below 300 (i
...
, 5 minutes) may indicate that you
have a problem in this area
...
The Cache Hit Ratio counter is
the ratio of cache hits to lookups—in other words, what percentage of issued
queries are already in the cache
...

Toward the end, you should see this number fairly near to 100, indicating that
almost all queries are cached
...
A low Cache Hit Ratio
combined with a high Cached Pages value means that you need to consider fixing
the dynamic SQL being used by the system
...


Tip SQL Server Profiler has the ability to import saved performance counter logs in order to correlate them with
traces
...


Dynamic Management Views (DMVs)
The dynamic management views exposed by SQL Server 2008 contain a variety of server- and databaselevel information that can be used to assist in performance measurement
...
dm_os_performance_counters DMV contains over 1,000 rows of high-level performance counters
maintained by the server, including common measures concerning I/O, locks, buffers, and log usage
...

In addition to sys
...
The following list shows a few of the DMVs
that I find to be most helpful for performance measurement and tuning:

62

CHAPTER 3

TESTING DATABASE ROUTINES

•

sys
...
When joined to sys
...
dm_exec_query_plan, it is possible to analyze the performance of individual
troublesome batches in relation to their SQL and query plan
...
dm_db_index_usage_stats, sys
...
dm_db_index_operational_stats: These three views display information that is
useful for index tuning, including reporting the number of seeks and scans
performed against each index, the degree of index fragmentation, and possible
index contention and blocking issues
...
dm_os_wait_stats: This records information regarding requests that were
forced to wait—in other words, any request that could not immediately be
satisfied by the server because of unavailability of I/O or CPU resource
...
dm_os_waiting_tasks and sys
...


Caution DMVs generally report any timings measured in microseconds (1/1,000,000 of a second), whereas
most other performance measuring tools report timings in milliseconds (1/1,000 of a second)
...
For
example, suppose that sys
...
g
...
Clearly, the performance issues in this case will not be resolved by upgrading the
server CPU—there is an I/O bottleneck and the appropriate solution involves finding a way to shorten
I/O response times, or reducing the I/O requirements of the query
...
In order to reset wait
statistics before running a performance test, you can use DBCC SQLPERF with the CLEAR option—for example,
DBCC SQLPERF ('sys
...


Extended Events
Extended events make up a flexible, multipurpose eventing system introduced in SQL Server 2008 that
can be used in a wide variety of scenarios, including performance testing
...
The payload (i
...
, the columns of data) collected
when an event fires can then be delivered to a variety of synchronous and asynchronous targets
...

One of the shortcomings of the monitoring tools introduced previously is that they tend to collect
performance indicators that are aggregated at predefined levels
...
dm_os_wait_stats, for example, might indicate that an I/O bottleneck is occurring at a server-wide
level
...
By leveraging extended events such as sqlos
...
wait_info_external, it is possible to gather specific wait statistics at the session or statement
levels
...

The following code listing illustrates how to create a new extended event session that records all
statements that encounter waits (triggered by the sqlos
...
wait_info(
ACTION(
sqlserver
...
plan_handle)
WHERE total_duration > 0
)
ADD TARGET package0
...
xel',
metadatafile = N'c:\wait
...
Notice that in this example I specify an
asynchronous file target, which means that SQL Server execution will continue while event data is sent
to and saved in the target file
...
The size of this buffer is determined by the
MAX_MEMORY event session variable, which by default is set to 4MB
...

Synchronous targets do not have this risk, but they may exhibit worse performance because any tasks
that fire events are made to wait for the last event to be fully consumed by the target before being
allowed to continue
...
To
analyze the data contained within this file, it can be loaded back into SQL Server using the

64

CHAPTER 3

TESTING DATABASE ROUTINES

sys
...
value('(/event/action[@name=''sql_text'']/value)[1]','varchar(max)')
AS sql_text,
xe_data
...
value('(/event/data[@name=''wait_type'']/text)[1]','varchar(50)')
AS wait_type,
xe_data
...
value('(/event/data[@name=''signal_duration'']/value)[1]','int')
AS signal_duration
FROM (
SELECT
CAST(event_data AS xml) AS xe_data
FROM
sys
...
xel', 'c:\wait_*
...
For more
information, and examples of other possible uses, see Books Online: http://msdn
...
com/enus/library/bb630354
...


Data Collector
The Data Collector allows you to automatically collect performance data from DMVs and certain system
performance counters and upload them to a central management data warehouse (MDW) at regular
intervals according to a specified schedule
...
The server activity collection set combines information from DMVs including
sys
...
dm_exec_sessions, sys
...
dm_os_waiting_tasks,
with various SQL Server and OS performance counters to provide an overview of CPU, disk I/O, memory,
and network resource usage on the server
...
If required, you can adjust the definition, frequency, and duration of counters collected by
creating your own custom data collection set based on the performance counter’s collector type
...
Historical performance data can be collected and persisted over a long
period of time, and comparisons can be drawn of the relative performance of a server over several
months, or even years
...
Figure 3-1 illustrates a
default report generated from the server activity collection set
...
The server activity collection report

66

CHAPTER 3

TESTING DATABASE ROUTINES

Analyzing Performance Data
In the previous section, I discussed how to capture SQL Server performance data using a number of
different tools and techniques
...

Note that while the techniques discussed here may help you identify bottlenecks and other
performance issues, they do not deal with how to fix those problems
...
I highly recommend that readers searching for a more detailed guide invest
in a book dedicated to the subject, such as SQL Server 2008 Query Performance Tuning Distilled, by Grant
Fritchey and Sajal Dam (Apress, 2009)
...
Performance tests should be repeatable and should be done in an
environment that can be rolled back to duplicate the same conditions for multiple test runs
...
I recommend using a test
database that can be restored to its original state each time, as well as rebooting all servers involved in
the test just before beginning a run, in order to make sure that the test starts with the same initial
conditions each time
...
Try each technique in your own environment to determine
which fits best into your testing system
...

Consistency is the key to validating not only that changes are effective, but also measuring how effective
they are
...
The metrics captured
during the baseline test will be used to compare results for later runs
...
g
...
Keep in mind that fixing issues in one area of an
application might have an impact on performance of another area
...
By fixing the I/O problems for the first query, you may
introduce greater CPU utilization, which in turn will cause the other query to degrade in performance if
they are run simultaneously
...
Serverside traces should be used to capture performance data, including query duration and resources used
...

In order to determine which resources are starved, performance counters can be used to track server
utilization
...


67

CHAPTER 3

TESTING DATABASE ROUTINES

Big-Picture Analysis
Once you have set up performance counters and traces, you are ready to begin actual performance
testing
...

A first step is to determine what kinds of unit and functional tests exist, and evaluate whether they
can be used as starting points for performance tests
...
However, most
commercial load tools are designed to exercise applications or web code directly
...
Absolute coverage is nice, but is
unrealistic in many cases
...

Depending on which load tool you are using, this can take some time
...
Nonrandom inputs can mask disk I/O issues caused by buffer cache recycling
...
If you are testing on a
system that mirrors the application’s production environment, try to test at a load equal to that which
the application encounters during peak periods
...
Note that it can be difficult to test against servers that aren’t scaled
the same as production systems
...
In this situation it might be advisable to
modify SQL Server’s processor affinity on the test system such that less processor power is available,
which will make the available processor to available disk I/O ratio fall into line with the actual
environment in which code needs to run
...
For example, ensure that the maximum
degree of parallelism is set similarly so that processors will be used the same way in queries on both the
test and production systems
...

Once user goals are set, load tests should generally be configured to step up load slowly, rather than
immediately hit the server with the peak number of users
...
Note that step testing
may not be an accurate figure if you’re testing a situation such as a cluster failover, in which a server may
be subjected to a full load immediately upon starting up
...
Try to look at general trends in
the performance counters to determine whether the system can handle load spikes, or generally
sustained load over long periods of time (again, depending on actual application usage patterns, if
possible)
...


Granular Analysis
If the results of a big-picture test show that certain areas need work, a more granular investigation into
specific routines will generally be necessary
...

While it is often tempting to look only at the worst offending queries—for instance, those with the
maximum duration—this may not tell the complete story
...
These procedures may be
responsible for longer user interface wait times, but the individual stored procedure with the longest
duration may not necessarily indicate the single longest user interface wait time
...
In these cases it is important to group procedures that are called together
and aggregate their total resource utilization
...


•

If, on the other hand, all of the procedures are called simultaneously in parallel
(for instance, on different connections), resource utilization should be totaled in
order to determine the group’s overall impact on the system, and the duration of
the longest-running individual query should be noted
...
Table 3-1 shows the average data collected for these
stored procedures
...
Stored Procedures Called After Login, with Averaged Data

Stored Procedure

Duration (ms)

CPU

Reads

Writes

LogSessionStart

422

10

140

1

GetSessionData

224

210

3384

0

GetUserInfo

305

166

6408

0

If the system calls these stored procedures sequentially, the total duration that should be recorded
for this group is 951 ms (422 + 224 + 305)
...
So we record 210
for CPU, 6408 for Reads, and 1 for Writes
...
Total duration will only be as much as the longest running of the three—422ms (assuming, of
course, that the system has enough resources available to handle all three requests at the same time)
...

By grouping stored procedures in this way, the total impact for a given feature can be assessed
...
This can also be an issue with cursors that are doing a large number of very small fetch

69

CHAPTER 3

TESTING DATABASE ROUTINES

operations
...

Another benefit of this kind of grouping is that further aggregation is possible
...
That information can be useful when trying to reach specific
scalability goals
...
However, some care should be taken to
make effective use of your time; in many cases what appear to be the obvious problems are actually side
effects of other, more subtle issues
...

Duration tells a major part of the story, but it does not necessarily indicate a performance problem with
that stored procedure
...
When performance tuning,
it is best to be suspicious of long-running queries with very low reported resource utilization
...

By using the granular analysis technique and aggregating, it is often possible to find the real
offenders more easily
...
This procedure reported a high duration,
which was interpreted as a possible performance problem
...
Each call to the second stored procedure reported a small enough duration that it did not
appear to be causing performance problems, yet as it turned out it was causing issues in other areas
...


Summary
Software testing is a complex field, but it is necessary that developers understand enough of it to make
the development process more effective
...

Database developers, like application developers, must learn to exploit unit tests in order to
increase software project success rates
...

During performance testing, make sure to carefully analyze the data
...
SQL Server 2008 ships with a variety of methods for
collecting and presenting performance counters, and I’ll be using these to analyze the relative
performance of different approaches taken in later chapters throughout the book
...
If you’re not already testing your software during the development process, the methodologies
presented here can help you to implement a set of procedures that will get you closer to the next level
...
But, alas, back in the real
world, hordes of developers sit in cubicle farms gulping acrid coffee, fighting bugs that are not always
their fault or under their control in any way
...
For instance, do you know what will happen if a janitor, while
cleaning the data-center floor, accidentally slops some mop water into the fan enclosure of the database
server? It might crash, or it might not; it might just cause some component to fail somewhere deep in the
app, sending up a strange error message
...
It is also imperative that SQL Server
developers understand how to work with errors—both those thrown by the server itself and custom
errors built specifically for when problems occur during the runtime of an application
...
Errors
The terms exception and error, while often used interchangeably by developers, actually refer to slightly
different conditions:
An error can occur if something goes wrong during the course of a program, even
though it can be purely informational in nature
...
However, this may
or may not mean that the program itself is in an invalid state
...
For example, if a network library is being used to send packets, and
the network connection is suddenly dropped due to someone unplugging a cable,
the library might throw an exception
...
If the caller does not
handle the exception (i
...
, capture it), its execution will also abort
...

Another way to think about exceptions and errors is to think of errors as occurrences that are
expected by the program
...
A dropped network connection, on the

71

CHAPTER 4

ERRORS AND EXCEPTIONS

other hand, could be caused by any number of circumstances and therefore is much more difficult to
handle specifically
...
The exception can then be
handled by a routine higher in the call stack, which can decide what course of action to take in order to
solve the problem
...
For example, some programmers choose to define a large number of different custom
exceptions, and then deliberately raise these exceptions as a method of controlling the flow of an
application
...
However convenient it may seem to use exceptions in such scenarios, they clearly represent
an abuse of the intended purpose of exception-handling code (which, remember, is to deal with
exceptional circumstances)
...
As a result, exception-laden code is more difficult to maintain when
compared to code that relies on more standard control flow structures
...
Furthermore,
due to the fact that exceptions can cause abort conditions, they should be used sparingly
...
If you’re designing an interface that
needs to ensure that the caller definitely sees a certain condition when and if it occurs, it might make
sense to use an exception rather than an error
...
Unlike many other programming languages, SQL Server
has an exception model that involves different behaviors for different types of exceptions
...

To begin with, think about connecting to a SQL Server and issuing some T-SQL
...
The connection also determines what
database will be used as the default for scope resolution (i
...
, finding objects—more on this in a bit)
...
A batch consists of one or more T-SQL statements,
which will be compiled together to form an execution plan
...
Let’s take a look
at some practical examples to see this in action
...
To see this behavior, you can use SQL
Server Management Studio to execute a batch that includes an exception, followed by a PRINT statement
...
000000
...
However, only the SELECT statement was aborted
...


Batch-Level Exceptions
Unlike a statement-level exception, a batch-level exception does not allow the rest of the batch to
continue running
...
An example of a batch-aborting exception is an invalid
conversion, such as the following:
SELECT CONVERT(int, 'abc');
PRINT 'This will NOT print!';
GO
The output of this batch is as follows:
Msg 245, Level 16, State 1, Line 1
Conversion failed when converting the varchar value 'abc' to data type int
...
The PRINT statement was not allowed to run, although if the batch had contained any valid
statements before the exception, these would have been executed successfully
...
For instance:
SELECT CONVERT(int, 'abc');
GO
PRINT 'This will print!';
GO
In this case there are two batches sent to SQL Server, separated by the batch separator, GO
...
This results in the following
output:
Msg 245, Level 16, State 1, Line 2
Conversion failed when converting the varchar value 'abc' to data type int
...
The exception
will bubble up to the next level of execution, aborting every call in the stack
...


74

CHAPTER 4

ERRORS AND EXCEPTIONS

Parsing and Scope-Resolution Exceptions
Exceptions that occur during parsing or during the scope-resolution phase of compilation appear at first
to behave just like batch-level exceptions
...
If the
exception occurs in the same scope as the rest of the batch, these exceptions will behave just like a
batch-level exception
...

As an example, consider the following batch, which includes a malformed SELECT statement (this is a
parse exception):
SELECTxzy FROM SomeTable;
PRINT 'This will NOT print!';
GO
In this case, the PRINT statement is not run, because the whole batch is discarded during the parse
phase
...

To see the difference in behavior, the SELECT statement can be executed as dynamic SQL using the
EXEC function
...
Try running the following T-SQL to observe the change:
EXEC('SELECTxzy FROM SomeTable');
PRINT 'This will print!';
GO
The PRINT statement is now executed, even though the exception occurred:
Msg 156, Level 15, State 1, Line 1
Incorrect syntax near the keyword 'FROM'
...
Essentially, SQL Server
processes queries in two phases
...
The second phase is the compilation phase, during which an execution plan is built
and objects referenced in the query are resolved
...

However, within the context of stored procedures, SQL Server exploits late binding
...


75

CHAPTER 4

ERRORS AND EXCEPTIONS

To see what this means, create the following stored procedure (assuming that a table called
SomeTable does not exist in the current database):
CREATE PROCEDURE NonExistentTable
AS
BEGIN
SELECT xyz
FROM SomeTable;
END;
GO
Although SomeTable does not exist, the stored procedure is created—the T-SQL parses without any
errors
...

Like the parse exception, scope-resolution exceptions behave similarly to batch-level exceptions
within the same scope, and similarly to statement-level exceptions in the outer scope
...

For instance:
EXEC NonExistentTable;
PRINT 'This will print!';
GO
leads to the following result:
Msg 208, Level 16, State 1, Procedure NonExistentTable, Line 4
Invalid object name 'SomeTable'
...
These types of connection- and server-level exceptions are generally caused by
internal SQL Server bugs, and are thankfully quite rare
...


The XACT_ABORT Setting
Although users do not have much control over the behavior of exceptions thrown by SQL Server, there is
one setting that can be modified on a per-connection basis
...
This means that control will always be
immediately returned to the client any time an exception is thrown by SQL Server during execution of a
query (assuming the exception is not handled)
...
Recall that the following integer overflow exception operates at the
statement level:
SELECT POWER(2, 32);
PRINT 'This will print!';
GO
Enabling the XACT_ABORT setting before running this T-SQL changes the output, resulting in the
PRINT statement not getting executed:
SET XACT_ABORT ON;
SELECT POWER(2, 32);
PRINT 'This will NOT print!';
GO
The output from running this batch is as follows:
Msg 232, Level 16, State 3, Line 2
Arithmetic overflow error for type int, value = 4294967296
...

Note that XACT_ABORT only affects the behavior of runtime errors, not those generated during
compilation
...


77

CHAPTER 4

ERRORS AND EXCEPTIONS

In addition to controlling exception behavior, XACT_ABORT also modifies how transactions behave
when exceptions occur
...


Dissecting an Error Message
A SQL Server exception has a few different component parts, each of which are represented within the
text of the error message
...
Error
messages can also contain additional diagnostic information including line numbers and the name of
the procedure in which the exception occurred
...
For example, the
error number of the following exception is 156:
Msg 156, Level 15, State 1, Line 1
Incorrect syntax near the keyword 'FROM'
...
However, there are times when knowing the
error number can be of use
...

The error number can also be used to look up the localized translation of the error message from the
sys
...
The message_id column contains the error number, and the language_id
column can be used to get the message in the correct language
...
messages
WHERE
message_id = 208
AND language_id = 1033;
GO
The output of this query is an error message template, shown here:

Invalid object name '%
...

See the section “SQL Server’s RAISERROR Function” for more information about error message
templates
...
This number can
sometimes be used to either classify an exception or determine its severity
...

The following exception, based on its error message, is of error level 15:
Msg 156, Level 15, State 1, Line 1
Incorrect syntax near the keyword 'FROM'
...
messages view, using the severity
column
...
If severity is 11 or greater, the
message is considered to be an error and can be broken down into the following documented
categories:
•

Error levels 11 through 16 are documented as “errors that can be corrected by the
user
...


•

Error levels 17 through 19 are more serious exceptions
...
Many of these are automatically logged to the SQL Server error
log when they are thrown
...
messages table
...
These
include various types of data corruption, network, logging, and other critical
errors
...


Although the error levels that make up each range are individually documented in Books Online
(http://msdn2
...
com/en-us/library/ms164086
...
For instance, according to documentation, severity level 11 indicates errors
where “the given object or entity does not exist
...
Many other errors have equally unpredictable levels, and it is recommended that you do not
program client software to rely on the error levels for handling logic
...
For instance, both errors 245 (“Conversion failed”) and 515 (“Cannot insert
the value NULL
...
However, 245 is a batch-level
exception, whereas 515 acts at the statement level
...
The values that SQL Server uses for this tag are not documented, so this tag is generally
not helpful
...


Additional Information
In addition to the error number, level, and state, many errors also carry additional information about the
line number on which the exception occurred and the procedure in which it occurred, if relevant
...

If an exception does not occur within a procedure, the line number refers to the line in the batch in
which the statement that caused the exception was sent
...
Consider the following TSQL:
SELECT 1;
GO
SELECT 2;
GO
SELECT 1/0;
GO
In this case, although a divide-by-zero exception occurs on line 5 of the code listing itself, the
exception message will report that the exception was encountered on line 1:
(1 row(s) affected)

(1 row(s) affected)
Msg 8134, Level 16, State 1, Line 1
Divide by zero error encountered
...
GO is an
identifier recognized by SQL Server client tools (e
...
, SQL Server Management Studio and SQLCMD) that
tells the client to separate the query into batches, sending each to SQL Server one after another
...
SQL Server does not know that on the client (e
...
, in SQL Server
Management Studio) these batches are all displayed together on the screen
...


SQL Server’s RAISERROR Function
In addition to the exceptions that SQL Server itself throws, users can raise exceptions within T-SQL by
using a function called RAISERROR
...
n ] ] )
[ WITH option [ ,
...
messages
...

The second argument, severity, can be used to enforce some level of control over the behavior of
the exception, similar to the way in which SQL Server uses error levels
...
User exceptions raised over level 20, just like those raised by
SQL Server, cause the connection to break
...


Note XACT_ABORT does not impact the behavior of the RAISERROR statement
...
It can be used to add additional coded information to be carried by the exception—but it’s
probably just as easy to add that data to the error message itself in most cases
...
For general exceptions, I usually use severity 16 and a value of 1 for
state:

RAISERROR('General exception', 16, 1);
This results in the following output:
Msg 50000, Level 16, State 1, Line 1
General exception
Note that the error number generated in this case is 50000, which is the generic user-defined error
number that will be used whenever passing in a string for the first argument to RAISERROR
...
This syntax is deprecated in SQL Server 2008 and
should not be used
...
For example, think
about how you might write code to work with a number of product IDs, dynamically retrieved, in a loop
...
If so, you might wish to define a custom exception that should be thrown when a
problem occurs—and it would probably be a good idea to return the current value of @ProductId along
with the error message
...
The first is to
dynamically build an error message string:
DECLARE @ProductId int;
SET @ProductId = 100;
/*
...
*/
DECLARE @ErrorMessage varchar(200);
SET @ErrorMessage =
'Problem with ProductId ' + CONVERT(varchar, @ProductId);
RAISERROR(@ErrorMessage, 16, 1);
Executing this batch results in the following output:
Msg 50000, Level 16, State 1, Line 10
Problem with ProductId 100
While this works for this case, dynamically building up error messages is not the most elegant
development practice
...
problem occurs
...
The %i

82

CHAPTER 4

ERRORS AND EXCEPTIONS

embedded in the error message is a format designator that means “integer
...
”
You can embed as many designators as necessary in an error message, and they will be substituted
in the order in which optional arguments are appended
...
problem occurs
...
For a complete list of the supported designators, see the
“RAISERROR (Transact-SQL)” topic in SQL Server 2008 Books Online
...
In addition, each of the exceptions would only be able to use the default userdefined error number, 50000, making programming against these custom exceptions much more
difficult
...
messages
...

To create a persistent custom error message, use the sp_addmessage stored procedure
...
In addition to
an error message, users can specify a default severity
...
When
developing new applications that use custom messages, try to choose a well-defined range in which to
create your messages, in order to avoid overlaps with other applications in shared environments
...

Adding a custom message is as easy as calling sp_addmessage and defining a message number and
the message text
...
This brings up an important point about severities of
custom errors: whatever severity is specified in the call to RAISERROR will override the severity that was
defined for the error
...

Changing the text of an exception once defined is also easy using sp_addmessage
...
The examples here do not show localization; instead, messages will be created
for the user’s default language
...


Logging User-Thrown Exceptions
Another useful feature of RAISERROR is the ability to log messages to SQL Server’s error log
...

In order to log any exception, use the WITH LOG option of the RAISERROR function, as in the following TSQL:

RAISERROR('This will be logged
...
The user executing the RAISERROR
function must either be a member of the sysadmin fixed server role or have ALTER TRACE permissions
...
In such cases it can be
extremely difficult to debug issues without knowing whether an exception is being thrown
...

In order to monitor for exceptions, start a trace and select the Exception and User Error Message
events
...
The Exception event will
contain all of the data associated with the exception except for the actual message
...
The User Error Message event will contain the formatted
error message as it was sent to the client
...

You may also notice error 208 exceptions (“Object not found”) without corresponding error message
events
...


Exception Handling
Understanding when, why, and how SQL Server throws exceptions is great, but the real goal is to actually
do something when an exception occurs
...


85

CHAPTER 4

ERRORS AND EXCEPTIONS

Why Handle Exceptions in T-SQL?
Exception handling in T-SQL should be thought of as no different from exception handling in any other
language
...
If an exception can be
caught at a lower level and dealt with there, higher-level modules will not require special code to handle
the exception, and therefore can concentrate on whatever their purpose is
...

Put another way, exceptions should be encapsulated as much as possible—knowledge of the
internal exceptions of other modules is yet another form of coupling, not so different from some of the
types discussed in the first chapter of this book
...
But the basic rule is, if you can “fix” the exception one way or another without letting the
caller ever know it even occurred, that is probably a good place to encapsulate
...
Any
exception that occurred would be passed back to the caller, regardless of any action taken by the code of
the stored procedure or query in which it was thrown
...


Note If you’re following the examples in this chapter in order, make sure that you have turned off the
XACT_ABORT setting before trying the following examples
...
If the last statement did throw an error, it returns the error number
...


86

CHAPTER 4

ERRORS AND EXCEPTIONS

and the second statement returns a result set containing a single value, containing the error number
associated with the previous error:
ErrorNumber
8134
By checking to see whether the value of @@ERROR is nonzero, it is possible to perform some very
primitive error handling
...
Many developers new to T-SQL
are quite surprised by the output of the following batch:
SELECT 1/0 AS DivideByZero;
IF @@ERROR <> 0
SELECT @@ERROR AS ErrorNumber;
GO
The first line of this code produces the same error message as before, but on this occasion, the result
of SELECT @@ERROR is
ErrorNumber
0
The reason is that the statement executed immediately preceding @@ERROR was not the divide by
zero, but rather the line IF @@ERROR <> 0, which did not generate an error
...
Of course, if even a single statement
is missed, holes may be left in the strategy, and some errors may escape notice
...
It is a simple,
lightweight alternative to the full-blown exception-handling capabilities that have been added more
recently to the T-SQL language, and it has the additional benefit of not catching the exception
...


SQL Server’s TRY/CATCH Syntax
The standard error-handling construct in many programming languages, including T-SQL, is known as
try/catch
...
The first section,
the try block, contains exception-prone code to be “tried
...
This is called the
catch block
...
This is also known as catching an exception
...

As a first example, consider the following T-SQL:
BEGIN TRY
SELECT 1/0 AS DivideByZero;
END TRY
BEGIN CATCH
SELECT 'Exception Caught!' AS CatchMessage;
END CATCH
Running this batch produces the following output:
DivideByZero
------------

CatchMessage
----------------Exception Caught!
The interesting things to note here are that, first and foremost, there is no reported exception
...
Second, notice that an
empty result set is returned for the SELECT statement that caused the exception
...
By sending back an empty result set, the implied
contract of the SELECT statement is honored (more or less, depending on what the client was actually
expecting)
...

Therefore, the following T-SQL has the exact same output as the last example:
BEGIN TRY
SELECT 1/0 AS DivideByZero;
SELECT 1 AS NoError;
END TRY
BEGIN CATCH
SELECT 'Exception Caught!' AS CatchMessage;
END CATCH

88

CHAPTER 4

ERRORS AND EXCEPTIONS

Finally, it is worth noting that parsing and compilation exceptions will not be caught using
TRY/CATCH, nor will they ever have a chance to be caught—an exception will be thrown by SQL Server
before any of the code is ever actually executed
...
These functions, a list of which follows, enable the
developer to write code that retrieves information about the exception that occurred in the TRY block
...

However, it is important to point out that unlike @@ERROR, the values returned by these functions are not
reset after every statement
...
Therefore, logic such as that
used in the following T-SQL works:
BEGIN TRY
SELECT CONVERT(int, 'ABC') AS ConvertException;
END TRY
BEGIN CATCH
IF ERROR_NUMBER() = 123
SELECT 'Error 123';
ELSE
SELECT ERROR_NUMBER() AS ErrorNumber;
END CATCH
As expected, in this case the error number is correctly reported:
ConvertException
----------------

ErrorNumber
----------245

89

CHAPTER 4

ERRORS AND EXCEPTIONS

These functions, especially ERROR_NUMBER, allow for coding of specific paths for certain exceptions
...


Rethrowing Exceptions
A common feature in most languages that have try/catch capabilities is the ability to rethrow exceptions
from the catch block
...
This is useful when you need to do some handling of the
exception but also let the caller know that something went wrong in the routine
...
However, it is fairly easy to create
such behavior based on the CATCH block error functions, in conjunction with RAISERROR
...
As functions are not allowed within calls to RAISERROR, it is necessary to
define variables and assign the values of the error functions before calling RAISERROR to rethrow the
exception
...


90

CHAPTER 4

ERRORS AND EXCEPTIONS

Keep in mind that, based on your interface requirements, you may not always want to rethrow the
same exception that was caught to begin with
...

For example, if you’re working with a linked server and the server is not responding for some reason,
your code will throw a timeout exception
...
This is something that
should be decided on a case-by-case basis, as you work out optimal designs for your stored procedure
interfaces
...
A primary example of this is logging of database exceptions
...

Another use case involves temporary fixes for problems stemming from application code
...
It might be simple to temporarily “fix” the problem by simply
catching the exception in the database rather than throwing it back to the application where the user will
receive an error message
...

It is also important to consider when not to encapsulate exceptions
...
There is definitely such a thing as too much exception handling,
and falling into that trap can mean that problems will be hidden until they cause enough of a
commotion to make themselves impossible to ignore
...
These situations are usually highlighted by a lack of viable backups because
the situation has been going on for so long, and inevitably end in lost business and developers getting
their resumes updated for a job search
...
Just use a little bit of
common sense, and don’t go off the deep end in a quest to stifle any and all exceptions
...
Although it’s better to try to find and solve the source of a deadlock than to code around
it, this is often a difficult and time-consuming task
...
Eventually
the deadlock condition will resolve itself (i
...
, when the other transaction finishes), and the DML
operation will go through as expected
...
A retry loop can be set up, within which the deadlock-prone code can be
tried in a TRY block and the deadlock caught in a CATCH block in order to try again
...
Each
time through the loop, the code is tried
...
Otherwise, execution jumps to the CATCH block, where a check is made to
ensure that the error number is 1205 (deadlock victim)
...
If the exception is not a deadlock, another exception is thrown so that the caller
knows that something went wrong
...


Exception Handling and Defensive Programming
Exception handling is extremely useful, and its use in T-SQL is absolutely invaluable
...
Whenever possible, code defensively—proactively look for problems, and if they can be
both detected and handled, code around them
...
If you can predict a
condition and write a code path to handle it during development, that will usually provide a much more
robust solution than trying to trap the exception once it occurs and handle it then
...
NET Framework provides its own exception-handling mechanism, which is quite separate from the
mechanism used to deal with exceptions encountered in T-SQL
...
SqlServer
...
SqlFunction()]
public static SqlDecimal Divide(SqlDecimal x, SqlDecimal y)
{
return x / y;
}
When cataloged and called from SQL Server with a value of 0 for the y parameter, the result is as
follows:
Msg 6522, Level 16, State 2, Line 1
A
...
DivideByZeroException: Divide by zero error encountered
...
DivideByZeroException:
at System
...
SqlTypes
...
op_Division(SqlDecimal x, SqlDecimal y)
at ExpertSQLServer
...
Divide(SqlDecimal x, SqlDecimal y)

...
That means that if the managed code throws an exception, it is caught by the
wrapper, which then generates an error
...

In this case, the original CLR exception, System
...

As previously stated, the best approach to deal with such exceptions is to tackle them at the lowest
level possible
...

One interesting point this raises is how to deal with exceptions arising in system-defined CLR
routines, such as any methods defined by the geometry, geography, or hierarchyid types
...
NET Framework error occurred during execution of user-defined routine or
aggregate "hierarchyid":
Microsoft
...
Types
...
Parse
failed because the input string '/1/1' is not a valid string representation of a
SqlHierarchyId node
...
SqlServer
...
HierarchyIdException:
at Microsoft
...
Types
...
Parse(SqlString input)

...
NET Framework error occurred during execution of user-defined routine or
aggregate "geography":
Microsoft
...
Types
...

Each geography instance must fit inside a single hemisphere
...

Microsoft
...
Types
...
SqlServer
...
GLNativeMethods
...
SqlServer
...
GLNativeMethods
...
SqlServer
...
SqlGeography
...
SqlServer
...
SqlGeography
...
SqlServer
...
SqlGeography
...
SqlServer
...
SqlGeography
...
SqlServer
...
SqlGeography
...


How do we create specific code paths to handle such exceptions? Despite the fact that they relate to
very different situations, as both exceptions occurred within managed code, the T-SQL error generated
in each case is the same—generic error 6522
...
Furthermore, we cannot easily add custom error-handling to the
original function code, since these are system-defined methods defined within the precompiled
Microsoft
...
Types
...

One approach would be to define new custom CLR methods that wrap around each of the systemdefined methods in SqlServer
...
dll, which check for and handle any CLR exceptions before
passing the result back to SQL Server
...
SqlServer
...
SqlFunction()]
public static SqlGeography GeogTryParse(SqlString Input)
{
SqlGeography result = new SqlGeography();
try
{
result = SqlGeography
...
The exceptions generated
by the system-defined CLR types have five-digit exception numbers in the range 24000 to 24999, so can
be distilled from the ERROR_MESSSAGE() string using the T-SQL PATINDEX function
...


Transactions and Exceptions
No discussion of exceptions in SQL Server can be complete without mentioning the interplay between
transactions and exceptions
...

SQL Server is a database management system (DBMS), and as such one of its main goals is
management and manipulation of data
...


The Myths of Transaction Abortion
The biggest mistake that some developers make is the assumption that if an exception occurs during a
transaction, that transaction will be aborted
...
Most transactions
will live on even in the face of exceptions, as running the following T-SQL will show:
BEGIN TRANSACTION;
GO
SELECT 1/0 AS DivideByZero;
GO
SELECT @@TRANCOUNT AS ActiveTransactionCount;
GO

96

CHAPTER 4

ERRORS AND EXCEPTIONS

The output from this T-SQL is as follows:
DivideByZero
-----------Msg 8134, Level 16, State 1, Line 1
Divide by zero error encountered
...
Alas, this is
also not the case, as the following T-SQL proves:
--Create a table for some data
CREATE TABLE SomeData
(
SomeColumn int
);
GO
--This procedure will insert one row, then throw a divide-by-zero exception
CREATE PROCEDURE NoRollback
AS
BEGIN
INSERT INTO SomeData VALUES (1);
INSERT INTO SomeData VALUES (1/0);
END;
GO
--Execute the procedure
EXEC NoRollback;
GO
--Select the rows from the table

97

CHAPTER 4

ERRORS AND EXCEPTIONS

SELECT *
FROM SomeData;
GO
The result is that, even though there is an error, the row that didn’t throw an exception is still in the
table; there is no implicit transaction arising from the stored procedure:
SomeColumn
1
Even if an explicit transaction is begun in the stored procedure before the inserts and committed
after the exception occurs, this example will still return the same output
...
It will simply serve as a message
that something went wrong
...
In addition to making
exceptions act like batch-level exceptions, the setting also causes any active transactions to immediately
roll back in the event of an exception
...
The
following T-SQL shows a much more atomic stored procedure behavior than the previous example:
--Empty the table
TRUNCATE TABLE SomeData;
GO
--This procedure will insert one row, then throw a divide-by-zero exception
CREATE PROCEDURE XACT_Rollback
AS

98

CHAPTER 4

ERRORS AND EXCEPTIONS

BEGIN
SET XACT_ABORT ON;
BEGIN TRANSACTION;
INSERT INTO SomeData VALUES (1);
INSERT INTO SomeData VALUES (1/0);
COMMIT TRANSACTION;
END;
GO
--Execute the procedure
EXEC XACT_Rollback;
GO
--Select the rows from the table
SELECT *
FROM SomeData;
GO
This T-SQL results in the following output, which shows that no rows were inserted:
Msg 8134, Level 16, State 1, Procedure XACT_Rollback, Line 10
Divide by zero error encountered
...
I recommend turning this setting on
in any stored procedure that uses an explicit transaction, in order to guarantee that it will get rolled back
in case of an exception
...
In this case the transaction is not automatically rolled back, as it is
with XACT_ABORT; instead, SQL Server throws an exception letting the caller know that the transaction

99

CHAPTER 4

ERRORS AND EXCEPTIONS

cannot be committed, and must be manually rolled back
...

COMMIT TRANSACTION;
END CATCH
GO
This results in the following output:
Msg 3930, Level 16, State 1, Line 10
The current transaction cannot be committed and cannot support
operations that write to the log file
...

Should a transaction enter this state, any attempt to either commit the transaction or roll forward
(do more work) will result in the same exception
...

In order to determine whether an active transaction can be committed or rolled forward, check the
value of the XACT_STATE function
...
It is a good
idea to always check XACT_STATE in any CATCH block that involves an explicit transaction
...

A solid understanding of how exceptions behave within SQL Server makes working with them much
easier
...

SQL Server’s TRY/CATCH syntax makes dealing with exceptions much easier, but it’s important to use
the feature wisely
...
And
whenever dealing with transactions in CATCH blocks, make sure to check the value of XACT_STATE
...


100

CHAPTER 5

Privilege and Authorization
SQL Server security is a broad subject area, with enough potential avenues of exploration that entire
books have been written on the topic
...

Broadly speaking, data security can be broken into two areas:
•

Authentication: The act of verifying the identity of a user of a system

•

Authorization: The act of giving a user access to the resources that a system
controls

These two realms can be delegated separately in many cases; so long as the authentication piece
works properly, the user can be handed off to authorization mechanisms for the remainder of a session
...
While
production DBAs should be very concerned with these sorts of issues, authentication is an area that
developers can mostly ignore
...

This chapter introduces some of the key issues of data privilege and authorization in SQL Server
from a development point of view
...
A related security topic is that
of data encryption, which is covered in detail in the next chapter
...
Development environments tend to
be set up with very lax security in order to keep things simple, but a solid development process should
include a testing phase during which full authentication restrictions are applied
...
Application Logins
The topics covered in this chapter relate to various privilege and authorization scenarios handled within
SQL Server itself
...
In such applications, users typically connect and log
into the application using their own personal credentials, but the application then connects to the database
using a single shared application login
...

There are some benefits to using this approach, such as being able to take advantage of connection
pooling between different sessions
...
If a bug were to exist in the application, or if the credentials associated with
the application login were to become known, it would be possible for users to execute any queries against
the database that the application had permission to perform
...


The Principle of Least Privilege
The key to locking down resources in any kind of system—database or otherwise—is quite simple in
essence: any given user should have access to only the bare minimum set of resources required, and for
only as much time as access to those resources is needed
...

Many multiuser operating systems implement the ability to impersonate other users when access to
a resource owned by that user is required
...
The most common example of this at an operating system level is UNIX’s su
command, which allows a user to temporarily take on the identity of another user, easily reverting back
when done
...
NET WindowsIdentity class
...
Granting
permission to a resource means adding a user to the list, after which the user can access the resource
again and again, even after logging in and out of the system
...
By taking control of
an account, the attacker automatically has full access to every resource that the account has permission
to access
...
In addition, rights to the resource will only be
maintained during the course of impersonation
...
e
...
In effect, this means that if an account is compromised,

102

CHAPTER 5

PRIVILEGE AND AUTHORIZATION

the attacker will akso have to compromise the impersonation context in order to gain access to more
secure resources
...
This is generally
implemented using proxies—users (or other security principals) that have access to a resource but
cannot be authenticated externally
...
Accessing
more valuable resources requires additional work on the part of the attacker, giving you that much more
of a chance to detect problems before they occur
...

•

At the server level, proxy logins can be created that cannot log in
...


The only way to switch into the execution context of either of these types of proxy principals is via
impersonation, which makes them ideal for privilege escalation scenarios
...
Certificates are
covered in more detail in Chapter 6, but for now think of a certificate as a trusted way to verify the
identity of a principal without a password
...
(Note that before a certificate can be created in any database, a master key must be created
...
)
USE master;
GO
CREATE CERTIFICATE Dinesh_Certificate
ENCRYPTION BY PASSWORD = 'stR0n_G paSSWoRdS, pLE@sE!'
WITH SUBJECT = 'Certificate for Dinesh';
GO
Once the certificate has been created, a proxy login can be created using the CREATE LOGIN FROM
CERTIFICATE syntax as follows:
CREATE LOGIN Dinesh
FROM CERTIFICATE Dinesh_Certificate;
GO
This login can be granted permissions, just like any other login
...
This is done by creating a user using the same certificate

103

CHAPTER 5

PRIVILEGE AND AUTHORIZATION

that was used to create the login, using the CREATE USER FOR CERTIFICATE syntax
...


Database-Level Proxies
Proxy principals that operate at the database level can be created by adding a user to the database that is
not associated with a server login
...
However, it is
impossible to log into the server and authenticate as Bob
...

Only then can you impersonate Bob, taking on whatever permissions the user is assigned
...


Data Security in Layers: The Onion Model
Generally speaking, the more levels that an attacker must penetrate in order to access a valuable
resource, the better the chance of being able to prevent their attack
...

The first layer of defense is everything outside of the database server, all of which falls into the realm
of authentication
...

From there, each user is authorized to access specific resources in the database
...
By assigning permissions only via
stored procedures, it is possible to maintain greater control over when and why escalation should take
place—but more on that will be covered later in this chapter
...

Figure 5-1 shows some of the layers that should be considered when defining a SQL Server security
scheme, in order to maximize the protection with which sensitive data is secured
...

A stored procedure layer provides an ideal layer of abstraction between data access methods and
the data itself, allowing for additional security to be programmed in via parameters or other inline logic
...
Likewise, a stored procedure might be used to force users to access data on a
granular basis by requiring parameters that are used as predicates to filter data
...


104

CHAPTER 5

PRIVILEGE AND AUTHORIZATION

Figure 5-1
...


Data Organization Using Schemas
SQL Server 2008 supports ANSI standard schemas, which provide a method by which tables and other
objects can be segmented into logical groups
...
This makes tasks such as managing authorization considerably easier since, by dividing your
database into schemas, you can easily group related objects and control permissions without having to
worry about what objects might be added or removed from that collection in the future
...

To create a schema, use the CREATE SCHEMA command
...
If an owner is not
explicitly specified, SQL Server will assign ownership to the user that creates the schema
...
SalesData
(
SaleNumber int,
SaleDate datetime
);
GO
If an object belongs to a schema, then it must be referenced with its associated schema name; so to
select from the SalesData table, the following SQL is used:

105

CHAPTER 5

PRIVILEGE AND AUTHORIZATION

SELECT *
FROM Sales
...
g
...
SalesData)
...


The beauty of schemas becomes obvious when it is time to apply permissions to the objects in the
schema
...
For instance, after the
following T-SQL is run, the Alejandro user will have access to select rows from every table in the Sales
schema, even if new tables are added later:
CREATE USER Alejandro
WITHOUT LOGIN;
GO
GRANT SELECT ON SCHEMA::Sales
TO Alejandro;
GO
It’s important to note that, when initially created, the owner of any object in a schema will be the
same as the owner of the schema itself
...
This is
especially important for ownership chaining, covered later in this chapter
...
By using ALTER SCHEMA with the TRANSFER option, you can specify that a table should be moved to
another schema:
--Create a new schema
CREATE SCHEMA Purchases;
GO
--Move the SalesData table into the new schema
ALTER SCHEMA Purchases
TRANSFER Sales
...
SalesData;
GO
Schemas are a powerful feature, and I recommend that you consider using them any time you’re
dealing with sets of tables that are tightly related to one another
...
The multiple databases can be consolidated to a single database that uses schemas
...


Basic Impersonation Using EXECUTE AS
Switching to a different user’s execution context has long been possible in SQL Server, using the SETUSER
command, as shown in the following code listing:
SETUSER 'Alejandro';
GO
To revert back to the previous context, call SETUSER again without specifying a username:
SETUSER;
GO
The SETUSER command is only available to members of the sysadmin or db_owner roles (at the server and
database levels, respectively), and is therefore not useful for setting up least-privilege scenarios
...

The EXECUTE AS command can be used by any user, and access to impersonate a given user or
server login is controlled by a permissions setting rather than a fixed role
...
SETUSER, on the
other hand, leaves the impersonated context active when control is returned to the caller
...


107

CHAPTER 5

PRIVILEGE AND AUTHORIZATION

To show the effects of EXECUTE AS, start by creating a new user and a table owned by the user:
CREATE USER Tom
WITHOUT LOGIN;
GO
CREATE TABLE TomsData
(
AColumn int
);
GO
ALTER AUTHORIZATION ON TomsData TO Tom;
GO
Once the user is created, it can be impersonated using EXECUTE AS, and the impersonation context
can be verified using the USER_NAME() function:
EXECUTE AS USER = 'Tom';
GO
SELECT USER_NAME();
GO

Note In order to use the EXECUTE AS statement to impersonate another user or login, a user must have been
granted IMPERSONATE permissions on the specified target
...

Any action performed after running EXECUTE AS will use Tom’s credentials
...
However, an attempt to create a new table will fail,
since Tom does not have permission to do so:
--This statement will succeed
ALTER TABLE TomsData
ADD AnotherColumn datetime;
GO
--This statement will fail with CREATE TABLE PERMISSION DENIED
CREATE TABLE MoreData
(
YetAnotherColumn int
);
GO
Once you have completed working with the database in the context of Tom’s permissions, you can
return to the outer context by using the REVERT command
...
e
...
The USER_NAME() function can be checked at any time to find out
whose context you are executing under
...
The user will be given the right to impersonate Tom, using
GRANT IMPERSONATE:
CREATE USER Paul
WITHOUT LOGIN;
GO
GRANT IMPERSONATE ON USER::Tom TO Paul;
GO
If Paul is impersonated, the session will have no privileges to select rows from the TomsData table
...
You will lose any permissions that the outer user has that
the impersonated user does not have, in addition to gaining any permissions that the impersonated user
has that the outer user lacks
...
Since both
the USER_NAME() function and the SUSER_NAME() function will return the names associated with the
impersonated user, the ORIGINAL_LOGIN() function must be used to return the name of the outermost
server login
...


109

CHAPTER 5

PRIVILEGE AND AUTHORIZATION

What is a module?
Each of the privilege escalation examples that follow use stored procedures to demonstrate a particular
element of functionality
...
A module is defined as any kind of code container that
can be created inside of SQL Server: a stored procedure, view, user-defined function, trigger, or CLR
assembly
...
If a database user has
access to execute a stored procedure, and the stored procedure is owned by the same database user that
owns a resource being referenced within the stored procedure, the user executing the stored procedure
will be given access to the resource via the stored procedure
...

To illustrate, start by creating and switching to a new database:
CREATE DATABASE OwnershipChain;
GO
USE OwnershipChain;
GO
Now create two database users, Louis and Hugo:
CREATE USER Louis
WITHOUT LOGIN;
GO
CREATE USER Hugo
WITHOUT LOGIN;
GO

Note For this and subsequent examples in this chapter, you should connect to SQL Server using a login that is
a member of the sysadmin server role
...
This option is one way of creating the kind of
proxy users mentioned previously
...
To create an access path without granting direct
permissions to the table, a stored procedure could be created, also owned by Louis:
CREATE PROCEDURE SelectSensitiveData
AS
BEGIN
SET NOCOUNT ON;
SELECT *
FROM dbo
...
However, this only works when the following conditions are met:
1
...


2
...


If either of those conditions were not true, the ownership chain would break, and Hugo would have
to be authorized another way to select from the table
...
For example, ownership chaining will not work
with dynamic SQL (for more information on dynamic SQL, refer to Chapter 8)
...
To set up cross-database ownership chaining, the user that
owns the stored procedure and the referenced table(s) must be associated with a server-level login, and
each database must have the DB_CHAINING property set using the ALTER DATABASE command
...

I recommend that you avoid cross-database ownership chaining whenever possible, and instead
call stored procedures in the remote database
...
For example, moving databases to separate servers is much easier if they do not depend on one
another for authentication
...
Consider avoiding multiple databases altogether, if at
all possible
...
In these cases, you’ll have to use one of the two other kinds of privilege escalation provided
by SQL Server: an extension to stored procedures using the EXECUTE AS clause, or module signing using
certificates
...
With
certificates, permissions are additive rather than impersonated—the additional permissions provided by
the certificate extend rather than replace the permissions of the calling principal
...


Stored Procedures and EXECUTE AS
As described in a previous section in this chapter, the EXECUTE AS command can be used on its own in TSQL batches in order to temporarily impersonate other users
...
The examples in this section only focus on stored procedures,
but the same principles can also be applied to the other object types
...
SensitiveData;
END;
GO
When this stored procedure is executed by a user, all operations within the procedure will be
evaluated as if they are being run by the Louis user rather than by the calling user (as is the default
behavior)
...
When the stored procedure has completed execution, context will be automatically
reverted back to that of the caller
...
For instance, consider the following two users and associated
tables:
CREATE USER Kevin
WITHOUT LOGIN;
GO
CREATE TABLE KevinsData

112

CHAPTER 5

PRIVILEGE AND AUTHORIZATION

(
SomeData int
);
GO
ALTER AUTHORIZATION ON KevinsData TO Kevin;
GO
CREATE USER Hilary
WITHOUT LOGIN;
GO
CREATE TABLE HilarysData
(
SomeOtherData int
);
GO
ALTER AUTHORIZATION ON HilarysData TO Hilary;
GO
Both users, Kevin and Hilary, own tables
...
Likewise, if the procedure was owned by Hilary, then ownership chaining would not permit
access to the KevinsData table
...
The following
stored procedure shows how this might look:
CREATE PROCEDURE SelectKevinAndHilarysData
WITH EXECUTE AS 'Kevin'
AS
BEGIN
SET NOCOUNT ON;
SELECT *
FROM KevinsData
UNION ALL
SELECT *
FROM HilarysData;
END;
GO
ALTER AUTHORIZATION ON SelectKevinAndHilarysData TO Hilary;
GO
Because Hilary owns the stored procedure, ownership chaining will kick in and allow selection of
rows from the HilarysData table
...
In this way, both permission sets can
be used, combined within a single module
...
For more complex
permissions scenarios, it is necessary to consider signing stored procedures using certificates
...

Creating a certificate-based proxy is by far the most flexible way of applying permissions using a stored
procedure, as permissions granted via certificate are additive
...

To create a proxy user for a certificate, you must first ensure that your database has a database
master key (DMK)
...


Then create the certificate, followed by the associated user using the FOR CERTIFICATE syntax:
CREATE CERTIFICATE Greg_Certificate
WITH SUBJECT='Certificate for Greg';
GO
CREATE USER Greg
FOR CERTIFICATE Greg_Certificate;
GO
Once the proxy user is created, it can be granted permissions to resources in the database, just like
any other database user
...
This is where stored
procedure signing comes into play
...
Even if granted permission to execute this stored procedure,
any user (other than Greg) will be unable to do so successfully, as the stored procedure does not
propagate permissions to the GregsData table:
CREATE USER Linchi
WITHOUT LOGIN;
GO
GRANT EXECUTE ON SelectGregsData TO Linchi;
GO
EXECUTE AS USER='Linchi';
GO
EXEC SelectGregsData;
GO

115

CHAPTER 5

PRIVILEGE AND AUTHORIZATION

This attempt fails with the following error:
Msg 229, Level 14, State 5, Procedure SelectGregsData, Line 6
The SELECT permission was denied on the object 'GregsData', database
'OwnershipChain', schema 'dbo'
...
This can be done by signing the procedure using the
same certificate that was used to create the Greg user
...

The flexibility of certificate signing becomes apparent when you consider that you can sign a given
stored procedure with any number of certificates, each of which can be associated with different users
and therefore different permission sets
...

Keep in mind when working with certificates that any time the stored procedure is altered, all
signatures will be automatically revoked by SQL Server
...

It is also important to know how to find out which certificates, and therefore which users, are
associated with a given stored procedure
...
The following query, which returns all
stored procedures, the certificates they are signed with, and the users associated with the certificates,
can be used as a starting point:
SELECT
OBJECT_NAME(cp
...
name AS certificate_name,
dp
...
crypt_properties AS cp
INNER JOIN sys
...
thumbprint = cp
...
database_principals dp
ON SUBSTRING(dp
...
thumbprint;
This query is somewhat difficult to understand, so it is worth explaining here
...
crypt_properties view contains information about which modules have been signed by certificates
...
certificates view
...

When this query is executed, the results show the signed module just created as follows:

signed_module
SelectGregsData

certificate_name
Greg_Certificate

user_name
Greg

Assigning Server-Level Permissions
The previous example showed only how to assign database-level permissions using a certificate
...
Doing so requires creation of a proxy login from a certificate, followed by creation of a
database user using the same certificate
...
Once the database user is
created, the procedure to apply permissions is the same as when propagating database-level
permissions
...
Unlike previous examples, this certificate
must be encrypted by a password rather than by the database master key, in order to ensure its private
key remains encrypted when removed from the database
...
The next step is to grant the appropriate permissions to the login:
GRANT ALTER ANY DATABASE TO alter_db_login;
GO
At this point, you must back up the certificate to a file
...

BACKUP CERTIFICATE alter_db_certificate
TO FILE = 'C:\alter_db
...
pvk',

117

CHAPTER 5

PRIVILEGE AND AUTHORIZATION

ENCRYPTION BY PASSWORD = 'YeTan0tHeR$tRoNGpaSSWoRd?',
DECRYPTION BY PASSWORD = 'stR()Ng_PaSSWoRDs are?BeST!'
);
GO
Once backed up, the certificate can be restored to any user database
...
cer'
WITH PRIVATE KEY
(
FILE = 'C:\alter_db
...

It is worth noting that at this point, the certificate’s physical file should probably be either deleted or
backed up to a safe storage repository
...
And since the certificate is being
used to grant ALTER DATABASE permissions, such an attack could potentially end in some damage being
done—so play it safe with these files
...
Create
a stored procedure that requires the privilege escalation (in this case, the stored procedure will set the
database to MULTI_USER access mode), create a user based on the certificate, and sign the stored
procedure with the certificate:
CREATE PROCEDURE SetMultiUser
AS
BEGIN
ALTER DATABASE alter_db_example
SET MULTI_USER;
END;
GO
CREATE USER alter_db_user
FOR CERTIFICATE alter_db_certificate;
GO
ADD SIGNATURE TO SetMultiUser

118

CHAPTER 5

PRIVILEGE AND AUTHORIZATION

BY CERTIFICATE alter_db_certificate
WITH PASSWORD = 'stR()Ng_PaSSWoRDs are?BeST!';
GO
The permissions can now be tested
...
So this time, CREATE USER WITHOUT LOGIN will
not suffice:
CREATE LOGIN test_alter WITH PASSWORD = 'iWanT2ALTER!!';
GO
CREATE USER test_alter FOR LOGIN test_alter;
GO
GRANT EXECUTE ON SetMultiUser TO test_alter;
GO
Finally, the test_alter login can be impersonated, and the stored procedure executed:
EXECUTE AS LOGIN='test_alter';
GO
EXEC SetMultiUser;
GO
The command completes successfully, demonstrating that the test_alter login had been able to
exercise the ALTER DATABASE permissions granted via the stored procedure
...


Summary
SQL Server’s impersonation features allow developers to create secure, granular authorization schemes
...
A stored procedure layer can be used to control security, delegating permissions as
necessary based on a system of higher-privileged proxy users
...
Schemas can also be used to make assignment of permissions an easier
task, as permissions may not have to be maintained over time as the objects in the schema change
...
That said, you should try
to keep systems as simple and understandable as possible, in order to avoid creating maintainability
nightmares
...
Many of the
techniques laid out in this chapter are probably not necessary for the majority of applications
...


119

CHAPTER 6

Encryption
Encryption is the process of encoding data in such a way as to render it unusable to anyone except those
in possession of the appropriate secret knowledge (the key) required to decrypt that data again
...
If all other security mechanisms fail,
encryption can prove to be a very effective last line of defense in protecting confidential data
...
The physical act of encrypting data is relatively
straightforward, but without the appropriate procedures in place to securely manage access to that data,
and the keys required to decrypt it, any attempt to implement encryption is worthless
...

Furthermore, the additional protection afforded by encryption has an associated performance cost
...
This means that, particularly in large-scale environments, careful attention must be
paid to ensure that encryption does not adversely affect application performance
...
Rather, I will
provide a practical guide to using the main encryption features of SQL Server 2008 to create a secure,
scalable database application that is capable of working with confidential data
...
I will particularly focus on the two main issues identified in the
preceding paragraphs—namely, how to securely store and manage access to encryption keys, and how
to design a database application architecture that protects sensitive data while minimizing the negative
impact on performance
...
Encryption is a method of protecting data, so the key
questions to ask are what data are you trying to protect, and who (or what) are you trying to protect it
from
...
If it were not, then we wouldn’t bother storing it in a database in the first place
...


121

CHAPTER 6

ENCRYPTION

Most of the encryption methods in SQL Server 2008 provide cell-level encryption, which is applied
to individual items of data, or columns of a table that contain sensitive information
...
Not only do these
approaches operate at a different scope, but they have significantly different implications for the design
of any application working with encrypted data
...
Some examples of
the types of data that might require encryption are as follows:
•

Many organizations define one or more levels of sensitive data—that is,
information that could negatively harm the business if it were allowed into the
wrong hands
...
If a competitor were to get hold of such
information, it could have severe consequences on future sales
...
A failure to meet
these standards may result in a company facing severe fines, or even being forced
to cease trading
...
In some cases, detailed requirements specify
exactly those data items that must be encrypted, the type of encryption algorithm
used, the minimum length of encryption key, and the schedule by which keys
must be rotated
...


Conducting a thorough analysis of business requirements to identify all data elements that require
protection and determining how that data should be protected are crucial to ensure that any proposed
encryption solution adequately addresses security requirements while balancing the need for
performance
...


What Are You Protecting Against?
Perhaps the most basic protection that encryption can offer is against the risk posed to data at rest
...
If an unauthorized third party could gain access to those files, it would be
possible for them to copy and restore the database onto their own system, allowing them to browse
through your data at their leisure
...

Besides physical theft of database files, hackers and external threats pose a continuing risk to
database security
...
By ensuring that all data transmitted
over a network is securely encrypted, you can be sure that it can only be understood by its intended
recipient, reducing the chance of it being successfully intercepted and used against you
...

Disgruntled DBAs and developers testing the limits of their authorization can access data they do not
require in order to do their jobs
...
If properly designed and deployed, an encryption
strategy can prevent this risk from occurring
...
However, if implemented correctly, it can provide an
additional level of security that might be the difference between your data becoming exposed and
remaining secret as intended
...
In many ways, this is a good thing: these
algorithms have been developed by some of the world’s leading information theorists and, in general,
have been proven to withstand concentrated, deliberate attack
...

Shannon’s maxim, as the preceding statement is commonly known, is a restatement of earlier work
by the Dutch cryptographer Auguste Kerchoffs
...
Originally published in a French journal, le Journal des Sciences Militaires, in 1883,
these principles are still very relevant in modern cryptography
...
If we assume that Shannon’s maxim holds (i
...
, that the details of any
encryption algorithm used are public knowledge), then the security of your encrypted data rests entirely
on protection of the secret key that the algorithm uses to decrypt and encrypt data
...
Indeed, the issues of secure key management and distribution are among the most important
areas of modern cryptography, as a failure to properly protect encryption keys compromises your entire
security strategy
...


The Automatic Key Management Hierarchy
The way in which SQL Server addresses the problem of secure key management is to implement a
hierarchy of encryption keys, with each key providing protection for those keys below it
...


123

CHAPTER 6

ENCRYPTION

Figure 6-1
...


Symmetric Keys, Asymmetric Keys, and Certificates
At the lowest level, items of data are encrypted with a symmetric key, the public key of an asymmetric
key pair, or a certificate
...
Each of these three methods offers different advantages and disadvantages, which will be
discussed later in this chapter
...
For example, the following code listing
demonstrates the syntax required to create an asymmetric 1,024 bit key using the Rivest, Shamir, and
Adleman (RSA) algorithm, owned by the database user Tom:
CREATE ASYMMETRIC KEY ExampleAsymKey
AUTHORIZATION Tom
WITH ALGORITHM = RSA_1024;
In this example, there is no explicitly specified method of protection, so the key will be protected by
the automatic key hierarchy
...


124

CHAPTER 6

ENCRYPTION

Database Master Key
Each user database on a server can have its own DMK, which is a symmetric key stored in both the
master database and the user database
...

Even though protected by both the SMK and a password, by default SQL Server can automatically
open the DMK when required by decrypting with the SMK alone, without needing to be supplied with
the password
...
Users only require permissions on the individual dependent
keys or certificates, as SQL Server will open the DMK to access those keys as necessary
...

A DMK can be created by running the CREATE MASTER KEY statement in the database in which the key
is to be stored, as follows:

CREATE MASTER KEY ENCRYPTION BY PASSWORD = '5Tr()ng_p455woRD_4_dA_DMK!!!';

Note Each database can have only one associated DMK
...


Service Master Key
The SMK is a symmetric key that sits at the top of the SQL Server encryption hierarchy, and is used to
encrypt all DMKs stored on the server, as well as protect logins and credentials associated with linked
servers
...
If either one of these credentials
becomes invalid (such as, for example, if the service account is changed), SQL Server will re-create the
invalid key based on the remaining valid key
...
There is no command to create an SMK; it is created automatically the first
time it is required and stored in the master database
...
The hierarchical structure protected by the
SMK provides sufficient security for most situations, and makes key access relatively easy: permissions
to use individual keys or subtrees of keys within the hierarchy are granted to individuals or groups using
the GRANT command, and such keys can be opened and used by authorized users as required
...
The flexible encryption key
management structure in SQL Server 2008 provides several ways for doing this, as I’ll now discuss
...

For example, a symmetric key may be encrypted by another symmetric key, which is then protected by a
certificate protected by the DMK
...

Key rotation using a flat model, which is based on only a single encryption key, is very cumbersome
...
Now suppose that, as a result of a new client requirement, that encryption key had to be rotated
every month
...
In a layered model, you can rotate higher keys
very easily, since the only encrypted data they contain is the symmetric key used by the next lower level
of encryption
...
However, bear in mind that, although
properly handled key rotation generally improves security, key rotation does involve a certain level of
inherent risk
...

When creating additional layers of keys, you should try to ensure that higher levels of keys have the
same or greater encryption strength than the keys that they protect
...


Removing Keys from the Automatic Encryption Hierarchy
When a DMK is protected by both the SMK and a password, as in the default hierarchy, you do not need
to explicitly provide the password in order to open the DMK; when a user with appropriate permissions
requests a key protected by the DMK, SQL Server opens the DMK automatically using the SMK to access
the required dependent keys
...

In cases where you want to restrict access to encrypted data from individuals in these roles, it is
necessary to drop the DMK protection by SMK, which will enforce the DMK protection by password
instead
...

The problem with removing keys from the automatic key hierarchy is that it then becomes
necessary to have a system that protects the passwords that secure those keys
...


Extensible Key Management
SQL Server 2008 supports management of encryption keys by extensible key management (EKM)
methods
...

The following code listing illustrates the syntax required to create an asymmetric key in SQL Server
mapped to a key stored on an EKM device:
CREATE ASYMMETRIC KEY EKMAsymKey
FROM PROVIDER EKM_Provider
WITH
ALGORITHM = RSA_1024,
CREATION_DISPOSITION = OPEN_EXISTING,
PROVIDER_KEY_NAME = 'EKMKey1';
GO
EKM provides a total encryption management solution using dedicated hardware, which means
that SQL Server can leave the process of encryption to the HSM and concentrate on other tasks
...

Figure 6-2 illustrates the different encryption configurations discussed in this section that provide
alternatives to the default automatic key management hierarchy
...
To understand more about these differences, it’s necessary to have a
more detailed look at the various methods by which individual items of data can be encrypted, which
will be covered in the next part of this chapter
...


127

CHAPTER 6

ENCRYPTION

Figure 6-2
...
I will not attempt to
describe the full details of every method here; readers who are interested in a detailed description of
each method should consult a book dedicated to the subject, such as Expert SQL Server 2008 Encryption,
by Michael Coles (Apress, 2009)
...

By “deterministic,” I mean that a given input into the hashing function will always produce the same
output
...
However, hashing methods
are often used in conjunction with encryption, as will be demonstrated later in this chapter
...
For any given input x, HASHBYTES(x) will
always produce the same output y, but there is no method provided to retrieve x from the resulting value
y
...

HASHBYTES is useful in situations where you need to compare whether two secure values are the
same, but when you are not concerned with what the actual values are: for example, to verify that the
password supplied by a user logging into an application matches the stored password for that user
...

Although hash algorithms are deterministic, so that a given input value will always generate the
same hash, it is theoretically possible that two different source inputs will share the same hash
...
In a totally secure environment, you cannot be certain that simply because two hashes are
equal, the values that generated those hashes were the same
...
Of these, SHA1 is the strongest, and is the algorithm
you should specify in all cases unless you have a good reason otherwise
...
The HASHBYTES function in action
This result is entirely repeatable—you can execute the preceding query as many times as you want
and you will always receive the same output
...


129

CHAPTER 6

ENCRYPTION

The advantage of obtaining consistent output is that, from a data architecture point of view, hashed
data can be stored, indexed, and retrieved just like any other binary data, meaning that queries of
hashed data can be designed to operate efficiently with a minimum amount of query redesign
...

This risk becomes even more plausible in cases where attackers can make reasonable assumptions in
order to reduce the list of likely source values
...
If a hacker were to obtain a dataset containing hashes of such
passwords generated using the MD5 algorithm, they would only have to search for occurrences of
0x2AC9CB7DC02B3C0083EB70898E549B63 to identify all those users who had chosen to use “Password1” as
their password, for example
...


Symmetric Key Encryption
Symmetric key encryption methods use the same single key to perform both encryption and decryption
of data
...


Figure 6-4
...
That is
to say, you will obtain different results from encrypting the same item of data on different occasions,
even if it is encrypted with the same key each time
...

The strongest symmetric key supported is AES256
...

The following example illustrates how to create a new symmetric key, SymKey1, that is protected by a
password:
CREATE SYMMETRIC KEY SymKey1
WITH ALGORITHM = AES_256
ENCRYPTION BY PASSWORD = '5yMm3tr1c_K3Y_P@$$w0rd!';
GO

130

CHAPTER 6

ENCRYPTION

Note By default, symmetric keys created using the CREATE SYMMETRIC KEY statement are randomly generated
...
Since SymKey1 is protected by a
password, to open this key you must provide the associated password using the OPEN SYMMETRIC KEY
DECRYPTION BY PASSWORD syntax
...

Once you are finished using the key, you should close it again using CLOSE SYMMETRIC KEY (if you fail to
explicitly close any open symmetric keys, they will automatically be closed at the end of a session)
...


Decrypting data that has been encrypted using a symmetric key follows a similar process as for
encryption, but using the DECRYPTBYKEY method rather than the ENCRYPTBYKEY method
...

The following code listing illustrates how to decrypt the data encrypted with a symmetric key in the
previous example:
OPEN SYMMETRIC KEY SymKey1
DECRYPTION BY PASSWORD = '5yMm3tr1c_K3Y_P@$$w0rd!';
DECLARE @Secret nvarchar(255) = 'This is my secret message';
DECLARE @Encrypted varbinary(max);
SET @Encrypted = ENCRYPTBYKEY(KEY_GUID(N'SymKey1'),@secret);
SELECT CAST(DECRYPTBYKEY(@Encrypted) AS nvarchar(255));
CLOSE SYMMETRIC KEY SymKey1;
GO
This results in the original plain text message being retrieved:

This is my secret message
On occasions, you may wish to encrypt data without having to deal with the issues associated with
creating and securely storing a permanent symmetric key
...

ENCRYPTBYPASSPHRASE generates a symmetric key using the Triple DES algorithm based on the value
of a supplied password, and uses that key to encrypt a supplied plain text string
...
The key itself is never
stored at any point, and is only generated transiently as and when it is required
...
However, remember that
ENCRYPTBYPASSPHRASE uses nondeterministic symmetric encryption, so you will obtain different results
every time you execute this query
...
As with all the asymmetric and symmetric encryption methods provided
by SQL Server, the resulting decrypted data is returned using the varbinary datatype, and may need to

132

CHAPTER 6

ENCRYPTION

be converted into the appropriate format for your application
...
However, it is not without its
drawbacks:
Firstly, ciphertext generated by ENCRYPTBYPASSPHRASE can be decrypted only when
used in conjunction with the passphrase used to encrypt it
...
This means that you need to consider strategies for how to
backup the passphrase in a secure manner; there is not much point encrypting
your data if the passphrase containing the only knowledge required to access that
data is stored in an openly accessible place
...
Although this is a well-recognized and widely used standard, some
business requirements or client specifications may require stronger encryption
algorithms, such as those based on AES
...

The strength of the symmetric key is entirely dependent on the passphrase from
which it is generated, but SQL Server does not enforce any degree of password
complexity on the passphrase supplied to ENCRYPTBYPASSPHRASE
...

The supplied clear text can only be varchar or nvarchar type (although this
limitation is fairly easily overcome by CASTing any other datatype prior to
encryption)
...
See the sidebar
entitled “Protecting Information from the DBA” for more information on this topic
...

DBAs have permissions to view, insert, update, and delete data from any database table, and any DBAs in
the sysadmin role have control over every key and certificate held on a SQL Server instance
...
This password can either be used with the ENCRYPTBYPASSPHRASE function, or it
can be used to secure a certificate or asymmetric key that protects the symmetric key with which the data
is encrypted
...


Asymmetric Key Encryption
Asymmetric encryption is performed using a pair of related keys: the public key is used to encrypt data,
and the associated private key is used to decrypt the resulting ciphertext
...
Figure 6-5 illustrates the
asymmetric encryption model
...
Asymmetric key encryption
In addition to asymmetric key pairs, asymmetric encryption may also be implemented using a
certificate
...
509 standard are used to bind a public key to a given identity
entitled to encrypt data using the associated key
...
Self-signed

134

CHAPTER 6

ENCRYPTION

certificates were used in the last chapter to demonstrate one method of assigning permissions to stored
procedures
...
For every method that deals with
asymmetric key encryption in SQL Server, an equivalent method provides the same functionality for
certificate encryption
...
Assuming equal
key length, certificate and asymmetric encryption will provide the same encryption strength, and there are
no significant differences between the functionality available with either type
...
Self-signed certificates in SQL Server can only have a key length of 1024 bits,
whereas stronger asymmetric keys may be created with a private key length of 512, 1024, or 2048 bits
...

I recommend asymmetric encryption using certificates rather than asymmetric key pairs, as certificates
allow for additional metadata to be stored alongside the key (such as expiry dates), and certificates can
easily be backed up to a CER file using the BACKUP CERTIFICATE command, whereas a method to provide
the equivalent functionality for an asymmetric key pair is strangely lacking
...

The following code listing illustrates how to create a 1024-bit asymmetric key using the RSA
algorithm:
CREATE ASYMMETRIC KEY AsymKey1
WITH Algorithm = RSA_1024;
GO
Encrypting data using the asymmetric key follows a slightly different process than for symmetric
encryption, as there is no need to explicitly open an asymmetric key prior to encryption or decryption
...
This is one of the reasons that, even though asymmetric encryption
provides stronger protection than symmetric encryption, it is not recommended for encrypting any
column that will be used to filter rows in a query, or where more than one or two records will be
decrypted at a time
...


Transparent Data Encryption
Transparent data encryption (TDE) is a new feature available in the Developer and Enterprise editions of
SQL Server 2008
...
In addition, generally speaking, it is not necessary to deal with any
key management issues relating to TDE
...
In other words, the MDF and LDF files in which database information is saved are stored
in an encrypted state
...
The query engine can therefore operate on it just as it does any other kind of
data
...

TDE provides encryption of (almost) all data within any database for which it is enabled, based on
the database encryption key (DEK) stored in the user database
...

To enable TDE on a database, you first need to create a server certificate in the master database
(assuming that the master database already has a DMK):
USE MASTER;
GO
CREATE CERTIFICATE TDE_Cert
WITH SUBJECT = 'Certificate for TDE Encryption';
GO

136

CHAPTER 6

ENCRYPTION

Then use the CREATE DATABASE ENCRYPTION KEY statement to create the DEK in the appropriate user
database:
USE ExpertSqlEncryption;
GO
CREATE DATABASE ENCRYPTION KEY
WITH ALGORITHM = AES_128
ENCRYPTION BY SERVER CERTIFICATE TDE_Cert;
GO

Note The preceding code listing will generate a warning message advising you to make a backup of the server
certificate with which the DEK is encrypted
...
Should the certificate become unavailable, you will be unable to decrypt the DEK,
and all data in the associated database will be lost
...
dm_database_encryption_keys view, as follows:
SELECT
DB_NAME(database_id) AS database_name,
CASE encryption_state
WHEN 0 THEN 'Unencrypted (No database encryption key present)'
WHEN 1 THEN 'Unencrypted'
WHEN 2 THEN 'Encryption in Progress'
WHEN 3 THEN 'Encrypted'
WHEN 4 THEN 'Key Change in Progress'
WHEN 5 THEN 'Decryption in Progress'
END AS encryption_state,
key_algorithm,
key_length
FROM sys
...

Disabling TDE is as simple as the process to enable it:
ALTER DATABASE ExpertSqlEncryption
SET ENCRYPTION OFF;
GO
Note that, even after TDE has been turned off all user databases, the tempdb database will remain
encrypted until it is re-created, such as when the server is next restarted
...
dm_database_encryption_keys, please consult
http://msdn
...
com/en-us/library/bb677274
...


The benefits of TDE are as follows:
•

Data at rest is encrypted (well, not quite all data at rest; see the following section)
...


•

TDE can be applied to existing databases without requiring any application
recoding
...


•

Performing encryption and decryption at the I/O level is efficient, and the overall
impact on database performance is small
...


TDE is certainly a useful addition to the cell-level encryption methods provided by SQL Server, and
if used correctly can serve to strengthen protection of your data
...
Although on the surface TDE appears to
simplify the process of encryption, in practice it hides the complexity that is necessary to ensure a truly
secure solution
...
However, the ease with which TDE can be enabled means that, in some cases, it is a
simpler option to turn on TDE than perform a proper analysis of each data element
...

There are also some important differences between TDE and the cell-level methods discussed
previously:
•

TDE only protects data at rest
...
All data that a user has
permission to access will always be presented to the user in an unencrypted state
...


•

There is a performance impact on every query run against a database that has TDE
enabled, whether that query contains “confidential” items of data or not
...


•

TDE negates any reduction in size of database files achieved from using SQL
Server 2008’s compression options
...


•

Not all data can be encrypted
...
Since this data does not pass through the SQL OS, it
cannot be encrypted by TDE
...
Cell-level encryption methods, such as ENCRYPTBYKEY and
ENCRYPTBYASYMKEY, encrypt an individual item or column of data
...
As such, TDE has more in common with encryption methods provided at the operating
system level, such as encrypting file system (EFS) technology and Windows BitLocker
...
I will therefore not consider it in the
remaining sections in this chapter
...
each have their own advantages and
disadvantages, so how do you choose between them? In this section I’ll describe an architecture that
means that you don’t have to make a compromise between different individual encryption methods;
rather, you can combine the best features of each approach into a single hybrid model
...


139

CHAPTER 6

ENCRYPTION

Figure 6-6
...

The elements of the hybrid encryption structure can be described as follows:
•

Data is encrypted using a symmetric key, which provides the best-performing
encryption method
...
Longer keys provide more
protection, but also require more processing overhead
...
I tend to create
one certificate for each user or group of users that need access to the symmetric
key
...


•

Each certificate is protected with a (strong) password, so that it can only be
accessed by individuals with knowledge of the password
...


•

Further layers of encryption can be added to this model by protecting the
symmetric key with further symmetric keys prior to encryption by the certificate
...


To illustrate how to create the hybrid encryption model just described, first create two users:
CREATE USER FinanceUser WITHOUT LOGIN;
CREATE USER MarketingUser WITHOUT LOGIN;
GO
Then create a certificate for each user, each with their own password:

140

CHAPTER 6

ENCRYPTION

CREATE CERTIFICATE FinanceCertificate
AUTHORIZATION FinanceUser
ENCRYPTION BY PASSWORD = '#F1n4nc3_P455w()rD#'
WITH SUBJECT = 'Certificate for Finance',
EXPIRY_DATE = '20101031';
CREATE CERTIFICATE MarketingCertificate
AUTHORIZATION MarketingUser
ENCRYPTION BY PASSWORD = '-+M@Rket1ng-P@s5w0rD!+-'
WITH SUBJECT = 'Certificate for Marketing',
EXPIRY_DATE = '20101105';
GO
We’ll also create a sample table and give both users permission to select and insert data into the
table:
CREATE TABLE Confidential (
EncryptedData varbinary(255)
);
GO
GRANT SELECT, INSERT ON Confidential TO FinanceUser, MarketingUser;
GO
Now that we have the basic structure set up, we will create a shared symmetric key that will be
encrypted by both the finance and marketing certificates
...
Instead, we will create the key with protection by the first certificate, and then open and
alter the key to add encryption by the second certificate too:
-- Create a symmetric key protected by the first certificate
CREATE SYMMETRIC KEY SharedSymKey
WITH ALGORITHM = AES_256
ENCRYPTION BY CERTIFICATE FinanceCertificate;
GO
-- Then OPEN and ALTER the key to add encryption by the second certificate
OPEN SYMMETRIC KEY SharedSymKey
DECRYPTION BY CERTIFICATE FinanceCertificate
WITH PASSWORD = '#F1n4nc3_P455w()rD#';
ALTER SYMMETRIC KEY SharedSymKey
ADD ENCRYPTION BY CERTIFICATE MarketingCertificate;
CLOSE SYMMETRIC KEY SharedSymKey;
GO

141

CHAPTER 6

ENCRYPTION

Finally, we need to grant permissions on the symmetric key for each user:
GRANT VIEW DEFINITION ON SYMMETRIC KEY::SharedSymKey TO FinanceUser
GRANT VIEW DEFINITION ON SYMMETRIC KEY::SharedSymKey TO MarketingUser
GO
To encrypt data using the shared symmetric key, either user must first open the key by specifying
the name and password of their certificate that is protecting it
...
The following listing demonstrates how FinanceUser
can insert data into the Confidential table, encrypted using the shared symmetric key:
EXECUTE AS USER = 'FinanceUser';
OPEN SYMMETRIC KEY SharedSymKey
DECRYPTION BY CERTIFICATE FinanceCertificate
WITH PASSWORD = '#F1n4nc3_P455w()rD#';
INSERT INTO Confidential
SELECT ENCRYPTBYKEY(KEY_GUID(N'SharedSymKey'), N'This is shared information
accessible to finance and marketing');
CLOSE SYMMETRIC KEY SharedSymKey;
REVERT;
GO
To decrypt this data, either the marketing user or the finance user can explicitly open the
SharedSymKey key prior to decryption, similar to the pattern used for encryption, or they can take
advantage of the DECRYPTBYKEYAUTOCERT function, which allows you to automatically open a symmetric
key protected by a certificate as part of the inline function that performs the decryption
...
Using the hybrid model described here, this is very easy to achieve—
simply create a new symmetric key that is protected by the existing finance certificate and grant
permissions on that key to the finance user
...

The beauty of this approach is that, since all of the keys to which a given user has access are
protected by a single certificate, the DECRYPTBYKEYAUTOCERT method can be used to decrypt the entire
column of data, whatever key was used to encrypt the individual values
...

In the following section, I’ll show you how to write efficient queries against encrypted data held in
such a model
...

The security of encryption always comes with an associated performance cost
...
However, the special nature of encrypted data has wider-ranging implications than accepting a
simple performance hit, and particular attention needs to be paid to any activities that involve ordering,
filtering, or performing joins on encrypted data
...
Most
data can be assigned a logical order: for example, varchar or char data can be arranged alphabetically;
datetime data can be ordered chronologically; and int, decimal, float, and money data can be arranged in
numeric order
...
This creates a number of issues for efficient query design
...
The
following code listing will create a table and populate it with 100,000 rows of randomly generated 16digit numbers, representing dummy credit card numbers
...
It is this known value that we will use to test various methods of searching
for values within encrypted data
...
spt_values a,
MASTER
...
Of course, the 100,000 randomly generated rows might by chance
contain valid credit card details, but good luck finding them!
To protect this information, we’ll follow the hybrid encryption model described in the previous
section
...

To begin, add a new column to the CreditCards table, CreditCardNumber_Sym, and populate it with
values encrypted using the CreditCard_SymKey symmetric key:
ALTER TABLE CreditCards ADD CreditCardNumber_Sym varbinary(100);
GO
OPEN SYMMETRIC KEY CreditCard_SymKey
DECRYPTION BY CERTIFICATE CreditCard_Cert
WITH PASSWORD = '#Ch0o53_@_5Tr0nG_P455w0rD#';
UPDATE CreditCards
SET CreditCardNumber_Sym =
ENCRYPTBYKEY(KEY_GUID('CreditCard_SymKey'),CreditCardNumber_Plain);
CLOSE SYMMETRIC KEY CreditCard_SymKey;
GO
Now let’s assume that we are designing an application that requires searching for specific credit
numbers that have been encrypted in the CreditCardNumber_Sym column
...
We’ll search for our chosen credit
card number by decrypting the CreditCardNumber_Sym column and comparing the result to the search
string:
DECLARE @CreditCardNumberToSearch nvarchar(32) = '4005550000000019';
SELECT * FROM CreditCards
WHERE DECRYPTBYKEYAUTOCERT(
CERT_ID('CreditCard_Cert'),
N'#Ch0o53_@_5Tr0nG_P455w0rD#',
CreditCardNumber_Sym) = @CreditCardNumberToSearch;
GO
A quick glance at the execution plan shown in Figure 6-7 reveals that, despite the index, the query
must scan the whole table, decrypting each row in order to see if it satisfies the predicate
...
A query scan performed on encrypted data
The need for a table scan is caused by the nondeterministic nature of encrypted data
...
Nor can we simply encrypt the search value 4005550000000019 and
search for the corresponding encrypted value in the CreditCard_Sym column, because the result is
nondeterministic and will differ every time
...
We can obtain a quick measure of the overall performance by inspecting
sys
...
text,
CAST(qs
...
execution_count / 1000
AS Avg_CPU_Time_ms,
qs
...
dm_exec_query_stats qs
CROSS APPLY sys
...
plan_handle) st
WHERE
st
...
079

71623

DECLARE @CreditCardNumberToSearch nvarchar(32)
= '4005550000000019';

--*SELECT-------------

The SELECT query is taking nearly 1
...
Also notice that the query text contained in the text column of sys
...

The appropriate solution to the problem of searching encrypted data differs depending on how the
data will be queried—whether it will be searched for a single exact matching row to the search string (an
equality match), a pattern match using a wildcard ( i
...
, LIKE %), or a range search (i
...
, using the BETWEEN,
>, or < operators)
...
Remember that the output of a hashing function is
deterministic—any given input will always lead to the same output, which is a crucial property to allow
encrypted data to be efficiently searched for any given search value
...

However, we don’t want to store the simple hash of each credit card number as returned by
HASHBYTES—as explained previously, hash values can be attacked using dictionaries or rainbow tables,
especially if, as in the case of a credit card number, they follow a predetermined format
...

HMACs, designed to validate the authenticity of a message, overcome this problem by combining a
hash function with a secret key (salt)
...


Message Authentication Codes
HMACs are a method of verifying the authenticity of a message to prove that it has not been tampered with
in transmission
...
The sender and the receiver agree on a secret value, known only to them
...

2
...

They then create a hash of the combined message with the salt value:

148

CHAPTER 6

ENCRYPTION

HMAC = HASH( “This is the original message” + “salt”)
3
...

4
...
They then check to make sure that the result is the
same as the supplied HMAC value
...
If somebody had attempted to intercept and tamper with the message,
it would no longer match the supplied HMAC value
...

The first step toward implementing an HMAC-based solution is to define a secure way of storing the
salt value (or key)
...

CREATE ASYMMETRIC KEY HMACASymKey
WITH ALGORITHM = RSA_1024
ENCRYPTION BY PASSWORD = N'4n0th3r_5tr0ng_K4y!';
GO
CREATE TABLE HMACKeys (
HMACKeyID int PRIMARY KEY,
HMACKey varbinary(255)
);
GO
INSERT INTO HMACKeys
SELECT
1,
ENCRYPTBYASYMKEY(ASYMKEY_ID(N'HMACASymKey'), N'-->Th15_i5_Th3_HMAC_kEy!');
GO
Now we can create an HMAC value based on the key
...
If the message length equals or exceeds this value, then any salt value
appended after the message content will be disregarded
...


•

The HMAC specification, as defined by the Federal Information Processing
Standard (FIPS PUB 198, http://csrc
...
gov/publications/fips/fips198/fips-198a
...
Key1 is
padded with the byte 0x5c and Key2 is padded with the byte 0x36
...

•

The HMAC standard does not specify which hashing algorithm is used to perform
the hashing, but the strength of the resulting HMAC is directly linked to the
cryptographic strength of the underlying hash function on which it is based
...
The hashing methods provided by the
...


It seems that SQL Server’s basic HASHBYTES function leaves a little bit to be desired, but fortunately
there is an easy solution
...
NET System
...
The following code listing illustrates the C# required to
create a reusable class that can be used to generate HMAC values for all supported hash algorithms for a
given key:
[SqlFunction(IsDeterministic = true, DataAccess = DataAccessKind
...
IsNull || PlainText
...
IsNull) {
return SqlBytes
...
Value)
{
case "MD5":
HMac = new HMACMD5(Key
...
Value);
break;
case "SHA256":
HMac = new HMACSHA256(Key
...
Value);
break;

150

CHAPTER 6

ENCRYPTION

case "SHA512":
HMac = new HMACSHA512(Key
...
Value);
break;
default:
throw new Exception( "Hash algorithm not recognised" );
}
byte[] HMacBytes = HMac
...
Value);
return new SqlBytes(HMacBytes);
}

Caution For completeness, the GenerateHMAC function supports all hashing algorithms available within the
Security
...
In practice, it is not recommended to use the MD5 algorithm in production

applications, since it has been proven to be vulnerable to hash collisions
...
Then create a new column in the CreditCards table
and populate it with an HMAC-SHA1 value using the GenerateHMAC function
...
These steps are demonstrated in the following code listing:
ALTER TABLE CreditCards
ADD CreditCardNumber_HMAC varbinary(255);
GO
-- Retrieve the HMAC salt value from the MACKeys table
DECLARE @salt varbinary(255);
SET @salt = (
SELECT DECRYPTBYASYMKEY(
ASYMKEY_ID('HMACASymKey'),
HMACKey,
N'4n0th3r_5tr0ng_K4y!'
)
FROM HMACKeys
WHERE HMACKeyID = 1);
-- Update the HMAC value using the salt
UPDATE CreditCards
SET CreditCardNumber_HMAC = (
SELECT dbo
...
Because I’m only concentrating on evaluating the performance of HMACs, I
won’t bother with this step
...
Queries issued against the CreditCards table will filter rows
based on CreditCardNumber_HMAC column, but we’ll want to return the CreditCardNumber_Sym column so
that we can decrypt the original credit card number
...

CREATE NONCLUSTERED INDEX idxCreditCardNumberHMAC
ON CreditCards (CreditCardNumber_HMAC)
INCLUDE (CreditCardNumber_Sym);
GO

Note A covering index contains all of the columns required by a query within a single index
...


Let’s test the performance of the new HMAC-based solution by searching for our known credit card
in the CreditCards table:
-- Select a credit card to search for
DECLARE @CreditCardNumberToSearch nvarchar(32) = '4005550000000019';
-- Retrieve the secret salt value
DECLARE @salt varbinary(255);
SET @salt = (
SELECT DECRYPTBYASYMKEY(
ASYMKEY_ID('HMACASymKey'),
MACKey,
N'4n0th3r_5tr0ng_K4y!'
)
FROM MACKeys);
-- Generate the HMAC of the credit card to search for

152

CHAPTER 6

ENCRYPTION

DECLARE @HMACToSearch varbinary(255);
SET @HMACToSearch = dbo
...
Having
found a matching row, it then decrypts the value contained in the CreditCardNumber_Sym column to
make sure that it matches the original supplied search value (to prevent the risk of hash collisions, where
an incorrect result is returned because it happened to share the same hash as our search string)
...


Figure 6-8
...
dm_exec_query_stats reveal that the query is
substantially more efficient than direct querying of the CreditCardNumber_Sym column, taking only
253ms of CPU time and requiring nine logical reads
...
This applies to searches of encrypted data just as with any other type of
data
...


153

CHAPTER 6

ENCRYPTION

The HMAC solution just proposed cannot be used in these cases, since the HMAC hash values are
unique to an exact value
...
In other
words, to support a query to find values LIKE 'The quick brown fox jumped over the lazy%', it would
be necessary to search for all rows matching the hash value of the trimmed string The quick brown fox
jumped over the lazy
...

To relate this back to the example used in this chapter, suppose that we wanted to create a method
of allowing users to search for a row of data in the CreditCards table based on the last four digits of the
credit card number
...
This
new column, CreditCardNumber_Last4HMAC, will only be used to support wildcard searches, while the
existing CreditCardNumber_HMAC column will continue to be used for equality searches on the whole
credit card number
...
GenerateHMAC(
'SHA256',
CAST(RIGHT(CreditCardNumber_Plain, 4) AS varbinary(max)), @salt));
GO
And, to support queries issued against the new substring HMAC, we’ll add a new index:
CREATE NONCLUSTERED INDEX idxCreditCardNumberLast4HMAC
ON CreditCards (CreditCardNumber_Last4HMAC)
INCLUDE (CreditCardNumber_Sym);
GO
Suppose that we wanted to issue a query to find all those credit cards ending in the digits 0019
...
GenerateHMAC(
'SHA256',
CAST(@CreditCardLast4ToSearch AS varbinary(max)),
@salt);

155

CHAPTER 6

ENCRYPTION

-- Retrieve the matching row from the CreditCards table
SELECT
CAST(
DECRYPTBYKEYAUTOCERT(
CERT_ID('CreditCard_Cert'),
N'#Ch0o53_@_5Tr0nG_P455w0rD#',
CreditCardNumber_Sym)
AS nvarchar(32)) AS CreditCardNumber_Decrypted
FROM
CreditCards
WHERE
CreditCardNumber_Last4HMAC = @HMACToSearch
AND
CAST(
DECRYPTBYKEYAUTOCERT(
CERT_ID('CreditCard_Cert'),
N'#Ch0o53_@_5Tr0nG_P455w0rD#',
CreditCardNumber_Sym)
AS nvarchar(32)) LIKE '%' + @CreditCardLast4ToSearch;
GO
The results obtained on my system list the following ten credit cards, including the
4005550000000019 credit card number that we were originally searching for (your list will differ since the
records contained in the CreditCards table were randomly generated, but you should still find the
intended matching value):
6823807913290019
5804462948450019
6742201580250019
7572953718590019
4334945301620019
7859588437020019
3490887629240019
5801804774470019
4005550000000019
2974981474970019

156

CHAPTER 6

ENCRYPTION

An examination of the query execution plan shown in Figure 6-9 reveals that, so long as the
substring chosen is sufficiently selective, as in this case, wildcard searches of encrypted data can be
achieved using an efficient index seek
...
Index seek of an HMAC substring
The performance of this query on my machine is almost identical to the full equality match against
the CreditCard_HMAC column, taking 301ms of CPU time and requiring nine logical reads
...
The CreditCardNumber_Last4HMAC column used in this example can
only be used to satisfy queries that specify the last four digits of a credit card
...

The more flexibility your application demands in its search criteria, the more untenable this
solution becomes, and the more additional columns of data must be created
...
Even when storing relatively secure HMAC hash values, every additional column
provides a hacker with more information about your data, which can be used to launch a targeted attack
...
For
example, how do we identify those rows of data that contain credit card numbers in the range of
4929100012347890 and 4999100012349999?
These types of searches are perhaps the most difficult to implement with encrypted data, and there
are few obvious solutions: The HMAC solution, or its substring derivative, does not help here—
sequential plain text numbers clearly do not lead to sequential ciphertext values
...

If the range to be searched were very constrained (e
...
, +/–10 of a supplied value), it might be
possible to forego encryption and store partial plain text values, such as the last two digits of the credit
card, in a new column
...

The only other alternative, if range queries absolutely must be supported, is to rely on one of the
forms of encryption performed at the I/O level, such as database encryption using TDE, EFS, or
Windows BitLocker
...


157

CHAPTER 6

ENCRYPTION

Summary
Encryption forms an important part of any security strategy, and can provide the crucial last line of
defense in thwarting attackers from gaining access to sensitive or confidential data
...
Added security also has a necessary negative impact on performance
...
This means that many standard database tasks, such as sorting, filtering, and
joining, must be redesigned in order to work effectively
...


158

CHAPTER 7

SQLCLR: Architecture and
Design Considerations
When Microsoft first announced that SQL Server would host the
...
Some of that
excitement was enthusiastic support voiced by developers who envisaged lots of database scenarios that
could potentially benefit from the methods provided by the
...
However, there was
also considerable nervousness and resistance from DBAs concerned about the threats posed by the new
technology and the rumors that rogue developers would be able to create vast worlds of DBAimpenetrable, compiled in-process data access code
...
Those hoping to use the SQLCLR features as a wholesale replacement for T-SQL were
quickly put off by the fact that writing CLR routines generally requires more code, and performance and
reliability suffer due to the continual cost of marshaling data across the CLR boundaries
...
NET developers to begin with, there was a somewhat steep learning curve involved
for a feature that really didn’t have a whole lot of uses
...
SQL Server
2008 lifts the previous restriction that constrained CLR User-Defined Types (UDTs) to hold a maximum
of only 8KB of data, which seriously crippled many potential usage scenarios; all CLR UDTs may now
hold up to a maximum 2GB of data in a single item
...
Indeed, SQL Server 2008 introduces three new systemdefined datatypes (geometry, geography, and hierarchyid) that provide an excellent demonstration of the
ways in which SQLCLR can extend SQL Server to efficiently store and query types of data beyond the
standard numeric and character-based data typically associated with SQL databases
...
This chapter, however, concentrates on design and
performance considerations for exploiting user-defined functions based on managed code in SQL
Server, and discussion of when you should consider using SQLCLR over more traditional T-SQL
methods
...


Note This chapter assumes that you are already familiar with basic SQLCLR topics, including how to create and
deploy functions and catalog new assemblies, in addition to the C# programming language
...
NET Framework and by SQL Server are in many cases similar, but
generally incompatible
...
NET
interoperability from the perspective of data types:
•

First and foremost, all native SQL Server data types are nullable—that is, an
instance of any given type can either hold a valid value in the domain of the type
or represent an unknown (NULL)
...
NET generally do not support this idea
(note that C#’s null and VB
...

Even though the
...


•

The second difference between the type systems has to do with implementation
...
For example,
...


•

The third major difference has to do with runtime behavior of types in
conjunction with operators
...
However, this is not
the same behavior as that of an operation acting on a null value in
...
Consider
the following T-SQL:
DECLARE @a int =
DECLARE @b int =
IF (@a != @b)
PRINT 'test is
ELSE
PRINT 'test is

10;
null;
true';
false';

The result of any comparison to a NULL value in T-SQL is undefined, so the
preceding code will print “test is false
...
Write("test is true");
else
Console
...
NET, the comparison between 10 and null takes place, resulting in the code
printing “test is true
...
For
instance, adding 1 to a 32-bit integer with the value of 2147483647 (the maximum
32-bit integer value) in a
...
In SQL Server, this behavior will never occur—
instead, an overflow exception will result
...
NET Framework
ships with a namespace called System
...
SqlTypes
...
NET
...
Furthermore, these types conform to the same range,
precision, and operator rules as SQL Server’s native types
...
It is my
recommendation that, whenever possible, all methods exposed as SQLCLR objects use SqlTypes types as
both input and output parameters, rather than standard
...
This will require a bit more
development work up front, but it should future-proof your code to some degree and help avoid type
incompatibility issues
...
NET
Framework for application development, is the ability to move or share code easily between tiers when it
makes sense to do so
...


The Problem
Unfortunately, some of the design necessities of working in the SQLCLR environment do not translate
well to the application tier, and vice versa
...
NET types support
...

Rewriting code or creating multiple versions customized for different tiers simply does not promote
maintainability
...
This is one of the central design goals of object-oriented programming, and it’s important to
remember that it also applies to code being reused inside of SQL Server
...
These wrappers should map the SqlTypes inputs and outputs to the
...
Wrappers are
also a good place to implement database-specific logic that may not exist in routines originally designed
for the application tier
...
First of all, unit tests will not need to be rewritten—the same tests that work in the
application tier will still apply in the data tier (although you may want to write secondary unit tests for
the wrapper routines)
...
”

161

CHAPTER 7

SQLCLR: ARCHITECTURE AND DESIGN CONSIDERATIONS

A Simple Example: E-Mail Address Format Validation
It is quite common for web forms to ask for your e-mail address, and you’ve no doubt encountered
forms that tell you if you’ve entered an e-mail address that does not comply with the standard format
expected
...

In addition to using this logic for front-end validation, it makes sense to implement the same
approach in the database in order to drive a CHECK constraint
...

Following is a simple
...
]\w+)*@\w+([-
...
\w+([-
...
IsMatch(emailAddress));
}
This code could, of course, be used as-is in both SQL Server and the application tier—using it in SQL
Server would simply require loading the assembly and registering the function
...
As-is, this method will return an
ArgumentException when a NULL is passed in
...
Another potential issue occurs in methods that require slightly
different logic in the database vs
...
In the case of e-mail validation, it’s difficult to
imagine how you might enhance the logic for use in a different tier, but for other methods, such
modification would present a maintainability challenge
...
Instead, create a wrapper method that uses the SqlTypes and
internally calls the initial method
...
Following is a sample that shows a wrapper method created over the
IsValidEmailAddress method, in order to expose a SQLCLR UDF version that properly supports NULL
inputs and outputs
...

[Microsoft
...
Server
...
IsNull)
return (SqlBoolean
...
IsValidEmailAddress(emailAddress
...
Given the nature
of SQLCLR, the potential for code mobility should always be considered, and developers should consider
designing methods using wrappers even when creating code specifically for use in the database—this
will maximize the potential for reuse later, when or if the same logic needs to be migrated to another tier,
or even if the logic needs to be reused more than once inside of the data tier itself
...

By properly leveraging references, it is possible to create a much more robust, secure SQLCLR solution
...


SQLCLR Security and Reliability Features
Unlike stored procedures, triggers, UDFs, and other types of code modules that can be exposed within
SQL Server, a given SQLCLR routine is not directly related to a database, but rather to an assembly
cataloged within the database
...
The CREATE ASSEMBLY statement
allows the DBA or database developer to specify one of three security and reliability permission sets that
dictate what the code in the assembly is allowed to do
...
Each increasingly permissive
level includes and extends permissions granted by lower permission sets
...
The EXTERNAL_ACCESS permission set adds
the ability to communicate outside of the SQL Server instance, to other database servers, file servers,
web servers, and so on
...

Although exposed as only a single user-controllable setting, internally each permission set’s rights
are actually enforced by two distinct methods:
•

Assemblies assigned to each permission set are granted access to perform certain
operations via
...


•

At the same time, access is denied to certain operations based on checks against
a
...
5 attribute called HostProtectionAttribute (HPA)
...
The
combination of everything granted by CAS and everything denied by HPA makes up each of the three
permission sets
...
Although violation of a permission enforced by either method will result in a runtime
exception, the actual checks are done at very different times
...
On the other hand, HPA permissions are
checked at the point of just-in-time compilation—just before calling the method being referenced
...


163

CHAPTER 7

SQLCLR: ARCHITECTURE AND DESIGN CONSIDERATIONS

Tip You can download the source code of the examples in this chapter, together with all associated project files
and libraries, from the Source Code/Download area of the Apress web site, www
...
com
...
Create a new assembly containing the
following CLR stored procedure:
[SqlProcedure]
public static void CAS_Exception()
{
SqlContext
...
Send("Starting
...
txt", FileMode
...

}
SqlContext
...
Send("Finished
...
This will result in the following
output:
Starting
...
NET Framework error occurred during execution of user-defined routine or
aggregate "CAS_Exception":
System
...
SecurityException: Request for the permission of type
'System
...
Permissions
...
0
...
0,
Culture=neutral, PublicKeyToken=b77a5c561934e089' failed
...
Security
...
Security
...
Check(Object demand,

164

CHAPTER 7

SQLCLR: ARCHITECTURE AND DESIGN CONSIDERATIONS

StackCrawlMark& stackMark, Boolean isPermSet)
at System
...
CodeAccessPermission
...
IO
...
Init(String path, FileMode mode, FileAccess access, Int32
rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options,
SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy)
at System
...
FileStream
...
CAS_Exception()

...
But the exception is not the only thing that happened; notice that the first
line of the output is the string “Starting
...
Send method used in the
first line of the stored procedure
...


Note File I/O is a good example of access to a resource—local or otherwise—that is not allowed within the
context connection
...


Host Protection Exceptions
To see how HPA exceptions behave, let’s repeat the same experiment described in the previous section,
this time with the following stored procedure (again, cataloged as SAFE):
[SqlProcedure]
public static void HPA_Exception()
{
SqlContext
...
Send("Starting
...

Monitor
...
Pipe);
//Release the lock (if the code even gets here)
...
Exit(SqlContext
...
Pipe
...
");

165

CHAPTER 7

SQLCLR: ARCHITECTURE AND DESIGN CONSIDERATIONS

return;
}
Just like before, an exception occurs
...
NET Framework error occurred during execution of user-defined routine or
aggregate "HPA_Exception":
System
...
HostProtectionException: Attempted to perform an operation that
was forbidden by the CLR host
...
Security
...
Security
...
ThrowSecurityException(Assembly
asm,
PermissionSet granted, PermissionSet refused, RuntimeMethodHandle rmh,
SecurityAction action, Object demand, IPermission permThatFailed)
at System
...
CodeAccessSecurityEngine
...
Security
...
CheckSetHelper(PermissionSet
grants,
PermissionSet refused, PermissionSet demands, RuntimeMethodHandle rmh, Object
assemblyOrString, SecurityAction action, Boolean throwException)

166

CHAPTER 7

SQLCLR: ARCHITECTURE AND DESIGN CONSIDERATIONS

at System
...
CodeAccessSecurityEngine
...
HPA_Exception()

...
”
message, indicating that the SqlPipe
...
As a
matter of fact, the HPA_Exception method was not ever entered at all during the code execution phase
(you can verify this by attempting to set a breakpoint inside of the function and starting a debug session
in Visual Studio)
...

You should also note that the wording of the exception has a different tone than in the previous
case
...
failed
...
” This difference in wording is not accidental
...
HPA
permissions, on the other hand, are concerned with server reliability and keeping the CLR host running
smoothly and efficiently
...


Note Using a
...
red-gate
...
For instance, the Monitor class is decorated with the following attributes that control host access:
[ComVisible(true), HostProtection(SecurityAction
...


A full list of what is and is not allowed based on the CAS and HPA models is beyond the scope of this
chapter, but is well documented by Microsoft
...
microsoft
...
aspx)

•

CLR Integration Code Access Security (http://msdn2
...
com/enus/library/ms345101
...
The fact is,
raising the permission levels will certainly work, but by doing so you may be circumventing the security
policy, instead of working with it to make your system more secure
...
Therefore, raising the permission of a given assembly in order to
make a certain module work can actually affect many different modules contained within the assembly,
giving them all enhanced access
...

You might now be thinking that the solution is simple: split up your methods so that each resides in
a separate assembly, and then grant permissions that way
...
But even in that case, permissions may not be granular enough to avoid code review
nightmares
...
By giving the entire module EXTERNAL_ACCESS permissions, it can now read the lines
from that file
...

Then there is the question of the effectiveness of manual code review
...


Selective Privilege Escalation via Assembly References
In an ideal world, SQLCLR module permissions could be made to work like T-SQL module permissions
as described in Chapter 5: outer modules would be granted the least possible privileges, but would be
able to selectively and temporarily escalate their privileges in order to perform certain operations that
require more access
...

The general solution to this problem is to split up code into separate assemblies based on
permissions requirements, but not to do so without regard for both maintenance overhead and reuse
...
The entire module could be granted a sufficiently high level of privileges to read
the file, or the code to read the file could be taken out and placed into its own assembly
...
As I’ll
show in the following sections, this solution would let you catalog the bulk of the code as SAFE yet still do
the file I/O operation
...

The encapsulation story is, alas, not quite as straightforward as creating a new assembly with the
necessary logic and referencing it
...
In the following sections, I’ll cover each of the permission types separately in order to illustrate
how to design a solution
...

However, as with any shared data set, proper synchronization is essential in case you need to update
some of the data after its initial load
...

For an example of a scenario that might make use of a static collection, consider a SQLCLR UDF
used to calculate currency conversions based on exchange rates:
[SqlFunction]
public static SqlDecimal GetConvertedAmount(
SqlDecimal InputAmount,
SqlString InCurrency,
SqlString OutCurrency)
{
//Convert the input amount to the base
decimal BaseAmount =
GetRate(InCurrency
...
Value;
//Return the converted base amount
return (new SqlDecimal(
GetRate(OutCurrency
...
AcquireReaderLock(100);
try
{
theRate = rates[Currency];
}
finally
{
rwl
...
This collection contains exchange rates for the given currencies in the system
...
Both the dictionary and the
ReaderWriterLock are instantiated when a method on the class is first called, and both are marked
readonly in order to avoid being overwritten after instantiation:
static readonly Dictionary
rates = new Dictionary();
static readonly ReaderWriterLock
rwl = new ReaderWriterLock();
If cataloged using either the SAFE or EXTERNAL_ACCESS permission sets, this code fails due to its use of
synchronization (the ReaderWriterLock), and running it produces a HostProtectionException
...
Because the host
protection check is evaluated at the moment of just-in-time compilation of a method in an assembly,
rather than dynamically as the method is running, the check is done as the assembly boundary is being
crossed
...


Note You might be wondering about the validity of this example, given the ease with which this system could
be implemented in pure T-SQL, which would eliminate the permissions problem outright
...

SQLCLR code will generally outperform T-SQL for even simple mathematical work, and caching the data in a
shared collection rather than reading it from the database on every call is a huge efficiency win
...


When designing the UNSAFE assembly, it is important from a reuse point of view to carefully analyze
what functionality should be made available
...
However, a
wrapping method placed solely around a ReaderWriterLock would probably not promote very much
reuse
...
This class could be used in any scenario in which a shared
data cache is required
...
Collections
...
Text;
System
...
AcquireWriterLock(2000);
try
{
dict
...
ReleaseLock();
}
}
public V this[K key]
{
get
{
theLock
...
dict[key]);
}
finally
{
theLock
...
AcquireWriterLock(2000);
try
{
dict[key] = value;
}
finally
{
theLock
...
AcquireWriterLock(2000);
try
{
return (dict
...
ReleaseLock();
}
}
public bool ContainsKey(K key)
{
theLock
...
ContainsKey(key));
}
finally
{
theLock
...
A reference to the UNSAFE assembly should be used in the exchange rates
conversion assembly, after which a few lines of the previous example code will have to change
...
Without this requirement, its code becomes greatly simplified:
private static decimal GetRate(string Currency)
{
return (rates[Currency]);
}
The exchange rates conversion assembly can still be marked SAFE, and can now make use of the
encapsulated synchronization code without throwing a HostProtectionException
...


172

CHAPTER 7

SQLCLR: ARCHITECTURE AND DESIGN CONSIDERATIONS

Note Depending on whether your database has the TRUSTWORTHY option enabled and whether your assemblies
are strongly named, things may not be quite as simple as I’ve implied here
...
See the section “Granting Cross-Assembly
Privileges” later in this chapter for more information
...


Working with Code Access Security Privileges
HPA-protected resources are quite easy to encapsulate, thanks to the fact that permissions for a given
method are checked when the method is just-in-time compiled
...
This means that simply referencing a second assembly is not enough—the
entire stack is walked each time, without regard to assembly boundaries
...
IO
...
IO
...
ReadLine()) != null)
theLines
...
ToArray());
}
Catalog the assembly in SQL Server with the EXTERNAL_ACCESS permission set
...
Edit the CAS_Exception assembly to
include a reference to the assembly containing the ReadFileLines method, and modify the stored
procedure as follows:
[SqlProcedure]
public static void CAS_Exception()
{
SqlContext
...
Send("Starting
...
ReadFileLines(@"C:\b
...
Pipe
...
");
return;
}
Note that I created my ReadFileLines method inside a class called FileLines; reference yours
appropriately depending on what class name you used
...

Running the modified version of this stored procedure, you’ll find that even though an assembly
boundary is crossed, you will receive the same exception as before
...

Working around this issue requires taking control of the stack walk within the referenced assembly
...
This is done by using the Assert method of the IStackWalk interface,
exposed in
...
Security namespace
...
Security
...
The FileIOPermission
class—in addition to other “permission” classes in that namespace—implements the IStackWalk
interface
...
The following code is a modified version of the ReadFileLines method that uses
this technique:
public static string[] ReadFileLines(string FilePath)
{
//Assert that anything File IO-related that this
//assembly has permission to do, callers can do
FileIOPermission fp = new FileIOPermission(
PermissionState
...
Assert();
List theLines = new List();
using (System
...
StreamReader sr =
new System
...
StreamReader(FilePath))
{
string line;
while ((line = sr
...
Add(line);
}
return (theLines
...
Unrestricted enumeration, thereby enabling all callers to do whatever file I/O–related
activities the assembly has permission to do
...
After making the modifications shown here
and redeploying both assemblies, the CAS exception will no longer be an issue
...
The most useful of these for this example uses an
enumeration called FileIOPermissionAccess in conjunction with the path to a file, allowing you to limit
the permissions granted to the caller to only specific operations on a named file
...
Read,
"C:\b
...
The
important thing is being able to identify the pattern
...
Security
...

Each class follows the same basic pattern outlined here, so you should be able to easily use this
technique in order to design any number of privilege escalation solutions
...
There are two other issues you need to be concerned with when working with cross-assembly
calls: database trustworthiness and strong naming
...
Marking a database as trustworthy is a simple matter of setting an option using
ALTER DATABASE:
ALTER DATABASE AdventureWorks2008
SET TRUSTWORTHY ON;
GO
Unfortunately, as simple as enabling this option is, the repercussions of this setting are far from it
...
This means access to the file system, remote database servers, and even other databases on the
same server—all of this access is controlled by this one option, so be careful
...
That said, I highly recommend leaving the TRUSTWORTHY option turned
off unless you really have a great reason to enable it
...

In the SQLCLR world, you’ll see a deploy-time exception if you catalog an assembly that references
an assembly using the EXTERNAL_ACCESS or UNSAFE permission sets in a nontrustworthy database
...

The assembly is authorized when either of the following is true: the database
owner (DBO) has UNSAFE ASSEMBLY permission and the database has the TRUSTWORTHY
database property on; or the assembly is signed with a certificate or an asymmetric
key that has a corresponding login with UNSAFE ASSEMBLY permission
...
If not, use sp_changedbowner to fix
the problem
...
To begin, create a certificate and a corresponding login in the master
database, and grant the login UNSAFE ASSEMBLY permission:
USE master;
GO
CREATE CERTIFICATE Assembly_Permissions_Certificate
ENCRYPTION BY PASSWORD = 'uSe_a STr()nG PaSSW0rD!'
WITH SUBJECT = 'Certificate used to grant assembly permission';
GO
CREATE LOGIN Assembly_Permissions_Login
FROM CERTIFICATE Assembly_Permissions_Certificate;
GO
GRANT UNSAFE ASSEMBLY TO Assembly_Permissions_Login;
GO
Next, back up the certificate to a file:
BACKUP CERTIFICATE Assembly_Permissions_Certificate
TO FILE = 'C:\assembly_permissions
...
pvk',
ENCRYPTION BY PASSWORD = 'is?tHiS_a_VeRySTronGP4ssWoR|)?',
DECRYPTION BY PASSWORD = 'uSe_a STr()nG PaSSW0rD!'
);
GO
Now, in the database in which you’re working—AdventureWorks2008, in my case—restore the
certificate and create a local database user from it:
USE AdventureWorks2008;
GO

176

CHAPTER 7

SQLCLR: ARCHITECTURE AND DESIGN CONSIDERATIONS

CREATE CERTIFICATE Assembly_Permissions_Certificate
FROM FILE = 'C:\assembly_permissions
...
pvk',
DECRYPTION BY PASSWORD = 'is?tHiS_a_VeRySTronGP4ssWoR|)?',
ENCRYPTION BY PASSWORD = 'uSe_a STr()nG PaSSW0rD!'
);
GO
CREATE USER Assembly_Permissions_User
FOR CERTIFICATE Assembly_Permissions_Certificate;
GO
Finally, sign the assembly with the certificate, thereby granting access and allowing the assembly to
be referenced:
ADD SIGNATURE TO ASSEMBLY::SafeDictionary
BY CERTIFICATE Assembly_Permissions_Certificate
WITH PASSWORD='uSe_a STr()nG PaSSW0rD!';
GO

Strong Naming
The other issue you might encounter has to do with strongly named assemblies
...
NET
security feature that allows you to digitally sign your assembly, allocating a version number and ensuring
its validity to users
...
However, vendors looking at distributing applications that include SQLCLR components will
definitely want to look at strong naming
...
NET Framework error occurred during execution of user-defined routine or
aggregate "CAS_Exception":
System
...
SecurityException: That assembly does not allow partially trusted
callers
...
Security
...
Security
...
ThrowSecurityException(Assembly asm,

177

CHAPTER 7

SQLCLR: ARCHITECTURE AND DESIGN CONSIDERATIONS

PermissionSet granted, PermissionSet refused, RuntimeMethodHandle rmh,
SecurityAction action, Object demand, IPermission permThatFailed)
at udf_part2
...

The solution is to add an AllowPartiallyTrustedCallersAttribute (often referred to merely as
APTCA in articles) to the code
...
In the case of the FileLines
assembly, the file looks like the following after adding the attribute:
using
using
using
using
using
using
using

System;
System
...
Data
...
Data
...
SqlServer
...
Collections
...
Security
...
Security
...
Keep in mind that this attribute must be specified for a reason, and by using it
you may be allowing callers to circumvent security
...


Performance Comparison: SQLCLR vs
...

It is important to realize that SQLCLR is not, and was never intended to be, a replacement for T-SQL
as a data manipulation language in SQL Server
...
Also, although I’ve stressed the importance of
creating portable code that may be easily moved and shared between tiers, you should not try to use
SQLCLR as a way of moving logic that rightly belongs in the application into the database
...
For a further discussion of the correct placement of data and application logic, refer back to
Chapter 1
...
NET Base Class Library, and situations where procedural code is more efficient than setbased logic
...


Creating a “Simple Sieve” for Prime Numbers
For this test, I created two simple procedures that return a list of all prime numbers up to a supplied
maximum value—one implemented in T-SQL, and one using SQLCLR
...
If the remainder of the division is 0 (in other words, we have found a factor), we
know that the current value is not a prime number
...
If the inner loop tests every possible divisor and has not found any factors, then we know the
value must be a prime
...
Here’s the T-SQL implementation:
CREATE PROCEDURE ListPrimesTSQL (
@Limit int
)
AS BEGIN
DECLARE
-- @n is the number we're testing to see if it's a prime
@n int = @Limit,
--@m is all the possible numbers that could be a factor of @n
@m int = @Limit - 1;
-- Loop descending through the candidate primes
WHILE (@n > 1)
BEGIN
-- Loop descending through the candidate factors
WHILE (@m > 0)
BEGIN
-- We've got all the way to 2 and haven't found any factors
IF(@m = 1)
BEGIN
PRINT CAST(@n AS varchar(32)) + ' is a prime'
BREAK;
END
-- Is this @m a factor of this prime?
IF(@n%@m) <> 0
BEGIN
-- Not a factor, so move on to the next @m
SET @m = @m - 1;
CONTINUE;
END

179

CHAPTER 7

SQLCLR: ARCHITECTURE AND DESIGN CONSIDERATIONS

ELSE BREAK;
END
SET @n = @n-1;
SET @m = @n-1;
END
END;
GO
And here’s the SQLCLR implementation using exactly the same logic:
[SqlProcedure]
public static void ListPrimesCLR(SqlInt32 Limit)
{
int n = (int)Limit;
int m = (int)Limit - 1;
while(n > 1)
{
while(m > 0)
{
if(m == 1)
{
SqlContext
...
Send(n
...
The example used here is intended to provide a simple procedure that can be implemented
consistently across both T-SQL and SQLCLR
...
The average execution time for each solution is shown in the graph illustrated in Figure
7-1
...
Comparison of prime number sieve implemented in T-SQL and SQLCLR
The results should come as no surprise—since the approach taken relies on mathematical
operations in an iterative loop, SQLCLR is always likely to outperform set-based T-SQL
...
If we were to compare simple inline, or nonprocedural, calculations then
there would likely not be such a stark contrast between the two methods
...
The most common example of such a linear
query is in the calculation of aggregates, such as running sums of columns
...
x,
SUM(T2
...
x >= T2
...
x;

181

CHAPTER 7

SQLCLR: ARCHITECTURE AND DESIGN CONSIDERATIONS

Unfortunately, the process required to satisfy this query is not very efficient
...


Figure 7-2
...
The
number of rows returned by this seek increases exponentially as more rows are processed
...
For a set of 200 rows, the query processor needs to process 20,100 total rows—four
times the amount of work required to satisfy the previous query
...

An alternative solution, which can yield some significant performance benefits, is to make use of a
cursor
...
However, there are a number
of good reasons why many developers are reluctant to use cursors, and I’m certainly not advocating their
use in general
...
An example of such a solution is
given in the following code listing:
[Microsoft
...
Server
...
Connection = conn;
comm
...
Int);
columns[1] = new SqlMetaData("RunningSum", SqlDbType
...
Pipe
...
Open();
SqlDataReader reader = comm
...
Read())
{
int Value = (int)reader[0];
RunningSum += Value;
record
...
SetInt32(1, RunningSum);
SqlContext
...
SendResultsRow(record);
}
SqlContext
...
SendResultsEnd();
}
}
I’ve used this solution on a number of occasions and find it to be very efficient and maintainable,
and it avoids the need for any temp tables to be used to hold the running sums as required by the
alternatives
...
7 seconds for the SQLCLR query, compared to over 5 minutes for the TSQL equivalent
...
The problem is that there are lots of ingenious techniques for working
with string data: in T-SQL, some of the best performing methods use one or more common table
expressions (CTEs), CROSS APPLY operators, or number tables; or convert text strings to XML in or order
to perform nontrivial manipulation of character data
...

I decided that, rather than try to define a scenario that required a string-handling technique, the
only fair test was to perform a direct comparison of two built-in methods that provided the equivalent
functionality in either environment
...
NET’s
String
...
For the purposes of the test, I created nvarchar(max) strings of different lengths, each composed
entirely of the repeating character a
...


183

CHAPTER 7

SQLCLR: ARCHITECTURE AND DESIGN CONSIDERATIONS

The following code listing demonstrates the T-SQL method:
CREATE PROCEDURE SearchCharTSQL
(
@needle nchar(1),
@haystack nvarchar(max)
)
AS BEGIN
PRINT CHARINDEX(@needle, @haystack);
END;
And here’s the() CLR equivalent:
[SqlProcedure]
public static void SearchCharCLR(SqlString needle, SqlString haystack)
{
SqlContext
...
Send(
haystack
...
IndexOf(needle
...
ToString()
);
}
Note that the starting position for CHARINDEX is 1-based, whereas the index numbering used by
IndexOf() is 0-based
...
I tested each procedure as follows, substituting different
parameter values for the REPLICATE method to change the length of the string to be searched:
DECLARE @needle nvarchar(1) = 'x';
DECLARE @haystack nvarchar(max);
SELECT @haystack = REPLICATE(CAST('a' AS varchar(max)), 8000) + 'x';
EXEC dbo
...


Figure 7-3
...
()IndexOf()

184

CHAPTER 7

SQLCLR: ARCHITECTURE AND DESIGN CONSIDERATIONS

As with the prime number sieve example given earlier, the logic required for string searching,
matching, and replacing is best suited to the highly efficient routines provided by the
...
If you currently have code logic that relies heavily on T-SQL string functionality including
CHARINDEX, PATINDEX, or REPLACE, I highly recommend that you investigate the alternative options
available through SQLCLR—you might be surprised by the performance gain you achieve
...
Service Broker is frequently mentioned as an excellent choice for helping to scale out
database services
...
In such a case, a request message would be sent to
the remote data service from a local stored procedure, which could do some other work while waiting for
the response—the requested data—to come back
...
In the following sections, I’ll guide you through my investigations
into XML and binary serialization using SQLCLR
...
Employee table from the AdventureWorks2008 database as a
sample data set, imagining a remote data service requesting a list of employees along with their
attributes
...
Employee
FOR XML RAW, ROOT('Employees')
);
GO
XML is, of course, known to be an extremely verbose data interchange format, and I was not
surprised to discover that the data size of the resultant XML is 105KB, despite the fact that the
HumanResources
...
I experimented with setting shorter column
names, but it had very little effect on the size and created what I feel to be unmaintainable code
...
The trace results revealed that the average execution time for
the preceding query on my system, averaged over 1,000 iterations, was a decidedly unimpressive 3
...

After some trial and error, I discovered that XML serialization could be made to perform better by
using the TYPE directive, as follows:
DECLARE @x xml;
SET @x = (

185

CHAPTER 7

SQLCLR: ARCHITECTURE AND DESIGN CONSIDERATIONS

SELECT *
FROM HumanResources
...
6687 seconds—an
improvement, but still not a very good result
...
The first problem was the code required to deserialize the XML back into a table
...
Furthermore, since the XQuery value
syntax does not support the hierarchyid datatype, the values in the OrganizationNode column must be
read as nvarchar and then CAST to hierarchyid
...
Employee
FOR XML RAW, ROOT('Employees'), TYPE
);
SELECT
col
...
value('@NationalIDNumber', 'nvarchar(15)') AS NationalIDNumber,
col
...
value('@OrganizationNode', 'nvarchar(256)') AS hierarchyid)
AS OrganizationNode,
col
...
value('@BirthDate', 'datetime') AS BirthDate,
col
...
value('@Gender', 'nchar(1)') AS Gender,
col
...
value('@SalariedFlag', 'bit') AS SalariedFlag,
col
...
value('@SickLeaveHours', 'smallint') AS SickLeaveHours,
col
...
value('@rowguid', 'uniqueidentifier') AS rowguid,
col
...
nodes ('/Employees/row') x (col);
GO
The next problem was performance
...
8157 seconds per iteration
...


186

CHAPTER 7

SQLCLR: ARCHITECTURE AND DESIGN CONSIDERATIONS

Binary Serialization with SQLCLR
My first thought was to return binary serialized DataTables; in order to make that happen, I needed a
way to return binary-formatted data from my CLR routines
...
NET’s
BinaryFormatter class, so I created a class called serialization_helper, cataloged in an EXTERNAL_ACCESS
assembly (required for System
...
Data;
System
...
SqlClient;
System
...
SqlTypes;
Microsoft
...
Server;
System
...
Permissions;
System
...
Serialization
...
Binary;

public partial class serialization_helper
{
public static byte[] getBytes(object o)
{
SecurityPermission sp =
new SecurityPermission(
SecurityPermissionFlag
...
Assert();
BinaryFormatter bf = new BinaryFormatter();
using (System
...
MemoryStream ms =
new System
...
MemoryStream())
{
bf
...
ToArray());
}
}
public static object getObject(byte[] theBytes)
{
using (System
...
MemoryStream ms =
new System
...
MemoryStream(theBytes, false))
{
return(getObject(ms));
}
}
public static object getObject(System
...
Stream s)
{
SecurityPermission sp =
new SecurityPermission(
SecurityPermissionFlag
...
Assert();

187

CHAPTER 7

SQLCLR: ARCHITECTURE AND DESIGN CONSIDERATIONS

BinaryFormatter bf = new BinaryFormatter();
return (bf
...
This
method first uses an assertion, as discussed previously, to allow SAFE callers to use it, and then uses the
binary formatter to serialize the object to a Stream
...

Deserialization can be done using either overload of the getObject method
...
Deserialization also
uses an assertion before running, in order to allow calling code to be cataloged as SAFE
...
The following code implements a UDF called GetDataTable_Binary,
which uses this logic:
[Microsoft
...
Server
...
Read)]
public static SqlBytes GetDataTable_Binary(string query)
{
SqlConnection conn =
new SqlConnection("context connection = true;");
SqlCommand comm = new SqlCommand();
comm
...
CommandText = query;
SqlDataAdapter da = new SqlDataAdapter();
da
...
Fill(dt);
//Serialize and return the output
return new SqlBytes(
serialization_helper
...
ToString(),
OrganizationLevel,
JobTitle,
BirthDate,

188

CHAPTER 7

SQLCLR: ARCHITECTURE AND DESIGN CONSIDERATIONS

MaritalStatus,
Gender,
HireDate,
SalariedFlag,
VacationHours,
SickLeaveHours,
CurrentFlag,
rowguid,
ModifiedDate
FROM HumanResources
...
GetDataTable_Binary(@sql);
GO

Note The hierarchyid CLR datatype is not marked as serializable, so in the preceding query I use the
ToString() method to serialize the string representation of the OrganizationNode value
...
1437 seconds—a massive improvement over the XML
serialization method
...
RemotingFormat = SerializationFormat
...
0576 seconds
...

Encouraged by the success of my first shot, I decided to investigate whether there were other
SQLCLR methods that would improve the performance still further
...
I worked on pulling the data out into object collections, and initial tests showed
serialization performance with SqlDataReader to be just as good as the DataTable, but with a reduced
output size
...

The advantage of a DataTable is that it’s one easy-to-use unit that contains all of the data, as well as
the associated metadata
...
Working with a SqlDataReader requires a
bit more work, since it can’t be serialized as a single unit, but must instead be split up into its
component parts
...
To
begin with, I set the DataAccessKind
...
A generic List is instantiated, which will hold one
object collection per row of data, in addition to one for the metadata
...
SqlServer
...
SqlFunction(
DataAccess = DataAccessKind
...
Connection = conn;
comm
...
Open();
SqlDataReader read = comm
...
A method called
GetSchemaTable is used to return a DataTable populated with one row per column
...

After populating the object collection with the metadata, it is added to the output List:
DataTable dt = read
...
Rows
...
Length; i++)
{
object[] field = new object[5];
field[0] = dt
...
Rows[i]["ProviderType"];
field[2] = dt
...
Rows[i]["NumericPrecision"];
field[4] = dt
...
Add(fields);
Finally, the code loops over the rows returned by the query, using the GetValues method to pull each
row out into an object collection that is added to the output
...

//Add all of the rows to the output list
while (read
...
FieldCount];
read
...
Add(o);
}
}

190

CHAPTER 7

SQLCLR: ARCHITECTURE AND DESIGN CONSIDERATIONS

//Serialize and return the output
return new SqlBytes(
serialization_helper
...
ToArray()));
}
Once this function is created, calling it is almost identical to calling GetDataTable_Binary:
DECLARE @sql nvarchar(max);
SET @sql = 'SELECT BusinessEntityID,
NationalIDNumber,
LoginID,
OrganizationNode
...
Employee'
DECLARE @x varbinary(max);
SET @x = dbo
...
If using this method to transfer data between broker instances on remote servers, the
associated decrease in network traffic could make a big difference to performance
...
0490 seconds
...

Continuing with my stress on reuse potential, I decided that a stored procedure would be a better choice
than a UDF
...

The first part of the stored procedure follows:
[Microsoft
...
Server
...
getObject(theTable
...
Length];
//Loop over the fields and populate SqlMetaData objects
for (int i = 0; i ...
This collection is
looped over item by item in order to create the output SqlMetaData objects that will be used to stream
back the data to the caller
...
decimal needs a precision and scale setting; character and binary types need a
size; and for other types, size, precision, and scale are all inappropriate inputs
...
Decimal:
cols[i] = new SqlMetaData(
(string)field[0],
dbType,
(byte)field[3],
(byte)field[4]);
break;
case SqlDbType
...
Char:
case SqlDbType
...
NVarChar:
case SqlDbType
...
VarChar:
switch ((int)field[2])
{
//If it's a MAX type, use -1 as the size
case 2147483647:
cols[i] = new SqlMetaData(
(string)field[0],
dbType,
-1);
break;
default:
cols[i] = new SqlMetaData(
(string)field[0],
dbType,
(long)((int)field[2]));
break;

192

CHAPTER 7

SQLCLR: ARCHITECTURE AND DESIGN CONSIDERATIONS

}
break;
default:
cols[i] = new SqlMetaData(
(string)field[0],
dbType);
break;
}
}
Once population of the columns collection has been completed, the data can be sent back to the
caller using the SqlPipe class’s SendResults methods
...
Pipe
...
Length; i++)
{
rec
...
Pipe
...
Pipe
...
The performance test
revealed that average time for deserialization of the SqlDataReader data was just 0
...

The results of the fastest refinements of each of the three methods discussed in this section are
shown in Table 7-1
...
Results of different serialization approaches

Method

Average Serialization Time

Average Deserialization Time

XML (with TYPE)

3
...
8157

Binary (DataTable)

0
...
0490

Size
105KB
68KB

0
...


193

CHAPTER 7

SQLCLR: ARCHITECTURE AND DESIGN CONSIDERATIONS

Summary
Getting the most out of SQLCLR routines involves a bit of thought investment
...

You should also consider reuse at every stage, in order to minimize the amount of work that must be
done when you need the same functionality six months or a year down the road
...
I used a common, core set of more highly
privileged utility assemblies in order to limit the outer surface area, and tried to design the solution to
promote flexibility and potential for use in many projects throughout the lifetime of the code
...
The first step in meeting this objective is
therefore to keep the application bug-free and working as designed, to expected standards
...
It’s common for software
customer support teams to receive requests for slightly different sort orders, filtering mechanisms, or
outputs for data, making it imperative that applications be designed to support extensibility along these
lines
...
This is especially true in web-based application development, where client-side grid controls
that enable sorting and filtering are still relatively rare, and where many applications still use a
lightweight two-tier model without a dedicated business layer to handle data caching and filtering
...
These solutions invariably seem to create
more problems than they solve, and make application development much more difficult than it needs to
be by introducing a lot of additional complexity in the database layer
...
Some DBAs and developers scorn dynamic SQL, often believing
that it will cause performance, security, or maintainability problems, whereas in many cases it is simply
that they don’t understand how to use it properly
...
There is a lot of misinformation floating
around about what it is and when or why it should be used, and I hope to clear up some myths and
misconceptions in these pages
...
For more information on how to capture these measures on your own system
environment, please refer to the discussion of performance monitoring tools in Chapter 3
...
Ad Hoc T-SQL
Before I begin a serious discussion about how dynamic SQL should be used, it’s first important to
establish a bit of terminology
...
When referring to these terms in this chapter, I define them as follows:
•

Ad hoc SQL is any batch of SQL generated within an application layer and sent to
SQL Server for execution
...


•

Dynamic SQL, on the other hand, is a batch of SQL that is generated within T-SQL
and executed using the EXECUTE statement or, preferably, via the sp_executesql
system stored procedure (which is covered later in this chapter)
...

However, if you are one of those working with systems that do not use stored procedures, I advise you to
still read the “SQL Injection” and “Compilation and Parameterization” sections at a minimum
...

All of that said, I do not recommend the use of ad hoc SQL in application development, and feel that
many potential issues, particularly those affecting application security and performance, can be
prevented through the use of stored procedures
...
Ad Hoc SQL Debate
A seemingly never-ending battle among members of the database development community concerns
the question of whether database application development should involve the use of stored procedures
...
I highly recommend that you search the Web to find these debates
and reach your own conclusions
...

First and foremost, stored procedures create an abstraction layer between the database and the
application, hiding details about the schema and sometimes the data
...
Reducing these dependencies and thinking of the database as a data API rather than a
simple application persistence layer enables a flexible application development process
...
For more information on concepts
such as encapsulation, coupling, and treating the database as an API, see Chapter 1
...

Furthermore, support for more advanced testing methodologies also becomes easier, not more difficult,
thanks to stored procedures
...
Mock objects can be substituted for real methods in testing scenarios so that any
given method can be tested in isolation, without also testing any methods that it calls (any calls made
from within the method being tested will actually be a call to a mock version of the method)
...

Another important issue is security
...
This means that by using ad hoc SQL,
your application may be more vulnerable to being hacked, and you may not be able to rely on SQL
Server to secure access to data
...
See the section “Dynamic SQL Security Considerations” for
further discussion of these points
...
Proponents of ad hoc SQL make the valid claim that, thanks to
better support for query plan caching in recent versions of SQL Server, stored procedures no longer have
a significant performance benefit when compared to ad hoc queries
...
Given
equivalent performance, I think the obvious choice is the more maintainable and secure option (i
...
,
stored procedures)
...
ad hoc SQL question is really one of purpose
...

In my eyes, a database is much more than just a collection of data
...

For these reasons, I believe that a decoupled, stored procedure–based design is the best way to go
...
This
is a righteous goal, but the fact is that dynamic SQL is just one means by which to attain the desired end
result
...
It is also possible not to go dynamic at all, by supporting static
stored procedures that supply optional parameters—but that’s not generally recommended because it
can quickly lead to very unwieldy code that is difficult to maintain, as will be demonstrated in the
“Optional Parameters via Static T-SQL” section later in this chapter
...
Keep in mind the questions of performance, maintainability, and most important, scalability
...
Remember that scaling
the database can often be much more expensive than scaling other layers of an application
...
In order for the database to sort data, the data
must be queried
...
Every time the data needs to be
resorted a different way, it must be reread or sorted in memory and refiltered by the database engine
...

Due to this issue, if the same data is resorted again and again (for instance, by a user who wants to
see various high or low data points), it often makes sense to do the work in a disconnected cache
...
This
can take a tremendous amount of strain off the database server, meaning that it can use its resources for
other data-intensive operations
...
I offer some suggestions in the section “Going Dynamic: Using
EXECUTE,” but keep in mind that procedural code may be easier to work with for these purposes than
T-SQL
...
In the database layer, the question of using dynamic SQL instead of static SQL
comes down to issues of both maintainability and performance
...
For the
best balance of maintenance vs
...


Compilation and Parameterization
Any discussion of dynamic SQL and performance would not be complete without some basic
background information concerning how SQL Server processes queries and caches their plans
...

Every query executed by SQL Server goes through a compilation phase before actually being
executed by the query processor
...
However, query compilation can be expensive for certain queries, and when the same queries
or types of queries are executed over and over, there is generally no reason to compile them each time
...

The query plan cache uses a simple hash lookup based on the exact text of the query in order to find
a previously compiled plan
...

If a compiled version of the query is not found, the first step taken is parsing of the query
...
The
parse tree is further validated and eventually compiled into a query plan, which is placed into the query
plan cache for future invocations of the query
...
To
demonstrate this, first use the DBCC FREEPROCCACHE command to empty out the cache:
DBCC FREEPROCCACHE;
GO
Keep in mind that this command clears out the cache for the entire instance of SQL Server—doing
this is not generally recommended in production environments
...
Employee table from the
AdventureWorks2008 database:

Note As of SQL Server 2008, SQL Server no longer ships with any included sample databases
...
codeplex
...


SELECT *
FROM HumanResources
...


(2 row(s) affected)

SQL Server Execution Times:
CPU time = 0 ms,

elapsed time = 1 ms
...
But subsequent runs produce the following output,
indicating that the cached plan is being used:

199

CHAPTER 8

DYNAMIC T-SQL

SQL Server parse and compile time:
CPU time = 0 ms, elapsed time = 1 ms
...


Thanks to the cached plan, each subsequent invocation of the query takes 11ms less than the first
invocation—not bad, when you consider that the actual execution time is less than 1ms (the lowest
elapsed time reported by time statistics)
...
If SQL Server determines
that one or more literals used in the query are parameters that may be changed for future invocations of
a similar version of the query, it can auto-parameterize the query
...
dm_exec_cached_plans dynamic
management view and the sys
...
The following query finds all cached queries
that contain the string “HumanResources,” excluding those that contain the name of the
sys
...

SELECT
cp
...
text
FROM sys
...
dm_exec_sql_text(cp
...
text LIKE '%HumanResources%'
AND st
...
dm_exec_cached_plans%';
GO

Note I’ll be reusing this code several times in this section to examine the plan cache for different types of
query, so you might want to keep it open in a separate Management Studio tab
...
Employee gives
the following results:
objtype
Adhoc

text
SELECT * FROM HumanResources
...
Queries that
cannot be auto-parameterized are classified by the query engine as “ad hoc” (note that this is a slightly
different definition from the one I use)
...
The following query, on the other hand, can be auto-parameterized:
SELECT *
FROM HumanResources
...
dm_exec_cached_plans as before results in the output shown following:
objtype

text

Adhoc

SELECT *

Prepared

(@1 tinyint)SELECT * FROM [HumanResources]
...
Employee

WHERE BusinessEntityId = 1;

WHERE [BusinessEntityId]=@1
In this case, two plans have been generated: an Adhoc plan for the query’s exact text and a Prepared
plan for the auto-parameterized version of the query
...

The benefit of this auto-parameterization is that subsequent queries submitted to SQL Server that
can be auto-parameterized to the same normalized form may be able to make use of the prepared query
plan, thereby avoiding compilation overhead
...
SQL Server 2008 includes a more
powerful form of auto-parameterization, called “forced parameterization
...
This can
be very beneficial to applications that use a lot of nonparameterized ad hoc queries, but may cause performance
degradation in other cases
...
microsoft
...
aspx for more
information on forced parameterization
...
Other forms of
parameterization are possible at the application level for ad hoc SQL, or within T-SQL when working
with dynamic SQL in a stored procedure
...

Every query framework that can communicate with SQL Server supports the idea of remote
procedure call (RPC) invocation of queries
...

Parameterizing queries in this way has one key advantage from a performance standpoint: the
application tells SQL Server what the parameters are; SQL Server does not need to (and will not) try to
find them itself
...
NET, by populating the Parameters collection on
the SqlCommand object when preparing a query
...
Open();
SqlCommand cmd = new SqlCommand(
"SELECT * FROM HumanResources
...
Int);
param
...
Parameters
...
Int);
param2
...
Parameters
...
ExecuteNonQuery();
sqlConn
...


Notice that the underlying query is the same as the first query shown in this chapter, which, when
issued as a T-SQL query via Management Studio, was unable to be auto-parameterized by SQL Server
...

Executing this code listing and then examining the sys
...
Employee
WHERE BusinessEntityId IN (@Emp1, @Emp2)

Just like with auto-parameterized queries, the plan is prepared and the text is prefixed with the
parameters
...
The object name is not
bracket-delimited, and although it may not be apparent, whitespace has not been removed
...
Otherwise, you will end up wasting
both the CPU cycles required for needless compilation and memory for caching the additional plans
...
The
cache lookup mechanism is nothing more than a simple hash on the query text and is case sensitive
...
It’s always a good idea when working with SQL Server to try to be consistent with
your use of capitalization and formatting
...

Suppose, for example, that we placed the previous application code in a loop—calling the same
query 2,000 times and changing only the supplied parameter values on each iteration:

203

CHAPTER 8

DYNAMIC T-SQL

SqlConnection sqlConn = new SqlConnection(
"Data Source=localhost;
Initial Catalog=AdventureWorks2008;
Integrated Security=SSPI");
sqlConn
...
Employee
WHERE BusinessEntityId IN (@Emp1, @Emp2)",
sqlConn);
SqlParameter param = new SqlParameter("@Emp1", SqlDbType
...
Value = i;
cmd
...
Add(param);
SqlParameter param2 = new SqlParameter("@Emp2", SqlDbType
...
Value = i + 1;
cmd
...
Add(param2);
cmd
...
Close();
Once again, return to SQL Server Management Studio and query the sys
...
There is only one plan in the cache for this form
of the query, even though it has just been run 2,000 times with different parameter values:
objtype

text

Prepared

(@Emp1 int,@Emp2 int)SELECT * FROM HumanResources
...

Now that a positive baseline has been established, let’s investigate what happens when queries are
not properly parameterized
...
Open();
for (int i = 1; i < 2000; i++)

204

CHAPTER 8

DYNAMIC T-SQL

{
SqlCommand cmd = new SqlCommand(
"SELECT * FROM HumanResources
...
ExecuteNonQuery();
}
sqlConn
...
Employee WHERE BusinessEntityId IN (1, 2)

Adhoc

SELECT * FROM HumanResources
...
Employee WHERE BusinessEntityId IN (3, 4)


...


Adhoc

SELECT * FROM HumanResources
...


Adhoc

SELECT * FROM HumanResources
...


Running 2,000 nonparameterized ad hoc queries with different parameters resulted in 2,000
additional cached plans
...
In SQL Server 2008, queries are aged out of the plan cache on a least-recently-used basis, and
depending on the server’s workload, it can take quite a bit of time for unused plans to be removed
...
This is obviously not a good thing!
So please—for the sake of all of that RAM—learn to use your connection library’s parameterized query
functionality and avoid falling into this trap
...
Although it is quite easy

205

CHAPTER 8

DYNAMIC T-SQL

to write static stored procedures that handle optional query parameters, these are generally grossly
inefficient or highly unmaintainable—as a developer, you can take your pick
...
There are a few different
methods of creating static queries that support optional parameters, with varying complexity and
effectiveness, but each of these solutions contains flaws
...
Employee table in the AdventureWorks2008 database:
SELECT
BusinessEntityID,
LoginID,
JobTitle
FROM
HumanResources
...

Executing the query produces the execution plan shown in Figure 8-1, which has an estimated cost of
0
...
This plan involves a seek of the table’s clustered index,
which uses the BusinessEntityID column as its key
...
Base execution plan with seek on BusinessEntityID clustered index
Since the query uses the clustered index, it does not need to do a lookup to get any additional data
...
Therefore, the following query, which uses only the
BusinessEntityId predicate, produces the exact same query plan with the same cost and same number
of reads:
SELECT
BusinessEntityID,
LoginID,
JobTitle
FROM
HumanResources
...
Employee
WHERE
NationalIDNumber = N'14417807';
GO
This query results in a very different plan from the other two, due to the fact that a different index
must be used to satisfy the query
...
This plan has an estimated cost of 0
...


Figure 8-2
...
Employee;
GO
As shown in Figure 8-3, the query plan in this case is a simple clustered index scan, with an
estimated cost of 0
...
Since all of the rows need to be returned and no
index covers every column required, a clustered index scan is the most efficient way to satisfy this query
...
Base execution plan with scan on the clustered index
These baseline figures will be used to compare the relative performance of various methods of
creating a dynamic stored procedure that returns the same columns, but that optionally filters the rows
returned based on one or both predicates of BusinessEntityID and NationalIDNumber
...
Employee
WHERE
BusinessEntityID = @BusinessEntityID
AND NationalIDNumber = @NationalIDNumber;
END;
GO
This stored procedure uses the parameters @BusinessEntityID and @NationalIDNumber to support
the predicates
...
However, this stored
procedure does not really support the parameters optionally; not passing one of the parameters will
mean that no rows will be returned by the stored procedure at all, since any comparison with NULL in a
predicate will not result in a true answer
...
Employee
WHERE
BusinessEntityID = @BusinessEntityID
AND NationalIDNumber = @NationalIDNumber;
END
ELSE IF (@BusinessEntityID IS NOT NULL)
BEGIN
SELECT
BusinessEntityID,
LoginID,
JobTitle
FROM
HumanResources
...
Employee
WHERE
NationalIDNumber = @NationalIDNumber;
END
ELSE
BEGIN
SELECT
BusinessEntityID,
LoginID,
JobTitle
FROM
HumanResources
...
Namely, taking this approach turns what was a very simple 10-line stored
procedure into a 42-line monster
...
Now consider what would happen if a third predicate were needed—the number of cases
would jump from four to eight, meaning that any change such as adding or removing a column would
have to be made in eight places
...
It is simply not a manageable solution
...
As a result, a lot of code has been written against it by developers who don’t
seem to realize that they’re creating a performance time bomb
...
Employee
WHERE
BusinessEntityID = COALESCE(@BusinessEntityID, BusinessEntityID)
AND NationalIDNumber = COALESCE(@NationalIDNumber, NationalIDNumber);
END;
GO
This version of the stored procedure looks great and is easy to understand
...
So if either of the arguments to the stored
procedure are NULL, the COALESCE will “pass through,” comparing the value of the column to itself—and
at least in theory, that seems like it should not require any processing since it will always be true
...
The result is that the function is evaluated
once for every row of the table, whatever combination of parameters is supplied
...
0080454 and nine logical reads
...

Similar to the version that uses COALESCE is a version that uses OR to conditionally set the parameter
only if the argument is not NULL:
ALTER PROCEDURE GetEmployeeData
@BusinessEntityID int = NULL,
@NationalIDNumber nvarchar(15) = NULL
AS
BEGIN
SET NOCOUNT ON;

210

CHAPTER 8

DYNAMIC T-SQL

SELECT
BusinessEntityID,
LoginID,
JobTitle
FROM
HumanResources
...
Depending on which parameters you use the first time you call it, you’ll see vastly
different results
...
If, however, you first call the stored procedure using only the
@BusinessEntityID parameter, the resultant plan will use only four logical reads—until you happen to
call the procedure with no arguments, which will produce a massive 582 reads
...

The final method that can be used is a bit more creative, and also can result in somewhat better
results
...
Employee
WHERE
BusinessEntityID BETWEEN COALESCE(@BusinessEntityID, -2147483648) AND
COALESCE(@BusinessEntityID, 2147483647)
AND NationalIDNumber LIKE COALESCE(@NationalIDNumber, N'%');
END;
GO
If you’re a bit confused by the logic of this stored procedure, you’re now familiar with the first
reason that I don’t recommend this technique: it’s relatively unmaintainable if you don’t understand
exactly how it works
...
And while that might be good for job
security, using it for that purpose is probably not a virtuous goal
...
This approach works as follows:
If @BusinessEntityID is NULL, the BusinessEntityID predicate effectively becomes
BusinessEntityID BETWEEN -2147483648 AND 2147483647—in other words, all
possible integers
...
This is
equivalent to BusinessEntityID=@BusinessEntityID
...
If
@NationalIDNumber is NULL, the predicate becomes NationalIDNumber LIKE N'%'
...
On the other hand, if
@NationalIDNumber is not NULL, the predicate becomes NationalIDNumber LIKE
@NationalIDNumber, which is equivalent to NationalIDNumber=@NationalIDNumber—
assuming that @NationalIDNumber contains no string expressions
...
However, that method is both
more difficult to read than the LIKE expression and fraught with potential problems
due to collation issues (which is why I only went up to nchar(1000) instead of
nchar(65535) in the example)
...
Unfortunately, this stored procedure manages
to confuse the query optimizer, resulting in the same plan being generated for every invocation
...
0033107, as
shown in Figure 8-4
...


Figure 8-4
...

If both arguments are passed, or @BusinessEntityID is passed but @NationalIDNumber is not, the
number of logical reads is three
...
This estimated plan really breaks down when passing
only @NationalIDNumber, since there is no way to efficiently satisfy a query on the NationalIDNumber
column using the clustered index
...
For the NationalIDNumber predicate this is quite a failure, as the stored procedure does over
twice as much work for the same results as the baseline
...
Building
dynamic SQL inside of a stored procedure is simple, the code is relatively easy to understand and, as I’ll

212

CHAPTER 8

DYNAMIC T-SQL

show, it can provide excellent performance
...
I’ll explain how to deal with these as the examples progress
...
The main
issue with the static SQL solutions, aside from maintainability, was that the additional predicates
confused the query optimizer, causing it to create inefficient plans
...

The simplest way to implement dynamic SQL in a stored procedure is with the EXECUTE statement
...
The following batch
shows this in its simplest—and least effective—form:
EXEC('SELECT
BusinessEntityID,
LoginID,
JobTitle
FROM HumanResources
...
This seems to be a de facto standard for SQL Server code; I very rarely see code that uses the full
form with the added “UTE
...

In this case, a string literal is passed to EXECUTE, and this doesn’t really allow for anything very
“dynamic
...
Employee
WHERE BusinessEntityID = ' + CONVERT(varchar, @BusinessEntityID));
GO
This fails (with an “incorrect syntax” exception) because of the way EXECUTE is parsed by the SQL
Server engine
...
But due to the fact that the first step does not include a stage for inline
expansion, the CONVERT is still a CONVERT, rather than a literal, when it’s time for concatenation
...
Define a variable and assign the dynamic SQL to it, and
then call EXECUTE:
DECLARE @BusinessEntityID int = 28;
DECLARE @sql nvarchar(max);
SET @sql = 'SELECT
BusinessEntityID,
LoginID,

213

CHAPTER 8

DYNAMIC T-SQL

JobTitle
FROM HumanResources
...
In
other words, forming the dynamic SQL is now limited only by the tools available within the T-SQL
language for string manipulation
...
Employee ';
IF (@BusinessEntityID IS NOT NULL AND @NationalIDNumber IS NOT NULL)
BEGIN
SET @sql = @sql +
'WHERE BusinessEntityID = ' + CONVERT(nvarchar, @BusinessEntityID) +
' AND NationalIDNumber = N''' + @NationalIDNumber + '''';
END
ELSE IF (@BusinessEntityID IS NOT NULL)
BEGIN
SET @sql = @sql +
'WHERE BusinessEntityID = ' +
CONVERT(nvarchar, @BusinessEntityID);
END
ELSE IF (@NationalIDNumber IS NOT NULL)
BEGIN
SET @sql = @sql +
'WHERE NationalIDNumber = N''' + @NationalIDNumber + '''';
END
EXEC(@sql);
GO
If this looks sickeningly familiar, you’ve been doing a good job of paying attention as the chapter has
progressed; this example has the same maintenance issues as the first shot at a static SQL stored
procedure
...
In addition, the SQL statement has been broken up into two component
parts, making it lack a good sense of flow
...

To solve this problem, I like to concatenate my dynamic SQL in one shot, using CASE expressions
instead of control-of-flow statements in order to optionally concatenate sections
...
Employee
WHERE 1=1' +
CASE
WHEN @BusinessEntityID IS NULL THEN ''
ELSE 'AND BusinessEntityID = ' + CONVERT(nvarchar, @BusinessEntityID)
END +
CASE
WHEN @NationalIDNumber IS NULL THEN ''
ELSE 'AND NationalIDNumber = N''' + @NationalIDNumber + ''''
END;
EXEC(@sql);
GO
In this example, the CASE expressions concatenate an empty string if one of the parameters is NULL
...

Thanks to the CASE expressions, the code is much more compact, and the query is still generally
formatted like a query instead of like procedural code
...
The query optimizer will “optimize
out” (i
...
, discard) 1=1 in a WHERE clause, so it has no effect on the resultant query plan
...
Each predicate can therefore be listed only once in the code, and combinations
are not a problem
...
Careful, consistent formatting can mean the
difference between quick changes to stored procedures and spending several hours trying to decipher
messy code
...
Employee
WHERE 1=1AND BusinessEntityID = 28AND NationalIDNumber = N'14417807'
Although this SQL is valid and executes as-is without exception, it has the potential for problems
due to the lack of spacing between the predicates
...
When I am
working with dynamic SQL, I concatenate every line separately, ensuring that each line is terminated
with a space
...
Following is an example of how I like to format my dynamic SQL:
DECLARE @BusinessEntityID int = 28;
DECLARE @NationalIDNumber nvarchar(15) = N'14417807';
DECLARE @sql nvarchar(max);
SET @sql = '' +
'SELECT ' +
'BusinessEntityID, ' +
'LoginID, ' +
'JobTitle ' +
'FROM HumanResources
...
Cutting everything up into individual tokens greatly
reduced the amount of whitespace, meaning that I could fit a lot more code in each variable
...


Now that the code fragment is properly formatted, it can be transferred into a new version of the
GetEmployeeData stored procedure:
ALTER PROCEDURE GetEmployeeData
@BusinessEntityID int = NULL,
@NationalIDNumber nvarchar(15) = NULL

216

CHAPTER 8

DYNAMIC T-SQL

AS
BEGIN
SET NOCOUNT ON;
DECLARE @sql nvarchar(max);
SET @sql = '' +
'SELECT ' +
'BusinessEntityID, ' +
'LoginID, ' +
'JobTitle ' +
'FROM HumanResources
...
At first glance, this might look
like a great solution, but it is still fraught with problems
...
Each set of input parameters produces the same execution plan as the baseline examples, with the
same estimated costs and number of reads
...
To illustrate this, execute the following T-SQL, which clears the query
plan cache and then runs the procedure with the same optional parameter, for three different input
values:
DBCC FREEPROCCACHE;
GO
EXEC GetEmployeeData
@BusinessEntityID = 1;
GO
EXEC GetEmployeeData
@BusinessEntityID = 2;
GO
EXEC GetEmployeeData
@BusinessEntityID = 3;
GO
Now, query the sys
...
Employee

@BusinessEntityID int = NULL
...
Employee
WHERE 1=1 AND BusinessEntityID = 2

Adhoc

SELECT BusinessEntityID, LoginID, JobTitle FROM HumanResources
...
This
means that every time a new argument is passed, a compilation occurs, which is clearly going to kill
performance
...
A stored
procedure implemented similarly to this one but with a minor modification would open a simple attack
vector that a hacker could exploit to easily pull information out of the database, or worse
...
The issue is a hacking technique called a SQL injection attack,
which involves passing bits of semiformed SQL to the database, typically via values entered in web
forms, in order to try to manipulate dynamic or ad hoc SQL on the other side
...
But what if
you were working with another stored procedure that had to be a bit more flexible? The following
example procedure, which might be used to search for addresses in the AdventureWorks2008 database,
gives an attacker more than enough characters to do some damage:
CREATE PROCEDURE FindAddressByString
@String nvarchar(60)
AS
BEGIN
SET NOCOUNT ON;
DECLARE @sql nvarchar(max);
SET @sql = '' +
'SELECT AddressID ' +

218

CHAPTER 8

DYNAMIC T-SQL

'FROM Person
...
The abridged list of results is as follows:
AddressID
16475
21773
23244
23742
16570
6

...
The WHERE clause for the query was
concatenated, such that it literally became WHERE AddressLine1 LIKE '%Stone%'
...
For instance,
consider what happens in the following case:
EXEC FindAddressByString
@String = ''' ORDER BY AddressID --';
GO
After concatenation, the WHERE clause reads, WHERE AddressLine1 LIKE '%' ORDER BY AddressID -%'
...
When supplied with this input, the results of the query now list every
AddressID in the Person
...

32521
This is, of course, a fairly mundane example
...
EmployeePayHistory --';
GO
Assuming that the account used for the query has access to the HumanResources
...
Address table, but the second lists all details from the EmployeePayHistory table:
BusinessEntityID

RateChangeDate

Rate

PayFrequency

ModifiedDate

1

1999-02-15

125
...
0769 2

2004-07-31


...
This includes viewing data,
deleting data, and inserting fake data
...

The solution is not to stop using dynamic SQL
...
Let me repeat that for effect: always, always, always parameterize your dynamic
SQL! The next section shows you how to use sp_executesql to do just that
...
Second, and perhaps more
important, is the threat of SQL injection attacks
...
Parameterization is a way to build a query such
that any parameters are passed as strongly typed variables, rather than formatted as strings and
appended to the query
...

The first step in parameterizing a query is to replace literals with variable names
...
Address ' +
'WHERE AddressLine1 LIKE ''%'' + @String + ''%''';
The only thing that has changed about this query compared to the version in the last section is that
two additional single quotes have been added such that the literal value of @String is no longer
concatenated with the rest of the query
...
Address WHERE AddressLine1 LIKE '%Stone%'
As a result of this change, the literal value after concatenation is now the following:

SELECT AddressID FROM Person
...

The reason for this is that EXECUTE runs the SQL in a different context than that in which it was
created
...
Since the value of the variable is not concatenated directly into the query, the type
of SQL injection described in the previous section is impossible in this scenario
...

The solution to this problem is to use the sp_executesql system stored procedure, which allows you
to pass parameters to dynamic SQL, much as you can to a stored procedure
...

The following T-SQL shows how to execute the Person
...
Address ' +
'WHERE AddressLine1 LIKE ''%'' + @String + ''%'''
EXEC sp_executesql
@sql,
N'@String nvarchar(60)',
@String;
GO
Running this batch will produce the same results as calling FindAddressByString and passing the
string “Stone
...

For an example that uses multiple parameters, consider again the GetEmployeeData stored
procedure, now rewritten to use sp_executesql instead of EXECUTE:
ALTER PROCEDURE GetEmployeeData
@BusinessEntityID int = NULL,
@NationalIDNumber nvarchar(15) = NULL
AS
BEGIN
SET NOCOUNT ON;
DECLARE @sql nvarchar(max);
SET @sql = '' +
'SELECT ' +
'BusinessEntityID, ' +
'LoginID, ' +
'JobTitle ' +
'FROM HumanResources
...
Note that you can use a string variable for the second parameter, which might make
sense if you are defining a long list—but I usually keep the list in a string literal so that I can easily match
the definitions with the variables passed in from the outer scope
...
This is perfectly OK! There is very little overhead in
passing parameters into sp_executesql, and trying to work around this issue would either bring back the
combinatorial explosion problem or require some very creative use of nested dynamic SQL
...

To verify that sp_executesql really is reusing query plans as expected, run the same code that was
used to show that the EXECUTE method was not reusing plans:
DBCC FREEPROCCACHE;
GO
EXEC GetEmployeeData
@BusinessEntityID = 1;
GO
EXEC GetEmployeeData
@BusinessEntityID = 2;
GO
EXEC GetEmployeeData
@BusinessEntityID = 3;
GO
After running this code, query the sys
...
The results should
show two rows, as follows:
objtype

text

Proc

CREATE PROCEDURE GetEmployeeData

Prepared

(@BusinessEntityID INT, @NationalIDNumber nvarchar(60))SELECT
...


One plan is cached for the procedure itself, and one is cached for the invocation of the dynamic
query with the @BusinessEntityID parameter
...
However, the maximum number of plans that can be cached for the stored procedure is five:
one for the procedure itself and one for each possible combination of parameters
...


223

CHAPTER 8

DYNAMIC T-SQL

To begin, create a renamed version of the best-performing (but hardest-to-maintain) static SQL
version of the stored procedure
...
Employee
WHERE
BusinessEntityID = @BusinessEntityID
AND NationalIDNumber = @NationalIDNumber;
END
ELSE IF (@BusinessEntityID IS NOT NULL)
BEGIN
SELECT
LoginID,
JobTitle
FROM HumanResources
...
Employee
WHERE
NationalIDNumber = @NationalIDNumber;
END
ELSE
BEGIN
SELECT
LoginID,
JobTitle
FROM HumanResources
...
It also has no additional overhead associated with context switching, which may
make it slightly faster than a dynamic SQL solution if the queries are very simple
...

To test the performance of the GetEmployeeData_Static stored procedure, we’ll call it from a simple
C# application via ADO
...
But first we need to obtain a set of values to supply for the
@NationalIDNumber and @BusinessEntityID parameters
...
Employee
table, we’ll call the procedure with every combination of those two parameters: once supplying just the
BusinessEntityID, once supplying just the NationalIDNumber, once supplying both, and once providing
NULL for both parameters
...
Employee table, so this provides (290
x 3) + 1 = 871 combinations
...

The following code listing illustrates the C# code required to perform the test just described
...
Open();
/* Grab every combination of parameters */
SqlCommand cmd = new SqlCommand(@"
SELECT BusinessEntityId, NationalIDNumber FROM HumanResources
...
Employee
UNION ALL SELECT BusinessEntityId, NULL FROM HumanResources
...
SelectCommand = cmd;
DataTable dt = new DataTable();
da
...
Rows)
{
SqlCommand cmd2 = new SqlCommand("GetEmployeeData_static", sqlConn);
cmd2
...
StoredProcedure;
SqlParameter param = new SqlParameter("@BusinessEntityID", SqlDbType
...
Value = r[0];
cmd2
...
Add(param);
SqlParameter param2 = new SqlParameter("@NationalIDNumber",
SqlDbType
...
Value = r[1];

225

CHAPTER 8

DYNAMIC T-SQL

cmd2
...
Add(param2);
cmd2
...
Close();
Before running the code, be sure to clear out the stored procedure cache by issuing the following
command:
DBCC FREEPROCCACHE;
GO
Then execute the C# code to call the static procedure in a loop
...
This time, we’ll amend our original cache
plan query to add in performance counter columns from the sys
...
objtype AS type,
COUNT(DISTINCT st
...
execution_count) AS execution_count,
CAST(SUM(qs
...
execution_count)
/ 1000 AS avg_CPU_ms
FROM sys
...
dm_exec_cached_plans cp ON cp
...
plan_handle
CROSS APPLY sys
...
sql_handle) st
WHERE
st
...
Employee%'
AND st
...
dm_exec_sql_text%'
GROUP BY
cp
...
00000000000000000000000000000

Proc

1

8710

0
...
Each static query took, on average, 0
...
Not bad
...
Employee ' +
'WHERE 1=1 ' +
CASE
WHEN @BusinessEntityID IS NULL THEN ''
ELSE
'AND BusinessEntityID = ' +
CONVERT(nvarchar, @BusinessEntityID) + ' '
END +
CASE
WHEN @NationalIDNumber IS NULL THEN ''
ELSE
'AND NationalIDNumber = N''' +
@NationalIDNumber + ''' '
END;
EXEC(@sql);
END
Testing this stored procedure against the static solution, and later, the sp_executesql solution, will
create a nice means by which to compare static SQL against both parameterized and nonparameterized
dynamic SQL, and will show the effects of parameterization on performance
...
Interrogating the DMV tables after the test runs on my
system gives the following results:
type
Adhoc

plans
872

execution_count
8711

avg_CPU_ms
0
...
The
overall average CPU time for this method is 0
...

The final stored procedure to test is, of course, the sp_executesql solution
...
This time, call it
GetEmployeeData_sp_executesql:

227

CHAPTER 8

DYNAMIC T-SQL

CREATE PROCEDURE GetEmployeeData_sp_executesql
@BusinessEntityID int = NULL,
@NationalIDNumber nvarchar(15) = NULL
AS
BEGIN
SET NOCOUNT ON;
DECLARE @sql nvarchar(max);
SET @sql = '' +
'SELECT ' +
'LoginID, ' +
'JobTitle ' +
'FROM HumanResources
...
When finished, check the performance results one last
time
...
00000000000000000000000000000

Prepared

4

8710

0
...
Runs with fewer iterations or against stored procedures that are more expensive for SQL
Server to compile will highlight the benefits more clearly
...
Again, more complex stored procedures with longer runtimes will naturally
overshadow the difference between the dynamic SQL and static SQL solutions, leaving the dynamic SQL
vs
...


Note When running these tests on my system, I restarted my SQL Server service between each run in order to
ensure absolute consistency
...
This kind of test can also be useful for general
scalability testing, especially in clustered environments
...


Output Parameters
Although it is somewhat of an aside to this discussion, I would like to point out one other feature that
sp_executesql brings to the table as compared to EXECUTE—one that is often overlooked by users who
are just getting started using it
...

Output parameters become quite useful when you need to use the output of a dynamic SQL statement that
perhaps only returns a single scalar value
...

To define an output parameter, simply append the OUTPUT keyword in both the parameter definition list
and the parameter list itself
...

Since this is an especially contrived example, I will add that in practice I often use output parameters with
sp_executesql in stored procedures that perform searches with optional parameters
...


229

CHAPTER 8

DYNAMIC T-SQL

Dynamic SQL Security Considerations
To finish up this chapter, a few words on security are important
...
In
this section, I will briefly discuss permissions issues and a few interface rules to help you stay out of
trouble when working with dynamic SQL
...
This is extremely important from an authorization perspective, because upon execution,
permissions for all objects referenced in the dynamic SQL will be checked
...

This creates a slightly different set of challenges from those you get when working with static SQL
stored procedures, due to the fact that the change of context that occurs when invoking dynamic SQL
breaks any ownership chain that has been established
...


Interface Rules
This chapter has focused on optional parameters of the type you might pass to enable or disable a
certain predicate for a query
...
These parameters involve passing table names, column lists, ORDER
BY lists, and other modifications to the query itself into a stored procedure for concatenation
...

As a general rule, you should never pass any database object name from an application into a stored
procedure (and the application should not know the object names anyway)
...

For instance, assume you were working with the following stored procedure:
CREATE PROC SelectDataFromTable
@TableName nvarchar(200)
AS
BEGIN
SET NOCOUNT ON;
DECLARE @sql nvarchar(max);
SET @sql = '' +
'SELECT ' +

230

CHAPTER 8

DYNAMIC T-SQL

'ColumnA, ' +
'ColumnB, ' +
'ColumnC ' +
'FROM ' + @TableName;
EXEC(@sql);
END;
GO
Table names cannot be parameterized, meaning that using sp_executesql in this case would not
help in any way
...
If you know in advance that this stored procedure will
only ever use tables TableA, TableB, and TableC, you can rewrite the stored procedure to keep those table
names out of the application while still providing the same functionality
...
The IF block validates that exactly one table is selected (i
...
, the value of the
parameter corresponding to the table is set to 1), and the CASE expression handles the actual dynamic
selection of the table name
...
SQL Server includes a function called QUOTENAME, which
bracket-delimits any input string such that it will be treated as an identifier if concatenated with a SQL
statement
...

By using QUOTENAME, the original version of the dynamic table name stored procedure can be
modified such that there will be no risk of SQL injection:
ALTER PROC SelectDataFromTable
@TableName nvarchar(200);
AS
BEGIN
SET NOCOUNT ON;
DECLARE @sql nvarchar(max);
SET @sql = '' +
'SELECT ' +
'ColumnA, ' +
'ColumnB, ' +
'ColumnC ' +
'FROM ' + QUOTENAME(@TableName);
EXEC(@sql);
END;
GO
Unfortunately, this does nothing to fix the interface issues, and modifying the database schema may
still necessitate a modification to the application code
...

However, it is important to make sure that you are using dynamic SQL properly in order to ensure the
best balance of performance, maintainability, and security
...


232

CHAPTER 9

Designing Systems for
Application Concurrency
It is hardly surprising how well applications tend to both behave and scale when they have only one
concurrent user
...
Alas, that feeling can be instantly ripped away, transformed
into excruciating pain, when the multitude of actual end users start hammering away at the system, and
it becomes obvious that just a bit more testing of concurrent utilization might have been helpful
...

Concurrency can be one of the toughest areas in application development, because the problems
that occur in this area often depend on extremely specific timing
...
Even worse
is when the opposite happens, and a concurrency problem pops up seemingly out of nowhere, at odd
and irreproducible intervals (but always right in the middle of an important demo)
...
The key is
to understand a few basic factors:
•

What kinds of actions can users perform that might interfere with the activities of
others using the system?

•

What features of the database (or software system) will help or hinder your users
performing their work concurrently?

•

What are the business rules that must be obeyed in order to make sure that
concurrency is properly handled?

This chapter delves into the different types of application concurrency models you might need to
implement in the database layer, the tools SQL Server offers to help you design applications that work
properly in concurrent scenarios, and how to go beyond what SQL Server offers out of the box
...
In the context of a
database application, problems arising as a result of concurrent processes generally fall into one of three
categories:
•

Overwriting of data occurs when two or more users edit the same data
simultaneously, and the changes made by one user are lost when replaced by the
changes from another
...
Additionally, a more serious potential consequence is
that, depending on what activity the users were involved in at the time,
overwriting may result in data corruption at the database level
...
If two sales terminals are running and each
processes a sale for the same product at exactly the same time, there is a chance
that both terminals will retrieve the initial value and that one terminal will
overwrite instead of update the other’s change
...
A common example of where this problem can manifest
itself is in drill-down reports presented by analytical systems
...
As the user clicks summarized data items on the report, the
reporting system might return to the database in order to read the corresponding
detail data
...


•

Blocking may occur when one process is writing data and another tries to read or
write the same data
...
However, excessive blocking can greatly decrease an application’s
ability to scale, and therefore it must be carefully monitored and controlled
...
But for the sake of this section, I’ll ignore the technical side for now and keep the
discussion focused on the business rules involved
...
Do not block
readers from reading inconsistent data, and do not worry about overwrites or
repeatable reads
...


CHAPTER 9

DESIGNING SYSTEMS FOR APPLICATION CONCURRENCY

•

Pessimistic concurrency control: Assume that collisions will be frequent; stop
them from being able to occur
...
To avoid overwrites, do not allow
anyone to begin editing a piece of data that’s being edited by someone else
...
Block readers
from reading inconsistent data, and let the reader know what version of the data is
being read
...
To avoid overwrites, do not allow any process to overwrite a
piece of data if it has been changed in the time since it was first read for editing by
that process
...
Block readers
both from reading inconsistent data and encountering repeatable read problems
by letting the reader know what version of the data is being read and allowing the
reader to reread the same version multiple times
...


Each of these methodologies represents a different user experience, and the choice must be made
based on the necessary functionality of the application at hand
...

On the other hand, many applications cannot bear overwrites
...
However, the best way to
handle the situation for source control is up for debate
...
Subversion uses an optimistic scheme in which
anyone can edit a given file, but you receive a collision error when you commit if someone else has
edited it in the interim
...

Finally, an example of a system that supports MVCC is a wiki
...
This means that if two
users are making simultaneous edits, some changes might get overwritten
...

In later sections of this chapter I will describe solutions based on each of these methodologies in
greater detail
...

Isolation levels are set in SQL Server in order to tell the database engine how to handle locking and
blocking when multiple transactions collide, trying to read and write the same data
...

SQL Server’s isolation levels can be segmented into two basic classes: those in which readers are
blocked by writers, and those in which blocking of readers does not occur
...
A special subclass of the SNAPSHOT isolation level, READ
COMMITTED SNAPSHOT, is also included in this second, nonblocking class
...

Transaction isolation levels do not change the behavior of locks taken at write time, but rather only
those taken or honored by readers
...
The following T-SQL creates a table called Blocker in TempDB and populates it
with three rows:
USE TempDB;
GO
CREATE TABLE Blocker
(
Blocker_Id int NOT NULL PRIMARY KEY
);
GO
INSERT INTO Blocker VALUES (1), (2), (3);
GO
Once the table has been created, open two SQL Server Management Studio query windows
...

In each of the three blocking isolation levels, readers will be blocked by writers
...
In order to release the locks, roll back the transaction by running the
following in the blocking window:

ROLLBACK;
In the following section, I’ll demonstrate the effects of specifying different isolation levels on the
interaction between the blocking query and the blocked query
...
Refer to the topic “Locking in
the Database Engine” in SQL Server 2008 Books Online for a detailed explanation
...
The primary difference
between these three isolation levels is in the granularity and behavior of the shared locks they take,
which changes what sort of writes will be blocked and when
...
In this isolation level, a reader will hold
its locks only for the duration of the statement doing the read, even inside of an explicit transaction
...
The reason is that as soon as the SELECT ended, the locks it held were released
...


REPEATABLE READ Isolation
Both the REPEATABLE READ and SERIALIZABLE isolation levels hold locks for the duration of an explicit
transaction
...
On the other hand, SERIALIZABLE transactions
take locks at a higher level of granularity, such that no data can be either updated or inserted within the
locked range
...
However, inserts such as
the following will not be blocked:
BEGIN TRANSACTION;
INSERT INTO Blocker VALUES (4);
COMMIT;
Rerun the SELECT statement in the blocking window, and you’ll see the new row
...

Once you’re done investigating the topic of phantom rows, make sure to issue a ROLLBACK in both
windows
...
Any key—existent or not at the time of the SELECT—that is
within the range predicated by the WHERE clause will be locked for the duration of the transaction if the
SERIALIZABLE isolation level is used
...
In either case, the operation will be
forced to wait for the transaction in the blocking window to commit, since the transaction locks all rows
in the table—whether or not they exist yet
...
When you’re done, be sure to issue a ROLLBACK
...
However, you might wish to selectively hold locks only on specific
tables within a transaction in which you’re working with multiple objects
...
In a READ COMMITTED
transaction, this will have the same effect as if the isolation level had been escalated just for those tables to
REPEATABLE READ
...


Nonblocking Isolation Levels
The nonblocking isolation levels, READ UNCOMMITTED and SNAPSHOT, each allow readers to read data
without waiting for writing transactions to complete
...


READ UNCOMMITTED Isolation
READ UNCOMMITTED transactions do not apply shared locks as data is read and do not honor locks placed
by other transactions
...
To see what this means, run the following in the blocking window:
BEGIN TRANSACTION;
UPDATE Blocker
SET Blocker_Id = 10
WHERE Blocker_Id = 1;
GO
This operation will place an exclusive lock on the updated row, so any readers should be blocked
from reading the data until the transaction completes
...
This can be especially problematic when users are shown
aggregates that do not add up based on the leaf-level data when reconciliation is done later
...


241

CHAPTER 9

DESIGNING SYSTEMS FOR APPLICATION CONCURRENCY

SNAPSHOT Isolation
An alternative to READ UNCOMMITTED is SQL Server 2008’s SNAPSHOT isolation level
...

This is achieved by making use of a row-versioning technology that stores previous versions of rows in
TempDB as data modifications occur in a database
...

However, this isolation level is not without its problems
...
And secondly, for many apps, this kind of
nonblocking read does not make sense
...
A SNAPSHOT read might cause the user to receive an invalid quantity, because the
user will not be blocked when reading data, and may therefore see previously committed data rather
than the latest updated numbers
...
There are many possible caveats with both approaches, and they are not right for every app, or
perhaps even most apps
...
One place to start is the
MSDN Books Online article “Understanding Row Versioning-Based Isolation Levels,” available at
http://msdn
...
com/en-us/library/ms189050
...


From Isolation to Concurrency Control
Some of the terminology used for the business logic methodologies mentioned in the previous section—
particularly the adjectives optimistic and pessimistic—are also often used to describe the behavior of
SQL Server’s own locking and isolation rules
...
From SQL Server’s standpoint, the only concurrency control necessary is
between two transactions that happen to hit the server at the same time—and from that point of view, its
behavior works quite well
...
In this sense, a purely transactional mindset fails to deliver enough
control
...
When using these isolation
levels, writers are not allowed to overwrite data in the process of being written by others
...
From a business point of view, this falls quite
short of the pessimistic goal of keeping two end users from ever even beginning to edit the same data at
the same time
...
This
comparison is far easier to justify than the pessimistic concurrency of the other isolation levels: with
SNAPSHOT isolation, if you read a piece of data in order to make edits or modifications to it, and someone

242

CHAPTER 9

DESIGNING SYSTEMS FOR APPLICATION CONCURRENCY

else updates the data after you’ve read it but before you’ve had a chance to write your edits, you will get
an exception when you try to write
...

This doesn’t scale especially well if, for instance, the application is web-enabled and the user wants to
spend an hour editing the document
...
The OPTIMISTIC options support a very similar form of optimistic concurrency to that of
SNAPSHOT isolation
...

Although both SNAPSHOT isolation and the OPTIMISTIC WITH ROW VERSIONING cursor options work by
holding previous versions of rows in a version store, these should not be confused with MVCC
...
The rows are not available later—for instance,
as a means by which to merge changes from multiple writers—which is a hallmark of a properly
designed MVCC system
...
This isolation level implements the anarchy business methodology mentioned in
the previous section, and does it quite well—readers are not blocked by writers, and writers are not
blocked by readers, whether or not a transaction is active
...
The goal of SQL Server’s
isolation levels is to control concurrency at the transactional level, ultimately helping to keep data in a
consistent state in the database
...
The following sections discuss
how to use SQL Server in order to help define concurrency models within database applications
...
The company sent out the forms to
each of its several hundred thousand customers on a biannual basis, in order to get the customers’ latest
information
...
To make matters worse, a large percentage of the customer files were removed from the filing
system by employees and incorrectly refiled
...
The firm has tried to remove the oldest of the forms and bring the newer ones to the top of
the stack, but it’s difficult because many customers didn’t always send back the forms each time they
were requested—for one customer, 1994 may be the newest year, whereas for another, the latest form
may be from 2009
...
The workflow is as follows: for each profile
update form, the person doing the data input will bring up the customer’s record based on that
customer’s Social Security number or other identification number
...
If
the dates are the same, the firm has decided that the operator should scan through the form and make
sure all of the data already entered is correct—as in all cases of manual data entry, the firm is aware that

243

CHAPTER 9

DESIGNING SYSTEMS FOR APPLICATION CONCURRENCY

typographical errors will be made
...

As is always the case in projects like this, time and money are of the essence, and the firm is
concerned about the tremendous amount of profile form duplication as well as the fact that many of the
forms are filed in the wrong order
...


Progressing to a Solution
This situation all but cries out for a solution involving pessimistic concurrency control
...
e
...
If another operator is currently editing that
customer’s data, a message can be returned to the user telling him or her to try again later—this profile is
locked
...
A scheme I’ve
seen attempted several times is to create a table along the lines of the following:
CREATE TABLE CustomerLocks
(
CustomerId int NOT NULL PRIMARY KEY
REFERENCES Customers (CustomerId),
IsLocked bit NOT NULL DEFAULT (0)
);
GO
The IsLocked column could instead be added to the existing Customers table, but that is not
recommended in a highly transactional database system
...

In this system, the general technique employed is to populate the table with every customer ID in
the system
...
The first and most serious problem is that
between the query in the IF condition that tests for the existence of a lock and the UPDATE, the row’s value
can be changed by another writer
...
In order to remedy this issue, the IF
condition should be eliminated; instead, check for the ability to take the lock at the same time as you’re
taking it, in the UPDATE’s WHERE clause:
DECLARE @LockAcquired bit;
UPDATE CustomerLocks
SET IsLocked = 1
WHERE
CustomerId = @CustomerId
AND IsLocked = 0;
SET @LockAcquired = @@ROWCOUNT;
This pattern fixes the issue of two readers requesting the lock at the same time, but leaves open a
maintenance issue: my recommendation to separate the locking from the actual table used to store
customer data means that you must now ensure that all new customer IDs are added to the locks table
as they are added to the system
...
Instead,
define the existence of a row in the table as indication of a lock being held
...
Imagine that a
buggy piece of code exists somewhere in the application, and instead of calling the stored procedure to
take a lock, it’s occasionally calling the other stored procedure, which releases the lock
...
This is very dangerous, as it will
invalidate the entire locking scheme for the system
...
Both of these
problems can be solved with some additions to the framework in place
...

This token is nothing more than a randomly generated unique identifier for the lock, and will be used as
the key to release the lock instead of the customer ID
...
By using a GUID, this system greatly decreases the chance
that a buggy piece of application code will cause the lock to be released by a process that does not
hold it
...
That way, if the caller receives this exception, it can take appropriate action—rolling back
the transaction—ensuring that the data does not end up in an invalid state
...

One final, slightly subtle problem remains: what happens if a user requests a lock, forgets to hit the
save button, and leaves for a two-week vacation? Or in the same vein, what should happen if the
application takes a lock and then crashes 5 minutes later, thereby losing its reference to the token?
Solving this issue in a uniform fashion that works for all scenarios is unfortunately not possible, and
one of the biggest problems with pessimistic schemes is that there will always be administrative
overhead associated with releasing locks that for some reason did not get properly handled
...
An external job must be written to poll the table on a regular basis,
“expiring” locks that have been held for too long
...
The actual interval depends on the amount of activity your system experiences, but once every 20 or
30 minutes is sufficient in most cases
...
The primary challenge is defining a timeout period that
makes sense
...

Unfortunately, no matter what timeout period you choose, it will never work for everyone
...
The user will
have no recourse available but to call for administrative support or wait for the timeout period—and of
course, if it’s been designed to favor processes that take many hours, this will not be a popular choice
...
I am happy to say that I have never received a panicked call at 2:00 a
...
from a user
requesting a lock release, although I could certainly see it happening
...
These notifications should update the lock date/time column:
UPDATE CustomerLocks
SET LockGrantedDate = GETDATE()
WHERE LockToken = @LockToken;
The application can be made to send a heartbeat as often as necessary—for instance, once every 5
minutes—during times it detects user activity
...
If this design is used, the timeout period can be shortened
considerably, but keep in mind that users may occasionally become temporarily disconnected while
working; buffer the timeout at least a bit in order to help keep disconnection-related timeouts at bay
...
This might improve the flexibility of the system a bit by letting callers request a
maximum duration for a lock when it is taken, rather than being forced to take the standard expiration interval
...
Should
you implement such a solution, carefully monitor usage to make sure that this does not become an issue
...
While that’s
fine in many cases, it is important to make sure that every data consumer follows the same set of rules
with regard to taking and releasing locks
...
If an application
is not coded with the correct logic, violation of core data rules may result
...

I have come up with a technique that can help get around some of these issues
...
However, when it is actually set to a certain value, that value must exist in the
CustomerLocks table, and the combination of customer ID and token in the Customers table must
coincide with the same combination in the CustomerLocks table
...
However, it does not enforce NULL values; the trigger takes care of that, forcing writers to set the
lock token at write time
...

This technique adds a bit of overhead to updates, as each row must be updated twice
...


Application Locks: Generalizing Pessimistic Concurrency
The example shown in the previous section can be used to pessimistically lock rows, but it requires some
setup per entity type to be locked and cannot easily be generalized to locking of resources that span
multiple rows, tables, or other levels of granularity supported within a SQL Server database
...
Application locks are programmatic, named locks, which behave much like
other types of locks in the database: within the scope of a session or a transaction, a caller attempting to
acquire an incompatible lock with a lock already held by another caller causes blocking and queuing
...
By default, the lock is tied
to an active transaction, meaning that ending the transaction releases the lock
...
To set a
transactional lock, begin a transaction and request a lock name (resource, in application lock parlance)
...
A caller can also set a wait timeout period,

250

CHAPTER 9

DESIGNING SYSTEMS FOR APPLICATION CONCURRENCY

after which the stored procedure will stop waiting for other callers to release the lock
...
The return will be 0 if the lock was successfully acquired without waiting, 1 if the lock was
acquired after some wait period had elapsed, and any of a number of negative values if the lock was not
successfully acquired
...
The
following call checks the return value to find out whether the lock was granted:
BEGIN TRAN;
DECLARE @ReturnValue int;
EXEC @ReturnValue = sp_getapplock
@Resource = 'customers',
@LockMode = 'exclusive',
@LockTimeout = 2000;
IF @ReturnValue IN (0, 1)
PRINT 'Lock granted';
ELSE
PRINT 'Lock not granted';
To release the lock, you can commit or roll back the active transaction, or use the sp_releaseapplock
stored procedure, which takes the lock resource name as its input value:
EXEC sp_releaseapplock
@Resource = 'customers';
SQL Server’s application locks are quite useful in many scenarios, but they suffer from the same
problems mentioned previously concerning the discrepancy between concurrency models offered by
SQL Server and what the business might actually require
...
This is clearly not a scalable option, so I set out to write a replacement, nontransactional
application lock framework
...
I especially wanted
callers to be able to queue and wait for locks to be released by other resources
...

When considering the SQL Server 2008 features that would help me create this functionality, I
immediately thought of Service Broker
...


Note For a thorough background on SQL Server Service Broker, see Pro SQL Server 2008 Service Broker, by
Klaus Aschenbrenner (Apress, 2008)
...
This token also happens to be the conversation handle for a Service Broker
dialog, but I’ll get to that shortly
...
Finally, the
LastGrantedDate column is used just as in the examples earlier, to keep track of when each lock in the
table was used
...

To support the Service Broker services, I created one message type and one contract:
CREATE MESSAGE TYPE AppLockGrant
VALIDATION = EMPTY;
GO
CREATE CONTRACT AppLockContract (
AppLockGrant SENT BY INITIATOR
);
GO
If you’re wondering why there is a message used to grant locks but none used to request them, it’s
because this solution does not use a lock request service
...


252

CHAPTER 9

DESIGNING SYSTEMS FOR APPLICATION CONCURRENCY

I created two queues to support this infrastructure, along with two services
...
Both the initiator and target conversation
handles for that dialog are used to populate the InitiatorDialogHandle and TargetDialogHandle
columns, respectively
...
When another caller wants to acquire a lock
on the same resource, it gets the target dialog handle from the table and waits on it
...

The AppLockTimeout_Queue is used a bit differently
...
This is because no messages—except perhaps Service Broker system messages—
will ever be sent from or to it
...

In addition to being used as the lock token, the dialog serves another purpose: when it is started, a
lifetime is set
...
Upon receipt, an activation procedure is used to release the lock
...
Whenever a lock is released by a caller, the conversation is ended,
thereby clearing its lifetime timer
...
As this stored
procedure is quite long, I will walk through it in sections in order to explain the details more thoroughly
...
Following are the first several lines of the
stored procedure, ending where the transactional part of the procedure begins:
CREATE PROC GetAppLock
@AppLockName nvarchar(255),
@LockTimeout int,
@AppLockKey uniqueidentifier = NULL OUTPUT
AS
BEGIN
SET NOCOUNT ON;

253

CHAPTER 9

DESIGNING SYSTEMS FOR APPLICATION CONCURRENCY

SET XACT_ABORT ON;
--Make sure this variable starts NULL
SET @AppLockKey = NULL;
DECLARE @LOCK_TIMEOUT_LIFETIME int = 18000; --5 hours
DECLARE @startWait datetime = GETDATE();
DECLARE @init_handle uniqueidentifier;
DECLARE @target_handle uniqueidentifier;
BEGIN TRAN;
The stored procedure defines a couple of important local variables
...
It is currently hard-coded, but could easily
be converted into a parameter in order to allow callers to specify their own estimated times of
completion
...
The @startWait
variable is used in order to track lock wait time, so that the procedure does not allow callers to wait
longer than the requested lock timeout value
...
There is a bit of a backstory to why I had to use them
...

This is counterintuitive if you’re used to working with first-in first-out (FIFO) queues, but is designed to
allow the minimum number of activation stored procedures to stay alive after bursts of activity
...

Obviously such a scheme, while useful in the world of activation stored procedures, does not follow
the requirements of a queued lock, so I made use of a transactional application lock in order to force
serialization of the waits
...
Once inside the scope of the
transactional lock, it’s finally time to start thinking about the Service Broker queues
...
If so, the stored procedure
starts a wait on the queue for a message:
--Find out whether someone has requested this lock before
SELECT
@target_handle = TargetDialogHandle
FROM AppLocks
WHERE AppLockName = @AppLockName;
--If we're here, we have the transactional lock
IF @target_handle IS NOT NULL
BEGIN
--Find out whether the timeout has already expired
...

It’s possible that a caller could specify, say, a 2500ms wait time, and wait 2000ms for the transactional
lock
...
Therefore, the reduced timeout is used as the RECEIVE timeout on the WAITFOR command
...
If it is an AppLockGrant message, all is good—
the lock has been successfully acquired
...
If
an unexpected message type is received, an exception is thrown and the transaction is rolled back:
IF @message_type = 'AppLockGrant'
BEGIN
BEGIN DIALOG CONVERSATION @AppLockKey
FROM SERVICE AppLockTimeout_Service
TO SERVICE 'AppLockTimeout_Service'
WITH
LIFETIME = @LOCK_TIMEOUT_LIFETIME,
ENCRYPTION = OFF;
UPDATE AppLocks
SET
AppLockKey = @AppLockKey,
LastGrantedDate = GETDATE()

255

CHAPTER 9

DESIGNING SYSTEMS FOR APPLICATION CONCURRENCY

WHERE
AppLockName = @AppLockName;
END
ELSE IF @message_type IS NOT NULL
BEGIN
RAISERROR('Unexpected message type: %s', 16, 1, @message_type);
ROLLBACK;
END
END
END
The next section of code deals with the branch that occurs if the target handle acquired before
entering the IF block was NULL, meaning that no one has ever requested a lock on this resource before
...
Since the target
conversation handle is required for others to wait on the resource, and since the target handle is not
generated until a message is sent, the first thing that must be done is to send a message on the dialog
...
conversation_endpoints
catalog view, and the sent message is picked up so that no other callers can receive it:
ELSE
BEGIN
--No one has requested this lock before
BEGIN DIALOG @init_handle
FROM SERVICE AppLock_Service
TO SERVICE 'AppLock_Service'
ON CONTRACT AppLockContract
WITH ENCRYPTION = OFF;
--Send a throwaway message to start the dialog on both ends
SEND ON CONVERSATION @init_handle
MESSAGE TYPE AppLockGrant;
--Get the remote handle
SELECT
@target_handle = ce2
...
conversation_endpoints ce1
JOIN sys
...
conversation_id = ce2
...
conversation_handle = @init_handle
AND ce2
...
Once that’s taken care of, the
stored procedure checks to find out whether the @AppLockKey variable was populated
...
Otherwise, a timeout is assumed to have occurred, and all work is rolled back
...

Luckily, the accompanying ReleaseAppLock procedure is much simpler:
CREATE PROC ReleaseAppLock
@AppLockKey uniqueidentifier
AS
BEGIN
SET NOCOUNT ON;
SET XACT_ABORT ON;
BEGIN TRAN;

257

CHAPTER 9

DESIGNING SYSTEMS FOR APPLICATION CONCURRENCY

DECLARE @dialog_handle uniqueidentifier;
UPDATE AppLocks
SET
AppLockKey = NULL,
@dialog_handle = InitiatorDialogHandle
WHERE
AppLockKey = @AppLockKey;
IF @@ROWCOUNT = 0
BEGIN
RAISERROR('AppLockKey not found', 16, 1);
ROLLBACK;
END
END CONVERSATION @AppLockKey;
--Allow another caller to acquire the lock
SEND ON CONVERSATION @dialog_handle
MESSAGE TYPE AppLockGrant;
COMMIT;
END;
GO
The caller sends the acquired lock’s token to this procedure, which first tries to nullify its value in
the AppLocks table
...
Otherwise,
the conversation associated with the token is ended
...
This message will be picked up by
any other process waiting for the lock, thereby granting it
...
The following TSQL creates the procedure and enables activation on the queue:
CREATE PROC AppLockTimeout_Activation
AS
BEGIN
SET NOCOUNT ON;
SET XACT_ABORT ON;
DECLARE @dialog_handle uniqueidentifier;
WHILE 1=1
BEGIN
SET @dialog_handle = NULL;
BEGIN TRAN;
WAITFOR
(
RECEIVE @dialog_handle = conversation_handle

258

CHAPTER 9

DESIGNING SYSTEMS FOR APPLICATION CONCURRENCY

FROM AppLockTimeout_Queue
), TIMEOUT 10000;
IF @dialog_handle IS NOT NULL
BEGIN
EXEC ReleaseAppLock @AppLockKey = @dialog_handle;
END
COMMIT;
END
END;
GO
ALTER QUEUE AppLockTimeout_Queue
WITH ACTIVATION
(
STATUS = ON,
PROCEDURE_NAME = AppLockTimeout_Activation,
MAX_QUEUE_READERS = 1,
EXECUTE AS OWNER
);
GO
This procedure waits, on each iteration of its loop, up to 10 seconds for a message to appear on the
queue
...
If no message is received
within 10 seconds, the activation procedure exits
...
Just as with other pessimistic schemes, it’s important for the application to
keep the returned key in order to release the lock later, using the ReleaseAppLock stored procedure
...
After a lock has been requested once, its row in the AppLocks
table as well as its associated grant dialog will not expire
...
If
this should become an issue, use the LastGrantedDate column to find locks that have not been recently
requested, and call END CONVERSATION for both the initiator and target handles
...
Indeed, even the word optimistic evokes a much nicer feeling, and the
name is quite appropriate to the methodology
...
This means none
of the administrative overhead of worrying about orphans and other issues that can occur with
pessimistic locks, but it also means that the system may not be as appropriate to many business
situations
...
” For that firm, an optimistic scheme would mean many hours of lost time and money—
not a good idea
...
Just like with the personal information forms, these cards have not been well
managed by the employees of the insurance firm, and there is some duplication
...

In this scenario, the management overhead associated with a pessimistic scheme is probably not
warranted
...

The basic setup for optimistic concurrency requires a column that is updated whenever a row gets
updated
...

At update time, the retrieved value is sent back along with updates, and its value is checked to ensure
that it did not change between the time the data was read and the time of the update
...
The rowversion type is an 8-byte binary string that is
automatically updated by SQL Server every time a row is updated in the table
...
The following T-SQL updates
one of the rows and then retrieves all of the rows in the table:
UPDATE CustomerNames
SET CustomerName = 'Pluto'
WHERE CustomerId = 456;
GO
SELECT *
FROM CustomerNames;
GO
The output of this query reveals the values in the table to be now as follows:
CustomerId

CustomerName

Version

123

Mickey Mouse

0x00000000000007E3

456

Pluto

0x00000000000007E5

It’s important to note that any committed update operation on the table will cause the Version
column to get updated—even if you update a column with the same value
...

Using a column such as this to support an optimistic scheme is quite straightforward
...
First of
all, every update routine in the system must be made to comply with the requirements of checking the
version
...
Secondly, this setup does not leave you with
many options when it comes to providing a nice user experience
...

The solution to both of these problems starts with a change to the version column’s data type
...
Therefore, my first suggestion is to switch to either uniqueidentifier or
datetime
...
This is done by requiring that any updates to the table include an update to the
Version column, which is enforced by checking the UPDATE function in the trigger
...
This way, the trigger can check
the value present in the inserted table against the value in the deleted table for each row updated
...
Finally, the trigger can set a new version value for the
updated rows, thereby marking them as changed for anyone who has read the data
...
CustomerId = d
...
Version <> d
...

Perhaps it would be better to extend this trigger to help provide users with more options when they get a
conflict
...
To create an output
document similar to an ADO
...
CustomerId = d
...
Version <> d
...

Since the document contains the newer version value that caused the conflict, you can let the end
user perform a merge or choose to override the other user’s change without having to go back to the
database to get the new rows a second time
...
Triggers are a great tool because the caller has no control over them—they will
fire on any update to the table, regardless of whether it was made in a stored procedure or an ad hoc
batch, and regardless of whether the caller has bothered to follow the locking rules
...

The major performance problems caused by triggers generally result from lengthened transactions and the
resultant blocking that can occur when low-granularity locks are held for a long period
...
However, these triggers will slow down updates a bit
...
In addition, each of these triggers incurs
additional index operations against the base table that are not necessary for a simple update
...
However, that’s generally
the difference between a few milliseconds and a few more milliseconds—certainly not a big deal,
especially given the value that they bring to the application
...
performance, I personally would always choose
the former
...

MVCC is not concerned with making sure you can’t overwrite someone else’s data, because in an
MVCC scheme there is no overwriting of data—period
...
This means that there’s no reason to check for data collisions; no data can get lost if
nothing is being updated
...

Generally speaking, to benefit from MVCC, the cost of blocking for a given set of transactions must
outweigh all other resource costs, particularly with regard to disk I/O
...

However, due to the fact that no updates are taking place, blocking becomes almost nonexistent
...
To begin with, create a table and populate it with some sample data, as shown in the following
T-SQL:
CREATE TABLE Test_Updates
(
PK_Col int NOT NULL PRIMARY KEY,
Other_Col varchar(100) NOT NULL
);
GO
INSERT INTO Test_Updates (
PK_Col,
Other_Col)
SELECT DISTINCT
Number,
'Original Value'
FROM master
...
25';
COMMIT;
This code simulates an UPDATE followed by a quarter of a second of other actions taking place in the
same transaction
...

To simulate a concurrent environment, we will execute the preceding query simultaneously across
several parallel threads
...
microsoft
...
aspx/kb/944837
...
sql
...
exe -Slocalhost -dtempDB -i"c:\update_test
...
sql file across 25 threads, each running the query
100 times
...
514
...
25';
COMMIT;
Save this query as insert_test
...
The
elapsed time of this test when run on my laptop is 00:00:29
...

The results are fairly clear: when simulating a massive blocking scenario, inserts are the clear winner
over updates, thanks to the fact that processes do not block each other trying to write the same rows
...


268

CHAPTER 9

DESIGNING SYSTEMS FOR APPLICATION CONCURRENCY

Of course, there is a bit more to MVCC than the idea of using inserts instead of updates
...
A query such as the following can be used to get
the latest version of every row in the Test_Inserts table:
SELECT
ti
...
Other_Col,
ti
...
Version)
FROM Test_Inserts ti1
WHERE
ti1
...
PK_Col
);
I will not cover MVCC queries extensively in this section—instead, I will refer you to Chapter 11,
which covers temporal data
...

As you might guess, MVCC, while an interesting concept, cannot be applied as described here to
many real-world applications
...


Sharing Resources Between Concurrent Users
So far in this chapter, we’ve looked at the business rules for protecting the integrity of data—preventing
overwrites and minimizing occurrences of blocking—and a brief consideration of performance
implications of different concurrent models
...

By default, when multiple requests are made to a SQL Server instance, the database engine shares
resources between all concurrent users equally
...
In practice, however, it is rare for all queries to be of equal importance
...
Maintenance tasks, in contrast, should sometimes run only in
the background, when there is idle resource to fulfill them
...

I don’t intend to cover every aspect of Resource Governor in these pages: for readers not familiar
with the topic, I recommend reading the introduction to Resource Governor on Books Online, at
http://msdn
...
com/en-us/library/bb895232
...
In this section, I’ll concentrate only on
demonstrating some of the ways in which Resource Governor can be applied to benefit performance in
high-concurrency environments
...
For this example, we will classify all requests from UserA into the
GroupA workload group, and all requests from UserB into the GroupB workload group, as follows:

270

CHAPTER 9

DESIGNING SYSTEMS FOR APPLICATION CONCURRENCY

CREATE FUNCTION dbo
...
All requests from UserA are classified into the
GroupA workgroup, and all requests from UserB will be classified into the GroupB workgroup
...

You can also make classification decisions based on other factors, including the name of the host or
application supplied in the connection string (retrieved using HOST_NAME() and APP_NAME(), respectively),
and whether the user is a member of a particular role or group
...
It is
therefore crucially important to make sure that this function is thoroughly tested and optimized or else
you risk potentially crippling your system
...
You may find that you need to connect to SQL Server 2008
via DAC in order to investigate and repair issues if a classifier function goes awry
...
RGClassifier );
GO
ALTER RESOURCE GOVERNOR RECONFIGURE;
GO
Now that we have set up the basic resource governor infrastructure, we need some way of
monitoring the effects on performance in high-concurrency environments
...
exe)
...
Scroll down the list of available counters, and, when you get to the SQL

271

CHAPTER 9

DESIGNING SYSTEMS FOR APPLICATION CONCURRENCY

Server:Workload Group Stats heading, click to add the CPU Usage % measure for both GroupA and
GroupB
...


Figure 9-1
...
To demonstrate the effect of applying
these limits, we first need to place some load on the server
...
We can confirm that the classifier function is working correctly

272

CHAPTER 9

DESIGNING SYSTEMS FOR APPLICATION CONCURRENCY

by checking the sys
...
dm_resource_governor_workload_groups
WHERE
name IN ('GroupA', 'GroupB');
The results show that there are five requests running under each workload group
...
On my system, this appears as shown in Figure 9-2
...
Balanced performance of two default workload groups
Perhaps unsurprisingly, as we have left all resource governor settings at their default values, the
graph shows an approximately equal split of CPU usage percentage between the two workloads
...
Firstly, we’ll try to limit the CPU resources given to both resource
pools:
ALTER RESOURCE POOL PoolA
WITH ( MAX_CPU_PERCENT = 10 );

273

CHAPTER 9

DESIGNING SYSTEMS FOR APPLICATION CONCURRENCY

ALTER RESOURCE POOL PoolB
WITH ( MAX_CPU_PERCENT = 10 );
ALTER RESOURCE GOVERNOR RECONFIGURE;
GO
Executing this code, you might be surprised to find, makes no difference to the graph
...

In this case, since the combined stated MAX_CPU_PERCENT of all resource pools is less than 100 percent,
there is no contention
...
The MAX_CPU_PERCENT of a
resource pool is only enforced in situations where exceeding that limit would prevent another resource
pool from being granted its requested maximum CPU settings
...
You should therefore always test in environments with competing concurrent processes
...
For this example, we’ll ensure that the total
MAX_CPU_PERCENT of the two resource pools adds up to 100 percent so that neither will be allowed to
exceed that limit (lest the other would be denied CPU resources):
ALTER RESOURCE POOL PoolA
WITH ( MAX_CPU_PERCENT = 90 );
ALTER RESOURCE POOL PoolB
WITH ( MAX_CPU_PERCENT = 10 );
ALTER RESOURCE GOVERNOR RECONFIGURE;
GO
Following the rules explained previously, you would perhaps now expect the CPU usage between
PoolA and PoolB to be split in the ratio 90:10, and depending on your system, this might be the outcome
that you now observe on the performance monitor graph
...


274

CHAPTER 9

DESIGNING SYSTEMS FOR APPLICATION CONCURRENCY

Figure 9-3
...
In other
words, the MAX_CPU_PERCENT setting is used to allocate the amount of CPU resource granted to a resource
pool, as a percent of the CPU on which the process is running
...
I can confirm this by executing the following:
SELECT
rgwg
...
text AS SQL,
ot
...
status
FROM
sys
...
dm_os_tasks ot on er
...
task_address
JOIN sys
...
group_id = rgwg
...
dm_os_schedulers os ON ot
...
scheduler_id
CROSS APPLY sys
...
sql_handle) est
WHERE
rgwg
...


0

0

running

GroupB

WHILE(1=1) SELECT REPLICATE('b'
...

In order to be able to demonstrate the effects of the Resource Governor more clearly, it is necessary
to constrain the execution of SQL Server threads to a single CPU by setting the affinity mask as follows:
sp_configure 'show advanced options', 1;
RECONFIGURE;
GO
sp_configure 'affinity mask', 1;
RECONFIGURE;
GO

Caution Calling sp_configure 'affinity mask' with the bitmask value 1 will assign all SQL Server threads
to Processor 0, irrespective of how many CPU cores are present on the system
...


The results of changing the CPU affinity mask on my system are shown in Figure 9-4
...
In this case, I am running the tests on a dual-processor machine, but I’ve
just set the SQL Server affinity mask to only allow SQL Server processor threads on one processor
...


276

CHAPTER 9

DESIGNING SYSTEMS FOR APPLICATION CONCURRENCY

Figure 9-4
...
In many cases, it is
a business requirement to ensure that certain defined groups of users or specific sets of queries receive
preferential treatment to guarantee fast response times
...

So far, I have demonstrated how the Resource Governor can be used to limit the maximum server
resource usage in certain scenarios, but a more practical situation involves guaranteeing the minimum
resource available to a query
...

To understand the interplay between settings of different pools, suppose that we want PoolA to be
our preferential pool and PoolB to be our normal pool
...
We will state a minimum of
0 percent CPU for PoolB, and a maximum CPU limit of 50 percent
...
PoolA
specifies a minimum server CPU usage of 30 percent
...
The effective maximum
CPU usage of each pool in this case is therefore the same as the requested max: 90 percent for PoolA and
50 percent for PoolB
...
PoolB has no stated minimum CPU percent, so all of the
effective maximum CPU usage of 50 percent comes from the shared resource pool
...
microsoft
...
aspx
...
The previous
effective maximum of 90 percent must be reduced to ensure that PoolC’s
MIN_CPU_PERCENT request of 15 percent can be granted
...

PoolB remains unchanged, with an effective maximum of 50 percent of CPU
resource, all of which is taken from the shared resource pool
...


Note that in this section I have only demonstrated the allocation of CPU resource using
MIN_CPU_PERCENT and MAX_CPU_PERCENT, but the same resource-balancing rules can be applied to memory
resource assigned to resource pools using MIN_MEMORY_PERCENT and MAX_MEMORY_PERCENT
...
Always remember
that these are not hard limits, but are used to calculate how resources are allocated when there is
contention
...


Controlling Concurrent Request Processing
Whereas the previous section discussed changing the amount of resources available to various resource
pools used by the Resource Governor, it is also important to consider the configuration of workload
groups into which query requests are placed
...
Once the maximum request limit has been reached, any
additional requests for this workload group will be placed into a wait state until capacity becomes
available
...

Then reset the query plan cache, the Resource Governor statistics, and wait statistics held in the DMV
tables by issuing the following T-SQL:
ALTER RESOURCE GOVERNOR RESET STATISTICS;
DBCC FREEPROCCACHE;
DBCC SQLPERF('sys
...

On my laptop, the time reported is 00:03:23
...

Back in SQL Server Management Studio, we can examine the performance of these queries using the
following T-SQL:
SELECT
total_request_count AS requests,
total_queued_request_count AS queued,
total_lock_wait_count AS locks,
CAST(total_cpu_usage_ms AS decimal(18,9)) / total_request_count
AS avg_cpu_time,
max_request_cpu_time_ms AS max_cpu_time

279

CHAPTER 9

DESIGNING SYSTEMS FOR APPLICATION CONCURRENCY

FROM
sys
...
52998019801980198019801980198

max_cpu_time
80

Now let’s alter the resource governor settings to limit the maximum allowed number of concurrent
requests for this workload group to three
...

ALTER WORKLOAD GROUP GroupA
WITH ( GROUP_MAX_REQUESTS = 3
USING PoolA;

)

ALTER RESOURCE GOVERNOR RECONFIGURE
GO
Before running the tests again, reset the tables:
ALTER RESOURCE GOVERNOR RESET STATISTICS;
DBCC FREEPROCCACHE;
DBCC SQLPERF('sys
...
This time the final execution time for all the queries on
my system is 00:03:03
...
By limiting the maximum number of concurrent requests for the workgroup,
the total time taken to process a queue of 50,000 queries in this example has been reduced by
approximately 10 percent
...
To understand what’s
occurring, rerun the previous query against the sys
...
The
results on my system are shown following:

requests queued locks
50500
48372 36727

avg cpu_time
3
...

However, when there are fewer concurrent requests, each query experiences substantially fewer
waits on resources that are locked by other threads
...
As a result, not
only was the overall elapsed time to process a queue of requests reduced, but the consistency between
the time taken to fulfill each query was increased (with the maximum CPU time of 54ms when there
were only three concurrent requests, compared to 80ms in the previous example)
...
dm_os_wait_stats DMV where wait_type is RESMGR_THROTTLED
...
In this chapter, I introduced the various
concurrency models that should be considered from a business process and data collision point of view,
and explained how they differ from the similarly named concurrency models supported by the SQL
Server database engine
...
Optimistic concurrency, while more lightweight, might not be so
applicable to many business scenarios, and multivalue concurrency control, while a novel technique,
might be difficult to implement in such a way that allowing collisions will help deliver value other than a
performance enhancement
...
The
discussion here only scratched the surface of the potential for this technique, and I recommend that
readers interested in the subject dedicate some time to further research this powerful feature
...
Although generally a novel concept for many SQL developers, the principles of working with
spatial data have been well established for many years
...
However, until recently, spatial data
analysis has been regarded as a distinct, niche subject area, and knowledge and usage of spatial data has
remained largely confined within its own realm rather than being integrated with mainstream
development
...
Customers’ addresses, sales regions, the area targeted by a local marketing
campaign, or the routes taken by delivery and logistics vehicles all represent spatial data that can be
found in many common applications
...

After demonstrating how to use these methods to answer some common spatial questions, I’ll then
concentrate on the elements that need to be considered to create high-performance spatial applications
...
If you’re interested in a more
thorough introduction to spatial data in SQL Server, I recommend reading Beginning Spatial with SQL Server 2008,
one of my previous books (Apress, 2008)
...
These objects might be
tangible, physical things, like an office building, railroad, or mountain, or they might be abstract features
such as the imaginary line marking the political boundary between countries or the area served by a
particular store
...

There are three basic types of geometry that may be used with the geometry and geography datatypes:
Point, LineString, and Polygon:

283

CHAPTER 10

WORKING WITH SPATIAL DATA

•

A Point is the most fundamental type of geometry, representing a singular
location in space
...


•

A LineString is comprised of a series of two or more distinct points, together with
the line segments that connect those points together
...
A simple LineString is one in which the path drawn
between the points does not cross itself
...
A LineString that is both simple and closed is known as a
ring
...
A polygon may also specify one or more
internal rings, which define areas of space contained within the external ring but
excluded from the Polygon
...
Polygons are two-dimensional—they have a length measured as the
total length of all defined rings, and also an area measured as the space contained
within the exterior ring (and not excluded by any interior rings)
...
To make the
distinction clear, I will use the word geometry (regular font) as the generic name to describe Points, LineStrings,
and Polygons, and geometry (code font) to refer to the geometry datatype
...
GeometryCollections may be homogenous or heterogeneous
...
As such, it could be represented as a MultiLineString—a homogenous collection of
LineString geometries
...
It is also possible to have a heterogeneous GeometryCollection, such as a collection containing a
Point, three LineStrings, and two Polygons
...

Having chosen an appropriate type of geometry to represent a given feature, we need some way of
relating each point in the geometry definition to the relevant real-world position it represents
...
So how do we do this?
You are probably familiar with the terms longitude and latitude, in which case you may be thinking
that it is simply a matter of listing the relevant latitude and longitude coordinates for each point in the
geometry
...


284

CHAPTER 10

WORKING WITH SPATIAL DATA

Figure 10-1
...
There are, in fact, many different systems of
latitude and longitude, and the coordinates of a given point on the earth will vary depending on which
system is used
...

In order to understand how to specify the coordinates of a geometry, we first need to examine how
different spatial reference systems work
...
This ability is essential to enable spatial data to store the coordinates of geometries
used to represent features on the earth
...
There are many different types of coordinate systems used in various
fields of mathematics, but when defining geospatial data in SQL Server 2008, you are most likely to use a
spatial reference system based on either a geographic coordinate system or a projected coordinate
system
...


•

The longitude coordinate measures the angle in the equatorial plane between a
line drawn from the center of the earth to the point and a line drawn from the
center of the earth to the prime meridian
...
As such, latitude can vary between –90°
(at the South Pole) and +90° (at the North Pole)
...

Figure 10-2 illustrates how a geographic coordinate system can be used to identify a point on the
earth’s surface
...
e
...
In simple terms, a projected coordinate
system describes positions on a map rather than positions on a globe
...

Figure 10-3 illustrates how the same point illustrated in Figure 10-2 could be defined using a projected
coordinate system
...
Describing a position on the earth using a geographic coordinate system

Figure 10-3
...
We need to know additional information, such as where to
measure those coordinates from and in what units, and what shape to use to model the earth
...


Datum
A datum contains information about the size and shape of the earth
...

The reference ellipsoid is a three-dimensional shape that is used as an approximation of the shape
of the earth
...
The degree by which
the spheroid is squashed may be stated as a ratio of the semimajor axis to the difference between the two
axes, which is known as the inverse-flattening ratio
...
For this
reason, spatial applications that operate at a regional level tend to use a spatial reference system based
on whatever reference ellipsoid provides the best approximation of the earth’s surface for the area in
question
...
In North America, the NAD83 ellipsoid is most
commonly used, which has a semimajor axis of 6,378,137m and a semiminor axis of 6,356,752m
...
By establishing a set of points with known coordinates, these points
can then be used to correctly line up the coordinate system with the reference ellipsoid so that the
coordinates of other, unknown points can be determined
...


Prime Meridian
As defined earlier, the geographic coordinate of longitude is the angle in the equatorial plane between
the line drawn from the center of the earth to a point and the line drawn from the center of the earth to
the prime meridian
...

It is a common misconception to believe that there is a single prime meridian based on some
inherent fundamental property of the earth
...
One commonly used prime meridian passes through Greenwich, London,
but there are many others
...


288

CHAPTER 10

WORKING WITH SPATIAL DATA

Figure 10-4
...
There are many ways of creating such
map projections, and each one results in a different image of the world
...

It is very important to realize that, in order to represent a three-dimensional model of the earth on a
flat plane, every map projection distorts the features of the earth in some way
...
Other projections preserve
the properties of features that are close to the equator, but grossly distort features toward the poles
...
The magnitude of distortion of features portrayed on the map is
normally related to the extent of the area projected
...

Since the method of projection affects the features on the resulting map image, coordinates from a
projected coordinate system are only valid for a given projection
...
257223563
...
This
system is used by handheld GPS devices, as well as many consumer mapping products, including Google
Earth and Bing Maps APIs
...
spatial_references table), the
properties of this spatial reference system can be expressed as follows:
GEOGCS[
"WGS 84",
DATUM[
"World Geodetic System 1984",
ELLIPSOID[
"WGS 84",
6378137,
298
...
0174532925199433]
]
Returning to the example at the beginning of this chapter, using this spatial reference system, we
can describe the approximate location of each corner of the US Pentagon building as a pair of latitude
and longitude coordinates as follows:
38
...
869,
38
...
873,
38
...
058
-77
...
053
-77
...
058

Note that, since we are describing points that lie to the west of the prime meridian, the longitude
coordinate in each case is negative
...
This spatial
reference system is based on the 1983 North American datum, which has a reference ellipsoid of
6,378,137m and an inverse-flattening ratio of 298
...
This geodetic model is projected using a
transverse Mercator projection, centered on the meridian of longitude 75°W, and coordinates based on
the projected image are measured in meters
...
257222101
]
],
PRIMEM["Greenwich",0],
UNIT["Degree", 0
...
0],
PARAMETER["False_Northing", 0
...
0],
PARAMETER["Scale_Factor", 0
...
0],
UNIT["Meter", 1
...
However, it would be quite cumbersome if we had to write out the full details of the datum,
prime meridian, unit of measurement, and projection details every time we wanted to quote a pair of
coordinates
...
The two spatial reference systems
used in the preceding examples are represented by SRID 4326 and SRID 26918, respectively
...
What’s
more, since SQL Server does not provide any mechanism for converting between spatial reference
systems, if you want to perform any calculations involving two or more items of spatial data, each one
must be defined using the same SRID
...


291

CHAPTER 10

WORKING WITH SPATIAL DATA

Note To find out the SRID associated with any given spatial reference system, you can use the search facility
provided at www
...
org
...
Geometry
Early Microsoft promotional material for SQL Server 2008 introduced the geography datatype as suitable
for “round-earth” data, whereas the geometry datatype was for “flat-earth” data
...
” A simple analogy might be that, in terms of geospatial data, the geometry
datatype operates on a map, whereas the geography datatype operates on a globe
...
Furthermore, since
SQL Server needs to know the parameters of the ellipsoidal model onto which
those coordinates should be applied, all geography data must be based on one of
the spatial reference systems listed in the sys
...


•

The geometry datatype operates on a flat plane, which makes it ideal for dealing
with geospatial data from projected coordinate systems, including Universal
Transverse Mercator (UTM) grid coordinates, national grid coordinates, or state
plane coordinates
...
The geometry datatype can also be used to store
any abstract nonspatial data that can be modeled as a pair of floating point x, y
coordinates, such as the nodes of a graph
...
In the following sections I’ll analyze some of the other differences in more detail
...
SQL Server
2008 also allows you to store Z and M coordinates, which can represent two further dimensions associated with
each point (typically, Z is elevation above the surface, and M is a measure of time)
...


292

CHAPTER 10

WORKING WITH SPATIAL DATA

Standards Compliance
The geometry datatype operates on a flat plane, where the two coordinate values for each point represent
the x and y position from a designated origin on the plane
...
For
example, the following code listing demonstrates how to calculate the distance between a Point located
at (50,100) and a Point at (90,130) using the STDistance() method of the geometry datatype:
DECLARE @point1 geometry = geometry::Point(50, 100, 0);
DECLARE @point2 geometry = geometry::Point(90, 130, 0);
SELECT @point1
...
So why the
fuss about the geometry datatype?
One key benefit of implementing such functionality using the geometry datatype instead of rolling
your own code is that all the methods implemented by the geometry datatype conform to the Open
Geospatial Consortium (OGC) Simple Features for SQL Specification v1
...
0
...
By using the geometry datatype,
you can be sure that the results of any spatial methods will be the same as those obtained from any other
system based on the same standards
...
For example, consider the two LineStrings illustrated in Figure
10-5
...
Two LineStrings that cross but do not touch

293

CHAPTER 10

WORKING WITH SPATIAL DATA

In normal English language, most people would describe these two LineStrings as touching, but not
crossing
...
You can test this for yourself by
examining the results of the STTouches() and STCrosses() methods, as shown in the following code
listing:
DECLARE @x geometry = geometry::STLineFromText('LINESTRING(0 0, 0 10)', 0);
DECLARE @y geometry = geometry::STLineFromText('LINESTRING(10 0, 0 5, 10 10)', 0);
SELECT
@x
...
STTouches(@y);
The result of the STCrosses() method is 1, indicating that the LineString x crosses over the
LineString y
...
In this case, the two LineStrings intersect at a single point (5,5),
so they are deemed to cross
...
e
...
In this case, the point (5,5) lies in the
interior of both LineStrings rather than in their boundary, so the result of STTouches() is 0 (i
...
, false)
...
The geometry datatype, however, operates on a flat plane
...
This is
not a limitation of the geometry datatype in itself, but rather of the inevitable distortions introduced
when using a projected coordinate system to represent a round model of the earth
...

For this reason, results obtained using the geometry datatype will become less accurate than results
obtained using the geography datatype over large distances
...
For storing
spatial data contained within a single country or smaller area, the geometry datatype will generally
provide sufficient accuracy, and comes with the benefits of additional functionality over the geography
type
...
This means that applications using the geography datatype
may experience slightly slower performance than those based on the geometry datatype, although the
impact is not normally significant
...

However, there are other more important implications arising between the different models on
which the two datatypes are based
...
In this context, the term hemisphere means one-half of the surface of
the earth, centered about any point on the globe
...
Nor is it possible to have a
geography LineString that extends from London to Auckland and then on to Los Angeles
...
In contrast, there is no limit to the size of a geometry instance, which
may extend indefinitely on an infinite plane
...
As defined earlier, the external ring of a Polygon defines an area of space
contained within the Polygon, and may also contain one or more internal rings that define “holes”—
areas of space cut out from the Polygon
...
However, a problem occurs when you try to apply this definition on a
continuous round surface such as used by the geography datatype, because it becomes ambiguous as to
which area of space is contained inside a Polygon ring, and which is outside
...
Does the area contained within the Polygon represent the
Northern Hemisphere or the Southern Hemisphere?

Figure 10-6
...
e
...
When
defining a geography Polygon, SQL Server treats the area on the “left” of the path drawn between the
points as contained within the ring, whereas the points on the “right” side are excluded
...
Whenever you define geography
polygons, you must ensure that you specify the correct ring orientation or else your polygons will be
“inside-out”—excluding the area they were intended to contain, and including everything else
...


295

CHAPTER 10

WORKING WITH SPATIAL DATA

A final technical difference concerns invalid geometries
...

However, as developers we have to reluctantly accept that spatial data, like any other data, is rarely as
perfect as we would like
...

Rather perversely, perhaps, the geometry datatype, which conforms to OGC standards, is also the
datatype that provides options for dealing with data that fails to meet those standards
...
All geography data, in contrast, is assumed to be valid at all times
...
Since SQL Server
cannot import invalid geography data, you may have to rely on external tools to validate and fix any
erroneous data prior to importing it
...
Unfortunately, the most commonly used spatial format, the ESRI
shapefile format (SHP), is not directly supported by any of the geography or geometry methods, nor by
any of the file data sources available in SQL Server Integration Services (SSIS)
...
For
readers who are interested, the structure is documented at http://msdn
...
com/enus/library/ee320529
...


Well-Known Text
WKT is a simple, text-based format defined by the OGC for the exchange of spatial information
...
It is also the format used in
the spatial documentation in SQL Server 2008 Books Online, at http://msdn
...
com/enus/library/ms130214
...

The following code listing demonstrates the WKT string used to represent a Point geometry located
at an x coordinate of 258647 and a y coordinate of 665289:
POINT(258647 665289)
Based on the National Grid of Great Britain, which is a projected coordinate system denoted by the
SRID 27700, these coordinates represent the location of Glasgow, Scotland
...
209 -33
...
212 -33
...
This is in contrast to the expression of a “latitude, longitude” coordinate pair, which most
people are familiar with using in everyday speech
...

The inevitable rounding errors introduced when attempting to do so will lead to a loss of precision
...


Well-Known Binary
The WKB format, like the WKT format, is a standardized way of representing spatial data defined by the
OGC
...
Every WKB representation begins with a header section
that specifies the order in which the bytes are listed (big-endian or little-endian), a value defining the
type of geometry being represented, and a stream of 8-byte values representing the coordinates of each
point in the geometry
...
23 and longitude 21
...
Additionally, since it is a binary format, WKB maintains the precision of
floating-point coordinate values calculated from binary operations, without the rounding errors
introduced in a text-based format
...


Note Although SQL Server stores spatial data in a binary format similar to WKB, it is not the same
...


297

CHAPTER 10

WORKING WITH SPATIAL DATA

Geography Markup Language
GML is an XML-based language for representing spatial information
...
The following code demonstrates an example of the
GML representation of a point located at latitude –33
...
21:

-33
...
21

GML, like WKT, has the advantages of being easy to read and understand
...
However, it is very verbose—the GML representation of an
object occupies substantially more space than the equivalent WKT representation and, like WKT, it too
suffers from precision issues caused by rounding when expressing binary floating-point coordinate
values
...


Importing Data
It is very common to want to analyze custom-defined spatial data, such as the locations of your
customers, in the context of commonly known geographical features, such as political boundaries, the
locations of cities, or the paths of roads and railways
...

SQL Server doesn’t provide any specific tools for importing predefined spatial data, but there are a
number of third-party tools that can be used for this purpose
...
Types
...
To demonstrate one method of
importing spatial data, and to provide some sample data for use in the remaining examples in this
chapter, we’ll import a dataset from the Geonames web site (www
...
org) containing the
geographic coordinates of locations around the world
...
geonames
...
zip
...
If you would like
to use a smaller dataset, you can alternatively download the
http://download
...
org/export/dump/cities1000
...


Caution The Geonames allCountries
...


To store the Geonames information in SQL Server, first create a new table as follows:

298

CHAPTER 10

WORKING WITH SPATIAL DATA

CREATE TABLE allCountries(
[geonameid] int NOT NULL,
[name] nvarchar(200) NULL,
[asciiname] nvarchar(200) NULL,
[alternatenames] nvarchar(4000) NULL,
[latitude] real NULL,
[longitude] real NULL,
[feature class] nvarchar(1) NULL,
[feature code] nvarchar(10) NULL,
[country code] nvarchar(2) NULL,
[cc2] nvarchar(60) NULL,
[admin1 code] nvarchar(20) NULL,
[admin2 code] nvarchar(80) NULL,
[admin3 code] nvarchar(20) NULL,
[admin4 code] nvarchar(20) NULL,
[population] int NULL,
[elevation] smallint NULL,
[gtopo30] smallint NULL,
[timezone] nvarchar(80) NULL,
[modification date] datetime NULL
);
GO
I’ve kept all the column names and datatypes exactly as they are defined in the Geonames schema,
but you may want to adjust them
...

There are a variety of methods of importing the Geonames text file into the allCountries table—for
this example, however, we’ll keep things as simple as possible by using the Import and Export Wizard
...
When
prompted to choose a data source, select the Flat File Source option, click the Browse button, and
navigate to and select the allCountries
...
From the ‘Code page’
drop-down, scroll down and highlight 65001 (UTF-8), and then click the Columns tab in the left pane
...
Then click Advanced from the left
pane
...


299

CHAPTER 10

WORKING WITH SPATIAL DATA

Figure 10-7
...
Column Properties for Geonames Data

Column

Name

DataType

OutputColumnWidth

0

geonameid

Four-byte signed integer
[DT_I4]

1

name

Unicode string [DT_WSTR]

200

2

asciiname

Unicode string [DT_WSTR]

200

3

alternatenames

Unicode string [DT_WSTR]

4000

4

latitude

Float [DT_R4]

5

longitude

Float [DT_R4]

6

feature class

Unicode string [DT_WSTR]

1

7

feature code

Unicode string [DT_WSTR]

10

8

country code

Unicode string [DT_WSTR]

2

9

cc2

Unicode string [DT_WSTR]

60

10

admin1 code

Unicode string [DT_WSTR]

20

11

admin2 code

Unicode string [DT_WSTR]

80

12

admin3 code

Unicode string [DT_WSTR]

20

13

admin4 code

Unicode string [DT_WSTR]

20

14

population

Four-byte signed integer
[DT_I4]

15

elevation

Two-byte signed integer
[DT_I2]

16

gtopo30

Two-byte signed integer
[DT_I2]

17

timezone

Unicode string [DT_WSTR]

18

modification date

Date [DT_DATE]

80

301

CHAPTER 10

WORKING WITH SPATIAL DATA

Note For a full description of the columns available from the Geonames export data listed in Table 10-1, please
consult http://download
...
org/export/dump/readme
...


Once the columns are set up, click Next to proceed, and check that the destination database is
correctly configured to import the records to the allCountries table
...

On my laptop, the import takes about 10 minutes, so take this opportunity to stretch your legs and
get a coffee, if you wish
...
Every record in the table has an associated
latitude and longitude coordinate value, but these are currently held in separate, floating point,
columns
...
The following code listing illustrates the TSQL required to add a geography column called location, and update it using the Point() method based
on the coordinate values held in the latitude and longitude columns for each row:
ALTER TABLE allCountries
ADD location geography;
GO
UPDATE allCountries
SET location = geography::Point(latitude, longitude, 4326);
GO
The approach described here can be used to load spatial data from a variety of tabular formats such
as gazetteers of place names
...


Tip Having populated the location column, the latitude and longitude columns could be dropped from the
allCountries table if so desired—the individual coordinate values associated with each point can be obtained
using the Lat and Long properties of the location column instead
...
Because spatial data types are not comparable, there are certain restrictions placed on
the types of query in which they can be used—for example:
•

You can’t ORDER BY a geometry or geography column
...
However, you can UNION ALL two datasets
...
shape =
t2
...
shape
...
shape) =
1
...

It is important to notice that the methods available in a given situation are dependent on the
datatype being used
...
The most noticeable example is that, although both geometry
and geography support the ability to test whether two geometries intersect (using the STIntersects()
method), only the geometry datatype provides the methods required to test the specific sort of
intersection—whether one geometry crosses, overlaps, touches, or is contained within another
...

In general, the methods available using either type can be classified into one of two categories:
Methods that adhere to the OGC specifications are prefixed by the letters ST (an
abbreviation for spatiotemporal)
...

SQL Server also provides a number of extended methods, which provide additional
functionality on top of the OGC standard
...

In this section, I won’t discuss every available method—you can look these up on SQL Server Books
Online (or in Beginning Spatial with SQL Server 2008)
...
Before we begin, let’s
create a clustered primary key and a basic spatial index on the allCountries table to support any queries
executed against it:
ALTER TABLE allCountries
ADD CONSTRAINT PK_geonameid
PRIMARY KEY (geonameid );
GO
CREATE SPATIAL INDEX idxallCountries
ON allCountries(location) USING GEOGRAPHY_GRID

303

CHAPTER 10

WORKING WITH SPATIAL DATA

WITH (
GRIDS =(
LEVEL_1 = MEDIUM,
LEVEL_2 = MEDIUM,
LEVEL_3 = MEDIUM,
LEVEL_4 = MEDIUM),
CELLS_PER_OBJECT = 16);
I’ll cover spatial indexes later in this chapter, so don’t worry if you’re not familiar with the syntax
used here—it will become clear soon!

Nearest-Neighbor Queries
Perhaps the most commonly asked question regarding spatial data is, “Where is the closest x to a given
location?” or in the more general case, “How do we identify the nearest n features to a location?” This
type of query is generally referred to as a nearest-neighbor query, and there are a number of ways of
performing such a query in SQL Server 2008
...
This approach is demonstrated in the following query:
DECLARE @Point geography;
SET @Point = geography::STPointFromText('POINT(0 52)', 4326);
SELECT TOP 3
name,
location
...
STDistance(@Point) AS Distance
FROM
allCountries
ORDER BY
location
...
However, it will take a
very long time to execute, as will be explained in the following section
...
It then orders the results of the query from the
allCountries table based on their distance from this point, and returns the top three results as follows:

304

CHAPTER 10

Name

WKT

Distance

Periwinkle Hill

POINT (-0
...
0055)

616
...
0166667 52)

1144
...
0166667 52)

WORKING WITH SPATIAL DATA

1144
...
This results in a table scan of 6
...

Performing a nearest-neighbor search like this on a table containing many rows is very slow and costly,
as you will no doubt have discovered if you tried executing the example yourself!
An alternative method for finding nearest neighbors involves a two-stage approach
...
The size of the buffer is chosen to be large
enough so that it contains the required number of nearest neighbors, but not so large that it includes lots
of additional rows of data that exceed the desired number of results
...
By being selective in
choosing the candidate records to sort, we can take advantage of the spatial index
...
STBuffer(25000);

--25km search radius

DECLARE @Candidates table (
Name varchar(200),
Location geography,
Distance float
);
INSERT INTO @Candidates
SELECT
name,
location,
location
...
Filter(@SearchArea) = 1;
SELECT TOP 3 * FROM @Candidates ORDER BY Distance;

305

CHAPTER 10

WORKING WITH SPATIAL DATA

Note Notice that in the preceding example query, I explicitly included an index hint,
WITH(INDEX(idxallCountries)), to ensure that the query optimizer chooses a plan using the idxallCountries

spatial index
...
SQL Server 2008 SP1 improves the situation, but there are
still occasions when an explicit hint is required to ensure that a spatial index is used
...
The advantage of
the buffer approach is that, by first filtering the set of candidate results to only those that lie within the
vicinity of the feature, the number of times that STDistance() needs to be called is reduced, and the
dataset to be sorted becomes much smaller, making the query significantly faster than the basic nearestneighbor approach described previously
...
On my laptop, the
top three nearest neighbors are now obtained in just 33ms, whereas the time taken to execute the first
query was over an hour!

Figure 10-8
...

However, the main problem with this approach is that it relies on an appropriate buffer size in
which to search for candidate results
...
If you set the buffer size too small, then there is a
risk that the search area will not contain any candidates, resulting in the query failing to identify any
nearest neighbors at all
...
This might be based on a
known, uniform distribution of your data—for example, you know that any item in the dataset will never
lie more than 25km from its nearest neighbor
...
”
Rather than identifying candidates that lie within a fixed search zone (which faces the risk of failing
to find any nearest neighbors at all), a better approach is to create a query that identifies candidates
within an expanding search range
...
These candidate results can then be sorted in the second stage to find the true n
nearest neighbors
...
If you don’t already have a numbers table, you can create one and populate it with the
integers between 0 and 1,000 using the following code:
CREATE TABLE Numbers (
Number int PRIMARY KEY CLUSTERED
);
DECLARE @i int = 0;
WHILE @i <= 1000
BEGIN
INSERT INTO Numbers VALUES (@i);
SET @i = @i + 1;
END;
GO
The Numbers table will be joined to the allCountries table to create a series of expanding search
ranges
...
All of the features in
this search area are returned as candidates, and then the TOP 3 syntax is used to select the three true
nearest neighbors
...
STDistance(@Point) AS Distance,
1000*POWER(2, Number) AS Range
FROM
allCountries
WITH(INDEX(idxallCountries))
INNER JOIN Numbers
ON allCountries
...
STDistance(@Point) < 1000*POWER(2,Numbers
...
Remember that the Numbers table
contains consecutive integers, starting at zero
...
Location
...
Number) specifies that the initial
criterion for a feature to be considered a nearest-neighbor candidate is that the distance between that
feature and @Point is less than 1000 * 2^0
...

If the requisite number of neighbors (in this case, we are searching for the top three) is not found
within the specified distance, then the search range is increased in size
...
Thus, the first range extends
to 1km around the point, the second range extends to 2km—then 4km, 8km, 16km, and so on
...

Once the search range has been sufficiently increased to contain at least the required number of
candidate nearest neighbors, all of the features lying within that range are selected as candidates, by
using a SELECT statement with the WITH TIES argument
...
The Range
column returned in the results states the distance to which the search range was extended to find the
nearest neighbor
...
While it is slightly more
complex, this last approach provides the most flexible solution for implementing a nearest-neighbor
query
...
To enhance the performance of this query further, we’ll need to look at the matter of spatial
indexes, which is covered later in this chapter
...
Whereas once considered gimmicks and
eye candy with little real value, these tools are now being used in increasingly important applications,
including for disaster response planning, logistics, epidemiology, and other health-reporting scenarios
...
Users pan and zoom the map to display a particular area of interest, and any data
contained within the visible map view is retrieved from the database to be plotted on the map
...

The first requirement is to identify the area visible within a given map view—in other words, the
bounding box of the map
...
Unfortunately, these
is no standardized view of which two corners to use: the GetMapView() method in Bing Maps returns the
coordinates of the points at the top-left and bottom-right corners of the map, whereas the equivalent
getBounds() method used by Google Maps returns the points at the southwest and northeast corners
...

Having identified the coordinate of two opposing points, we next need to construct a Polygon
representing the bounding box enclosed by those points
...
The coordinate values of the top-left and
bottom-right corners of the map are (41,–109) and (37,–102), respectively
...
Typical viewport of a web-mapping application
The coordinates of the bounding box returned from Bing Maps/Google Maps are expressed using
latitude/longitude values measured using the SRID 4326
...
A first shot at doing so might look something like
this:
DECLARE @TopLeft geography = geography::Point(41, -109, 4326);
DECLARE @BotRight geography = geography::Point(37, -102, 4326);
DECLARE @BoundingBox geography;
SET @BoundingBox = geography::STPolyFromText('POLYGON(('
+ CAST(@TopLeft
...
Lat AS varchar(32)) + ','
+ CAST(@TopLeft
...
Lat AS varchar(32)) + ','
+ CAST(@BotRight
...
Lat AS varchar(32)) +','

309

CHAPTER 10

WORKING WITH SPATIAL DATA

+ CAST(@BotRight
...
Lat AS varchar(32)) + ','
+ CAST(@TopLeft
...
Lat AS varchar(32))
+ '))',
4326);
SELECT @BoundingBox
...

Although the coordinate values of the corner points of the map bounding box are expressed in
geographic coordinates of latitude and longitude, the default two-dimensional map view presented in
the browser is flat
...

Given a little thought, this fact should be obvious—unless you have a very clever monitor capable of
displaying images in three-dimensional space, all geospatial data on a computer display must have been
projected
...

Unlike most projected reference systems, SRID 3785 uses a Mercator projection based on a perfectly
spherical model of the earth
...
In this example, this means that
every point lying along the bottom edge of the map, which connects the points at (37,–109) and (37,–
102), all have a latitude of 37
...
Remember that the geography datatype operates on an
ellipsoidal model, so the edge between the points (37,–109) and (37,–102) represents the shortest
distance between those two points on the surface of the reference ellipsoid in question (SRID 4326, in
this case)
...

What does this all mean for the application in this example? Consider a point located at latitude
37
...
419
...
Since the
latitude of 37
...
419 lies between –102 and –
109, this point is contained in the map view used in this example, and we would expect it to be included
in the result set retrieved from the database
...
STAsText(),
location
...
The
reason might become more obvious if we take a visual look at what’s happening here
...


Figure 10-10
...
The result is that the point
representing Arboles falls outside the Polygon, and is not included in the results of the geography
STIntersects() method
...
One solution to this problem is to create additional
anchor points along the bottom and top edges of the Polygon, which will lead to a closer approximation
of the projected map view, but will make our Polygon more complex and potentially slower to use in
spatial queries
...
In other words, execute a
query directly on the latitude and longitude columns as follows:
SELECT *
FROM allCountries
WHERE
latitude > 37 AND latitude < 41
AND
longitude > -109 AND longitude < -102;
Not only will this avoid the problem of mixing flat/round data, but it will also perform significantly
faster than the previous solution using the geography datatype if we were to add an index to the
longitude and latitude columns
...
If we wanted to extend the application so that users could draw an irregular
search area on the map, or if we wanted to search for LineStrings or Polygons that intersected the area,
this approach would not work since we could no longer perform a query based on simple search criteria
...
The latitude
coordinate is mapped directly to the y coordinate, and the longitude coordinate is mapped to the x
coordinate
...
To implement this approach, execute the following code listing:
ALTER TABLE allCountries
ADD locationgeom geometry;
GO
UPDATE allCountries
SET locationgeom = geometry::STGeomFromWKB(location
...
STSrid);
GO
Selecting those geometry records that intersect the Polygon POLYGON ((-109 41, -109 37, -102 37,
-102 41, -109 41)) now gives the results expected
...
Other situations in which
this can occur are when you need to rely on a function that is only available within the geometry
datatype, such as STConvexHull() or STRelate()
...


312

CHAPTER 10

WORKING WITH SPATIAL DATA

However, you should exercise great caution when using the geometry datatype to store geographic
data in this way, because you may receive surprising results from certain operations
...
If using the geometry datatype to store
coordinate values expressed in geographic coordinates of latitude and longitude, this means that lengths
will be measured in degrees, and areas in degrees squared, which is almost certainly not what you want
...
Nowhere is this truer than in the realm of spatial indexes, where
it is not uncommon to witness performance improvements exceeding 1,000 percent by creating a spatial
index on even a small table of data
...

Spatial indexes operate very differently compared to the clustered and nonclustered indexes used
for more conventional datatypes
...
To understand why, in
this section I’ll first provide an overview of spatial indexing, and then look at some of the ways of
optimizing a spatial index to provide optimal performance for your spatial applications
...
The result set obtained from the primary filter is a superset—while it is
guaranteed to contain all of the true results, it may also contain false positives
...
The secondary filter is
more accurate, but slower than the primary filter
...
To do this, spatial indexes in SQL Server utilize a
multilevel grid model, with four levels of grid nested inside each other, as illustrated in Figure 10-11
...
Rather than describing the detailed shape of the associated
geometry, each entry in a spatial index comprises a reference to a grid cell, together with the primary key
of the geometry that intersects that cell
...
For more information on these topics, please refer to a book
dedicated to the subject, or refer to Books Online
...
The multilevel grid used by spatial indexes

314

CHAPTER 10

WORKING WITH SPATIAL DATA

DECLARE @BoundingBox geography;
SET @BoundingBox = geography::STPolyFromText(
'POLYGON ((-109 41, -109 37, -102 37, -102 41, -109 41))',
4326);
SELECT
name,
location
...
STIntersects(@BoundingBox) = 1;
The location column is included in the idxallCountries geography index, and the predicate
location
...
To do so, the @BoundingBox
parameter is first tessellated according to the same grid as the idxallCountries index on the location
column
...
The outcome of the primary filter can lead to one of three results:
•

If a geometry in the location column has no index cells in common with the cells
occupied by @BoundingBox, it can be discarded by the primary filter, since the
geometries themselves cannot intersect
...
This row
can therefore definitely be included in the result set without need to call the
secondary filter
...


•

If the geometry only partially occupies a grid cell occupied by @BoundingBox, it
cannot be determined for certain whether that cell intersects the @BoundingBox
...


To get the best performance from a spatial query, the ideal goal is to get as much of the processing
done by the primary filter, and reduce the number of times that the secondary filter needs to be called
...
Achieving this goal requires tuning the grid
properties to best match the data in the underlying dataset and the type of queries run against that data
...
How well an
index succeeds in meeting these two aims is largely determined by the values chosen for the grid
resolution, the bounding box, and the cells per object parameters of the index
...
However, in the following section I’ll give you some general ideas to bear
in mind when determining the settings for a spatial index
...
You may find this useful in order to index unevenly
distributed data
...
The resolution at each level of the grid may
be set independently to one of three resolutions: LOW corresponds to a 4×4 grid,
MEDIUM corresponds to an 8×8 grid, and HIGH corresponds to a 16×16 grid
...
e
...
These
false positives will lead to more work having to be done by the secondary filter,
leading to query degradation
...
e
...

Another effect of a high resolution may be that the number of cells required to fully
tessellate the geometry exceeds the CELLS_PER_OBJECT limit, in which case
tessellation will not be fully complete
...
Specifying a smaller bounding box but maintaining
the same number of grid cells will lead to each individual grid cell being smaller,
creating a more precise fit around any features and making the primary filter more
accurate
...
For
the geography datatype, there is no explicit bounding box, as every geography index
is assumed to cover the whole globe
...
The optimum number of cells per object is intricately linked to the
resolution of the cells used at each level; a higher-resolution grid will contain
smaller cells, which may mean that more cells are required to fully cover the object
at a given level of the grid
...
In such
cases, the grid cells will not be fully subdivided and the index entry will not be as
accurate as it can be
...
This may lead to a
more accurate index, but a slower query, thereby negating the purpose of using a
spatial index in the first place
...
This index used the default parameters of MEDIUM grid

316

CHAPTER 10

WORKING WITH SPATIAL DATA

resolution at all four levels, and 16 cells per object
...

To assess the effectiveness of the idxallCountries index, we could simply obtain some performance
timings using queries with different index settings
...

To use either of these procedures, you supply parameters for the table and index name to be tested,
together with a query sample—a geography or geometry instance that will be tessellated according to the
settings used by the index
...
The important thing to bear in mind is that, without a spatial
index on this table, a query to find out which rows lie within the chosen query sample would have to call
the STIntersects() method on every one of these 6
...
Fortunately, this is not the case,
because the spatial index can provide a primary filter of these records, as shown in the following rows
returned by the procedure:
Number_Of_Rows_Selected_By_Primary_Filter

49018

Number_Of_Rows_Selected_By_Internal_Filter

38913

Number_Of_Times_Secondary_Filter_Is_Called

10105

Based on the primary filter of the table, 49,018 records were selected as candidates for this query
sample
...
71 percent of the total number of rows in the table
...

For example, in the case of an intersection predicate, if a point lies in a grid cell that is completely
covered by the query sample, the point is certain to be intersected by that geometry, and so the
STIntersects() method need never be used
...
In this case, 79
...


317

CHAPTER 10

WORKING WITH SPATIAL DATA

The remaining 10,105 records selected by the primary filter lay in index cells that were only partially
intersected by the geometry specified in the query sample, and so had to rely on the secondary filter to
confirm whether they should be selected or not
...
Note that the final number-of-rows output is less than the number of rows initially selected by
the primary filter, as some of those rows would have been false positives that were then eliminated by
the secondary filter
...
7818911685995

Primary_Filter_Efficiency

91
...
The only difference is in how those rows
are identified
...
476 percent of rows
selected by the primary filter were included in the final results
...
78 percent, indicating the percentage of output rows selected just from the internal filter
...

Now let’s consider what happens when we create a new index, using HIGH resolution at all four grid
levels:
CREATE SPATIAL INDEX idxallCountriesHigh
ON allCountries(location) USING GEOGRAPHY_GRID
WITH (
GRIDS =(
LEVEL_1 = HIGH,
LEVEL_2 = HIGH,
LEVEL_3 = HIGH,
LEVEL_4 = HIGH),
CELLS_PER_OBJECT = 16);
Once again, we’ll examine the properties of this index using the sp_help_spatial_geography_index
procedure:
EXEC sp_help_spatial_geography_index
@tabname = allCountries,
@indexname = idxallCountriesHigh,
@verboseoutput = 1,
@query_sample = 'POLYGON ((-109 41, -109 37, -102 37, -102 41, -109 41))';
Unsurprisingly, the Base_Table_Rows value remains unchanged at 6,906,119, as does the total
number-of-rows output from the query, 44,840
...
4766 to
91
...
However, there is a very important distinction between the indexes, as shown in the following
rows:

318

CHAPTER 10

WORKING WITH SPATIAL DATA

Number_Of_Rows_Selected_By_Internal_Filter

25333

Number_Of_Times_Secondary_Filter_Is_Called

23824

Percentage_Of_Primary_Filter_Rows_Selected_By_Internal_Filter

51
...
4964317573595

When using a MEDIUM grid resolution, 79
...
The secondary filter therefore only needed to be
called on the remaining 10,105 rows
...
53 percent of the
primary filter could be preselected by the internal filter
...

At first consideration, this may seem illogical
...
Each Point
geometry would only require a single cell in the index, and by using the HIGH resolution, that cell would
be as granular as possible
...
In this case,
@BoundingBox is quite a large Polygon representing the state of Colorado, which requires a great number
of cells to fully cover
...
In this example, the MEDIUM resolution grid provides a more accurate
index, and hence better performance, than the HIGH resolution grid
...
Tuning spatial indexes requires a
large degree of trial and error, but the stored procedures introduced here can provide valuable statistics
to assess the performance of an index to help the process
...
As more applications and services
become location-aware, there is a requirement for all types of data to be stored with associated spatial
information in a structured, searchable manner
...
However, the complexity and uniqueness of spatial data means that specific
approaches must be taken to ensure that spatial queries remain performant
...


319

C H A P T E R 11

Working with Temporal Data
It’s probably fair to say that time is a critical piece of information in almost every useful database
...
Without a time axis, it is impossible to describe the number of purchases
made last month, the average overnight temperature of the warehouse, or the maximum duration that
callers were required to hold the line when calling in for technical support
...

In this chapter, I will delve into the ins and outs of dealing with time in SQL Server
...


Modeling Time-Based Information
When thinking of “temporal” data in SQL Server, the scenario that normally springs to mind is a
datetime column representing the time that some action took place, or is due to take place in the future
...
Some of the categories of time-based information that may be modeled in SQL Server are
as follows:
•

Instance-based data is concerned with recording the instant in time at which an
event occurs
...
Scenarios in which you might model an instance include the moment
a user logs into a system, the moment a customer makes a purchase, and the exact
time any other kind of event takes place that you might need to record in the
database
...


•

Interval-based data extends on the idea of an instance by describing the period of
time between a specified start point and an endpoint
...
A subset of interval-based data is the idea of duration, which

321

CHAPTER 11

WORKING WITH TEMPORAL DATA

records only the length of time for which an event lasts, irrespective of when it
occurred
...

•

Period-based data is similar to interval-based data, but it is generally used to
answer slightly different sorts of questions
...
” Although these are similar to—and can be
represented by—intervals, the mindset of working with periods is slightly
different, and it is therefore important to realize that other options exist for
modeling them
...


•

Bitemporal data is temporal data that falls into any of the preceding categories,
but also includes an additional time component (known as a valid time, or more
loosely, an as-of date) indicating when the data was considered to be valid
...
When querying the database
bitemporally, the question transforms from “On a certain day, what happened?”
to “As of a certain day, what did we think happened on a certain (other) day?” The
question might also be phrased as “What is the most recent idea we have of what
happened on a certain day?” This mindset can take a bit of thought to really get;
see the section “Managing Bitemporal Data” later in this chapter for more
information
...
Prior to SQL Server 2008, there wasn’t
really a whole lot of choice when it came to storing temporal data in SQL Server—the only temporal
datatypes available were datetime and smalldatetime and, in practice, even though it required less
storage, few developers used smalldatetime owing to its reduced granularity and range of values
...
The full list of supported temporal datatypes is listed in Table 11-1
...
Date/Time Datatypes Supported by SQL Server 2008

Datatype

Resolution

Storage

datetime

January 1, 1753, 00:00:00
...
997

3
...
0000000–December 31,
9999, 23:59:59
...
0000000–December 31,
9999, 23:59:59
...
0000000–
23:59:59
...
What developers actually need to understand when
working with SQL Server’s date/time types is what input and output formats should be used, and how to
manipulate the types in order to create various commonly needed queries
...


Input Date Formats
There is really only one rule to remember when working with SQL Server’s date/time types: when
accepting data from a client, always avoid ambiguous date formats! The unfortunate fact is that,
depending on how it is written, a given date can be interpreted differently by different people
...

It’s nearly 12:35 p
...
Why is this of particular interest? Because if I write the current time and date, it
forms an ascending numerical sequence as follows:

12:34:56 07/08/09
I live in England, so I tend to write and think of dates using the dd/mm/yy format, as in the preceding
example
...
And if you’re from one of various Asian countries (Japan, for instance), you might
have seen this sequence occur nearly two years ago, on August 9, 2007
...

Luckily, there is a solution to this problem
...

ISO 8601 is an international standard date/time format, which SQL Server (and other software) will
automatically detect and use, independent of the local server settings
...
mmm
yyyy is the four-digit year, which is key to the format; any time SQL Server sees a four-digit year first,
it assumes that the ISO format is being used
...
According to the standard, the hyphens and the T
are both optional, but if you include the hyphens, you must also include the T
...
However, one
important point to note is that whatever datatype is being used, both the time and date elements of any

323

CHAPTER 11

WORKING WITH TEMPORAL DATA

input are optional
...
In a similar vein, if a time component
is provided as an input to the date datatype, or a date is supplied to the time datatype, that value will
simply be ignored
...
To demonstrate the importance of this
character, compare the results of the following: SET LANGUAGE British; SELECT CAST('2003-12-09
00:00:00' AS datetime), CAST('2003-12-09T00:00:00' AS datetime)
...

Remember that SQL Server does not store the original input date string; the date is converted and stored
internally in a binary format
...

Unfortunately, it’s not always possible to get data in exactly the right format before it hits the
database
...

To use CONVERT to create an instance of date/time data from a nonstandard date, use the third
parameter of the function to specify the date’s format
...
” By using these styles, you can more easily control how date/time input is processed, and explicitly

324

CHAPTER 11

WORKING WITH TEMPORAL DATA

tell SQL Server how to handle input strings
...

The other commonly used option for controlling the format of input date strings is the DATEFORMAT
setting
...
The following T-SQL is equivalent to the previous example
that used CONVERT:
--British/French style
SET DATEFORMAT DMY;
SELECT CONVERT(date, '01/02/2003');
--US style
SET DATEFORMAT MDY;
SELECT CONVERT(date, '01/02/2003');
There is really not much of a difference between using DATEFORMAT and CONVERT to correct
nonstandard inputs
...
In the
end, you should choose whichever option makes the particular code you’re working on more easily
readable, testable, and maintainable
...
This may cause a performance problem in some cases, so make sure to test carefully
before deploying solutions to production environments
...
It is also
commonly used to format dates for output
...
By formatting dates into strings in the data layer, you may reduce the
ease with which stored procedures can be reused
...
Such additional work on the part of the application is probably unnecessary, and
there are very few occasions in which it really makes sense to send dates back to an application
formatted as strings
...

Just like when working with input formatting, the main T-SQL function used for date/time output
formatting is CONVERT
...
The following T-SQL shows how to format the current date as a
string in both US and British/French styles:

325

CHAPTER 11

WORKING WITH TEMPORAL DATA

--British/French style
SELECT CONVERT(varchar(50), GETDATE(), 103);
--US style
SELECT CONVERT(varchar(50), GETDATE(), 101);
The set of styles available for the CONVERT function is somewhat limited, and may not be enough for
all situations
...
The
...
DateTime class includes extremely flexible string-formatting capabilities that can be harnessed
using a CLR scalar user-defined function (UDF)
...
Value;
return new SqlString(theDate
...
ToString()));
}
This UDF converts the SqlDateTime instance into an instance of System
...
The method accepts a wide array of
formatting directives, all of which are fully documented in the Microsoft MSDN Library
...
FormatDate(GETDATE(), 'MM yyyy dd');
Keep in mind that the ToString method’s formatting overload is case sensitive
...


Efficiently Querying Date/Time Columns
Knowing how to format dates for input and output is a good first step, but the real goal of any database
system is to allow the user to query the data to answer business questions
...

To start things off, create the following table:
CREATE TABLE VariousDates
(
ADate datetime NOT NULL,
PRIMARY KEY (ADate) WITH (IGNORE_DUP_KEY = ON)
);
GO
Now we’ll insert some data into the table
...
spt_values
WHERE number BETWEEN 1001 AND 1256
)
INSERT INTO VariousDates ( ADate )
SELECT
CASE x
...
number, 2) * b
...
number-1000, '20100201'))
WHEN 2 THEN
DATEADD(millisecond,
b
...
number-1000, '20100213'))
END
FROM Numbers a, Numbers b
CROSS JOIN
(
SELECT 1
UNION ALL
SELECT 2
) x (n);
GO
Once the data has been inserted, the next logical step is of course to query it
...
000
...
A first shot at that query might be
something like the following:
SELECT *
FROM VariousDates
WHERE ADate = '20100213';
GO
If you run this query, you might be surprised to find out that instead of seeing all rows for February
13, 2010, zero rows are returned
...
When this query is evaluated and
the search argument ADate = '20100213' is processed, SQL Server sees that the datetime ADate column
is being compared to the varchar string '20100213'
...
000 is used
...
000
...

There are many potential solutions to this problem
...
Doing so would facilitate easy queries on a
particular date, but would lose the time element associated with each record
...

A better solution is to try to control the conversion from datetime to date in a slightly different way
...
The following query is an example of one such way of doing this:
SELECT *
FROM VariousDates
WHERE CONVERT(varchar(20), ADate, 112) = '20100213';
Running this query, you will find that the correct data is returned; you’ll see all rows from February
13, 2010
...
The table’s index on the
ADate column is based on ADate as it is natively typed—in other words, as datetime
...
As a
result, this query is unable to seek an index, and SQL Server is forced to scan every row of the table,
convert each ADate value to a string, and then compare it to the date string
...
229923
...
Converting the date/time column to a string does not result in a good execution plan
...

Generally speaking, performing a calculation or conversion of a column in a query precludes any
index on that column from being used
...

To demonstrate this unusual but surprisingly useful behavior, we can rewrite the previous query as
follows:
SELECT *

328

CHAPTER 11

WORKING WITH TEMPORAL DATA

FROM VariousDates
WHERE CAST(ADate AS date) = '20100213';
This query performs much better, producing the execution plan shown in Figure 11-2, which has a
clustered index seek with an estimated cost of 0
...
Querying date/time columns CAST to date type allows the query engine to take advantage of
an index seek
...
A query based on this data might look like this:
SELECT *
FROM VariousDates
WHERE ADate BETWEEN '20100213 12:00:00' AND '20100214 00:00:00';
This query, like the last, is able to use an efficient clustered index seek, but it has a problem
...
If there happens to be a row for February 14, 2010 at midnight (and the data in the sample table
does indeed include such a row), that row will be included in the results of both this query and the query
to return data for the following shift
...
Instead, always use the fully expanded version, inclusive of the
start of the interval, and exclusive of the end value:
SELECT *
FROM VariousDates
WHERE
ADate >= '20100213 12:00:00'
AND ADate < '20100214 00:00:00';
This pattern can be used to query any kind of date and time range and is actually quite flexible
...


Date/Time Calculations
The query pattern presented in the previous section to return all rows for a given date works and returns
the correct results, but is rather overly static as-is
...
By using

329

CHAPTER 11

WORKING WITH TEMPORAL DATA

SQL Server’s date calculation functions, input dates can be manipulated in order to dynamically come
up with whatever ranges are necessary for a given query
...
The first returns the difference between two dates; the second adds (or subtracts) time from
an existing date
...

DATEDIFF takes three parameters: the time granularity that should be used to compare the two input
dates, the start date, and the end date
...
Note that I mentioned that this query compares the two dates,
both at midnight, even though neither of the input strings contains a time
...

It’s also important to note that DATEDIFF maintains the idea of “start” and “end” times, and the result
will change if you reverse the two
...

The DATEADD function takes three parameters: the time granularity, the amount of time to add, and
the input date
...
000:

SELECT DATEADD(hour, 24, '20100113');
DATEADD will also accept negative amounts, which will lead to the relevant amount of time being
subtracted rather than added, as in this case
...
As a result, developers came up with a number of methods to “truncate”
datetime values so that, without changing the underlying datatype, they could be interrogated as dates
without consideration of the time component
...

Although, with the introduction of the date datatype, it is no longer necessary to perform such
truncation, the “rounding” approach taken is still very useful as a basis for other temporal queries
...


2
...
For instance, if you want to remove the seconds and milliseconds of
a time value, you’d round down using minutes
...

Once you’ve decided on a level of granularity, pick a reference date/time
...


CHAPTER 11

WORKING WITH TEMPORAL DATA

3
...


4
...
The result will be the
truncated value of the original date/time
...
Assume that you want to start with
2010-04-23 13:45:43
...
The granularity used will be days, since that is the lowest level of granularity above the units
of time (milliseconds, seconds, minutes, and hours)
...
233';
SELECT DATEDIFF(day, '19000101', @InputDate);
Running this T-SQL, we discover that 40289 days passed between the reference date and the input
date
...
000
...
Of course, you don’t have to run this T-SQL step by step; in a real
application, you’d probably combine everything into one inline statement:

SELECT DATEADD(day, DATEDIFF(day, '19000101', @InputDate), '19000101');
Because it is a very common requirement to round down date/time values to different levels of
granularity—to find the first day of the week, the first day of the month, and so on—you might find it
helpful to encapsulate this logic in a reusable function with common named units of time, as follows:
CREATE FUNCTION DateRound (
@Unit varchar(32),
@InputDate datetime
) RETURNS datetime
AS
BEGIN
DECLARE @RefDate datetime = '19000101';
SET @Unit = UPPER(@Unit);
RETURN
CASE(@Unit)
WHEN 'DAY' THEN
DATEADD(day, DATEDIFF(day, @RefDate, @InputDate), @RefDate)
WHEN 'MONTH' THEN
DATEADD(month, DATEDIFF(month, @RefDate, @InputDate), @RefDate)
WHEN 'YEAR' THEN
DATEADD(year, DATEDIFF(year, @RefDate, @InputDate), @RefDate)
WHEN 'WEEK' THEN
DATEADD(week, DATEDIFF(week, @RefDate, @InputDate), @RefDate)
WHEN 'QUARTER' THEN
DATEADD(quarter, DATEDIFF(quarter, @RefDate, @InputDate), @RefDate)

331

CHAPTER 11

WORKING WITH TEMPORAL DATA

END
END;
GO
The following code illustrates how the DateRound() function can be used with a date/time value
representing 08:48 a
...
on August 20, 2009:
SELECT
dbo
...
DateRound('Month', '20090820 08:48'),
dbo
...
DateRound('Week', '20090820 08:48'),
dbo
...
000
2009-08-01 00:00:00
...
000
2009-08-17 00:00:00
...
000

Note Developers who have experience with Oracle databases may be familiar with the Oracle PL/SQL TRUNC()
method, which provides similar functionality to the DateRound function described here
...
Suppose, for example, that you want to find the last day of
the month
...
For instance, you can use a reference date of 190012-31:

SELECT DATEADD(month, DATEDIFF(month, '19001231', @InputDate), '19001231');

332

CHAPTER 11

WORKING WITH TEMPORAL DATA

Note that when using this approach, it is important to choose a month that has 31 days; what this TSQL does is to find the same day of the month as the reference date, on the month in which the input
date lies
...
Had I used February 28 instead
of December 31 for the reference date, the output any time this query was run would be the 28th of the
month
...
For example, a common requirement in
many applications is to perform calculations based on time periods such as “every day between last
Friday and today
...
In this case, to find the nearest Friday to a
supplied input date, the reference date should be any Friday
...

The following T-SQL finds the number of days between the reference Friday, January 7, 2000, and
the input date, February 9, 2009:
DECLARE @Friday date = '20000107';
SELECT DATEDIFF(day, @Friday, '20090209');
The result is 3321, which of course is an integer
...
Currently, the result of the inner
DATEDIFF is divided by 7 to calculate a round number of weeks, and then multiplied by 7 again to
produce the equivalent number of days to add using the DATEADD method
...
If you really want to return the “last” Friday every time, and never the input date itself—even if it is
a Friday—a small modification is required
...
By calculating the number of days elapsed
between this second reference date and the input date, the rounded number of weeks will be one week
lower if the input date is a Friday, meaning that the result will always be the previous Friday
...
To find the “next” one of a given day (e
...
, “next Friday”), simply add one week
to the result of the inner calculation before adding it to the reference date:
DECLARE @InputDate datetime = GETDATE();
DECLARE @Friday
datetime = '2000-01-07';
SELECT DATEADD(week, (DATEDIFF(day, @Friday, @InputDate) / 7) +1, @Friday);
As a final example of what you can do with date/time calculations, a slightly more complex
requirement is necessary
...
The group meets on the second Thursday of each month
...
The earliest date on
which the second Thursday can fall occurs when the first day of the month is a Thursday
...
The latest date on which the second
Thursday can fall occurs when the first of the month is a Friday, in which case the second Thursday will
be the 14th
...
The following T-SQL uses this
approach:
DECLARE @InputDate date = '20100101';
DECLARE @Thursday date = '20000914';
DECLARE @FourteenthOfMonth date =
DATEADD(month, DATEDIFF(month, @Thursday, @InputDate), @Thursday);
SELECT DATEADD(week, (DATEDIFF(day, @Thursday, @FourteenthOfMonth) / 7),
@Thursday);
Of course, this doesn’t find the next meeting; it finds the meeting for the month of the input date
...
Otherwise, the next month’s second Thursday is four weeks away
...
The following T-SQL combines all
of these techniques to find the next date for a New England SQL Server Users Group meeting, given an
input date:
DECLARE @InputDate date = GETDATE();
DECLARE @Thursday date = '20000914';
DECLARE @FourteenthOfMonth date =
DATEADD(month, DATEDIFF(month, @Thursday, @InputDate), @Thursday);
DECLARE @SecondThursday date =
DATEADD(week, (DATEDIFF(day, @Thursday, @FourteenthOfMonth) / 7), @Thursday);

334

CHAPTER 11

WORKING WITH TEMPORAL DATA

SELECT
CASE
WHEN @InputDate <= @SecondThursday
THEN @SecondThursday
ELSE
DATEADD(
week,
CASE
WHEN DATEPART(day, @SecondThursday) <= 10 THEN 5
ELSE 4
END,
@SecondThursday)
END;
Finding complex dates like the second Thursday of a month is not a very common requirement
unless you’re writing a scheduling application
...
” Combining the range techniques discussed in the previous section with the
date/time calculations shown here, it becomes easy to design stored procedures that both efficiently and
dynamically query for required time periods
...
The obvious answer is of course the following:

SELECT DATEDIFF(year, @YourBirthday, GETDATE());
Unfortunately, this answer—depending on the current day—is wrong
...
On March 25, 2010, that person’s 45th birthday should be celebrated
...
Happy New Year and happy birthday combined, thanks to the magic of SQL Server? Probably
not; the discrepancy is due to the way SQL Server calculates date differences
...
This feature
makes the previous date/time truncation examples work, but makes age calculations fail because when
differencing years, days and months are not taken into account
...
The following T-SQL both accomplishes the
primary goal, and as an added bonus, also takes leap years into consideration:
SELECT
DATEDIFF (
YEAR,
@YourBirthday,
GETDATE()) CASE
WHEN 100 * MONTH(GETDATE()) + DAY(GETDATE())

335

CHAPTER 11

WORKING WITH TEMPORAL DATA

< 100 * MONTH(@YourBirthday) + DAY(@YourBirthday) THEN 1
ELSE 0
END;
Note that this T-SQL uses the MONTH and DAY functions, which are shorthand for DATEPART(month,
) and DATEPART(day, ), respectively
...
For the most part, using the date/time calculation and rangematching techniques discussed in the previous section will yield the best possible performance
...
It is quite
likely that more technical business users will request direct access to query key business databases, but
very unlikely that they will be savvy enough with T-SQL to be able to do complex date/time calculations
...
A lookup table can be created that allows users to derive
any number of named periods from the current date with ease
...

The basic calendar table has a date column that acts as the primary key and several columns that
describe time periods
...
A standard example can
be created using the following code listing:
CREATE TABLE Calendar
(
DateKey date PRIMARY KEY,
DayOfWeek tinyint,
DayName nvarchar(10),
DayOfMonth tinyint,
DayOfYear smallint,
WeekOfYear tinyint,
MonthNumber tinyint,
MonthName nvarchar(10),
Quarter tinyint,
Year smallint
);
GO
SET NOCOUNT ON;
DECLARE @Date date = '19900101';
WHILE @Date < '20250101'
BEGIN
INSERT INTO Calendar
SELECT
@Date AS DateKey,
DATEPART(dw, @Date) AS DayOfWeek,
DATENAME(dw, @Date) AS DayName,
DATEPART(dd, @Date) AS DayOfMonth,

336

CHAPTER 11

DATEPART(dy, @Date) AS
DATEPART(ww, @Date) as
DATEPART(mm, @Date) AS
DATENAME(mm, @Date) AS
DATEPART(qq, @Date) AS
YEAR(@Date) AS Year;

WORKING WITH TEMPORAL DATA

DayOfYear,
WeekOfYear,
MonthNumber,
MonthName,
Quarter,

SET @Date = DATEADD(d, 1, @Date);
END
GO
This table creates one row for every date between January 1, 1990 and January 1, 2025
...
Although
this sounds like it will potentially produce a lot of rows, keep in mind that every ten years worth of data
will only require around 3,652 rows
...

The columns defined in the Calendar table represent the periods of time that users will want to find
and work with
...
You might, for example, want to add
columns to record fiscal years, week start and end dates, or holidays
...

Once the calendar table has been created, it can be used for many of the same calculations covered
in the last section, as well as for many other uses
...
DateKey = CAST(GETDATE() AS date);
Once you’ve identified “today,” it’s simple to find other days
...
DateKey < GETDATE()
AND LastFriday
...
If you select a different first day of the week, you’ll have to
change the DayOfWeek value specified
...
The DayName
column was populated using the DATENAME function, which returns a localized character string
representing the day name (i
...
, “Friday,” in English)
...

Since the calendar table contains columns that define various periods, such as the current year and
the week of the year, it becomes easy to answer questions such as “What happened this week?” To find
the first and last days of “this week,” the following query can be used:
SELECT
MIN(ThisWeek
...
DateKey) AS LastDayOfWeek
FROM Calendar AS Today
JOIN Calendar AS ThisWeek ON
ThisWeek
...
Year
AND ThisWeek
...
WeekOfYear
WHERE
Today
...
For instance, you may wish to identify “Friday of
last week
...
*
FROM Calendar AS Today
JOIN Calendar AS FridayLastWeek ON
Today
...
Year
AND Today
...
WeekOfYear
WHERE
Today
...
DayName = 'Friday';
Unfortunately, this code has an edge problem that will cause it to be somewhat nonfunctional
around the first of the year in certain cases
...
The query also joins on the Year column, making the
situation doubly complex
...
A good alternative solution is to add a WeekNumber column
that numbers every week consecutively for the entire duration represented by the calendar
...
DateKey AS StartDate,
(
SELECT TOP(1)
EndOfWeek
...
DateKey >= StartOfWeek
...
DateKey
) AS EndDate,
ROW_NUMBER() OVER (ORDER BY StartOfWeek
...
The StartOfWeek CTE selects each day from the
calendar table where the day of the week is 1, in addition to the earliest date in the table, in case that day
is not the first day of a week
...
The SELECT list includes the DateKey represented for
each row of the StartOfWeek CTE, the lowest DateKey value from the EndOfWeek CTE that’s greater than
the StartOfWeek value (which is the end of the week), and a week number generated using the
ROW_NUMBER function
...

Once this T-SQL has been run, the calendar table’s new column can be populated (and set to be
nonnullable), using the following code:
UPDATE Calendar
SET WeekNumber =
(
SELECT WN
...
DateKey BETWEEN WN
...
EndDate
);
ALTER TABLE Calendar
ALTER COLUMN WeekNumber int NOT NULL;
Now, using the new WeekNumber column, finding “Friday of last week” becomes almost trivially
simple:
SELECT FridayLastWeek
...
WeekNumber = FridayLastWeek
...
DateKey = CAST(GETDATE() AS date)
AND FridayLastWeek
...
There are a couple of ways
that a calendar table can be used to address this dilemma
...
The following T-SQL is one way of doing so:
WITH NextTwoMonths AS

339

CHAPTER 11

WORKING WITH TEMPORAL DATA

(
SELECT
Year,
MonthNumber
FROM Calendar
WHERE
DateKey IN (
CAST(GETDATE() AS date),
DATEADD(month, 1, CAST(GETDATE() AS date)))
),
NumberedThursdays AS
(
SELECT
Thursdays
...
MonthNumber ORDER BY DateKey)
AS ThursdayNumber
FROM Calendar Thursdays
JOIN NextTwoMonths ON
NextTwoMonths
...
Year
AND NextTwoMonths
...
MonthNumber
WHERE
Thursdays
...
*
FROM NumberedThursdays
WHERE
NumberedThursdays
...
ThursdayNumber = 2
ORDER BY NumberedThursdays
...
Then, in the NumberedThursdays CTE, every Thursday for those two months is
identified and numbered sequentially
...

Luckily, such complex T-SQL can often be made obsolete using calendar tables
...
There is, of
course, no reason that you can’t add your own columns to create named periods specific to your
business requirements
...

A much more common requirement is figuring out which days are business days
...
Although you could simply count out the weekend days, this would fail to take
into account national holidays, state and local holidays that your business might observe, and company
retreat days or other days off that might be specific to your firm
...

If you do not need to record a full description associated with each holiday, then you could instead
populate the column with a set of simple flag values representing different types of holidays
...
DateKey = CAST(GETDATE() as date)
AND Today
...
Year
AND Today
...
MonthNumber
);
This query counts the number of days in the current month that are not flagged as holidays
...

If your business is seasonally affected, try adding a column that helps you identify various seasonal
time periods, such as “early spring,” “midsummer,” or “the holiday season” to help with analytical
queries based on these time periods
...

Using calendar tables can make time period–oriented queries easier to perform, but remember that
they require ongoing maintenance
...
You may want to add an additional year of days on the first of
each year in order to maintain a constant ten-year buffer
...
Language barriers aside, one of the most important issues
arises from the problems of time variance
...

In 1884, 24 standard time zones were defined at a meeting of delegates in Washington, DC, for the
International Meridian Conference
...
This central time
zone is referred to either as GMT (Greenwich Mean Time) or UTC (Universel Temps Coordonné, French
for “Coordinated Universal Time”)
...


341

CHAPTER 11

WORKING WITH TEMPORAL DATA

Figure 11-3
...

As I write these words, it’s just after 8:15 a
...
in England, but since we’re currently observing British
Summer Time (BST), this time represents UTC + 1 hour
...

The Eastern United States, for example, is normally UTC – 5, but right now is actually UTC – 4, making it
3:15 a
...


Note Different countries switch into and back from daylight savings time at different times: for example, the
time difference between the United Kingdom and mainland Chile can be three, four, or five hours, depending on
the time of year
...
m
...
Unfortunately, not all of the countries in the world use the
standard zones
...
m
...
5
...

There are three central issues to worry about when writing time zone–specific software:
When a user sees data presented by the application, any dates should be rendered
in the user’s local time zone (if known), unless otherwise noted, in which case data
should generally be rendered in UTC to avoid confusion
...
All date/time data
in the database should be standardized to a specific zone so that, based on known
offsets, it can be easily converted to users’ local times
...
There are various ways of modeling
such data, as I’ll discuss shortly
...
If you will need to report or query
based on local times in which events occurred, consider persisting them as-is in
addition to storing the times standardized to UTC
...
Consider a user in New York
asking the question, “What happened between 2:00 p
...
and 5:00 p
...
today?” If
date/time data in the database is all based in the UTC zone, it’s unclear whether
the user is referring to 2:00 p
...
to 5:00 p
...
EST or UTC—very different questions!
The actual requirements here will vary based on business requirements, but it is a
good idea to put a note on any screen in which this may be an issue to remind the
user of what’s going on
...
It’s a good idea to handle as much of the work as possible in the application layer, but
some (or sometimes all) of the responsibility will naturally spill into the data layer
...


Storing UTC Time
The basic technique for storing temporal data for global applications is to maintain time zone settings
for each user of the system so that when they log in, you can find out what zone you should treat their
data as native to
...

This approach requires some changes to the database code
...
However, this rule only applies unconditionally for inserts; if you’re
converting a database from local time to UTC, a blind find/replace-style conversion from GETDATE to
GETUTCDATE may not yield the expected results
...
SalesOrderHeader table:
CREATE PROCEDURE GetTodaysOrders
AS
BEGIN
SET NOCOUNT ON
SELECT
OrderDate,

343

CHAPTER 11

WORKING WITH TEMPORAL DATA

SalesOrderNumber,
AccountNumber,
TotalDue
FROM Sales
...
SalesOrderHeader table contains date/time values defined in UTC, it might
seem like changing the GETDATE calls in this code to GETUTCDATE is a natural follow-up move
...
Although for the most part CAST(GETDATE() AS date) will return the same as
CAST (GETUTCDATE() AS date), there are 4 hours of each day (or sometimes 5, depending on daylight
savings settings) in which the date as measured using UTC will be one day ahead of the date measured
according to EST
...
m
...
”
The time portion will be truncated, and the query won’t return any of “today’s” data at all—at least not if
you’re expecting things to work using EST rules
...
After it is converted to local time, then truncate the time portion
...
Depending on
whether or not you’ve handled it in your application code, a further modification might be required to
convert the OrderDate column in the SELECT list, in order to return the data in the user’s local time zone
...


Using the datetimeoffset Type
The new datetimeoffset datatype is the most fully featured temporal datatype in SQL Server 2008
...
Thus, a single value can
contain all the information required to express both a local time and the corresponding UTC time
...
Calculations on datetimeoffset values take account of both the time component and the offset
...
In other
words, midday in Moscow occurred 3 hours before midday in London
...
Consider the following:
CREATE TABLE TimeAndPlace (
Place varchar(32),

344

CHAPTER 11

WORKING WITH TEMPORAL DATA

Time datetimeoffset
);
GO
INSERT INTO TimeAndPlace (Place, Time) VALUES
('London', '2009-07-15 08:15:00 +0:00'),
('Paris', '2009-07-15 08:30:00 +1:00'),
('Zurich', '2009-07-15 09:05:00 +2:00'),
('Dubai', '2009-07-15 10:30:00 +3:00');
GO
To find out which event took place first, we can use a simple ORDER BY statement in a SELECT query—
the output will order the results taking account of their offset:
SELECT
Place,
Time
FROM TimeAndPlace
WHERE Time BETWEEN '20090715 07:30' AND '20090715 08:30'
ORDER BY Time ASC;
The results are as follows:
Place

Time

Paris

2009-07-15 08:30:00
...
0000000 +03:00

London

2009-07-15 08:15:00
...
In UTC terms, the Zurich time corresponds to 7:05 a
...
which lies outside of the range of
the query condition and so is not included in the results
...
m
...
m
...
The application still needs to tell the database the correct offset for the time zone in
which the time is defined
...
For example, my operating system reports my current time zone as “(GMT)
Greenwich Mean Time : Dublin, Edinburgh, Lisbon, London
...
Such information can be extracted from a
lookup table based on the system registry, and newer operating systems recognize and correctly allow
for daylight savings time, adjusting the system clock automatically when required
...
NET application, you can use TimeZoneInfo
...
Id to retrieve the ID
of the user’s local time zone, and then translate this to a TimeZoneInfo object using the
TimeZoneInfo
...
Each TimeZoneInfo has an associated offset from UTC
that can be accessed via the BaseUtcOffset property to get the correct offset for the corresponding
datetimeoffset instance in the database
...
For example, suppose that you
wanted to know the time in London, 24 hours after a given time, at 1:30 a
...
on the March 27, 2010:
DECLARE @LondonTime datetimeoffset = '20100327 01:30:00 +0:00';
SELECT DATEADD(hour, 24, @LondonTime);
This code will give the result 2010-03-28 01:30:00
...
However, at
1:00 a
...
on Sunday, March 28, clocks in Britain are put forward 1 hour to 2:00 a
...
to account for the
change from GMT to BST
...
0000000 +01:00
...

What’s more, the offset corresponding to a given time zone does not remain static
...
5 hours behind UTC
...

Time zone issues can become quite complex, but they can be solved by carefully evaluating the
necessary changes to the code and even more carefully testing once changes have been implemented
...
Once inserted, there is no way to ask the database whether a time was
supposed to be in UTC or a local time zone
...
Time is continuous, and any given state change
normally has both a clearly defined start time and end time
...
” But you really didn’t drive only at 10:00, unless you happen to be in
possession of some futuristic time/space-folding technology (and that’s clearly beyond the scope of this
chapter)
...
A column called OrderDate is an almost ubiquitous feature in databases that handle
orders; but this column only stores the date/time that the order ended—when the user submitted the
final request
...
Likewise, every time we check our e-mail, we see a Sent Date field,
which captures the moment that the sender hit the send button, but does not help identify how long that
person spent thinking about or typing the e-mail, activities that constitute part of the “sending” process
...
For most
sites, it really doesn’t matter for the purpose of order fulfillment how long the user spent browsing
(although that information may be useful to interface designers, or when considering the overall
customer experience)
...
The important thing is, it was sent (and later received, another data point that many e-mail
clients don’t expose)
...
Take for instance your employment history
...
Failing to include both the start and end dates with this data can create some
interesting challenges
...
Each row in this case would represent a subinterval during which some status change
occurred
...
Start with the following table and
rows:
CREATE TABLE JobHistory
(
Company varchar(100),
Title varchar(100),
Pay decimal(9, 2),
StartDate date
);
GO
INSERT INTO JobHistory
(
Company,
Title,
Pay,
StartDate
) VALUES
('Acme Corp', 'Programmer', 50000
...
00, '20001005'),
('Better Place', 'Junior DBA', 82000
...
00, '20071114');
GO
Notice that each of the dates uses the date type
...
m
...
m
...
What
matters is that the date in the table is the start date
...
The end date of
the final job, it can be assumed, is the present date (or, if you prefer, NULL)
...
*,
COALESCE((
SELECT MIN(J2
...
StartDate > J1
...
00

1997-06-26

2000-10-05

Software Shop Programmer/Analyst

62000
...
00

2003-01-08

2007-11-14

Enterprise

Database Developer

95000
...
If no such start date exists, the current date is used
...
This table may, for instance, hide the
fact that the subject was laid off from Software Shop in July 2002
...

Despite the lack of support for gaps, let’s try a bit more data and see what happens
...
00, '19980901'),
('Acme Corp', 'Programmer 2', 58000
...
00, '20000901'),
('Software Shop', 'Programmer/Analyst', 62000
...
00, '20020101'),
('Software Shop', 'Programmer', 40000
...
00, '20040601'),
('Better Place', 'DBA', 87000
...
A few raises and title adjustments—including one title adjustment with no
associated pay raise—and an unfortunate demotion along with a downsized salary, just before getting
laid off in 2002 (the gap which, as mentioned, is not able to be represented here)
...
The Subject’s Full Job History, with Salary and Title Adjustments

Company

Title

Pay

StartDate

Acme Corp

Programmer

50000
...
00

1998-09-01

Acme Corp

Programmer 2

58000
...
00

2000-09-01

Software Shop

Programmer/Analyst

62000
...
00

2000-10-05

Software Shop

Programmer/Analyst

67000
...
00

2002-03-01

Better Place

Junior DBA

82000
...
00

2004-06-01

Better Place

DBA

87000
...
00

2007-11-14

Ignoring the gap, let’s see how one might answer a resume-style question using this data
...

The first step commonly taken in tackling this kind of challenge is to use a correlated subquery to
find the rows that have the maximum value per group
...
Pay =
(
SELECT MAX(Pay)
FROM JobHistory AS J3
WHERE J3
...
Company
);

349

CHAPTER 11

WORKING WITH TEMPORAL DATA

One key modification that must be made is to the basic query that finds start and end dates
...
The following T-SQL finds the correct start and end dates for each company:
SELECT
J1
...
StartDate) AS StartDate,
COALESCE((
SELECT MIN(J2
...
Company <> J1
...
StartDate > MIN(J1
...
Company
ORDER BY StartDate;
A quick note: This query would not work properly if the person had been hired back by the same
company after a period of absence during which he was working for another firm
...
Company,
J1
...
StartDate)
FROM JobHistory AS J2
WHERE
J2
...
Company
AND J2
...
StartDate),
CAST(GETDATE() AS date)
) AS EndDate
FROM JobHistory AS J1
WHERE
J1
...
Company
FROM JobHistory J3
WHERE J3
...
StartDate
ORDER BY J3
...
Company,
J1
...
StartDate;

350

CHAPTER 11

WORKING WITH TEMPORAL DATA

This example complicates things a bit too much for the sake of this chapter, but I feel that it is
important to point this technique out in case you find it necessary to write these kinds of queries in
production applications
...

Getting back to the primary task at hand, showing the employment history along with peak salaries
and job titles, the next step is to merge the query that finds the correct start and end dates with the query
that finds the maximum salary and associated title
...
The following T-SQL shows how to accomplish this:
SELECT
x
...
StartDate,
x
...
Pay,
p
...
Company,
MIN(J1
...
StartDate)
FROM JobHistory AS J2
WHERE
J2
...
Company
AND J2
...
StartDate)),
CAST(GETDATE() AS date)
) AS EndDate
FROM JobHistory AS J1
GROUP BY J1
...
StartDate >= x
...
StartDate < x
...
Pay =
(
SELECT MAX(Pay)
FROM JobHistory AS J3
WHERE J3
...
Company
)
) p
ORDER BY x
...
The

351

CHAPTER 11

WORKING WITH TEMPORAL DATA

StartDate/EndDate pair for each period of employment is a half-open interval (or semiopen, depending
on which mathematics textbook you’re referring to); the StartDate end of the interval is closed (inclusive
of the endpoint), and the EndDate is open (exclusive)
...
The results of this query
are as follows:
Company

StartDate

EndDate

Pay

Title

Acme Corp

1997-06-26

2000-10-05

58000
...
00

Programmer 3

Software Shop 2000-10-05

2003-01-08

67000
...
00

DBA

Enterprise

2007-11-14

2009-07-12

95000
...
The solution is to select the
appropriate row by sorting the result by the Pay column, in descending order
...
StartDate >= x
...
StartDate < x
...

In terms of query style, the main thing to notice is that in order to logically manipulate this data,
some form of an “end” for the interval must be synthesized within the query
...
This will make querying much more straightforward
...
I’ve already mentioned the issue with
gaps in the sequence, which are impossible to represent in this single-column table
...
What if the subject took on some after-hours contract work during the same time
period as one of the jobs? Trying to insert that data into the table would make it look as though the
subject had switched companies
...
There are many situations in which
gaps and overlaps may not make sense, and the extra bytes needed for a second column would be a

352

CHAPTER 11

WORKING WITH TEMPORAL DATA

waste
...
Systems are often used by IT departments that ping
each monitored server on a regular basis and record changes to their status
...
000'),
('DBServer', 'Available', '2009-04-20T03:00:00
...
100'),
('DBServer', 'Available', '2009-06-12T14:38:52
...
593'),
('WebServer', 'Available', '2009-06-15T09:28:17
...
ServerName,
S1
...
StatusTime)
FROM ServerStatus AS S2
WHERE
S2
...
StatusTime),
GETDATE()
) AS EndTime
FROM ServerStatus AS S1
WHERE S1
...
100

2009-06-12 14:38:52
...
593

2009-06-15 09:28:17
...
The
monitoring system might insert additional “unavailable” rows every 30 seconds or minute until the

353

CHAPTER 11

WORKING WITH TEMPORAL DATA

target system starts responding again
...
To get around this problem, the query could be modified as follows:
SELECT
S1
...
StatusTime) AS StartTime,
p
...
StatusTime)
FROM ServerStatus AS S2
WHERE
S2
...
StatusTime
AND S2
...
Status = 'Unavailable'
GROUP BY
S1
...
EndTime;
This new version finds the first “available” row that occurs after the current “unavailable” row; that
row represents the actual end time for the full interval during which the server was down
...


Modeling and Querying Independent Intervals
In many cases, it is more appropriate to model intervals as a start time/end time combination rather than
using a single column as used in the previous section
...
Therefore, both gaps and overlaps can be
represented
...

Going back to the employment example, assume that a system is required for a company to track
internal employment histories
...
Ignore the obvious need for a
table of names and titles to avoid duplication of that data—that would overcomplicate the example
...
The
primary issue is that although I did include one CHECK constraint to make sure that the EndDate is after
the StartDate (we hope that the office isn’t so bad that people are quitting on their first day), I failed to
include a primary key
...
Employee alone is
not sufficient, as employees would not be able to get new titles during the course of their employment
(or at least it would no longer be a “history” of those changes)
...
What if an employee leaves the company for a while, and later
comes to his senses and begs to be rehired with the same title? The good thing about the table structure
is that such a gap can be represented; but constraining on both the Employee and Title columns would
prevent that situation from being allowed
...
An employee cannot be in two places (or offices) at the same time, and the combination of
the three columns would allow the same employee to start on the same day with two different titles
...

As it turns out, what we really need to constrain in the primary key is an employee starting on a
certain day; uniqueness of the employee’s particular title is not important in that regard
...
In SQL Server, unique constraints allow for one NULL-valued row—so only one
NULL EndDate would be allowed per employee
...
Again, this is probably not what
was intended
...
Therefore, I will return to
this topic in the next section, which covers overlapping intervals
...
Ignore for a moment the lack of
proper constraints, and consider the following rows (which would be valid even with the constraints):
INSERT INTO EmploymentHistory
(
Employee,
Title,
StartDate,
EndDate
) VALUES
('Jones', 'Developer', '20070105', '20070901'),
('Jones', 'Senior Developer', '20070901', '20080901'),
('Jones', 'Principal Developer', '20080901', '20081007'),
('Jones', 'Principal Developer', '20090206', NULL);
The scenario shown here is an employee named Jones, who started as a developer in January 2007
and was promoted to Senior Developer later in the year
...
However, a few months after that he decided to rejoin the
company and has not yet left or been promoted again
...
Real-world scenarios include such
requirements as tracking of service-level agreements for server uptime and managing worker shift
schedules—and of course, employment history
...
The following T-SQL accomplishes that goal:
SELECT
theStart
...
Employee = 'Jones'
AND NOT EXISTS
(
SELECT *
FROM EmploymentHistory Previous
WHERE
Previous
...
StartDate
AND theStart
...
Employee
);

356

CHAPTER 11

WORKING WITH TEMPORAL DATA

This query finds all rows for Jones (remember, there could be rows for other employees in the table),
and then filters them down to rows where there is no end date for a Jones subinterval that matches the
start date of the row
...

The next step is to find the ends of the covering intervals
...

To match the end rows to the start rows, find the first end row that occurs after a given start row
...
StartDate,
(
SELECT
MIN(EndDate)
FROM EmploymentHistory theEnd
WHERE
theEnd
...
StartDate
AND theEnd
...
Employee
AND NOT EXISTS
(
SELECT *
FROM EmploymentHistory After
WHERE
After
...
EndDate
AND After
...
Employee
)
) AS EndDate
FROM EmploymentHistory theStart
WHERE
theStart
...
EndDate = theStart
...
Employee = Previous
...
e
...
First, find the
end date of every subinterval using the same syntax used to find end dates in the covered intervals
query
...
Make sure to filter out rows where
the EndDate is NULL—these subintervals have not yet ended, so it does not make sense to include them as
holes
...
The following T-SQL demonstrates this approach to find noncovered intervals:
SELECT
theStart
...
StartDate)
FROM EmploymentHistory theEnd

357

CHAPTER 11

WORKING WITH TEMPORAL DATA

WHERE
theEnd
...
EndDate
AND theEnd
...
Employee
) AS EndDate
FROM EmploymentHistory theStart
WHERE
theStart
...
EndDate IS NOT NULL
AND NOT EXISTS
(
SELECT *
FROM EmploymentHistory After
WHERE After
...
EndDate
);

Overlapping Intervals
The final benefit (or drawback, depending on what’s being modeled) of using both a start and end date
for intervals that I’d like to discuss is the ability to work with overlapping intervals
...

To begin with, a bit of background on overlaps is necessary
...
Interval A is overlapped by each of the other intervals B through E, as follows:
•

Interval B starts within interval A and ends after interval A
...


•

Interval D both starts and ends within interval A
...


Figure 11-4
...
StartDate
C
...
StartDate
E
...
StartDate AND B
...
EndDate AND B
...
EndDate
< A
...
EndDate > A
...
EndDate <= A
...
StartDate AND D
...
EndDate
< A
...
EndDate > A
...
Let us first consider the situations in which an interval X does not overlap interval
A
...
StartDate > A
...
EndDate < A
...
StartDate < A
...
EndDate > A
...
” This is illustrated in Figure 11-5
...
If X starts before A ends and X ends after A starts, the two intervals overlap
...
A single employee cannot have two titles
simultaneously, and the only way to ensure that does not happen is to make sure each employee’s
subintervals are unique
...
The following query finds all
overlapping rows for Jones in the EmploymentHistory table, using the final overlap expression:
SELECT *
FROM EmploymentHistory E1
JOIN EmploymentHistory E2 ON
E1
...
Employee
AND (
E1
...
EndDate, '99991231')
AND COALESCE(E1
...
StartDate)
AND E1
...
StartDate
WHERE
E1
...
StartDate <>
E2
...
Thanks to the primary key on the Employee and StartDate columns,
we know that no two rows can share the same StartDate, so this does not affect the overlap logic
...
This avoids the possibility of inserting an interval starting in the
future, while a current interval is still active
...
Since this logic can’t
go into a constraint, there is only one possibility—a trigger
...
The
following T-SQL shows the trigger:
CREATE TRIGGER No_Overlaps
ON EmploymentHistory
FOR UPDATE, INSERT
AS
BEGIN
IF EXISTS
(
SELECT *
FROM inserted i
JOIN EmploymentHistory E2 ON
i
...
Employee
AND (
i
...
EndDate, '99991231')
AND COALESCE(i
...
StartDate)
AND i
...
StartDate
)
BEGIN
RAISERROR('Overlapping interval inserted!', 16, 1);
ROLLBACK;
END
END;
GO
The final examples for this section deal with a common scenario in which you might want to
investigate overlapping intervals: when monitoring performance of concurrent processes in a database
scenario
...
Uncheck all of the events except for SQL:BatchCompleted and leave the default columns selected
...
Enter the following query:

ostress -Q"SELECT * FROM sys
...
The run should take
approximately 1 minute and will produce 10,000 Profiler events—one per invocation of the query
...

Profiler trace tables include two StartTime and EndTime columns, both of which are populated for
many of the events—including SQL:BatchCompleted and RPC:Completed
...

The first and most basic query is to find out which time intervals represented in the table had the
most overlaps
...
The following T-SQL does this using the previously discussed overlap
algorithm:
SELECT
O1
...
EndTime,
COUNT(*)
FROM Overlap_Trace O1
JOIN Overlap_Trace O2 ON
(O1
...
EndTime AND O1
...
StartTime)
AND O1
...
SPID
GROUP BY
O1
...
EndTime
ORDER BY COUNT(*) DESC;
Much like the EmploymentTable example, we need to make sure that no false positives are generated
by rows overlapping with themselves
...

Running this query on an unindexed table is a painful experience
...
Creating the following index on the table
helped a small amount:
CREATE NONCLUSTERED INDEX IX_StartEnd
ON Overlap_Trace (StartTime, EndTime, SPID)
However, I noticed that the index was still not being effectively used; examining the query plan
revealed an outer table scan with a nested loop for an inner table scan—one table scan for every row of
the table
...
The first algorithm returns overlaps of intervals B and D, whereas the second
algorithm returns overlaps of intervals C and E
...
The solution to the performance issue is to merge the two
algorithms, not into a single expression, but rather using UNION ALL, as follows:
SELECT
x
...
EndTime,
SUM(x
...
StartTime,
O1
...
StartTime >= O2
...
StartTime < O2
...
SPID <> O2
...
StartTime,
O1
...
StartTime,
O1
...
StartTime < O2
...
EndTime > O2
...
SPID <> O2
...
StartTime,
O1
...
StartTime,
x
...
theCount) DESC
OPTION(HASH GROUP);
This query is logically identical to the previous one
...
Note that I was forced to add the HASH GROUP option to the end of the query to make the query
optimizer make better use of the index
...


Time Slicing
Another way to slice and dice overlapping intervals is by splitting the data into separate periods and
looking at the activity that occurred during each
...

Although it’s easy to answer those kinds of questions for dates by using a calendar table, it’s a bit
trickier when you need to do it with times
...
Instead, I recommend
dynamically generating time tables as you need them
...

As an example of using the function, consider the following call:
SELECT *
FROM dbo
...
003', '2010-01-02T12:34:48
...

StartDate

EndDate

2010-01-02 12:34:45
...
000

2010-01-02 12:34:46
...
000

2010-01-02 12:34:47
...
000

2010-01-02 12:34:48
...
100

To use the TimeSlice function to look at the number of overlapping queries over the course of the
sample trace, first find the start and endpoints of the trace using the MIN and MAX aggregates
...
The following T-SQL shows how to do that:
SELECT
Slices
...
TimeSlice(StartEnd
...
EndTime)
) Slices;

364

CHAPTER 11

WORKING WITH TEMPORAL DATA

The output of the TimeSlice function can then be used to find the number of overlapping queries
that were running during each period, by using the CROSS APPLY operator again in conjunction with the
interval overlap expression:
SELECT
Slices
...
thecount
FROM
(
SELECT MIN(StartTime), MAX(EndTime)
FROM Overlap_Trace
) StartEnd (StartTime, EndTime)
CROSS APPLY
(
SELECT *
FROM dbo
...
StartTime, StartEnd
...
StartDate < OT
...
EndDate > OT
...
This can be especially useful
for tracking sudden increases in blocking, which often will not correspond to increased utilization of any
system resources, which can make them difficult to identify
...


Modeling Durations
Durations are very similar to intervals, in that they represent a start time and an end time
...
However, in some cases, you may wish to store
durations using a greater precision than the 100ns resolution offered by SQL Server’s datetime2 type
...

There are several examples of cases when you might want to model durations rather than intervals
...
Another example is data that may not require a date/time component at all
...
The moment at which the run took place does not matter; the only important fact is how long the
runner took to travel the 300 yards
...
SQL Server will round the duration down to the nearest 100ns—the
lowest time resolution supported by the datetime2 type
...

What this table does not address is the issue of formatting, should you need to output precise data
rendered as a human-readable string
...
As such, you will have to roll your own code to do so
...
However, if you do need to format data in the database tier (and
you have a very good reason to do so), the best approach to handle this scenario would be to create a
SQLCLR UDF that uses the properties of
...

The following UDF can be used to return a duration measured in nanoseconds in the string format
HH:MM:SS
...
SqlServer
...
SqlFunction]
public static SqlString FormatDuration(SqlInt64 TimeInNanoseconds)
{
// Ticks = Nanoseconds / 10
long ticks = TimeInNanoseconds
...
Hours
...
Minutes
...
Seconds
...
"
+ (TimeInNanoseconds % 1000000000)
);
}
This function could easily be amended to return whatever time format is required for your
particular application
...
Sometimes we’re forced to work with incomplete
or incorrect data, and correct things later as a more complete picture of reality becomes available
...

But in systems that require advanced logging and reproducibility of reports between runs for auditing
purposes, a straightforward UPDATE, INSERT, or DELETE may be counterproductive
...

As an alternative to performing a simple alteration of invalid data, some systems use the idea of
offset transactions
...
For example, assume that part of a financial reporting system has a table that describes
customer transactions
...
However, due to a teller’s key
error that was not caught in time, by the time the reporting data was loaded, the amount that made it
into the system was $5,000:
INSERT INTO Transactions VALUES
(1001, 'Smith', '2009-06-12', 'DEPOSIT', 5000
...
Updating the transaction row itself would
destroy the audit trail, so an offset transaction must be issued
...
The first method is to issue an offset transaction dated the same as the incorrect transaction:
INSERT INTO Transactions VALUES
(1001, 'Smith', '2009-06-12', 'OFFSET', -4500
...
Properly dating the offset record is imperative for data auditing purposes:
INSERT INTO Transactions VALUES
(1001, 'Smith', '2009-06-13', 'OFFSET', -4500
...
After properly
dating the offset, a query of the data for customer Smith for all business done through June 12 does not
include the correction
...
And
although a correlated query could be written to return the correct summary report for June 12, the data
is in a somewhat strange state when querying for ranges after June 12 (e
...
, June 13 through 15
...

To get around these and similar issues, a bitemporal model is necessary
...
The following modified version of the
Transactions table shows the new column:
CREATE TABLE Transactions
(

367

CHAPTER 11

WORKING WITH TEMPORAL DATA

TransactionId int,
Customer varchar(50),
TransactionDate datetime,
TransactionType varchar(50),
TransactionAmount decimal(9,2),
ValidDate datetime
);
When inserting the data for Smith on June 12, a valid date of June 12 is also applied:
INSERT INTO Transactions VALUES
(1001, 'Smith', '2009-06-12', 'DEPOSIT', 5000
...
00
...
Instead, a
corrected deposit record is inserted, with a new valid date:
INSERT INTO Transactions VALUES
(1001, 'Smith', '2009-06-12', 'DEPOSIT', 500
...
But the important
difference is that the transaction still maintains its correct date—so running a report for transactions
that occurred on June 13 would not return any rows, since the only rows we are looking at occurred on
June 12 (even though one of them was entered on June 13)
...
Rather than use an offset, queries should always find the last update for any given
transaction within the valid range
...
The person running the report wants the most “correct” version of the data—that
is, all available corrections should be applied
...
TransactionId,
T1
...
TransactionType,
T1
...
TransactionDate = '2009-06-12'
AND T1
...
TransactionId = T1
...
For instance, assume that this same report was run on the evening of June 12
...
00 for transaction 1001
...
TransactionId,
T1
...
TransactionType,
T1
...
TransactionDate = '2009-06-12'
AND T1
...
TransactionId = T1
...
ValidDate = '2009-06-12'
...

Using this same pattern, data can also be booked in the future, before it is actually valid
...
By
working with the valid date, Smith can make a request for an outgoing transfer on June 14, but ask that
the transfer actually take place on June 16:
INSERT INTO Transactions VALUES
(1002, 'Smith', '2009-06-16', 'TRANSFER', -1000
...
But a business manager can query on June 15 to find out which
transactions will hit in the coming days or weeks, and audit when the data entered the system
...
This can be tremendously useful in many scenarios—especially in the
realm of financial reconciliation when you can be forced to deal with backdated contracts that change
the terms of previous transactions and business booked in the future to avoid certain types of auditing
issues
...
For example, the system may have a policy whereby
transactions are said to be closed after 90 days
...
Another example would be data that has
been used to generate an official report, such as for a government agency
...
In that case, a
trigger would be needed in order to verify the date against a table containing the report run history
...
Managing temporal data successfully begins with an
understanding of the different types of temporal data: instance-based, interval-based, period-based, and
bitemporal
...


370

C H A P T E R 12

Trees, Hierarchies, and Graphs
Although at times it may seem chaotic, the world around us is filled with structure and order
...
One of the natural
hierarchies here on earth is the food chain that exists in the wild; a lion can certainly eat a zebra, but
alas, a zebra will probably never dine on lion flesh
...
but more on that later!
We strive to describe our existence based on connections between entities—or lack thereof—and
that’s what trees, hierarchies, and graphs help us do at the mathematical and data levels
...
However, sometimes the database hierarchy
needs to be designed at a more granular level, representing the hierarchical relationship between
records contained within a single table
...
Rather, you’d put all of the
employees into a single table and create references between the rows
...
I will describe each
technique individually and compare how it can be used to query and manage your hierarchical data
...
A graph is defined as a
set of nodes (or vertices) connected by edges
...
If all of the edges in a graph are directed, the graph itself is said to be directed (sometimes
referred to as a digraph)
...
A graph without cycles is called an acyclic graph
...


371

CHAPTER 12

TREES, HIERARCHIES, AND GRAPHS

Figure 12-1
...
Each intersection can be
thought of as a node, and each street an edge
...
Therefore, a street system can be said to be a cyclic, directed
graph
...
And in software development,
we typically work with class and object graphs, which form the relationships between the component
parts of an object-oriented system
...
Figure 12-2 shows a simple tree
...
Exactly one path exists between any two nodes in a tree
...


A hierarchy is a special subset of a tree, and it is probably the most common graph structure that
developers need to work with
...
This
means that a certain node is designated as the root, and all other nodes are said to be subordinates (or
descendants) of that node
...
Multiple parents are not allowed, nor are multiple root nodes
...
Figure 12-3 shows a
hierarchy containing a root node and several levels of subordinates
...
A hierarchy must have exactly one root node, and each nonroot node must have exactly one
parent
...
Another important term is siblings, which describes nodes that share the same
parent
...


The Basics: Adjacency Lists and Graphs
The most common graph data model is called an adjacency list
...
This is an extremely flexible way of modeling a
graph; any kind of graph, hierarchy, or tree can fit into this model
...
In this section, I will show you
how to work with adjacency lists and point out some of the issues that you should be wary of when
designing solutions around them
...
Note that X and Y are assumed to be references to some valid table of
nodes
...
It can also be used to reference
unconnected nodes; a node with a path back to itself but no other paths can be inserted into the table for
that purpose
...
The net effect is the same, but in my opinion the nullable Y column
makes some queries a bit messier, as you’ll be forced to deal with the possibility of a NULL
...


Constraining the Edges
As-is, the Edges table can be used to represent any graph, but semantics are important, and none are
implied by the current structure
...

Traversing the graph, one could conceivably go either way, so the following two rows may or may not be
logically identical:
INSERT INTO Edges VALUES (1, 2);
INSERT INTO Edges VALUES (2, 1);
If the edges in this graph are supposed to be directed, there is no problem
...
If, on the
other hand, all edges are supposed to be undirected, a constraint is necessary in order to ensure that two
logically identical paths cannot be inserted
...
The most obvious solution to this problem is to create a trigger that checks the rows when
inserts or updates take place
...

Before creating the trigger, empty the Edges table so that it no longer contains the duplicate
undirected edges just inserted:
TRUNCATE TABLE Edges;
GO
Then create the trigger that will check as rows are inserted or updated as follows:
CREATE TRIGGER CheckForDuplicates
ON Edges
FOR INSERT, UPDATE
AS
BEGIN
IF EXISTS
(
SELECT *
FROM Edges e
WHERE
EXISTS
(

374

CHAPTER 12

TREES, HIERARCHIES, AND GRAPHS

SELECT *
FROM inserted i
WHERE
i
...
Y
AND i
...
X
)
)
BEGIN
ROLLBACK;
END
END;
GO
Attempting to reinsert the two rows listed previously will now cause the trigger to end the
transaction and issue a rollback of the second row, preventing the duplicate edge from being created
...
You can take advantage of the fact that an indexed view has a unique index, using it as a constraint
in cases like this where a trigger seems awkward
...
The
following code listing creates such a table, populated with every number between 1 and 8000:
SELECT TOP (8000)
IDENTITY(int, 1, 1) AS Number
INTO Numbers
FROM master
...
spt_values b;
ALTER TABLE Numbers
ADD PRIMARY KEY (Number);
GO

Note We won’t actually need all 8,000 rows in the Numbers table (in fact, the solution described here requires
only two distinct rows), but there are lots of other scenarios where you might need a larger table of numbers, so it
doesn’t do any harm to prime the table with additional rows now
...
spt_values table is an arbitrary system table chosen simply because it has enough rows
that, when cross-joined with itself, the output will be more than 8,000 rows
...
However, in this case, its utility
is fairly simple: a CROSS JOIN to the Numbers table, combined with a WHERE condition, will result in an
output containing two rows for each row in the Edges table
...
The
following view encapsulates this logic:
CREATE VIEW DuplicateEdges
WITH SCHEMABINDING

375

CHAPTER 12

TREES, HIERARCHIES, AND GRAPHS

AS
SELECT
CASE n
...
X
ELSE e
...
Number
WHEN 1 THEN e
...
X
END Y
FROM Edges e
CROSS JOIN Numbers n
WHERE
n
...
Both techniques have similar
performance characteristics, but there is admittedly a certain cool factor with the indexed view
...


Note Once you have chosen either the trigger or the indexed view approach to prevent duplicate edges, be sure
to delete all rows from the Edges table again before executing any of the remaining code listings in this chapter
...
Figure 12-4 shows two graphs: I
is undirected and J is directed
...
Directed and undirected graphs have different connection qualities
...
This is easy enough to represent as a query (in this case, starting at node 1):
SELECT Y
FROM Edges e
WHERE X = 1;
For an undirected graph, things get a bit more complex because any given edge between two nodes
can be traversed in either direction
...
We need to consider all
edges for which node Y is either the start or endpoint, or else the graph has effectively become directed
...
The problem can be fixed
to some degree by creating multiple indexes (one in which each column is the first key) and using a
UNION ALL query, as follows:
SELECT Y
FROM Edges e
WHERE X = 1
UNION ALL
SELECT X
FROM Edges e
WHERE Y = 1;
This code is somewhat unintuitive, and because both indexes must be maintained and the query
must do two index operations to be satisfied, performance will still suffer compared with querying the
directed graph
...

The remainder of the examples in this chapter will assume that the graph is directed
...
For this section, a
more rigorous example data set is necessary
...


Figure 12-5
...

The next requirement is a table of intersections—the nodes in the graph
...

Note that I haven’t included any constraints on this table, as they can get quite complex
...
However, it’s difficult to say whether this would apply to all cities, given that many older cities

378

CHAPTER 12

TREES, HIERARCHIES, AND GRAPHS

have twisting roads that may intersect with each other at numerous points
...

CREATE TABLE IntersectionStreets
(
IntersectionId int NOT NULL
REFERENCES Intersections (IntersectionId),
StreetId int NOT NULL
REFERENCES Streets (StreetId),
PRIMARY KEY (IntersectionId, StreetId)
);
GO
INSERT INTO IntersectionStreets VALUES
(1, 1), (1, 5), (2, 2), (2, 5), (3, 3), (3, 5), (4, 4), (4, 5);
GO
The final table describes the edges of the graph, which in this case are segments of street between
each intersection
...
In both cases, the street is also included in the key
...

The CK_Intersections constraint ensures that the two intersections are actually
different—so you can’t start at one intersection and end up at the same place after
only one move
...
However, doing so would clearly not
help you traverse through the graph to a destination, which is the situation
currently being considered
...
The
GetIntersectionId function returns the intersection at which the two input streets intersect
...
It works by searching
for all intersections that the input streets participate in, and then finding the one that had exactly two
matches, meaning that both input streets intersect
...
IntersectionId
FROM dbo
...
Streets
WHERE StreetName IN (@Street1, @Street2)
)
GROUP BY i
...
The basic technique of
traversing the graph is quite simple: find the starting intersection and all nodes that it connects to, and
iteratively or recursively move outward, using the previous node’s ending point as the starting point for
the next
...
The following is
a simple initial example of a CTE that can be used to traverse the nodes from Madison and 1st Avenue to
Madison and 4th Avenue:
DECLARE
@Start int = dbo
...
GetIntersectionId('Madison', '4th Ave');
WITH Paths

380

CHAPTER 12

TREES, HIERARCHIES, AND GRAPHS

AS
(
SELECT
@Start AS theStart,
IntersectionId_End AS theEnd
FROM dbo
...
theEnd,
ss
...
StreetSegments ss ON ss
...
theEnd
WHERE p
...
The recursive part uses the anchor’s output as
its input, finding all connected nodes from there, and continuing only if the endpoint of the next
intersection is not equal to the end intersection
...
First of all, the ordering of the output of a CTE—just like any other query—is not
guaranteed without an ORDER BY clause
...
On
a bigger set of data and/or with multiple processors, SQL Server could choose to process the data in a
different order, thereby destroying the implicit output order
...
What
if there were more than one path? Figure 12-6 shows the street map with a new street, a few new
intersections, and more street segments added
...
This is on purpose, as I’m going to use those segments to illustrate
yet another complication
...
A slightly more complete version of the street map
Once the new data is inserted, we can try the same CTE as before, this time traveling from Madison
and 1st Avenue to Lexington and 1st Avenue
...
GetIntersectionId('Madison', '1st Ave'),
@End int = dbo
...

To solve this problem, the CTE will have to “remember” on each iteration where it’s been on
previous iterations
...
This can
be done using a materialized path notation, where each previously visited node will be appended to a
running list
...
GetIntersectionId('Madison', '1st Ave'),
@End int = dbo
...
StreetSegments
WHERE
IntersectionId_Start = @Start
UNION ALL
SELECT

383

CHAPTER 12

TREES, HIERARCHIES, AND GRAPHS

p
...
IntersectionId_End,
CAST(p
...
StreetSegments ss ON ss
...
theEnd
WHERE p
...
If node A (IntersectionId 1) is specified as the
start point, the output for this column for the anchor member will be /1/2/, since node B
(IntersectionId 2) is the only node that participates in a street segment starting at node A
...
Note that the columns in both the anchor and recursive members are CAST to make sure
their data types are identical
...
The output of
the CTE after making these modifications is as follows:
theStart

theEnd

thePath

1

2

/1/2/

2

3

/1/2/3/

2

6

/1/2/6/

6

5

/1/2/6/5/

3

4

/1/2/3/4/

4

8

/1/2/3/4/8/

8

7

/1/2/3/4/8/7/

7

6

/1/2/3/4/8/7/6/

6

5

/1/2/3/4/8/7/6/5/

The output now includes the complete paths to the endpoints, but it still includes all subpaths
visited along the way
...
After making that addition, only the two paths that actually visit both the
start and end nodes are shown
...
Figure 12-7 shows a completed version of the map, with
the final two street segments filled in
...
A version of the map with all segments filled in
Rerunning the CTE after introducing the new segments results in the following partial output
(abbreviated for brevity):
theStart theEnd thePath
6

5

/1/2/6/5/

6

5

/1/2/3/4/8/7/6/5/

6

5

/1/2/3/4/8/7/3/4/8/7/6/5/

6

5

/1/2/3/4/8/7/3/4/8/7/3/4/8/7/6/5/

6

5

/1/2/3/4/8/7/3/4/8/7/3/4/8/7/3/4/8/7/6/5/

6

5

/1/2/3/4/8/7/3/4/8/7/3/4/8/7/3/4/8/7/3/4/8/7/6/5/

6

5

/1/2/3/4/8/7/3/4/8/7/3/4/8/7/3/4/8/7/3/4/8/7/3/4/8/7/6/5/

6

5

/1/2/3/4/8/7/3/4/8/7/3/4/8/7/3/4/8/7/3/4/8/7/3/4/8/7/3/4/8/7/6/5/


...

The maximum recursion 100 has been exhausted before statement completion
...
The problem can be seen to start
at the fourth line of the output, when the recursion first visits node G (IntersectionId 7)
...

Following the first route, the recursion eventually completes
...

Eventually, the default recursive limit of 100 is reached and execution ends with an error
...
In this case, 100 is a good limit because it quickly tells us that there is a
major problem!
Fixing this issue, luckily, is quite simple: check the path to find out whether the next node has
already been visited, and if so, do not visit it again
...
thePath NOT LIKE '%/' + CONVERT(varchar, ss
...
This will make it impossible for the
recursion to fall into a cycle
...
The full code for the fixed CTE
follows:
DECLARE
@Start int = dbo
...
GetIntersectionId('Lexington', '1st Ave');
WITH Paths
AS
(
SELECT
@Start AS theStart,
IntersectionId_End AS theEnd,
CAST('/' +
CAST(@Start AS varchar(255)) + '/' +
CAST(IntersectionId_End AS varchar(255)) + '/'
AS varchar(255) ) AS thePath
FROM dbo
...
theEnd,
ss
...
ThePath +
CAST(IntersectionId_End AS varchar(255)) + '/'

386

CHAPTER 12

TREES, HIERARCHIES, AND GRAPHS

AS varchar(255)
)
FROM Paths p
JOIN dbo
...
IntersectionId_Start = p
...
theEnd <> @End
AND p
...
IntersectionId_End) + '/%'
)
SELECT *
FROM Paths;
GO
This concludes this chapter’s coverage on general graphs
...
Although hierarchies are much more specialized than graphs,
they tend to be more typically seen in software projects than general graphs, and developers must
consider slightly different issues when modeling them
...
I have had the pleasure of working fairly
extensively with a production system designed to traverse actual street routes and will briefly share some
of the insights I have gained in case you are interested in these kinds of problems
...
A big city has tens of thousands of street
segments, and determining a route from one end of the city to another using this method will create a
combinatorial explosion of possibilities
...

First of all, each segment can be weighted, and a score tallied along the way as you recurse over the
possible paths
...
For example, in the system I
worked on, weighting was done based on distance traveled
...
This scoring also lets the system determine the shortest possible routes
...
For instance, traveling from
one end of the city to another is usually most direct on a freeway
...
Put these routes together, including the freeway travel, and you have an optimized path from
the starting point to the ending point
...

If you’d like to try working with real street data, you can download US geographical shape files (including
streets as well as various natural formations) for free from the US Census Bureau
...
census
...
html
...


387

CHAPTER 12

TREES, HIERARCHIES, AND GRAPHS

Adjacency List Hierarchies
As mentioned previously, any kind of graph can be modeled using an adjacency list
...
Adjacency list hierarchies are very easy to model,
visualize, and understand, but can be tricky or inefficient to query in some cases since they require
iteration or recursion, as I’ll discuss shortly
...
This is a nice
feature, since it makes your code shorter, easier to understand, and more efficient
...
See “Constraining the Hierarchy” later in this section for
information on how to make sure that your hierarchies don’t end up with cycles, multiple roots, or
disconnected subtrees
...
Since it’s such a common and easily
understood example, this is the scenario that will be used for this section and the rest of this chapter
...
Employee table of the AdventureWorks database
...
Most of the time,
adjacency list hierarchies are modeled in a node-centric rather than edge-centric
way; that is, the primary key of the hierarchy is the key for a given node, rather
than a key representing an edge
...


•

ManagerID is the key for the employee that each row reports to in the same table
...
e
...
It’s common when modeling adjacency list hierarchies to use either
NULL or an identical key to the row’s primary key to represent root nodes
...


You can use the following T-SQL to create a table based on these columns:
USE AdventureWorks;
GO
CREATE TABLE Employee_Temp
(
EmployeeID int NOT NULL
CONSTRAINT PK_Employee PRIMARY KEY,
ManagerID int NULL
CONSTRAINT FK_Manager REFERENCES Employee_Temp (EmployeeID),
Title nvarchar(100)
);
GO
INSERT INTO Employee_Temp
(
EmployeeID,

388

CHAPTER 12

TREES, HIERARCHIES, AND GRAPHS

ManagerID,
Title
)
SELECT
EmployeeID,
ManagerID,
Title
FROM HumanResources
...
For adjacency lists as well as the other
hierarchical models discussed in this chapter, we’ll consider how to answer the following common
questions:
•

What are the direct descendants of a given node? In other words, who are the
people who directly report to a given manager?

•

What are all of the descendants of a given node? Which is to say, how many people
all the way down the organizational hierarchy ultimately report up to a given
manager? The challenge here is how to sort the output so that it makes sense with
regard to the hierarchy
...


Finding Direct Descendants
Finding the direct descendants of a given node is quite straightforward in an adjacency list hierarchy; it’s
the same as finding the available nodes to which you can traverse in a graph
...
To find all employees
that report directly to the CEO (EmployeeID 109), use the following T-SQL:
SELECT *
FROM Employee_Temp
WHERE ManagerID = 109;
This query returns the results shown following, showing the six branches of AdventureWorks,
represented by its upper management team—exactly the results that we expected
...
Considering that this column is not indexed, it should
come as no surprise that the query plan for the preceding query involves a scan, as shown in Figure 12-8
...
Querying on the ManagerID causes a table scan
...
However, choosing
exactly how best to index a table such as this one can be difficult
...
However, in an actual production
system, there might be a much higher percentage of queries based on the EmployeeID—for instance,
queries to get a single employee’s data—and there would probably be a lot more columns in the table
than the three used here for example purposes, meaning that clustered key lookups could be expensive
...

In order to show the best possible performance in this case, change the primary key to use a
nonclustered index and create a clustered index on ManagerID, as shown in the following T-SQL:
ALTER TABLE Employee_Temp
DROP CONSTRAINT FK_Manager, PK_Employee;
CREATE CLUSTERED INDEX IX_Manager
ON Employee_Temp (ManagerID);
ALTER TABLE Employee_Temp
ADD CONSTRAINT PK_Employee
PRIMARY KEY NONCLUSTERED (EmployeeID);
GO

390

CHAPTER 12

TREES, HIERARCHIES, AND GRAPHS

Caution Adding a clustered index to the nonkey ManagerId column might result in the best performance for
queries designed solely to determine those employees that report to a given manager, but it is not necessarily the
best design for a general purpose employees table
...


Traversing down the Hierarchy
Shifting from finding direct descendants of one node to traversing down the entire hierarchy all the way
to the leaf nodes is extremely simple, just as in the case of general graphs
...
The following CTE, modified from the section on graphs, traverses the
Employee_Temp hierarchy starting from the CEO, returning all employees in the company:
WITH n AS
(
SELECT
EmployeeID,
ManagerID,
Title
FROM Employee_Temp
WHERE ManagerID IS NULL
UNION ALL
SELECT
e
...
ManagerID,
e
...
EmployeeID = e
...
EmployeeID,
n
...
Title
FROM n;
GO
Note that this CTE returns all columns to be used by the outer query—but this is not the only way to
write this query
...
EmployeeID
FROM Employee_Temp e
JOIN n ON n
...
ManagerID
)
SELECT
e
...
ManagerID,
e
...
EmployeeID = n
...
The latter query tends to perform better as the output
row size increases, but in the case of the small test table, the former query is much more efficient
...


Ordering the Output
Regardless of the performance of the two queries listed in the previous section, the fact is that we haven’t
really done much yet
...
In order to add value, the output should be sorted such
that it conforms to the hierarchy represented in the table
...
By ordering by the path, the output will follow the same nested order as the hierarchy itself
...
EmployeeID,
e
...
Title,
CONVERT(varchar(900),
n
...
EmployeeID), 10) + '/'
) AS thePath
FROM Employee_Temp e
JOIN n ON n
...
ManagerID
)
SELECT
n
...
ManagerID,
n
...
thePath
FROM n
ORDER BY n
...
This
ensures that, for instance, the path 1/2/ does not sort higher than the path 1/10/
...
Note that siblings in this case are ordered based on their EmployeeID
...

Instead of materializing the EmployeeID, materialize a row number that represents the current ordered

393

CHAPTER 12

TREES, HIERARCHIES, AND GRAPHS

sibling
...
The following modified version of the CTE enumerates the path:
WITH n AS
(
SELECT
EmployeeID,
ManagerID,
Title,
CONVERT(varchar(900),
'0000000001/'
) AS thePath
FROM Employee_Temp
WHERE ManagerID IS NULL
UNION ALL
SELECT
e
...
ManagerID,
e
...
thePath +
RIGHT(
REPLICATE('0', 10) +
CONVERT(varchar, ROW_NUMBER() OVER (ORDER BY e
...
EmployeeID = e
...
EmployeeID,
n
...
Title,
n
...
thePath;
GO
The enumerated path representing each node is illustrated in the results of the query as follows:

394

CHAPTER 12

TREES, HIERARCHIES, AND GRAPHS

EmployeeID

ManagerID

Title

thePath

109

NULL

Chief Executive Officer

00000001/

140

109

Chief Financial Officer

00000001/00000001/

139

140

Accounts Manager

00000001/00000001/00000001/

216

139

Accountant

00000001/00000001/00000001/00000001/

178

139

Accountant

00000001/00000001/00000001/00000002/

166

139

Accs Payable Specialist

00000001/00000001/00000001/00000003/

201

139

Accs Payable Specialist

00000001/00000001/00000001/00000004/

130

139

Accs Recvble Specialist

00000001/00000001/00000001/00000005/

94

139

Accs Recvble Specialist

00000001/00000001/00000001/00000006/

59

139

Accs Recvble Specialist

00000001/00000001/00000001/00000007/

103

140

Assistant to the CFO

00000001/00000001/00000002/

71

140

Finance Manager

00000001/00000001/00000003/

274

71

Purchasing Manager

00000001/00000001/00000003/00000001/

Tip Instead of left-padding the node IDs with zeros, you could expose the thePath column typed as varbinary
and convert the IDs to binary(4)
...
The downside is that this makes the IDs more difficult to visualize, so for the purposes of this
chapter—where visual cues are important—I use the left-padding method instead
...
For
instance, simply looking at the thePath column in the results of the first query in this section, we can see
that the path to the Engineering Manager (EmployeeID 3) starts with EmployeeID 109 and continues to
EmployeeID 12 before getting to the Engineering Manager
...


395

CHAPTER 12

TREES, HIERARCHIES, AND GRAPHS

Are CTEs the Best Choice?
While CTEs are possibly the most convenient way to traverse adjacency list hierarchies in SQL Server
2008, they do not necessarily deliver the best possible performance
...

To highlight the performance difference between CTEs and iterative methods, a larger sample
hierarchy is necessary
...
This means that
the hierarchy will maintain the same depth, but each level will have more siblings
...
The following T-SQL
accomplishes this, running in a loop five times and doubling the width of the hierarchy on each
iteration:
DECLARE @CEO int;
SELECT
@CEO = EmployeeID
FROM Employee_Temp
WHERE ManagerID IS NULL;
DECLARE @width int = 1;
WHILE @width <= 16
BEGIN
INSERT INTO Employee_Temp
(
EmployeeID,
ManagerID,
Title
)
SELECT
e
...
ManagerID
WHEN @CEO THEN e
...
ManagerID + (1000 * @width)
END,
e
...
ManagerID IS NOT NULL;
SET @width = @width * 2;
END;
GO
There are two key factors you should pay attention to in this example
...
Second,
look at the CASE expression in the SELECT list, which increments all IDs except that of the CEO
...


396

CHAPTER 12

TREES, HIERARCHIES, AND GRAPHS

Once this code has been run, the Employee_Temp hierarchy will have 9,249 nodes, instead of the 290
that we started with
...
To increase the depth, a slightly
different algorithm is required
...
Next, update the preexisting managers in the
table to report to the new managers
...
EmployeeID - (1000 * @depth) INTO @OldManagers
SELECT
e
...
ManagerID,
'Intermediate Manager'
FROM Employee_Temp e
WHERE
e
...
ManagerID = e
...
I’ve found that depth has a
much greater performance impact than width, and extremely deep hierarchies are not especially
common—for instance, even the largest companies do not normally have more than 20 or 30 levels of
management
...
Applying this logic iteratively requires the following table variable:
DECLARE @n table
(
EmployeeID int,
ManagerID int,
Title nvarchar(100),
Depth int,
thePath varchar(900),
PRIMARY KEY (Depth, EmployeeID)
);
The Depth column maintains the level for nodes as they are inserted
...

To start things off, prime the table variable with the node you wish to use as the root for traversal
...
For each
level of depth, find the subordinates
...
EmployeeID,
e
...
Title,
@depth + 1,
n
...
ManagerID ORDER BY e
...
EmployeeID = e
...
Depth = @depth;
IF @@ROWCOUNT = 0
BREAK;
SET @depth = @depth + 1;
END
Finally, the output can be queried from the table variable
...
However, its performance is quite a bit
better than the CTE
...
6 seconds on my
laptop against the enhanced Employee_Temp table
...
2 seconds
...
I feel that the maintainability issues overshadow the performance benefits in all
but the most extreme cases (such as that demonstrated here)
...
However, you should be able to convert any of the examples so that they
use iterative logic
...


399

CHAPTER 12

TREES, HIERARCHIES, AND GRAPHS

Note If you’re following along with the examples in this chapter and you increased the number of rows in the
Employee_Temp table, you should drop and re-create it before continuing with the rest of the chapter
...
Instead
of using ManagerID as a key at each level of recursion, use EmployeeID
...
ManagerID,
CONVERT(varchar(900),
n
...
EmployeeID),
10) + '/'
) AS thePath
FROM Employee_Temp e
JOIN n ON n
...
EmployeeID
)
SELECT *
FROM n
WHERE ManagerID IS NULL;
This query returns the path from the selected node to the CEO as a materialized path of employee
IDs
...
In order to do
that, change the outer query to the following:
SELECT
COALESCE(ManagerID, 217) AS EmployeeID
FROM n
ORDER BY

400

CHAPTER 12

TREES, HIERARCHIES, AND GRAPHS

CASE
WHEN ManagerID IS NULL THEN 0
ELSE 1
END,
thePath;
In this case, the COALESCE function used in the SELECT list replaces the CEO’s ManagerID—which is
NULL—with the target EmployeeID
...
All other sorting is based on the materialized
path, which naturally returns the CEO’s row last
...
Inserting a leaf
node (i
...
, a node with no subordinates) requires simply inserting a new node into the table
...
This is effectively the same as inserting a new node and then relocating
the old node’s subtree under the new node, which is why I’ve merged these two topics into one section
...
To reflect these changes in the
Employee_Temp table, first insert the new CTO node, and then update the VP’s node to report to the new
CTO:
INSERT INTO Employee_Temp
(
EmployeeID,
ManagerID,
Title
)
VALUES
(
999,
109,
'CTO'
);
GO
UPDATE Employee_Temp
SET ManagerID = 999
WHERE EmployeeID = 12;
GO
That’s it! This same logic can be applied for any subtree relocation—one of the advantages of
adjacency lists over the other hierarchical techniques discussed in this chapter is the ease with which
data modifications like this can be handled
...
This time,
the first step is to relocate any subordinates that report to the node to be deleted—the key is that
subordinates is plural this time, as there may be more than one
...

Suppose that on her first day at the office, the new CTO won the lottery and decided that she would
rather race her yacht than continue to work at AdventureWorks
...
Due to the self-referencing foreign key on the table, nonleaf nodes cannot be deleted—this is
another nice fringe benefit of adjacency lists
...
Once the update is complete, the node is a leaf, and so can
be removed
...
The main benefit of this assumption is that the resultant code can be made a
lot simpler; the problem is that the simpler code is prone to various problems should bad data creep in—
and as most readers are no doubt aware, bad data can and will creep in if given the opportunity
...
From a defensive point of view, I highly
recommend the second approach
...

Forests occur when there are multiple root nodes in the hierarchy
...
Cycles occur when, somewhere
downstream from a given node, that node is suddenly referenced again
...
The primary key, which is

402

CHAPTER 12

TREES, HIERARCHIES, AND GRAPHS

on the EmployeeID column, guards against most cycles by making it impossible for a given employee to
have more than one manager
...

The first thing that must be constrained against is multiple root nodes
...
Employee_Temp
WHERE
ManagerID IS NULL;
GO
CREATE UNIQUE CLUSTERED INDEX IX_OnlyOneRoot
ON OnlyOneRoot (ManagerID);
GO
The view returns all rows in the table with a NULL manager ID, and the index, because it is UNIQUE,
only allows one such row to be inserted
...
For example, it
doesn’t stop someone from assigning a manager to the root node (the unique constraint can only
enforce rows that exist, and by updating the table, the NULL manager ID would no longer exist at all)
...
To begin with, the simplest cycle—and one that’s not constrained against by either the primary
key or the foreign key—is an employee managing herself
...
For instance, if George manages Ed and Ed manages Steve, someone could issue an update
to the table so that Ed manages George
...
In order to solve this problem, a trigger can be employed
...
Should it encounter the same employee a
second time before hitting the root, it is apparent that there is a cycle
...
EmployeeID, et
...
ManagerID = et
...
ManagerID IS NOT NULL
AND e
...
EmployeeID
)
SELECT @CycleExists = 1
FROM e
WHERE e
...
EmployeeID;
IF @CycleExists = 1

404

CHAPTER 12

TREES, HIERARCHIES, AND GRAPHS

BEGIN
RAISERROR('The update introduced a cycle', 16, 1);
ROLLBACK;
END
END
GO
This type of cycle can only be caused by either updates or multirow inserts, and in virtually all of the
hierarchies I’ve seen in production environments, there were no multirow inserts
...
Remember to change the trigger definition if you need to work with
multirow inserts in your environment
...
While the solution
proposed in this section solves the problem of cycles using triggers in the database, you might want to consider
enforcing this kind of business logic in an application layer instead, if it makes more sense to do so
...
In this section we’ll look at an alternative
technique that avoids this problem: persisted materialized paths
...
In the following sections, I will show you how to maintain both hierarchy
types alongside each other in the Employee_Temp table, using a series of triggers
...
This same path can be persisted in the table to
allow you to answer all of the same hierarchical questions as with an adjacency list, but without the
necessity for recursion or iteration
...
EmployeeID,
CONVERT(varchar(900),
n
...
EmployeeID) + '/'
) AS thePath
FROM Employee_Temp e
JOIN n ON n
...
ManagerID
)
UPDATE Employee_Temp
SET Employee_Temp
...
thePath
FROM Employee_Temp
JOIN n ON n
...
EmployeeID;
GO
varchar(900) is important in this case because the materialized path will be used as an index key in
order to allow it to be efficiently used to traverse the hierarchy
...
This is also a bit of a limitation for persisted materialized paths; a path to navigate an
especially deep hierarchy will not be indexable and therefore will not be usable for this technique
...
That said, I can all but
guarantee that a clustered index will never be the right choice
...
The path doesn’t bring any value in its nonindexed form, so I
don’t recommend trying that technique
...
In the case of the Employee_Temp table, the index will include the Title and
EmployeeID columns so that the same output shown before can be most efficiently produced via the
materialized path (the table is clustered on ManagerID, so that does not have to be explicitly included):
CREATE NONCLUSTERED INDEX IX_Employee_Temp_Path
ON Employee_Temp (thePath)
INCLUDE (EmployeeID, Title);

Finding Subordinates
Since the materialized path is a string, we can take advantage of SQL Server’s LIKE predicate to traverse
down the hierarchy
...
Looking back at the results of the enumerated path contained in the last section, notice
that since all nodes are descendants of EmployeeID 109 (the CEO), every path starts with the string 109/
...

Therefore, searching for all subordinates of a given employee is as simple as using the LIKE
predicate and adding the wildcard character, %
...
To find all subordinates of the Vice President using the CTE, the query engine must perform
187 logical reads
...

Finding only the direct reports for a given node is just a bit trickier
...
To eliminate
the input node, we can change the predicate to thePath LIKE @Path + '%/'
...
However, this still includes all
children nodes, as each has a path suffixed by a stroke
...
Essentially, this predicate
says that the target path can only have one more stroke than the input path—and therefore, that path is
one level of depth below
...
However, the materialized path itself already contains all of the information necessary—
it’s just that the data needs to be manipulated a bit to get it into a usable format
...
In order to generate a table from the list, it must be split up based on its
delimiters
...
By putting this logic in a
recursive CTE, we can take the substring of each node in the path on each iteration
...
The
following CTE finds the path to the CEO starting at the Research and Development Manager:
WITH n AS
(
SELECT
CONVERT(int,

407

CHAPTER 12

TREES, HIERARCHIES, AND GRAPHS

SUBSTRING(thePath, 2, CHARINDEX('/', thePath, 2) -2)) AS EmployeeID,
SUBSTRING(thePath, CHARINDEX('/', thePath, 2), LEN(thePath)) AS thePath,
1 AS theLevel
FROM Employee_Temp
WHERE EmployeeID = 217
UNION ALL
SELECT
CONVERT(int,
SUBSTRING(thePath, 2, CHARINDEX('/', thePath, 2) -2)),
SUBSTRING(thePath, CHARINDEX('/', thePath, 2), LEN(thePath)),
theLevel + 1
FROM n
WHERE thePath LIKE '/%/'
)
SELECT *
FROM n
ORDER BY theLevel;
The output of this query is as follows:
EmployeeID

thePath

theLevel

109

/12/3/158/217/

1

12

/3/158/217/

2

3

/158/217/

3

158

/217/

4

217

5

Aside from the expression used to pull out the current first node, another expression I used in both
the anchor and recursive members cuts the first node out of the path, so that the path progressively
shrinks as the CTE recurses at each level
...

Although the EmployeeID column is probably the only one necessary in the output, I’ve left the other
columns in so that you can see how the path is affected by the CTE’s logic at each level
...
This logic can be encapsulated in a trigger such that whenever new nodes
are inserted into the adjacency list, their paths will automatically be updated
...
thePath =
Managers
...
EmployeeID),
e
i
...
EmployeeID
Managers ON Managers
...
ManagerID;

The logic of this trigger is relatively simple: find the updated row in the Employee_Temp table by
joining on the EmployeeID columns of both it and the inserted virtual table, and then join back to
Employee_Temp to get the manager’s path
...

The most important thing to mention about this trigger is its limitation when it comes to multirow
inserts
...
For instance, try disabling the
row count check and inserting a subordinate first, followed by a manager, in the same statement:
INSERT INTO Employee_Temp
(
EmployeeID,
ManagerID,
Title
)
VALUES
(1000, 999, 'Subordinate'),
(999, 109, 'Manager');
Since the order in which the UPDATE processes rows is not guaranteed, the result of this operation is
nondeterministic
...
It may be possible to solve this problem
by traversing any hierarchy present in the inserted table using a cursor, but I decided not to attempt this
as I have never seen a situation in a real-world project in which this limitation would be a barrier
...
Any time you
affect a node’s path, you must cascade the new path to all of its subordinates
...
thePath LIKE e
...
EmployeeID
) x;
Relocating a materialized path’s subtree involves finding the new manager’s path and replacing it in
the updated node as well as all of its child nodes
...
Her path will change to the CEO’s path with her employee ID concatenated to the
end: 0000000109/0000000003/
...
So the same
operation—replacement of the beginning of the path—has to happen for all three nodes
...

Once again, a trigger can be employed to automatically perform this update when a subtree is
located based on the adjacency list
...
thePath =
REPLACE(e
...
thePath,
Managers
...
EmployeeID),
10
) + '/'
)
FROM Employee_Temp e
JOIN inserted i ON e
...
thePath + '%'
JOIN Employee_Temp Managers ON Managers
...
ManagerID
END
END
GO
There are a few things to discuss in this trigger
...
Just like when dealing with inserting new nodes,
relocation of subtrees must be serialized to one node at a time in order to avoid logical ambiguities
...
” As long as the update is not to the hierarchy, multirow updates are certainly allowed
...
Next, the trigger checks to see whether the ManagerID column is being updated
...
If so, it then throws an error if multiple rows have been affected
...
The logic used is to find the previous path of the updated node—using the inserted virtual table,
which will still have that original path because direct updates to the path are not allowed—and replace it
in all nodes with the new path
...


Note If you’re using the tg_AtLeastOneRoot trigger in conjunction with the tg_Update trigger, you’ll have a
problem if you need to swap the root node, because to satisfy the tg_AtLeastOneRoot trigger’s logic, the update
must end with a root note in place, and this will require a multirow update
...


Deleting Nodes
Thanks to the fact that the adjacency list is being used in conjunction with the materialized path,
deleting nodes requires no additional logic
...
Leaf nodes have no subordinates and therefore there is nothing to

411

CHAPTER 12

TREES, HIERARCHIES, AND GRAPHS

cascade—the row will be deleted, and no further logic is necessary
...


Constraining the Hierarchy
All of the logic mentioned in the previous “Constraining the Hierarchy” section (which dealt with
adjacency lists) still applies to materialized paths
...
Instead, a simple check constraint
should be used that makes sure the given employee only appears once, at the end of the path:
ALTER TABLE Employee_Temp
ADD CONSTRAINT ck_NoCycles
CHECK
(
thePath NOT LIKE
'%' +
RIGHT(REPLICATE('0', 10) + CONVERT(varchar, EmployeeID), 10) +
'/%' +
RIGHT(REPLICATE('0', 10) + CONVERT(varchar, EmployeeID), 10) +
'/'
);
GO

Note There is a subtle difference between the logic expressed here and the logic used to prevent endless loops
in the section “Traversing the Graph
...
This was done to prevent a
false alarm in case of a path like 123/456/3
...


The hierarchyid Datatype
In the preceding section, I discussed one of the limitations of the materialized path approach—namely,
that the string encoding makes it difficult to work with deep hierarchies
...
While this doesn’t provide much additional functionality
on top of what was possible using the materialized path approach described previously, it does make
querying hierarchical data much easier, as I will demonstrate in this section
...
Fortunately for us,
the string format expected by the Parse() method is exactly the same as we used in the thePath column
created in the previous section—using an oblique stroke between each node in the path
...

ALTER TABLE Employee_Temp
ADD hierarchy hierarchyid;
UPDATE Employee_Temp
SET hierarchy = hierarchyid::Parse (thePath);

Finding Subordinates
Having populated the hierarchyid column, let’s put it to the test by performing the same set of common
queries as we did for the other methods
...
To do this, we can use the IsDescendantOf() method, as shown in
the following query:
SELECT * FROM Employee_Temp
WHERE hierarchy
...

This style of syntax is probably familiar to all developers who have previously used
XQuery functionality in SQL Server or elsewhere
...

To get the string representation of a hierarchyid value, you must call the
ToString() method
...

This is an interesting quirk of the hierarchyid type in that, for any given node x,
x
...
This is by design, and is useful in some circumstances,
but may not seem that intuitive
...


To restrict the results to only return direct descendants of a node (and also to exclude the node itself
from the results), we can take advantage of the GetLevel() method, which returns an integer
representing the “level” of a node in the hierarchy
...
Therefore, the direct subordinates of a
given node will always be one level greater than their parent
...
IsDescendantOf(@Parent) = 1
AND hierarchy
...
GetLevel() + 1;
This is a very flexible structure that can be used to specify exactly how many levels of descendants to
traverse under a given node—for example, to return children and grandchildren, you can include all
those nodes where the difference between the two levels is less than or equal to 2
...
In order to display this
path in a readable format, we simply need to call the ToString() method
...
ToString() AS hierarchyPath
FROM Employee_Temp
WHERE EmployeeID = 100;
The results of this query are as follows:

EmployeeID ManagerID Title
hierarchyPath
100
143
Production Technician - WC20 /109/148/21/143/100/

414

CHAPTER 12

TREES, HIERARCHIES, AND GRAPHS

Notice how the path obtained from the hierarchyid ToString() method is exactly the same as the
path we created in the thePath column earlier
...
Rather than returning a Boolean response indicating whether a given node is descended
from another node, the GetAncestor() method is invoked on a child element and used to return a
hierarchyid node representing an ancestor of that child
...

To demonstrate, the following code listing uses that GetAncestor() method with the argument 1 to
determine the immediate parent of each node in the hierarchy column
...
GetAncestor(1) = @Parent;

Inserting Nodes
In order to insert a new node, we need to calculate the appropriate hierarchyid value representing the
path to that node
...
GetDescendant() accepts two arguments that determine the lower
and upper values of the range in which the allocated hierarchyid node will lie
...
By providing null values for both arguments, the following code uses the
GetDescendant() method to allocate an arbitrary hierarchyid value for the new employee:
DECLARE @Parent hierarchyid;
SELECT @Parent = hierarchy
FROM Employee_Temp
WHERE EmployeeID = 85;
SELECT @Parent
...
ToString();
The result, /109/148/21/85/1/, is guaranteed to be a new node that is a direct descendant of the
supplied parent node, but the actual identity of the new node may vary
...
For example, consider instead what would happen if the Production Supervisor,
EmployeeID 16, were to hire a new member of staff
...
GetDescendant(null, null)
...
It is assigned to EmployeeID 1, who is a
technician reporting to this production supervisor
...
To do this, we will find out the maximum currently assigned child node
for the chosen parent, as follows:
DECLARE @Parent hierarchyid;
SELECT @Parent = hierarchy
FROM Employee_Temp
WHERE EmployeeID = 16;
DECLARE @MaxChild hierarchyid;
SELECT @MaxChild = MAX(hierarchy)
FROM Employee_Temp
WHERE hierarchy
...
GetDescendant(@MaxChild, null)
...
The
second parameter that can be supplied to GetDescendant() represents the maximum node value that the
newly created node must lie before
...

Of course, in the current model, the value of each node in the materialized path is based on the
EmployeeID of the associated employee
...
However, there are many other scenarios in which such an
identity value is not available, in which case, the GetDescendantOf() method becomes a very useful way
to allocate values for new child nodes
...


Relocating Subtrees
To relocate a hierarchyid node, we use the GetReparentedValue() method
...
To move the node, you must then update the
hierarchyid value to be equal to this result
...

To start with, we need to identify the nodes that represent both the old parent that nodes are
moving from, and the new parent to which they will be reporting, as follows:
DECLARE @FromParent hierarchyid;
SELECT @FromParent = hierarchy
FROM Employee_Temp
WHERE EmployeeID = 64;

416

CHAPTER 12

TREES, HIERARCHIES, AND GRAPHS

DECLARE @ToParent hierarchyid;
SELECT @ToParent = hierarchy
FROM Employee_Temp
WHERE EmployeeID = 74;
Having identified the two parents between which nodes are to be moved, we can then update all
those descendants of the old parent using the GetReparentedValue() method as follows:
UPDATE Employee_Temp
SET hierarchy = hierarchy
...
IsDescendantOf(@FromParent) = 1
AND EmployeeID <> 64;
This query uses the IsDescendantOf() method to identify all those rows that report to the old parent,
EmployeeID 64, and updates them to report to the new parent
...

As with row inserts, if you wanted to maintain this solution in a production environment alongside
the adjacency list, you would need to modify the tg_Update procedure to update the hierarchy column
when nodes were moved
...
As such, any node may be removed from the hierarchy, potentially leading to orphan nodes if
the node removed is a nonleaf node
...
These constraints mean that only leaf
nodes can be deleted, which reduces the chances of orphans or multiple root nodes being introduced
into the structure of the hierarchy
...
A further
consideration is how best to index columns of hierarchyid data
...

By default, SQL Server 2008 indexes hierarchyid values in a depth-first order
...
This sort of indexing is
efficient for fulfilling queries that involve traversing up (or down) several levels of subtrees, such as “Find
the managers two levels immediately above this employee
...
This kind of
index is best suited for queries such as “Find all the employees who report directly to this manager
...
To
create a breadth-first index, we will make use of the GetLevel() method once more to populate a
persisted column representing the level of each node
...
GetLevel();
GO
CREATE UNIQUE INDEX idxHierarchyBreadth
ON Employee_Temp(hierarchyLevel, hierarchy);
GO

Summary
Graphs and hierarchies are extremely common throughout our world, and it is often necessary to
represent them in databases
...
Hierarchies—special types of
graphs—can also be modeled using adjacency lists, but other techniques can be employed to make
querying them much more efficient, without the need for recursion or iteration
...
As always, the most important thing you can do as a
developer is to carefully consider your options, and test whenever possible to find the optimal solution
...
See also
persistent materialized paths
constraining, 374–376, 388, 402–405, 412
finding ancestors, 407–408
finding descendants, 389–391, 400–401,
406–407
graph queries, 376–377
nodes
deleting, 402, 411
inserting new, 401, 408–409
overview, 373, 388–389
relocating subtrees, 401, 410–411
traversing, 388, 391–393, 396–399, 412
Advanced Encryption Standard (AES), 130, 140
affinity mask, 276
age, calculating, 335–336
agile development, 25, 55
allCountries table, 299, 302–303
allCountries
...
zip file, 298
AllowPartiallyTrustedCallersAttribute (APTCA),
178
ALTER AUTHORIZATION command, 106
ALTER DATABASE command, 111, 175
ALTER SCHEMA command, 107
anarchic concurrency control, 236–237, 243
ancestor relationships, 373
anti-patterns, 20

APIs (application programming interfaces), 18
application lifespan, 27
application locks
acquiring, 250
described, 250
drawbacks of, 251
nontransactional, 251–259
releasing, 250
application logic, 12
application logins, 102
application programming interfaces (APIs), 18
application-level parameterization, 202–203
APTCA (AllowPartiallyTrustedCallersAttribute),
178
architecture, software
cohesion, 2, 4–5
coupling, 2–3, 5
encapsulation, 2, 5, 9
evolution, 8–9
flaws, 12
AS keyword, 7
Aschenbrenner, Klaus, 252
as-of data component, 322
assemblies
cataloging, 163
granting privileges between, 175–178
privileges, raising selectively, 168–
175
strong naming, 177
Assert method (IStackWalk interface), 174
assertions, debug, 52
assets, need for encryption, 121–122
assumptions, identifying in code, 29–33
asymmetric keys, 124, 127, 134–136
attitudes to defensive programming, 24–
27
authentication
defined, 101
overview, 101, 104
software release issues, 101
authentication tests, 56

419

INDEX

authorization
...
See also temporal data
described, 322, 367
managing, 367–369
black box testing, 49–50
BLOB data, 139
blocking, 236
...
See Code Access Security
CAST operation, 34, 133
catch block, defined, 87
cell-level encryption, 122, 138–139,
147
CELLS_PER_OBJECT limit, 316, 319
certificates
backing up, 117–118
creating, 103, 117
cryptographic hash, 116
identifying modules signed, 116
identifying users, 116
module signing, 112, 114–116
overview, 124
restoring, 117–118
certification authority (CA), 134
ChangeLog table, 31–32
CHARINDEX, 183–184
classes, compared to tables, 13–14
client/server-based architecture, 9
CLOSE SYMMETRIC KEY method, 131
closed LineString object, 284
CLR (common language runtime)
...
See SQLCLR
Common Table Expressions (CTEs)
compared to persistent materialized paths,
407
limitations, 383
ordering output, 381
traversing graphs, 380–381, 386
traversing hierarchies, 396, 399
compilation, query, 75, 198–200
compression algorithms, 139
computers, evolution of, 8–9
concurrent processes, monitoring
performance, 360
concurrent use
...
NET, 56
CROSS APPLY operator, 351–352, 365
CROSS JOINs, 37
cross-database ownership chaining, 111–112
crossing, 294
CTEs
...
See encryption
Data Encryption Standard (DES), 125
data logic
business modeling, 10
described, 10
location, 10–11
data massage, 41
Data Protection API (DPAPI), 125

421

INDEX

data types, 160–161
database applications, 9–10
database bugs, 27–28
database development
...
See encryption
database encryption key (DEK), 136
database integrity, risks to, 28
database interfaces, 18
database mail feature, 43
database master key (DMK), 124–125
database queries
...
See also dynamic SQL
accessing information, 13–14
design goals, 8
flexibility, 195
inheritance represented in, 14–16
integrating with object-oriented systems, 8,
12–13, 16–18
purpose, 197
role in applications, 19
security responsibilities, 21
data-dependent applications, 1
data-driven applications, 1
DataTable class, 187, 189
date datatype, 35, 323
date format function, 45
DATEADD function, 36, 330–336, 366
DATEDIFF function, 330–336, 365
DATEFORMAT settings, 324–325
dates
...
See also optional parameters
advantages of, 197, 213
defined, 196
formatting, 215–216
injection attacks, 218–219
justification for, 197–198
ownership chaining with, 111
security, 230–232
uses for, 205

E
edges, graph, 371
effective maximum CPU usage, 278

INDEX

EFS (encrypting file system), 139
EKM (extensible key management), 127
ellipsoidal calculations, 294
ELSE clause, 46
encapsulation
challenges to determining, 2
database interface, 18
defined, 2
example of, 5
importance, 5, 9
ENCRYPTBYASYMKEY method, 135, 139
ENCRYPTBYKEY method, 131, 139, 142–143
ENCRYPTBYPASSPHRASE function, 132–134
encrypting file system (EFS), 139
encryption
balancing performance and security, 139–
144
implications for query design
equality matching with hashed message
authentication codes, 148–153
overview, 145, 158
range searches, 157
wildcard searches with HMAC
substrings, 153–157
methods of
asymmetric key encryption, 134–136
hashing, 129–130
overview, 128, 139
symmetric key encryption, 130–133
transparent data encryption, 136–139
need for
assets, 121–122
overview, 121–123
threats, 122–123
overview, 121, 158
encryption key hierarchy
asymmetric keys, 124
certificates, 124
database master key, 125
extensible key management, 127
overview, 123–127
removing keys from automatic encryption
hierarchy, 126–127
service master key, 125
symmetric key layering, 126
symmetric key rotation, 126
symmetric keys, 124
enumerating the path, 393–395
equality matching, 148–153
error 208 exceptions, 85
error level, 79, 81

error message templates, 78
error messages, 78, 82–84
error numbers, 78
error state, 79
ERROR_LINE function, 89–90
ERROR_MESSAGE function, 89–90
ERROR_NUMBER function, 89–90
ERROR_PROCEDURE function, 89–90
ERROR_SEVERITY function, 89–90
ERROR_STATE function, 89–90
errors, defined, 71
ESRI shapefile format (SHP), 296
events
...
See also EXECUTE
command
EXECUTE AS command, 107–109, 112–114

423

INDEX

EXECUTE command
drawbacks of, 218–220
process, 221
uses for, 196, 213
expanding search range, 306
expected behavior, 45
ExpertSqlEncryption user database, 137
explicit bounding box, 294
explicit contracts, 6
exposure limiting, 43
extended events, 64–65
extended methods, 303
extensible key management (EKM), 127
exterior ring, 284
EXTERNAL_ACCESS permission set, 43, 163
extreme programming (XP), 23, 55

F
fail fast methodology, 27
farms, server, 9
feature creep, 22
Federal Information Processing Standard
(FIPS), 149
file I/O, 165
FileIOPermission class, 175
FileIOPermissionAccess enumeration, 175
filestream datatype, 139
Filter( ) method, 303
filtering, 294
FinanceSymKey property, 143
FinanceUser class, 142
finding locations in bounding boxes, 308–313
FIPS (Federal Information Processing
Standard), 149
fixed search zone, 306
flat plane, 292
flexible modules/interfaces, 20
fn_trace_gettable function, 61
forced parameterization, 202
forests, 372, 402
format designators, error message, 82–83
formatting in the database, 366
functional tests, 50–52
future-proofing code, 42–43

G
GenerateHMAC function, 151, 154
geographic coordinate systems, 286, 310

424

geographic information systems (GISs), 283
geography
accuracy, 294
overview, 292–296
performance, 294–296
standards compliance, 293–294
technical limitations, 294–296
geography datatype, 292, 294
Geography Markup Language (GML), 296, 298
geometry
accuracy, 294
overview, 292–296
performance, 294–296
standards compliance, 293–294
technical limitations, 294–296
geometry datatype, 292
GeometryCollection geometry, 284
GetAggregateTransactionHistory stored
procedure, 51
getBounds( ) method, 308
GetMapView( ) method, 308
GETUTCDATE command, 344
GISs (geographic information systems), 283
global positioning system (GPS), 288
GML (Geography Markup Language), 296, 298
GMT (Greenwich Mean Time), 341
GO identifier, 80
GPS (global positioning system), 288
GRANT IMPERSONATE command, 109
granular analysis, 69, 71
graphs
...
)
symmetric
encryption, 130–133
layering, 126
overview, 124
rotating, 126
KISS (keep it simple, stupid) principle, 22, 24

L
Lat property, 302
late binding, 75
latitude, 284, 286
latitude column, 302
layering symmetric keys, 126
least privilege, principle of, 102–103
lifespan of applications, 27
limiting exposure, 43
LineString objects, 284, 294, 312
load testing, 69
localization error messages, 85
location column, 302
locations, finding in bounding boxes, 308–
313
lock tokens, 246–247, 249–250
locking data
...
See persistent materialized
paths
MAX_CPU_PERCENT value, 274–276, 278
MD (Message Digest) algorithms, 129
MEDIUM grid resolution, 317, 319
meridians, 310
Message Digest (MD) algorithms, 129
Microsoft Bing Maps, 308
mock objects, 196
modeling spatial data, 283–291
modules, code
defined, 110
privilege escalation and, 110
purpose, 5
MONTH function, 336
MultiLineString geometry, 284
multiple databases
consolidating, 107
need for, 112
MultiPoint instance, 294
MultiPolygon geometry, 284
multirow inserts
causing cycles, 405
in hierarchies, 409
multivalue concurrency control (MVCC)
advantages of, 266–268
data retention, 243
drawbacks of, 281
example uses for, 237
overview, 237, 266
querying, 269
uses for, 269

N
national grid coordinates, 292
nearest-neighbor queries, 304–308

...
NET interoperability, 160–161

...
Security namespace, 150

INDEX


...
Text
...
Regex
class, 40
nodes, 373–374
nonrepeatable reading, 236
normalization, 12–13
NULL values, 37, 160
Number_Of_Rows_Output stored procedure,
318
numbers table, 307
NUnit unit testing framework, 53–54

O
Object Explorer pane, 299
object-oriented systems
accessing information, 13–14
design goals, 8
integrating with databases, 8, 12–13, 17–
18
Object-Relational Mappers (ORM), 17
objects
creating, 105
owners, 105–106
referencing, 105
offset transactions, 367
OGC (Open Geospatial Consortium), 293
one-way encryption, 129
onion model of security, 104–105
Open Geospatial Consortium (OGC), 293
OPEN SYMMETRIC KEY DECRYPTION BY
PASSWORD syntax, 131
optimistic concurrency control
drawbacks of, 281
example environment, 260
implementation, 260–263
overview, 237, 242, 260
SNAPSHOP isolation level support, 242
updateable cursors, 243
OPTIMISTIC isolation options, 243
optimizing grid, 315
optional parameters
dynamic SQL, 212–218, 220
static in stored procedures, 208–213
static T-SQL, 206–208, 213
ORDER BY query, 303, 381
Orders table, 29–30
ORIGINAL_LOGIN function, 110
ORM (Object-Relational Mappers), 17
ostress tool, 272, 280
OUTPUT keyword, 229

output parameters, 229
overwriting data, 236
ownership chaining, 106, 110–114

P
parallels, 310
parameterization, query
...
See also optional parameters
ad hoc SQL, 197
balancing with security, 139–144
dynamic SQL
overview, 217
versus static SQL, 198
using EXECUTE, 226–229
using sp_executesql, 227–229
geography versus geometry, 294–296
parameterization/caching benefits, 203–
205
software, 19–20
SQL Server service, effect of restarting,
229
static SQL stored procedure, 225, 228–229
performance testing
additional information, 68
counters, 61–62
Data Collector, 65, 67
DMVs, 62–63
extended events, 64–65
granular analysis, 69, 71
identifying problems, 71
importance, 57–58
process, 68
profiling server activity, 59–61
running, 69–71
period-based data, 322
permission sets, assembly code
enforcement, 163
raising, effects of, 168
setting, 163
types, 163

427

INDEX

permissions system
database-level, 114–116
order of use, 104
server-level, 103
system-level, 117–119
persistent materialized paths
compared to CTE method, 407
deleting nodes, 411
drawbacks of, 410
indexing, 406
moving subtrees, 410–411
navigating up hierarchy, 407–408
overview, 405
uses for, 405–407
pessimistic concurrency control
...
Disk Queue Length counter,
61
PhysicalDisk:Disk Read Bytes/sec counter,
62
planar calculations, 294
Point( ) method, 302
points, 284, 312
polygons, 284, 295, 312
polymorphism, 13
primary filter, 313, 315
Primary_Filter_Efficiency stored procedure,
318
prime meridian, 286, 341
principle of least privilege, 102–103
PRINT statement, 27
private key, 134
privilege, resource
escalation, 102
goals, 102–103
in non-Windows systems, 102
in Windows-based systems, 102
privilege escalation
...
See also
performance
QUOTENAME function, 232

R
RAISERROR function, 81, 83–85, 90
Range column, 308
range searches, 157
READ COMMITTED isolation level, 238–239,
241–242
READ COMMITTED SNAPSHOT isolation level,
238
READ UNCOMMITTED isolation level, 238,
241, 243
Reads column, 59
rectangular search area, 312
Reduce( ) method, 303
refactoring, 2
reference ellipsoid, 288
reference frame, 288
reference identifiers, 290–291
RegExMatch function, 40
regression bugs, 55
regression suite, 55
regression testing, 55, 57
Remote Procedure Calls (RPCs), 202

INDEX

removing keys from automatic encryption
hierarchy, 126–127
REPEATABLE READ isolation level, 238–242
Resource Governor, 269, 271–272, 274–277,
279–281
rethrowing SQL Server exceptions, 90–91
retry loops, 91–92
reuse, code, 161
REVERT command, 109
ring orientation, 295
Rivest, Shamir, and Adleman (RSA) algorithm,
124
root nodes, 372–373
rotating symmetric keys, 126
routing systems, 387
row versioning technique, 62
row-level security, 104
rowversion column, 37
ROWVERSION type, 260–262
RPC:Completed events, 59
RPCs (Remote Procedure Calls), 202
RSA (Rivest, Shamir, and Adleman) algorithm,
124

S
SAFE permission set, 43, 163
salt value, 130
Scan:Started events, 60
schemas
advantages of, 107
applying permissions to objects within,
106
creating, 105
creating objects within, 105
described, 105
security features, 105
specifying owner, 105–106
scope resolution, 75
secondary filter, 313
Secure Hash Algorithm (SHA) algorithms, 129
security
...
See intervals, half-open
sensitive data, 122
SERIALIZABLE isolation level, 238–242
server activity collection set, 65, 67
server farms, 9
server-level principals
...
See SQLCLR
SQL Server Books Online, 303
SQL Server Integration Services (SSIS), 296, 302
SQL Server Profiler, 60, 62, 360
SQL Server:Workload Group Stats heading, 272
SQL:BatchCompleted events, 59
SQLCLR (SQL common language runtime)
...
NET interoperability, 160–161
performance of, versus T-SQL, 170, 178–184
process, 93–96
resources on, 159
security/reliability features, 163–170, 172–
175
serialization example, 186, 188–193
string-formatting capabilities, 326
as T-SQL replacement, 159
uses for, 159
SqlDataReader class, 189
SQLQueryStress performance testing tool, 228
SQLServer:Buffer Manager:Page life expectancy
counter, 62
SQLServer:Cache Hit Ratio counter, 62
SQLServer:Cached Pages counter, 62
SQLServer:Locks:Average Wait Time (ms)
counter, 62
SqlServer
...
dll library, 298
SqlTypes
...
IndexOf( ) method, 183–184
strong naming, 177
STTouches( ) method, 294
STxxxxFromWKB( ) method, 297
su command, UNIX, 102
subordinate nodes, 372
Subversion, 237
SUSER_NAME function, 109
SymKey1 key, 130
symmetric keys, 124, 126–127, 130–133, 140
sysadmin role, 134
sys
...
crypt_properties view, 116
sys
...
dm_db_index_operational_stats DMV, 63
sys
...
dm_db_index_usage_stats DMV, 63
sys
...
dm_exec_query_stats DMV, 63
sys
...
dm_os_performance_counters DMV, 43, 63
sys
...
dm_os_waiting_tasks DMV, 63
sys
...
spatial_reference_systems system table, 292
sys
...
Data
...
NET namespace, 160–
161
System
...
See also dates; intervals; times
categories of, 322
data types, 322
durations, 365–366
importance, 321
querying, 269
time zone issues, 341–344, 346
tessellated, 313
Test-Driven Development (TDD) methodology,
23, 196
testing
performance
additional information, 68
counters, 61–62
Data Collector, 65, 67
DMVs, 62–63
extended events, 64–65
granular analysis, 69, 71
identifying problems, 71
importance, 57–58
process, 68
profiling server activity, 59–61
running, 69–71
software
benefits, 58
best practices, 36–39
databases, shortage of, 49
reasons for, 56
stored procedures and, 196
techniques for, 55–56
testability, 20–21
timing, 55
types of, 49–52, 55–58
volume of tests needed, 57–58
thumbprint, certificate
...
See temporal data
times
...
)
default, 324
input formats, 323–324
querying, 326–329, 337–341, 362, 364–365
ToString method, 326
touching, 294
tracing SQL Server exceptions, 85
transactional locks, 250
transactions
doomed, 100
exceptions and, 96–99
rolling back, 96–100
stored procedures and, 97–98
transparent data encryption, 136–139
trees
defined, 372
overview, 371
triggers, 266
Triple DES, 130
trustworthy databases
marking, 175
security ramifications, 175
turning off, 175
try block, defined, 87
try/catch exception handling, 87–92, 100
T-SQL, versus SQLCLR, 178–184
T-SQL function, 40, 137
T-SQL stored procedures, 6
TSQLUnit unit testing framework, 52
two-part naming, 105

U
UDFs (user-defined functions), 191
undirected edges, 371
undirected graphs, 372
unexpected behavior, 45
UNION ALL query, 303
UNION query, 36, 303
unit testing
advantages of, 54
limits of, 51
uses of, 50, 55
unit testing frameworks
advantages of, 52
debug assertions, 52
tips for using, 52–54
variety, 52
Universal Transverse Mercator (UTM) grid
coordinates, 290, 292

432

Universel Temps Coordonné (UTC), 341–344
UNSAFE permission set, 163
UPDATE statement, 152
UPDATE trigger, 31
updateable cursors, 243
updates, causing cycles, 405
US geographical data, accessing, 387
User Error Message events, 85
user interface data, 12
USER_NAME function, 109
user-defined functions (UDFs), 191
users
creating, 103–104
defined, 103
impersonating, 102–104, 107–110
proxy, 114
use of, 103–104
UTC (Universel Temps Coordonné), 341–344
UTM (Universal Transverse Mercator) grid
coordinates, 290, 292

V
valid time component, 322
validating input, 40–42
varchar datatype, 34
varchar type, 34
Visual SourceSafe, 237
Visual Studio Team System 2008, 69

W
WAITFOR command, 252
warnings, 79, 85
web services, 18
Well-Known Binary (WKB) format, 296–297
Well-Known Text (WKT) format, 290, 296–297
white box testing, 49–50, 52
wildcard searches with HMAC substrings, 153–
157
WindowsIdentity class, 102
WITH TIES argument, 308
WKB (Well-Known Binary) format, 296–297
WKT (Well-Known Text) format, 290, 296–297
wrapper methods/classes
advantages of, 161, 163
example of, 162–163
uses for, 161, 163
Writes column, 59

INDEX

X
x coordinate, 312
XACT_ABORT setting, 77–78, 98–99
XACT_STATE function, 100
XandY table, 37, 39
XML format documents, 185, 263

XML serialization, 185–186
XP (extreme programming), 23, 55

Y
y coordinate, 312

433



---

Title: APress Expert SQL Server 2008 Development
Description: APress Expert SQL Server 2008 Development

Buy These NotesPreview