The question outlined in the title can be interpreted in many ways. The typical things in my head are the following:

• Programming: set-based, declarative approach vs sequential, procedural programming
• Operations: RDBMS is quite different from OS and other standard apps (including server products)

Going through the items: a typical problem with application development is that most developers just can’t digest that SQL is not procedural language. It hates when you tell it how to do something. And it will punish you with amazing performance hits. Believe it or not but I really got T-SQL code like this:

DECLARE @tranid bigint, @processor int
DECLARE tranitem CURSOR FOR
SELECT transaction_id, processor
FROM transaction_history;
OPEN tranitem;
FETCH NEXT FROM tranitem INTO @tranid, @processor;
WHILE @@FETCH_STATUS = 0
   BEGIN
      IF @processor BETWEEN 31 AND 44
         UPDATE transaction_history SET processed = 1 WHERE transaction_id = @tranid;
      FETCH NEXT FROM tranitem INTO @tranid, @processor;
   END;
CLOSE tranitem;
DEALLOCATE tranitem;

Needless to say, transaction_history had 300 million lines and the column processor had an index on it. With the supplied script, we couldn’t wait for the completion. With the rewritten one, all the few thousand matching rows were updated in less than a minute. Oh, yes, this is what I wrote:

UPDATE transaction_history SET processed = 1 WHERE processor BETWEEN 31 AND 44

So a DBA is someone who can write effective T-SQL code? Sounds like a database developer… :)

Looking at the second item: why is a relational database engine so special? Well, because I like them :) Seriously, they should use a different approach in some cases than an OS or a webserver. Most obvious part is the scheduling I think. OSes usually use preemptive scheduler, that is, every process gets a time slice to run, then they should wait in the queue. With SQL Server that would be disastrous as the SQL processes would wait in the queue holding locks and it would bring down server performance. Instead, SQL Server goes for a cooperative scheduler, trying to give enough room for a process to finish completely as quick as possible. In general, SQL Server thrills to minimize lock holding time. And as a DBA, you should be good at understanding the locking mechanism, lock compatibilites and understand how online exactly is online indexing.

Also, if you’re a DBA, you should know that SQL Server is optimistic. It expects that if a transaction is started, it’ll be committed and in 99.99% of the cases, it’s right. In the remaining 0.01%, rollback is a bit more expensive than it could be if someone had optimized for it. Needles to say, you won’t restart your server if you abort a batch running for two hours and the rollback is not done after 30/60/90/120 minutes. Do you? You would wait a bit more to get back your database…

So far, the DBA is a guy who can write well-performing T-SQL code and understand the SQL Server architecture.

Let me add one more thing: they should be good at operational thinking as well – this differentiate a DBA from a developer and a Windows admin. Operational mindset is that you’re sensitive to potential pitfalls and you know how to avoid them. So if you see the T-SQL code, your only question is not if it’s good as it looks but you also ask if it’ll run fine in 25 instances in parallel. You know that things shoudl be tested first and test environment should be as mush similar to production as possible. (You know you should have a test environment, do you?) And you know when you can/should make bold moves, going with something untested.

Why are DBA skills necessary? Because otherwise you screw up your database server. Either by bringing it down and/or wasting its capacity. I’m no angel here. I was an involuntary DBA a few years ago and there was an issue with the backup partition in one of our clusters. To suspend all tranlog backups, I decided to stop the SQL Agent instead of disabling all the backup jops one by one. Smart idea, you may say. So I said SQL Agent service stop in Computer management and started to work. A few unexpected things happened in the next few minutes (e.g. I saw new backups appearing on the drive), so I checked the cluster and found that the agent started magically. Then I found that the whole cluster failed over. Later I realized that the cluster service detected that the agent was dead and did a failover. I should have used the cluster administrator or cluster.exe for this task. Since then, I’m very well aware of it.

One interesting thing is that people has a very dishonesting picture about SQL Server and what I see is that it partially comes from the fact that if you picked a SQL 7/2000 installer without any idea about all the things I wrote, you could still install and use it (and got surpised on the big transaction log). And these guys ran into serious problems – not because SQL Server was crap but because they had no idea about how it works. On the other side of the wall, if you had no experience with Oracle, no way on earth you could install one. That was a separate art. This way if you had an up and running Oracle server, there must have been someone with at least average skills around. (For the record: I’m not challenging the fact that Oracle had a better database engine at that time.)

Msg 18456, Level 14, State 1, Server DEMOSQL1, Line 1
Login failed for user ‘tygger’

Actually, something better gets in to the log, but first, let’s analyze the first line, that is ,the first three numbers of the error. the first one is the error number, the identifier of the error; the second is the severity, that is, how bad it is. The bigger the number the worse it is. If you see something above 19, you might be in real trouble. The third number is the state, which is an interesting species. This can be used to provide diagnostic information, like throwing an error with state 1 from an SP and with state 2 from parameterized query. It can make DBAs (and developers) life easier. Now back to that errorlog!

2009-07-27 14:02:02.21 Logon     Error: 18456, Severity: 14, State: 8.
2009-07-27 14:02:02.21 Logon     Login failed for user 'tygger'. [CLIENT: 10.10.10.1]

Feel the difference: that state value is obviously different from what the client saw – and this is intentional. SQL Server won’t disclose any info which may help a malicious person during an attack. But if there’s a real need for help, you can go to the sysadmins and ask them to check the log what was the reason of the failure. They can check for the status values in the errorlog. And here’s the resolution:

ERROR	DESCRIPTION
2, 5	Invalid username
6	SQL authentication was attempted with Windows login name
7	Disabled login
8	Bad password
9	Invalid password
11, 12	correct login but no server access
13	SQL Server service paused
16	Correct login but no access to the selected (or default) database
18	User must change password
23	Server is shutting down, only sysadmins can log in
27	Correct login but server cannot determine an initial database
38	[2008] Opening explicitly specified database failed (16 in SQL 2005)
40	[2008] Opening login default database failed (16 in SQL 2005)

Some of these are not obvious (some of them I have no idea about:), so here’s some help:

State 16 in SQL 2005, 38/40 in 2008 is caused most commonly by auto close databases. This is a great feature – for MSDE or Express but not for production. So I recommend you checking if you have any database with auto close enabled and if you have, turn this off – you have just problems with it. Other than this login issue (which will disappear if you retry as the database will be reopened in a few seconds, so a subsequent user can use it immediately) it pollutes your precious errorlog with messages like this: “Starting up database XYZ” .

State 11/12 are typical Windows login issues: SQL Server knows who tries to log in, authentication is successful, but there’s no SQL login for the Windows login, authorization fails.

This post was written based on the SQL Protocol team blog post and the discussion after it, as it was pretty embarrassing that there’s not a single consolidated list of errors, but many of them should be collected from the comments – so I put the list together.

However, it it lesser known that if you start the SQL Server in a single user, minimal mode (sqlservr -m -c started from the directory of the program directory, you can see it at the installed service), the local admins become sysadmins, independently of their normal privileges. This can be a failsafe solution, for example in an over-hardened environment where someone accidentally lost the password of the renamed and disabled sa account, which was the only sysadmin. You can reset passwords, add new sysadmins or whatever you want.

Many people faced the upleasant message below while trying to connect to SQL Server with SQL authentication: “Login failed for user ‘(null)’. Reason: Not associated with a trusted SQL Server connection” The most typical reason for this is that the server is configured to allow only Windows logins, not SQL logins. The cure is pretty simple, open up server properties and change the authentication option from Windows only to SQL and Windows – and there you go.

Recently, I met the error message in a different situation. A few guys couldn’t connect to a server with Windows integrated authentication. They got the error message above. Others could connect without any issue. I tried to figure out what was going on: my first candidate was MTU size but as the authentication token size was the same size (actually, it was bigger for those who had no problem), I ruled this out. Then I checked if they could connect to a file share on the machine. This action failed and I found this in the security eventlog:

Event Type: Failure Audit
Event Source: Security
Event Category: Logon/Logoff
Event ID: 534
Date: 8/3/2008
Time: 3:04:30 PM
User: NT AUTHORITY\SYSTEM
Computer: SQL10
Description:
Logon Failure:
Reason: The user has not been granted the requested
logon type at this machine
User Name: erik
Domain: MYDOMAIN
Logon Type: 3
Logon Process: NtLmSsp
Authentication Package: NTLM
Workstation Name: ERIKPC

So what the heck does this mean? I asked my old pal Google, who said he read about it once, and people say it’s nothing else than network logon failure. So I decided to check the local security policy. And I found that in the User Rights Assignment node the Access this computer from the network privilege was granted only the local Administrators group. At that point I banged my head into the wall why I hadn’t found the pattern sooner and also extended this privilege to the users group as well. After a policy refresh, it worked.

Lookink back, it was pretty simple. As the peons users were not authorized to access the computer via the network, the OS tore and dropped their token so the SQL Server didn’t get anything from their identity. Once again I learnt something new.

select * from sysprocesses
where
spid in (select blocked from sysprocesses where spid <> blocked)
and dbid = db_id('MyDatabase')

(The where spid <> blocked clause was needed because from SQL 2000 SP4, the “self-blocking” is also listed as a blocking but it wasn’t interesting in this case.) I expected to spot one of our report writers running a bloated query with massive joins ans sophisticated filters but it wasn’t the case. I found nothing. I had a suspicion that if I restarted the server, this phenomenon would disappear but as the server was still in a usable state, I decided to dig deeper cause I hate mysteries unresolved.

I started to check the distribution agent history in Replication Monitor but the only thing I found was that the agents were very slow. I checked them in the Activity Monitor and noticed that they were spending lot of time in RESOURCE_SEMAPHORE wait state. I checked the excellent white paper about SQL Server wait types, and saw that it usually means that due to high concurrent load, the query should wait for memory grant. So I checked the memory usage a bit.

In SQL 2000, I used DBCC MEMUSAGE, but it disappeared in SQL 2005. Good news is that there’s something even better. While DBCC MEMUSAGE told you the to 30 memory eater object, the sys.dm_os_buffer_descriptors DMV will tell you this info for every object you want. I picked the sample scripts below from Books Online at sys.dm_os_buffer_descriptors and queried both the server and the database level memory usage.

-- server level
SELECT count(*)AS cached_pages_count
    ,CASE database_id
        WHEN 32767 THEN 'ResourceDb'
        ELSE db_name(database_id)
        END AS Database_name
FROM sys.dm_os_buffer_descriptors
GROUP BY db_name(database_id) ,database_id
ORDER BY cached_pages_count DESC

-- and database level
SELECT count(*)AS cached_pages_count
    ,name ,index_id
FROM sys.dm_os_buffer_descriptors AS bd
    INNER JOIN
    (
        SELECT object_name(object_id) AS name
            ,index_id ,allocation_unit_id
        FROM sys.allocation_units AS au
            INNER JOIN sys.partitions AS p
                ON au.container_id = p.hobt_id
                    AND (au.type = 1 OR au.type = 3)
        UNION ALL
        SELECT object_name(object_id) AS name
            ,index_id, allocation_unit_id
        FROM sys.allocation_units AS au
            INNER JOIN sys.partitions AS p
                ON au.container_id = p.hobt_id
                    AND au.type = 2
    ) AS obj
        ON bd.allocation_unit_id = obj.allocation_unit_id
WHERE database_id = db_id()
GROUP BY name, index_id
ORDER BY cached_pages_count DESC

I got back that the 95% of the total server memory is used by my troublesome database and inside that, 95% (approx. 140K pages) is used by the table MSreplication_subscriptions. That surprised me as I was under the impression that that is a small table. I decided to query the table with some stats:

set statistics io on
select * from MSreplication_subscriptions

And got this:

(13 row(s) affected)
Table 'MSreplication_subscriptions'. Scan count 1, logical reads 142543, physical reads 1234, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

Shocking… 13 rows, 142000 reads and no LOB in the table… it wasn’t a perfect day for a broken database… I ran a DBCC on the table:

DBCC CHECKTABLE ('MSreplication_subscriptions')

It returned this info, 5 hours and 43 minutes later:

DBCC results for 'MSreplication_subscriptions'.
There are 13 rows in 147460 pages for object 'MSreplication_subscriptions'.
DBCC execution completed. If DBCC printed error messages, contact your system administrator.

I started to get confused… I thought something was broken, but DBCC didn’t scream and I couldn’t imagine how it could go so wrong that even DBCC failed to spot the error. So I decided to allow myself a break and left the computer. Needless to say, my brain couldn’t stop so in two minutes it came into my mind that I didn’t checked the physical table structure. I went back to query the index stats because the clustered index contains the records themselves on the leaf level, so querying the clustered index stats provides info about the table itself. For a moment I was thinking if the table just got heavily fragmented, but it was a silly idea, given that the page numbers significantly outnumbered the record numbers (and still no LOB). So I queried the index details:

SELECT * FROM
sys.dm_db_index_physical_stats(db_id(), object_id('dbo.MSreplication_subscriptions'),
null, null, 'detailed')

As you can see, it wasn’t even a DMV, but a DMF, that is, a dynamic management function. It returned everything I needed, and even more:

database_id	object_id	index_id	partition_number	index_type_desc	alloc_unit_type_desc	index_depth	index_level	avg_fragmentation_in_percent	fragment_count	avg_fragment_size_in_pages	page_count	avg_page_space_used_in_percent	record_count	ghost_record_count	version_ghost_record_count	min_record_size_in_bytes	max_record_size_in_bytes	avg_record_size_in_bytes	forwarded_record_count
6	2073058421	1	1	CLUSTERED INDEX	IN_ROW_DATA	5	0	98.1227621828284	147942	1.00244014546241	148303	47.3060785767235	13	1	1608002	314	374	347.846	NULL
6	2073058421	1	1	CLUSTERED INDEX	IN_ROW_DATA	5	1	99.7432605905006	3895	1	3895	48.5910180380529	148303	0	0	11	118	101.346	NULL
6	2073058421	1	1	CLUSTERED INDEX	IN_ROW_DATA	5	2	99	100	1	100	50.2309117865085	3895	0	0	11	118	102.433	NULL
6	2073058421	1	1	CLUSTERED INDEX	IN_ROW_DATA	5	3	0	2	1	2	64.6466518408698	100	0	0	11	118	102.69	NULL
6	2073058421	1	1	CLUSTERED INDEX	IN_ROW_DATA	5	4	0	1	1	1	1.42080553496417	2	0	0	11	102	102.119	NULL

At first sight, it’s a mess but it has a logic. Every row in the result set corresponds to a level of an index on a table (recall, indexes are in B-tree structure in SQL Server). We have a single, clustered index here, with 5 levels (see the idex_depth and index_level columns). The page_count cloumn shows the pages used by the level and the record_count returns the number of records. On level 0, that is, on the leaf level, we have 13 records but 148303 pages (table was growing all the time). For the solution, let’s check the two additional alternative record count columns.

The ghost_record_count tells you how many records are in the pages physically which are already not there logically due to deletion (or update which requires more space). Such records are deleted by the GHOST_CLEANUP system process, so it might happen that we can’t see a value other than 0 in the column if the cleanup was wicked fast. The other column called version_ghost_record_count and counts those records which are logically not in the table but cannot be deleted yet due to an open transaction using row versioning (running on snapshot isolation level) and the original row has been changed. In our case, we have a single ghost record and 1608002 version ghost. It’s pretty tough for a table with 13 rows, even if 113 distribution agents belong to those 13 rows and they run continuously, updating at least once in a minute. So I checked for the oldest transaction:

DBCC OPENTRAN

Transaction information for database 'MyDatabase'.

Oldest active transaction:
    SPID (server process ID): 74
    UID (user ID) : -1
    Name          : user_transaction
    LSN           : (21:201:1)
    Start time    : May  13 2009  1:48:22:733PM
    SID           : 0x010500000000000515000000119d203ab2692ec2891a29a8ed030000
DBCC execution completed. If DBCC printed error messages, contact your system administrator.

And all of it happened on 28th May. I checked the process and found that it was a distribution agent, ran by a DISTRIB.EXE OS process. Given that I restarted the SQL Agent a few hours before and there was no replication latency a day before, I was pretty sure that it was a zombie agent since 13th May.

I killed the processz and the ghost_record_count went up to 1.5M, then an hour later when I checked back, the whole table was on 8 pages. Happy end.

Although it was a bit long post, I hope someone will benefit from this demonstration of the pretty huge armory of a DBA.