What are sendmail debug switches and how can they be used to troubleshoot deferred messages?

The problem: You need to troubleshoot low-level details concerning a deferred message in sendmail but don't want to necessarily increase global log levels (or restart services).

The solution: Use sendmail debug flags/switches. This blog post describes how.

First, some basics... the sendmail command can be used to resend queued messages (among other things) and includes the ability to generate debug logs/output about those events. The beginning syntax for the command is:

sendmail -D <path/file to write debug output to> -d<debug category digit(s)>.<log level detail for the category> <other sendmail switches/options>

• -D tells sendmail that you want to write the log info to a separate file. If this is missing, sendmail will write to stdout but it might not be visible on screen. You can substitute -v to ensure that output shows on-screen.
• -d is by far the most important part. It tells sendmail what specific debug category you want to log. The idea here is that you can generate extra debug info for a very specific part of the delivery process (e.g. DNS queries, current load averages, etc). Each category had an id number and a level of detail.

So what ARE the debug categories and their respective numbers? That's constantly changing depending on the version of SendMail you're using. If you want to know the specifics for your version, you would like need to examine the code (using sendmail/TRACEFLAGS as a reference - yuck).

The good news is that there are a lot of well known categories that don't change much. Here's the list (as per the famous "Bat Book" from O'Reilly):

-d0 Display system configuration information.
-d1 Show sender information.
-d2 Tracesendmail’s exit information.
-d3 Print the load average.
-d4 Trace disk-space calculations.
-d5 Trace timed events.
-d6 Show failed mail.
-d7 Trace the queue filename.
-d8 Trace hostname canonicalization.
-d9 Traceidentdexchanges.
-d10 Trace recipient delivery.
-d11 Trace delivery generally.
-d12 Trace mapping of relative host.
-d13 Trace the envelope and envelope splitting.
-d14 Show header field commas.
-d15 Trace incoming connections.
-d16 Trace outgoing connections.
-d17 Trace MX record lookups
-d18 Trace SMTP replies.
-d19 Show ESMTP MAIL and RCPT parameters.
-d20 Show delivery agent selection.
-d21 Trace rules and rule sets.
-d22 Show address tokenization.
-d23 Unused.
-d24 Trace assembly of address tokens.
-d25 Trace the send-to list.
-d26 Trace recipient queueing.
-d27 Trace aliasing,~/.forwardfile handling, and controlling user.
-d28 Trace the User Database.
-d29 Tracelocaladdrrule set rewrite of local recipient.
-d30 Trace header processing.
-d31 Trace header validation.
-d32 Show collected headers.
-d33 Watchcrackaddr().
-d34 Trace header generation and skipping.
-d35 Trace macro definition and expansion.
-d36 Trace the internal symbol table.
-d37 Trace setting of options and classes.
-d38 Trace database processing.
-d39 Displaydigitdatabase mapping.
-d40 Trace processing of the queue.
-d41 Trace queue ordering.
-d42 Trace connection caching.
-d43 Trace MIME conversions.
-d44 Tracesafefile().
-d45 Trace envelope sender.
-d46 Showxffile’s descriptors.
-d47 Trace effective/real user/group IDs.
-d48 Trace calls to thecheck_rule sets.
-d49 Tracecheckcompat().
-d50 Trace envelope dropping.
-d51 Trace unlocking and prevent unlink ofxffile.
-d52 Trace controlling TTY.
-d53 Tracexclose()
-d54 Show error return and output message.
-d55 Trace file locking.
-d56 Trace persistent host status.
-d57 Monitorvsnprintf( ) overflows.
-d58 Trace buffered filesystem I/O.
-d59 Trace XLA fromcontrib.
-d60 Trace database map lookups insiderewrite().
-d61 Tracegethostbyname().
-d62 Log file descriptors before and after all deliveries.
-d63 Trace queue processing forks.
-d64 Trace Milter interactions.
-d65 Trace nonallowed user actions.
-d66 Unused.
-d67 Unused.
-d68 Unused.
-d69 Queue scheduling.
-d70 Queue quarantining.
-d71 Milter quarantine on errors.
-d72 Unused.
-d73 Queue shared memory updates.
-d74 Unused.
-d75 Unused.
-d76 Unused.
-d77 Unused.
-d78 Unused.
-d79 Unused.
-d80 TraceContent-Length:header (Sun version).
-d81 Trace > option for remote mode (Sun version).
-d82 Unused.
-d83 Collection timeout.
-d84 Delivery timeout.
-d85 The internaldprintfdatabase map.
-d86 Unused.
-d87 Unused.
-d88 Unused.
-d89 Unused.
-d90 Unused.
-d91 Log caching and uncaching connections.
-d92 Unused.
-d93 Unused.
-d94 Force RSET failure.
-d95 Trace AUTH= authentication.
-d96 AllowSSL_CTX_set_info_callback( ) call.
-d97 Trace setting of auto mode for I/O.
-d98 Trace timers (commented out in the code)
-d99 Prevent backgrounding the daemon.

OK, cool. So now we know which debug flags to use but how do we reprocess a queued message? That can be done with the -qI, -qR, and -qS switch.

• -qI lets you specify the QID pattern of the messages you want to resend.
• -qR lets you specify the recipient address pattern of the message(s) you want to retry.
• -qS lets you specify the sender address pattern of the message(s) you want to retry.

Thorougly confused yet? Alright! Let's go through some examples:

Example 1: You want to retry any queued messages going to yahoo.com recipients and want to display MX lookup information for those transactions. You want the output to be displayed on-screen.

Command: `sendmail -v -d17.9 -qR@recipientfoo.com`

The -d17 is the mx lookup flag and the .9 indicates that I want details consistent with log level 9.

Example 2: You want to retry any queued messages from bob@gmail.com and want to display system configuration info and sender info. You want the output to be saved to a file in /root:

Command: `sendmail -D /root/foo -d0-1.9 -qSbob@gmail.com`

The -D /root/foo specifies the destination for the error details. The -d0-1.9 tells sendmail that I want to include BOTH -d0 and -d1 error logs... both with a log level detail of 9.

Awesome stuff!